War stories: Logs are where the dead things dwell

Over the years there has been one love hate relationship that I could never truly get away from entirely. That was logging on systems and anything else that had something to say. I got so silly that at one point when I was doing work for a DoD customer I had a monitor on my desk that was simply tailing the perimeter router logs. I had gone full matrix and no, I never once thought I was Neo.

One company that I did work for in the past had a syslog server that was purported to be collecting logs from production systems. This was an environment where there was so much work to do that I relegated the syslog system to the back burner. I didn’t like logging systems. I didn’t want to have anything to do with them. I knew in my heart of hearts that this was a necessary aspect of the job but, it ranked right up there with a home lobotomy kit.

Finally I realized that I needed to address the problem when I found a system that was in a failed state and no one seemed to know what it did at all. I looked at the system logs and it had been down for a week already when I noticed it. It was reporting into the syslog system and there were no monitoring alerts for the operations team to act on. That was a curious development.

I picked through the logs again and all of the messages that I could find were of minimal use. Case in point, failed logins were being recorded but, little else. The sound of my teeth grinding registered on a seismic meter some place I’m certain.

At this point I decided that it was far easier to ask forgiveness than to plead my case. Especially taking into consideration that no one seemed to know who was responsible for the system in the first place.

So, I pulled the network cable.

I tagged a note to the front of the server in an envelope and walked back to my office. I waited for the storm to come. But, nothing. No calls, no raging developers, admins…not even crickets.

I got back to working on transitioning the logs from other systems to report in to a log aggregator from a company that has long since past into the mists of time. The transition went well. It slurped up the logs that were previously being sent to the syslog system without an issue. I just took over the IP address and presto, all of the systems started sending in their messages.

Then I recalled the errant system. It had been months at this point since I pulled the cable. Not two days later an irate application team lead stormed into my office with his hair on fire. “This is a level one severity!” he bellowed. “We need all hands on deck to deal with this system. It’s not accessible.”

I smiled. I said, “Come with me.” We walked down to the server floor. The crisp air greeted us as we opened the door. I could barely contain my smile. We walked to the back corner of the floor and I pointed to the system. “Is this the server you are talking about?”

“Yes!” he almost shrieked. “Why is it unplugged?! This is a production system.” I pointed to the note. He looked confused as he reached for the envelope. He opened it and read the note and then he paused. His hands shook a little and he cleared his throat. “I owe you an apology. Obviously this system needs to be removed from the racked and wiped for another project. This project is officially done.”

The note simply had the date I took it offline…nine months earlier.