War stories: the water shut off valve

Years ago I worked for a company that had some manner of connection to the goings on for the power grid. *cough*

It was a job that afforded me all sorts of different projects as security had previously been more of an afterthought that anything else of note. Intrusion detection systems that were racked and powered but, couldn’t catch a cold. A firewall that my mother could tunnel through and so forth. But, rather than whinge about it (to anyone other than my therapist) it was a great opportunity.

I had some great experiences working there and a few moments that caused me to question the fabric of reality as I perceived it. One such day my co-worker, let’s call him James, and I were wandering towards the door. It was the end of the day and we were clocking out. Or, so we thought.

As we made our way down the hall, we discovered one of the physical security staff and the head of Ops working feverishly with buckets on a problem in the janitor closet. The hot water valve had broken and was spraying water everywhere. We dropped out bags and got to work helping.

First question, “where is the water shut off valve?” No one knew the answer. No one knew where the shutoff for the water was in the entire building. That was somewhat alarming. We started a bucket brigade. But, that wasn’t going to be a sustainable solution as at some point we were going to run out of energy and the leak was getting progressively worse.

Phone calls were made and people tried to figure out how to shut the water off. Eventually, mercy did come in the form of a water shut off. We were drenched, tired and thankful that we were able to work together to solve the problem.

Why? On the floor just below this particular janitorial closet was the switch room for the entire network. Remember, this was a power related company. The cascade failure that could have happened in the event the water made it through the ceiling could have been horrendous.

When you are in information security there are times where you have to roll up you sleeves and get messy, literally. I recall one intern who refused to pull Cat 5 cable under a floor because that was beneath him. No pun intended. The best part was that the IT Director was under the floor at the time doing exactly that job at the moment.

In the case of a potential disaster like a janitorial closet flood it is important to jump in and help if it is safe to do so. You are part of a team. The moment you forget that things start to fall apart.

In addition to being part of a team, you should check around your office. Do you know what is on the floor immediately above your IT assets such as servers? Might seem like a simple question but, one worth checking. Be sure to do that AFTER you puzzle out where the water shutoff for the building is located. Just a friendly tip.