Screaming Machines And Situational Awareness

I’m always amazed at the reaction I garner from folks when I’m talking about situational awareness and using the tools that people already have at their disposal.

Usually I’m asked the question, “What do you mean by that?” which causes me to pause. I ask them if they have routers, switches, servers and databases as an example. The answer is always yes. I ask them if they have logging enabled on those systems? 

Blank stare. 

“I think that we do” is the answer I get after they consider it for a moment. There is the problem. Too many organizations are not aware of what systems are on their network or what state those systems might be in. They “think” that the answer is yes. There is an absence of clarity. 

Imagine this scenario. You’ve hired a third party company of penetration testers to run an assessment against your Internet facing assets. In the initial meeting they ask what type of testing will be done. If you say white box they ask the questions such as “What are your IP ranges?”, “How many systems are to be tested?” and so on. It doesn’t bode well if you’re either not prepared to answer these questions or are unaware as to what those answers might be at all.

Situational awareness is something that needs to be receive better attention in any organization. Of course this is not an absolute. There are some companies that do an excellent job of this but, by no means are they they rule. You have a network that is absolutely littered with systems that are dying for someone to talk too. The problem is that these needy machines cry out for attention and in many instances most people just aren’t willing to listen. 

Jerks. Systems just want to be loved. Is that so wrong?

I have had the good fortune to have worked for numerous companies and have dealt with many clients over the years. Along the way I have encountered some defined repeatable foolishness.

Examples:

1) “The system ran out of disk space.” No one was monitoring the drive. 

2) “The application is blocked by the firewall.” The daemon had failed. Again, no monitoring.

3) “We have a class one outage. How do you know? The CEO can’t get his/her email.”

You get the idea.

If your early warning system is an executive screaming for someone’s head on a stick you’re behind the game from the outset. Wrap your head around the fact that IT systems are not designed to fail. The issue here is that they should be. Systems that are built and deployed should be done so with the expectation that they can and will fail. Having a mission critical SAP database with no backups or redundancy is a prime example that I like to harp on. The database fails and finance is hobbled causing hardship for the company. The need to be aware of what is happening is key.

Monitoring systems on your network is required if you’re to have any hope of keeping ahead of the issues. I worked at one company that had no less than seven monitoring solutions. They were paying annual maintenance for all of them. The problem was that only four of them were even remotely in a functional state and each one was doing something different. After a review it turned out that each system had been purchased for a specific feature. All seven were consolidated down to one system that did a great job, once it was configured. 

Short story, work on your centralized logging. Go beyond the tiresome “failed logins” and look into what these systems are trying to tell you. Ensure that your monitoring systems are actually doing what they are designed to do. A greater level of situational awareness will help to lessen the failure rate and improve resiliency. Defending the castle against attackers isn’t easy. A defender has to be right every time. A hacker only has to be right once. 

(Image used under CC from Official U.S. Navy Imagery)