Security on-call nightmares

The sleep deprivation brought on by two children who refuse to rest begins to take its toll on me. I clutch at my cup of coffee in desperation and remember the good old days as an on-call staff member. Wait, no…that doesn’t sound quite right.

It wasn’t uncommon for me to get a phone call at 3 am from a help desk staffer obviously annoyed that they had been roused from his crossword puzzle and/or nap. 

There were several types of calls that I would get called in for that I could lump into three buckets. First off, the firewalls were down / broken / interfering / melancholy, you name it. These were more often than naught actual problems that needed to be attended to in short order. Only once in my memory was the firewall actually they problem. I had naively decided to use the SMTP (email daemon) on my CheckPoint firewalls to process email for the company. That was an absolute failure at the time. The support person said, “Please, don’t do that.”

Lesson learned.

The lion’s share of the incidents were due to application and network changes that were not properly documented. They would try to skirt change control by jamming in a last minute change that would be “just a process flow and no code changes” which would break horribly in untold ways and they implementers would throw their hands up in the air and say, “Not it! Must be the firewall.” There is a special place in hell for them. 

The second category was “I can’t access this site for research” which invariably had exactly nothing to do with their assigned duties or even anything remotely connected to it. One such incident went as follows,

Me: “I’m sorry, could you repeat that?”

Helpdesk: “The user says they can’t access a warez site to download a crack of $product. He said he has a deadline.” 

Me: “HAHAHAHAHAHAHAHAHAHAHAHAHA”

Helpdesk: “So what should I tell the user?”

Me: “Tell them no. That is a violation of company policy for starters.”

Helpdesk: “But, he is a director. Can you tell them?”

Me: “No. Good night.”

…at this point I actually hung up. That type of call happened far too often. 

The third and final act. The last on-call incident that I had to respond to was a whopper. I had been working for a company that had outsourced their first line support to a firm that was based in India. I often wondered what their hiring standard was and I realized quickly that they would check candidates for a pulse. They would hire the lot. Even the occasional one whose pulse could not be readily determined. 

The staff were given training.They were provided with processes to follow. They were given escalation trees in the event of a problem. I remember this call vividly. I was asleep on the sofa. my TV was complaining that the signal could not be found since the Apple TV at least had the sense to go to sleep. 

The phone rang, “Mr. Lewis David?” This was off to a good start. 

Me: “This is Dave.” 

Muffled whispers, “I was looking for a Mr. David.” I grind my teeth for a moment. “What can I do for you?”

Helpdesk: “OK. We have a user who has a complaint. Please do the needful and revert.”

Me: “Um, OK, this will go smoother if you could actually tell me what the problem is.” 

Helpdesk: “The problem is in ticket number….” I glazed over. 

Me: “Please, in the interest of expediency, please…please, just tell me what the issue is.”

Helpdesk: “Yes sir.” Another pause. “Sir, the Internet is broken.” 

./headasplode

Me: “Could I have a little more detail?”

Helpdesk: “That is what the user told us in an email.”

For the sake of decorum I will not share the rest of that conversation. Suffice as to say there was nothing actually wrong beyond a user that could not surf to his favorite porn site. 

I checked the logs.

So why share all of this? Simple. Companies and organizations need to have defined repeatable processes. There needs to be proper training on these processes to ensure that people know what their job is in an emergency situation or even in their day to day role. 

End users need to better understand that they are part of the solution and why, as an example, asking for access to bootleg software is frowned upon.

Helpdesk operators need to have a clear understanding of how and when to escalate a problem. They should not be the human equivalent of the ‘drive thru’ window in a restaurant. Some basic understanding should be leveraged here otherwise what are you spending money on this service for?

Security awareness is an essential piece of the puzzle. Empower your staff so that folks who carry the on-call albatross don’t ever need to get a phone call at 3 am again unless absolutely necessary.