War stories: just shut off telnet

Real-world devops failures -- and how to avoid them
Credit: Gadini via Pixabay

Years ago I was working on a project that had a rather interesting premise. It was a way to send a file between two parties that was stamped as verified by a third party intermediary. Pretty basic stuff but, in the 90s it was rather neat. One of the things that I discovered was that I could issue junk commands to the application simply by launching a telnet client and connecting to the “encrypted” listening port.

Yeah, that was how the developers described it. I smiled. I was able to get the application to answer various queries that, by the documentation, should have only been possible using the client application that was purpose built for said task. The client and server were supposed to have some manner of key exchange but, it did not work as advertised.

Lucky for the team, this had not gone into production yet. Although there was a large paying customer that was expecting delivery. At this point, the threats from different team leads started being fired around the office email. This is rather typical of some IT shops when something goes wrong. Rather than pull together and fix the problem there is a knee jerk response to make threats. This fails in the long term version of things.

Bad blood always finds it’s way back to the surface.

Now, the pièce de résistance in this case came when the Chief Technology Officer weighed in with his brilliant plan. I opened the email and read it. I sat back in my chair thinking that it was a joke and I looked at my co-worker, “He is kidding, right?” My co-worker shook his head and said, “I think he actually thinks that is a bonafide solution to this problem."

His email read, “Just turn off telnet so we can ship the product."

What? Turn off telnet clients on every machine on the Internet? I wrote him back seeking clarification and yes, he was actually serious. Recall that this was the Chief Technology Officer. Suffice as to say I had a sinking feeling like I had not experienced in a long time. This simply wasn’t feasible. No matter how many threatening phone calls and emails transpired over the course of the day it wasn’t changing the point that this wasn’t a fixable problem.

Sure, there would be firewall rules in place to ensure that connections were strictly controlled between client and application but, realistically this was going to be opened up to more clients in the future. A grim prospect that I didn’t look forward to at all. Thankfully, after some long conversations with the lead developer, he took it to heart that this needed to be fixed. The delivery was ultimately delayed but, in the end a better product was the end result.

This could have all been avoided had security been included in the outset of the project. Instead, this problem came to light because my co-worker and I said, “what happens if we do this, this and this?" after we discovered the project.

Years later I’m sitting here scribbling away and wonder whatever happened to that CTO. The product was eventually shelved and I moved on to a new job. I did learn a valuable lesson that security is far cheaper to include at the outset of a project than to try and route around in the end with delivery looming. Don't set yourself up for failure.

Cybersecurity market research: Top 15 statistics for 2017