Open Rsync server exposed EMS computers in Greensboro, NC

EMS computers in Guilford County, NC were exposed for an unknown length of time, because the server managing system updates was publicly available on the internet. The problem was discovered earlier this month by a researcher scanning the internet for Rsync servers.

Salted Hash was alerted to the server’s existence by MacKeeper’s Chris Vickery, a security researcher who is known for finding exposed systems and databases.

Vickery said he was scanning the internet for exposed Rsync servers when he discovered one being used by EMS systems in Greensboro, NC – or all of Guilford County, to be exact.

Shortly after discovering the server, Vickery contacted the Greensboro Police Department. When he brought the issue to our attention, Salted Hash alerted other county officials, including the director for Guilford County Emergency Services.

“At first I thought I had discovered something related to an enterprise email backup server,” Vickery told Salted Hash. In fact, these types of servers are what he normally finds when scanning.

Vickery downloaded the software on the server and examined the configuration scripts. In them, he discovered the administrator password (Lpdw223$), which could be used to access local EMS systems individually.

Additionally, Vickery said the server contained SunGard MCT software. This software runs on the computers that are inside of police, fire, and ambulance vehicles.

“The software I’ve downloaded contains mapping files for all of Guilford County, as well as all the images and sound notifications that can appear through this dispatch-based software. The installation configuration file sets up a Windows scheduled task that checks this Rsync server for updates at regular intervals and deletes previous files.” Vickery explained.

The concern was that a malicious actor could upload corrupted or blank files in place of the real ones, triggering system crashes that could have serious physical consequences to the 507,000 people who live in the county.

A Guilford County official told Salted Hash that the problem only impacted EMS, and it didn’t affect other public safety users. The statement also confirmed that the server Vickery discovered was used to update devices in the field.

“The server houses these updated files which are synced regularly to a local folder on our field devices. This is its only function. Users manually initiate the update process via a shortcut on their desktops when notified by us that an update exists,” the statement explained.

However, the county also said server logs show that no one outside of emergency services had accessed the server. Yet, at the very least, the logs should show Vickery accessing the server and downloading files. It isn’t clear why the county missed that, or why the server didn’t log Vickery’s actions.

Also, the statement overlooks Vickery’s entire point, which was that the individual systems were at risk, as they accessed the update server remotely. In fact, Vickery has a copy of the script that would prompt a user to update the device, so an attacker would have had no problems getting emergency personnel to update.

The county said that the files on the server were read-only, which Vickery couldn’t confirm as doing so would place him in legal hot water, but “the fact that there was no username or password necessary to access the server strongly hints that the files could have been replaced,” Vickery said.

Still, the county took the disclosure seriously. The local administrator passwords were changed, and the Rsync server itself is no longer publicly available.

“It may be convenient to bypass authentication requirements for rollout update servers, but that leaves you open to disastrous consequences (not to mention professional embarrassment). Never skimp on security,” Vickery said via email, when asked for his final thoughts on the incident.

But sometimes, skimping is exactly what happened. Often, the networks powering emergency services, as well as police and fire departments, are chaotic. They have outdated hardware and software, and they use servers such as the one Vickery discovered not because they’re lazy or incompetent, but because they’re what’s available.

Budget crunch is a real problem for many IT teams working in emergency services. In fact, the term team is being generous; many counties operate with just a single IT person doing the job of six people.

In those situations, the county will make do using whatever is affordable or gets the job done.

“You’ll see a lot of cities using AVG free, because they can’t afford anti-virus,” said Nick Selby, a Texas police detective and information security consultant.

“Anything’s that free, they’ll do, because they don’t have the budget. There’s another big problem, which is that a lot of things that are being used in public safety were invented somewhere else, for something else,” Selby added.

He recalled a situation that came to light a few years ago, where police video cameras were found to be vulnerable.

Those cameras were originally sold to the school bus industry, but later they were sold to police departments with no additional security or protection. As a result, police were using cameras that could be controlled remotely.

Around the same time he found the EMS server, Vickery also discovered an Rsync server that belonged to a law firm. The video footage on that server raised questions about the official report issued by the La Habra police department, after an inmate took his own life while in custody.