Ransomware, healthcare and incident response: Lessons from the Allscripts attack

On January 18, 2018, at around 2:00 a.m. EST, the security operations center (SOC) at electronic health record (EHR) and practice management software provider Allscripts detected abnormal activity.

Four hours later, at 6:00 a.m. EST, the SOC started their investigation and determined the abnormal activity was in fact a full-blown ransomware incident due to SamSam, a family of ransomware that is known to target healthcare organizations. A short time later, teams from Microsoft, Mandiant and Cisco were called in to help.

Getty Images

At this point, as is the case for any organization facing a large-scale incident, Allscripts said the situation turned into a “crisis event.” It was quickly determined that the Professional EHR (Pro EHR) and Electronic Prescriptions for Controlled Substances (EPCS) services were the hardest hit. Allscripts told Salted Hash in a statement that approximately 1,500 medical practices were impacted by the incident.

Incident response is something that most IT and security professionals are drilled on. It’s a process that sometimes requires a specific mindset to deal with stressful situations; and a process with challenges that are unique to each business.

We examine the recent SamSam ransomware attack at Allscripts, following the phases of incident response as documented by SANS.

Preparation

As the heading suggests, the first step in building out an incident response plan is preparation. Incident response teams have to prepare as best as they can for a number of incidents, from power outages, hardware failures and malware to tornados, floods and earthquakes — and everything in between.

By all accounts Allscripts did this, and in a statement the company told Salted Hash that they “prepare and drill” for various possible incidents ahead of time. In addition, the company conducts regular blue team and red team exercises.

However, it isn’t clear to what extent ransomware was part of their drills, as the company didn’t offer any insight into its incident response program or the mindset behind its development when asked about it.

Based on comments made to the public by Allscripts, posts to Allscripts ClientConnect portal by customers and company representatives, status update calls during the incident, and conversations with customers, Salted Hash is confident that Allscripts had an incident response plan in place (as they should), but there were some snags once the plans were actually put to the test during a real-world incident.

The SamSam ransomware attack against Allscripts started on a Thursday, and most customers reported they were offline or continued to have access problems until Thursday the following week.

For example, after Allscripts reported that systems for Allscripts Practice Management (PM) and Pro EHR “were restored in the East, Central, Mountain, and Pacific regions” on Tuesday, January 23, many clients in those areas reported zero access.

“We are currently working to restore permissions for all users. Once permissions are restored, users will have access to their core applications. We are continuing to work on restoration of interfaces,” an Allscripts update note explained.

So, while customers could access the cloud — as one customer observed — most couldn’t actually access the database. This was a key problem for many of the customers impacted by the attack, as they’re hosted customers. All of their practice information lives in the cloud, from patient records to billing details, lab results and more.

A number of customers turned to the ClientConnect forum to vent their frustrations in response to daily updates:

“A lot of talking but not really saying anything,” one customer wrote early on Tuesday morning.

“I am not restored yet despite the promise yesterday of 18- 24-hour restoration. Who cares if you can access the cloud if that just takes you to a dead site? It has been 6 days people, I do not want to hear that you are diligently working on it around the clock. I want results not mumbo jumbo.”

Salted Hash asked Allscripts about the disconnect between the status update and the reality for customers, such as why customers were still having issues even after the company said systems were restored.

“Allscripts serves a wide range of clients in a variety of individual circumstances. Accordingly, they experienced different effects as a result of this incident,” the company said. “There were a range of circumstances involved with getting particular systems back online and we addressed each of them as quickly as possible.”

Identification and containment

CSO

Allscripts identified the problem, meeting the requirements of the identification phase, when they determined the detected issues were ransomware related and furthered this identification by confirming the ransomware was SamSam, a family of ransomware known for targeting healthcare organizations.

In order to limit damage and prevent further damage, as required by the containment phase, Allscripts started severing connections with their data centers in Raleigh and Charlotte, N.C. No business wants to cut its own backbone, but sometimes the hard calls have to be made.

As mentioned, Allscripts detected something strange at around 2:00 a.m. EST on January 18, and by 6:00 a.m. EST had determined that it was a full-blown ransomware incident, requiring the assistance of teams from Cisco, Mandiant and Microsoft.

In a statement, Allscripts said that hundreds of personnel worked on the response, calling the first 24-hours an “intense swirl of many technical, business, and other practical challenges.”

Allscripts stated that they drill for various security incidents, and when asked, confirmed they participate in data sharing programs with other medical organizations.

In a statement, the company said they have data sharing relationships with the FBI, the North Carolina Healthcare Information & Communications Alliance, Inc. (NCHICA), and a group of healthcare industry CISOs. They also use outside advisors who “serve as additional eyes and ears.”

When asked how prepared they were for the eventuality of a ransomware attack, especially since SamSam had previously been used against two other healthcare organizations, Allscripts responded: “Keep in mind that there were no antivirus signatures available for this SamSam variant at the time it struck Allscripts. This was an entirely new, zero-day variant of SamSam ransomware that had never been identified previously by Cisco, Microsoft or the FBI. We were able to contain it within minutes, and then begin the intense work of restoring those client services that were affected.”

This is concerning.

By their own account Allscripts has access to a large pool of threat intelligence and information sharing. Cisco is one of their vendors, and their threat intelligence teams have done extensive work on SamSam.

Threat intelligence sources within the healthcare industry have known for some time that the group behind SamSam constantly makes modifications (variants) to the ransomware’s code in order to alter how it loads on a victim’s machine, thus bypassing antivirus and other endpoint defenses.

Moreover, those same sources also know that the actors behind SamSam don’t rely on spam-like delivery tactics to infect victims, opting instead to infect individual systems and manually deploy the ransomware.

As such, the threat intelligence sources who spoke to Salted Hash say the key to defending against SamSam, something that would be central to any threat model and corresponding threat map, is the realization that signatures and endpoint defenses alone are not enough.

Instead, the key to stopping SamSam is a mix of endpoint defenses, patch management, limiting system functionality (disable everything that isn’t needed for the system to perform its core role or function), and limiting user permissions.

Was Allscripts aware of this? Salted Hash asked Allscripts a number of SamSam-related questions in order to determine where the information gaps were and how up-to-date their systems were, but the company declined to answer any of them. Instead Allscripts simply stated, “this was a new ransomware variant for which no antivirus or patch was available.”

It’s important to note, though, that each organization is different, so achieving harmony between endpoint defenses, system functionality, user permissions and patch management isn’t as easy as it sounds when dealing with large enterprise operations. SamSam isn’t a case of “if you only do this, you’ll be fine” or “work this checkbox magic here please.”

Dealing with threats like SamSam requires effort and a lot of bridge building because multiple units within the organization have to come together in order to develop a workable solution.

Eradication and recovery:

With the threat identified and containment achieved, Allscripts had to turn its focus to eradication and recovery. This is where most of the pain lives for any organization dealing with an incident, as speed is the goal, but that isn’t always possible when critical systems are involved. You have to clean and restore systems and then test them before putting them back into production. It’s a daunting task at times.

At this stage it’s critical that Allscripts understand how the attack happened and take steps to ensure it can’t happen again, such as patching vulnerable systems or resetting compromised passwords — two things the actors behind SamSam often leverage in an attack.

It isn’t clear what happened during this phase, though, as Allscripts has declined to discuss it.

Allscripts moved quickly to restore systems, reporting that more than a dozen services already were or would be up and running by January 19. Each day, sometimes more than once a day, Allscripts updated customers on the recovery efforts and added new services to the list of those restored.

On January 26, in a letter to customers, Allscripts CEO Paul Black said that the company would be accelerating their plans to replicate Professional EHR across multiple data centers and a technology refresh “to shorten our recovery time in the event of any future disruption.”

When asked about the data replication and the use of virtualization, Allscripts told Salted Hash that they use virtualization when it’s reasonable and appropriate but added that they didn’t attempt to spin up any of their replication servers to help with the restoration process.

While the incident response plan developed by Allscripts worked (after all they were able to recover from the SamSam attack), there was still a good deal of pain during this process for their customers.

Lessons learned

Often during the containment phase an organization needs to do some sort of public contact with partners and customers, perhaps even with industry peers or regulatory bodies. But for Allscripts, the communication was a pain point.

Updates and answers from support representatives posted in the ClientConnect forum conflicted with reports from customers.

For example, customers were told on ClientConnect that the EPCS application, called ePrescribe, was online. That same day, customers reported intermittent access or lock-ups once the output icon was clicked. When a support ticket was filed, the customer was told it was a known issue related to the ransomware and that the company was working on it.

This frustrated customers, who perceived that Allscripts wasn’t being truthful. But Allscripts was being truthful; the services were up, the problem was access. So, the internal definition of up or online and what the customer is expecting those terms to mean just wasn’t jiving.

Customers, for the most part, are forgiving — if you’re open and communicative. The key, though, is to communicate in a way they understand.

Tech professionals could make the argument that Allscripts did restore services and the systems were online within 24 hours, but the reality is that customers were down or experienced intermittent issues for at least six days, and longer in some cases.

These may be outlier cases, but as long as they exist, the incident isn’t resolved.

Editor’s Note: This is the first story in a series on the Allscripts Ransomware attack that CSO will be launching this week. Tomorrow we look at the impact of the attack from the customer point of view — what happens when your SaaS provider is hit by ransomware?

Register now to download a free PDF of the complete series.