A ransomware attack against a SaaS provider hurts customers, but when it's a healthcare company that’s hit, patients suffer. Such was the case with January's attack against Allscripts, one of the largest electronic health record and practice management technology vendors.
By all accounts, Allscripts did a lot right. They had an incident response plan in place. They got outside help. They recovered their systems. They communicated with customers.
Getty Images
But as our reporting found, there was a communication gap.
When Allscripts reported a restoration to a given application or service, customers immediately reported the opposite. The root of the problem appears to be that while services were restored, access was a different matter entirely. While Allscripts acknowledged the access issues in support tickets and in some customer communications, the message being conveyed sounded like misdirection to some customers. But that wasn't Allscripts' intent at all (far from it), which is why communication during an incident is so vital.
Crisis communications
The key to communication is to explain things in such a way that the intent of the message and its objectives are clear. To put it another way, you need to speak on your customers' level.
Crisis communications is an important aspect of incident response when the organization experiencing the incident needs to communicate with partners, regulatory bodies, or its customers. It's also a good idea to have one channel of communications, a single voice, in order to prevent conflicting statements.
In Allscripts' case, when the company reported that systems were brought back online January 23 (five days after the incident started), customers rejected that notion because they couldn't access the tools and services needed to do their jobs.
Allscripts certainly brought those services back online, but the actions didn't meet customer expectations. To Allscripts' customers, online meant that things were back to normal, as if the ransomware attack never happened, which wasn't the case.
Impact at scale
Allscripts was victimized by the actors behind the SamSam family of ransomware on Thursday, January 18. While the company was able to activate its incident response plan and restore some services within hours, many customers were without access until the following Thursday.
The company said their Professional EHR and Electronic Prescriptions for Controlled Substances (EPCS) services were the hardest hit by the ransomware attack. In addition, customers also reported issues with Allscripts Practice Management (PM), and other tools within the suite.
Allscripts says that about 1,500 medical practices were impacted by the ransomware attack. It's likely that this figure represents the number of practices hosted in the Raleigh and Charlotte data centers, which were disconnected during the incident.
However, any organization operating in the healthcare space should realize that 1,500 practices could represent hundreds of doctors and other medical practitioners, as well as thousands of patients. On its website, Allscripts claims a "client base of 180,000 physicians across approximately 45,000 ambulatory facilities, 2,500 hospitals and 17,000 post-acute organizations."
What follows is a look at the pain experienced by the secondary victims of the SamSam attacks against Allscripts, that is, the medical practitioners and the office staff who use Allscripts' services. These accounts illustrate the real-world impact of such attacks. We are publishing them in order to stress the need to properly prepare for the eventuality of a devastating attack, as well to show how widespread and serious the impact could be.
Customer communications
On January 23, five days after the incident started, Allscripts told customers via the ClientConnect portal that they were making "material progress" in restoring affected services:
"Since our last update, Allscripts PM and Professional EHR systems in the East, Central, Mountain, and Pacific regions have been brought back online. We are currently working to restore permissions for all users. Once permissions are restored, users will have access to their core applications. We are continuing to work on restoration of interfaces."
Immediately, customers began questioning the update, asking for ETAs and clarification concerning the wording of the message. "What does this really mean," one customer asked, "when will we [be] able to access 'core applications'?"
In response to that question another customer stated: "It means you can access the cloud but you cannot access the database yet. That is not really restored then."
Other customers chimed in, calling the update misleading: "Stop misleading your clients and give them credible facts. Either we are up or we are not. The updates from you remain the same only worded differently."
CSOOn January 24, six days after the incident started, Allscripts announced that access to hosted Pro EHR and PM was now available to "nearly all clients both through the desktop application and the Pro Mobile solution."
At the same time, customers were having problems with the mobile offering (those that could use it, as it was only available for customers on iPads and iPhones) not meeting their needs. Moreover, the messaging from Allscripts was once again causing problems.
"We need access to PM in order to bill anything," one customer said, venting frustration. "Your mobile solution is no solution at all. What good does that do? It does not function like the actual EMR…You are hurting this practice financially, not to mention how our patients feel right now. We are not able to bill anything, post payments, run statements, etc."
Another customer, responding to a phone call from an Allscripts representative, advising them that "there have been some outages" simply had to laugh out of frustration.
"We have been down a week at this point and they JUST call to say SOME OUTAGES?! This is the FIRST phone call we received in a week from them! 3 or 4 employees can get in here and that is out of about 30 people. Our patient care is suffering," the customer continued.
"We are an OB/GYN office — How many days of your doctor not being able to access your records during pregnancy is ok for you? How long do you trust them when they can't help you with darn near anything? We are starting to look incompetent and Allscripts is to blame!"
Salted Hash asked Allscripts about the problems customers were experiencing, even after announcements were made that services were restored. "Allscripts serves a wide range of clients in a variety of individual circumstances. Accordingly, they experienced different effects as a result of this incident," the company said.
"There were a range of circumstances involved with getting particular systems back online and we addressed each of them as quickly as possible."
Everyone's eggs in one basket
It's a painful reality. A core vendor is taken offline after a serious incident, leaving the medical practitioners and their patients between a rock and a hard place. Most of Allscripts' customers selected the hosted option (going paperless) because it was economical.
"The issue is that a lot of pressure has been put on practices all over to make the move to hosted solutions and it seems great, but this very risk is almost always overlooked," one customer said, speaking to the pain experienced in the days after SamSam was first detected.
And yet, even when such a risk is understood, the customer expectation is that the vendor responsible for the hosted environment will protect them from such problems. Strictly speaking, Allscripts did this, but the customers certainly didn't feel as if that were the case.
Before moving to hosted EHR/EMR solutions (or really any hosted solutions), it's important to ask the vendor (such as Allscripts) to discuss their business continuity and disaster recovery plans — specifically how they plan to respond to threats like ransomware or hardware failure, and how quickly they can get your office back up and running.
One Allscripts customer said they requested a business continuity plan from the company months before the SamSam attack. Nothing ever came of that request.
Vendors aside, it's also important for the practice to consider internal business continuity plans. One practice manager (and Allscripts customer) shared their office's process for keeping their business moving during the Allscripts outage:
"We have copies of the forms we used before going to EHR years ago. Since there are frequent outages with Allscripts we immediately convert to paper and keep on moving forward. This time I created an appointment schedule on Excel and we manually put in several days of the schedule from the Mobile App and everyone has access to the Excel spreadsheet to see who is coming in and to add appointments."
Unfortunately, as others pointed out, the process defeats the whole reason many moved to hosted EHR/EMR to begin with. Not to mention this process duplicates their workflow.
This particular customer said they would scan the paper chart note into the system and be done with it. It isn't pretty, but it works, and that's what counts.
Moving forward
A letter to customers from Allscripts CEO Paul Black, dated January 26, said the company would be accelerating their plans to replicate Professional EHR across multiple data centers. This is good, because often cloud providers use multiple storage area networks (SANs) and virtual environments to speed up recovery in the event of an outage. It's certainly advisable to inquire about such features when considering hosted solutions.
However, the letter's remarks were also confusing and a bit disappointing. Specifically, the CEO's letter said that Pro EHR would be replicated across multiple data centers and that the company would perform a technology refresh to shorten recovery time in the event of future disruptions. The initial efforts are expected to be completed by September 2018.
Why will it take so long? Why wasn't this done sooner? Allscripts would only say that it "is a complex process that takes time to implement."
When asked about the data replication and their use of virtualization, Allscripts told Salted Hash that they use virtualization when it's reasonable and appropriate but didn't get into specifics.
As such, it isn't clear if their VM usage hurt or helped during the recovery process. However, given the issues and the length of the outage experienced by their customers, it's unlikely the VMs played any valuable role. The company's statement also said they didn't attempt to spin up any of their replication servers during the recovery phase.
In the aforementioned letter to customers, Allscripts CEO Paul Black, said the company would offer customers a 33-percent credit for fees on "all hosted applications that were affected by the ransomware incident."
One customer who spoke to Salted Hash called the credit a joke, stating that it wouldn't really compensate their office for the downtime they experienced. On the Client Connect forum, another customer said it was "a slap in the face" to Allscripts' clients.
"We all lost way more money with the system being down for 8 days then 33% of our monthly bill (sic)," the customer wrote.
Salted Hash asked how Allscripts arrived at the 33-percent figure, but the company declined to answer.
Editor's note: This is the second story in our series on the Allscripts ransomware attack. Yesterday we published a timeline of the attack and lessons learned. Tomorrow's story is a deep dive into the SamSam ransomware.
Register now to download a free PDF of the complete series.