On July 3, employees at Arbor Dental in Longview, Washington, noticed glitches in their computers and couldn't view x-rays. Arbor was one of dozens of dental clinics in Oregon and Washington stymied by a ransomware attack that disrupted their business and blocked access to patients' records.
But the hackers didn't target the clinics directly. Instead, they infiltrated them by exploiting vulnerable cybersecurity at Portland-based PM Consultants, which handled the dentists' software updates, firewalls, and data backups. Arbor's frantic calls to PM went to voicemail, said Whitney Joy, the clinic's office coordinator.
"The second it happened, they ghosted everybody," she said. "They didn't give us a heads up."
A week later, PM sent an email to clients. "Due to the size and scale of the attack, we are not optimistic about the chances for a full or timely recovery," it wrote. "At this time we must recommend you seek outside technical assistance with the recovery of your data."
On July 22, PM notified clients in an email that it was shutting down, "in part due to this devastating event." The contact phone number listed on PM's website is disconnected, and the couple that managed the firm did not respond to messages left on their cellphones.
The attack on the dental clinics illustrates a new and worrisome frontier in ransomware — the targeting of managed service providers, or MSPs, to which local governments, medical clinics, and other small- and medium-sized businesses outsource their IT needs. While many MSPs offer reliable support and data storage, others have proven inexperienced or understaffed, unable to defend their own computer systems or help clients salvage files. As a result, cybercriminals profit by infiltrating dozens of businesses or public agencies with a single attack, while the beleaguered MSPs and their incapacitated clients squabble over who should pay the ransom or recovery costs.
Cost savings are the chief appeal of MSPs. It's often cheaper and more convenient for towns and small businesses with limited technical needs to rely on an MSP rather than hire full-time IT employees. But those benefits are sometimes illusory. This year, attacks on MSPs have paralyzed thousands of small businesses and public agencies. Huntress Labs, a Maryland-based cybersecurity and software firm, has worked with about three dozen MSPs struck by ransomware this year, its executives said. In one incident, 4,200 computers were infected by ransomware through a single MSP.
Last month, hackers infiltrated MSPs in Texas and Wisconsin. An attack on TSM Consulting Services of Rockwall, Texas, crippled 22 cities and towns, while one on PerCSoft of West Allis, Wisconsin, deprived 400 dental practices around the country of access to electronic files, the Wisconsin Dental Association said in a letter to members. PerCSoft, which hackers penetrated through its cloud remote management software, said in a letter to victims that it had obtained a key to decrypt the ransomware, indicating that it likely paid a ransom. PerCSoft did not return a message seeking comment.
TSM referred questions about the Texas attack to the state's Department of Information Resources, which referred questions to the FBI, which confirmed that the ransomware struck the towns through TSM. One of the 22 Texas municipalities has been hit by ransomware twice in the past year while using TSM's services.
FBI spokeswoman Melinda Urbina acknowledged that MSPs are profitable targets for hackers. "Those are the targets they're going after because they know that those individuals would be more apt to pay because they want to get those services back online for the public," she said.
Beyond the individual victims, the MSPs' shortcomings have a larger consequence. They foster the spread of ransomware, one of the world's most common cybercrimes. By failing to provide clients with reliable backups or to maintain their own cybersecurity, and in some cases paying ransoms when alternatives are available, they may in effect reward criminals and give them an incentive to strike again. This year, ProPublica has reported on other industries in the ransomware economy, such as data recovery and insurance, which also have enriched ransomware hackers.
To get inside MSPs, attackers have capitalized on security lapses such as weak passwords and failure to use two-factor authentication. In Wisconsin and elsewhere, they also have exploited vulnerabilities in "remote monitoring and management" software that the firms use to install computer updates and handle clients' other IT needs. Even when patches for such vulnerabilities are available, MSPs sometimes haven't installed them.
The remote management tools are like "golden keys to immediately distribute ransomware," said Huntress CEO Kyle Hanslovan. "Just like how you'd want to push a patch at lightning speed, it turns out you can push out ransomware at lightning speed as well."
Otherwise, the hacker may spread the ransomware manually, infecting computers one at a time using software that normally allows MSP technicians to remotely view and click around on a client's screen to resolve an IT problem, Hanslovan said. One Huntress client had the "record session" feature of this software automatically enabled. By watching those recordings following the attack, Huntress was able to view exactly how the hacker installed and tracked ransomware on the machines.
In some cases, Hanslovan said, MSPs have failed to save and store backup files properly for clients who paid specifically for that service so that systems would be restored in the event of an attack. Instead, the MSPs may have relied on low-cost and insufficient backup solutions, he said. Last month, he said, Huntress worked with an MSP whose clients' computers and backup files were encrypted in a ransomware attack. The only way to restore the files was to pay the ransom, Hanslovan said.
Even when backups are available, MSPs sometimes prefer to pay the ransom. Hackers have leverage in negotiations because the MSP — usually a small business itself — can't handle the volume of work for dozens of affected clients who simultaneously demand attention, said Chris Bisnett, chief architect at Huntress.
"It increases the likelihood that someone will pay rather than just try to fix it themselves," Bisnett said. "It's one thing if I have 50 computers that are ransomed and encrypted and I can fix them. There's no way I have time to go and do thousands of computers all at the same time when I've got all these customers calling and saying: ‘Hey, we can't do any business, we're losing money. We need to be back right now.' So the likelihood of the MSP just saying, ‘Oh I can't deal with this, let me just pay,' goes up."
Because there are so many victims, the hacker can make a larger ransom demand with greater confidence that it will be paid, Hanslovan said. Attacking the MSP "gives you hundreds or even thousands more computers for the same cost of infection," he said. The "support cost of negotiating the ransom is low" since the attacker typically corresponds with the MSP rather than its individual clients.
Before this year's ransomware spree, MSPs were susceptible to other kinds of cybercrime. Last October, the U.S. Department of Homeland Security warned in an alert about attacks on MSPs for "purposes of cyber espionage and intellectual property theft." It added that "MSPs generally have direct and unfettered access to their customers' networks," and that "a compromise in one part of an MSP's network can spread globally, affecting other customers and introducing risk."
The first spate of ransomware attacks on MSPs, early this year, deployed what is called the GandCrab strain. Then, in an online hacking forum, the hackers behind GandCrab announced their retirement in May. After that, another strain of ransomware known as Sodinokibi ransomware sprung up and began targeting MSPs.
Sodinokibi ransom amounts are "scaled to the size of the organization and the perceived capacity to pay," according to Connecticut-based Coveware, which negotiates ransoms for clients hit by ransomware. Sodinokibi will not run on systems that use languages including Russian, Romanian and Ukrainian, according to security firm Cylance, possibly because those are native languages for hackers who don't want to draw the attention of local law enforcement.
Sodinokibi was the strain used in the attack on TSM Consulting Services that encrypted the computers of 22 Texas municipalities, leaving them unable to fulfill tasks such as accepting online payments for water bills, providing copies of birth and death certificates and responding to emails. Most of the towns have not been publicly identified. More than half have returned to normal operations, the Texas Information Resources Department said in an update posted on its website. The hackers sought millions of dollars. The department is "unaware of any ransom being paid in this event," according to the update.
TSM began operations in 1997, and it provides equipment and support to more than 300 law enforcement agencies in Texas, according to its website. It is unclear why the 22 municipalities, and not TSM's other clients, were affected by the August attack.
One of the 22 Texas municipalities hit last month was Kaufman, a city about 30 miles southeast of Dallas. An attack last November on Kaufman, which forced its police department to cease normal operations, was mentioned in a ProPublica article about two data recovery firms that purported to use proprietary technology to disable ransomware but in reality often just paid the attackers. TSM had enlisted one of the firms, Florida-based MonsterCloud, to help Kaufman recover from the November intrusion.
MonsterCloud waived its fee in exchange for a video testimonial featuring the Kaufman police chief, the president of TSM and the TSM technician who worked with Kaufman. In the testimonial, TSM technician Robby Pleasant said that the attackers had "reset everyone's password, including the administrator," and that the data "was locked up and not functioning." Pleasant said in the video that MonsterCloud was able to "recover all the data" and "saved the day."
"They can come in and recover even if someone does find a hole in our armor," Pleasant said in the video.
Last month, attackers again found a hole in TSM's armor. Using a third-party software vendor, rather than TSM, Kaufman had strengthened its backup system since the first attack, so it was able to restore much of the lost data, City Manager Michael Slye said. Kaufman's computer systems were down for 24 hours, and the city handled municipal business such as writing tickets and taking payments on paper during that time, Slye said.
But backup safeguards were less effective for Kaufman's police department, which uses a different type of software than other city offices, Slye said. The department's dashcam video storage lost months of footage, and it still isn't working, he said.
"It was not a fun experience to get this twice," he said.
A TSM employee who declined to be named said the November attack may have been caused by "someone clicking on a bad email. We don't have definitive information on that. We went into recovery mode immediately."
PM Consultants, the Oregon provider of IT services to dental clinics, was run by a husband and wife, Charles Gosta Miller and Ava Piekarski, out of their home, according to state records. The firm didn't employ enough technicians, said Cameron Willis, general manager of Dentech LLC in Eugene, Oregon, which took on many of PM's former clients. Some former PM clients have complained to Willis that it was unresponsive to their requests for help, he said.
"A lot of dental office facilities don't want to spend the money on IT infrastructure the way they should," and they lack the technical know-how to vet providers, Willis said. They "don't know any better. They don't have the time to research. If you have someone who does provide some service, it's very, very easy to see how some of the fly-by-nights would attract such a large clientele. ... When one office finds something that works, they scream it to the hills."
In the July 22 email announcing its closure, PM said it had been "inundated with calls" on the morning of the ransomware attack, "and we immediately started investigating and trying to restore data. Throughout the next several days and into the weekend, we worked around the clock on recovery efforts. ... However, it was soon apparent the number of PC's that needed restoration was too large for our small team to complete in any reasonable time frame." The company was also "receiving hundreds of calls, emails and texts to which we were unable to respond."
PM said that it had retained counsel to "assist with recovery of any available insurance, payment and billing proceeds," and that it would be "sending out final invoices in the next two weeks." Its formal dissolution, it continued, "will include an option to submit a claim" against the company.
Austin Covington, director of Lower Columbia Oral Health, a Longview, Washington, clinic affected by the attack, said it plans to take legal action against PM and declined to comment further. Other victims have not been publicly identified.
Some dentists "did not lose any data" because they had good backup files, Willis said. "Some clients lost some. Some lost a lot." He doesn't know whether clients paid ransoms, he said.
Dentech takes a different approach than PM did, Willis said. To prevent ransomware and other breaches, even its own staff has limited access to the remote management software favored by hackers, he said. It has 14 technicians, who often handle services such as software updates in person, he said. Dentech requires clients to use best practices, Willis said. If they decline, the firm requires them to sign a waiver releasing Dentech of liability in case of ransomware or other data loss.
Without such explicit terms, it's often unclear whether the MSP or its clients are responsible for paying ransoms or recovery costs associated with an attack. Chris Loehr, executive vice president of Texas-based Solis Security, which helps victims negotiate ransom payments, was called in when GandCrab ransomware struck an MSP and encrypted some of its clients' backup files several months ago. The MSP paid the ransom only for those that used its data backup service, which had failed, Loehr said. Clients who did not buy the backup service had to decide themselves whether to pay the ransom.
This summer, in a separate incident, Loehr negotiated with hackers on behalf of a New York-based MSP that was hit by Sodinokibi ransomware. The MSP didn't want to pay the total ransom of about $2 million in bitcoin to unlock the files of all its clients, who were primarily architectural and engineering firms. Instead, each of the 200 affected clients was left to decide whether to pay about $10,000 in bitcoin. The MSP's owner refused for legal reasons; he was worried that, if he was sued over the attack, a payment might be construed as an admission of fault, Loehr said.
The preponderance of low-quality MSPs has fostered the current ransomware onslaught, Loehr said. He noted that little experience or funding is needed to open an MSP; the barriers to entry are few.
"The startup costs are low," Loehr said. "It doesn't take much. The way the MSP world works, it's not like you have to go out and buy $1 million of software. You can operate out of your house. These guys charge their clients upfront. There is little cash flow to get this stuff off the ground."
"Every IT guy thinks he can do this," Loehr said. "‘Hey, I'm a technology guy.'
"No."
Renee Dudley is a tech reporter at ProPublica.
Copyright © ProPublica. Licensed under CC BY-NC-ND 3.0 US.
