Underfunded Water Utilities Contend with Leaky Defenses
What would you do if your toilet stopped flushing? What if the taps ran dry and you were unable to access safe drinking water for days or even weeks?
You likely have a plan for a power outage, but what about the long-term unavailability of clean water or sewage services? This is the scenario that keeps Lesley Carhart and their colleagues at risk management firm Dragos Inc. up at night.
“Not having sewage that works is a big deal,” says Carhart, who is technical director of incident response at Dragos. “We’re not used to seeing that in the United States. Not having clean water to drink is not something we’re used to in most of the United States and Canada.”
Water systems limited defense, monitoring, and detection abilities, along with the potential to make a big impact when struck, make them a “juicy target when you have that overarching goal—which a lot of these adversary groups do—of having a big negative impact and psychological impact on an opposing country,” Carhart adds.
It’s hardly unheard of. Threat actors struck water treatment plants using cyberattacks in recent years, including when a hacking group tied to Russian state actors infiltrated a Texas water facility in January 2024, causing a system malfunction and forcing a water tank to overflow. The same group took credit for an attack at a wastewater treatment facility in Indiana in April. In late 2023, an Iranian-linked group targeted multiple organizations that used Israeli-designed equipment, including a Pennsylvania town’s water provider, forcing it to switch from a remote pump to manual operations.
Electric grids and major utility providers have increased their resources and cybersecurity defenses in recent years, but most U.S. water and wastewater utilities are municipal. As such, they are likely working out of nondescript office buildings with just a few computers and IT professionals controlling systems that supply water for cities and towns, Carhart says. This lack of robust resources and support puts these organizations at risk.
The U.S. Environmental Protection Agency (EPA) warned in May 2024 that water systems are being targeted by cyberattackers. The agency also said that 70 percent of utilities inspected by federal officials in the past year violated standards meant to prevent breaches, such as failing to change default passwords or revoke system access for former employees.
“Implementing basic cyber hygiene practices can help your utility prevent, detect, respond to, and recover from cyber incidents,” the EPA said. “Because water utilities often rely on computer software to operate their treatment plants and distribution systems, protecting information technology and process control systems from cyberattacks is vital. Small water systems are not immune from cyberattacks.”
The interconnectivity between information technology (IT) and operational technology (OT) systems makes a holistic risk management approach essential, says Roya Gordon, executive industry consultant for OT Cyber at Hexagon’s asset lifecycle intelligence division. Although recent high-profile attacks have targeted OT infrastructure—such as the systems that add chemicals to treat drinking water—she warns that organizations can’t abandon IT security to focus on OT because adversaries will use IT weak links (including poor credentialling or cyber hygiene) to access OT systems.
“We're now in a paradigm shift,” Gordon says. “Before, we saw there being security layered on top of these legacy systems. You can’t just rip and replace and upgrade systems in OT environments because it’s a live environment. You can’t afford downtime.”
So, instead of replacing insecure systems, operators would use workarounds that layer security systems on top by adding firewalls, threat intelligence solutions, and monitoring.
The industry is shifting toward security-by-design, but it’s slow going because of the longevity of OT. Systems are designed to last for decades, meaning it could easily be 40 years before a less-secure system can feasibly be replaced, Gordon explains. In addition, downtime in utilities, even just to change a small configuration, could result in service interruptions or even break other, older OT systems. A 2024 SecurityWeek report found that maintenance is the primary source of OT security incidents. So, operators must proceed with caution when upgrading systems, making investments in both IT and site security to protect OT even more vital.
Utilities have long relied on a method of “security by obscurity,” hiding OT procedures and systems behind layers of other systems and tools, Carhart says. Sifting through all of that is a long-term, expensive investment for an attacker looking for a way in.
“At the most fundamental level, adversaries will use the least amount of resources and effort to reliably accomplish their goals,” Carhart explains. “If you want to take out power to some geographic area, if you are there locally, it’s probably easier to take a paint can and throw it on a transformer than learn how to infiltrate a system over a long period of time. That’s a hefty investment that takes a lot of time, a lot of people, and a lot of expertise. If you have physical access to that thing and you’re not going to get caught, you’re going to take a paint can, throw it on a transformer, and cause a big disruption. If you’re across the planet and you can't do that, well, the most efficient, effective way to do that same type of thing might be to do this reconnaissance, learn about an environment, and do your best job in doing a disruptive attack.”
It’s probably easier to take a paint can and throw it on a transformer than learn how to infiltrate a system over a long period of time.
Consider it like a rubric: assailants are looking for the most effective, least expensive pathway to accomplish their goal, whether it’s financial (ransomware attacks, thieves, insider threats), reputational (hacktivists), or political (nation-state attacks and espionage).
When measured on that rubric, water facilities are logical targets, especially for cyber adversaries looking to turn a quick profit.
“Commercial ransomware groups are in it to make money, and they know two things about wastewater,” Carhart says. “They know, first of all, it’s incredibly undefended because there are very, very few cybersecurity resources there. Second of all, they know that it’s a really important thing to society and people really notice when it doesn’t work, so people are likely to pay up.”
Adversaries focused on sabotage—including nation-state actors—are also looking for easier targets with large potential impacts.
“The overall objective is to cause a disruption to society, cause a psychological impact, cause a human impact for these state-style adversaries,” Carhart explains. “Looking at the utilities they have available to them to attack, oil and gas could have a huge impact, yes, but it tends to be very well resourced in defense. You’re going to have to invest a lot in attacking them and causing an impact… and same with a lot of electric utilities.
“Now, if we look at water, it’s municipal,” they continue. “There are next to no cybersecurity teams. Some of these organizations have one IT person. And from my perspective at least, [water and sewage utilities] cause just as much impact to our daily lives and the health and wellness of our society as electricity.”
Although there are some resources available, especially around information sharing and guidance through the Water Information Sharing and Analysis Center, they don’t necessarily build budgets for small municipalities. It can also be a tough sell to city councils and voters when potential attacks feel far-fetched or detached from the region, Carhart adds.
“For your local town or suburb, that’s not an easy sell when you’re talking about, ‘Do we repair the streets or do we hire a cybersecurity person?’” they say.
Some vendors offer discounted or free security tools for municipal services or nonprofits. Carhart recommends taking advantage of as many educational opportunities and information-sharing sessions as possible so utility security teams can improve their operations and awareness at minimal cost.
“If there are cyber exercises in your vertical or for your region that you can be a part of, they are usually funded by somebody else,” Carhart says. “If you can participate just as an observer in them, that’s a really good idea.”
Some exercises are hosted by ISACs, government agencies, or other utilities, and it can be beneficial to see how other organizations handle similar challenges. Free resources are also available through the Cybersecurity and Infrastructure Security Agency (CISA, part of the U.S. Department of Homeland Security), the EPA, and cybersecurity companies. The Biden administration also released a National Security Memorandum in April about critical infrastructure security and resilience that sets up future requirements and resources for utility security.
“Absolutely leverage those if you can. Make sure you know what’s available for you,” Carhart says.
That’s not an easy sell when you’re talking about, ‘Do we repair the streets or do we hire a cybersecurity person?’
Gordon recommends breaking down OT protection improvement into four key steps:
- Create an asset inventory. Know what you have, what’s vulnerable, what’s at the end of its life, and what’s critical to operations.
- Track vulnerability identification. Security organizations and government agencies release new OT and IT system vulnerabilities daily. Keep track of which vulnerabilities could affect your systems and operations. Triage based on which vulnerabilities could produce notable damage or allow unauthorized system entry.
- Conduct risk management and mediation. Once you understand which vulnerabilities you face, work with other stakeholders in the organization and community to identify which risks need to be mitigated, which can be accepted, and which can be transferred. Mediate risks as needed and able.
- Develop OT incident response and forensics. Spend some time developing an incident response plan for OT-impacting attacks, including how you will handle forensics and investigations of the incident while simultaneously pursuing recovery.
The fourth item—response and recovery—is often missing from many smaller utilities’ preparation, Carhart says. This produces panic during the incident and often results in extremely expensive emergency sessions with outside security professionals and consultants.
“I get a lot of frantic calls in the middle of the night from organizations that have never planned for a cybersecurity incident,” Carhart adds. “I promise you it is much more expensive to call me in the middle of the night when you’ve had no plan, no retainer, and nothing invested in cybersecurity than it is to spend a little bit of time and effort writing down a plan and knowing who you’re going to call. It’s going to save you a lot of time, stress, and money if you plan things out in advance, even if it’s on the back of a napkin. Who are you going to call, what are you going to try to accomplish, and where’s the funding going to come from?”
Because any disruption in service has significant effects on a utility’s community, there can be intense pressure to recovery quickly, so it also behooves organizations to know what steps to take to evaluate compromised systems, check for signs of infiltration, and ensure that a speedy recovery doesn’t enable an adversary to slip through the cracks, says Manny Cancel, CEO of the Electricity ISAC.
“The landscape is unprescedented, to the point of almost being overwhelming,” he says. “Just on the cyber side, the amount of vulnerabilities that you have to respond to and be worried about is incredible. And the adversaries have figured this out in what we call the ‘one to many’ problem—rather than attacking an individual company, I’m going to attack a platform that you use, and then let me see what I can get access to or how I can compromise the sector.”
In addition, some well-funded adversaries—such as Volt Typhoon, a stealthy state-sponsored actor supported by China—have focused on using system vulnerabilities to gain access to critical infrastructure systems and then sit and wait. These adversaries carefully observe utilities’ operations, configurations, and defenses in a “living off the land” technique, blending in and harvesting data to put the attackers in a position to conduct a more devastating attack later, according to a 2023 Microsoft security blog.
Adversaries can also use a splashier incident to hide their infiltration, banking on utility defenders missing it during the rush to restore.
“There isn’t a tolerance for outages, and other sectors are very dependent on utilities,” Cancel says. “We will always look to restore as soon as possible, but it’s worth stepping back for a brief bit of time and thinking about the ramifications. The other thing is having the appropriate monitoring. So, if you think you eradicated the problem but, in fact you missed it or you made it worse, and there’s a backdoor that the adversary put in to wreak more havoc… That’s something we need to look out for.”
The future is electric. Our February issue of Security Technology takes a look at the opportunities and risks this poses for security practitioners around the world. Read the issue: https://t.co/SlxXcUsR11
— Security Management (@SecMgmtMag) February 2, 2024
When it comes to OT risks, Gordon recommends three areas utility security professionals should focus on.
Network monitoring. “You want to make sure there’s not an external device trying to communicate with your device,” Gordon says. Fortunately, many solutions exist to help security professionals watch for external contact and communication from OT networks. These solutions “tap the network, so they listen in on the network traffic. They take a copy of it. It’s non-invasive,” she adds. “Then they do a deep packet inspection of all the communications going on in the OT network. They’re able to pull asset inventory, but they are also able to create signatures to search for those indicators of compromise and even block malicious activity.”
Physical monitoring. Gordon recommends diving into the Purdue Reference Model for industrial control systems, especially when it comes to checking on physical assets.
“Those devices aren’t always communicating on the OT network,” she says. “You have systems that are isolated. You have safety systems that aren’t going to be continuously communicating on the OT network. They’re going to communicate when they are triggered on an as-needed basis.”
If those environments aren’t being consistently checked, the organization only has visibility into half of its key systems. This can be particularly dangerous for insider threats because attackers with inside knowledge know which systems are party to network monitoring and which ones might be overlooked.
Backups. Having effective backups to restore systems is the general guidance to defeat ransomware attacks, but new wiper malware puts that maxim to the test. This malware—which can be embedded in ransomware attacks as a two-wave strike, Gordon says—can permanently delete or corrupt data on targeted systems, rendering them inoperable. One benefit of OT environments, though, is that they don’t change often. A backup from two weeks or a month ago should be perfectly fine, so recovery won’t need to involve reconfiguring the system from scratch, she says.
Overall, water utility security is an underfunded area with a high potential impact, and it requires multiple parties and functions to come together and share resources and responsibility.
“There's not one silver bullet,” Carhart says. “There’s not one party there that can fix everything. It’s a combination of all of those things put together that are going to solve this problem, which is a big problem. Water is the thing that keeps me and my colleagues up at night. Sewage, fresh drinking water—those are the things that keep us awake and keep us worried.”
Claire Meyer is managing editor for Security Management. Connect with her on LinkedIn or via email at [email protected].
