Mass Emergency Planning Exercise Tests Resilience to Coordinated Converged Attacks
How do you test the resilience of a multinational electric grid system that 40 million people rely on? You play.
Every other year, thousands of people from 250 North American electricity providers, gas and telecommunication partners, government agencies, and other organizations participate in the North American Electric Reliability Corporation (NERC) and Electricity Information Sharing and Analysis Center (E-ISAC) GridEx exercise.
The November 2023 exercise included two days of distributed play, with planning and exercise materials developed by the E-ISAC’s GridEx Planning Team. Local planners then used these materials to design and conduct exercises, as well as hold an in-depth executive tabletop exercise with industry executives and government leaders.
These exercises are designed to stretch critical infrastructure providers’ resilience and emergency response plans, finding gaps and uncovering ways that government partners can better support electricity providers during crises, says Manny Cancel, senior vice president at NERC and CEO of the E-ISAC.
This can result in tangible benefits for organizations. For instance, a previous GridEx produced the concept of extending mutual assistance agreements to cyberattacks. In response to natural disasters, electricity providers often roll repair trucks across the United States to help other grid operators get back up and running. The same concept is being applied to cybersecurity, where electric companies can deploy cybersecurity professionals and share insights to help another company respond to and recover from an incident, Cancel says. This assistance can prove invaluable, especially because only around 100 out of the 3,300 utilities in North America are large, investor-owned companies. The rest are state, local, and cooperative operations with limited cybersecurity capabilities, he explains.
“That’s why we have to work as a sector to share information, because we have to make sure the whole sector is as prepared as it can be,” he says.
The E-ISAC shares threat intelligence and risk mitigation advice daily, but GridEx is a chance to put plans to the test.
The latest exercise, GridEx VII, simulated a multi-vector threat that organizations had to contend with during an extended period.
“We actually do try to break the grid,” Cancel says. “And the reason we do that is not to set people’s hair on fire; it is really to start to answer some of the tough questions about: how do we communicate? How does this change our incident response? What should we think about in this converged attack scenario? What’s the communication like with the government? What are the restoration priorities?”
Cancel has participated in every GridEx, either as a player (he was previously chief information officer at Con Edison) or as a facilitator. He attests that the drills are quite comprehensive, albeit a bit daunting.
“At the end of the day, I will say it’s still realistic—it’s pretty intense,” he adds. “It can be overwhelming, but it is a realistic set of scenarios that we set forth.”
So, what scenarios did players have to contend with in GridEx VII?
Distributed play. The two-day distributed play exercise included incidents ranging from cyber and physical attacks on substations to disinformation on social media. Cyberattacks hampered participants’ ability to respond to threats, disrupting communications, reducing generation capacity, and softening the target for a coordinated physical attack, where assailants targeted multiple substations, firing guns at critical transformer components. This caused electricity outages across a large operating area. Then, a distributed denial-of-service attack against the corporate virtual private network (VPN) system made remote access to systems intermittent or impossible. Meanwhile, organizations had to contend with misinformation and disinformation on social media.
Local government officials and the public pressured participants to begin recovery, but assailants struck again, including by detonating a vehicle-borne improvised explosive device at a telecommunications facility.
Frustrated by the outages and disinformation, protesters started to harass utility personnel. Then, explosives detonated at equipment storage and staging areas, damaging spare equipment needed to restore service. Jumping forward a week in time, players explored recovery and long-term considerations, including global supply and diesel fuel shortages that could further delay restoration.
Tabletop exercise. The tabletop exercise involved 75 organizations, including U.S. and Canadian electricity industry leaders, government officials, and law enforcement representatives.
The exercise tested how the industry and government agencies would respond to a sophisticated, well-coordinated physical and cyberattack. It simulated a compromise in key software that grid operators use to monitor and control the grid. Simultaneously, an incident degraded communications in large areas of the United States. Then, a coordinated kinetic and cyberattack damaged high-voltage transformers, circuit breakers, and remote terminal units at transmission substations in Louisiana and Texas. The ensuing power outages disrupted operations at several nationally important natural gas hubs.
Impacts of the attacks were long-lasting, especially after the Midcontinent Independent System Operator found that its website was defaced, criminals sent a ransom demand, its backup systems appeared corrupted, and a critical IT staff member could not be reached. The exercise wrapped up by assessing recovery one month after the attack, which included ongoing repairs.
Why did the E-ISAC choose to test companies on converged attacks? North American companies aren’t currently seeing many intentional converged attacks (although physical and cyber threats may overlap coincidentally), but “where you do see it is in the geopolitical context, and particularly in warfare,” Cancel says. “So, you’ve seen it in Gaza, you’ve seen it in Ukraine, and probably other parts of the world where there are conflicts. It is using both kinetic and physical attacks and—at the same time—leveraging cybersecurity-related attacks too. It’s a ‘let me take advantage of everything I can to help defeat my enemy here or to advance my position’ approach.”
Although those attacks are not commonly seen so far in private industry in the United States and Canada, they are worth preparing for. This is especially important because threat actors are looking for the easiest path into an organization—whether that’s leveraging a physical attack or a cyberattack to cause damage or gain a payout, Cancel says.
Some of those incidents are singular, while others mark a long-term strategy. In multiple incidents in late 2022, assailants in the United States physically attacked electric substations, shooting at equipment with firearms to cause damage and outages. A pro-Iran group targeted industrial equipment that water utilities use in pump stations. Chinese state-backed hacking groups have successfully infiltrated multiple critical infrastructure organizations in the United States, FBI Director Christopher Wray said in April, and they could use that access to study activity, configurations, and defenses, waiting for an opportune moment to launch an attack.
“All critical infrastructure is adopting an all-hazards approach,” Cancel says. “It’s no longer acceptable to just focus on one aspect or one vulnerability. You have to recognize that you can be compromised by severe physical attacks and certainly cyberattacks… While this hasn’t happened on a wide scale, we practice this—this is exactly what we do in GridEx. We simulate a converged scenario, not necessarily by the same actors but just for the sake of recognizing that we may one day have to do this right, on a pretty severe scale.”
That’s not to say that the GridEx VII scenarios are out of the realm of possibilities today. The E-ISAC team uses real-life vulnerabilities, weather events, and recent attacks worldwide to design the exercises. Then, the team collects lessons learned and recommendations from both the exercises and participants to drive future improvements and resilience.
It’s no longer acceptable to just focus on one aspect or one vulnerability.
GridEx VII produced multiple recommendations, including:
Evaluate how to improve software security and Inter-Control Center Communications Protocol (ICCP) telemetry exchange between control centers. The tabletop exercise prompted participants to consider that current ICCP infrastructure—which utilities use to monitor and control operations at transmission and generation facilities—may not be resilient against single-point-of-failure vulnerabilities.
Further refine communications, response protocols, and redundancy in a hybrid work environment. The scenarios in both tabletop and distributed play exercises tested utilities’ ability to cope with remote work challenges, including unavailable key personnel and the inability to access key systems remotely if the VPN fails.
“After COVID-19 began, organizations made different decisions about managing in-person and remote work, and it may no longer be possible for all responders to meet in a single room to coordinate,” the GridEx VII summary report said. “Consequently, it is important to continue developing best practices for response in a hybrid environment. GridEx VII provided an opportunity to test hybrid response protocols during two days of intense exercise play. For example, one organization planned for remote and in-person response by identifying a secure information-sharing system for its executive team. However, during GridEx VII, some players realized that not everyone who needed to share sensitive information had the necessary access to the software, demonstrating the challenge of hybrid environments when responding to extraordinary operational circumstances.”
Cancel adds that using hybrid workforces during the GridEx drill made the exercises more realistic because it is unlikely that every key personnel member will be perfectly situated at the start of every emergency. But being able to connect with them in a crisis, ensuring that new personnel are adequately trained on response procedures, and staying situationally aware of incidents when personnel are outside of a command center can be challenging, he says.
The exercise stress-tested communication tools and networks, including issues around differing security protocols across agencies and companies.
Augment response planning so teams and partners comprehend technical information. Common terminology is essential during incident response because jargon can slow comprehension and introduce opportunities for misunderstanding, the report said.
“GridEx provides the opportunity to convene personnel from different business units and organizations, including non-electric industry stakeholders, to discuss complex response and restoration of the electric grid,” the report explained. “However, communicating technical information can be challenging for non-technical personnel, potentially hampering the response to an incident. Government responders may not have a background in the electric industry and need more support understanding the implications of certain attacks or response actions. The same may be true of non-electric utility partners, such as those responsible for water/wastewater. These organizations may need to have power restored as a priority but not be able to interpret communications from electric utilities.
“Just as technical knowledge could make it difficult to communicate with external partners, some planners found that it was a challenge internally as well,” the report continued. “Corporate functions such as communications, security, and customer care were not always comfortable with grid security incident response processes, making it difficult for them to support responders and the operational needs of the organization.”
Along with creating documents and guides in more common and unified language, the report also recommended that asset owners and operators consider implementing internal training for non-technical personnel to provide a basic understanding of technical, critical topics.
Evaluate how industry and government will handle conflicting priorities during a complex and prolonged outage and restoration. Large-scale crises affecting electricity and critical infrastructure during a long period will likely reveal conflicting priorities between industry, government, and citizens.
“Industry should determine the need to develop an improved restoration framework that considers government requests that may conflict with pre-established restoration priorities and recommend guiding principles for coordinating with other critical infrastructures as needed,” the report noted.
In some incidents, restoring power could require a great deal of specialized equipment—often exceeding what utilities have on hand. Suppliers may be unwilling or unable to give special priority to customers unless doing so benefits them or meets contractual obligations. Supply chain struggles identified during the COVID-19 pandemic heightened awareness of this potential chokepoint in incident recovery.
“Industry should leverage its supply chain efforts and determine the need to improve processes to address equipment and supply shortfalls during a large-scale crisis and identify where government authorities can resolve supply chain issues,” the report said.
Claire Meyer is managing editor of Security Management. Connect with her on LinkedIn or via email at [email protected].
