The year of 2020 was a string of regional disaster records: the worst wildfire season in California, again; Nashville Christmas day bombing shuts down phone services in four states; more hurricanes that reached landfall in the United States (12) since 1916 (9); and, of course, there’s that pesky worldwide pandemic which technically isn’t a regional disaster.
For 2021, the recent Texas state-wide power outage resulted in between 3.3 million to 4 million Texas homes losing power.
Separately, natural disasters in the U.S. have grown to such a level there’s a Wikipedia page.
Looking at an article published by the American Petroleum Institute in response to the Texas power outage regional disaster, Joshua Rhodes from the Webber Energy Group, a research group at the University of Texas, Austin, refers to the Texas power situation as a “black swan event.”
Maybe.
Black Swans are Truly Rare
For clarity, a black swan scenario is an extremely rare event, or an event perceived to not even exist. Yet, in 2020 and now in 2021, the unthinkable continues to occur. It’s time to expand our perspective because yesterday’s impossible has become today’s reality.
In 2018 and again in 2019, I wrote two blogs about black swan events. To really understand a black swan event, and how it effects your business, is to: 1) understand the most valuable assets and processes that power your business, and 2) find your business stress points with scenarios so wickedly terrible you thoroughly understand how disruption effects your business, your clients and your employees.
Demonstrating how bad these black swan scenarios should be, pre-Covid-19 pandemic planning scenarios were more like a zombie apocalypse than our current state of play where over 68% of the US workforce is working from either home or an office.
Regardless, once you understand the critical business process breaking points, and you understand where risk exists combined with the magnitude of that risk, you can develop a plan, and a corresponding budget, to reduce and perhaps even eliminate risk.
Resiliency Planning Saves the Day
For an example on how this works, look at a recent report published by the Christian Science Monitor looking back at how the Texas power grids responded to cold snaps in 1989 and 2011.
“After a 1989 cold snap, the Public Utility Commission of Texas, the agency that oversees ERCOT, recommended improvements in the winterization of electricity generators.” These non-mandatory recommendations were not implemented, and these same generators failed in 2011 during another round of record low temperatures. In fact, some of the failures from 1989, 2011 and again in 2021 were reported to be repeat events.
Meanwhile, El Paso Electric, which is not connected with ERCOT, also suffered devastating winter related outages in 2011. Afterwards, the local utility invested $4.5 million improving both the technology required to run their infrastructure and winterized delivery supply systems. Costs were ultimately forwarded to customers of El Paso Electric, but these customers saw the investment benefits during a week when most of the rest of Texas went dark.
El Paso Electric determined by spending $4.5 million, they were able to alleviate the events of 2011 or worse. The cost to El Paso Electric’s 437,000 client base: $10.30 per customer which most likely was spread over one or two decades.
In the Texas ERCOT example, we know the worst-case winter peak is roughly 70 gigawatts and the summer peak is 75 gigawatts. How much will it cost to build a fully reliable infrastructure to support at least those respective peaks? And more importantly, are the people of Texas willing to support that investment?
Looking at the 4 million households impacted by most recent Texas outage, and using the same $10.30 investment per customer metric, I’ll bet $41 million buys a lot of infrastructure stability. Meanwhile, insurance estimates are approaching $19 billion.
Simulate Your Black Swan
Now, let’s review how one company prepared for a statewide disaster.
Coincidentally, I spent a single day in Dallas in 2019 with a group of clients where we simulated a very similar power outage scenario. In our scenario, the entire ERCOT infrastructure covering Houston, Austin, Dallas, and San Antonio were offline for days. The team didn’t know when power would be restored, only that with the passing of each simulated day, their situation grew worse.
Most mobile phones were dead within 36 hours. Almost all laptops stopped working after 6 hours. After three days, data centers dependent on fuel delivery commitments were running out of fuel because a pretend martial law confiscated statewide supplies. As you can guess, the situation only grew worse from there.
Yet, from this exercise, the team became resourceful.
Creating Business Value
The team used the exercise to identify employees and processes critical to the business, and post-simulation, executed a series of actions. Here are three:
The team purchased portable power systems and battery backup units for critical employees, and implemented policies confirming these devices are regularly charged and tested.
The team pre-selected remote workplaces in other parts of the state and in neighboring Oklahoma and Louisiana where employees could relocate at time of crisis.
The team registered with the Cybersecurity and Infrastructure Security Agency’s (CISA) Wireless Priority Service allowing for prioritized wireless access when most wireless circuits may be unavailable.
These recommendations were quick hits, but there were dozens more that surfaced from this 2019 black swan exercise that would have paid big dividends during February’s real-life disaster.
Unfortunately, many businesses view these what-if planning sessions as unproductive and not adding to the company’s value.
To the contrary, at a minimum, by running these crisis simulations, employees recognize management is concerned for their employees and their jobs. By documenting and promoting these exercises with clients, clients recognize a viable business partner actively positioning for an unforeseen crisis which protects their partnership.
Which business partner would you rather align yourself to? The firm who plans for disaster and continuous operations, or the firm who does nothing?
By looking at the bigger picture, such as understanding the cost of an outage in terms of lost revenue, lost clients, and lost productivity, spending a few pennies today will add dollars of valuation later.
Want to learn more about disaster preparedness? Check out our podcast:
ABOUT THE AUTHOR
Michael Puldy is the founder and CEO of Puldy Resiliency Partners, LLC, where he helps companies improve their risk posture against internal and external business threats by building and managing business resiliency programs. He has over three decades of disaster recovery, information risk management, and operations experience including two-plus decades in senior leadership roles at IBM.
Michael has the distinction of being named a Ponemon Fellow by the Ponemon Institute. He is an award-winning speaker and author of professional and peer reviewed papers, blogs, and has published two books. Michael has a patent pending with the United States Trademark and Patent Office, and he is currently writing The Renaissance of Resiliency, discussing the evolution of data center centric IT disaster recovery to business continuity to a future where total resiliency is a way of life.
Michael’s contact information:
1.310.606.0291
Leveraging the Texas Power Outage to Add Business Value
Michael Puldy
CEO, Puldy Resiliency Partners, LLC