Does your business continuity plan handle a derecho?

First of all, just what is a derecho? It's an unusual type of storm stretching over a wide area. The storm we got hit by caused widespread damage that extended over a path 1,000 km (600 miles) long. The meteorology and physics behind it are quite complex (Wikipedia has a long and barely comprehensible article explaining derechos), and they are sufficiently uncommon and serious enough that Wikipedia also has a list of derecho events.

The experience is somewhere between being hit by a minor tornado and a hurricane. The important planning points are:

• Short notice: we received storm warnings less than two hours before the full force of the storm hit.
• Wide area: damage to overhead power lines etc. is spread over a wide area, making recovery by power companies slow due to the sheer number of separate outages.

This storm raked as the sixth costliest in Canadian history: over 1,900 electricity poles were destroyed and over a million customers were without power, many for several days. (Wikipedia again).

Some important observations:

• The notice period was short (and the storm occurred during a weekend) giving businesses inadequate time to perform any special preparations. If your business continuity plan assumed that you would have time to fully secure outdoor assets before a severe storm, it wouldn't have worked. This is in contrast to a hurricane or tropical storm, which may be tracked for days before it hits you.
• Major power and telecom infrastructure was recovered within a couple of days: less urgent infrastructure took two or more weeks as there was insufficient capacity to make the needed repairs.
• Cell phone and landline infrastructure failed, sometimes as a result of storm damage, but often as a result of lack of electrical power when backup power supplies were exhausted.
• Local government relied on the internet to provide emergency information to people: this didn't work very well without working telecom infrastructure.
• Local radio isn't local at weekends. Only one channel broadcast here had any useful news. Local (and national) radio staff aren't employed at weekends and the prepared programming carries on regardless of what is happening in the real world.

The second-order effects were quite interesting:

• Most gas (petrol) stations closed, lacking any backup power.
• Supermarkets closed, lacking backup power to either run checkouts or refrigerate stock.
• Without adequate backup power, water treatment plants and airport services were affected.
• Deliveries were disrupted, due to a lack of fuel for delivery vehicles.
• Electric vehicle owners could not recharge vehicles.
• Emergency generators were initially hard to obtain. (Two weeks later a lot of slightly used generators were available at a discount).
  
• We had an internet outage for a number of days which started the day after the storm.

The point about delayed internet outages when (locally) power had been restored is especially interesting. A report report by the Californian Public Utilities Commission explains why. Due to the cost of provisioning generators and storing fuel on site, the (typical) requirements placed on companies are:

• 24 hours of fuel storage at a Central Office (telephone exchange).
• 4 hours minimum power backup at a remote site, with a recommendation of 8 hours for a more critical site. (If you are more than a short distance from the exchange, you are probably connected to one of these).
  

The assumption made is that the company can successfully deliver additional fuel (or install an emergency generator) within these time periods. It's easy to see why this is an unrealistic expectation in the event of a widespread disruption like a derecho.

What should we learn for planning purposes?

• Don't assume adequate time to prepare for a major storm. Forecasting just isn't that good. Keep outside areas free of anything that will be converted into a missile during high winds. Inside, keep critical cabinets, doors, etc. closed when not in use to reduce losses if the building is damaged by extreme weather.
• Be prepared for an extended power outage, particularly if you won't rank highly on your power company's triage list.
• The logistics of obtaining additional fuel for generators may be disrupted: this is a mitigation cost / risk tradeoff.
  
• Be prepared for telecom outages due to infrastructure power loss in the hours following a wide area power disruption, even if you were initially unaffected.