Is DR Planning Better After 9/11? Hurricane Irene Shows that Companies Have a Long Way to Go

By Zerto, on 12 September, 2011

The 30-second post:

  • Virtualization makes DR more difficult to implement
  • Some companies used Hurricane Irene as a test-case for failover to a DR site (seriously?!)
  • Others relied on backup power without a real DR solution
  • We feel hypervisor-based replication is an easy and cost-effective way to be prepared for any disaster scenario

The complete story:

As a company that builds disaster recovery solutions, Zerto has some understanding of the post 9/11 task that the WTC-area companies went through to get corporate life back in order in those terrifying days after 9/11.  The task of rebuilding after a disaster is challenging and frankly, has gotten more challenging in the 10 years since 9/11, mostly due to virtualization.

In fact, the larger the enterprise, and the more virtualized an enterprise is, the more complicated their ability to adequately do DR.  Applications are managed and provisioned as virtual machines (VMs) and virtual disks (VMDKs) – but existing storage-based replication and disaster recovery technologies are still LUN and volume oriented. This results in complex, labor-intensive setup and management of DR.

This problem is most striking when looking at the coverage from Hurricane Irene that hit the East Coast of the US two weeks ago.  According to a recent interview with Forrester Research Analyst Rachel Dines in SC magazine:

“Many of the New York and Boston-based clients she has spoken with this week plan to use the event as an opportunity to gauge the effectiveness of their disaster recovery (DR) and BC programs.

“They’ve told me they’re treating this somewhat as a test scenario,” she said. “You know it’s coming, so there are some things you can do to be ready.”

This may include preparing systems to fail-over to a backup data center in case of an interruption, such as flooding or a long-term power outage, Dines said.”

Frankly, we feel that companies need to have regular test failovers long before disaster strikes, and we’re surprised to hear that established companies would use a hurricane as an “opportunity” to do a failover test.  Zerto’s customers do regular failover tests, long before a crisis – even a small one – occurs.

Other companies choose a different route, using backup generators to power their production data center.  This is another DR solution that seems ineffective.  According to Mike Vizard, in an article on IT Business Edge,

“…a lot of frustrated business owners will be looking into the … merits of buying a generator.”

He mentions that downed power lines caused major work disruptions on the East Coast, and many took a full week to be fixed by the utility companies.

“The business lesson to be learned from this is that public utilities can’t be counted on in any kind of major emergency.”

Generator power as a DR plan seems to be popular.  Many companies use generators to keep their offices open and their data center running in a disaster scenario.  According to an article on datacenterknowledge.com:

“One of the areas that experienced widespread power outages was Richmond, Virginia, where [company name removed] operates a large data center.

“As Hurricane Irene made landfall, Dominion Power suffered extensive outages,” said [the company]. “However, the company’s Richmond Data Center transferred from utility to generator power without impacting our customers. Diesel deliveries will continue to fuel the data center’s generators until electricity is restored, ensuring our customers’ mission-critical data is secure.

“Dominion Power has assured us they are working to restore power to the company’s Richmond Data Center as soon as possible. In the meantime, we have scheduled fuel deliveries to keep our generators operating efficiently.”

Unfortunately, we feel this is one of the least reliable disaster recovery planning tactics for data centers.  Most obvious is the fact that a true disaster – fire, earthquake, tsunami, major hurricane – could wipe out a company’s offices, networks and even employees.  In such cases, the fact that the power is still on is the least important part of the company’s ongoing functioning.

Plus, generators rely on diesel fuel being trucked in to keep the generators running, which is costly and dangerous.  One building, 60 Hudson St. in New York City, nearly had a major disaster on its hands when a small fire broke out.  The building, a hub for telecom operators in NYC, stores 80,000 gallons of diesel fuel in the basement to enable generators to run in case of a major power outage.  One small fire in 2010 nearly created a far bigger disaster than any random power outage could.

Generator power as a DR plan seems to be popular, but it’s a foolhardy solution to use in order to keep a data center running when there are easier and cheaper solutions to enable fail over of company data to a safe location.

So what’s the right solution?

Mike Vizard, in the article mentioned earlier, thinks more companies will rely on the cloud for their DR plan.

“Most providers of cloud computing services expect that Hurricane Irene will wind up being a boon for cloud computing services. After all, nothing demonstrates the need to be able to access remote servers better than massive flooding and a loss of power that prevents people from returning to work.”

We agree.

By using Zerto’s hypervisor-based approach to disaster recovery, companies can simply and cost-effectively migrate their data to the cloud, and continue to work remotely in the event of a serious disaster scenario.  In fact, a cloud provider need not use the same storage vendor in order to serve as a company’s replication site.  Zerto offers the ability to mix storage from any vendor using any technology. And instead of using Hurricane Irene to test recovery (!), with Zerto, failover testing is performed at your convenience, with the click of a button, and has no impact on the production environment.

Time will tell what the impact is of hurricane Irene, in the meantime, we’re pleased that Zerto’s customers were not impacted at all, whatsoever.