Devastation caused by Hurricane Sandy forced at least two major hospitals and a data center in lower Manhattan to resort to backup generators fueled by diesel for power.
In these cases the backup processes failed.
In two of the outages, bucket brigades were formed to carry diesel fuel up many flights of stairs to feed generators. At the two hospitals, hundreds of people, including patients in critical condition and mothers with newborn babies, had to be evacuated.
One of the more ironic cases occurred at Bellevue Hospital, located along the East River in lower Manhattan, where excessive water caused a backup pumping system to fail even though it had been sealed to protect against flooding.
Bellevue, a public hospital, had to evacuate hundreds of patients on Wednesday after floods caused the failure of pumps that were moving diesel fuel from the basement to backup generators on the 13th floor.
At one point, the National Guard was called in to mount a bucket-brigade to lug the diesel fuel up the stairs to the generators.
The hospital clearly had a working continuity plan with the backup generators when it lost power on Monday from Con Edison,
However, the diesel pumps shorted out from floodwater even though they were encased with submarine doors that had rubber gaskets, said Alan Aviles, president of the Health and Hospitals Corp., which runs Bellevue.
"It's easy to judge from a ringside seat ... but this [kind of storm] has never happened before," Aviles said when asked about the adequacy of the hospital's continuity planning on CNN's Piers Morgan Tonight show last night.
Bellevue officials will investigate exactly how the electrical short occurred, Aviles said on the show.
Aviles and other Bellevue officials did not respond to requests for further comment.
New York University's Langone Medical Center in lower Manhattan also evacuated patients, including newborn babies, on Monday night after its backup generators failed. Con Edison had preemptively shut off power to some parts of the lower island, and the utility also experienced an explosion at one substation in the area, according to reports.
Meanwhile, a data center at 75 Broad Street in lower Manhattan operated by Peer1 Hosting also faced a shutdown after diesel pumps in the building's basement failed due to flooding caused by the storm.
However, data center customers and others showed up to form a bucket-brigade to get the fuel to the rooftop generator on the 17-story building. The brigade was able to transport enough fuel to the rooftop to keep the data center running.
Continuity experts say that diesel pumps should be located high enough to stay operational during floods. "The diesel tank pumps probably should have been higher, but nobody anticipated this much water," said Al Berman, president of DRI International, the oldest disaster recovery education and certification organization with operations in 100 countries.
He noted that sealing the pumps, as was done at Bellevue, "never get tested either,"
"There was no way to anticipate this storm's damage two weeks. [Bellevue] did what they could," Berman said in a telephone interview from Manhattan.
He suggested that a test of the sealant around the pumps could have made a difference, perhaps using water from fire hoses poured over them. "Unfortunately, we all test under the best circumstances and we execute under the worst," he said.
In lower Manhattan and especially at the two older hospitals, "you're looking at a lot of antiquated infrastructure," which probably contributed to problems there, Berman said.
In past years, offices and hospitals on Manhattan would rely on backup generators being turned on by Con Edison to guard against brown-outs and black-outs. Today, many organizations can turn on their own backup generators as needed, Berman said.
Cell phone towers have long operated mostly with two-hour backup batteries, though wireless carriers are now realizing the need for portable generators at cell sites, he said. At the height of Hurricane Sandy, about 25% of all cell phone towers in 10 states hit by the storm had lost service, according to the Federal Communications Commission.
The lesson from Hurricane Sandy, Berman said, is that hospital accreditation organizations and businesses need to stick to the basics of continuity planning and to revise testing procedures.
"Somebody should say, 'What's the backup?.' We should also change the way we do testing and the scenarios for testing," Berman added.
"You're seeing in Manhattan that people are living through something they never lived through before, and the learning process is costly," he said. "Fortunately, the hospitals did a great job with the evacuations. We learn a lot of lessons from disasters and some are tougher than others."
Matt Hamblen covers mobile and wireless, smartphones and other handhelds, and wireless networking for Computerworld. Follow Matt on Twitter at @matthamblen, or subscribe to Matt's RSS feed . His e-mail address is mhamblen@computerworld.com.
Read more about data center in Computerworld's Data Center Topic Center.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.