The Contingency Plan That Wasn't
By traditional IT contingency planning standards, HP had already gone beyond the call of duty. Most IT contingency planning focuses on preparing extra code rather than extra products. Convention says that to ensure a problem-free transition to a new system, there should be a redundant system ready to handle things if the rollout goes sour. At the very least, there should be a "rollback" strategy to go back to the old system if there is an issue.
But both Bouchard and Hanger maintain that neither strategy would have worked in HP's case, considering the scale of its server business. "If we had had two systems running, that would have meant that every supplier would have had an order for us in the old system and the new system," says Hanger. "When it was received into the manufacturing line, it would have had to be received twice and then [be reconciled] twice." Says Bouchard: "You would have created a backlog of orders because of the cumbersomeness of the duplication effort."
What HP should have done was to create a plan for taking orders and shipping products that assumed the IT system it had planned to use didn't exist. "Contingency planning is not about IT," says AMR Research's Swanton. "It's [about] having people who are watching what's happening, able to detect if there's a problem and working out some simple manual way around it until you're ready to work with the system. If that takes a team working a bunch of overtime, fine. It will be a lot less disruptive than losing sales." Taking your order-to-cash process back to the 1950s requires a mind-set change among all the people involved in that process, from customer service representatives to warehouse clerks, because it will feel silly and unnecessary - sort of like imagining what will happen if the sun doesn't come up tomorrow.
Next Time, Imagine the Worst
Even extending HP's business contingency plan to bank an additional few weeks' worth of servers might have seemed risky because those extra servers might not have sold. In most companies facing such decisions, CIOs feel more comfortable trying to eliminate project risk by perfecting their project management skills. This route appeals to our optimism and quest for competence, says Robert Charette, president of Itabhi, a risk management consultancy. Business contingency planning, on the other hand, is gloomy and expensive.
But if companies are ever going to perfect IT project management, Charette says, it will require a revolution to overthrow the principles that rule most IT projects: budget and schedule. "Cost and schedule are what let people rationalize away crucial pieces of project management like application testing and training," he says.
Worse, a cost-and-schedule approach never holds up during a crisis, says Charette. When HP saw that the order management system wasn't working properly, it pulled out all the stops to get the code working properly - cost and schedule be damned. Charette says when using this event-driven approach, "the project doesn't move forward until you've got each step right". Yet event-driven project management has its own pitfalls. It's not infallible, and if not carefully managed, projects can drag on forever.
Here's what Bouchard would have done differently. He would have expanded the contingency plan to cover five or six weeks. Instead of trying to prevent IT problems that were too small and rolled up in too many strange combinations, it would have been easier to bring the backup factory in Europe online earlier and stockpile more generic servers. In his interview with CIO, he did not address whether additional manual workarounds, such as a manual order-entry process, would have helped.
If that sounds like passing the buck, Bouchard is only passing it from one hand to the other. In December 2003, he became CIO and executive vice president of global operations - one of those rare CIOs who also runs the supply chain. Besides emphasizing business contingency planning more strongly, he is in the process of reorganizing the operations and IT groups of HP's businesses. Bouchard replicated his dual role at the regional level too. Hanger runs both IT and operations for the Americas. The more consolidated approach should improve communication between IT and the business, he believes, and make it easier to identify how IT projects affect operations.
One message that needs to be communicated more strongly within HP - and within every company these days, Bouchard believes - is the message that is implicit in his dual role: "There is big leverage between IT and the business processes when you deal with a large supply chain," he says. "Just looking at contingency planning from an IT point of view would be a big mistake. It has to be looked at from an integrated view of IT processes and the business.
Three Steps to a Business Contingency Plan
- CREATE A CROSS-FUNCTIONAL TEAM to engage business people and educate them about the supply chain risks of a major system rollout.
- DEVELOP A TRANSITION PLAN to the new system that assumes the system will fail during final rollout. Create a conservative time estimate for the period it could be down.
- DEVISE MANUAL PROCESSES for keeping orders and deliveries flowing during the problem period. Have extra people and factory capacity on call to handle the extra workload.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.