Planning for the Worst

Comments

Reader ROI

Read this article to learn

-Why no company can afford to be without a Y2K contingency plan -Why Y2K contingency planning should be led by senior business executives -Specifics of real-world contingency plans Better safe than sorry.

That's one of childhood's earliest lessons, and the reason for bicycle helmets, seat belts and the moratorium on running with pencils. Like the errant pencil-wielding child, though, the state of most Y2K projects is that of an accident waiting to happen. Project teams are hurtling headlong through remediation and testing and doing little to prepare for Y2K mishaps that could sneak past their corrections and trip up their business. Contingency planning has -- perhaps tragically -- been relegated to a back burner issue.

When Y2K remediation projects were first conceptualised, the path seemed quite clear: from awareness to triage, then on to repairs and replacements, followed by testing and finally contingency planning for the unlikely event that anything was missed. With less than nine months until 2000, such a linear approach to the problem is no longer a viable option, and companies cannot afford to wait until repairs and testing are finished to consider backup strategies. Although fixing noncompliant systems must still be the primary focus, companies have no choice but to approach Y2K from both sides of the problem: preparation and recovery. Contingency planning is as critical for the company with a project nearing completion as for the company lagging behind.

Even companies whose systems appear to be 100 per cent compliant face risk and need to prepare for any eventuality. According to a 1998 study by International Data Corp (US), about half of CIOs planned to finish their year 2000 preparations in 1998; 44.5 per cent planned to finish in 1999. Considering that historically almost 75 per cent of IS projects miss their deadlines and that tight Y2K project schedules leave little or no room for slippage, the potential for casualties is great. And even if you are done, there's a good chance your suppliers -- or your suppliers' suppliers -- won't be.

Many companies already have disaster recovery plans in place detailing an emergency response routine for fires, floods and similar catastrophes (see, "Plan of Attack", page 26). These plans make a good starting point for Y2K contingency planning, but companies should not rely on them exclusively. Y2K creates vulnerabilities that make it different from the run-of-the-mill disasters most companies are prepared to handle. Recovery plans are usually designed to assist with localised business disruptions like power outages or flooding caused by weather-related events. The problems likely to occur as a result of Y2K are pandemic as opposed to localised, and companies have to be ready to deal with simultaneous problems on several fronts. Therefore, companies need to create contingency plans specific to Y2K risk. Thomas McAndrew, vice president and managing director of the national Y2K practice for Computer Sciences Corp's Consulting Group in Waltham, Massachusetts, drives this point home with a hypothetical example of a manufacturing company that has multiple locations. If a hurricane knocks out power and services at one plant, the company's disaster recovery plan might be to temporarily reroute business through another plant. "In the case of Y2K, however, if an embedded chip takes out one plant, chances are that all 10 are going to go," he explains.

Contingency Planning Teams

Mobilising the business community is an immense challenge, observes William Ulrich, president of Tactical Strategy Group in Soquel, California, and author of two books on Y2K issues. "CEOs and CFOs need to have meetings with their entire staff of direct reports to make this a number-one priority," he says, "and they need to ensure that there is a business unit owner for each group to champion it." The key contacts for each business unit should collectively form the company's contingency planning team and be responsible for creating plans, communication within their units, plan maintenance and understanding when the contingency plans should be triggered. This group of people should also transition into a rapid response team that deals with problems that arise before or after the date change. The team should include representatives from all the disciplines in the enterprise, such as IS, facilities and operations as well as representatives from individual business units. When a failure in your network has global implications, the pressure to have every contingency covered is pretty intense. John Pasqua, vice president of the Year 2000 Program at AT&T in New York City, explains that AT&T will have "business resumption SWAT teams" on call during the changeover so that if a problem occurs they can isolate and resolve the situation right away as opposed to waiting until the first regular workday of the new year.

In many cases, contingency plans will be enacted by employees rather than executives, so it is important that employees are informed about the plans that are in place and roles that they may be asked to play. To ensure that the necessary leadership is in place when the clock turns, one company is actually dispatching key staff members to various locations throughout the organisation so that there will be a corporate officer to take charge in the East, West and Midwest in case of significant business disruptions.

Prioritise the Enterprise

Once the contingency planning team has been formed, it must first prioritise the business's infrastructure, which includes systems and key suppliers.

McAndrew recalls a contingency planning discussion with top executives of one company: "We met with six vice presidents and directors of various business departments within the company and asked them to identify the central process for their company. They came up with four different answers," he says. In order to begin the process of contingency planning companies must first decide, at the highest level, what areas of the company must be functional on 1 January 2000, and then create a plan that provides backup options for the systems, departments, people and technologies within those areas. For example, at PaineWebber in New York City, executive vice president and CIO Scott Abbey notes that the company has prioritised the relationships with all its business partners by their strategic importance to PaineWebber's main business functions. A company that has a high-impact business relationship with PaineWebber as well as a high risk of failure will be monitored very closely by the company (if not cut off). "We have already identified the relationships and categorised them by risk: high impact or low impact," he says. "We are in the process of developing contingency plans for high-risk and high-impact partners." This assessment applies to internal functions and processes, not just outside relationships.

Capers Jones, chief scientist of Artemis Management Systems in Burlington, Massachusetts, recounts the true story of a major pharmaceutical company that encountered a quintessentially high-impact Y2K problem. At the pharmaceutical production plant, the system that tracks expiration dates for drugs was thrown by the date change. The company was making batches of drugs, putting them into inventory the same day and then destroying them at night because the tracking system read the batch's expiration date as having passed. The problem was not as ruinous as it could have been because the company could focus all its efforts on fixing it. However, if this situation had cropped up post-Y2K alongside several similar malfunctions within the company, it could have been disastrous.

When assessing the importance of a given system, companies also need to consider how long they can afford to have that system down. If downtime of more than 30 days will cause a significant slowdown in the business, then the company needs a solid plan for coping with the problem -- and that plan must have a clearly defined trigger date when it will kick in.

Start at the Top

In many companies the Y2K project falls under the umbrella of the IS department, but contingency planning is effective only when run by corporate executives. The lack of high-level leadership in contingency planning results from the lingering notion that Y2K is largely an IS problem and the reluctance of many executives to deal with the ramifications of a Y2K-related failure.

Apart from a few CEOs in complete denial, most executives have accepted the fact that Y2K is a serious business issue first and an IS problem second. That distinction is especially important when developing contingency plans. In order to assess which systems and suppliers are business critical, the executive staff has to work very closely with the individual business units and maintain strong lines of communication. When executives are not involved in the process, budget battles over Y2K contingency planning are likely to get bloody.

"If implementation of the plan requires upfront expenditures of $US30 million, for example, that's where the sell gets tough," says John Dougherty, director of assessment and contingency planning for EDS's CIO Services in Plano, Texas.

Project leaders include executives only in the financial discussion, they can't expect them to understand the larger issues at stake and the need for contingency plans.

Chicago-based consultancy A T Kearney has focused on demystifying the year 2000 for executives. Joel Goldhammer, vice president at A T Kearney, points out that "inside companies there is a lot of management of information going on rather than management of the process." He compares it to the children's story about the emperor's new clothes. CEOs and CFOs are getting worried about their lack of information about year 2000 and the validity of the information that they present to the board. "[Executives] basically want to know if they have clothes on or not," says Goldhammer. Instead of debating the legal ramifications of who knows what, companies should be focused on creating well-designed contingency plans and communicating them to their employees and suppliers.

Develop Workarounds

Often, contingency planning will involve the gross simplification of a process rather than a technical solution. For example, AT&T is highly automated, and so the company has had to explore alternatives to many of its systems. For the company's billing, order entry and provisioning, Pasqua's Y2K team (which consists of hundreds of employees across the company) has prepared alternatives like the use of manual workarounds for which they have selected and trained certain employees. Businesses may find themselves reinstituting manual processes for a period of time after 2000, and it's important to train personnel in advance. If a company's payroll system goes down, for instance, payroll staff needs to know how to issue cheques manually while still maintaining an accurate and orderly accounting system. Companies will find that many of their employees are unfamiliar with the practice of creating a paper trail that actually involves paper. Part of contingency planning is simply looking for ways to minimise the risk of failure. AT&T will be taking advantage of the cyclical nature of the demand for telecom services by deactivating and checking parts of its network during low usage times. Because 1 January 2000 falls on a Saturday and network usage tends to decline over the weekend, AT&T will be able to look at systems that are usually inactive at that time to ensure that they made it through the date change without any problems. If a problem has occurred within one of the systems, engineers can fix it without ever having to bring it up, thereby mitigating the risk of a wider failure.

Pasqua also notes that in some cases the company will simply eliminate the need for systems to process cross-millennium transactions by deactivating them just before the date change and reactivating them at 12:01am, 1 January 2000. If system A sends a transaction to system B during the rollover, system B must use a logic function to recognise that the transaction was initiated in 1999 but completed in 2000. Although both systems can process the date correctly, shutting down the system and restarting it provides an additional safeguard because that logic never has to kick in. As simplistic as shutting a system down and then restarting it after midnight may seem, it is a way for AT&T to reduce the risk of problems. "We believe that the systems have been [fixed], but unburdening them is a form of risk mitigation," says Pasqua.

In assessing its date-dependent systems AT&T has faced the same options as other companies: retire the systems, repair them or replace them. However, in situations in which it has decided to go with a replacement it has also developed contingency plans in case the project falls behind. "If the [replacement] plan is starting to slip," Pasqua says, "then we will repair the system on a parallel path [with the replacement] just in case the system is not available and compliant when needed. It's a duplication of effort, but we have to be ready for Y2K. We will abandon the repair if the replacement is ready." Get Suppliers Prepared The maxim that a supply chain is only as strong as its weakest link injects uncertainty into the most carefully crafted Y2K strategies. And for many companies, handling the issue of supplier compliance is the toughest part of contingency planning. As General Motor's 1998 strike illustrated, companies are increasingly at the mercy of their suppliers. Within weeks, the strike at two parts plants forced GM to shut down 25 of 29 assembly plants because they couldn't operate without the parts. Since a midsize company has tens if not hundreds of external service providers, the level of exposure to potential Y2K problems is enormous. Many upper-level IS executives will opt to cut off suppliers that are not Y2K compliant. For example, Sandy Lutrario, IT director of year 2000 for Xerox in Stamford, Connecticut, says that if her company's suppliers do not have compliant versions of their product to her by set dates, Xerox will request that those products not be used on its desktops worldwide.

One contingency option is to have a backup supplier for each important service, a go-to company that can jump in and pick up the slack if the original supplier fails. Like many companies, AT&T is having its alternate suppliers deliver compliant versions of their software products to be tested along with the product of AT&T's current supplier. AT&T is also doing extensive testing with other carriers and customer groups in the telecom industry so that if other carriers appear to be at risk, AT&T can be ready to expand its capacity to absorb traffic from other suppliers. Companies that are considering contracting backup suppliers need to make those arrangements as soon as possible; resources in the marketplace are drying up, and as year 2000 looms closer they will undoubtedly get more expensive.

Companies should avoid forming too strong a dependence on any one external supplier whose Y2K noncompliance could bring down the business. In cases where the perceived risk is great enough, some companies are arranging for multiple service providers for each area of its business and each geographical location.

Another possible option for mitigating the risk posed by a key service provider is to "stock up", if possible. For example, one large bank has plans to beef up its inventory of cheques before the date change because the company that normally provides its cheques uses a digitised printer, and bank executives fear that reorders may be slowed if the cheque company experiences a problem.

Ulrich of Tactical Strategy Group cautions organisations to look at their contracts with suppliers very carefully. He notes that almost all supplier contracts have a 10 to 30 day "cure" period, during which client companies are legally required to allow the supplier to fix their systems and resume service before terminating a contract. In cases where 10 to 30 days is too long and the supplier's service is critical to the business, legal staff may have to step in.

Y2K experts deride the compliance letters companies bombard their suppliers with, which often receive little or no response. Compliance letters in most cases are driven by the legal department, which hopes that the letters will stand up in court as proof of due diligence. For companies that are especially interested in the status of their suppliers' projects, Vin Culhane, CEO of information services at Willis Corroon, an insurance brokerage and consulting services firm in Nashville, Tennessee, says that a meeting will get the job done more efficiently. "We have identified 16 carriers that represent 65 per cent to 70 per cent of our business, and we are visiting those companies face to face," he says. "We take three or four hours, we present our program and they present theirs, and it gives us a much greater sense than a letter would of how seriously they are taking the problem." Consider Your Infrastructure Problems that arise from software, hardware and other IT systems are not the only issues that contingency planners need to consider. Y2K has the potential to cause large infrastructural failures. Companies need to prepare for potential lapses in their power and telecom services and the effect that will have on their employees and customers. Brian Wengenroth, a vice president in the IT Group at Booz, Allen and Hamilton in New York City, recalls the brutal ice storm in Canada last year. "People were running out of cash because there was no power [to run banks and ATMs], and that speaks to a larger problem," he says. "Had the storm started 70 miles west it would have wiped out the banking systems in Canada by hitting Ottawa and Toronto. Companies need to deal with the potential for systemic failures, not just point failures." A T Kearney's Goldhammer has a client in Eastern Europe where concerns persist over whether the utilities will be able to service their customers. "We have a manufacturer whose product must be kept refrigerated. How are they going to keep stuff refrigerated if the power grid doesn't come up?" A T Kearney's client plans to have refrigerated trucks standing outside each of its plants during the date change in case the power goes out. While this solution may be the right one for that manufacturer, for another company the expense of hiring trucks to wait outside each plant might outweigh the risk. In short, contingency plans should balance the risk of a failure with the cost of a backup.

To circumvent potential communications problems, one insurance company plans to have an offsite command centre with multiple means of communication. It is setting up a network of diverse telecom routings with several different long-distance carriers as well as radio connections in an attempt to guarantee that the company's communications won't go down. Again, if communications are critical to the business and the risk seems high, it may be worth the cost.

Companies also need to ensure that that they have contacted the various vendors that provide critical building services like security and HVAC to make certain they will be able to provide additional services if necessary.

Prepare for the Halo Effect

Adding to the legitimate Y2K-related problems companies will contend with is the likely boom of customer service calls and other indicators of the halo effect. Companies may want to consider hiring extra staff for help desks or customer service centres. Booz, Allen's Wengenroth adds a cautionary note: The influx of calls could pose more than a simple annoyance; if transactions are handled over those lines, it could cost the company. He suggests that companies rework automated telephone routing systems to deal with the calls. "If your call centre is also used to process transactions, you may want to change the process so that paying customers don't get caught up in the rush of [worried] little old ladies calling." Another insurance company plans to add staff to its customer service division for the release of account statements around 2000. Aside from worrying whether their preparations will prevent all system problems, company leaders anticipate that public awareness of Y2K is going to be so high that people will be calling about their statements and questioning things that have always been there.

Join Industry Groups

Like other companies, AT&T sees security in numbers and participates actively as a member of industry groups to keep attention focused on the year 2000 problem. The Network Reliability and Interoperability Council (NRIC) is a telecom advisory board to the FCC and government comprising 40 to 50 of the largest providers in the industry. Every few years it is rechartered with a different focus and a different chairperson; currently the group is concentrating on preparations for Y2K. NRIC members are focusing specifically on compliance issues and assessing the readiness plans of all the major players in the industry, reviewing interoperability testing and looking at what the industry is doing individually and collectively to develop contingency plans.

Pasqua stresses that everyone has a stake in helping one another make it through the year 2000 successfully. "We don't look at Y2K as a competitive opportunity. We want to do what we can to help [other companies] and then move forward and compete on business terms." Question Your Assumptions Contingency planning requires modelling fairly elaborate scenarios. Larry McArthur, president and CEO of Ascent Logic in San Jose, California, tells about a company that designed centralised data processing systems for banks in Europe. The company needed a new power-generation system that would allow it to keep working independently of the power grid for 30 days. During the plans, McArthur asked who occupied the building next door to the data processing company. It turned out that the company's neighbour was a hospital that had only 24 hours of fuel. "What do you think would happen if the hospital ran out of fuel?" McArthur asks. Knowing that it would come under pressure to share its fuel with the hospital in the event of an outage, the company ended up buying a fuel-reserve system for the hospital. The more companies get into the process of contingency planning, the more risks they see. But they can no longer wait to prepare for possible failures. As AT&T's Pasqua observes, "Time is our worst enemy. Trying to solve the problem in a serial fashion-contingency planning after remediation -- is adding risk to your overall program; it's the wrong approach. Every day we are a day closer, and waiting to work on contingency planning is a dangerous trap."After the Ball Drops: A One-Year Contingency Y2K management efforts won't end at the stroke of midnight on 31 December, 1999. Capers Jones, chief scientist of Artemis Management Systems in Burlington, Massachusetts, created a checklist of tasks companies must be prepared to undertake after the clock has turned.

January 2000 Immediate Damage Control

- Monitor all critical software systems, database queries, suppliers and contractors for year 2000 compliance - Correct physical problems (elevators, security systems, etc) - Deploy manual backup methods if needed - Perform emergency Y2K repairs for previously undetected problems - Correct client-reported Y2K problems - Notify year 2000 repair vendors of problems that they missed - Analyse legal and liability status of the company and its directors and officersNow Till July 2000 Damage Assessment and Litigation Filing - Prepare for possible filing of lawsuits by clients, shareholders, employees and tax authorities for year 2000 problems - Replace temporary system repairs (that is, date encapsulation) with long-term fixes - Phase out manual backup methods if possible August Till December 2000 Recovery and Replacement - Litigation disclosures and depositions begin - Deploy hardware upgrades to restore lost performance - Tally overall year 2000 costs with estimates for remaining costs - Disclose overall year 2000 costs in financial reports to shareholders - Resume some suspended software projects