Enterprise Software Upgrades: Less Pain, More Gain

Prologue: The Scream

Upgrading software from one version to the next appears like a small step - after all, you're not ripping the stuff out and replacing it, you're just improving it a little bit - but Nextel senior vice president and CIO Dick LeFave finds himself about to step off a precipice with his Oracle ERP software upgrade.

The numeric change - from version 10.7 to 11i - sounds minor, but in this case the numbers and letters lie. "This isn't just doing an upgrade; it's a whole new system," says LeFave. "It's a different design; it's a whole different set of solutions; it's a whole different way of doing business. You may as well start out from scratch."

This is not what LeFave expected. And he's not alone. Denise Quinlan, an assistant vice president and PeopleSoft product manager at Boston-based MFS Investment Management, is still recovering from the sticker shock she got when her ERP vendor quoted her the price for dispatching its consultants to do an upgrade. When Quinlan heard the estimated cost to go from version 7.5 to version 8 - $US490,000 for the HR software and $US872,000 for the financial software - she blurted, "For an upgrade?"

For PeopleSoft's part, a spokeswoman says the price quote was preliminary and that there were no discussions about using MFS staff to reduce costs (see "Vendors: Â'It's Not That Bad'", page 49).

CIOs who have to wield the wrench in these efforts are shocked when they look under the hood of major software upgrades, not just from Oracle and PeopleSoft but from JD Edwards, SAP and Siebel - all the enterprise software vendors - and see that they're facing overhauls, not tune-ups. The CFOs and CEOs who foot the bills for these upgrades are equally nonplussed. They ask questions like, Didn't we just pay millions for this stuff?

Yes. Between $US40 million and $US250 million for an enterprise software system for a Fortune 500 company, according to AMR Research.

And the CFOs and CEOs want to know why they're paying again.

Why It Hurts So Bad

This story focuses on ERP users, but the same problems and advice applies to CRM, supply chain management systems and other major enterprise software products. They have become so complex, so expensive and so integral to business processes that the term upgrade is a misnomer. Enterprise software upgrades can cost up to 30 per cent of the original software installation price, according to Gartner, take more than a year to complete and require companies to revamp their technology infrastructures and business practices. CIOs have to present a strong business case for why, in these difficult economic times, their company should go through the trouble and expense. A tough sell to most corporate boards.

Three factors further complicate the upgrade process.

1: Upgrades are unforgiving when it comes to customisation.

Enterprise software is the one-size-fits-all suit that you tailor to be a 38-short on one side for your manufacturing group and a 54-portly on the other side for your salespeople - along with some extra pockets from other vendors stitched onto the suit. Don't expect the vendors to touch that suit with a 10-foot needle when it comes time to upgrade. It's up to you to redo the customisations and connections with third-party software on the new version. Customisations that need to be carried over from one version of enterprise software to the next are the biggest technology headache and ROI killer that CIOs face in upgrades.

Page Break

2: The move to Internet architecture changes everything for the IT staff and end users.

All the big vendors have moved their software from a client/server architecture, in which PC-based software accesses a central server through a company's network, to an Internet architecture, in which a Web server joins the mix and the software is accessed through the public Internet and a Web browser. It's new turf for both IT staffers and end users. Staffers suddenly must see the Internet as the platform for the company's most important applications. End users must learn different ways to access programs and absorb all the business process changes that come when they can collaborate with people outside the company. These added capabilities are a good thing, of course. But the planning and change management that is required make them a migraine minefield.

The desupport date is the ticking time bomb of enterprise software upgrades. A vendor says, We won't support version XYZ of our software after this date. Technology changes and customers - especially new ones - demand fresh functionality. Vendors can't afford to support six different versions of their software simultaneously.

But CIOs object to what they call surprise desupport announcements that most major vendors have made in the past few years. They say they aren't getting enough time to do one upgrade before a vendor announces a desupport date for an older version.

Nextel's LeFave has been through this. "When the vendor comes to you and says the product is at the end of its life cycle, that's code for saying, I'm going to be putting my maintenance and development resources on a new product, and you're not going to get anything unless you pay for it yourself," he says. Even if the vendor continues to support old software versions, it will shift the bulk of its people and resources to new versions so that finding someone knowledgeable about your old version becomes like trying to find a department store salesperson on red-tag clearance day.

"I'd like to have a road map of when upgrades are planned and when to expect them so that we can do our planning better," says LeFave.

The average time between upgrades has shrunk from three years in the early 1990s to 18 to 24 months, according to AMR Research, and CIOs have lost the ability to keep up. "Vendors are pushing new code out as fast as they can - so rapidly that you may have updates coming at you almost monthly," says Pat Phelan, an ERP analyst for Gartner (US). "The vendors don't seem sensitive enough to the fact that the average buyer can't absorb that kind of change."

With all these difficulties, a few brave CIOs are fighting to push back the desupport dates. In July 2001, 58 members of the 2200-member independent Oracle Applications Users Group (OAUG) signed a petition urging Oracle to extend the support date for version 10.7 of its ERP software from June 2002 to December 2004. This petition came after Oracle had already extended the desupport date for 10.7 from June 2001 after customers complained bitterly about all the bugs - about 5000 of them, according to customers - that appeared in the initial release of the 11i software in June 2000. (Oracle declined to comment on the number of bugs in 11i.) In the end, Oracle and the OAUG compromised, and the desupport date was extended to June 2003. The other major enterprise vendors - JD Edwards, PeopleSoft, SAP and Siebel, which, like Oracle, have released ambitious upgrades of their software in the past few years - have all extended desupport dates for previous versions in response to customers' complaints about bugs and performance. (In addition, some companies have found third-party support for software after a vendor-announced desupport date.) Download These Scream Savers.

CIOs have two choices when it comes to upgrades: go along or fight. But both options require more planning than most CIOs do now. Even if upgrades aren't a continuous process (and some overworked CIOs may think they are), planning for them must be. It's the only way you can make upgrade decisions without your back against the wall of a desupport date. We've uncovered best practices for building a solid internal governance structure for managing upgrades and staying on top of your vendor's upgrade schedule; for minimising customisations to ease future upgrades; and for organising with your peers to negotiate for desupport dates that won't cripple you.

Page Break

Strategy No. 1 - Don't Stop Thinking About Tomorrow.

Companies put off upgrading their enterprise software because the process is a shock to their system - not just the IT system but the business system. Some postpone the process until they will lose support for an old software version if they don't upgrade (see "It's the Functionality, Stupid", below).

To avoid desperate upgrades, CIOs need to create a continuous planning process that includes users, not just IT.

At Ventura Foods, a Los Angeles-based food distributor, Richard Chamberlain has set up a cross-functional team of IT and businesspeople who are responsible for each module of the company's JD Edwards ERP system. Those module leads are the superusers who help their business departments understand and use the system. The committee has a flowchart of the entire ERP system that shows all the business processes affected by the different components. When an update is released, the committee can map the new functions to the current system and the business processes that are affected to determine whether the updates are worth incorporating.

The module leads, along with IT managers, are responsible for tracking the updates. That keeps them involved and makes it easier to enlist their support for doing an upgrade - or resisting it. "The different module leads come together and say, OK, here's an update, what does this mean to all of us?" says Chamberlain, who is director of innovations and e-solutions.

Like most ERP customers, Chamberlain keeps three different ERP environments running at once: the system that the company uses day-to-day - the production environment; a much smaller development environment for playing with new software and any customisations that are necessary; and a testing environment to see if new versions of the software will work properly before they are brought to the production system.

But in a testament to how complex enterprise software upgrades can be, Dennis Upton, CIO at Brother International, a New Jersey-based office equipment maker, had to set up a second test environment when he upgraded from SAP 3.0E to the SAP 4.5B product because the new software was so different from the old. "You have to create a duplicate environment for your upgrade," he says. "It's a fact of life."

It was also about $US200,000 in extra cost.

Strategy No. 2 - Flush the Custom Code.

CIOs know that putting an upgrade through its paces in a test environment is critical because most companies wind up customising their enterprise software to fill gaps in the functionality, adapt to a specific industry or legal requirement, or simply because the users want the software to act differently than it does out of the box. It's not that customisations represent a failure - they are often legitimate and may even add a bit of competitive advantage to an otherwise vanilla software program. But companies tend to customise overmuch in an attempt to appease the businesspeople who use the system.

That's why some CIOs see upgrades as a prime opportunity to flush custom code from the system and replace it with a standard software program or interface from a vendor. Removing custom code makes it easier to upgrade because you don't have to rewrite the code each time a new enterprise version comes out. It also makes it easier to integrate the software with other software systems inside the company.

"I think the effort required to upgrade is directly related to the amount of custom code you have," says Rafael Sanchez, CIO of Burger King. The Miami-based restaurant chain built a custom software program to handle its real estate dealings when it first installed SAP R/3 in 1995. But this year Sanchez replaced it with standard functionality from SAP. "We reduced the custom code in the real estate package by 90 per cent, in finance we got rid of 60 per cent, and in HR we'll lose 60 to 80 per cent," he says. "All that means is that we'll have fewer software applications to support internally."

Page Break

But CIOs must be careful before they get rid of their customised code during an upgrade. Business users may love that code, so you need their backing - or at least their understanding - for why you're dropping it, says Cheryl T Smith, the former senior vice president and CIO of KeySpan, a New York-based gas and electric company. (Smith became CIO at McKesson in September.) During recent upgrades to KeySpan's Oracle ERP system, Smith and Don Stahlin, director of IT, identified all the customised components of the software and the opportunities to go to standard functions in the package's new version. They met with users to see which customisations they were willing to give up. When the two groups disagreed, they brought the discussion before a steering committee composed of functional and business leaders from across KeySpan.

Smith and Stahlin came up with costs for two options: modifying the actual ERP code of the new version (expensive and difficult) or adding a custom program outside of the ERP system to perform the function. That helped users understand just how much effort IT had to put into writing and maintaining those customisations, says Stahlin. But the users also had to do some research before they could go before the steering committee to plead their case. They had to come up with a "no-technology" option, usually a change to the business process to fit with the software out of the box.

"Sometimes the users had valid reasons for keeping the customisations, like labour contracts that required things to be done a certain way," Smith says. "But we found nine and a half times out of 10 we could change the way we do business because it wasn't that critical; it was just habit. You just unfreeze the organisation and rearrange the way people do things, and now you just do them a slightly different way and that's OK."

The steering committee tended to shoot down the requests for customisations, says Smith, because they were invariably more expensive than the no-technology options. "When someone from HR came to them and said: Â'We have to do it this way', the committee would say: Â'No you don't - change your process. I'd rather spend this money in my business.'"

Strategy No. 3 - Do It Yourself (Forget Consultants).

Having a continuous internal planning system for upgrades makes it easier for CIOs to limit the number of outside consultants they need to bring in to help with upgrades. An AMR study found that companies that handed over responsibility for their upgrade projects to outside consultants spent twice as much ($US2.3 million versus $US1.5 million) and took longer (10 months versus six) than those that kept the project leadership and as much of the work as possible in-house. "The costs skyrocket because you will have people on the project who don't know your business," says Judy Bijesse, an analyst at AMR Research, "and you'll have a lot of consultants who are being trained while you're paying them."

By retaining leadership yourself and tracking the fortunes of colleagues who are upgrading, "you can avoid being the one that bleeds on that first release," says Nextel's LeFave. Indeed, most enterprise software is so bug-ridden in its first release that CIOs can wind up installing the upgrade all over again when the vendor comes out with a "point release" to fix the initial bugs. Manish Khadepau waited until Oracle was on the second point release of Oracle 11i before implementing it at Infogrames, a New York-based video game publisher. (There have been six point releases of 11i since it was first introduced in 2000 - each requiring a complete reinstall if the customer has customised it.) But it was still bleeding edge at the time, Khadepau says, and full of bugs.

Each time Oracle would send Khadepau a new bug fix, the fix would destabilise the rest of his system and require him to rewrite the customisations his company had made. "Oracle 11i is so big and so interconnected that when they fixed one piece, three others would break somewhere else in the system," he says. The earlier version of Infogrames' ERP software was so heavily customised that the upgrade wound up costing as much as a new installation - between $US400,000 and $US500,000, according to Khadepau.

Quinlan, the MFS systems manager who blanched at the cost of consulting fees for her company's upgrade, has decided to install a new HR application herself (she has a background in HR and is a former PeopleSoft consultant) with support from a few internal staffers. But the financial upgrade is bigger and much more complex than she and her staff can handle. "We haven't decided what to do there yet," she says.

Page Break

Strategy No. 4 - Wait for the Bugs to Pass You by.

Companies that can afford to wait for an upgrade are best positioned to get it done quickly - at least from a technical standpoint, says Khadepau, the Oracle ERP veteran who is now manager of financial systems for Multilink Technology, a New Jersey-based maker of optical network components.

Khadepau knows because he is the volunteer director of the New Jersey chapter of the OAUG and has seen his colleagues struggle with 11i. He says one New Jersey company (which declined to be identified) bought the licence for 11i and did not install it. Instead, it sent an IT staffer to monitor meetings of his group until the wails of despair changed into guarded optimism about the software. The company is now installing version 11.55 of the software, which works well, according to Khadepau, and has new functionality that early releases of 11i did not. "Now they come to me and say things like, What was all that fuss you were making about?" he laughs.

Companies have to pick their time to upgrade carefully - especially small companies that don't pull much weight with their vendors. KeySpan's Smith remembers that when she was with Verizon, the telecom giant used to take on upgrades earlier than most customers because it had enormous clout with its vendors. "The vendor would put a big staff of its top people onsite to help us through it," she says. "At KeySpan, we have a tough enough time getting our vendors' attention without risking an early upgrade."

Epilogue - The Upgrades Bottom Line

There is one other strategy that CIOs could pursue: treat the upgrade process as a greenfield. At Nextel, making the shift to the new version of Oracle's ERP software will cost enough and take long enough that LeFave is calling it a new installation and inviting Oracle's main competitors, SAP and PeopleSoft, to bid on installing their system at Nextel. "We've worked hard to build an integrated architecture with ERP" that is heavily integrated with software from other companies, says LeFave. "All of a sudden you have to make this major leap of faith, and that opens up a discussion around should we be doing it with these guys or should we look into the future with someone else?"

The stakes have risen for enterprise upgrades. These projects need to add value, and value is no longer defined by squeaking through an upgrade before the desupport date for an older version. A good planning process, along with regular meetings with fellow users at other companies who can help you organise to get what you want from vendors, is critical.

But old habits die hard. Amazingly, 21 per cent of the AMR survey respondents sold their enterprise software upgrades to their business based on vendors' announcing desupport dates for the software. That's not planning. That's desperation.

As CEOs and CFOs tighten the reins on IT spending, Brother International's Upton pities the fool that tries to get away with such a sales pitch in the future. "If you've got a multimillion-dollar ERP system, there better be a better reason for upgrading than the fact that the vendor won't support it any more," he says.

Page Break

SIDEBAR: Vendors: It's Not That Bad

Enterprise software makers defend themselves against user complaints "IT'S NOT AS BAD AS YOU THINK." That was the most common response from the major ERP software vendors when confronted with customers' complaints about their software upgrade practices.

The vendors dispute AMR Research's estimate that the cost of upgrading amounts to an average of 18 per cent of the original software and Gartner's assertion that it sometimes reaches 30 per cent. PeopleSoft says its averages are 10 per cent to 20 per cent, JD Edwards says it's 10 per cent to 15 per cent, and SAP claims 5 per cent to 10 per cent. Upgrading gets more efficient the more users you have, responds AMR Research, because you still have to do the same planning and testing whether you're upgrading 50 or 5000 people, which may help account for SAP's low percentage - its customers are primarily big companies.

The dramatic shift in the underlying architecture of these products - from client/server to Web based - has a lot to do with the hassle that customers are seeing, say the vendors. Bug problems that early on plagued Oracle and JD Edwards' upgrades can be attributed, at least in part, to this change, the vendors say.

Major ERP upgrades also spark hardware shopping sprees that drive up costs, says Brian Vogt, vice president of professional services for Germany-based enterprise software vendor SAP. Companies say: "As long as we're upgrading our ERP system we may as well upgrade our databases and operating systems too," says Vogt. "It becomes part of an overall hardware upgrade that companies do every three to five years anyway." While Vogt acknowledges that SAP's latest ERP upgrade, MySAP.com, demands more horsepower than its forebears, he insists that it's possible to stay with the OS and database from the previous version.

Customers also bump up upgrade costs when they choose different best-of-breed vendors instead of a single vendor for ERP and its dependent systems (like supply chain and CRM), say vendors.

Vogt argues that companies could upgrade quicker and cheaper if they paid more attention to the changes in jobs and work methods that come with upgrades. "IT can do the technology testing, but companies need to be more proactive in having set testing procedures for new business processes," he says. "The primary reason the cost of upgrades go up is not the technology; it's because customers can't make decisions about what they want to do with their business," echoes Mike Gregoire, senior vice president of PeopleSoft Global Services (US).

The vendors all acknowledge that they are hearing complaints but say the economy - not the time window between releases - is the primary aggravant. "The pressure has increased dramatically to lengthen the support period for older versions," says Hank Bonde, COO of JD Edwards (US). In response, all the vendors have extended support for their older software to an average of about five years, but they all defended what they cited as an industry standard time window of about three years. Especially in new categories of enterprise software, such as CRM, customers are pushing for more frequent upgrades, adds Bonde, because they are hungry for new functionality.

Page Break

SIDEBAR: It's the Functionality, Stupid

Upgrades can be worth it if first you focus on the business benefits. Many CIOs have begun planning for upgrades as they would a new software installation. In a recent study of 109 companies that had upgraded their ERP software, AMR Research found that most of them spent more time selling the project internally and getting approval and funding than they did doing the upgrade. According to Judy Bijesse, an analyst at AMR Research who co-authored the study, it took many companies quite a long time - as much as a year - to build the business case for upgrading.

Lower cost of software ownership is a dubious component of those business cases. Although 49 per cent of the AMR survey respondents used that as a justification for upgrading, only 13 per cent actually saw their IT costs go down. In part, that's because infrastructure costs inevitably go up. For example, to go from SAP R/3 version 3 to version 4 requires 87 per cent more CPU speed, a 72 per cent increase in memory requirements and 33 per cent more disk storage space on each computer that uses the software, according to Gartner.

But it is possible to save money as long as you use the upgrade as a launching pad for adding new functionality. Simply moving from an old version to a new one without doing that is what CIOs sneeringly refer to as a "technical upgrade", and it's what CIOs do when they are forced into it by a desupport date.

By far the largest number of respondents in the AMR survey, 61 per cent, said the most valuable result of their upgrades came from adding new functionality, and they spent an average of $US200,000 on additional software to get it. Web portals were the most popular add-on, followed by procurement applications, data warehouses, HR applications and CRM. (Interestingly, only 27 per cent cited new functionality as their primary reason for upgrading at the start of the project. It was during the course of their projects that their belief in the value of new functionality developed.)

SIDEBAR: The Seven Lively Steps to an Upgrade.

The time spent on upgrading enterprise software varies little between small- and large-scale projects. Expect to spend a year or more from the time you begin pondering an upgrade to getting it running.

GET PERMISSION - Enterprise software upgrades are expensive, so they require well-thought-out business cases and a thorough examination of your options. For example, will the upgrade be so different from the previous version that you should consider going to a different vendor's package that better fits your needs? Duration: 6 to 8 months.

PLAN - Most companies wind up customising their enterprise software to fit their business practices. Upgrades are an opportunity to strip out some of that customisation and replace it with standard functionality from the vendor. But it takes time to figure out what stays and what goes - and to make sure the business won't revolt at the changes. Duration: 6 to 7 weeks.

INSTALL - The (relatively) easy part. Get the new hardware and networks up and running and the software loaded in all your different locations. Duration: 4 to 7 weeks.

TEST - The hardest part. Enterprise software packages are highly integrated. Make a change in one place and it ripples through the rest of the system. That - combined with the fact that new enterprise releases are always buggy - makes testing a nightmare, but it's critical to avoiding breakdowns once you go live. Duration: 5 to 9 weeks.

MIGRATE THE DATA - This is where you reconcile three different versions of the same customer record in your database and put a single correct version into the new system's database. Time and expense depend on the rigour and quality control of the data entry processes in your company. Duration: 2 to 4 weeks (or longer depending on data quality).

TRAIN - Even if all the technical steps go well, an upgrade can fall apart if users don't like the new screens they see or can't figure out how to do their job with the new system. Even small changes to the system can mean big changes to business processes and drive users nuts. Duration: 2 to 5 weeks.

CUT-OVER - This is the tenuous stage when you turn on the upgrade and turn off your old system. Most CIOs time their cut-over for a weekend, preferably a long weekend, when few employees are there to see and feel the chaos. Duration: 3 to 4 days.

- SOURCE: AMR RESEARCH AND CIO REPORTING.

Page Break

$US1000 a Seat

Upgrades cost more for smaller companies. A major ERP upgrade will run about 18 per cent of the original installation cost, according to an AMR Research survey. For companies with 500 users of ERP software, the average cost was $US533,594 or about $US1000 per user. For companies with 2000 users, the cost was $US1.2 million or about $US595 per user. For large companies with more than 8400 users, the total came to $US2.6 million or $US300 per user.

- SOURCE: AMR RESEARCH JANUARY 2002 SURVEY OF 109 COMPANIES WITH AT LEAST 2500 EMPLOYEES

SIDEBAR: SAP's Big Freeze: Market leader stabilises some enterprise code

In response to customer complaints, SAP, which invented and now leads the market for enterprise software, has decided to stabilise the core components of its R/3 ERP software suite, which it will call R/3 Enterprise, for at least five years. Customers can upgrade to the newest SAP components, such as its APO supply chain product or its CRM software, without having to touch R/3. In the past, SAP upgrades have required replacing all components of the software each time.

Of course, SAP's new policy also means that R/3 users cannot expect any significant new functionality. They must also pay a licence fee for the new components, such as supply chain and CRM. SAP argues that these components require a significant investment to develop and represent technology that is outside traditional ERP functionality. Such enhancements merit new licence fees, even from existing customers (who get a credit for the R/3 software they've already purchased), says Arne Schmidthals, vice president of product management for R/3 Enterprise. But Schmidthals acknowledges that customers have argued that the APO supply chain product, which is heavily integrated with and reliant upon the core ERP software, should be included in R/3 Enterprise.

In a sense, SAP is a victim of its own success. It blew away its competitors by combining a bunch of formerly standalone products for different functional areas of the company (finance, HR, manufacturing) into an integrated package. So should every new software component it develops be lumped together in the traditional way? Schmidthals thinks not. Still, it's a shift from the days when customers that paid their yearly maintenance fees got new functionality free in the next release. As all the major ERP vendors move to "componentise" their products - in part to make it easier to upgrade individual pieces rather than the whole package - the "maintenance fee buys all" philosophy is vanishing.