Nothing eats time or blows budgets like a runaway data integration project.
Here are seven tips to help protect your investment Reader ROI -- Data integration is all about new operating efficiencies and improved business processes -- neither of which comes cheap. Integration projects are also prone to run-on expenses in unexpected, or hidden, areas.
Here you'll learn about those hidden costs and how to avoid them.
A lot of IT organisations have integrated a lot of data and in the process shelled out a lot of money. Too much? Maybe so, judging from the comments of several CIOs who've recently been through major integration projects. Business plans. Data transfer. Project leadership and user training. Each of these devils has risen up from the details of integration projects and bitten CIOs squarely in the back pocket. Now perhaps these IS executives weren't fully abreast of the latest and greatest integration tools on the market. Or maybe there are just so many different ways to integrate data and thus so many ways to fumble the project management fundamentals common to any IT initiative.
Either way, integration has clearly become a basic block-and-tackle endeavour for many organisations -- a low-profile service that user groups and senior management have come to take for granted. Even some CIOs are tempted to delegate or outsource data integration, then walk away until the work is finished. In other words, it's a project waiting to bloat.
The Bottom(less) Line
At its most basic, data integration is whatever set of processes takes place when two applications built by different developers exchange information. This process almost always involves four steps: extraction, transport, transformation and loading.
One might have a separate program to perform each of these functions, or one might instead be among the growing legions who invest in message-brokering software that can automatically manage all aspects of the data flow, from broad business considerations to discrete technical minutiae. One might also be implementing an enterprise resource planning (ERP) solution and looking for that to solve all the integration headaches (if that's you, don't hold your breath), or be somewhere in between -- either using some combination of two or three tools or a partial message-brokering solution (see "Message Brokering: Plan Today or Pay Tomorrow"). No matter the approach, some things can be done upfront to keep unnecessary costs from turning integration projects into a money pit. For starters, recognise that it takes more than a software installation to integrate data successfully and cost-efficiently. Message brokering, for example, like most software marketed as a complete solution, requires development and ongoing management by humans, which often means investment in additional software tools for such tasks as code management, version control and customisation of transformation adapters.
And don't forget: It may not sound expensive to say you have to figure out what data is needed, how to convert it and how to clean it up so that it can be loaded into new applications. But data integration remains, for many, a difficult and costly task. So what should integration cost? Obviously the dollar figure varies widely, from a few thousand dollars for basic data extraction to millions for a full message-broker-plus-tools complement. And one must also take into account that integration isn't necessarily an isolated project; it has a ripple effect across systems, often creating new pseudo-applications. "They're applications that sit between your existing applications," says Ross Altman, a research director in GartnerGroup's applications, integration and middleware strategies service in the US. "They're slightly different in style and architecture, but they're definitely applications." And building applications requires a development environment, training, process definitions, architecture-all the things that you'd associate with any other development project. "Most people don't realise that," warns Altman. "They just think it's a matter of stitching together databases. And then those stitchings end up becoming mission-critical." What else can one do to keep an integration project's cost contained? Here are seven "gotchas" identified by CIOs.
The No.1 Gotcha:
Business Objectives: Nail Them Upfront or Get Nailed LaterA very thorough understanding of the whole information flow is critical as a starting point, says Yusef Akyuz, vice president of information services for Timberland, the rugged shoe and apparel maker in New Hampshire. He oversaw a considerable investment in IBM's MQSeries message-brokering software. He then set out to customise the flow of information using the IBM software so that different computing platforms and applications were invisible to business users. His goal: use the installation process as an opportunity to re-engineer business processes so that people wouldn't simply be transferring files but rather everything needed to perform particular functions more efficiently.
"You can pick and choose the information coming across [using message brokering] and tailor it toward the destination it's going to, but if you don't have the big picture in mind, what good would that be?" Akyuz asks. "One should not look at what is needed today, but one, two, three years ahead." And then the focus should not only be on interfacing but on the entire supply chain function. "Think ahead," says Akyuz. "How can different departments benefit from other departments' data?" The No.2 Gotcha: Weak Project Specs Will Blow More Than Your Budget Nail down specifics on data sources and targets, including what needs to be moved and how it will be used -- or else prepare to pay more later, when you're doing the data mapping you should have done upfront. "You don't want to spend an inordinate amount of time somewhere in the middle of the project trying to specify what it is you really need to chew off," says Ashwin Rangan, senior vice president and CIO of Conexant Systems, a semiconductor maker in California. He learned his lessons when he set out to supply the company's sales staff with data from an SAP billing system. Often, Rangan found, when IT people look at integration projects, their focus tends to be on the technology -- particularly the middle layer that permits two discrete systems to talk to one another. But in the long run, that is a secondary consideration. The primary question is, Why does the business want a pipeline between these two systems? "Oftentimes that question is never asked, let alone answered," says Rangan.
Messaging and middleware is a very complex set of technologies -- and a nontrivial investment of time, energy and money. If you forge ahead without measurable mileposts and a firm plan, your integration project will become an IT initiative, not a business initiative. "The hurt in that kind of mistake goes beyond the project itself because it's the credibility of the organisation and the people that get put at stake. You can't put a dollar figure on integrity. Losing it is an unbearable cost." The No.3 Gotcha: Most Data Is Dirtier Than You Think Cleansing data of redundancies and irrelevancies so that it can be moved efficiently from a source system to a target system is time-consuming and costly. Yet people continue to be optimistic on the front end, setting themselves up for creeping costs as the integration project moves along, according to John Carrow, CIO of Unisys in Pennsylvania, and former CIO of the city of Philadelphia. "It almost always ends up taking a brute-force, intensive, manual data-entry effort to sort out the data," says Carrow. "It's common to optimistically assume that data is clean, but that's usually a very bad and costly assumption." Carrow also points out that off-the-shelf products have made it easy to provide the right kinds of interfaces for moving data once it is clean. "But doggone if it isn't still one of those things where, despite the fact that it's been focused on across many industries and businesses, you've still got to ask yourself, Have we been thorough enough in defining the interfaces? Do we really understand everything that has to be converted? And have we sized it right in terms of what it's going to take to do that?" The No.4 Gotcha: Once Is Never EnoughAccording to Conexant's Rangan, it pays to take a snapshot of the data to be transformed and hang onto it for the duration of the integration project.
Integration bridges are built under certain assumptions, and those assumptions are likely to change over time. If you don't freeze the data, you may later need to go back to the design phase to make it transferable. In the process of deploying its new sales-force automation tool, Conexant wanted to dig data out of an SAP billing system in a different environment. It was a case of different technologies, products and platforms -- one was client/server, the other a replicated Notes database. "Just before we got to the point of cutover, we found that we had to do one more wash [because some of the data formats had changed]," he says. "It was one of those gotcha moments." Rangan's group referred back to the design cycle, took another look at the information sources being dug out for sales and rewashed it. "The second scrub is usually easier because you've already been down the path once," he says. "And it's not as difficult to go down the path a second or subsequent times. Nonetheless, it is a delay in the overall integration process." The No.5 Gotcha: Take Charge of Project Management Too many cooks in the kitchen can indeed spoil the integration broth, says Lauris Ann Nance, CIO of the Public Service Company of North Carolina, a natural gas supplier based in Gastonia. Nance's 60-person IT group was charged with integrating financial and HR applications from a PeopleSoft ERP package with a materials-management package from American Software and with a legacy customer information system that also needed Y2K compliance work. That meant there were many opinions on whose methodologies would rule the day. Such a situation may or may not ring up financial expenses, but who wants to pay even the nonmonetary costs incurred by a cloud of "who's in charge?" tension hanging over a workforce every day? "On the pro side, our people learned the importance of having strong project management capabilities in-house," she says. "But in the future, we'll size up consultants on how well they'll fit with us before we bring them in based almost exclusively on their technical skills." The No.6 Gotcha: Monitor Transmissions to Avoid Fatal Bottlenecks End-of-month increases in integration activity, such as regular billings or salespeople rushing to meet quotas, can fill a messaging system to its capacity, slowing down transfers and backing up business processes. At Timberland, Akyuz used to have IT resources dedicated to monitoring data transmissions, segmenting large transmissions into many smaller ones. IBM's MQSeries software and a transmission monitoring application that Timberland developed alert staff to problems early on or as they happen and automatically retry stalled transformations. Akyuz is also in the process of procuring a pricey object-oriented middleware solution for integration, which, along with MQSeries, will segregate and send information to different locations, regardless of hardware platform, application, time zone or language.
Akyuz offers a word on monitoring: Even if you've invested in a comprehensive messaging solution, you may need to spend a little more to obtain automated monitoring capabilities -- from either the messaging vendor or a third party.
Otherwise you may not know there's a problem with a transmittal until it's created a backlog; nor could you assume that there'll be an automatic retry if the transmission stops midstream. Most messaging vendors offer monitoring tools as expensive options. If you do go with monitoring tools, make sure they work with all the different computing platforms your enterprise uses.
The No.7 Gotcha:
Don't Forget Training and Support
Not only does data reside in any number of disparate sources, but it also often gets accessed with any number of interfaces. It's impractical to support integrated architectures with several specialists, so you need individual staff members with a broad range of skills. Naturally, that means training --Êof users and IT pros alike. If you don't have the skills in-house to provide user training, you'll need to hire someone to do it. Plus you'll need to fork over sufficient funds for the creation of printed training materials -- and of some marketing tools to sell people on what's in it for them. "These are soft costs," says Charlie Lacefield, former vice president and executive director of business processes and IT at Dow Corning in Michigan, who retired just after completing a multimillion-dollar ERP implementation at the chemical-products company. "They're very hard to budget for because it's so difficult to gauge how much it's going to take." He adds that so far there's been no need for additional investment in integration-specific tools to complement the ERP implementation since Dow Corning went with the full suite of SAP modules. As for the trainers, Lacefield favours technicians with good communication skills, never general educators with only a rote understanding of technology. Naturally, finding qualified trainers can be a job in itself for a member of the IT team; the same can be said of coordinating training schedules. "If people are going to need to adapt to new ways of doing their jobs in order for your integration project to work, not only are you going to have to train them, but you'll also have to hold their hands," says Lacefield. "You have to keep reminding people that they're part of a big team. That's where you get into T-shirts, coffee mugs, mouse pads. Briefcases worked well for us. The more you give, the more you get back. And the cost to do it right is not incidental."Message Brokering: Plan Today or Pay Tomorrow The same old integration issues apply to new-wave EAI tools For most IT organisations, integrating applications by simply batching and transferring files is no problem -- that's what they've done for a long time.
But taking advantage of the new message-brokering products on the market, also called enterprise application integration tools (EAI), which allow integration on a more real-time basis, means reckoning with some of the same old issues.
Doing messaging right begins with retraining staff and understanding that the program may not be as complete as you'd like right out of the box, according to Ross Altman, a research director in GartnerGroup's applications, integration and middleware strategies service in the US. "In many cases, you're going to have to buy third-party management and performance monitoring tools in order to get some real control over this potentially broadly distributed implementation," Altman says. "I think a lot of people are surprised by the ongoing management requirements that messaging brings."Other items that might run up your integration bill when you go the EAI route: EAI Tools. They reduce the amount of time and effort it takes to build your application, but they don't make the work go away. "You still have to figure out the business semantics of the source and target applications," says Altman.
"What does the data look like? What are the business rules that apply to that data?" Preparation. EAI doesn't negate the need to do some analysis and design work.
"Somebody's going to have to sit in a conference room and put a lot of flip-chart pages on the wall before they're ready to use the EAI tool that they bought," says Altman. "That cost tends to get overlooked." Management. The best EAI tools have built-in code-management, version control, impact analysis and configuration-creation components. But they still have to be managed by a living, breathing human being. "No matter how you choose to do it, integrating systems and applications effectively takes a dedicated team," says Altman. "And the team has to include developers, project leaders, business-process experts, and systems and administration people. In other words, you can't get there on the cheap." -- D Pearson
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.