Fighting Fragmentation

Internet commerce could be the most divisive issue companies will ever face.

Not since the introduction of the telephone in 1876 have companies had to deal with a new two-way communications channel to their customers.

The whippersnapper Web joins the established telephone and in-person interactions as a point of contact between customers and vendors. Given that dynamic, an e-commerce site, despite its promise, can actually lower the quality of a company's customer service. Without thoughtful integration of those distinct channels, people who place orders on the Web will be offered only apologies for incompatible systems when they call toll-free customer service lines to check their orders. This trend toward fragmentation is already evident. Forrester surveyed 50 Global 2000 Internet commerce sites and found that only 16 per cent were integrated with existing customer service systems.

To combat this rift, Forrester believes companies must adopt a new, loosely coupled integration strategy that we call federated integration. Using products available today, it is key to supporting fast-moving business opportunities and is the weapon to slay the next Y2K-scale problem: application integration. If ignored, that problem will cripple business flexibility and undermine IT's credibility with the business units. Federated integration is largely home-grown, requiring a serious commitment, but it will produce a system that can manage costs and avoid gratuitous reinventing.

The Challenge

As e-commerce sites evolve, grow and tie in to existing systems, corporations will find themselves with more complications than solutions. Forrester's research shows that simple point-to-point integration between Web sites and back-end systems will not scale to support the real-time trading promise of e-commerce. Not only that, the volume of data transfer is expected only to grow, and the quality of that data will need to be top-notch.

Sixty-six per cent of the IT executives we surveyed said that hourly or daily batch updates are sufficient today. But 81 per cent of these early e-commerce adopters said that in two years they will have Web transactions that will require real-time connectivity with corporate systems (see "Synchronising Web Data"). Yet they expressed reservations about opening back-office data stores, particularly in businesses with sensitive data and privacy issues. One survey respondent from a health-care services provider notes, "Our goal is to access data in real time, but no one feels comfortable running Web transactions against our main corporate databases." Cost is another impediment to real-time connectivity. "We fear that the cost and complexity of synchronisation will increase by an order of magnitude and become prohibitive to our subsidiaries. But batch replication hasn't allowed us to reflect sudden changes -- and that's something we need," says a representative from a business services company.

Forty-three per cent of the IT executives we interviewed expect the amount of data moving between Web and corporate systems to increase by a factor of 100 or more over the next two years.

These online front-runners told us they spend an average of 25 per cent of their commerce development and software budgets integrating with existing systems. The money adds up because of a host of nagging problems. The most troubling are the mismatch between back-end architectures and Web front ends and the dubious integrity of back-end data. "The Web front end is advanced and the back end isn't," says one computer company executive. "Screen scraping [adding a graphical user interface] is an option, but it's like taking a Ferrari dashboard and sticking it into a Daewoo Nubira -- it doesn't make the car any better." These escalating demands will drive IT's current one-off integration approach into crisis. To avoid missing business opportunities, companies must rationalise their integration investments under a new architecture that will leverage existing applications to solve new problems, enable efficient implementation of new processes and limit disruptions to the business.

A Solution: Federated Integration

Federated integration spans packaged applications, custom development, systems management, data warehousing and internal ad hoc efforts. It lets a company use its existing applications as building blocks to support rapidly changing market dynamics. With integration logic extracted from applications, large-grained business process changes -- such as adding a new partner's customers to a consolidated account balance -- become easier to implement. And changes required for integration don't intrude on an application's existing purpose.

Ideally, federated integration yields a four-tier foundation of services for connecting, transforming, applying and managing data.

In the first tier, connection services provide basic network connectivity, communications protocols and security. Applications can be integrated by exchanging data or events; for example, "update customer address" or "place order". Connection services encompass both data-level and event-level integration in a consistent way, regardless of the transport method used.

Second-tier services transform data and events so that they are understood by all applications. Each application translates its native format into the common format. This approach enables new applications to be integrated but pre-existing transformations are not affected. These services are available from vendors such as Constellar and Active Software.

The third tier, application services, consists of the business logic needed to implement new processes or augment existing applications. The logic runs on servers from vendors like Netscape Communications and NetDynamics or message brokers from vendors like NEON or Vitria Technology. These keep improving as technologies like Enterprise JavaBeans and XML roll out.

Finally, federated integration is an enterprise-scale activity requiring robust administration tools. In the fourth tier, management services reach into all integration tiers to provide functions like exception-based alerts, system testing and automated performance tuning. Vendors like NasTel Technologies, NetIQ and Computer Associates International build on standards like SNMP and XML (which provide a clear interface to management data) to deliver these services.

Getting to the Solution

Firms must build federated integration themselves -- it doesn't exist as a single product in the marketplace. Although systems integrators have a thriving business, CIOs should use them only on a tactical basis. Integration skills are a competitive advantage that should remain in IS's control.

To get started, companies need to recognise that integration clusters are the efficient solution. Loose affiliations of vendors anchored by a strategic middleware supplier form federated integration clusters. To make integration work, companies will have to make deep investments in tools, technologies and software. And to get the most advantage from these, firms should focus on a single cluster. Four clusters are emerging, each aligned with a particular component platform: BEA Systems, Microsoft, IBM and Oracle. Although partnerships are nascent and commitments largely untested, market momentum will push these clusters to coordinate their technical support plans and market their offerings.

A company will select a cluster based on its component platform choice and the level of commitment it has to packaged applications or database vendors. If these factors don't naturally lead to a single integration cluster, the company should partition its application portfolio to use separate clusters. Forcing a particular application to touch more than one integration cluster will forfeit any economic efficiencies or speed-to-market benefits.

Once a company selects an integration cluster, it must decide on a case-by-case basis whether to integrate applications by exchanging data or by issuing and responding to events. This decision involves balancing four factors: response time, quantity of data, accessibility and cost (see "Choosing Data-Level Versus Event-Level Integration"). Each approach has its own challenges.

Data-Level Integration

Firms must concoct a single way to cope with the huge variety of back-end data created over the past 20 years. Figuring out how to extract the right data is only the starting point. The harder part is creating -- and enforcing -- common data formats. The common format is required for exchanging data, but more important it lays the groundwork for a consistent view of data across the enterprise.

To start, adopt a hub-and-spoke data architecture for moving data. The trick in data-level integration is to put as much transformation, application and management logic as possible in hubs and not in the original application. This leaves applications unscathed while centralising responsibility and control.

Data hubs like Constellar's Hub and Reliant Data Systems' DCLE can handle this task.

The lack of a universal data "language" has been a huge barrier to data exchange until the advent of XML, which is supported by vendors including IBM, Microsoft, Netscape and Oracle. Firms will gain tremendous efficiencies by labelling their internal and external data using XML tools and servers from vendors like Microsoft and DataChannel.

Event-Level Integration

Applications that must work together in real time require event-level integration. It begins with an underlying messaging backbone such as IBM's MQSeries to provide connection services. Organisations must put messaging interfaces on applications and use message brokers to issue and respond to events. That takes programming. Vendors like Active Software and Netscape sell "adapters" -- configurable software connectors that provide a messaging interface -- for popular packaged applications, but companies must still code interfaces for custom applications. This task is made a bit easier with adapter toolkits, which Forrester expects to proliferate -- unfortunately, without a standard -- in the next year.

Event messages are routed to applications using message brokers from vendors like Tibco Software and Saga Software. Changes in business processes can be implemented as business rules on the message broker.

Price and Payoff

Federated integration has a $1+ million get-started price. This initial cost is consumed by servers, software and labour to create data formats, reverse-engineer existing systems and manage the whole effort. Remember that companies already spend 30 per cent of the massive price of packaged application implementation on integration. And the cost of not doing it at all is lost customers, unhappy business units and, perhaps, replaced CIOs.

To make it easier for suppliers, partners and customers to tap into corporate systems, innovative organisations will clarify their data models and publish them on the Web. By removing the barriers to understanding corporate applications, wise companies will enable deep integration across their supply chain.

Companies on advanced integration trajectories are creating a commerce application program interface (API) that will surface their business processes to the Net. To maximise API's value, firms must choose a component platform -- Com/ Dcom or JavaBeans/Corba -- strategic for their commerce efforts. This discipline will set the stage for cost-effective development and site operation.

Meanwhile, companies undergoing back-end restructuring with ERP systems must bring commerce techies into the ERP team early. The extra effort will be worth the trouble in two ways: The ERP customisation plan will include the commerce requirements, and the commerce investment can be trimmed to just the new middle-tier development that is complementary to the new corporate systems.

Despite all that seems new or changed, commerce site integration actually gives legacy systems a new lease on life. Therefore CIOs must find a way to keep legacy skills alive and hang on to people who understand the existing business systems -- and ensure that they work well with the Internet commerce cowboys who will be driving commerce integration.

Federated integration will unite investments, energise the firm to tackle integration challenges and enable rapid rollouts of new business processes. All of which will head off the hidden killer of incompatible commerce channels.

Bob Chatham, a senior analyst at Forrester Research in Cambridge, Massachusetts, can be reached at