Clean Up Your Act

Your special report on integration in the March issue of CIO highlighted only too well the myriad of issues that come to light once an integration project has been embarked upon. A common theme amongst the articles was the reliance, or lack thereof, on the quality of an organisation's data.

A recent PricewaterhouseCoopers study showed that corrupt data costs global businesses over $US1.4 billion a year in lost revenues, overpayments, mismanaged inventory, duplicated purchases, lawsuits and lost customers. This was certainly highlighted in your articles.

I did find it interesting though, that although most of your interviewees acknowledged that data integrity and quality were problems, they also believed correcting or understanding the errors in the data was not something that could be automated.

I suspect that many CIOs are unaware of the data profiling technologies that are now available. These highlight data anomalies and provide a process to manage the rectification of these errors. This, to all intents and purposes, "de-risks" large-scale data migration or integration initiatives, bringing them in on-time and on-budget.

Essentially, data profiling provides complete understanding of data by identifying inconsistencies, redundancies and inaccuracies in your data from the data itself, without requiring or relying on documentation, experts or source code. Data profiling is becoming an essential element in any large data integration exercise. It identifies all the anomalies in your data before they become an issue later in the project development life-cycle or, worse still, after deployment.

Most CIOs are aware that when they call in one of the big consulting companies to handle their data integration, the contract typically contains a clause that leaves data quality as the responsibility of the customer. What is often not apparent is the potential cost. Research shows that if the customer doesn't address quality issues prior to the start of the integration process, it will cost between 10 and 100 times more to fix in the development and deployment phases of the project.

Data integrity has previously not been considered a strategic issue. Historically, the emphasis has been on data collection and storage rather than maximising the value of the data by ensuring its quality. High-quality data is the foundation of all good business decisions - and companies are beginning to appreciate the ramifications of making decisions on data of questionable integrity.

I wonder how many of your readers have had projects in their organisations in the last 12-18 months that have been delayed, aborted or have had cost overruns due to data integration issues. This has almost become an accepted part of project lifecycle and has been managed by the building in of contingency into the project plan.

Organisations using automated profiling techniques have reported cost savings of between 20-40 per cent on all data integration projects.

Automated data profiling allows you to measure and understand the magnitude of your data integrity problems. And as we all know, what we can measure we can manage.

Steve Hitchman.

Managing Director.

Management Information Principles Limited.

North Sydney, NSW.

Hitchman@mip.com.au.

WHAT DO YOU THINK?

Send your thoughts and feedback to linda_kennedy@idg.com.au. Letters may be edited for length or clarity.