The Good . . .
The good news is that any organisation should be able to improve the quality of its data by treating data as a strategic corporate resource; by developing a program for managing data quality with a commitment from the top; by installing appropriate technologies; by relying on experienced data quality professionals to oversee and carry out the program; and, most importantly, by constantly working at the issue.
Some companies - particularly those where the data quality issues are relatively straightforward - are making real progress. For instance Associated Food Stores in the US has moved away from "bat-fingered" manual data entry to adopt a wireless real-time locating system from WhereNet Corporation to enable it to locate, track and manage assets across its 243-hectare distribution centre. Now internal logistics manager Tim Van de Merwe says the company has achieved 100 per cent accurate data capture.
"Our data collection no longer requires any keyboard entry, no scanning because in the RF [radio frequency] sense - it's transmitted. We have automatic entries of locations, and then status association with the IDs that are located based on areas. Areas are predetermined and predefined priorities are connected to those areas. So any time an asset shows up in that area it's automatically labelled as available or unavailable and some kind of action that needs to be taken with those assets has been predefined. Once you set up the system, it just rolls," de Merwe says.
However for many companies the data quality issues are far more complex, and these organisations must draw on analytic solutions to measure and proactively manage data quality. Insightful Corporation, for instance, provides Fortune 1000 companies with scalable data analysis solutions that drive better decisions faster by revealing patterns, trends and relationships. Block says Insightful uses a stepped approach to data quality that starts with an assessment of the number of missing values, zeroes, undefined values, missing primary keys, missing foreign keys and domain outliers found in corporate data.
"The multi-level approach involves checking a number of observations in a certain table and its variation with time, frequency of missing values, outliers, and so on, compared to previous months and other factors," Block says.
"Finally the work moves to the level of business quality, which involves checking reasonable bounds on purchase orders, transaction sizes, the order of dates - you cannot close a cheque account before you even open it, the number and variation of customers in a certain region, and so on." Insightful also might check time variations of data fields such as a birthday, name and gender (Block has often found gender variations over time as a consequence of bad data).
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.