Actually that is a bit unfair.
While measuring intangibles - such as feelings, happiness and comfort levels - might be akin to juggling jelly, there are valiant efforts to do just that. In the age of transformational relationships, this becomes even more pressing, especially when these measures impact on supplier remuneration.
Tony Clasquin, CIO of Colonial First State, expresses a long-term interest in the subject of intangible measures, but admits he's unaware of any formal industry process. "Most of my SLAs are focused on behaviour rather than performance," he says. "The way you set an SLA is to first define what is the behaviour you want. Once you have the target behaviour clear in your head, then - and only then - do you set the SLA with a view to drive the behaviour. The SLA also needs to try and motivate the service provider to perform, rather than punishing non-performance.
"You can achieve significant improvement in performance by setting competing targets - better, faster, cheaper. These look mutually exclusive, but if you put pressure on all three at the same time, you encourage breakthrough innovation," Clasquin says. "I also find that there is an extraordinary power in just asking for reports. These might not be attached to any SLA, but few people give you a report that highlights a problem without doing something about it themselves. Again, this drives the right behaviour."
Another CIO listed "proactive value add, communication, trust, commitment to work with us to change, and probably 10 or a dozen sort of phrases and words that are values of a working relationship - a really good working relationship" as behavioural intangibles that could be measured.
Andrew Switala of Hewlett-Packard Managed Services provides a hit list of possible intangible measures that can be broken up into management satisfaction and end-user satisfaction measures (see "Measuring the Warm Fuzzies", page 86). "Generally, management is focused on the bottom line and anything that will impact it. End users are focused on how the delivery of services will hinder or enhance their ability to do their day-to-day jobs. Interestingly, 'wow factors' seem to be of greater interest to 'tech heads' in vendor organizations than customers," he says.
Satisfaction Surveys
Notwithstanding the doubt some CIOs expressed of the existence of satisfaction metrics, one supplier who has instituted surveys of intangibles for several years and claims to have developed a working and effective model is EDS.
"We instituted our service excellence client dashboard in 1999," says Catherine O'Gorman, EDS's Asia Pacific service excellence director.
The dashboard, a sort of balanced scorecard, comprises an online client survey using seven key and 13 more specific questions, which are designed to elicit responses along the lines of excellent, good, average, fair or poor. EDS regards average as a negative response. "If you're sitting on the fence," O'Gorman says, "you're not doing well."
Any number of participants can take part in the survey, and surveys can be instigated by clients at any time, although EDS recommends at least once a year. A minimum number of responses is three, although some clients have as many as 20 or 25 respondents. EDS asks that at least one respondent is a CXO. O'Gorman says, allowing for individual respondents' characters, there is generally little disparity in the responses within a client survey.
The survey questions, which are common to all clients, are reviewed every year. The current seven key questions cover: overall response, references, renewability, value, competitive advantage (that is, EDS's competitors), innovation and thought leadership. A typical question runs: "Innovation is the successful introduction of new ideas. The extent which EDS provides innovation that adds value to your business . . . "
While EDS is loath to give too much detail for its proprietary system, the other questions cover such areas as: "doing things in a timely manner, do we listen to them [clients], do we act upon their input, are we collaborative, do we collaborate with other suppliers, EDS as a long-term partner, and so on".
Negative responses raise the alarm bells, prompting follow-up interviews and discussions. Negative responses also impact on EDS employees' bonuses. As their at-risk component is about 20 percent of total remuneration, this can amount to a significant punitive measure. The loss also impacts on the entire EDS client team, encouraging a sort of EVA-style team effort in ensuring good results.
EDS is looking at revising its remuneration scheme - "making it more sophisticated, O'Gorman says" - as well as the operations of the dashboard itself. While the latter is largely a look and feel exercise and will leave the operating principles largely intact, when launched (it was slated for June) it will provide greater detail and some improved functionality, especially for internal EDS use.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.