In my 20-plus years in IT I’ve always been stunned by how technologists measure the value of the services delivered to the business.
It’s not that the metrics they have are all wrong; it’s just that they don’t help anyone make good decisions – and that’s a big problem.
We’re so vain in IT, aren’t we? For years we’ve taken pride in telling our business counterparts how we’ve maintained 99 per cent system availability, how we’ve fixed hundreds of problems, or how we’ve had thousands of hits on our wonderful new website.
But I’ll let you into a secret: all of these are just vanity metrics. They’re not just useless, they can be extremely costly.
I’ve seen many web and online initiatives during my career, but even the ones that have dramatically increased site activity haven’t always been successful.
Too often cross-functional teams across IT and marketing do victory laps when they report an uptick in page views. But really, unless those page views convert into something tangible – like customer capture and retention, revenue and profit – they mean absolutely nothing.
Vanity metrics are not just limited to web efforts; now we’re gauging mobility success through metrics like app downloads. But again, what does that really tell us from a business perspective? Not much at all, since apps only become valuable when they facilitate business engagement.
The problem with today’s IT-centric metrics is that they’re not actionable; they don’t help organisations make good sound business decisions.
Consider, for example, releasing additional features to the e-commerce functionality of your website. Traditionally, the only time you measure success is by periodically reporting revenues accrued.
But what if the increase in page views only resulted in a small increase in profit? Has the profit been worth the additional development effort?
Or is it profitable at all when you’ve invested in additional infrastructure in anticipation on the online cash register ringing?
My advice to IT and marketing leaders is to stop fleetingly admiring yourself in the tech mirror and take a full length view of your business capabilities. Sure, feel good about the technology, but also start developing sets of metrics that drive actions.
Here are some tips.
You assume, you lose
I’m a big proponent of stakeholder engagement, but don’t ever assume that you or your stakeholders know everything. I’ve seen many examples where IT and business leaders have made assumptions about what customers and employees wanted from applications, only to be painfully rebuked later.
My advice is to start testing assumptions where it really makes most sense – at the sharp end, with customers and users. This is especially important with mobile computing, since customers can engage your business at many more points in time.
Learn your A, B and Cs
So you’ve added a new whiz bang feature to your website, or believe you’ve dramatically improved the user design of a mobile app.
Before releasing it en masse and then holding your breath, consider developing more actionable metrics through techniques like A/B (split testing) and cohort analysis.
For example, develop the means to release the new functionality to 50 per cent of your customers and then compare the results against those customers who don’t have the feature.
Did the new feature lead to more customers tapping the buy button? Obviously if the new feature led to more business you can release the feature to 100 per cent of your customers, but it also helps guide other actions, such as developing similar functionality in other applications, or eliminating features that don’t add any value.
Remember too that these types of techniques are a great way to confirm that costly cosmetic website design changes don’t always lead to anything beneficial (what I call avoiding the “lipstick on a pig” syndrome).
Quality trumps quantity
How many metrics have you developed in IT? 50, 100, more? Now ask yourself how many are useful. In my experience five to ten actionable metrics are far more valuable than 100 technology-centric diagnostics.
For every metric you develop, look in the mirror and start asking ‘so what does this mean to the business?’ versus how good they make your technical teams look.
Remember too that metrics such as percentage of sales conversions to website visits, revenue per customer, or churn ratios actually mean something and are a great way to initiate business conversations and ongoing alignment.
When you’ve worked behind the firewall with transactional based systems, it’s always been easy to impress how technically great you’ve been at running IT.
But cloud, mobility, and the consumerisation of IT are increasing the expectations on business technologists to demonstrate the value of technology in clear, unambiguous business terms.
Unfortunately for those traditionalists who use vanity metrics there’s now a clear message: stop looking at yourself in the technical mirror because people will see right through your technical shallowness. Remember, beauty is in the eye of the business beholder, so start developing actionable metrics.
Miriam Waterhouse is the CIO at the National Film and Sound Archive, Canberra.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.