Psst. See that great stack of paper on the corner of your desk? You know the one. It contains all the latest research from your IT analyst firm - the research that costs you hundreds of thousands of dollars per year.
Get up right now, pick up the pile and toss it in the garbage.
Dump it because most of the written material - especially the market projections - is not worth your time.
Why? Because more often than not, it is simply wrong. Remember GartnerGroup's late 80s prediction that OS/2 - Microsoft and IBM's jointly developed operating system - would dominate the desktop within five years? Such predictions go wildly awry because typically they are based on little hard data and they try to cover a vast sweep of time. "Anyone who predicts a market will be a certain size five years out, put a muzzle on them. It's utterly irresponsible," says Evan Quinn, chief research officer for the Hurwitz Group (US).
"We used to joke that every market would be $(US)1 billion by the year 2000," says Scott McCready, who logged nearly 12 years as an analyst at Yankee Group, IDC and Giga before becoming CEO of CIOview, a software vendor headquartered in Massachusetts. "Once you come up with some kind of growth rate, all you have to do is compound that over several years and you're going to come up with a pretty big number. But [analysts] don't do a good job getting the constraints of the market right. I honestly think [most forecasters] sit around a campfire."
And some analyst firms are joined around that campfire by vendors. They accept payment from vendors to create forecasts (as well as other types of research). McCready says that during his analyst days, vendor sponsors pressured him to change unfavourable forecasts to their advantage.
All that might seem more benign if the research weren't so expensive. So why spend celestial sums on advice that is less than divine? Because it's the price of admission to get you in to talk with these people one-on-one. If you're looking for value from your analysts, you have to pick up the phone. Choose your analyst firm carefully, then cultivate a one-on-one relationship with your designated guru. Get him or her up to speed on your situation, including all maddening intricacies and details, then ask for an opinion. Face-to-face and over the phone, analysts can be enormously valuable.
At a previous job, Ken Kantor (now IS director at musical system manufacturer Peavey Electronics in Mississippi) saved hundreds of thousands on a computer hardware purchase because a Gartner analyst shared street prices with him over the phone - something he could not have done in a research report. Kantor also got the analyst in on the conference call with the vendor. "The analyst was telling it like it was, right in front of the vendor. They weren't very pleased with each other by the end of the call, but I was," Kantor says.
Maybe someday a brilliant analyst firm will stop basing its business model on that useless stack of paper sitting in your office. Instead, your analyst will give you a call every week to share what he or she has heard from vendors and your colleagues out there in the trenches. But until that day comes, here is our list of six essential best practices for dealing with analyst firms - straight from the mouths of CIOs who use them.
1Manage Your Analyst Company Subscriptions MOST COMPANIES (ESPECIALLY LARGE ONES) do not have anyone watching their consumption of analyst services. As a result, many different divisions buy the same analyst services, creating redundancy and wasting money. And the analysts aren't likely to clue you in, says Norma LaRosa, president and cofounder of California-based Kensington Group, which analyses the analysts. "CIOs need to put together an enterprisewide contract so they don't pay double and triple what they actually need and use," LaRosa says.
But simply knowing what you are buying isn't enough. Before signing up with any analyst firm, you need to establish exactly who the research consumers in your company are and exactly what they need. "It's essential to understand the user group that will use the research. You can do this through internal survey work," says Perry Gartner, director and lead analyst of IT research reports and services for California-based Outsell, an analyse-the-analysts firm. (His father is Gideon, the progenitor of Gartner and now Giga Information Group.) And one size doesn't fit all. In one case Perry Gartner knows of, a CIO was so impressed with Forrester Research's information that he bought subscriptions for his entire IT staff. But the people in the trenches were not the right audience for Forrester's high-level view, and they soon clamoured for more tactical, hands-on advice that wasn't available from Forrester's reports.
It's a good idea to designate someone to manage the analyst subscriptions and disseminate that information throughout the organisation. One caveat: this person should be from the group that uses the research (normally IT), not from the corporate library. "If the librarian is the gatekeeper of the information, you'll get absolutely no value at all," says David Cearley, who is senior vice president and co-research director for Meta Group (US).
2 Find an Analyst Who Knows Your IndustryWHEN CONSIDERING YOUR ANALYST options, there is a huge temptation to buy based on reputation. Resist it. Gartner, Meta, Forrester and the like are considered one-stop shops, but they all have different backgrounds and emphases and may not provide the kind of depth you need in your industry. The antidote to broad generalities: hire a boutique research firm - there's bound to be at least one - that covers your industry. Seek names of boutique firms from your peers in your industry.
3 Demand Real-World Experience
SOME ANALYST FIRMS HIRE KIDS right out of university. We're sure they're bright as all get out, but wisdom is the child of experience. Don't be afraid to dig in to the credentials of the analysts who advise you - and ask to be matched with someone who has been a CIO or at least worked in IT. James Lance, senior vice president of IS and CIO at Ohio-based The Elder-Beerman Stores, appreciates the fact that his Meta Group analyst used to be a CIO in the retail industry. "He has good know-how in dealing with the issues that are foremost in my mind." When Lance sought advice on an upcoming telecom project, he was thankful his analyst understood the constraints and communications needs of a retail store.
If your analyst should leave his firm and strike out on his own, it is worthwhile to use him for additional consulting. But if you cancel your previous analyst service in favour of a newly independent contractor, watch out: most analysts sign noncompete agreements that bar them from working for a competitor or "stealing" clients for at least a year.
4 Don't Be Afraid to Negotiate
BUYING ANALYST SERVICES IS LIKE BUYING an airline ticket - everyone pays a different price, though the analysts don't like to admit that. Meta's Cearley says the onus is on the CIO to ask the sales rep (not the analyst) what other pricing packages are available, especially if they are heavy users. "CIOs generally don't spend enough time [trying to] understand what all the different options are," says Cearley.
Educate yourself, and don't be afraid to negotiate, Perry Gartner advises. "It pays to be insistent on what your limits are. It's very easy to get intimidated. But CIOs need to say, My budget is X, my needs are Y, and if you want my business, you have to work with me to configure a service that meets these parameters," says Perry Gartner. "Analyst firms are experiencing increasing downward pressure on prices for content services." The CIOs interviewed for this article said they do not attempt to negotiate price with their analyst firms. But it pays to be bold. The client who complains might just be the one who gets the discount.
5 Get Underneath the Reports and Case Studies "IT'S LESS IMPORTANT WHAT THE ANALYST has written and more important what they will tell you on the phone. The research is the opening salvo. The real relationship starts when you pick up the phone," McCready says.
Mike Conlon, CIO at the University of Florida Health Science Centre in Gainesville, prepares himself for all his analyst phone calls. He does research ahead of time, puts one of his employees on the line with him in case he forgets something and presses analysts hard on technical issues. "They're supposed to be the experts. Try to smoke them out a little bit," he advises.
While you're on the phone, it pays to ask your analyst to dish the dirt on the latest case study he or she wrote. "Get the analyst on the phone and ask for the real scoop," says Efrem Mallach, CEO and cofounder of the Kensington Group. Analysts gather a lot of dirt that they don't end up using in the case study. But they can tell you that information on the phone, says Hurwitz Group's Quinn. You're looking for the not-for-publication tidbits that could save you hassle and money during your implementation.
6 Don't Make Your Analyst Stretch Too FarWHEN CUSTOMERS PRESS ANALYSTS TO REACH beyond their area of expertise, bad things can happen. "There are a few phrases analysts have difficulty speaking: I don't know' and I don't cover that in depth'," says Quinn.
McCready recalls once getting into trouble when a large brokerage house asked him to do some custom research that was outside his domain. His recommendation to the brokerage turned out to be wrong, and they were very angry. "Analysts get stretched. They're being asked to do all these things. If you say, I don't know the answer to that', there would be this awful silence. They would be wondering, What else doesn't he know?" says McCready.
Don't expect analysts to have flawless insight in every area. Have your BS meter on during every exchange with your analyst. Don't let him gloss over something. Push him for depth, and if you don't get enough, don't follow the advice. Seek your information elsewhere.
THE VERY BEST PRACTICE OF ALL is being aggressive in helping your analysts get to know you. Find out how much one-on-one time you're allowed with your analysts and bump it up if it's sparse. Get on the phone, listen to their recommendations, then keep your own counsel, using as many different sources of information as possible. Expect all those sources to melt away when accountability time arrives. Bad advice doesn't qualify you for a refund or exemption from the consequences within your own company. Says Conlon, "I'm the accountable person. They're advising. I'm deciding. I'm not forced to do what they tell me." vWhat's Behind Their Numbers All too often, you don't want to know When reading up on a new technology, the last thing CIOs have time for is drilling down into the methodologies analysts use in preparing their forecasts and surveys. The CIO glances at the paragraph summarising how many people participated in the survey, or she scans some of the basic forecasting assumptions, and that's about it. And to some degree, that's exactly what the analyst firms are counting on.
The fact is, most analysts' methodologies are something less than scientific. Just how much less than scientific is a matter of some mystery. Analysts don't like to reveal what's going on behind the curtain. If you are curious, however, don't be afraid to push them on this.
"If you're paying for the research, you have every right to know how those numbers are generated. They usually say: Well, we have this model that we use.' You can demand to walk through every single algorithm if you want to," says Scott McCready, a long-time analyst and now president of CIOview, a Massachusetts-based software vendor.
- L G Paul
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.