Cloud performance metrics: No standards, so mileage varies

How well will a particular application perform in a given cloud computing environment? That's a basic question for IT groups who are comparison shopping among cloud optionsand one that's tough to answer at the moment. Apples to apples comparisons prove almost impossible.

Unfortunately, there's not an independent source that can give an individual company a good idea of how well a cloud service will perform, nor a comprehensive set of tools to help evaluate a service while it's being tested, says Bernard Golden, CEO of consultancy HyperStratus and a CIO.com blogger.

In fact, it's not even easy to compare the costs among cloud providers, let alone measure and map performance, notes Geva Perry, cloud-market strategy blogger and chief marketing officer at high-performance Web application developer GigaSpaces.

Wanted: Cloud Benchmarks

In cloud computing there isn't a standard industry benchmark against which to compare all services or even a standard network across which all end users access them, according to a June benchmarking study from Bitcurrent.

Most end-user companies don't have the skills or resources to do that for their own application testing, so they have to retrofit their normal cost- and performance analyses to fit the cloud according to Sean Hackett, research director at The 451 Group.

The Bitcurrent study monitored response time, internal I/O, network performance and other typical metrics and concluded that the services themselves varied so much that the programming requirements, location and internal IT infrastructures of its customers could make results almost too large to be comparable. "Your milage may vary greatly," the report warned.

Within the last year, and especially within the last month, a crop of cloud-measurement services have cropped up to supply that need, with results that range from the ultra-specific to the generic.

"People are using the cloud for a lot of very performance-intensive applications, VoIP, videoconferencing, high-definition video, disaster recovery, they're finding out ISVs adn carriers are really good at interoperability and connectivity, but can't tell you much about performance," according to Jim Melvin, CEO of network performance-metrics provider Apparent Networks.

"With those types of applications performance is highly dependent on location and the path the traffic takes from your site to theirs. It's not enough to know your bandwidth, there's also latency, jitter and data loss rates to consider," Melvin says.

New Services Try To Fill Gap

This week, Apparent Networks launched a free service designed to give IT managers a detailed look at the average wide-area network performance between two cities.

The service relies on a dozen network-measurement points throughout the U.S. that track network performance to a selected group of cloud providers, including Rackspace, Salesforce.com and other leading brands.

Apparent's paid version of the service costs $100 per month per site, and uses a micro-appliance that plugs into any customer network port to identify specific locations between which to track performance.

Other performance measurement services include:

Webmetrics, which offers a SaaS version of broad-spectrum performance monitoring designed to track Web site performance from the point of view of the customer;

CloudCMP, a joint effort of researchers at Duke University and Microsoft Research that offers tools to benchmark a range of cloud services so users can compare them;

CloudSpeedTest, a test-your-bandwidth type tool that offers a quick but shallow view of cloud performance, from cloud-benchmarking specialists |CloudHarmony|, which provides a range of deeper, more specific benchmarks and analyses for customers.

Compuware Gomez, a well-known Web- and Web-applications monitoring company is expanding its network of performance-monitoring agents to cover the cloud as well, with

CloudSleuth, a set of analysis tools and educational resources.

"There are a lot of IT people who are a little afraid of the cloud, afraid it will either take away their job or afraid about the security," according to John Barnes of Model Metrics, a cloud-integration and consulting company that used to focus on performance metrics alone. "That's why you see so much interest in private cloud, it sounds safer.

Today, cloud providers are trying to provide online monitoring of performance and security to provide a level of transparency to customers, Barnes says.

But that doesn't mean IT will be able to compare one vendor's metrics to another's.

Follow everything from CIO.com on Twitter @CIOonline.