Riverbed wins 7-vendor WAN optimization test

As applications move to the cloud, network managers are seeing increasing requirements to optimize and manage WAN connections. Most enterprises have migrated to web-based applications and make heavy use of Internet services for day-to-day business. All of this makes network performance a key factor for productivity and end-user satisfaction.

Vendors have responded with a bevy of devices aimed at improving network performance. Many focus on just one function, such as compression, bandwidth management or flow visibility. Our last test of network optimization products was in 2007, so we wanted to find out if things had moved forward. Could network managers improve WAN performance without buying a stack of diverse boxes for every office?

We invited every major network optimization vendor, and ended up with seven contenders:1. Blue Coat Mach5 editions of the SG300-25 and SG900-102. Cisco Wide Area Virtualization Engine WAVE-7541, Cisco 4451-AX ISR and 2900-AX ISR.3. Citrix CloudBridge 20004. Exinda Networks Model 6862 and 10862 running x800-series software5. Ipanema Technologies ip|engine 1000ax and ip|engine 20ax6. Riverbed Steelhead CXA-5050 and CXA-5557. Silver Peak Systems VX-1000 and VX-5000.

We put each product in the lab for an intensive round of testing in three key areas of network optimization: performance, visibility, and control. We also looked at evolving parts of the network optimization market, including application-layer controls and active data link management. Finally, we evaluated each product for its enterprise suitability, flexibility, and ease of use.

Our Clear Choice Test winner is Riverbed, which excels at the core WAN optimization functions of compression and de-duplication. From a pure performance perspective, they know what they're doing.

However, WAN optimization has evolved to encompass other features, such as traffic management and visibility. Here, we find that Riverbed Steelhead could use some work. Traffic management is good, but not great. Visibility is limited in a way that pushes network managers to use Riverbed's own tools, rather than opening up to the growing world of standards-based flow analysis products. And new features, such as WAN path selection, don't live up to Steelhead's traditional technology leadership.

If you're looking for innovation, you'll be as impressed, as we were, with Ipanema Technologies ip|engines and Exinda Networks x800-series. These two products offer a rounded approach to network optimization that we didn't see in Steelhead. These vendors are clearly thinking beyond the basics to what the next generation of network optimization products should look like. As newcomers, though, we found glitches and holes in the products. For example, Exinda's management system is weak, while Ipanema's network integration and traffic management features are too rigid to work well with some networks.

For great performance, we were again impressed with Silver Peak, a top scorer in our 2007 test. Tied for first place in our compression and de-duplication tests, Silver Peak's VX-series and NX-series simply makes things go faster. However, Silver Peak is having a hard time shaking their data center-to-data center heritage, and the VX-series and NX-series need updates to handle the requirements of networks with many branch offices.

Cisco's WAAS product line is so broad that it's difficult to know what to test, and then how to rank them. As Cisco WAAS moves from standalone devices to ISR-integrated software, we gain all the power, sound, and fury of IOS, but at a cost in complexity. If you're happily committed to IOS on ISR devices at the edge of your branch network, adding WAAS is a no-brainer with big benefits at moderate cost. However, if you're mixing Cisco with other vendors at the branch edge, the decision rarely weighs in favor of Cisco for network optimization.~~

We were disappointed with Blue Coat's Mach5 and Citrix Systems' CloudBridge 2000. Both cover the basics and work as advertised -- although CloudBridge 2000 probably needs some shakeout before being deployed. However, Blue Coat has failed to learn from its own technology experts how to integrate traffic management with compression. And CloudBridge 2000 only includes the barest of features required to compete in this area. Neither company is a standard-bearer for how to push a broad spectrum of network optimization features into branch offices.

Here's a more detailed analysis of the products, broken up by functional areas:

PERFORMANCE TESTING: Silver Peak, Riverbed come out on top

Performance is one of the first reasons network managers start looking at network optimization products, which use a combination of techniques, including caching (usually called "de-duplication" to distinguish it from the kind of caching that web proxies do), in-line compression, TCP and IP protocol optimization, and application-specific optimization.

Each technique works in different ways, and, depending on your application mix, may be more or less beneficial. For example, if you move large text or database files around, in-line compression saves a lot of bandwidth. If you move a large data file that only changes a little bit each day, de-duplication helps. If your operating systems are tuned with small TCP window sizes or are using some types of congestion control, TCP/IP optimization helps.

We decided to focus on the one feature that really counts for WAN network managers: the end-user experience.

We focused on five types of traffic that we thought would be representative of many enterprise WANs: encrypted and unencrypted web traffic, email, remote terminal (specifically Citrix Xen Desktop), and Voice over IP. We ran each test across five types of WANs, using an InterWorking Labs Maxwell link emulator to vary latency and loss to simulate traffic ranging from across a data center to through a satellite link.

We felt that high latency links were an important part of our testing. Pure WAN compression and de-duplication rarely pays for itself in areas where bandwidth is inexpensive and abundant -- meaning much of the Americas, Europe, Asia and Australia. But when WANs are intercontinental, prices skyrocket and compression hardware can often be justified on a pure cost basis, ignoring other factors.

We also decided that we could not fairly test CIFS file sharing, even though we know that many enterprises still use CIFS across their WANs. Because of the huge variation in CIFS optimization, we knew that our results would be very skewed based on small variations in how we tested, and we felt that publishing results on CIFS might be misleading.

We started with HTTP and HTTPS traffic: internal web sites, SharePoint, web-based POS and ERP applications. Boiling down the hundreds of statistics available from Spirent's WebAvalanche product, we focused on transactions completed: If we installed this product in this network, how much more work could get done? To do this, we compared the transactions we could complete on a fully utilized 45Mbps circuit with and without optimization.

Here are the key findings:

HTTP TRAFFIC: Silver Peak and Blue Coat lead the way

• For pure HTTP, Blue Coat Mach5 excelled in high-latency networks with an amazing 260% improvement in completed transactions compared to the baseline.
• Silver Peak VX-series helped the most in low-latency networking, improving transaction count by 234%.  
• Overall, every product except Citrix CloudBridge did well, boosting performance by at least 170%.

For the type of traffic that we used in our tests (a mix of HTTP objects), we think that most network managers will find that end users report the best performance with Silver Peak VX-series and Blue Coat Mach5 and slightly lower performance, but not significantly different, between Riverbed Steelhead, Ipanema ip|engine, Exinda x800-series, and Cisco WAAS.

HTTPS TRAFFIC: Riverbed comes in first

• Different vendors have different approaches to HTTPS, however, it's pretty clear that the approach Blue Coat used wasn't the right one, as it dropped from being the best performing in low-latency HTTP networks to the worst performing in HTTPS networks. Plug a Blue Coat Mach5 into a typical HTTPS session, and you'll cut your performance by more than half.
• Riverbed shot far in front of everyone else in HTTPS acceleration: an average of 185% improvement in transaction rate compared to non-optimized traffic.
• Cisco WAAS and Silver Peak VX-series were next in line.
• Citrix CloudBridge was at the back of the pack.
• And Ipanema doesn't claim to support HTTPS optimization at this point.

The vast disparity between HTTP and HTTPS traffic is an important factor for network managers to consider. If your traffic is all HTTP and HTTPS and performance is your number one criteria, Riverbed Steelhead certainly leads across both protocols, with Silver Peak VX-series a solid contender.

EMAIL TRAFFIC: SilverPeak edges the competition

• When we tested email traffic, we found significantly less variability between products and protocols. The performance increases ranged from 140% of baseline to 166%.
• Silver Peak VX-series beat the competition in email compression, both in low-latency and high-latency environment.
• Blue Coat, Citrix, Exinda, Ipanema, and Riverbed are essentially indistinguishable, falling within a 5% range. 

CITRIX TRAFFIC: It's Silver Peak

• We tested Citrix XenDesktop, the most popular enterprise remote desktop protocol for end-user applications, but didn't expect to see much improvement because the Citrix application layer is already designed for WAN efficiency, and because Citrix XenDesktop includes both compression and encryption technology.
• We weren't surprised to find that Silver Peak VX-series led the pack in performance, but gave us only an 11% bump. 
• Cisco WAAS, Citrix CloudBridge, Exinda x800-series, and Riverbed Steelhead essentially came in with identical performance of 102% to 103%.
• Blue Coat Mach5 and Ipanema ip|engine trailed, in some cases actually causing a 2% to 4% performance decrease. 

Network managers may want to consider exempting Citrix XenDesktop traffic from optimization to avoid potentially slowing things down.

VOIP TRAFFIC: Silver Peak and Riverbed get the call

We looked at voice traffic (SIP-established Voice over IP) to see what happens with network optimization devices. Generally, the answer is supposed to be "nothing" for two reasons: VoIP traffic shouldn't be compressible, because the CODEC in the VoIP phone has already done compression. Also, connectionless UDP traffic doesn't fall into the "bump in the wire" model of the products we tested. To properly compress UDP traffic, the two devices would need to have an explicit tunnel so that they would know about the traffic, rather than auto-detecting compression via TCP options. ~~

• Silver Peak VX-series doesn't exactly operate in a bump-in-the-wire model, so it was able to get some performance gains out of our UDP traffic, an average of 9% improvement over baseline. 
• Riverbed Steelhead, even without specific tunnel configuration, was also able to offer a 7% increase in performance. The rest of the pack turned in expected numbers, ranging from 99% to 103%.
• However, these increases in performance came at some cost: increased jitter, which can be a serious problem for VoIP call quality. Both Silver Peak VX-series and Riverbed Steelhead kicked jitter up by about 10%. Our testing wasn't able to include call quality scores, but, generally, more jitter is worse.
• Citrix CloudBridge was able to reduce jitter across our test network by 16% compared to the baseline. 

Looking across all protocols, Silver Peak VX-series and Riverbed Steelhead were neck-and-neck in performance improvement, with Cisco WAAS next, followed by Exinda x800-series and Ipanema ip|engine. Our testing found that in performance, Blue Coat Mach5 and Citrix CloudBridge were at the back of the pack.

TRAFFIC MANAGEMENT: Ipanema shines

If making applications go faster is one goal of network optimization, selecting which applications get priority, reserving bandwidth, and policing non-critical applications are all equally valid goals. We lumped those features under the banner of "traffic management."

• Ipanema ip|engine came out on top, with a very sophisticated global application management system.
• On the other hand, Cisco WAVE engines have no built-in traffic management features. Cisco ISR-integrated optimization, though, gains all of IOS traffic management capabilities.
• Network managers who think that traffic management is important will want to focus on Exinda x800-series, Ipanema ip|engine, and Riverbed Steelhead as the products with the most sophisticated feature sets.  

Ipanema ip|engine offers the most innovative traffic management portfolio of any product we tested. Ipanema's ip|engine technology offers global traffic management in their Salsa management tool.

In a network which is a pure star, such as branch offices clustered around a single data center, traffic management is easy because the network is really a series of point-to-point lines, all of which can be controlled on both ends.

But in a modern network with multiple data centers, support centers, cloud-based communications, and branch-to-branch flows, the virtual point-to-point circuits disappear. Instead, each site may be communicating with multiple other sites, with potential for congestion and unhappy users.

Traffic management in this environment gets very difficult without some sort of global view of traffic flows on a moment-by-moment basis.

Ipanema's ip|engines are in constant communication with each other and can apply global traffic management when multiple sites, for example, threaten to overpower a single branch office with limited bandwidth. Ipanema's ip|engine technology really works, but it's not perfect.

We saw multiple Ipanema ip|engines devices coordinate traffic flows and provide coordinated traffic management across diverse sites. However, we also saw some anomalies pointing to a few bugs to be worked out. For example, we had locked a particular streaming application down to 256kbps, but saw our Ipanema ip|engines let through 50% more traffic for long periods of time. Our Ipanema support team initially blamed this on a lack of global device synchronization, but Ipanema later told us that synchronization was not required -- which left us more in the dark about the inaccurate traffic management behavior.

Network managers also need to be aware that Ipanema ip|engine only supports a very rigid network architecture and traffic management model. When defining global traffic flows, Salsa manages bandwidth on an application basis, not on a per-site basis. Thus, you can say "video conferencing gets a guarantee of 512K per call" or "Citrix XenDesktop gets a guarantee of 64K per session" but there is no easy way to say how many video calls or Citrix XenDesktop sessions can be going in or out of a particular site or what percentage of a site's bandwidth can be taken up by a particular application.

We think that some network managers will find Ipanema's ip|engine traffic management technologies the answer to their dreams. However, Ipanema's model of networking and traffic management doesn't have room for negotiation -- either your network and traffic model looks like Ipanema wants, or they won't be able to solve your traffic management and application assurance problems.

For the other six products, we verified correct operation of outbound and inbound traffic management on a site-to-site basis (since no other product included Ipanema's global traffic view). We based all testing on a typical WAN traffic policy. All of the products we tested passed these tests, subject to differences in what each product could support. In some cases, we had to vary our policy slightly. For example, our test policy called for policing of recreational Internet usage, but not every product could identify applications such as BitTorrent and differentiate them from non-recreational Internet usage.

Riverbed's Steelhead and Exinda's x800 appliances showed sophisticated traffic management capabilities that made us rank them below Ipanema, but above the rest.

One key feature that stood out with both products was layer 7 application identification: the ability to apply traffic management rules based on the application being used and not just a simple tuple of IP address and port number.

Exinda's heritage is as a traffic management company, which made it natural that it would excel in this part of our test. For example, the Exinda x800 can apply both per-site and per-host traffic management rules, giving the network manager the opportunity to provide some "fairness" in sites where heavy streaming usage -- or someone downloading the 1Gb iOS7 upgrade -- can impact business traffic on Internet links.

With Exinda x800 per-host "dynamic virtual circuits," hosts (specified by subnet, VLAN or other qualifiers) can be given a chunk of bandwidth to share fairly, on top of other per-site, per-subnet or per-application limitations. Our testing found this to work properly, with one exception: traffic management and Exinda's Internet web cache feature don't mix, something Exinda told us they would fix in a future version of the product.

Both Exinda x800 and Riverbed Steelhead have inbound and outbound traffic management capabilities, although the Steelhead is strangely asymmetric in its configuration. Riverbed recently added a nice hierarchical model for traffic management, but this was only extended to outbound traffic. Exinda x800 has the same hierarchical model in both directions.

We also preferred Exinda's x800 traffic management for a host of smaller features not in Riverbed Steelhead, but which could be very helpful in defining a corporate network policy. For example, the Exinda x800 has time-of-day rules for traffic management and the ability to express bandwidth both in percentages and in absolute values.

The Exinda x800 also has a feature that higher education network managers will appreciate: users who exceed a certain quota of traffic over some time period can be sent to a temporary "jail" that limits the amount of bandwidth they can use. Riverbed's Steelhead doesn't have any of these capabilities.

While Blue Coat Mach5, Citrix's CloudBridge 2000 and Silver Peak's VX-series all had traffic management features, they brought a less capable set of tools. For example, the Citrix CloudBridge 2000 has a sophisticated way to prioritize and police applications, but doesn't let you guarantee a particular bandwidth slice to any application.~~

WAVE not WAAS

Cisco deserves special mention here. When we started testing, Cisco only had its standalone WAVE appliances, devices that had sophisticated WAN de-duplication and compression features, but no way to perform any type of traffic management.

Midway through our test, Cisco released a new mid-sized router in its ISR family, the 4451-X. This new hardware includes IOS-XE and the capability to run internal virtual machines, including the Cisco Wide Area Application Services (WAAS) engine. At the same time, Cisco introduced a new licensing variation in the ISR family called "AX" (for "Application Experience"), which includes four options bundled together: WAAS, Application Visibility and Control, Security and the normal Data license. This new license, combined with some automatic configuration wizards on the 4451-X, drastically simplifies the process of adding network optimization features to the 4451-X ISR platform.

So, we decided to focus on the IOS-integrated WAAS offering rather than Cisco's standalone WAVE devices. Network managers considering Cisco for network optimization need to be careful not to confuse the Cisco standalone options, which have been the flagship of its WAN acceleration offering, with the new bundled ISR devices. Standalone WAVE still has its place in certain environments, but most network managers should aim for the IOS-integrated appliances for both features and cost-effective deployment.

We based our testing of Cisco's acceleration performance on the standalone WAVE appliances, which should behave the same as the new integrated WAAS systems, since they are running identical software. Standalone Cisco WAVE devices have absolutely no traffic management capabilities and very limited visibility features.

IOS and IOS-XE both have a comprehensive set of traffic management tools, which puts Cisco roughly in the same ranking in our testing as Blue Coat Mach5, Citrix CloudBridge, and Silver Peak VX-series. However, Cisco IOS also has application identification features (NBAR) that can be combined with traffic management to go beyond this baseline.

On the other hand, using NBAR to affect traffic management with a WAAS engine in the middle results in an unholy mess of unmaintainable configurations. While the CCIEs out there may be able to handle such a thing, we feel that IOS or IOS-XE combined with the WAAS engine don't deliver a long-term best choice for traffic management.

VISIBILITY: Exinda, Riverbed excel

When dealing with hundreds or thousands of branch offices, the ability to drill down quickly and diagnose network and application problems is a major competitive edge. Visibility helps in other ways besides problem solving -- capacity planning, trend analysis, and usage tracking are all improved by good visibility.

Since a network optimization device sees WAN traffic before it gets VPNed and NATed by firewalls, this is a great place to get visibility on traffic leaving the branch office into the corporate network or to the Internet.

Our testing focused in evaluating the capabilities of products for both short-term and long-term traffic analysis. We found that Riverbed Steelhead and Exinda x800 offered the strongest and broadest visibility feature set -- although getting the full value out of Riverbed Steelhead also means budgeting for the Riverbed Cascade analysis tool.

Next in line were Ipanema ip|engine and Silver Peak VX-series, with Blue Coat Mach5, Citrix CloudBridge, and Cisco WAAS bringing up the rear.

SHORT-TERM ANALYSIS: Exinda, Ipanema offer richest context

We started out by asking the question: "Who is doing what on my network right now?" We found all the products to be similar. All give an overview of network performance, including compression, and have the option to see currently open connections. Some even give you a drill-down -- Cisco WAAS will tell you, for example, everything you ever wanted to know about an open connection. That's great for debugging the network optimization tools. The biggest difference between devices at this level was their ability to use application identification technology to add more context: only Ipanema ip|engine and Exinda x800-series provided this type of information.  (Riverbed will be adding its application identification information to flow data starting with the next version, 8.5, of the Steelhead operating system RiOS.)

With Blue Coat Mach5, Cisco WAAS, Citrix CloudBridge and Ipanema ip|engine, your per-site information essentially stops there, and there's no real information about past sessions stored on their appliances.

Silver Peak VX-series, Riverbed Steelhead and Exinda x800-series all go further, providing flow history data on the device that lets you pivot through different views, such as by IP address, application group, and application. Exinda x800-series adds a bonus: mapping user names to IP addresses through a Windows Active Directory Domain Controller connector, which enables reporting and traffic management based on user name as well as IP address. Of course, there are scalability issues with this approach, but in a branch office with a single Windows Domain and a single network optimization box, it all works rather well.

Our testing of on-box traffic information focused on short-term flow and traffic information. For long-term traffic data, we went looking for other solutions.~~

LONG-TERM ANALYSIS: It's Riverbed, Blue Coat and Silver Peak

We believe that on-site appliances shouldn't be asked to be long-term repositories of information for two reasons: performance impact on the end device and lack of global view from a single station. The performance requirements to handle storage and analysis of traffic can be tricky and asking a device sitting in-line in the branch to store more than a month of data isn't a great idea.

A global view is also a critical reason to look at long-term reporting. Any one appliance can tell you a lot about what happened at that site, but when a network or application manager is hunting problems with a larger scope, or is looking at trend information across the whole network, a larger view is important. The problem is compounded at the data center end, where multiple devices may be present in multiple data centers and need aggregation.

The obvious solution for exporting flow, traffic, and optimization data over the long term is to use a standard such as NetFlow v9 (or the IETF replacement, IPFIX). When we started this test, we assumed we'd get 100% coverage, but we were surprised: Blue Coat Mach5, Citrix CloudBridge, and Ipanema ip|engine don't go there yet -- Blue Coat told us they were adding the capability, and Citrix said they would get equivalent functionality through a partnership with Splunk.

Cisco's standalone WAVE systems -- from the company that invented NetFlow -- don't do NetFlow either, but the WAAS integrated into IOS can export NetFlow based on IOS' capabilities.

Riverbed Steelhead, Exinda x800-series, and Silver Peak VX-series all export NetFlow. However, network managers have to be careful here, just because the optimization information is sent out in NetFlow doesn't mean that every NetFlow collector and analyzer can do anything with it. Riverbed clearly understood that problem when it bought Mazu Networks in 2009 and re-launched Mazu's analysis tool as Riverbed Cascade, giving Riverbed a top-notch analysis tool with extensive application analytics.

We looked briefly at Exinda's analysis tool, NRC (an OEM product from Vineyard Networks), but they told us that the tool will be replaced by a new central management and analysis platform at year's end. This leaves Exinda x800-series at a disadvantage, between long-term flow analysis platforms, compared to Riverbed Steelhead, Blue Coat Mach5, and Silver Peak VX-series.

Along the way, we tripped across a nifty feature in Ipanema and Exinda products (and available as part of Riverbed Cascade): application performance monitoring. With application monitoring, network optimization appliances can become first steps at answering the question: "Is this a network or a server problem?" For example, Ipanema ip|engine appliances and Salsa management tools collect network delay and jitter, packet loss rates, server response time, and TCP retransmission statistics for each defined application. This fits beautifully in Ipanema's model, which is all application-based, and allows the Ipanema Salsa management system to report on overall user experience. Since one of the main reasons to install network optimization is to improve end-user experiences, application performance monitoring struck us as a great complementary feature.

WAN PATH SELECTION: Tread lightly

When WAN branch offices have multiple links for reliability, network managers understandably want to use every bit of outbound bandwidth they're paying for to improve performance. Outbound load sharing is a feature common in branch office firewalls, but the load sharing is usually fairly simplistic. Recently, the term "WAN Path Selection" has been used to talk about a more sophisticated type of outbound load balancing.

With WAN Path Selection, the outbound path for a network flow is selected based on configured-in application performance goals and constantly measured link information. For example, if video conferencing is a business critical application requiring low jitter and high throughput, WAN Path Selection might pick whichever link happens to have the lowest jitter and highest throughput at the moment. And, while the videoconference is going on, WAN Path Selection might also keep other flows from running over the same circuit.

Because the Network Optimization devices we tested all include some type of traffic management controls, WAN Path Selection is an obvious additional feature to help improve overall network performance.

WAN Path Selection is such an interesting feature that when we started our testing, only one product supported the option (Ipanema ip|engine) and by the time we were finished writing, both Cisco and Riverbed had added it to the products we were testing. Since only Ipanema ip|engine included WAN Path Selection when we started, we did not do rigorous testing. Network managers should also note that the comparative pricing for Ipanema's product provided in this test does not include the additional license required for Dynamic WAN Selection (Ipanema's name for the feature), at $8,700 for the central site and $725 for each branch office.

While WAN Path Selection sounds like an interesting idea, when we looked deeply at how the network optimization vendors were implementing it, we found some significant gotchas. The biggest issue is that WAN Path Selection can only really be properly done in the edge firewall device, not a network optimization device that sits behind the firewall. The conflicting goals of the optimization and the firewall are to blame here.

Normally, the network optimization device sits inside the firewall (towards the LAN side of the branch office). This is because the firewall will have VPN tunnels to the rest of the network, and the optimization device can only work effectively on unencrypted traffic. However, for the network optimization device to perform WAN Path Selection directly, it has to be outside the firewall (towards the WAN side of the branch office) and in-line with each link leaving the branch office.

Working around this fundamental problem has smart people thinking overtime about potential solutions. The Ipanema ip|engine has a Rube Goldbergian scheme to perform WAN Path Selection by indicating a preferred route to the upstream firewall or router using some of the DiffServ bits in the IP header or hard-coding MAC addresses of upstream routers -- solutions that can cause more problems than they solve. Riverbed's engineers suggested a different approach of running the traffic through the network optimization box twice, straddling the firewall. Inside the firewall their device would optimize the traffic and mark priorities. Outside the firewall, the traffic would pass through again, and the Steelhead could look at the markings left over from the first pass to shape the traffic and control WAN routing.

In the short term, until companies such as Ipanema can actually fully replace edge firewall/router devices, the only real contender in this space is Cisco with a fully integrated router, firewall, and network optimization device in one chassis. Since ISR-integrated version of Cisco's WAAS has a similar technology (Cisco's name for it is Cisco Performance Routing, which they acronymize as PfR), the integrated ISR can make a WAN path selection routing decision, apply traffic management, and then pass the network flows through the WAAS engine for optimization.

Some network managers with different network topologies from the one we used may find that Ipanema ip|engine and Riverbed Steelhead work great out-of-the-box. However, WAN Path Selection introduces a very different paradigm for these devices, because all but Cisco ISR-integrated WAAS currently operate as a "bump in the wire." Considerable engineering and expertise has gone into making the bump very innocuous. When a device suddenly starts looking like a router more than a bridge, that's a very different kind of design and very different behavior that is incompatible with most current deployments. You're throwing out a lot of experience and knowledge and venturing into unknown territory.

For this reason, we'd advise extreme caution when looking at this feature. Fundamentally, IP routing is incompatible with the idea of keeping a single flow running across a particular network path, especially when asymmetric routing is possible. This doesn't mean that WAN Path Selection can't work, just that the places where it can work may be very limited or extremely restricted network environments.~~

WAN MANAGEMENT: Cisco, Ipanema, Riverbed offer strongest tools

We prepared a set of testing criteria for enterprise suitability but found little difference in products except when it came to management. Areas such as high availability and clustering were all nearly identical.

When we looked at SNMP support we also found some variation. For example, Citrix left out SNMP v3 support and Ipanema doesn't approve of SNMP for device management, sending you to the global management system instead. Our testing shows that Riverbed Steelhead has the most sophisticated SNMP management available in its devices. But while these differences indicate a relative lack of maturity in some products, it's hard to get too excited about these differences.

Instead, we chose to focus on centralized management and reporting. We found a dramatic spectrum from almost nonexistent (Exinda) to nearly unusable (Blue Coat and Citrix) to chaotic (Silver Peak) to comprehensive and well-thought-out (Cisco, Ipanema, and Riverbed).

In many cases, we also found a strong distinction between performance monitoring and network management, often with completely different products needed to solve the two problems. We found that all products have a very low management requirement: get them set up, and as long as your environment doesn't change, you really don't have to touch them.

Certainly the strongest management tools came from Cisco and Ipanema, with Riverbed a very close second place. Cisco and Ipanema both take a systemic approach to management of network optimization, meaning that you rarely have to dive into individual device configurations. Instead, you configure a general set of policies that apply across the entire deployment and the management systems take care of keeping configurations in synchronization.

Ipanema's Salsa management tool is definitely the most sophisticated in the network optimization market, automatically handling a wide variety of configuration tasks focused on application performance. With Salsa, the network manager lays out applications on the network, bandwidth required for each, and, well, that's about it. Everything else is automatic. However, Salsa is missing other types of management that we'd expect at this level. For example, software updates are unreasonably complicated, especially for a company making rapid changes to their software. Salsa goes way, way beyond what everyone else does in this space, but then it hits the wall so severely that it makes a dent where other areas, such as element management and reporting, are concerned.

Cisco's management tools also take a desirable systems approach to setting up network optimization, although this approach is being fragmented by Cisco's new ISR-integrated approach. With standalone Cisco WAVE boxes, a single management tool, the WAAS Central Manager, is all you want and all you need. For a company that has consistently had problems with network management, the WAAS Central Manager is amazingly good and stands out as one of the strongest parts of the product and one of the most elegant management systems in this space.

However, when the ISR-integrated WAAS is used, Cisco now requires two management systems: the legacy WAAS Central Manager for basic WAAS acceleration operations, and Cisco PRIME for management of the IOS wrapper that handles other parts of network optimization, such as traffic management and visibility. Cisco PRIME lacks the smooth polish of the WAAS standalone manager, and because it is early in PRIME's life cycle, still has a number of glitches. Network managers selecting Cisco PRIME should plan on losing time to minor bugs and an aggressive upgrade cycle the first few years.

Riverbed's Central Management Console is more element-focused than either Ipanema's Salsa or Cisco's WAAS Central Manager tool, but is a very strong, stable, and capable management system. With full coverage across most product features in a single tool, Riverbed has the strongest and most consistent management system. The lack of a "full network" view of policy and traffic management will create extra work for network managers, but the stability, solidity, and comprehensive view of each device makes up for this deficit. Any network manager choosing Riverbed Steelhead for their network optimization will be impressed by the quality of the management system, including ease of use and consistency.

Compared with Riverbed, Ipanema and Cisco, the rest of the network optimization products we tested look amateurish in their management systems. Silver Peak decided that it was unhappy with its existing Global Management System (GMS) management tool and re-wrote it, but not completely, leaving network managers to run two separate clients (one in Java) with very different interfaces in order to get any real work done.

Citrix didn't even bother to write a real management system for its network optimization devices -- it had one left over from its load balancer and grafted the network optimization configuration on top of the existing tool, leaving much of the management system and reporting engine dangling in space without really restructuring to handle the network optimization case. Similarly, Blue Coat re-purposed its proxy management tool with a bevy of obscure details and confusing sub-options to handle the network optimization product.

Exinda's management, like Silver Peak's, is in transition except that Exinda is nowhere near as far along as Silver Peak. Instead, Exinda has a cobbled-together a set of third party, cloud, and on-premises solutions which together don't cover the bases. Network managers choosing Exinda should be prepared for a lot of command line configuration until Exinda's new management system comes out. And from the preview that Exinda showed us, the new management system is not going to be complete anytime soon. Reporting is in good shape in the beta we saw, but configuration is non-existent.

EASE OF USE: Riverbed rules

As an early master of the field of WAN compression and de-duplication, Riverbed set the stage for the marketplace with products that are both flexible and easy to use. Riverbed Steelhead devices don't require complicated configuration to define network topologies. Instead, if you drop a Steelhead in-line with an existing network connection, it runs with almost no programming and no continuous management or updating.

With Riverbed Steelhead setting the bar, everyone has fallen into place and done a pretty good job. The laggards here are Silver Peak and Cisco.

With Silver Peak VX-series, the connection between two devices is more explicitly managed because a tunnel is always established between devices. This means that you have to put VX-series outside of any firewall function, because a firewall on the WAN side of the VX-series cannot apply normal controls. Silver Peak VX-series will automatically create tunnels if you want, but then the tunnels have to be managed if you want to use traffic management features, creating an additional and unnecessary burden, especially in a large WAN. Silver Peak's heritage as a data center-to-data center company, where the number of devices in the network is tiny, is showing through here with this design defect.

We found two significant issues with Cisco WAAS and network transparency, depending on whether you use the standalone WAVE devices or the WAAS integrated into IOS. With standalone WAVE devices, Cisco's WAAS software is incompatible with any good firewalls that check TCP sequence numbers. If you have a good firewall, you need to turn off TCP sequence number checking -- a security function that was put into the firewall for a good reason. Cisco's own ASA firewall has a special WAVE-compatible mode to handle this problem, so if you have ASA firewalls and want to put in standalone WAVE devices, you don't have to turn off this important security feature for all connections, just those through the WAVE device.

On the other hand, if you choose to use IOS-integrated WAAS, then you have a different transparency problem because the IOS-integrated version cannot operate as a bump in the wire transparently to existing networks. That works well with Cisco's positioning of IOS and IOS-XE as the do-it-all box for branch offices, but network managers who want to keep separate devices or who haven't committed to an all-Cisco WAN won't be able to deploy the IOS-integrated WAAS without significant re-engineering.

We also marked down Citrix CloudBridge 2000 in the ease-of-use area because device-to-device communications are only automatically managed when you're not accelerating SSL connections. Once you get SSL into the picture, then Citrix CloudBridge requires the network manager to do much more explicit management and configuration of topology, complicating deployments unnecessarily.

We ran into another significant usability issue with CloudBridge 2000 when we rebooted our test system and the SSL stopped working. Why? Because Citrix CloudBridge requires the network manager to log into each device after it has been rebooted and manually unlock the SSL key store before it will start accelerating SSL connections again. Yes, that's more secure, but you can't turn it off, and if you put these devices in a branch office that has less than perfect AC power, you're gaining a world of aggravation. SSL seems to be a particular weak spot for the CloudBridge 2000.

Another flexibility area we evaluated was virtualization. Not every network manager will want it, but for those that do, it's an important feature. Every product we tested supports deployment as a virtual machine. In fact, some prefer it: Silver Peak told us that it would rather run the VX-series as a virtual machine unless you're hitting their top performance tiers. Others are running as VMs whether you want to or not. For example, the Citrix CloudBridge 2000 we tested is really a virtual machine sitting on a Citrix hypervisor, while the Cisco ISR-XE WAAS is a virtual machine running inside of IOS.

Even more interesting is the option to use the optimization device as a virtualization host all by itself. This might let you run a small file and print server or Windows domain controller in a branch office, combining two pieces of hardware into one. Riverbed and Cisco have already shipped this capability (although Cisco does require additional hardware in the form of a UCS-E blade), and both Citrix and Exinda told us that they have plans in this direction on some hardware platforms.~~

Wan Optimization: The Virtual Option

All of the vendors we tested now offer virtual versions of their products to run in branch offices. Many also offer virtual versions for data center deployments as well, and there are a number of true believers in the pack.

Citrix, for example, ships their product on their own hypervisor.  If you buy an appliance, they'll send you a server -- but the CloudBridge software is running in a VM.

Silver Peak is so confident in their virtualization performance that it didn't even ship us dedicated appliances -- it had us run their software as a VM within VMware. Because our testing proved that Silver Peak kept up or beat the rest of the market when it came to performance improvements, we're true believers now too. If you want to run this stuff in a VM, go ahead.  It works great.

How we tested network optimization products

We tested network optimization devices in our lab to evaluate their capabilities in eight areas: performance, traffic management, visibility, application controls, data link management, enterprise suitability, network flexibility, and ease of use.

We designed a small network based on Cisco routers and Juniper firewalls to simulate the operation of a large enterprise wide-area network.  To reproduce the conditions of an intercontinental network, we used Interworking Labs Maxwell line emulators to selectively introduce bandwidth bottlenecks, latency, and errors into the network. Our goal was to simulate a network of about 100 sites connected via standard IPSec tunnels using approximately 45Mbps of bandwidth over the WAN.

At the edges of the network, we used virtualization software from VMware to host various network performance testing tools, including open source products as well as our go-to solution, Spirent's WebAvalanche. To get the performance we needed to take our network to 45Mbps, we spun up eight separate Spirent virtual machines and spread the load across all eight systems to ensure that the network was the bottleneck, not the testing systems.

We asked each vendor participating in this test to provide two devices that would be sufficient to handle 2,000 users and an aggregate bandwidth of 45Mbps for our performance testing. We also asked for a lower-speed device to be used for functional testing and evaluation of any central management capabilities.

Once we had our test lab put together, we worked with Spirent and David Newman at Network Test to develop testing plans for the most common enterprise protocols: web browsing (both encrypted and unencrypted) over HTTP, electronic mail, thin client using Citrix XenDesktop, Voice over IP using SIP, and basic bulk data transfer. We tried to mix compressible traffic with non-compressible traffic to see how these devices would perform in an enterprise setting.

Although we acknowledge that many enterprises are still using CIFS (SMB) file sharing across their WAN, we chose not to try to test the performance of CIFS optimization because of the immense variation in use patterns. Network managers who are looking at these products as a way of optimizing CIFS traffic should design their own tests to tease out the huge differences in CIFS support across products.

We ran baseline tests across our testbed to determine transaction rates without any specific network optimization or traffic management technology. We ran each test three times, taking the average of the three runs and re-doing tests as needed until we achieved consistent and repeatable results. Our testing was done at five round-trip latencies: 0 ms, 50 ms, 100 ms, 200 ms, and 700 ms. These represented across-town, regional, national, international and satellite latencies.

Then, using configurations designed and approved by each vendor, we re-ran our tests to saturate the testbed's WAN circuits at the five latency levels. Our goal was to measure the increase in performance in the presence of optimization technology. Our main metric was transaction count rather than raw throughput, as we feel that this better represents the experience that end users have in a WAN environment. Network managers looking at this technology for other reasons, such as data center-to-data center replication, should not consider our WAN testbed a good simulation of inter-data center traffic.

For performance testing, we looked at only one type of traffic at a time. When it came time to test traffic management, we mixed up different types of traffic, prioritizing and allocating bandwidth to prefer real-time traffic over bulk transfers.

To test visibility capabilities, we gathered approximately 10Gb of traffic from a live corporate WAN, mixing both corporate and Internet applications together. We played this traffic back through the optimization devices. This gave us a nice mix of applications so that we could see how different applications were identified and categorized in the various management systems.

We chose to test standards-based mail protocols, rather than encrypted MAPI, because we believe that most security-conscious enterprises have switched from direct MAPI connections to either RPC-over-HTTP (available since Outlook 2003) or standards-based email protocols IMAP and SMTP. However, for enterprises that are still using old-style MAPI protocol, several of the vendors we tested have specific support for encrypted MAPI. Network managers looking to add optimization technology should make sure they have a very clear discussion with their email team to be sure everyone understands exactly what protocol is in use. A few packet dumps wouldn't hurt either.

THANKS

Network World would like to express its thanks to Spirent, for providing WebAvalanche software and support, and Interworking Labs, for providing Maxwell link emulators.

Snyder, a Network World Test Alliance partner, is a senior partner at Opus One in Tucson, Ariz. He can be reached at Joel.Snyder@opus1.com

Read more about lan and wan in Network World's LAN & WAN section.