A fast-growing teleradiology firm found that a slowing storage-area network (SAN) interfered with its business of getting medical images out to radiologists for interpretation.
Virtual Radiologic offers radiology services to hospitals and radiological practices nationwide. The company provides gap coverage for evenings and weekends - in some cases, it actually provides a healthcare organizations entire radiology department.
The company, also referred to as vRad, builds virtual private network (VPN) tunnels over the Internet to its healthcare customers. Customers send their images - X-rays, magnetic resonance imaging (MRI) scans and the like - to vRAd's Minneapolis data center. The data center then pushes the images to vRad's 450 radiologists, who read the scans and dictate reports that go back to the customers. The company has more than 3,000 radiology group and hospital customers.
Analysis: Why Telemedicine Is Finally Ready to Take OffFeature: 12 Telemedicine Innovations That Will Shape Healthcare's Future
A SQL Server database application essentially matches each incoming image to the doctor best suited to review it. The application must quickly crunch a number of variables in real time to determine where the images should go. The company's SAN, however, began to slow as vRad's radiological workload expanded. Read/write times lengthened to 30 milliseconds, according to Patrick Williamson, IT manager at vRad.
"As time went on, and our database grew and our volume grew, the SAN just couldn't hold up," Williamson said. "We were outgrowing the Compellent SAN."
Against that backdrop, vRad deployed a new SAN, built on hybrid disk-flash storage arrays from Nimble Storage. The iSCSI SAN operates within a heavily virtualized environment, with all development servers and a growing portion of production servers having been converted to virtual machines in a VMware environment.
The new SAN, which went into production earlier this year, addresses the company's performance issues- latency is now less than a millisecond with Nimble - and boosts storage capacity. It also contributes to a reduction in data center floor space requirements and the resulting cost savings.
Outgrowing SAN 'Classic Story' of Latency, High Database Demand
Matthew Ushijima, principal consultant at GlassHouse Technologies, an IT infrastructure consulting based in Southborough, Mass., has seen vRad's situation emerge a number of times in his experience as an IT manager and consultant. As a business grows, demands on key applications increase, he says, and the database load becomes higher than the initial specification. Organizations employing a mid-tier array in such a setting may lack sufficient head room to expand.
"It is a classic story in IT," Ushijima said.
In vRad's case, its Compellent SAN worked well initially, when the company was conducting 9,000 studies a day. "We got the performance out of it that we needed," Williamson says.
As the volume edged upwards toward vRad's current peak of 24,000 studies per day, storage performance suffered. "Latency was a huge issue," he says. (Latency is a measure of the time it takes storage to respond to an I/O request - to read or write data - from an application.)
News: Flash Breakthrough Promises Faster Storage, Terabytes of MemoryMore: Flash Storage Offers Fast Path to VDI, Better App Performance
At vRad, storage performance problems impacted its SQL databases. Williamson says databases were getting slow and maintenance jobs were taking longer. The situation deteriorated to the point where database maintenance jobs overlapped and didn't always complete successfully, he notes.
The teleradiology operation had reached a key decision point: Rewrite databases to run on a slower SAN or deploy a new SAN. The company opted for the latter choice, rolling out Nimble CS460 GX2 hybrid storage arrays and the storage vendor's SmartStack for Business-Critical Applications. The latter is a converged infrastructure reference architecture, which includes Nimble's CS-Series arrays, Cisco Systems' Unified Computing Systems B-Series blade servers and VMware vSphere 5.1
Taking this storage approach resulted in a 5x reduction in write latency and a 25x reduction in read latency, according to vRad.
Jacob Wilde, lead systems engineer at vRad, says Nimble's take on shared SAN storage marks a departure from other storage vendors. Storage providers traditionally add more disk to get more performance out of a SAN, he says. Nimble, in contrast, achieves performance through CPUs, solid state drive caches, RAM and the Nimble file system.
"Nimble eliminates spindle count as being the traditional performance bottleneck for storage performance," Wilde says. He also notes that customers can further expand performance with version 2.0 of the Nimble operating system, which lets customers take advantage of the performance of up to four arrays in a scale-out cluster.
The teleradiology company can use fewer drives on the Nimble SAN compared with its former storage system, as it now obtains more IOPS per drive. In testing, the Compellent SAN had 114 x 15,000 rpm NL drives and 42 x 7,000 rpm NL drives, while the Nimble SAN had 25 x 7,000 rpm NL drives, according to Wilde. Using the SQLIO benchmarking tool, vRad found the Compellent SAN averaged 84 write IOPS per drive versus 820 on the Nimble SAN.
New SAN Also Meant No 'Massive Database Rewrite'
Performance isn't the only story, however. The overhaul also let vRAD avoid an expensive technical exercise. The SAN, Williamson says, "prevented us from going through a massive database rewrite effort, which would have been very costly from a personnel perspective." He estimates it would have taken vRad's development group six months to perform the rewrite at cost of at least $100,000.
The SAN has other cost-saving implications, too. The company has been able to virtualize more servers in light of the SAN's improved speed. "There were apps and servers we were never able to virtualize because local disk in the server was faster than the SAN performance we were getting," Williamson says.
Related: Will Fault Tolerant Servers Improve Your Virtualized Environment?Also: How to Manage Data Center Complexity
The company's development environment runs about 350 servers, all of which are virtualized. On the production side, about 250 out of 350 servers are now virtual machines. The virtualization program continues, Williamson says: "We are virtualizing as many servers as we can."
Virtualization, combined with a move to Cisco blade servers in 2010, has helped vRad reduce its data center footprint by half. Prior to the Cisco and Nimble gear, the company's data center was filled with rack-mounted servers, Williamson recalls. Now, vRad has stacks of decommissioned servers waiting to be sold or disposed.
The data center floor space reduction translates into an annual savings of $300,000 for vRad. That calculation takes into account factors including power and cooling in addition to real estate.
The company also saves on hardware acquisition. Williamson says vRad spins up a dozen or so virtual machines each month. If the company were to acquire physical servers at the same rate, it would end up spending $50,000 a month, he adds.
Williamson also notes that vRad was able to deploy more SANs due to Nimble's price point. The organization has deployed four SANs - two in production, one in development and one in its disaster recovery environment - for the same price as one Compellent SAN.
For companies such as vRad, storage plays a behind-the-scenes role as it helps deliver an organization's services. But it's a technology that emerges front and center when overtaxed.
"It's one of those things that isn't very visible unless there's a problem," says Dan Leary, vice president of worldwide marketing at Nimble.
John Moore has written on business and technology topics for more than 20 years. His areas of focus include mobile app development, health IT, cloud computing, government IT and distribution channels. Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn.
Read more about server hardware in CIO's Server Hardware Drilldown.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.