John Oliver's recent net neutrality diatribe, which has attracted more than 3.8 million YouTube views, spotlights Netflix ISP Speed Index data showing Netflix streaming quality deteriorating and then improving after Netflix and Comcast began to directly connect. But what does the Netflix Index data really show?
We analyzed Netflix service performance swings during the period under scrutiny using Netflix's own data, and found that a dozen ISPs experienced a common pattern of Speed Index deterioration and improvement. We set out to understand why, and here's what we found.
The Netflix content delivery system has undergone four architectural configurations, each with a different critical server-to-player streaming path that affects Netflix ISP Speed Index results. In mid-2013, Netflix shifted its content delivery system from public Content Delivery Networks (CDNs) like Akamai to its own CDN called Open Connect. To connect the Netflix CDN to the Internet, Netflix uses a variety of "middle-mile" ISPs, including Cogent Communications and Level 3 Communications. As the amount of traffic Netflix served from its own CDN increased, some of its "middle-mile" Internet connectivity suppliers apparently lacked sufficient capacity to service the Netflix load, and video performance levels, as documented by the Netflix ISP Speed Index, dropped.
NetForecast analyzed the Speed Index data on the Netflix website to gain insight into this performance issue. The performance degradation, which peaked in January, was clearly evident in the data. We gathered Speed Index data for all U.S. ISPs Netflix reports on for the period covering four months preceding and four months after the congestion peak, and found 12 ISP networks for which performance degraded from September to January and improved from January to May. The adversely-affected ISP networks included:
- AT&T -- DSL
- AT&T -- Uverse
- Bright House -- Cable
- Centurylink -- DSL
- Clearwire -- Wireless
- Comcast Cable
- Frontier -- DSL
- Fiber Mediacom -- Cable
- Time Warner Cable
- Verizon -- DSL
- Verizon FIOS -- Fiber
- Windstream - DSL
The problem did not affect last-mile ISPs that embedded Netflix CDN servers in their network, because the video content destined for their users did not traverse the congested middle-mile ISP networks.
The figure below shows the Netflix Index normalized for each affected ISP relative to their Index value in September. All curves are negative by January and improved by May relative to their January values. By April, half of the curves show improvement in video performance relative to September, the start of the analysis period.
In aggregate, the 12 affected ISPs serve more than 70 million homes. Although not all of the homes subscribe to Netflix, the population contains a very substantial portion of Netflix customers.
A Netflix-Comcast arrangement directly connected the Comcast and Netflix networks, thus bypassing the Netflix middle-mile ISPs. As far as NetForecast knows, no other changes were made in either the Netflix server-to-player system or the Comcast network. The only change resulting from this arrangement was to remove the middle-mile ISP bottleneck between Comcast subscribers and the Netflix datacenters. Since Comcast has the largest user population among the dozen impacted networks, direct connection between Comcast and Netflix removed about 30% of the total Netflix traffic destined for the 12 ISPs via the middle-mile ISP networks used by Netflix.
The takeaway is that the Comcast-Netflix interconnection arrangement not only improved performance for Comcast-connected Netflix users, but it also improved performance for the other 70% of the Netflix user population accessing videos over the Netflix Open Connect CDN.
You can find more information about the NetForecast analysis in a NetForecast report on the subject here.
In the interest of full disclosure, Comcast partially funded the NetForecast data analysis.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.