Verizon has called a truce in its ongoing war of words with Netflix over who is responsible for poor quality of experience over its network for the latter’s programming. But this is unlikely to end either this immediate battle or the larger war between major OTT content providers and ISPs over who should pay for delivery, which has flared up particularly in the US with respect to Netflix.
This simply reflects Netflix dominance there, accounting for one third of all Internet download traffic according to a study by Canadian networking equipment vendor Sandvine. Comcast and AT&T have also this year been involved in similar verbal jousting, although with differing outcomes so far in each case. Netflix agreed to pay Comcast for improved interconnection but then complained that it should not have had to do so. This led AT&T to weigh into the debate in June by accusing Netflix of double talk in feeding ISPs and biting them at the same time. Netflix has so far refused to enter into a similar interconnection agreement with AT&T because it puts the blame for poor streaming performance squarely on the latter’s U-Verse service.
To make the story more confusing, Netflix did reach an interconnection agreement with Verizon in April, but since then performance has continued to deteriorate. Matters came to a head in June when Verizon received an email from one of its customers in Los Angeles asking why experience watching Netflix was so poor an experience on a FiOS connection that appeared to be running at 75 Mbps. This came amid claims that Verizon was deliberately ‘throttling’ Netflix traffic, which would have contravened net neutrality.
Such accusations cranked up Verizon’s publicity machine and prompted its Vice President for Federal Regulatory Affairs David Young to write a blog ‘setting the record straight’. This coincided with Netflix publishing its monthly data on average bit rate obtained by its users over each of the major 16 US ISPs. Verizon’s DSL service was bottom at just 0.91 Mbps, while even FiOS was only 12th at 1.58 Mbps, having fallen from 1.9 Mbps in May and 1.99 Mbps in April.
This is barely good enough for what Netflix defines as its baseline quality standard, which requires 1.5 Mbps. It is worth noting though that only the number one ISP on the list Cablevision, with an average of 3.1 Mbps, surpassed Netflix’ recommended bit rate for standard definition programming, with all of them way short of the 5 Mbps for HD. It is a bit of a joke therefore that Netflix has been bragging about streaming in 4K, which even with the help of HEVC encoding would be out of the question at these lamentable bit rates.
It is also worth noting that it is the smaller ISPs that are delivering the best performance for Netflix traffic, with Cablevision followed on the June league table by Cox at 2.99 Mbps, while Suddenlink and Charter are joint third on 2.86 Mbps. The explanation for this goes to the heart of the issue, since Cogent, which provides IP transit service for Netflix in the US, finds it easier to move traffic into smaller ISP networks. In turn such networks have seized on the issue as a way of gaining subscribers, aiming to offer high quality of service for Netflix programming by ensuring there is sufficient capacity at the interconnects. Mostly they are not charging Netflix either, certainly not Cablevision or Time Warner, although that may change for the latter following its acquisition by Comcast.
It is a big issue for the larger ISPs since a lot of money is at stake, as well as control over the infrastructure. Netflix is building out its own CDN, which will progressively replace those provided by the likes of Cogent and will be seeking favorable interconnection agreements. The ISPs meanwhile are holding out for the most they can get, as well as engaging in a publicity offensive as regulators start paying closer attention to interconnection.
In that blog ‘to set the record straight’ Verizon’s Young argued that the poor and declining bit rate experienced by people viewing Netflix over its network was caused by the poverty of the interconnects that had been deployed. He insisted there was no congestion at all over Verizon’s network as a result of carrying Netflix traffic even at peak times. But at such times the interconnects between Verizon’s network and the links chosen by Netflix to download video were at times 100% loaded. Verizon had even drilled down to the links supplying the Los Angeles customer who had complained about poor Netflix performance and found that they were frequently congested too.
Young mused over why Netflix was alone in suffering from these congestion issues. ‘What it boils down to is this,’ he wrote. ‘Other transit and content providers took steps to ensure that there was adequate capacity for their traffic to enter our network. In some cases, these are settlement-free peering arrangements, where the relative traffic flows between an IP network provider and us, remain roughly equal, and both parties invest in sufficient facilities to match these roughly equal flows. That is the traditional basis for such deals. In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.’
We had supposed Netflix and Verizon had entered into such an agreement, as had been widely reported in April. But clearly Netflix was not prepared to pay the price demanded by Verizon, anticipating perhaps that the ISP would be compelled to bring the interconnections up to scratch itself rather than stand accused of throttling traffic. We are now in a battle for the moral high ground, with Young concluding his blog with an assurance that even though the ISP was not to blame it would ensure that its customers received the best quality of experience for all the services they access. His wording was interesting. ‘We are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network. This doesn’t ‘prioritize’ Netflix traffic in any way, but it ensures that their traffic gets on our network through direct connections’not middleman networks’that are up to the task.’
It seems then that Verizon will be connecting directly to Netflix in future but will be charging extra for that. Young suggested that already tests were showing improved performance. July’s ISP league table from Netflix will therefore be interesting. For all Netflix’ complaints about having to pay for interconnection it will be smaller OTT service providers that risk falling behind as they will not be able to afford direct interconnection and may go on having to rely on these middleman networks that Verizon’s Young derides.
But for precisely such reasons regulators are now starting to delve deeper into the Internet access structure as they appreciate its relevance for competition and consumer interests. In June Federal Communications Commission (FCC) chair Tom Wheeler announced that it would no longer be confining its attention to the ‘last mile’ but would be looking at regulating the ‘rest of the Internet’, including interconnection arrangements. This was taken as implicit support for the Netflix argument that such connections should be free, but Verizon contends that if the big content providers do pay, then this actually benefits small emerging players, since it will alleviate congestion at the existing interconnects that are open to everyone.
That was certainly the case at Verizon’s border router whose congestion had been responsible for the customer complaint. To an extent the dominant Netflix position is bringing the US debate over interconnection to a head, for otherwise it is likely the country’s ISPs would still be quite happily giving free access to all OTT players. Netflix is lobbying hard for ISPs to extend such free access to itself and gave us the following slightly ironic statement. ‘We’d like to thank Verizon for laying out the issue so nicely. Congestion at the interconnection point is controlled by ISPs like Verizon. When Verizon fails to upgrade those interconnections, consumers get a lousy experience despite paying for more than enough bandwidth to enjoy high-quality Netflix video. That’s why Netflix is calling for strong net neutrality that covers the interconnection needed for consumers to get the quality of Internet they pay for.’