Today’s post is the second in a three-part series on net neutrality, with the first part yesterday discussing the basics of internet routing, ISPs, and content distribution.

Below, part two will build on yesterday’s post and discuss how Netflix evolved their business model from a technical standpoint to overcome ISP throttling issues, and how ISPs maintain such a good negotiating position. 

---

Netflix Intervenes Directly

I’m talking about Netflix a lot in this series, and for good reason. Netflix has been largely praised as the reason net neutrality was brought to the table in the first place. Its business model has become contingent on high-speed internet being available to most consumers, moving away from the DVD delivery model used at its inception to high-definition streaming video direct to computers, televisions, cell phones, and game consoles. Here’s the rub: most high-speed internet providers straddle at least one other market. In the case of Comcast, it provides both cable television and cable internet. Netflix, which was beginning to gain consumers with its direct streaming service, began eating into Comcast’s cable television subscriber base. At the same time, more Comcast broadband users were switching to Netflix and “cutting the cord”, something that is still happening today. That additional bandwidth requirement put additional strain on the cable company’s resources.

In 2012, Netflix began transitioning away from external CDNs, whose costs had risen thanks to peering disputes such as the one between Level 3 and Comcast, to transit providers like Cogent, a Tier 1 provider with “settlement-free” (no cost to anyone) routing onto Comcast’s network. In theory, that should have been sufficient, as in years prior, Comcast and Cogent exchanged large amounts of traffic with no issue.

In reality, however, after only a short time of Netflix working with Cogent, Comcast stopped augmenting capacity on its interconnection with the Tier 1 provider. Cogent’s CEO was shocked, saying “for most of Cogent’s history with Comcast…[as] Comcast’s subscribers demanded more content from Cogent’s customers, Comcast would add capacity to the interconnection points with Cogent to handle that increased traffic.” This was no longer the case with Netflix on board.

To placate customers on Comcast who were being met with heavy throttling and service issues, Netflix purchased additional transit agreements with Level 3 and Tata, among others, to add to Netflix’s ability to communicate directly to Comcast. Cogent and Level 3’s routes both became saturated with data, and in 2013, the congestion was so significant that users were seeing increased performance problems for Netflix, which brought about a huge number of Netflix subscribers making angry phone calls to Comcast’s customer support, who directed them instead to Netflix as the source of the problem.

At this point, Netflix had purchased every possible transit route into Comcast’s network, yet quality continued to degrade for Comcast users. As a result, Netflix was forced into negotiations with Comcast, and signed a deal. The difference in video quality is striking. Here was the outcome:

In that same year, Verizon (by the way, they owned half of Redbox at the time) jumped on the same bandwagon and began to throttle traffic from Cogent’s peering link, also apparently targeting Netflix customers. At this time, Netflix has negotiated similar interconnection payment deals with Comcast, Time Warner, AT&T, and Verizon. These represent 1% of the ISPs that Netflix provides service to, but they encompass the dominant market share of Netflix customers.

I would be remiss if I missed noting the update of this that was revealed in late 2014 by data collected by M-Labs, in which we see that Cogent implemented a QoS structure in February and March of 2014 to alleviate congestion on its network by sorting traffic into “retail” and “wholesale” positions. This has been cited as at least part of the “actual” reason for the increase in performance seen in the image above, but given that Netflix was also peered with other transit providers who have experienced similar frustrations with congestion to ISPs, I personally remain unconvinced that this is a significant contributor to the situation’s resolution.

Comcast’s Market Position

Comcast’s actions betray the way tiered network service providers are supposed to work. Rather than paying for links to transit providers to support the increasing needs of its own paying customers, Comcast instead used its advantageous market position with customer coverage to force content providers and hosting companies to pay Comcast for the privilege of serving their customers. As a result of these successes in battling both Level 3 and Netflix directly, regulation-free peering has been raised as a demand of other ISPs, likely in an attempt to set more precedent for paid peering agreements.

So then why can Comcast get away with this? It has a ton of users. 56.8% of the US is on Comcast, in fact. When more than half of consumers in the United States share an ISP, they can heavily dictate how providers like Netflix work, and can be directly responsible for the success or failure of their product depending on their intervention. And that’s actually one problem the net neutrality rules do fail to change, for the worse.

In 2013, SoftBank reported that 28% of American households had at most one provider of broadband, and 37% have only two. Chances are, when you go to buy a car you can think of more than two automobile manufacturers to choose from. So why can over half of the households in the US solve their question of which ISP to use with a coin-flip?

Many municipalities have, over the years as fiber networks have grown and the Internet has been more widely adopted, struck agreements with large companies like Time Warner or Comcast that either turn over a city-built fiber network to the provider for exclusive use, or that prevent the municipality from turning on the fiber at all. For example, I went to college in a city where a large incumbent ISP paid the city a huge sum of money every year to simply keep things disconnected from the massive fiber network that was built under the city. It sat, unused, in an area that could have strongly benefited from improved network speeds and access.

However, if the ISP is building the network instead of inheriting one, they could pay egregious sums of money for rights-of-way, a term used to describe an easement granted for the construction of utilities or transportation. They also need to work with local utility companies to rent space on poles or in underground conduits to attach wires. These fees, which have charitably been seen as ‘overvalued’, make starting a smaller, competing ISP virtually impossible, and rights to the fiber itself are generally reserved for a single ISP due to capacity issues. Further, if a new ISP is entering an area controlled by a large incumbent ISP, they could face potentially baseless legal claims in an effort to tie up the new company’s resources in court injunctions and expensive legal proceedings.

While municipalities are concerned about losing revenue from these lucrative deals, they forget that those costs are ultimately borne by their taxpayers anyways, when the bill for the broadband service comes around. This year, however, the FCC may be forced to take a look at this problem, as municipal broadband is on the rise in some cities and we are starting to see it run up against the legislation designed to impose these monopolies. In the meantime, it leaves most consumers with only a couple of options in the broadband marketplace, making the idea of “free market” and “open competition” a fallacy among ISPs.

Netflix Strikes Back with “Open Connect”

After settling with Comcast, Netflix took another evolutionary step in its business model and created its own content delivery network in an attempt to get away from paying CDN fees to companies like Level 3 and Akamai. This idea involved placing multiple appliances in different service providers’ racks to localize the large cache of Netflix data to a specific, smaller region, in order to incur less overhead routing traffic from Netflix’s main servers to the end user. To incentivize ISPs to switch to utilizing this Netflix-exclusive CDN instead of continuing to collect fees via intermediary CDNs, they restricted 3D movies and “Super HD” content to its proprietary CDN only. Time Warner took some issue with this, claiming it discriminated against Time Warner customers, as TWC did not opt-in to be a participant with Netflix’s “Open Connect” CDN.

Backing up a moment, this claim would suggest that Time Warner’s preferable alternative would be for Netflix to pay service fees to (let’s say) Akamai, who then pays service fees to Time Warner, rather than Netflix simply handing off data to Time Warner via its already widely-peered CDN closer to Time Warner’s users. Using this strategy would mean that Time Warner incurs no additional load across its main network, as all Netflix’s data would be entering at nodes very close to a user. (For example, a user in Brooklyn might have their Netflix Open Connect appliance in Manhattan.) This means Time Warner’s incremental cost for routing the data (that is, why CDNs more traditionally pay ISPs) would be extremely low.

Another option provided was to allow Time Warner itself to place Netflix Open Connect appliances for even cheaper delivery to its users, by skipping the need to peer with an Open Connect-participating service provider. The cost required for this would be the leasing cost of rack space plus power and cooling, which was still likely to be a cheaper, better performing solution than routing over a national network. The appliance syncs with Netflix’s main servers daily, moving popular and commonly accessed content to the appliance.

As a side note, the Open Connect CDN is completely free to any participating ISP and is maintained by Netflix’s own support engineers. All of these strategies are designed explicitly to reduce congestion on these ISPs’ networks. After about a year, however, Netflix stopped the policy of withholding higher-definition content from ISPs as negotiations with large ISPs like Time Warner continued to fail, and they finally agreed to pay the ISP interconnection fees due to, again, increasing congestion and poor quality for Netflix customers.

---

Please check back tomorrow for the link to the final part of my series on net neutrality — or click the button at the bottom of the page to follow my blog, and receive new articles instantly via email! 

Tomorrow’s post will discuss some of the major questions people have about net neutrality, and why concerns about it are frequently misplaced.

(Header image via Creative Commons 3.0 license, courtesy Electronic Frontier Foundation)

Just as a reminder, all views expressed in this blog are mine only, and do not necessarily reflect the views of Microsoft or its vendors or customers.