Your browser is not supported. Please update it.

16 July 2020

Comcast rushed out an untested AI in lockdown network panic

It goes without saying that the pace in which the Covid-19 pandemic took hold did not allow for network operators to go capex crazy by laying fiber infrastructure like there was no tomorrow. Other means of coping with insane bandwidth traffic spikes were implemented, such as calling in specialist technology suppliers and rallying together to agree on widespread bitrate reductions across streaming platforms.

For Comcast though, there was another hero in the great network bandwidth crunch of 2020 – artificial intelligence. Details on what makes Comcast’s AI tick are typically vague, leaning our view towards the achievement sounding like a glorified network capacity forecasting tool above a layer of network automation features.

Nevertheless, the world’s largest cable operator has given itself a massive pat on the back, crediting its in-house developed Comcast Octave software for managing traffic spikes as high as 60% and adding 35 Tbps of peak capacity to regional networks during March and April, according to a recent interview with VentureBeat.

Panic must have enveloped Comcast’s Octave department as lockdowns were called into action, considering Octave had only been deployed to part of the network at the time. This warranted radical measures as a team of 25 Octave engineers worked 7-day weeks to accelerate the deployment process. It worked, according to Comcast execs, bringing a 36% capacity increase to subscribers.

Where the magic happens is a little indistinct, as we said, but we have come to accept such ambiguity as inherent with most software based on AI and machine learning.

Octave is built on top of Comcast’s Smart Network Platform, which automates network functions to reduce the frequency and duration of network outages – cutting outage times from hours to minutes. Here, in the Smart Network Platform, is our first clue as to what actually makes Comcast Octave tick. At the heart of this Smart Network Platform is a machine learning tool called NetIQ that makes thousands of measurements on the core network every hour to monitor service degradation. NetIQ detects core network issues in minutes rather than hours – reducing the average from 90 minutes down to 5 minutes.

Only by having a grasp on core network issues can Octave then do its thing – which is in both identifying when network modems are not using bandwidth efficiently, and in making this bandwidth available ahead of time. Comcast Octave monitors some 4,000 telemetry data points on 50 million network modems every 20 minutes, tweaking speed and capacity accordingly based on oodles of data points that only an AI-based platform could monitor, let alone act upon before an issue has even occurred.

Our initial stance was critical mainly because of the poor job people do at explaining these things, so credit where credit’s due to the data science used to understand trends and calculate correlations between events on the network. Comcast says it uses telemetry and automation to operate equipment without manual interference from engineers – mitigating issues before degradation is allowed to hit the network.

Of course, it helps that Comcast has its own CDN too, part of $12 billion in network investments made since 2017, the majority of which has gone towards 33,331 new route miles of fiber optic cable to allow for doubling capacity every 2.5 years, according to Jan Hofmeyr, chief network officer at the Comcast Technology Center in Philadelphia, speaking to VentureBeat.

It is typical for tier 1s like Comcast to have in-house CDNs, often supported by mainstream CDNs, giving them extra control of what is delivered and how it is delivered. Since lockdowns came in, we have seen more companies going all-out on cloud infrastructure and exploring the possibilities of the network edge, as well as techniques like P2P networking. Lockdowns have also highlighted how building a residential ISP network in a dense metropolitan area is a wildly different prospect from building a residential ISP network in a sparsely populated rural area. Some ISPs build their networks with a substantial amount of excess capacity, while others do not, with Comcast clearly falling into the former category – much to the envy of many smaller service providers.

Inevitably, what goes up, must come down – and this week evidence emerged for the first time showing that SVoD viewing has tailed off since lockdowns began easing. Netflix registered a 4% dip in viewing time, with Amazon Prime suffering a slump of 12% from the heights of April and May. It comes after the likes of Netflix, Amazon Prime Video, YouTube, Disney+, Apple TV+, Facebook and many more all committed to bitrate squeezes earlier this year to help ease what became a monolithic traffic load.

Certainly, more ISPs will wish they had the resources to weather the storm, and while many will be funneling capex into fiber, others will be looking for a more software-centric, AI-based approach like Octave.