There is no way that some of the largest technology businesses in the world were going to allow Apple (and by association, Samsung) to control the entire video world, simply by controlling a codec patent pool, based on patents which it didn’t even invent – just because more copies will run on those brands phones than on anything else.
Which is the political explanation we are happiest with around the current battle of the codecs – and this week the Microsoft, Google, Cisco, Mozilla, Intel, Netflix, Amazon phalanx that makes up AOMedia, has taken the next step along the road to preventing that kind of power being usurped by Apple. It is only a collection of these powerful, but for the most part pre-smartphone businesses (with the obvious exception of Google) which could be strong enough to stand up against the forces of MPEG LA and its HEVC patent pool and the rival HEVC Advance pooling.
AOMedia’s next step is to annex the hardware power which drives these devices – not content to initially land Intel, which could be seen as potentially out of step with the low MIPS per Watt thinking which drives the smartphone business, and it has added ARM, the core designer which has designs in well over 2 billion phones, and virtually every smartphone, as well as AMD and Nvidia, considerable chip vendors in their own rights.
The only major chip makers now not at the AOMedia party are Samsung, Broadcom, Qualcomm and Apple – and every one of these relies on core designs from ARM, with the possible exception of Broadcom, which also has many MIPS designs (ads well as an ARM license) among its millions and millions of set tops. Recently Broadcom has begun to take more and more ARM designs into its video SoCs.
If you were Apple weighing up a decision about how to claim at least partial ownership, through the MPEG LA patent pool, to HEVC, the supposed next in line to the codec throne, you would have to consider how you could do that while ignoring the close association between these chip designers and specifically ARM, and AOMedia, representing the cluster of majors which are insisting that the days of codecs being first designed, and then later delivered on expensive, unworkable terms of business, are definitively over.
AOMedia has now issued its first cut of the raw code base, but only so that more and more 100s of codec designers can get behind it, hoping to be ready with a definitive bitstream specification by year’s end, so that 2017 can be the last year that anyone has to rely on HEVC. It has gone out open source as of this week, and simply takes the best bits out of VC1, VP9, Cisco’s Project Thor, Mozilla’s Daala, and plugged them together. So far, by the admission of Gabe Frost, a long term video man at Microsoft, who is also the Executive Director, at the Alliance for Open Media, it’s only about as good as HEVC, and he hasn’t really even bothered to measure it yet, because that’s not the point.
That’s because the code is about 9 months away from being ready. According to the original timeline issued by AOMedia, there was to be no codec for 18 months, and we pointed out when we chatted to Frost this week, that he seems to be about 3 months ahead of that curve. He said, “That’s because we are working with the hardware partners where this Bitstream specification will run, and they are helping us, and we have had to get in-step with their product cycles. Normally with codecs, you write a version in pure software and then the hardware adaptation takes place subsequently, but by partnering with the key hardware players we can do both at the same time, and go straight to hardware implementations.”
“But you mustn’t think this is the finished article. There are lots of tools to add, things like de-blocking filters and ring filters, that will be in the final version and we need the open source community to help us work on those.”
The copyright will operate under a BSD license, and the patents will be administered by W3C, creating a completely “free” codec that is updateable in the same way that an operating system can download updates, throughout its life.
So how will real time encoding firms take to the new codecs, having just put a ton of effort into distinguishing themselves through HEVC? “You must remember we are writing both a decoder AND an encoder, and there are many intellectual property opportunities in encoding, it is only the decoder that becomes a static feature which has to be compliant with the bitstream definition.”
Frost will not go so far as saying that companies like Harmonic, Elemental and Ericsson (Envivio) will do what their clients want, and having no reasons to pay for a royalty license is going to push major operators and broadcasters unerringly to the doors of AOMedia.
By our calculation that means that chips with appropriate hardware acceleration will be with us first quarter of 2017, and the entire eco-system will then begin a switch towards it, from HEVC.
We can see that this announcement does not signify the end of the warring factions in codec design, but the first weapon of choice is to undermine your opposition with future promises and right now when MPEG LA or HEVC Advance come knocking at your door requesting you sign a long term patent license, dissembling for a year or so, could mean you can pay nothing and then switch to a royalty free deal, or at least only commit to a short term license.
This is in direct contravention to the flight to 4K that we are witnessing in the market where set top boxes are coming out with 4K running over HEVC – and at least some of these must be signing deals with one authority or another. The last move in these patent wars was for MediaTek to join the HEVC Advance faction, backed by Technicolor, Dolby, Philips, GE and Mitsubishi.
While MediaTek’s addition is not enough to swing the vote in one direction or another, MediaTek owns MStar which has around a 70% market share of smart TV chips globally, so it is clearly influential.
We conclude that the Broadcom chips that drive 4K set tops and the MediaTek chips that drive smart TVs, will have to be separately targeted by AOMedia, retrospectively – it will have to work with them, in the background, so as not to undermine their existing HEVC efforts – to get the AOMedia codec base to work acceptably well on those chips. This will be a tougher effort, because it might harm those companies’ sales if this was done in public,
HEVC was designed, like all previously successful codecs, by committee, operating in a purely conceptual fashion, but with a full knowledge of how the current generation of chips work.
The MPEG LA by contrast has 32 members but seems to be dominated by the two major smartphone camps Apple and Samsung, as well as Asian broadcasters and operators (NHK of Japan, NTT DoCoMo and Korea Telecom) and so can boast the majority of devices out there and the largest payees into royalty streams.
With this move AOMedia has managed to claim the high ground on smartphones with the ARM core, and if later it can show a working design with Broadcom and MediaTek chips under it, that will be an effective slam dunk for this grouping.
The honest truth is that since the original MPEG committee sat in the 1980s, people have been trying to “own” the codec space and the AOMedia stance to make it royalty free, should do away with 35 years of arguing.
Microsoft was originally asked to help on MPEG to create a fully software version of one of the original specifications, and then it came up with VC1 and tried to license it – to its own consternation it turned out than an MPE LA patent pool awarded Microsoft very few of the original patents, instead agreeing that companies like Technicolor, Sony and Philips owned the bulk of the ideas behind it. Google was next with VP9, trying to re-write history by making the On2 Technologies MPEG copy look like it was all original work, and trying to get people to sign its own license. For its own part Cisco has acquired teleconferencing business that have dealt in codecs since the bad old H.263 days.
Each of them have since made significant contributions, admitted in the attempt to gain a lead and land a huge royalty check. But you can see why these companies see Apple as an interloper, unworthy or being in a patent pools, which in essence will use theirs, and others’ inventions. Apple is not an inventor of core technologies, but a shrewd user of technologies – it’s not the same thing.
This has united Microsoft and Google and led to them burying the hatchet over software patents. And a lot of money has changed hands to get this far. In 2011 a consortium of Apple, EMC, Ericsson, Microsoft, RIM and Sony collectively outbid Google for Nortel’s patent hoard, which made it ramp its efforts and land the Motorola patent hoard with a $12.5 billion bid to acquire it, keeping the bulk of its 17,500 patents so that it could disrupt markets with impunity.
All this history makes it impossible for Apple to successfully challenge the AOMedia codec patents in court. The rival grouping HEVC Advance may well try to put a legal challenge together, but some of the original MPEG patents will by now be exhausted, and HEVC Advance can hardly offer complete protection against AOMedia held patent suits, without this cluster of companies.
To Faultline this means that HEVC will continue assaulting the 4K mountain of business for another year or so, before the edifice that is HEVC comes tumbling down.
Frost talks constantly about the “Uncertainty of terms” as the major reason for AOMedia stepping in, and says that “right now recruiting new members is not our problem” implying there is a queue to join AOMedia, but will not be drawn on how many 1,000s of codec writers the creation of an open source will trigger.
“Many broadcasters are keen,” he insists and we believe him, and there will be “big improvements” in performance over the coming 6 months, and of course he reminds us that some AOMedia members in Netflix and Amazon, are among the biggest users of content, who must have been appalled at the initial HEVC Advance terms, subsequently dropped, to charge content creators and distributors royalties in what would have been the biggest smash and grab in video history.
“Gaming services like Twitch have delivered more GB of video data, than all TV ever delivered over the internet,” and we will be adding things like a Media TimeLine controller into our video systems, so that two or more video streams can be viewed simultaneously over the internet. Sony with its Playstation franchise will be sure to want to harness that kind of capability, so perhaps will be less inclined to brandish its own codec patents in opposition.