Apple’s move last month to join the Alliance for Open Media and rumors that Samsung may follow have thrown the codec world into turmoil, prompting MPEG’s founder and chairman Leonardo Chiariglione into writing a blog urging desperate action to repair the broken codec licensing model before it collapses completely.
This model enshrined by MPEG has served the digital media industry for 30 years, encouraging participating companies to invest in and donate Intellectual Property (IP) with high expectations of making a healthy profit. This has spawned a succession of standards, including MPEG-2 for digital TV in 1994, followed by AVC for reduced bitrate video in 2003 and finally in 2013 MPEG-H Part 2 or H.265, which has become better known as HEVC, as well as DASH for Internet streaming at the same time. This model worked well enough right up until HEVC was on the drawing board, with technology licensors pooling their IP so that it was relatively easy to acquire and at the time just about affordable for service providers and device makers. But then the process broke down as fundamental disagreements erupted between license holders over strategy and the underlying valuation of the IP.
This led to the licensing process fragmenting into three pools around HEVC, MPEG LA, HEVC Advance and Jelos Media. This has confused the industry and complicated the process of acquiring licenses to deploy HEVC that will guarantee immunity from prosecution. As a result, while it has some traction among the traditional video sector it has had little take up on the web. In fact, as Chiariglione has admitted, the fractured licensing model created the void for the Alliance for Open Media (AOM) to occupy from the web side. Founded by Amazon, ARM, Cisco, Facebook, Google, IBM, Intel Corporation, Microsoft, Mozilla, Netflix, and Nvidia, AOM set about creating a royalty free codec, while Chiariglione has attempted to counter this creating an alternative to the traditional MPEG licensing model resulting in lower prices. This has largely failed and he has since accused some licensors of being blind to the reality of the new challenge from the royalty free movement.
The reality is that both existing users of the last generation AVC and many emerging players on the OTT and mobile fronts have been deterred from considering HEVC by the greed and overzealous pursuit of patent infringements by some licensors. With classic Italian melodrama, Chiariglione has described this situation as tragic not just for MPEG but for the whole video industry. It is tragic he argued for MPEG because collective investments running into hundreds of millions of dollars will go up in smoke. For the whole industry it will block access to innovation as contagion from the royalty free model spreads to other segments and stifles investment in new technologies.
Chiariglione is clearly mourning over what he could be the demise of MPEG which he started, he can take comfort from the fact that the royalty free movement have has had to fork out for some of the technology. Google had learnt long before AOM was formed in September 2015 that there was a price to pay for its royalty free approach. Google had decided early on it did not want to be beholden to MPEG licensors and so set about developing its own codecs, as Microsoft had done earlier with its VC-1 around 2003. This led to Google buying On2 Technologies in February 2010 for an estimated $124.6 million, acquiring a company that had specialized in codecs since its foundation in 1992. On2’s VP8 technology became the core of Google’s WebM format, but it stacked up badly against H.264 in tests, as well as turning out to be very similar in several respects. This led Apple’s late founder and CEO Steve Jobs to comment that the codec was both underwhelming technically and exposed customers to risk of litigation for IP infringement. Such litigation never happened but sources close to the process revealed that Google had to pay substantial amounts to secure additional IP sufficient both to remove any risk of infringement and ensure that it was well placed to become competitive with MPEG and the emerging HEVC codec. After all even a royalty free codec would be unlikely to gain much ground if it was demonstrably inferior to the latest codec form the MPEG stable.
While this was not achieved with VP9, Google’s royalty free successor to VP8, even Chiariglione admitted in his recent blog that the AOM’s first joint codec due for release soon was likely to outperform HEVC. But again it looks like this has required purchase of some IP from HEVC patent holders.
Chiariglione’s most contentious point was his suggestion that the prospect of royalty free codecs undermines the whole video industry. It is true that the likes of Fraunhofer, Dolby and Technicolor would not have contributed so much directly to codec IP without royalty payments, but there are other ways of recouping technology investments within an open source community. Technology has to be seen more as a commodity or enabler for revenues that ultimately achieves payback in consultancy fees or data. The latter of course lies at the heart of the codec battle and is why there is some disharmony within AOL as well as MPEG and the HEVC patent groups. It comes down largely to Google which stands to gain from commoditizing codec technology just as it has from Android, for data is a major part of its business. That is why other major Internet technology players are determined to prevent Google having things all its own way in the codec sphere, even if it is royalty free, and partly explains why Apple joined AOL in January 2018. This was greeted as a U-turn by Apple which had previously appeared to have pledged its allegiance to HEVC, but if so the turn had been visible for some time. It was clear that Apple, like Google earlier, had become fed up being exposed as a net payer to what had become an increasingly fractured licensing scene around HEVC. It wants to gain access to key patents itself and has according to sources within that world been scratching around lately trying to buy some. At the same time joining AOL gives it access to the rival royalty free codec world with scope for influencing the course of their development. In any case Apple is moving increasingly into OTT video and needs to support Apple TV alongside iPad and iPhone and so will need multiple codecs, so for both strategic and tactical reasons is likely to remain a member of MPEG LA.
This raises the question of whether Samsung will follow suit and join AOM, given that it is currently a member of HEVC advance, the other major HEVC patent pool group. Samsung owns plenty of patents and so has no need to join AOL for access to technology but could be interested in getting on board to influence developments around AV2, the successor to AV1 currently in early stages of development. It may be that Samsung will join as AV2 development gathers steam.
This hinges on another point not always appreciated, which is that one reason for HEVC’s failure to gain traction is lack of enthusiasm over its technical performance. Although it has succeeded in halving the bit rate for a given video quality compared with its predecessor, the arrival of ultra HD coupled with proliferation of OTT services over bandwidth-constrained networks calls for more. That is why the MPEG group itself has pushed ahead with a successor called Joint Exploration Model (JEM), which is closely related to HEVC but achieves around a 60% improvement in efficiency through various techniques, including intelligence in the decoder to exploit optimizations in successive frames, which imposes a processing ahead in clients that will require tuning at the hardware level. It so happens that relevant South Korean technology players including Samsung are putting their weight behind JEM, but it is still likely Samsung will hedge its bets and spread its wings by coming into AOL.
So while AOL and its royalty free model may be favorite to succeed, with Korean support for JEM, Chiariglione reckons there is still hope for saving the MPEG licensing model but only if it is totally overhauled. His main point is that the whole MPEG standard development process should be streamlined to separate coding tools into clear pools of ownership without defining set profiles and with the ability to switch them on and off. Then customers would be able to select profiles suitable for them on the basis of available technologies with easily identifiable owners. They would also be able to switch on tools once when they become usable, for example because the relevant owner has just joined a patent pool.
The fact such measures were not taken well before HEVC was released makes us wonder why harmony should suddenly be expected to break out now. Perhaps the specter of Google is haunting MPEG licensees so much that they will bury their differences and unite against a common enemy.