Your browser is not supported. Please update it.

6 December 2018

Versatile Video Coding already stymied by patent quagmire

With the codec soup we have now, it has become almost meaningless to talk about a next generation when there are so many overlapping timescales involved. The current situation is if there is a codec war it is squaring up into a straight fight between HEVC from the MPEG/ITU camp and AV1 from the powerful Alliance for Open Media (AOM) comprising tech giants and browser companies including Amazon, Apple, Facebook, Google, Intel, Microsoft and Netflix.

The situation on the ground tends to lag the perception promulgated by the media so that H.264/AVC still dominates current video codec usage, used by 92 percent of survey respondents, according to the recent Video Developer Report compiled by Austrian encoder technology firm Bitmovin. However that is now in decline and this survey based on the views of 456 video developers in 67 countries found that penetration of H.265/HEVC has grown to 42 percent, compared to 28 percent in 2017. AV1 has gained more groundswell in percentage terms, with 29 percent of respondents planning to use the codec in the next year, compared with 14 percent in the 2017 Bitmovin Video Developer Report.

Bitmovin argues these figures are not indicative of a war but more a trend towards a multi-codec world where all content providers select the best technology for different scenarios. It claims to have found AVC, HEVC, VP9 and AV1 co-existing in live deployments.

There is some truth in this but what we find is a clear dichotomy by industry sector. Broadcasters relatively struck for cash are the ones most committed to HEVC because that was the only realistic option on the table at the time they made their decisions. They could not afford to back more than one horse. Meanwhile many content owners and OTT service providers have sided with AV1, as witnessed by a glimpse at the latest list of AOM members which includes almost anyone of significance on that side. Of those only Apple of the big players really stands out by having a foot in both camps.

Taking in the bigger picture the MPEG/ITU camp have cause for concern, but this is not for technical reasons, but because of the deterioration in the licensing situation that coincided with the roll out of HEVC. This has always been a bugbear for the MPEG range, but was at least contained during the MPEG-2 and then H.264 eras, while with HEVC patents have fragmented into three pools, from MPEG-LA, HEVC Advance and Velos, presenting a confusing and risky picture for potential users. This has given oxygen to the AV1 movement far more than arguable and sometimes spurious claims of performance gains over HEVC. This can be seen by comparing the relative penetrations of HEVC v AV1 against their predecessors H.264 v VP9. The number of respondents in the Bitmovin survey planning to use HEVC over the next months has actually decreased from 40% in 2017 to 36% today, only 7% more than are planning to use AV1.

Yet H.264’s 92% compared with only 15% for VP9. So there has already been a huge swing away from HEVC towards AV1 compared with their predecessors. Although AV1 is not a straight descendant of VP9 they follow a clear lineage, promoted as alternatives to the MPEG/ITU camp.

At least this has roused MPEG to raise its game technically with its proposed sequel to HEVC called VVC (Versatile Video Coding). This has turned in some juicy performance figures given its early stage of development following announcement of the project in April 2018, comfortably outgunning both HEVC and AV1 in several benchmarks.

One of the most impressive and realistic benchmarks compiled by the Joint Collaborative Team on Video Coding (JCT-VC) team set up to develop VVC compared the immature version against HEVC at different bit rates. The objective of the exercise was to discover how much lower the bit rate could be turned down for VVC while maintaining the same perceived video quality, across all of the principle display formats including Standard Dynamic Range with 4K, and the principle flavors of HDR (High Dynamic Range). It turned out that VVC at 1 Mbps was equal to HEVC at 1.6 Mbps. On the other hand at 2.8 Mbps VVC was equal to HEVC at 4.6 Mbps. These are bit rate reductions of around 40%, which is comfortably within range of the 50% target even when a lot of the work refining the encoding and decoding procedures is still to be done. It is true AV1 slightly outperforms HEVC on some benchmarks, but only by a few percentage points at most, so VVC is well ahead of that too.

VVC is very much in the MPEG mold, taking some of the improvements already made in HEVC and running with them further. It has the same block structure comprising Coding Tree Units (CTUs), which equate to the macroblocks of earlier standards such as H.264 from which prediction blocks exploiting similarities between successive frames are devised. Under VVC the CTUs can be larger, up to 256×256, which is an important step forward given ever higher screen resolutions, comprising more and more pixels. Large CTUs give greater scope for exploiting redundancy within and between frames. The CTUs are divided into coding units (CUs) and one of the advances in VVC is that these can be arranged more flexibly and can be rectangular as well as square.

So the principal architectural features of VVC are in place and clearly working, as revealed by the various benchmarks. But more detailed execution and aspects of the codec syntax describing the portioning of images in detail has yet to be defined. For that reason VVC is certainly in good shape technically and may well exceed expectations when complete.

But all this will come to little unless the licensing situation is resolved, which is why some in the field have questioned whether it is worth investing money and effort in VVC until that is at least clearer. Even Harmonic’s Vice President of Solutions and Strategy Thierry Fautier, who on the whole supports the MPEG brigade, has talked about VVC suffering from “HEVC syndrome” with reference to the patent quagmire. The riposte might be that at least by ploughing ahead with VVC MPEG will be ready to hit the ground running if the swamp is ever drained.