The future of video codecs is as uncertain as ever with little consensus emerging among vendors or service providers over which ones will prevail, at least on the distribution side. The situation has not been helped by MPEG’s own decision to proceed with a successor to HEVC even while that codec is still beset by wrangling over licensing and intellectual property. This is tantamount to an admission that HEVC’s performance improvement over its H.264 predecessor has been underwhelming in the face of unprecedented demand for encoding efficiency driven by the advent of ultra HD and proliferation of OTT services.
This proposed HEVC successor called JEM (Joint Exploration Mode) arose from an evaluation team set up jointly by MPEG and the ITU’s Video Coding Experts Group (VCEG) on the basis of concern over the performance of HEVC. This MPEG-VCEG team concluded after conducting basic tests that there was potential to build a new standard that would significantly outperform HEVC. That evaluation included testing and an assessment of the nascent JEM compression technology, which indicated significant performance gains over HEVC.
This breaks with the tradition established with MPEG-2 that new codecs only arise once a decade and also that they are based on a radical advance in the underlying algorithms. Neither of these hold for JEM. HEVC did not begin to emerge until 2013 and yet JEM deployments are already being slated for 2020. Secondly and perhaps surprisingly JEM does not represent a radical advance on HEVC and in fact is architecturally very similar. The difference is that it is optimized much better to exploit the greater than anticipated advances in processing capability since HEVC was conceived.
JEM is also being designed to suit 5G applications, with the advent of ubiquitous wireless video sensors, as well as large scale streaming of UHD video content and in particular the requirements of High Dynamic Range (HDR) and Wide Colour Gamut (WCG). HDR and WCG again were barely on the radar screen during the conception of HEVC. In general JEM is being built to cater for emerging immersive TV formats, such as virtual/augmented reality, free viewpoint and light field.
Free viewpoint is the natural equivalent of virtual reality, casting the viewer as director with the ability to select the vantage point within a 3D scene. Light field is more of an underlying technology, involving use of multiple sensors in cameras to collect a greater depth of field so that images can be presented in more realistic perspective. Advanced codecs should take account of these to avoid losing the quality benefits upon compression.
While JEM was on the drawing board we also had the Alliance for Open Media’s (AOM) AV1 standard emerging, built on Google’s VP9 codec with addition of new coding tools from other open source royalty-free codecs such as Google’s VP10, Cisco’s Thor, and Mozilla/Xiph.org’s Daala. The primary motivation was to bypass all the licensing issues and costs associated with HEVC to produce a royalty free codec, although there have been threats of legal action from holders of HEVC patents.
However AV1 could never fly without also achieving some tangible performance or efficiency gain over HEVC and current evidence suggests that this will be about 20% to 30% at best. This is important because Google’s YouTube and Netflix are by far the two largest champions of AV1 but their support is qualified by the requirement for greater efficiency. Netflix has stated it would expect at least 20% while YouTube has not made any such formal statement and given its stake in the codec may allow more latitude. This is especially the case given Google’s concern over the high licensing costs of HEVC/H.265, with patents initially pooled by two separate bodies, HEVC Advance and MPEG LA. They were joined mid 2017 by a third body, Velos Media, offering HEVC-related patents owned by its members, including Ericsson, Panasonic, Qualcomm, Sharp and Sony. The licensing situation around HEVC is still as much of a dog’s breakfast as ever.
JEM though has put a spanner in AOM’s works by promising greater efficiency improvements. The ITU and MPEG consortium have published results suggesting improvements in the range 35% to 60% over HEVC, which is striking given the codec is still being defined. This is now being followed by other tests, notably from the EBU (European Broadcasting Union), which has commissioned the University of Applied Sciences Wiesbaden to perform a subjective evaluation of the quality and practicality of competing video codecs. These include AV1, HEVC and JEM, comparing speed of encoding/decoding as well as efficiency and quality. The stated aim is to help EBU Members choose the best codec for their individual needs, with results due to be published at the end of January 2018.
Earlier tests have highlighted JEM’s Achilles Heel, which is encoding time, which can in extreme cases be as much as 60 times higher than for HEVC on a given processor. Resources required for decoding are also greater by a factor of up to 10. But that is the whole point – JEM was designed to mop up processor resources in order to gain efficiency and so the real question is whether it can decode fast enough on today’s mobile chips and encode fast enough for say live streaming on affordable hardware. In any case these early implementations of JEM run in software, so performance will improve with hardware acceleration.
Certainly given all these developments with AV1 just coming available, it looks likely that deployment of HEVC will slow down. After all why go through the upheaval of deploying that when a royalty free alternative is coming, providing legal barriers are sidestepped, or when MPEG’s own anointed successor may be just two years away.