Ultra HD masterclass part 3: Expanded vision for ultra HD

The Ultra HD Forum’s master class at IBC 2017 gave a comprehensive overview of key standards developments over the next five years under its phase B umbrella but left open questions over other areas of immersive TV.

It was not clear how or whether the Forum, which is responsible for infrastructure, or its sister group the UHD alliance dealing with content, production and CPE, would embrace Virtual Reality (VR), Augmented Reality (AR) and even holographic projection.

Yet VR and AR are now fast-growing industries set to generate $4 billion sales in devices alone by 2020 according to CCS Insight, (although until we seen an advanced light field version we’re not convinced), while the first convincing demonstrations of holographic displays could be seen at IBC 2017 from the likes of Silicon Valley startup Light Field Lab.

While VR and AR are still confined chiefly to gaming and not yet part of mainstream TV, they are likely to feature in trials and early deployments within the five-year time frame covered by the Forum’s recent survey of operator attitudes to Ultra HD.

Already Ultra HD has expanded its remit greatly to include HDR (High Dynamic Range), HFR (High Frame Rate) and NGA (Next Generation Audio) on top of the original 4K 2160 x 3840 resolution. Given that its role is to nurture and promote standards from other key bodies rather than develop them directly, both the Ultra HD Forum and UHD alliance should be looking at standards for VR and AR given that these intersect with their own domains. There is also 360-degree video, which is often mistakenly regarded as synonymous with VR when it is just an enabling technology. It is necessary but not sufficient on its own to generate a full VR experience, which should also track the viewer’s motions and position to allow real time interaction within the virtual environment, which 360-degree video on its own does not do.

Defining standards for capture, display and transmission of 360 video would be a good starting point, but at present proprietary standards are jostling for position, with Google even peddling a stripped-down version limited to a 180-degree field of view on the grounds that this is easier to produce while also allowing greater depth of field by not requiring the equirectangular projection widely promoted for full 360-degree video captured by cameras simultaneously in every direction. Equirectangular projection simply maps a curved surface onto a flat rectangle, introducing distortion, which reaches an extreme when extended say to the whole globe so that the polar regions are extended in width, making Greenland look bigger than the whole of South America.

When it comes to video associated with VR and AR in general as well as 360-degree projection, some of the familiar challenges associated with HD content are amplified. Quality losses and artifacts resulting from compression are magnified and are not readily addressed through existing standards. So new compression practices will be required and also stringent specifications over the measures for resolution, frame rate, bit rate and possibly audio formats.

Whether this will be addressed by a Phase C for Ultra HD to follow the Forum’s current Phase B remains to be seen. If this does happen, the Forum will probably embrace standards still at an early stage of development from the IEEE, covering AR and VR together since they have similar requirements for quality and interactivity. The difference is that AR overlays elements that are either computer generated or captured from the natural world onto the user’s surrounding world, while VR instead uses the apparatus such as head set combined with TV screen to simulate a real-world environment.

The IEEE has so far done little more than define nine areas for VR/AR standards, having just set up its VRAR Working Group. But it is much more ambitious and wide ranging in scope than the few other previously established AR/VR standards ventures, most notably the Khronos VR initiative largely confined to developing a common API called OpenXR for interoperability between devices and games.

The IEEE working group aims to go much further and cover all aspects of AR/VR devices, content, production and interoperability. It is starting with standards for video quality, user interfaces, and file formats, where again we would assume there is a lot of overlap with the two Ultra HD standards bodies. One of the nine areas is immersive video file and stream formats, while another is the user interface. It also covers interoperability between virtual objects and the real world, where the aim is to establish a framework for realism in the virtual world so that things look believable and again ultimately will boil down to specifications for the key components including video and sound, but potentially extending to the other senses.

Another of the nine areas is the immersive user interface, which may have its roots in gaming but where the IEEE is already looking to take account of not just TV but also the Internet of Things and especially future medical applications. There is obvious potential for helping with stroke rehabilitation for example by enabling users to interact via touch or even motions of the eye, as well as subtle movements of the body.

Another key aspect of VR/AR covered by one of the nine IEEE work areas is user identity, which has a host of applications. In a VR shopping application the user might “walk” down the aisle of a superstore and then end up paying at the checkout. With automatic identification by vision the user can avoid having to enter credit card details. So here the IEEE is designing standards, which could exploit 3D head images for users to identify themselves automatically to a site or service, as well as authenticate pieces of information, without needing to provide any additional details.

The IEEE does not yet cover holography, but probably will in future as this becomes part of VR and AR. Currently this is the preserve of a few specialist companies or laboratories, such as Light Field Lab, which has developed a small projector capable of displaying content as 3D objects hovering in space and the massively funded Magic Leap. This will require standards for capture, display and interactivity. Light Field is ahead of the game here too by working on an API which it says will assist in projecting existing content holographically.

From the perspective of the Ultra HD movement then the question is whether it will expand its remit further to include not just AR and VR but also aspects of immersive UIs including voice recognition, or whether on the other hand it will fold into a larger standards movement as all these elements of next generation TV come together.