At last some sense came in this week from the Virtual Reality and Mixed Reality marketplace, in the form of a white paper from InterDigital which proposed a fairly radical new way of encoding and supplying 360 degree video to devices.
At no stage did the report from InterDigital’s Innovation Partners user the term 360 video, but instead made it abundantly clear that it would use the right terms for VR experiences. It referred to this as 3DoF+ or Three Degrees of Freedom Plus.
In essence it proposed two things – the first are two very similar ways of breaking up 3D video into concentric layers or spheres, where all the obejcts are the same distance from the viewer and treating each layer in the same way as regards motion parallax and occluded content. In other words each sphere of video would shift the same amount as a viewer’s head moved, with an object in a closer sphere moving further and more rapidly relative to us, and with sphere’s further away moving more slowly. This is to provide visual clues for depth processing for the brain. In reality each object has its own distance from your eyes, and as you move – even slightly, the change in its placement creates depth clues which your brain knows how to decipher. This is a way of fooling the brain into doing much the same, but without anything like so much data having to be sent to a VR headset.
In brief, for the uninitiated, 6 Degrees of Freedom (6DoF) is the ability to see along 3 axes from your viewpoint, but also to move along each of these axes, so that your view is changed dramatically. This is seen as something that would be great for full VR or MR, but not obtainable right now.
We have said on Faultline Online Reporter before that a codec would have to encode about a screen of data for what your eyes alight on in the center of their view, and another 8 further screens worth of data for the spaces above and below and to the sides for your peripheral vision, making even stationary VR require 9 times as much data as normal video.
Actually it is worse than this, quite a lot worse. You also need to know what each 3D object looks like from every side, and top and bottom, if you have 6 Degrees of Freedom. But this is only to view from a single location, even then this is not a 3D viewing scheme, so that it does not look like a flat version of reality.
If you walked along, objects at different distances should move at different speeds relative to you, and objects that were behind something (occluded) should suddenly appear. Also light and shading would change dynamically as you move. So all that data needs to be encoded and delivered and available and work with a system which tells the video sources your movements accurately (tracking). A full 6DoF VR might amount to terabits per second.
What InterDigital is proposing is that existing VR which offers only a single viewing point, where you can rotate, but not move, is the best thing to aim for, for now. We call this 360 Video, although it is correctly called 3 Degrees of Freedom. But we can do better than this and allow the head to not only move around slightly, but also shift just a little left and right – getting slightly closer and further to the edges of each layer of video or each sphere – as if you were fixed in a seat at a cinema. This it calls 3DoF+ where the Plus represents some slight improvement which can make the viewing more realistic with this small amount of movement.
The two slightly differing techniques were to break each distance from you into a sphere of viewing and send that data as one part of a view, and the other was to send both this plus 3D objects. It is a lot less data than 6DoF. One of the techniques shares the workload between the central server and the end device.
But then it addressed its second innovation which helps on the practical side of getting content to devices. A machine learning program can calculate how far away everything is in an existing 2D movie, using perspective, shading, light and parallax effects, as well noting what happens as the camera pans. And it can then create these different distances spheres dynamically from 2D movie material, and project them to VR headsets. So old TV content can be made into VR content with a once over from a trained AI instance.
Actually the final step that InterDigital describes is massively significant and equates to the creation of an Adaptive Bit Rate delivery of VR experiences, in 3DoF+. However the content is broken into these spheres, the number of spheres that are created (the more spheres, the more photorealistic the outcome) is down to how much bandwidth and processing power is at each end. Let’s call this Adaptive Bitrate VR or ABVR.
If your connection is poor, you get fewer spheres of lower video density, if it is good you get the maximum number of spheres or video layers, at higher density.
InterDigital seems to agree with us, that trying to recreate VR experiences through what are effectively phones strapped to your head, will only create a 2D effect anyway and that the shift towards light fields, which fool our eyes into observing different focal lengths, is the way forward (think Hololens and Magic Leap and some Google experiments). But still, cutting down the amount of content that has to be sent is vital for the real world.
InterDigital holds some key standards in cellular relating to signal strength being determined to each phone, and when InterDigital is involved in a new technology expect the end point to be a patent or two and a filing for a standard at the 3GPP at the end of it. This seems to be generic new method for bringing new 360 degree video which is fundamentally more convincing than what’s out there right now.
Finally it is worth noting that the true definition of mixed reality is to see the world as it is, but with virtual elements overlaid on it. This requires a lot less data than full reality, but it will stop phone users walking into busy roads when playing a fully immersive game (you can still see the road). However such virtual objects inside the real world still need shading and parallax calculations in real time and any process that reduced the stress on the network by sending less data, will accelerate the arrival of fully immersive experiences even when most of what you are seeing is real.