Your browser is not supported. Please update it.

3 May 2018

From chapel to cloud, AWS Elemental claims first deep learning live event

What better platform than the Royal Wedding to show off the technological advances achieved when you combine the clout of AWS with Elemental? While many of our readers, and indeed us here at Faultline Online Reporter, may have minimal interest in the event itself, the viewing figures will be through the roof and any company involved in the delivery and processing of major live events must watch and scrutinize proceedings with eagle-eyes.

Elemental pioneered encoding in the live streaming space while AWS has cemented itself as a cloud and AI leader – together handling what is claiming to be the world’s first live online video user experience with deep learning, at Sky News. When showing this month’s wedding, Sky News will use AI technology to identify celebrity guests as they arrive and spin up information and facts about each attendee for viewers via the new Who’s Who Live app – available inside the existing Sky news app and on the company website.

That’s not all, there are two third parties involved for extra data punch and front-end expertise. Sky News is using the GrayMeta data analysis platform, combined with the Amazon Rekognition video and image analysis service for real-time identification of celebrity faces – enabling the information tagging feature. UK firm UI Centric has developed the front-end application and video player, which it says leverages the Sky machine learning workflow.

AWS Elemental gave Faultline Online Reporter a brief demonstration of its celebrity data archive system a year or so ago, at the time claiming machine learning capabilities, but without an explanation of the algorithms and learning processes in play to back this up.

Essentially this is facial recognition software connected to an image archive in the cloud, according to AWS Elemental CMO Keith Wymbs, speaking to Faultline Online Reporter this week. He explained the deep learning algorithms are applied to increasing accuracy, so as more images and videos are uploaded, the more accurate the results – with algorithms intensely studying faces. Customers upload all their own images, so in the case of the Royal Wedding, Sky News has uploaded the full guest list to AWS cloud infrastructure, for Amazon Rekognition to process and then, in essence, come back with metadata tied in with more metadata (tags tied with information), said Wymbs.

To avoid any potentially risky scenarios, for example a celebrity being tagged with raunchy information detailing their personal life, there is a human barrier to get past, which obviously creates a delay in the stream, but Wymbs assured us this only incurs a small delay. As for the information accompanying the celebrity tags, Wymbs could not confirm the source or in fact who decides which information is pulled, but he suspects GreyMeta handles this part of the process, almost – but not entirely – certain that the wholly unreliable and easily editable Wikipedia is the source.

Wymbs explained to us how the image recognition system was originally built for products, until it was actually the engineers, not the marketing team, who suggested applying the technology to video content.

We recently discussed how and where AWS Elemental will expand next, having reached such dizzying heights in the market by establishing a seemingly insurmountable lead in cloud services and encoding. This week’s announcement is certainly a statement of how deep learning algorithms can be applied to real-world TV use cases, perhaps laying the foundations for a more interactive TV experience in years to come. Building on what AWS and Sky News are doing, there could be scope for viewers to interact with a live event which uses tagging, for example dismissing tags which are of no interest, while requesting additional information for the tags which are of interest.

Such an experience would be most suited to live sports events, where there is an obvious viewer divide concerning information such as statistics or team news. Location-based personalization to assume a viewer’s allegiance would be inaccurate, so providing the option to choose which team’s data is prioritized would create a hyper-personalized viewing experience. Throwing Alexa or other voice-activated smart assistants into the mix would add an extra ingredient. From the pay TV operator perspective, we believe developing these types of TV experiences would sway subscribers from cutting the cord.

“The beauty of a company like AWS is it’s all about building blocks,” said Wymbs, agreeing that video will look much more interactive in the future. AWS is investigating applying tracking and tagging to high-brow events, not just for key moments but for more subtle instances, combining data with social interactions — so over time the viewer decides what they see.

Sky may already be investigating putting these deep learning algorithms to use in live sports, given AWS Elemental’s experience of capturing and analyzing baseball plays for Major League Baseball Advanced Media’s (MLBAM) Player Tracking System. In addition, David Gibbs, who is the Director of Sports Products and Digital News at Sky News said: “This new functionality allows Royal Wedding viewers greater insight into one of the biggest live events of the year, wherever they are. We’re excited by the software’s potential and ability to give audiences new ways of consuming content.”

But first, AWS Elemental has a Royal Ceremony to attend, for which ingest and video processing will be handled by an AWS Elemental Live small form factor, single-channel video encoder on the ground, before being beamed up to cloud-based AWS Elemental Media Services for processing of live and on-demand multiscreen content. The crucial security element is dealt with by AWS Elemental MediaPackage, integrating DRM over the Amazon CloudFront CDN, and MediaPackage is also tasked with processing live-to-VoD apps such as catch-up TV.

Sky is a long-term customer of Elemental, with Sky News boasting 23 million viewers across the UK and Europe. The Who’s Who app unfortunately won’t be available to US viewers, only in Europe.

AWS Elemental GM Alex Dunlap said: “The high visibility and unpredictable audience size for this type of event made AWS cloud services, including those for media, a great solution by giving Sky the ability to test quickly, only pay for what was used, and produce a reliable, high-quality experience in a matter of weeks. Sky News is making their content even more engaging to viewers by enriching content with AWS machine learning services.”

John Motz, CTO for GrayMeta said: “By harnessing the power of automated metadata creation, this experience exemplifies the future of how content will be viewed across platforms and technologies.”