Cloud has been on everyone’s lips since the pandemic hit. A large chunk of the worthwhile events at virtual IBC were on the topic, with plenty of discussions on cloud-native services, multi-CDN, edge compute and remote production tools.
It was therefore fitting that this week, Haivision hosted a webinar to flaunt an impressive case study of its cloud-centric gear in action. Those who tuned in were treated to an in-depth look at the blueprints behind Fox Sports’ coverage of NASCAR and Major League Baseball (MLB) amid unprecedented working conditions at the height of the pandemic.
The crowning achievement? Half-second latency for remote production teams on the other side of the country.
Fox Sports has now named the resulting setup Production Anywhere and has since transferred it across its portfolio. Brad Cheney, VP of Field Operations and Engineering explained that it was an amalgamation of previous cloud efforts – “we took pieces and parts of things we’ve done in the past and linked it all together.”
Production Anywhere is the latest product of a common chain of events we have observed in the past few months. Live production workforces become stranded by lockdowns, forcing cloud migration initiatives off the backburner.
Initially launched for NASCAR, Production Anywhere had to connect four sets of remote workforces. On the East Coast, a mobile unit stationed at the NASCAR Darlington Raceway held directors, camera and video operators and the audio team, while on-air commentators and producers were working from the Fox Sports Network Centre in Charlotte, NC.
All other supporting roles for delivering the races live was either delivered from the LA Broadcast Centre, over 2500 miles away, where replay operators, graphics operators, producers and assistant directors would hone the video content, or from remote home workers – of which Cheney was one. These included audio teams, broadcast engineers, producers and executives.
Cheney explained how, in the weeks leading up to the race, Fox tested the capacity of its main link between Darlington and Charlotte. Increasing amounts of video were sent over a private fiber line, as ever more Haivision encoder hardware pads were added to Fox’s pre-existing Makito X4 and JPEG infrastructure to meet the demand.
A private WAN was set up to link together three stationary units straddling both sides of the US, with each location holding direct connections to the other two in order to safeguard against any outages.
An SRT H.264 feed was transported from Darlington at 8 Mbps to Haivision Makito X4 decoders in LA and Charlotte, while a secondary 2 Mbps SRT stream was sent to a Haivision SRT Gateway in LA from Charlotte and Darlington.
Cheney told us how the workers in LA were viewing content within half a second of the event, while remote home workers received their feed in under a second. “The goal was collaboration everywhere,” he explained.
Staff in LA were able to remotely control operations at the Darlington mobile unit via a PCoIP connection.
It sounds like Fox has managed to keep its books down, or at least close to, pre-pandemic levels of spending. Cheney accepted that while lack of staff on site certainly cut some production costs, “there’s an increased cost in IP connectivity, so the cost nets out. We can reduce costs properly when we use it for multiple events, but it’s a fine line.”
Aside from the technological feats, NASCAR was a success in terms of viewership. One of the first calendar sporting events since the pandemic hit, it had 6.3 million viewers, up 38% year on year.
Cheney told us that the initiative was such a success that it quickly permeated all of Fox Sport’s live event coverage, including Fox Baseball, Fox NFL, Fox College Football, Fox PBC, and Fox Soccer.
The system for MLB ran on similar hardware and network infrastructure. Each stadium had a full mobile studio extension, dubbed Home Production Kits. These fed into AVP2000 contribution encoders and MK RXI cloud video processing software, as well as the Haivision hardware.
Production for all games – such as replay and graphics – was carried out from the LA Broadcast Centre, with this feed then sent to remote analysts and commentators in studios across the country, as none were allowed on site.
Analysts were able to view the game with under half a second of latency and collaborate with others in completely different locations. “Not a single announcer or analyst was sat next to someone they were collaborating with,” Cheney explained.
As for Fox’s vendor choice? “We knew Haivision was the only way we were going to move that much video,” Cheney said.
Cheney seemed adamant that this new workflow was not going anywhere. “We don’t see this stopping. We’re using the best workflows to protect our workforce,” he said.
Pandemic aside, Cheney also noted that the cloud has relieved production staff from their often-brutal work schedules when they are shuttling between onsite locations – “it’s often a labor of love.” Safe to say this was a great example of the sheer possibilities of cloud migration, whether the stimuli are technological or social.