A deep-dive into the relative performance of five encoding providers by the Streaming Learning Center (SLC) has uncovered a concerning gap between the major cloud providers and their upstart rivals. Brightcove and Bitmovin appear to knock lumps out of AWS Elemental and Microsoft Azure, and worse for the incumbent cloud platforms, even Tencent beats them.
This is the case for HEVC, at least, and the report’s author, the venerable Jan Ozer, notes that the trend of AWS being the most expensive option has continued. In the 2021 analysis, AWS was twice as pricey, and Ozer noted in 2022 that this has continued – likely due to AWS’ encoding technique producing the highest or second highest top rung bitrate, which will hike its own internal bandwidth and storage costs. This really does look like AWS blasting itself in the foot with both barrels, and passing that cost onto its customers.
Speaking to Ozer, it was clear that the legwork done to produce these tests was strenuous. What they seem to have unearthed is the chasm between the largest cloud platforms and the smaller vendors, which the end-users would struggle to evaluate in an operational workflow. So, while the tests are damning for AWS and Microsoft, it remains unclear if they care.
Ozer pointed to a previous project with Dolby and Hybrik, which illustrated the disparity in the costs of encoding services. In the tests, the Dolby-Hybrik approach was deployed on AWS infrastructure, and compared to a number of other services, including AWS MediaConvert.
The cost of running 1,000 hours a month through the AWS MediaConvert service was $252,600. The next closest rival was Azure, on $121,170, shortly followed by Bitmovin, on $92,383. The cost of the DIY-esque Dolby-Hybrik approach for those same thousand hours? A comical $14,260, illustrating that AWS apparently does not care at all – and worse, seems to be making a killing doing so.
Ozer concludes, “Sometimes when you work with established ‘leaders,’ there’s a presumption that while the pricing might be a touch premium, the output quality and feature set are market leading. This report challenges that, and companies spending more than a few thousand dollars a month on encoding with any of these services, particularly AWS, could really benefit from this sort of analysis. The report’s findings can be confirmed, at least partially, via the Moscow results and various pricing lists, but none of this should come as a surprise to AWS.”
The high-cost approach is also seen with Microsoft’s results. Microsoft has confirmed that it deliberately prioritizes the upper end of the encoding ladder, and spreads the layers or bitrates more uniformly in the high bitrate area. Again, this increases its bandwidth and storage costs, but interestingly, Microsoft was the cheapest option for H.264 and second cheapest for HEVC.
Here it looks like Microsoft is not passing those costs along to its customers, or is more computationally efficient behind the scenes. However, Ozer’s testing threw up some weird errors on Microsoft’s part, where its encoding ladder was overly conservative and would drop down to 360p from 1080p, resulting in some very poor-quality test results.
This HEVC report is a follow-up to an earlier AVC report. In both, the testing focused on how each service handled 23 video files, chosen from the entertainment, sports, animation, and office/presentation genres. The tests would simulate how these services would handle delivery in three scenarios – top-heavy (aiming for best quality), mobile (focused more at the middle of the encoding ladders), and IPTV (only looking at the top-two bitrates in the encoding ladder).
With 11 evaluation criteria, the HEVC report then weights and ranks the results – which see Tencent perform best in both the top-heavy and mobile tests, with Brightcove being the best for IPTV suitability. AWS is worst for mobile, and Microsoft is worst for top-heavy and IPTV. Bitmovin looks decent, with Brightcove performing a little better across the board, but Tencent is only 2% off the mark in the IPTV testing from having a complete clean sweep in the three categories.
While the performance differences between the platforms vary somewhat, the more striking difference is in the costs. The report notes that the cost to encode these files using AVC ranges from $3.09 to $9.05, implying a difference of around 2.9x.
For those same files using HEVC, the range jumps from $7.32 to $39.14, which is over a 5.3x difference. It is also notable that for all the time that has passed, the relative upper threshold of HEVC encoding costs compared to AVC is still 4.3x. Of course, HEVC requires more compute power, with the benefit being the lower transportation bandwidth needed, but given the rate of silicon improvements seen in the industry since HEVC’s 2013 launch, that 4.3x difference still feels high.
The report’s examination of the storage, encoding cost, and streaming costs turn up a surprise of sorts. Brightcove’s top-heavy scenario HEVC encoding costs were the lowest by far, at just $7.32. Microsoft was second, on $17.11, shortly followed by Bitmovin’s $18.59, and then Tencent’s $19.62. Here, the big question is how Brightcove is pulling that off, given that the other three results imply a relatively competitive market.
But perhaps the bigger question is how AWS is managing to score that aforementioned $39.14? Ignoring Brightcove, AWS is nigh-on double the cost of its most expensive rival, and while it performs better in the storage costs, with a much smaller gap between its rivals, it is also worst in the streaming costs.
The report is pretty damning for AWS, but we do not expect to get an answer to our enquiries for some explanation from the cloud titan. Of course, migrating a video workflow is a major headache, but there will be some video service leadership teams that will wince when they see these tables. AWS’s reach is so vast in the cloud segment that it may well be worth sticking with it, from a total cost of ownership (TCO) perspective, and while Tencent looks alluring, moving to a Chinese vendor in this geopolitical climate also sounds like a potential migraine.
The report uses VMAF for its quality testing criteria, explaining that it has the closest correlation with the subjective scores generated by humans. On this basis, a weighted average was used, and as Tencent had recently introduced a new per-title encoding technique that is closest to Netflix’s Convex Hull method, Tencent ended up performing best – in both H.264 and H.265. Notably, SLC’s results line up very closely with the Moscow State University results, which are held to be a gold standard in quality assessment.
At all the tested bitrates, for H.264 encoding, Tencent scored most VMAF points. Bitmovin was second, but was a considerable distance from Tencent. Brightcove is third, followed closely by Microsoft, with AWS a long way from the top. So, not only is AWS more expensive than its rivals, but it also appears to be the worst in quality.
The H.265 graphs are hidden behind the purhcase paywall, as this sort of work can’t be done for free, but the HEVC report teaser notes that the ordering is the same. Netflix’s Convex Hull is plotted in a graph, which shows the ideal mapping of vertical pixels (resolution) to the bandwidth availability in Adaptive Bitrate (ABR) delivery. Tencent and Bitmovin are much closer to the Netflix curve, meaning that they most closely follow the ABR ladders, and therefore should provide the optimal resolution at all times.
Brightcove is close to Netflix at lower data rates, but strays from the path as the bitrate increases. Here, it ends up very close to AWS’s ladder, with AWS a long way from Netflix’s ideal scenario all the way through.
Worse, because Microsoft has the most conservative ABR ladder, it ends up with some of the worst quality scores – as shown in this image (below) comparing the Azure 1 Mbps choice of 360p with the Netflix choice of 1080p, in a mobile-targeted encode for a soccer game.