How Suitest solved a bittersweet symphony of QA app testing

Despite our frequent analyses on the state of the video analytics market and criticism of its over-saturation, the technology has traveled a long way this year. It was therefore only fitting we should sign off our final issue of 2018 by revisiting a start-up in the space, although not in our typical realm of QoE. Speaking to Czech quality assurance (QA) vendor Suitest this week, we discussed possible trends for 2019, potential breakthroughs at US movie studios, and the difficulties in transitioning to the video quality testing market.

In short, Suitest came out of TV app development and shifted to QA testing for OTT video and HbbTV applications on smart TVs, set tops and media streaming devices. After a number of technical failures, the company proved there was a way to do object-based test automation, a technique which comes as standard on desktop browsers through open source tools but has been amiss from the TV app ecosystem. The result is automated testing integrated into a DevOps process – where software updates and QA testing run harmoniously together in the cloud.

“We have an OTT video customer in Asia who runs automation tests four times a day using Suitest, so as soon as a new software update arrives it can pick up any flaws straight away,” said Chief Strategy Officer Mirko Nedeljkovic.

It took us a while, but eventually there was a lightbulb moment, that the real beauty of Suitest is its affordability. Its tools are even made affordable for individual developers, scaling right up to something like South East Asian SVoD firm iflix which has north of 15 million subscribers in a market where streaming devices are incredibly varied (so probably the customer he just mentioned). Its big claim is the ability to test hundreds of devices and apps in parallel.

“All of our competitors are doing image-based testing, using HDMI encoders which can cost hundreds of dollars each, as well as cameras, lots of data processing and other expensive hardware, particularly when smart TVs are involved. We are the first to go software only,” said Nedeljkovic. Cost savings for customers are potentially huge, he assured, although later admitted Suitest does have some hardware, in the form of its Candybox product, a web-based programmable universal remote control that hooks up to TVs and set tops.

We are probably not alone in the mindset of associating QA video testing with one company in particular, Eurofins Digital Testing. According to Nedeljkovic, Eurofins attempted to build a similar object-based testing system to Suitest but gave up the ghost after learning about Suitest’s US and European patent applications about three and a half years ago. The applications are still being processed, but the company is hopeful of an early 2019 grant.

Faultline Online Reporter first met Suitest at IBC when Nedeljkovic said the next frontier is integrated QA and video quality testing, which until now have been separate domains. At the time, he said Suitest had plans to locate partner companies shortly, but still this venture remains a work in progress, as he admitted this week that video quality testing is a complex topic. Suitest wants to eventually add these tools to its portfolio, but admitted in doing so it would mark its entrance into a heavily congested market. Many of Suitest’s customers are reportedly asking for video quality testing on their streaming apps to be provided by the same company supplying them with QA testing on these same apps, which we think is another good example of the ongoing commoditization of OTT video technologies.

In recent developments, last week Digital UK, the Freeview channel provider jointly owned by the BBC, ITV, Channel 4 and Arqiva, added two new on-demand players from CBS to its Freeview Play platform. Nedeljkovic spoke enthusiastically about its latest developments in HbbTV achieved by working closely with Digital UK, and will be making a few tests available to device manufacturers next year.

Speaking of consumer hardware, Suitest recently said it was one of the world’s first companies to offer a comprehensive test suite for Roku devices. Bemused, we asked why its rivals hadn’t bothered. Nedeljkovic explained, because Roku is based on a unique programming language called BrightScript, while most other streaming devices use JavaScript, it makes object-based testing more complex. Meanwhile, rivals offering image-based testing are not as limited due to just utilizing the HDMI encoder. So, despite the clear cost saving capabilities of object-based testing, there remains some advantages to image-based techniques.

We told Nedeljkovic it was interesting he hadn’t yet mentioned AI or machine learning, given the interest in certain techniques expressed by others in the space. This prompted a tentative confession that Suitest is in the process of applying off the shelf machine learning algorithms to test creation, to enhance the detection of problems, but not going so far as to tell an app how to function correctly, for example by allowing algorithms to alter an app’s core code base. He reckons the new features will roll out to customers in the region of six to twelve months.

The majority of Suitest’s business is in Europe, but CuriosityStream, the ad-supported nonfiction VoD service created by Discovery Channel founder John Hendricks, is an interesting prospect in the US, having struck partnerships with Amazon, Comcast, Sling TV, StarHub and YouTube over the years since its founding in 2015, the same year Suitest was born. But one of the most interesting projects are the trials Suitest has running at “all” the major movie studios in the US, which Nedeljkovic unfortunately couldn’t speak much about. We’ll be sure to probe during our catch up and much needed live demo while attending the NAB show in April.