While many eyes in the broadband and TV worlds were tuned last week to the Mobile World Congress in Barcelona, they might have done better to hop across the pond and drop in on the IBM Interconnect Conference running at the same time in Las Vegas. That was also largely about mobility as the name half indicates, but gave a remarkably comprehensive insight into the state of the science of big data analytics and where it is going over the next few years. As far as TV was concerned there was both a reality check and a view over the horizon of how much better things could become in a few years if operators get their act together and start engaging with the times. The truism that traditional pay TV is a laggard when it comes to applying analytics not just to viewer data but myriad external sources, was clearly upheld.
There are naturally though some evangelists among operators and not just as would be expected specialist TV analytics firms. Among the latter it is the younger breed of OTT providers rather than legacy operators that have analytics more in their blood and not surprisingly the name of Netflix cropped up frequently at the IBM conference. Netflix has built its success on big data and analytics and is still at the cutting edge seeking ways it can zoom in ever more closely on individual preferences and exploit those in as close to real time as possible.
This comes back to that IBM conference and it was the host itself, once known as Big Blue when it dominated the enterprise computer industry through the 1970s and 80s, which stole the show with its launch of the open source Quarks platform designed to analyze data on the fly at the network edge, pitched particularly at Internet of Things (IoT) devices or endpoints. Quarks enables rapid real time processing of streams but crucially also the ability to make sense of the data as it goes. This involves both adequate cleansing or pre-processing of data on the fly and equally importantly the ability to incorporate data from other sources within the network, generated by packages such as IBM’s own Streams, to provide what IBM calls comprehensive distributed “edge to center” analytics.
IBM Streams itself was designed to enable applications developed by customers to ingest, analyze and correlate information quickly from multiple “big data” sources, but focused more on very high data throughput at up to millions of events or messages per second within the center of an operation. Quarks complements this for the network edge, but may well more often be working with other equivalent, more specialized packages in the case of pay TV and broadcasting.
The context here begins with Apache Hadoop, which is now being deployed by just about every major enterprise in broadcasting and outside, with the market forecast to grow from $2 billion in 2013 to $50 billion in 2020 at a CAGR of 58.2 % according to Allied Market Research. Hadoop is open source software written in Java for distributed storage and processing of very large data sets, with some built in fault tolerance, so that it can run on off the shelf commodity hardware. As such it is very much in the spirit of virtualization and cloud computing and has now become the de facto standard for managing and processing hundreds of terabytes or even petabytes of data from multiple sources. It has two core components, the storage part called the Hadoop Distributed File System (HDFS), and the processing part called MapReduce. Files are split into large blocks and distributed across nodes in a cluster, each of which runs pre-packaged code in parallel according to what data needs to be processed. This takes advantage of data locality, so that each node only manipulates data it has immediate access to in order to accelerate processing.
It would be interesting to see statistics on the rate of growth of data volumes managed by HDFS, but it has certainly been explosive for several years, above all among Internet and social media companies. At the Hadoop Summit 2015, Twitter said it had over 300 petabytes spread across Hadoop clusters comprising more than 1000 servers.
But while HDFS has become widely deployed, Map Reduce has been overtaken to some extent by more recent processing engines that are more efficient for some use cases, most notably interactive queries and algorithms that involve iteration to revisit the same data set many times, as well as streaming applications. For such applications Map Reduce can act as a bottleneck rather than an accelerator with its locality and so alternative architectures have emerged with one of the most popular being the Apache Spark. Developed at UC Berkeley in 2009 and commercialized by Databricks, Spark was designed from the ground up for interactive queries and iterative algorithms, as these were two major use cases poorly served by MapReduce with its design for large scale batch processing. Spark achieves faster performance through a more open cluster combined with incorporation of dedicated engines for specific application groups, including streaming and graph computations.
The latter is relevant for TV and is not just about visualization but also analyzing relationships and trends within subscriber data. Essentially entities such as customers are depicted as nodes and relationships between them as lines joining those nodes. Graph computing has proved almost uniquely able to perform exquisite feats of data mining, teasing out hidden, but potentially valuable correlations or relationships inside very large data sets.
Spark has achieved spectacular performance gains for these use cases, running up to 100 times faster than Hadoop MapReduce in main memory, or 10 times faster on disk where the gains are constrained by the distances the disk head has to travel when accessing non-local data. In solid state memory all data reads take essentially the same time, while on hard drives they are dependent on physical location. Netflix was one of the first video operators to deploy Hadoop and then also to replace MapReduce with Spark, partly for the speed but also to gain greater flexibility for big data analysis.
IBM’s Quarks is part of this movement towards expanded Hadoop based architectures, and when combined with a central analytic system, can yield an analytic hierarchy with processing distributed across the whole span of a network. Quarks itself is an SDK for deploying lightweight embeddable runtime software to perform streaming analytics at the edge. It allows devices to react locally to events and then send data to central systems if that event is deemed of wider interest. Most crucially it allows in principle all relevant data to be delivered the moment an event of interest occurs, rather than just relying on a periodic heartbeat of information, which may not include all data needed to take actions, or which might arrive too late.
This gives an idea of where analytics is going, but most pay TV operators are nowhere near that point yet and in any case this all ignores the reality that a lot of big data is not yet fit for purpose. TV audience data in particular is complex with large numbers of inter-dependencies among overlapping data sets that need to be ironed out before the data can be loaded via traditional ETL (Extract, Transform and Load) techniques into a data warehouse ready for analytics processing.
Some operators have been disappointed so far with experiences of data analytics because of failure to integrate different data sets. This is a problem that didn’t happen so much when analysis was confined to traditional panels and immediate set top return path data, but has grown with the proliferation of multiscreen viewing on connected devices bringing in data from additional sources. In some cases, operators are getting much better data from the online side, especially if they still rely on traditional TAM (TV Audience Measurement) data rather than more comprehensive return path data.
For example, a panel will give a decent enough view of how bigger channels are performing, while an operator may at the same time be getting good feedback at the household or individual level about multiscreen consumption. But the joined up picture is lost because the two data sets do not interact and they provide information that on the surface is incompatible.
IBM for one at least claims to be up with the problem. The future definitely lies with more automation of big data preparation but for TV it will be some time before it is possible to combine all sources on the fly for determination of real time actions.