No longer is the simple speed test an accurate gauge of network performance, as faster and more intelligent networks mean ways of measuring actual performance becomes an increasingly complex endeavor. The emergence of standards like the Broadband Forum’s QED (Quality of Experience Delivered) aim to help ISPs better understand network data and therefore ultimately improve end user experiences, particularly via WiFi, but are at the same time contributing to the network performance measurement conundrum.
Once upon a time, upgrading line capacity from 2Mbps to 4Mbps generally meant your network was noticeably twice as good. Today, jumping from a 1Gbps to 2Gbps line would go completely unnoticed by most end users. So, while the speed test is an important measure, it becomes less important as a measure of a good experience, which is where QED measurements come in – adding more dimensions to the performance landscape with an application-specific approach.
Following on from the lifting of the QED lid in Q4 2020, after some three years in incubation, details have since emerged showing how Vodafone, which is one of the main operators driving the standard along with Orange and Telenor, is implementing the fledgling framework.
Jonathan Newton, principal network architect at Vodafone’s Fixed Access Center of Excellence, admitted that QED might indeed be a daunting prospect to begin with in terms of an ISP changing its existing methods of producing end user analyses. This is something the community is keen to dispel.
Speaking during a recent BBF webinar, Newton served up three approaches. Firstly, ISPs can leverage existing capabilities such as dedicated smartphone applications or network probes that are deployable on customer sites. Simply sending small upgrades to these apps or probes can create the capability to provide results that can be analyzed under QED.
Operators can also take advantage of the large number of network-level probes using standardized test protocols like STAMP (Simple Topology Announcement and Management Protocol). “You have a prerequisite to do the monitoring of sample data, and just a small increment is required to feed that into a QED-style analysis,” explained Newton.
The third and final step is the deployment of virtualized probes, which is enabling testing on-demand in edge cloud locations and supports the introduction of additional testing capabilities.
The overarching message is that ISPs are increasingly responsible for WiFi in the home – and QED increases the capability to monitor and understand this performance.
Diving deeper into how this relates to the large and unpredictable latency of WiFi, Njorn Ivar Teigen, Head of Research at Domos, presented some important findings. The Norwegian machine learning specialist has collected a large dataset from running a QED sampler at scale, which it has applied to finding samples that stand out with higher latency and loss than what is normal or desired over WiFi. From here, Domos has been able to analyze the samples to find plausible causes to those problems and investigate further.
On a side note, Teigen said that once you have a QED sample running at scale, it becomes quite easy to do A/B testing of new services, for instance.
Bringing up an exert from a Domos dataset comparing 2.4GHz and 5GHz networks, it was obvious that the latter performed significantly better, which was expected. However, as signal strength increased, Domos noticed that latency problems were not going away. You would expect that as the signal strength increases, you also get a higher rate (rate being the throughput speed of the WiFi link) because the radio can push more data through when the signal is stronger.
However, Domos found that – especially in the upstream direction – there were a lot of cases where signal strength was very good, but the rate was very low. This is clearly a problem as a lot of available airtime on the radio was being used up, causing queuing that translates to latency and packet loss for end users. Having identified this problem, Domos was then able to design tests to diagnose and configure the network to mitigate this problem.
Providing another perspective on QED was Peter Thompson, CTO at network performance science outfit Predictable Network Solutions (PNSol). Attempting to cut through the complexity, Thompson noted how the naming of things is always tricky. He outlined how QED is a program of BBF, a workstream if you like, that is generating technical and marketing reports which are based on notions of quality attenuation. But QED goes beyond that by defining use cases and ultimately conformance tests.
In other words, measuring QED means measuring quality attenuation. There is always some delay and some chance of failure in the network, and therefore some attenuation of quality, which the industry also calls ΔQ. “Quality attenuation and ΔQ are the same thing. The latter is just a bit more of a mathematical notation that we can more easily put in formulas,” he said.
“ΔQ combines delay and loss into a single measure. If you have to wait too long for something it might as well never happen, so blending two things into a single measure makes life easier. The whole achievement of layered network protocols has insulated the application from most of the network details, leaving just ΔQ. This goes beyond bandwidth because it incorporates the notion of bandwidth as an additional case,” continued Thompson.
Ending back where we started, we should reiterate that QED doesn’t replace the speed test, it complements the speed test.