Cali DMV disengage report shows how far self-driving has to go

The annual release of the Californian DMV’s self-driving car testing report has arrived, publishing the progress of autonomous vehicle development on public roads. Local law requires that companies tell the DMV how many times they had to override a car in autonomous mode, providing a crude benchmark of sorts. As expected, Google’s Waymo still leads the pack, but based on these numbers alone, the rest of the industry has a lot to catch up on.

Of course, the early-stage testing is going to incur the higher number of disengagements, which the DMV defines as ‘a deactivation of the autonomous mode when a failure of the autonomous technology is detected or when the safe operation of the vehicle requires that the autonomous vehicle test drive disengage the autonomous mode and take immediate manual control of the vehicle.’

However, one thing we can extrapolate from the data is that the leading approach doesn’t seem to have come on all that far. While that might sound gloomy, it should be noted that there still hasn’t been a crash death in one of these test vehicles. In 2016, there were 37,461 deaths on US roads, equating to 1.18 fatalities per million vehicle miles traveled. Waymo alone has driven over 4-million miles in the US, with half of those covered last year. So far, it seems to be noticeably ahead of that death-per-million-miles figure.

Compared to when Riot last covered the DMV report, in January 2016, Waymo has only improved slightly on the intervention-per-mile basis – rising from 5,318 in 2015 to the 5,956 in 2017. The Waymo testing in 2014 saw an intervention every 785 miles, and so the jump in that year was pretty remarkable. However, the great leap forward appears to have come to a halt, and from here on in, we might only witness incremental improvements in the order of 10% or so.

On this current data alone, it seems that GM is currently sitting at around 2014 Waymo capability, and it seems that it should be able to get up to Waymo’s level in time. GM says that it has been testing its cars in more challenging environments, the middle of San Francisco to be precise, and that its disengagement rating would be higher if it stuck to highways – although it did meet some of the local wildlife, and is currently facing a lawsuit from a motorcyclist.

But the lingering big question is the potential severity of each of these disengagements, had there not been an intervention. Whether the incident would have led to car damage, occupant or pedestrian injury, or even death, is not touched on – although the data from the vehicles could be passed through simulations to estimate the potential fallout.

Similarly, there isn’t really a human-equivalent for the disengagements, meaning that it’s hard to gauge how the self-driving modes measure up against an attentive human driver. Again, there’s no way of comparing the figures for a distracted (texting, talking, playing with the IVI system) or unfit (drunk, high, old) driver, so we’re still lacking a yardstick in that regard.

However, it appears that the automakers and OEMs are making significant progress in their approaches. While the likes of Intel’s Mobileye and BMW are planning for SAE Level 5 vehicles (entirely autonomous) by 2021, the wider tech community thinks it will take longer. The current state-of-the-art cars could be let loose on the roads today and would probably not kill anyone – but they would not play nice with the norms of driving, and could cause piles of congestion. They might not be very good at getting from point A to point B in a timely fashion.

The DMV report outlines the filings from all the companies currently testing vehicles in California, where the local DMV asks that each company provide data for each time a human test driver has to intervene. California is unique in this regard, among US states, but it does not require reports for tests conducted on private land – only public roads.

Of the 50 companies that have testing permits, 20 were required to publish results – but 19 filed their paperwork on time. Faraday Future did not, and is being probed by the DMV, but that behavior sounds pretty par for the course for Faraday Future – which has been beset by problems since launch. Some 7 companies did not test on public Californian roads, including BMW, Ford, Honda, Tesla, and Volkswagen.

Unsurprisingly, the Alphabet-owned Waymo (which is currently suing the pants off Uber) clocked the most testing mileage. Traveling 352,544 miles (567,365km), Waymo reported 63 disengagements across its fleet of 75 cars – an average of one manual override per 5,596 miles. Waymo was involved in 3 traffic incidents.

GM was second-place in distance traveled, clocking in at 131,675 miles (211,910km), and reporting 105 disengagements across its fleet of 86 Cruise vehicles. That’s an average of 1,254 miles per override, around a fifth (22.4%) the capability of Waymo. GM was also involved in 22 collisions through 2017, which is relatively rather a lot, but it improved its per-mile range from 300-miles to that 1,254-mile level. Its fleet also has some great car names, with Bolty McBoltface a highlight.

However, Waymo and GM were way out in front of all the other testers, and their number of disengagements suggest that they are much further along when it comes to testing on public roads. Because the testers aren’t publishing the results of their private tests, it’s hard to draw any solid conclusions from the data, but it would seem logical that the more advanced a project was, the more happy a company would be to test it on the open roads.

Mercedes-Benz drove three vehicles a combined 1,087 miles, resulting in 842 disengagements – one every 1.3 miles. Bosch drove three cars for 1,454 miles, but disengaged 598 times – once per 2.43 miles. Other total distances were similarly small, with clocking 6,127 miles, Nissan hitting 5,007 miles, Zoox reaching 2,244 miles, and Delphi (now Aptiv) traveling 1,810 miles.