In a bid to prove that its robot drivers are safer than humans, Waymo simulated dozens of real-world fatal crashes that took place in Arizona over nearly a decade. The Google spinoff discovered that replacing either vehicle in a two-car crash with its robot-guided minivans would nearly eliminate all deaths, according to data it publicized today.

The results are meant to bolster Waymo’s case that autonomous vehicles operate more safely than human-driven ones. With millions of people dying in auto crashes globally every year, AV operators are increasingly leaning on this safety case to spur regulators to pass legislation allowing more fully autonomous vehicles on the road.

But that case has been difficult to prove out, thanks to the very limited number of autonomous vehicles operating on public roads today. To provide more statistical support for its argument, Waymo has turned to counterfactuals, or “what if?” scenarios, meant to showcase how its robot vehicles would react in real-world situations.

Last year, the company published 6.1 million miles of driving data in 2019 and 2020, including 18 crashes and 29 near-miss collisions. In those incidents where its safety operators took control of the vehicle to avoid a crash, Waymo’s engineers simulated what would have happened had the driver not disengaged the vehicle’s self-driving system to generate a counterfactual. The company has also made some of its data available to academic researchers.

That work in counterfactuals continues in this most recent data release. Through a third party, Waymo collected information on every fatal crash that took place in Chandler, Arizona, a suburban community outside Phoenix, between 2008 and 2017. Focusing just on the crashes that took place within its operational design domain, or the approximately 100-square-mile area in which the company permits its cars to drive, Waymo identified 72 crashes to reconstruct in simulation in order to determine how its autonomous system would respond in similar situations.

Some of these crashes involved one vehicle, while most involved two. For crashes with two vehicles, Waymo ran separate experiments simulating its autonomous vehicles in the role of each vehicle — first replacing the vehicle that initiated the crash and then replacing the vehicle that responded to the other vehicle’s actions. For crashes with one vehicle, Waymo only simulated the single vehicle. That left it with 91 simulations in total.

The company reconstructed these crashes, systematically aligning the vehicle’s trajectory to make sure its Waymo vehicles are exposed to a similar situation than in the actual fatal crash. Waymo used the same simulation platform it uses to train and evaluate its autonomous vehicles on virtual roads in ordinary operations.

The results show that Waymo’s autonomous vehicles would have “avoided or mitigated” 88 out of 91 total simulations, said Trent Victor, director of safety research and best practices at Waymo. Moreover, for the crashes that were mitigated, Waymo’s vehicles would have reduced the likelihood of serious injury by a factor of 1.3 to 15 times, Victor said.

“That means that even if it didn’t avoid the crash completely, it took action to reduce the severity of the impact,” Victor said. “If the severity was reduced, it made it less likely that the driver would have died.”

When Waymo replaced the vehicle responding to the instigator of the crash, it found that it “completely avoided” 82 percent of simulated crashes — the vast majority of which without the need for hard braking or evasive action. In another 10 percent as a responder, the Waymo vehicle’s urgent maneuvers helped mitigate the severity of the crash. Those crashes all took place at an intersection where the other vehicle either turned left across or cut straight across the Waymo vehicle’s path, leaving little time to react, the company said.

“We’re not saying that we will eliminate all fatalities, but we are saying that the best way to reduce the chances of a serious injury would be to take an evasive maneuver when that’s possible,” Victor said. “And in all of these crash simulations, the Waymo driver took evasive maneuvers.”

There were three incidents of a person dying after being rear-ended by another vehicle. In simulation, Waymo was not able to avoid these crashes when its vehicle was the one being rear-ended. “In rear crashes specifically, there’s not a lot that the responder role can do,” said Matthew Schwall, head of field safety at Waymo. “So the Waymo driver really suffers from the same challenge that humans do in those situations, that it’s hard to make the prediction soon enough to be able to make an evasive maneuver.”

Twenty simulated crashes involved a pedestrian or cyclist being struck by a driver. Waymo’s autonomous vehicles avoided 100 percent of those crashes in simulation, the company said.

There’s no standard approach for evaluating AV safety. A recent study by RAND concluded that, in the absence of a framework, customers are most likely to trust the government — even though US regulators appear content to let the private sector dictate what’s safe. In this vacuum, Waymo hopes that by publicizing this data, policymakers, researchers, and even other companies may begin to take on the task of developing a universal framework.

To be sure, Waymo did not make its findings subject to a peer-reviewed analysis for publication in an academic or scientific journal, though it would be open to publishing it in the future, a spokesperson said. The simulations were not conducted independently of the company, nor were they reviewed by any third-party for verification prior to the company making them public.

The company did share its findings with a select group of academic experts to get their reactions. Daniel McGehee, director of the National Advanced Driving Simulator Laboratories at the University of Iowa, said Waymo is pushing safety analyses and transparency “to a new level.”

Jonas Bargman, associate professor for vehicle safety at Chalmers University of Technology in Sweden, said that Waymo’s use of simulation-based assessment is at “the scientific forefront” but only one of the components needed to assess the safety of automated vehicles.

After reviewing the report, Bargman concluded that the “Waymo simulation platform is highly sophisticated,” citing the company’s 3D sensor-level perception models. He singled out the use of counterfactual what-if scenarios as “highly relevant for the introduction of Waymo self-driving vehicles to the public.”