Self-driving software has to handle the unexpected. In a 2015 TED talk, Chris Urmson, then head of Google’s self-driving car project and now CEO of self-driving vehicle technology firm Aurora, showed footage of a woman in an electric wheelchair chasing a duck in circles on the road. The vehicle slowed and navigated the encounter safely, he said.
Urmson’s scene was unusual. Animals on the road are not. Collisions with animals killed 210 people in the U.S. in 2024, according to the Insurance Institute for Highway Safety, and a 2008 Federal Highway Administration report to Congress estimated 1 million to 2 million collisions between vehicles and large animals each year. For developers of automated driving systems, a central challenge is finding those encounters in fleet sensor recordings and turning them into tests they can rerun whenever the software changes.
During NVIDIA’s GTC Berlin developer livestream, Tin Stribor Sohn, a tech lead at Porsche, and Dan Gural, head of technical partnerships at Voxel51, which provides a data and model platform for physical AI, robotics and computer vision, described how their teams pull rare scenarios from exabytes of fleet data and rerun them in simulation instead of re-driving them. Voxel51 helps Porsche find relevant driving scenes and prepare them for reconstruction and simulation with NVIDIA tools.
Some miles of fleet data are more useful than others. What Porsche needs are the rare scenes that show whether a software update made a driving function better or quietly broke something that used to work. “Not all of the drives we have have the same value,” Sohn said. “We need to identify corner cases, and one of these corner cases can be some animal running over the highway, or some helicopter landing on the highway.”
Each of those scenes becomes a regression test, and the collection keeps growing. “Every time we add a new release of an AI function, we don’t only need to test new scenarios we encounter, but also the whole set of scenarios we already encounter,” Sohn said. Re-driving all of them on real roads, he added, “would grow exponentially over time.” Using NuRec, NVIDIA’s neural reconstruction tools, the team turns recorded drives into 3D scenes and runs them overnight in simulation, so engineers know by the next day whether a release is ready to go into the vehicle, Sohn said.
One rare encounter can also become many tests. NVIDIA’s Cosmos Transfer model generates variations of a reconstructed scene in different weather and lighting. “I want to test it in the rain. I want to test it in the snow. I want to test it at night, daytime, foggy, whatever you can imagine,” Gural said.
Before those recordings become simulations, the team checks the underlying sensor data. Gural gave the example of a vehicle hitting a pothole and a camera shifting out of position. Such problems can compromise the material used to train a model. “We want to make sure that the data that we’re training the models on are as high quality as possible,” he said. The math and photogrammetry behind those checks are complex, but the principle is simple: “The best data makes the best models,” Gural said.
Porsche helps identify which driving problems need attention, while Voxel51 integrates the tools used to investigate them. “They know the issues where the car is doing well, where the car is doing poorly,” Gural said of Porsche and its parent, the Volkswagen Group. Sohn said the resulting library of driving scenarios is meant to serve the whole group: it can be reused across development projects, shared with partners that train models, and expanded as the fleet encounters new situations.
Sohn judges automatic braking by how it performs on real roads, and that means cataloging every situation that should trigger it. “We want to identify all of the events that cause braking,” he said, including animals and people entering the road.