Driving Bench: GPT-6 Astra Becomes First Model To Drive A Real Car Through A Course

OpenAI's GPT-6 Astra became the first model to complete DrivingBench, a new benchmark from Aditya Ramabadran, Simon Mahns and Tobias Gessler that lets frontier language models drive a real 2022 Toyota Corolla through a roughly 130-metre cone course in a parking lot. Astra finished the course on its second attempt in 5 minutes 22 seconds, while the other three models tested — Claude Fable 5.1, Grok 4.6 and GPT-5.6 Sol — all failed to get past the halfway mark. The car used comma.ai's comma four hardware running open-source openpilot software, with models controlling it through MCP tools observe(), set_motion() and stop_now() at a speed cap of 3.5 m/s (about 8 mph).

Not all AI benchmarks run purely on GPUs — some are moving https://officechai.com/ai/glm-5-2-places-2nd-on-vending-bench-ahead-of-all-google-openai-models/ into the real world. A new benchmark has handed frontier AI models the steering wheel, throttle and brakes of a real Toyota Corolla and asked them to navigate a cone course in a parking lot. Only one model made it to the end: OpenAI’s GPT-6 Astra. The benchmark, called DrivingBench, comes from Aditya Ramabadran, Simon Mahns and Tobias Gessler. It is designed to test whether general-purpose language models, not purpose-built self-driving systems https://officechai.com/ai/waymo-raises-16-billion-at-126-billion-valuation/ , can handle the messy business of controlling a physical vehicle in the real world. Astra completed the roughly 130-metre course on its second attempt in 5 minutes 22 seconds. Every other model tested failed to get past the halfway mark. What is DrivingBench? The premise is simple: can a frontier LLM drive a real car? The creators are explicit that this is a harder problem than it might sound, and different from both robotics demos and driving simulators. In a robot pick-and-place demo, a model can drop an object and simply try again. In a simulator, controls tend to be simpler and the feedback more predictable. On a real course, a wrong turn means a collision, and the model has to reason about a physical world it only sees through camera frames. The benchmark therefore tests perception, planning, control and latency all at once. Latency matters more than usual here, since the car keeps moving while the model is thinking. The creators note that models served with lower latency and higher throughput could have an edge, because they can close the feedback loop with the world faster. The four models tested were GPT-6 Astra, Claude Fable 5.1, Grok 4.6 and GPT-5.6 Sol. Each ran inside its native agent harness: Astra and Sol in Codex, Fable in Claude Code, and Grok in Cursor, all at medium reasoning effort. Astra is OpenAI’s newest flagship, built to be state of the art at computer use https://officechai.com/ai/gpt-6-astra-benchmarks/ , while Fable 5.1 https://officechai.com/ai/fable-5-1-benchmarks/ is a recent Anthropic release while Grok 4.6 https://officechai.com/ai/grok-4-6-benchmarks/ is xAI’s latest agent-focused release. The setup The test vehicle is a 2022 Toyota Corolla fitted with a comma four, the hardware from comma.ai that runs the open-source openpilot driver-assistance software. The comma four connects to the car’s CAN bus, which lets it send steering and speed commands. The team built a harness on top of openpilot, exposing the car to models through MCP tools. There are three tools: - observe returns the current camera frames along with speed, steering angle and remaining motion. - set motion takes a direction, a steering percentage, a speed, a duration and a short stated reason. It replaces whatever command is currently active. - stop now brakes immediately. Commands don’t queue; a new one overrides the old one, and when a command expires the car begins braking rather than continuing on its own. Because the car keeps moving while a model reasons, models have to choose durations that account for their own thinking time, and ideally send a replacement command before the current one runs out. Every tool response carries timestamps, so a model can in principle work out its own latency. Steering is expressed as a percentage, where 100% maps to 180 degrees of steering-wheel angle. The team originally had models specify curvature and distance in metres, but found the models weren’t well enough calibrated on distance, and switched to the percentage scheme, which worked much better. The system caps speed at 3.5 m/s about 8 mph , and the steering wheel can turn no faster than 100 degrees per second, so models also had to learn how the car actually responds. A human operator sat in the driver’s seat throughout with a foot over the brake, ready to stop the car if it left the course or was about to hit something. openpilot’s driver monitoring and other safety features were left intact, and the code enforces multiple layers of speed limits, including an emergency stop that cancels motion if speeds exceed 6 m/s. The course The course is laid out with mini-cones in a large, mostly empty parking lot in the Bay Area, and it runs about 130 metres. It includes a left entry turn, a long straight aisle, gentle bends, a right turn into a cross aisle, another right turn into a final aisle, and a straight approach to a finish zone marked by blue cones, where the car has to park. The creators say they designed it to include a full range of manoeuvres while avoiding turns too tight for the comma-and-Corolla setup to handle, and admit they were surprised any model completed it at all. Finding the lot was its own challenge. The team sent ChatGPT and Grok Bot agents to scan Google Maps satellite imagery for suitable lots, with mixed results, and the team was asked to leave two lots along the way. Methodology Each model got up to three attempts in one continuous chat. After a failed attempt, the operator sent a generic reflection prompt asking the model to think about why it ended and what to do differently, followed by a continuation prompt granting another try. The point was to test in-context learning: could a model improve its driving using only its own experience within the conversation? Scoring is based on progress along the course centerline, counted only while the car stays within 4 metres of it. Progress can only increase, so driving the wrong way or sitting still earns nothing, and a collision keeps the progress reached before it. Finish time is a secondary metric for runs that complete the course. The creators say the leaderboard ranks models by their best of up to three attempts, with a first-attempt view also available. All runs are published as open traces with video, and the harness and prompts are open-sourced. Results | Model | Harness | Attempt 1 | Attempt 2 | Attempt 3 | Best | |---|---|---|---|---|---| | GPT-6 Astra | Codex | 49% | 100% 5:22 | — | 100% | | Claude Fable 5.1 | Claude Code | 9% | 10% | 45% | 45% | | Grok 4.6 | Cursor | 8% | 11% | 10% | 11% | | GPT-5.6 Sol | Codex | 6% | 6% | 6% | 6% | Astra’s first attempt covered about 67 metres before ending, roughly halfway. On its second, it drove the full 134.7 metres and parked in the blue finish zone. At that pace, it averaged well under 1 m/s, or about 1 mph. The report says it never exceeded 0.8 m/s on that run. The two Astra attempts cost about $9.75 in tokens at list prices. Fable 5.1 was the only other model to get anywhere, reaching 45% on its third attempt after scoring 9% and 10% on the first two. Grok 4.6 and GPT-5.6 Sol never got much past the first corner in any attempt. Where models failed The most common failure was perceptual. Most attempts ended at the first corner, where models had to work out which side of a diagonal line of cones the lane was on. Fable’s reflection after its second attempt acknowledged picking the wrong side of the boundary again. Sol’s reflection concluded it had wrongly assumed that cone colour indicated which side of the lane a cone marked, though the cones were deliberately multicoloured. Grok noted that the car is wider than the camera view makes it look. Latency and control cadence mattered too. Astra checked in about every 5 to 6 seconds and issued around six commands a minute. Astra and Sol were the only models to actually replace a motion command before it expired. By contrast, in Fable’s second attempt, the car was moving for only 31 seconds out of about 190, with most of the time spent reasoning while the car sat braked. Grok’s own reflection identified the problem, telling itself to overlap commands, yet in its next attempt it still left gaps of around 10 seconds. Signs of in-context learning Astra and Fable both showed real improvement between attempts. After its first run, Astra reflected that it had declared the car aligned too early and resolved to use shorter movement increments at lower speeds near bends and islands. On its second attempt it stayed at or below 0.8 m/s and used full steering lock on 20 of its 24 commands, and finished. Across both attempts, it held 100% steering on 20 of 32 commands, having learned that the car needed it. Fable moved in the opposite direction on steering. It began with commands of 60 to 100%, decided that was too much, and settled on a default of 30%. In its third attempt, it worked out the lane on the long straight correctly, but left too little room for the following right turn. Caveats The creators are upfront about the limitations. Each model was evaluated only once, and although models had multiple attempts, those attempts aren’t independent since they share a context. The comma setup limits how tightly the car can turn, and openpilot requires the human to press a button before the first command can move the car, so some first commands took effect a few seconds late. The forward camera also can’t see objects very close to the car or anything to the sides or rear. The prompt’s steering example was conservative, too: it described a 90-degree turn at full lock taking about 60 seconds at 1 m/s, when GPS tracks put it closer to 25 seconds. Every model received the same text. One quirk stood out. Astra sometimes refused to drive the physical car, citing safety, even in an empty lot with speed caps and a human ready to brake. Calling it a simulation didn’t work, because models sometimes realised from the images that the car was real. What worked, oddly, was renaming the MCP server “DrivingBench Sandbox”. The creators plan to run models multiple times, test different reasoning efforts, slim down the prompt, and add more models and a longer, harder course in a future version. Their conclusion is that frontier models can now drive real vehicles at sufficiently low speeds, and that the result calls for more work on safety and evaluation. The result adds another data point to the case that Astra’s strength lies in physical and computer-use tasks, even as independent indexes have shown it barely ahead of its predecessor https://officechai.com/ai/openai-gpt-6-astra-scores-a-disappointing-61-on-artificial-analysis-intelligence-index-same-as-gpt-5-6-sol/ on general intelligence and its ARC-AGI-3 results https://officechai.com/ai/gpt-6-astra-major-breakthrough-on-arc-agi-3-with-score-of-62/ point to a big jump in novel-task reasoning. Meanwhile, its predecessor GPT-5.6 Sol https://officechai.com/ai/openai-launches-gpt-5-6-sol-beats-mythos-on-terminalbench/ , now succeeded by the cheaper GPT-6 Sol https://officechai.com/ai/openai-releases-gpt-6-sol-and-gpt-6-luna-with-50-lower-prices-than-gpt-5-6/ , never advanced past 6% on the course.