{"slug": "bugs-that-broke-driving", "title": "Bugs That Broke Driving", "summary": "Comma.ai's openpilot driver-assistance system has been plagued by bugs that caused dangerous driving behaviors, including drifting right, detecting false lanelines, and cutting turns, all traced to flaws in sensor calibration, ground-truth labeling, and visual odometry. The company documented these issues in a blog post detailing fixes that improved driving performance more than new machine-learning techniques.", "body_md": "#\n[Bugs that broke driving](https://blog.comma.ai/driving-bugs/)\n\nAll software is riddled with bugs, and usually bad software is bad because of those bugs, not because of bad algorithmic choices. openpilot is no different; we’ve made great progress over the years by implementing new exciting ML techniques, but most of the driving improvements came from discovering and fixing blatant bugs. Here I walk through some of the most entertaining bugs through the years.\n\n## openpilot drifts right[Permalink](#openpilot-drifts-right)\n\nIn 2017 comma was not a mature company. We had few users, low quality logging, no metrics in training other than a unitless val loss, and no closed-loop simulation tests. As a result we generally had no clue what was going on with openpilot beyond obvious issues we noticed while driving with it. This is clear from these Slack messages showing we were aware of the driving model’s tendency to bias driving slightly to the right for months, with no root cause or satisfactory explanation.\n\nBy the end of the year we did figure out the major cause of this bias. We trained models to predict the future driving trajectory, and lanelines. These targets were ground-truthed with a localizer that relied on gyro measurements, as well as other sensors. Gyros are usually biased, the Android stack we used automatically corrected for the bias when it detected the device was at rest. This meant that when openpilot started and the car was stationary the bias was corrected appropriately. However, when the car then started moving, the openpilot device would heat up causing the bias to change, but the device never came to rest again so the bias was never corrected. The bias correlation with temperature is similar for every device, so all data would skew the same direction. A more comprehensive localizer that accounted for bias accurately fixed this.\n\n## Laneline predictions in the middle of the lane[Permalink](#laneline-predictions-in-the-middle-of-the-lane)\n\nIn 2019 we started getting mysterious complaints of cars departing the lane on roads with clearly marked lanelines. This happened because at the time openpilot still relied on the lanelines for driving decisions, and as the image shows, in these cases it would inexplicably detect a laneline in the middle of the lane. After looking for some more cases like this, we found all examples were in situations with bright daylight.\n\nThe issue became obvious when looking at several examples of the training data with this same flaw. In certain lighting conditions the sun can reflect on the hood showing a bright white area, this gets detected in the ground-truth stack as a laneline marking that is always just in front of the car, and thus gets reconstructed as a laneline in the middle of the road. By adding better ground-truth of specifically these examples to the laneline detector, we retrained the detector and trained a new driving model with the fix.\n\n## Turn-cutting in highway turns[Permalink](#turn-cutting-in-highway-turns)\n\nIn October 2020 there were several complaints about openpilot crossing the inner laneline in turns. After looking at the examples they all appeared to happen in long sweeping turns at night, in all cases the driving model predictions for the path and lanelines that were biased towards the inside of the turn.\n\nAn investigation into the ground-truth showed the localizer incorrectly computed the car’s motion in these cases. Some localizer debugging revealed that the headlights shining onto long continuous lanelines creates problems for the visual odometry. The ORB feature detector selected edges based on the headlight illumination that move vehicle with the vehicle. This means that in scenes where there are little other features for the ORB detector to track, the localizer relies only on these erroneous features causing the localizer to be consistently biased towards those inside of the turn. Smarter ORB feature matching and outlier rejection fixed this, and the retrained model addressed the complaints.\n\n## Braking blurs the image[Permalink](#braking-blurs-the-image)\n\nPhone cameras use a voice coil actuator to move the lens for focus adjustment. This means the lens is not rigidly mounted to a focus position, but instead mounted on a spring and focus movements are achieved by applying a force with the actuator. This also means that if there are rapid accelerations of the device, focus will shift due to the inertial forces, and to maintain focus a compensatory force needs to be applied with the actuator.\n\nWe had an algorithm that would compensate for this focus shift for the OnePlus phones we were using in comma devices in 2019. Later we switched to Leeco phones, but never re-evaluated the focus algorithm. As it turns out our compensation algorithm was causing significant blur during acceleration on these newer devices, and these phones needed no focus compensation at all. This algorithm caused exactly what it was designed to prevent!\n\n## Driving model never converges to good highway speeds[Permalink](#driving-model-never-converges-to-good-highway-speeds)\n\nWhen experimental mode was introduced in openpilot, openpilot did not decisively accelerate or brake to a good speeds on the highway. Instead in steady-state, it would just keep driving the speed it was initialized at. Which could for example cause it to drive 55mph forever on an empty long highway.\n\nopenpilot driving models are trained on-policy in a driving simulator. The inputs to the simulator are 6dof pose changes, we use this instead of actions (acceleration and curvature) so we have more fine-grained control over the physics of the simulation. We noticed that we needed to add a lot of pitch noise to the simulation for the model to learn to brake and accelerate.\n\nAfter investigation this was due to the tight coupling of pitch and acceleration. In any normal vehicle, acceleration causes pitch movements. This means acceleration must be simulated with an appropriate pitch change. This lack of realism in the simulator can be exploited by the student model that is trained in it. As the results below show, fixing these artifacts by simulating realistic pitch movements during simulated acceleration made the model perform much better at our speed convergence tests.\n\n## Runtime performance doesn’t match train-time tests[Permalink](#runtime-performance-doesnt-match-train-time-tests)\n\nThis bug is so good we wrote it 4 times. Over the years openpilot has supported various cameras and mounting configurations. As a result sometimes the models were trained to take in bigger images than were available in some configurations. To make this work, the images needed to be padded to fit to the model’s desired input shape. It is essential that the padding is the same during train-time and runtime, but the inference and training stacks are completely separate. Over the years they each changed frequently, and were poorly tested to match each other.\n\nHere are all the varieties of bugs with padding making runtime different from train-time causing regressed outputs:\n\n- in 2021:\n`fc19544`\n\n(#22971) - in 2024:\n`8656ef1`\n\n(#31495) - in 2026\n`f24ad7e`\n\n(#37986) - also in 2026\n`4585e93`\n\n(#38091)\n\n## Looking back[Permalink](#looking-back)\n\nA lot has changed since 2017, many of these bugs would be unlikely to make it to production with our current tooling and infrastructure. We will still write many silly bugs though, and if you like the idea of fixing those bugs, [maybe you should come work here.](https://comma.ai/jobs)\n\n*Harald Schäfer*\n\n*CTO @ comma.ai*\n\n#### Share on\n\n[X](https://x.com/intent/tweet?via=comma_ai&text=Bugs+that+broke+driving%20https%3A%2F%2Fblog.comma.ai%2Fdriving-bugs%2F)\n\n[Bluesky](https://bsky.app/intent/compose?text=Bugs+that+broke+driving%20https%3A%2F%2Fblog.comma.ai%2Fdriving-bugs%2F)", "url": "https://wpnews.pro/news/bugs-that-broke-driving", "canonical_source": "https://blog.comma.ai/driving-bugs/", "published_at": "2026-07-29 02:55:44+00:00", "updated_at": "2026-07-29 03:22:35.592456+00:00", "lang": "en", "topics": ["autonomous-vehicles", "computer-vision", "machine-learning", "ai-products"], "entities": ["comma.ai", "openpilot"], "alternates": {"html": "https://wpnews.pro/news/bugs-that-broke-driving", "markdown": "https://wpnews.pro/news/bugs-that-broke-driving.md", "text": "https://wpnews.pro/news/bugs-that-broke-driving.txt", "jsonld": "https://wpnews.pro/news/bugs-that-broke-driving.jsonld"}}