Bugs that broke driving: Machine Learning edition Comma.ai published a blog post detailing machine learning bugs that degraded its self-driving model's training and inference, including a BF16 plan-output precision issue that limited speed representation to 0.125 m/s steps around 30 m/s, an FP16 ConvNeXt overflow where a residual multiplier near 54 at stages[2].blocks[29] exceeded FP16's 65,504 limit, and a desire-loss bug in which NaN targets were converted to class 0 by argmax. The company said the fixes promote the plan head and its inputs to FP32, divide the affected ConvNeXt stage's residual stream by four, and mask NaN targets before computing cross-entropy loss. Comma.ai framed the retrospective as a way to understand where its code tends to break and catch problems earlier. Bugs that broke driving: Machine Learning edition https://blog.comma.ai/ml-bugs/ “We have solved self driving, we just need to fix the bugs.” Rumor has it that the Linux kernel averages around 0.5 bugs per 1,000 lines of code. Some bugs are easy to avoid or catch with tests and metrics. Others are very good at hiding. This is a sequel to Bugs that broke driving https://blog.comma.ai/driving-bugs/ with more Machine Learning related bugs. Some of these bugs affected correctness, others slowed things down or used up extra memory. They all made training or inference worse. Looking back at these bugs helps us understand where our code tends to break and how to catch problems earlier. Output layers need to be run in high precision Permalink output-layers-need-to-be-run-in-high-precision September 2026. The world model is a diffusion transformer DiT which predicts the next frame and an “expert” driving plan. During inference, the model runs in mixed precision bf16 fp8 nvfp4 : - Transformer MLP linears use NVFP4 most FLOPs - Attention projections, cached attention Q/K/V, and the KV cache use FP8 saves VRAM - Other layers use BF16 not too many FLOPs - Norm reductions, attention scores, and sinusoidal timestep features use FP32 standard safe mixed precision In particular, the plan output layers ran in BF16. Around 30 m/s, BF16 can only represent speeds in steps of 0.125 m/s. The acceleration calculation from the plans subtracts nearby speed predictions and divides by a short time interval , magnifying the rounding errors: The fix is to promote the plan head and its inputs to FP32 before running it, Casting the finished plan to FP32 just gives the staircase more decimal places. +plan head.float -plan = plan head features .float +plan = plan head features.float ConvNeXt FP16 Permalink convnext-fp16 August 2026. The driving model’s vision encoder is a ConvNeXt, with a learned multiplier on each residual branch: x + gamma f x . At runtime, it runs in FP16. The pretrained checkpoint we used https://huggingface.co/timm/convnext xxlarge.clip laion2b soup we pick had a value of almost 54 at stages 2 .blocks 29 . After fine-tuning, some inputs push the residual past FP16’s 65,504 limit leading to inf . The annoying thing is that this only triggered in some seemingly random cases, and didn’t happen in the various checks we run during model exporting to FP16. The workaround was to divide that stage’s residual stream by four and carry the scale through its biases, multipliers, and LayerNorm epsilons: stage = encoder.stages 2 +stage.downsample 1 .weight.div 4 +stage.downsample 1 .bias.div 4 +for block in stage.blocks: + block.conv dw.bias.div 4 + block.gamma.div 4 + block.norm.eps /= 16 +encoder.stages 3 .downsample 0 .eps /= 16 Missing desire targets Permalink missing-desire-targets July 2026. The driving model predicts “driving desire”, like turning or changing lanes. For a while, some training data had no desire targets: we use NaNs to distinguish missing targets, and our loss helpers normally handle this by masking those targets out. But the desire loss took an argmax first, and did not have the NaN handling. python import torch import torch.nn.functional as F target = torch.full 1, 8 , torch.nan target.argmax -1 tensor 0 F.cross entropy torch.zeros 1, 8 , target.argmax -1 tensor 2.0794 All those NaNs became class 0 , “no desire”, with a perfectly normal loss. The fix replaces NaNs with zeros while keeping a mask, then uses that mask to ignore missing targets: python def num from nan target : valid = ~torch.isnan target return target.masked fill ~valid, 0 .detach , valid +target, valid = num from nan target loss = F.cross entropy logits, target.argmax -1 , reduction="none" +loss = valid.all -1 loss Redundant biases drifted apart across distributed ranks Permalink redundant-biases-drifted-apart-across-distributed-ranks April 2026. The driving model has many nn.Linear layers. During training, we use torch.compile , BF16 autocast and Distributed Data Parallel DDP as “acceleration” recipes. Sometimes, harmless redundant biases are written in the architecture: a = nn.Linear 8, 8192 b = nn.Linear 4, 8192 y = a x + b x :, :4 Note that the 8/4 is just illustrative, and keeps the weight gradients different while the bias gradients are identical. a.bias and b.bias will be merged by the compiler into one bias, so they get the same gradient. The compiler went one step further: it made both .grad tensors share the same memory. DDP synchronized them correctly, then clip grad norm used its foreach implementation to scale that shared buffer from two CUDA blocks at once. Depending on timing, an entry could be multiplied by the clipping coefficient once or twice, giving each rank a different update. In one recorded run, rank 0 clipped all 8,192 entries once; rank 1 clipped 384 of them twice. The drift can reach a magnitude of 0.0056 between ranks after only 50 SGD steps, while the weight matrices stayed identical. Both biases receive the same faulty update within a rank, their sum drifts too, which can change the model’s outputs. DDP synchronizes gradients during training, so it doesn’t repair those parameter differences: the GPUs are now training different copies of the model. We worked around it by removing the redundant biases. This was fixed in PyTorch 2.13.0 https://github.com/pytorch/pytorch/blob/v2.13.0/torch/ inductor/fx passes/joint graph.py L801-L804 as a side effect of a compiler optimization change https://github.com/pytorch/pytorch/commit/6f94c6e16e1bf562f8f5514953cf9b8d47d07cdd . It preserved FP32 → BF16 → FP32 cast chains so smaller intermediate tensors could still be materialized: in our backward graph, that happened to keep the two bias gradients in separate FP32 buffers. Note: The foreach kernel itself still assumes the gradient buffers don’t overlap… DDP random seed bug Permalink ddp-random-seed-bug August 2026. Torchtitan https://github.com/pytorch/torchtitan uses the same seed across DDP replicate , this is correct because weight initialization should match across replicate ranks. But torchtitan then keeps that rank identical across during for training. With ordinary tensors not DTensors , different ranks could get identical dropout masks and diffusion noise. The fix is to reseed each data-parallel rank after initialization essentially : torch.manual seed seed model.init weights +torch.manual seed seed + dp rank train model Flux does this after initialization https://github.com/pytorch/torchtitan/blob/68c97b0c54b0d853fefefab4ecf9e41e6a82a6b7/torchtitan/models/flux/trainer.py L38-L50 , setting distinct seed mesh dims= "cp", "dp shard", "dp replicate" . AllNorm Permalink allnorm June 2021 to April 2026. Normalization layers are widely used in Machine Learning. BatchNorm normalizes each channel using the mean and variance across the batch and spatial dimensions. During training, it uses the current batch statistics and updates running averages. During inference, it uses those running averages. The driving models are multi-task learners, with some tasks being more sparse than others, for example: hard brake events. One failure mode we encountered was when a channel specializes in a rare feature. Through batches without that feature, the channel stays quiet and its BatchNorm running variance shrinks. When validation activates that channel again, it gets divided by a stored standard deviation that is too small, blowing up the outputs. Training can still look fine because its statistics come from the current batch: when the feature returns, the denominator grows with it. LayerNorm fixes this, but needs per input statistics at inference. On device, we want normalization folded into convolutions. Recalibrating BatchNorm running statistics offline also fixes this, but adds overhead and another training pass. Same for lowering momentum which needed extra tuning. The best fix we came up with was pooling the whole batch, channels, and spatial dimensions into one set of statistics. We called this AllNorm. python class AllNorm nn.Module : def init self : super . init self.bn = nn.BatchNorm1d 1 def forward self, x : return self.bn x.reshape -1, 1 .reshape as x All channels share statistics, making it robust to this failure, and keeping the ability to fold the statistics into the convolutions. Note: With the extra compute that the chestnut https://blog.comma.ai/chestnut/ gives, we can now afford using LayerNorm and not deal with this Gigashuffle’s reusable CPU buffers Permalink gigashuffles-reusable-cpu-buffers July 2026. Gigashuffle https://github.com/commaai/gigashuffle returns tensors backed by reusable CPU buffers. Each reader allocates these once and fills them again for every batch: Once per reader: reader buffer i key = torch.empty shape, dtype=dtype .share memory For every batch: torch.index select shuffle buffer i key , 0, indices, out=reader buffer i key Normally, each batch is copied to the GPU before the next read. Our gradient accumulation loop collected several microbatches first, so later reads overwrote batches already in the list. One way to fix this is to clone both inputs and targets before fetching the next microbatch: for in range grad accum : inputs, targets = next loader + inputs = {k: v.clone for k, v in inputs.items } + targets = {k: v.clone for k, v in targets.items } microbatches.append inputs, targets FSDP2 init bugs Permalink fsdp2-init-bugs April 2025. With FSDP2, we construct the model on meta , shard it, then use to empty to allocate the weights. Initialization has to run after allocation: not defining an init function for a parameter initializes it to the arbitrary memory value it received during allocation. python import torch with torch.device "meta" : ... model = torch.nn.Module ... model.weight = torch.nn.Parameter torch.ones 4 = model.to empty device="cpu" model.weight.detach tensor 3.7189e-38, 0.0000e+00, -6.3894e-37, 4.5160e-41 No clear fix here but to be paranoid about not forgetting init methods NCCL timeout increase didn’t do anything Permalink nccl-timeout-increase-didnt-do-anything July 2026. Our RL training runs collect rollouts from the cluster. If the cluster is busy, some ranks can wait for data longer than PyTorch’s default NCCL timeout of 10 minutes https://docs.pytorch.org/docs/2.14/distributed.html torch.distributed.init process group . To work around this, we added a barrier with a timeout of one hour before each training step, thinking we have fixed it. But that timeout was passed around, only to never be used: opts = BarrierOptions opts.timeout = timeout opts.asyncOp = async op work = group.barrier opts=opts The C++ dispatcher passed it along, but NCCL implemented the barrier as an all-reduce and only forwarded asyncOp : // C++ dispatcher BarrierOptions opts; opts.timeout = std::chrono::milliseconds timeout ; opts.asyncOp = asyncOp; backend- barrier opts ; // ProcessGroupNCCL::barrier AllreduceOptions arOpts; arOpts.asyncOp = opts.asyncOp; auto work = allreduce impl barrierTensor, "nccl:all reduce barrier", arOpts ; // opts.timeout goes nowhere. That all-reduce got its deadline from a different options object: the process group’s configuration. The watchdog checked that deadline; our barrier timeout=... argument never entered the calculation: // Creating the collective's work: options belongs to the process group. assignTimeoutToWork work, options ; // Inside assignTimeoutToWork work, option , omitting timeout extensions: work- opTimeout = option- timeout; // Watchdog calls work.checkTimeout with no timeout override: if timeElapsed = opTimeout { // Report a collective timeout. } The fix was to give the wait for data its own Gloo group, with the timeout set on that group: + Once at startup, on every rank: +ready = dist.new group backend="gloo", timeout=timedelta hours=1 Before each training step: -dist.barrier timeout=timedelta hours=1 +dist.barrier group=ready Memory leak with AdamW’s foreach updates on DTensors Permalink memory-leak-with-adamws-foreach-updates-on-dtensors March 2026. AdamW batches updates into calls like torch. foreach mul params, 1 - lr weight decay . FSDP2 exposes its parameters as DTensors, so those calls go through DTensor’s dispatcher to work out how the operation applies to the shards. For foreach’s tensor lists, the dispatcher built a fresh OpSchema on every call, then constructed a cache key from the tensors’ shape and sharding metadata. In the C++ key builder, PyTuple Pack increments its arguments’ reference counts, but .release stopped the C++ wrapper from decrementing its own reference afterward. That left an extra reference to the argument tuple on every call, so the tuples and their metadata kept piling up in CPU RAM. The fix https://github.com/pytorch/pytorch/pull/176010 was to simply not release: // torch/csrc/autograd/python variable.cpp // Inside DTensor OpSchema recompute comparison key impl simplified : py::tuple args to hash tup args to hash.size ; // ... fill the tuple with argument metadata ... comparison key = PyTuple Pack 2, op, - args to hash tup.release .ptr ; + args to hash tup.ptr ; The bug affected PyTorch 2.10.0 through 2.12.1; the same fix shipped via a later PR https://github.com/pytorch/pytorch/pull/181792 in 2.13.0. CUDA cache allocation Permalink cuda-cache-allocation February 2026. When doing inference in our distributed cluster, we use an inference server with dynamic batching: triton inference server https://github.com/triton-inference-server . We usually benchmark at the largest batch size in isolation, and assume this will be the max VRAM the model will use. But dynamic batching reserved more VRAM, and we didn’t know why. We traced it to how PyTorch caches freed GPU allocations, so the order of batch sizes matters. With its default native allocator, this takes just one tensor at a time: python import torch def reserved after batch sizes : torch.cuda.empty cache Reset between experiments. for bs in batch sizes: x = torch.empty bs, 1024, 1024, device="cuda", dtype=torch.float32 del x return torch.cuda.memory reserved // 2 20 print reserved after 3, 4 28 MiB print reserved after 4, 3, 4 16 MiB Batch size 3 leaves a cached 12 MiB block, batch size 4 needs 16 MiB, so the allocator keeps both. Starting with 4 lets batch 3 use 12 MiB of the larger 16 MiB block. We flipped the warmup order to start with the largest batch: -for bs in range 1, max batch size + 1 : +for bs in range max batch size, 0, -1 : warmup bs CUDA graphs don’t capture Adam’s CPU work Permalink cuda-graphs-dont-capture-adams-cpu-work June 2022. When training with CUDA graphs, we used to capture the whole training step, optimizer step included. CUDA graphs replay only GPU work, so CPU code inside the capture runs just once By default, PyTorch’s eager Adam increments its step counter and computes its bias corrections, 1 - beta1 step and 1 - beta2 step , on the CPU: torch/optim/adam.py — Adam. init group simplified state = self.state p if len state == 0: Initialize this parameter's optimizer state once. state "step" = torch.zeros , device=p.device if group "capturable" or group "fused" else torch.tensor 0.0, device="cpu" cpu? ? Back then, replay kept updating the GPU moment estimates and weights with those frozen corrections, silently reusing the value of step from the capture-time value. We avoided this by running the optimizer outside the graph. PyTorch 1.12.0 later added Adam ..., capturable=True , putting the step counter and bias-correction math on the GPU so replay advances them too. Today, manual capture with capturable=False raises an error https://github.com/pytorch/pytorch/blob/v2.14.0/torch/optim/optimizer.py L440-L449 . Looking back Permalink looking-back The recurring themes were precision, leaked or reused memory, and distributed training. Some bugs were hard to spot but preventable with better code practices. Others became obvious once we visualized model outputs during inference or inspected model state during training. Come fix the bugs that are still hiding https://comma.ai/jobs