The P100 has been doing silently noisy math in llama.cpp for years. Three lines fix it for free.

A developer discovered that the Tesla P100 GPU has been performing inaccurate floating-point arithmetic in llama.cpp for years due to a missing architecture check. A three-line patch that excludes the P100 from fast fp16 paths reduces median Kullback-Leibler divergence by 2300× and increases top-token agreement from 96.53% to 99.89%, with no performance cost. The fix is now available in community pull requests and an upstream issue has been filed.

The P100 has been doing silently noisy math in llama.cpp for years. Three lines fix it for free. llama.cpp's CUDA backend has a FAST FP16 AVAILABLE flag: "this GPU is fast at fp16, so do quality-sensitive math in fp16." The GTX 10-series sm 61 was exempted from it long ago. The Tesla P100 sm 60 never was — because GP100 is the one Pascal chip with fast fp16 hardware. Hardware can , therefore software did . Nobody measured what it cost. Measured against an fp32-arithmetic truth base Qwen3.6-27B Q6 K, wikitext-2, 2048 ctx, 32 chunks, KLD over full logit distributions : | median KLD vs truth | top-token agreement | | |---|---|---| | stock llama.cpp on P100 | 0.002298 | 96.53% | | 3-line patch | 0.000001 | 99.89% | That's ~2300× tighter, and roughly 1 in 29 next-token predictions were changing outright. The speed cost, benchmarked at pp8192/tg32 @ depth 8192 across three model classes 27B hybrid, 4B dense, 36B MoE : zero . Prefill inside noise on all three; decode +1.4% faster patched. The patch is merged in llama-cpp-turboquant PR 212 https://github.com/TheTom/llama-cpp-turboquant/pull/212 and open in buun-llama-cpp PR 80 https://github.com/spiritbuun/buun-llama-cpp/pull/80 . GGML upstream issue: ggml-org/llama.cpp 25593 https://github.com/ggml-org/llama.cpp/issues/25593 . ggml/src/ggml-cuda/common.cuh has three gates that all say the same thing: if defined FP16 AVAILABLE && CUDA ARCH = 610 // device macro ggml cuda highest compiled arch cc = 610 // fast fp16 available cc = GGML CUDA CC PASCAL && cc = 610 // fast fp16 hardware available sm 61 GTX 1070/1080, P40 is carved out of all three — someone measured or knew that chip's fp16 story and made the right call. sm 60 P100 sailed through all three gates because GP100 has genuine 2:1 fp16 hardware 18.7 TF . The fix extends the existing sm 61 idiom: && CUDA ARCH = 600 / && cc = 600 . Three lines. That's the whole patch. What those gates control on sm 60 verified in dispatch source, not headers : Flash-attention tile and vec kernels — all of attention math ran in half2 vector arithmetic. Yes, llama.cpp has real working FA on Pascal — a fact the Python world, where FA2 requires sm 80, mostly doesn't know. cuBLAS compute type — and this one's subtle: on sm 60, MMQ the int8 quantized matmul path is hard-disabled should use mmq returns false below the DP4A gate — GP100 missed the DP4A instruction by one minor arch version . So ALL quantized-weight prefill runs dequantize + cuBLAS GEMM, and this flag chose fp16 GEMM. Your entire weight path, in half precision, silently. mmvf — f16-weight matrix-vector arithmetic type. Not by reading code — by measuring GPUs against each other. While validating a KV-cache codec buun's TCQ/VBR work across a P100 rig and a 3090, the same model, same weights, same test corpus kept producing systematically different KLD floors. Chasing that difference through a controlled decomposition panel KV storage precision × attention arithmetic × weight-path arithmetic, each isolated pointed at platform arithmetic , not the codec. The codec was innocent. The platform wasn't. Key methodology point for anyone reproducing this class of bug: within-build comparisons cannot see it. If your truth base and your test run share the same fp16 arithmetic, the error is common-mode and cancels. Every rung of a KV-quant ladder measured on stock sm 60 looks clean relative to its own f16 baseline while the whole ladder sits ~0.005 median KLD away from the fp32 answer with a 95.0% same-top floor — the weight-path error is ~2× the attention-path error . You must generate the reference logits with fp32 arithmetic to see it. sm 60: measured, fixed. sm 61: was already exempt nothing changes . sm 62 Jetson TX2 : same gate, same silicon family, unmeasured — deliberately NOT patched. Measure before you carve. Everything Volta+: unaffected by this patch. Independently verified by the turboquant maintainer pre-merge: these three gates are the only 600-vs-610 distinction in the CUDA tree, and a Blackwell build produced bit-identical PPL. Whether other arches have their own distance-to-fp32-truth is a separate, open research question. Different flag, different kernels, different measurement. Don't extrapolate this bug to your 3090. P100s are flooding the used market at ~$80 shipped while the DRAM crisis prices everything else into orbit. That's 16GB of HBM2 at 732 GB/s per card. The market had it half-right: sm 61 P40s clean math all along soared to ~$300 while the P100 floundered — the price gap was partly pricing in a software bug. Post-patch, the P100 does fp32-clean inference at fp32 speed that was always there; the "fast" fp16 path it loses was buying nothing real prefill is bound by cuBLAS GEMM and memory bandwidth, not the fp16 vector path . - Build stock llama.cpp, -DCMAKE CUDA ARCHITECTURES=60 . - Generate truth base: llama-perplexity --kl-divergence-base out.kld with fp32 KV, FA off, on a patched build fp32 arithmetic — this is the step everyone skips. - Score stock vs patched runs against that base with --kl-divergence , 2048 ctx, 32 chunks. - Speed: llama-bench pp8192/tg32 @ d8192, patched vs unpatched, same host, nothing else on the GPUs. Full panel design, predictions-logged-before-runs, and receipts: https://github.com/apollo-mg/Project-Apollo/blob/1a41c11439e6661c84c83bec333ccdf964663da6/data/Apollo%20Docs/Pascal FAST FP16 Carveout Results.md https://github.com/apollo-mg/Project-Apollo/blob/1a41c11439e6661c84c83bec333ccdf964663da6/data/Apollo%20Docs/Pascal FAST FP16 Carveout Results.md https://github.com/apollo-mg/Project-Apollo/tree/main/data/carveout panel https://github.com/apollo-mg/Project-Apollo/tree/main/data/carveout panel . - buun TCQ/VBR codec — the cross-GPU codec validation that surfaced the discrepancy, and the first fork PR review. - TheTom llama-cpp-turboquant — fast merge with independent cross-arch verification. - Found and isolated by running Fable 5 through my custom P/ReAct/R agent loop. It wrote the scripts, the hardware provided the receipts.