Did FP8 make the model dumber? A per-prompt regression check for quantized serving

An engineer at Jahn AI ran a per-prompt regression check before recommending FP8 quantization for Qwen3-8B serving on an RTX PRO 6000 Blackwell, which delivered a 1.5x throughput gain (1,725 to 2,597 tok/s at concurrency 32 with vLLM). The check compares greedy-decoding outputs between BF16 and FP8 across 20 prompts, using exact equality and difflib similarity to flag material differences, and passed for the tested workload. The engineer notes the method validates a prompt profile, not the model generally, and that a kernel assertion in the default FP8 path on sm_120 required a workaround.

FP8 gave us a clean 1.5x on Qwen3-8B serving throughput on an RTX PRO 6000 Blackwell 1,725 to 2,597 tok/s at concurrency 32, vLLM . The uncomfortable question is always the same: did the model get dumber. This post is the exact check we ran before recommending the switch, with numbers, so you can run the same one. Standard benchmarks MMLU and friends are noisy instruments for quantization deltas at 8B scale. Score movement inside the error bars tells you nothing about whether YOUR prompts changed behavior. What you actually want to know is narrower: on the workload you serve, does the FP8 checkpoint produce materially different outputs than BF16, and are any of the differences wrong. That is answerable directly, cheaply, and per prompt. Both configurations run the same fixed workload: 20 prompts covering reasoning, code, summarization, translation, extraction, classification, math, and instruction following. Greedy decoding, temperature 0, 256-token cap, streamed. Greedy matters: it removes sampling noise, so any output difference is attributable to the numerics. Then a three-stage comparison: outputs bf16 i == outputs fp8 i . Anything identical is settled. difflib.SequenceMatcher.ratio sorts near-identical wording drift from real divergence.The core loop is small: python import difflib, json bf16 = json.load open "vllm bf16 conc1.texts.json" fp8 = json.load open "vllm fp8 conc1.texts.json" for i, a, b in enumerate zip bf16, fp8 : if a == b: print i, "identical" continue r = difflib.SequenceMatcher None, a, b .ratio print i, f"similarity {r:.3f}" non-identical pairs go to side-by-side review That last line is the acceptance bar. Minor stylistic wobble is expected from a numerics change; a flipped fact is a blocker. This profile passed, so the 1.5x was free for this workload. Two honest caveats. First, this validates a prompt profile, not the model in general: different domains, longer contexts, or sampled decoding need their own pass. Second, greedy-decoding equality is a strict signal but not a complete one; if you serve with sampling, run the review stage on sampled pairs too and expect more benign divergence. Getting FP8 to run at all on workstation-class Blackwell sm 120 required routing around a kernel assertion in the default FP8 path. If you are on the same silicon and vLLM refuses to load the FP8 checkpoint, that is a known class of problem rather than something wrong with your setup. The full report this check belongs to, with the raw CSVs, environment manifest, and the one-command reproduction script, is here: https://conatus.jahn.ai/ai-engineering/sample-report https://conatus.jahn.ai/ai-engineering/sample-report