We run a self-hosted ~300B reasoning model in production. It writes macroeconomic desk reports in Azerbaijani and English.
Every so often, it stops.
Not crashes. Not refuses. Mid-sentence, a competent economic analysis turns into a fabricated Chinese news article. Or a software README, complete with pip install
instructions. Or a Persian name, repeated seven times. Or a wall of spreadsheet cells starting with #REF!
.
The user is watching this happen, token by token.
We call this decoding corruption*, and it is not hallucination. Hallucination is the model being wrong about the world. This is the model no longer producing an answer at all. There is a large literature on detecting the first one. There is almost nothing on the second, which is strange, because anyone who operates a self-hosted model has seen it.
This post is about what we learned building a detector for it. The headline lesson is not the one I expected going in.
Most hallucination detectors run after generation completes. For factuality that is fine. For this, it is not.
Two reasons. First, streaming: the user already watched the model derail. A verdict that arrives after the fact has failed. Second, cost: on a reasoning model, one corrupt generation burns minutes of GPU time. We measured a 13,200-character table echo that took 269 seconds to generate to completion. Every one of those seconds was wasted.
So the detector has to run inside the stream, cheap enough to be invisible next to decoding.
One pass over the character stream. A single state object ingests one character at a time and incrementally maintains every statistic anything downstream will need. Detectors are stateless reads over that shared state, evaluated only at checkpoints.
That constraint matters more than it sounds. It means adding a sixth detector costs zero additional passes. Everything is O(1) amortized per character with bounded memory, so a runaway generation is processed in constant space.
Five detectors sit on top:
Rules. Deterministic thresholds on the snapshot. Digit fraction, symbol density, longest identical-character run. Ships day-one protection with zero training, and every alarm carries a human-readable reason like numeric dump digit_frac=0.57
.
Character n-gram surprise. An online n-gram model trained on the stream's own clean prefix, scoring each incoming character's surprise before observing it. No pretraining, no access to the generator's logits. The stream explains itself. The signature is nice: repetition loops collapse surprise toward zero, while drift and regurgitation spike it.
Count-Min Sketch repetition. Character shingles counted in a 4×2048 sketch. Constant memory regardless of stream length. Reads the repeat rate, because a real loop hammers many shingles, not one.
Rolling SimHash. A 64-bit fingerprint over a sliding word window, compared against the fingerprint frozen at the end of the clean prefix. This is the one that catches the hardest class: fluent, same-script regurgitation that is invisible to character statistics but lands ~0.5 normalized Hamming away in fingerprint space.
Character entropy. Table dumps and degenerate loops flatten the distribution.
All five read from one state. Throughput is ~50,000 characters per second on a single CPU core, two to three orders of magnitude faster than the model produces text. No GPU anywhere.
Detection was never the bottleneck. Every configuration I tried caught every corrupt stream. TPR 1.000, immediately, without effort.
The entire difficulty was in not flagging clean text.
Our clean text is economic prose dense with figures, percentages, variable codes and markdown tables. It is, statistically, the most corruption-looking legitimate text you could ask for.
I started with noisy-OR fusion. It is the textbook rule for combining independent evidence:
p = 1 - Π(1 - sᵢ)
It produced a 42.9% false positive rate on clean production text.
Think about what that means operationally. Four out of every ten healthy desk reports killed and regenerated. Completely unusable.
Here is why it happens. Legitimate prose mildly excites several detectors at once. A digit-dense sentence nudges the rules tier. A formulaic passage nudges compressibility. A section header nudges the sketch. None of them is alarmed. But noisy-OR multiplies survival probabilities, so five detectors at 0.3 each produce a confident 0.83.
Real corruption does not look like five weak signals. It looks like one or two saturated signals. A cross-lingual drift does not gently raise five statistics; it slams the foreign-script fraction to the ceiling while the others barely move.
So we replaced the combination rule with strongest-signal dominance plus a small corroboration bonus when two or more detectors independently cross 0.5:
p = min(1, max(max(sᵢ), w·ℓ) + b·[#{i : sᵢ ≥ 0.5} ≥ 2])
Same five detectors. Same data. Same protocol.
FPR dropped from 0.429 to 0.024.
Then per-detector fixes took it to 0.000:
The lesson generalizes past our system: if you are building a multi-signal guard over stylistically varied text, your combination rule is probably a bigger deal than your detector roster.
Thresholds obvious in the abstract were wrong. I set the hard digit-fraction rule at 0.18, which felt generous. Real quantitative answers sustain digit fractions near 0.3. Moved it to 0.28. Separately, a markdown horizontal rule is twenty-plus identical characters, which tripped the run detector on every document with a ---
in it. Separator characters are now excluded.
Both fixes came from replaying the guard over the real corpus. I would argue that step is non-optional for any detector you intend to deploy.
Aborting is an integration hazard. We implement abort as an exception raised from inside the token sink. The first version tripped the HTTP circuit breaker, because the transport layer read an abort as an upstream failure. Three catches would have taken the entire assistant offline. The transport has to treat it as caller-side cancellation.
The protocol has three phases. During hold, the first 350 characters are buffered and inspected, and nothing is forwarded. A corrupt verdict here kills the request with zero characters leaked. This matters because corrupt-from-the-start is the dominant production mode. On a clean verdict, the buffer flushes, the self-calibrated baselines freeze, and the stream goes live with rechecks every 400 characters and two-hit hysteresis before aborting.
The cost is a one-time ~350-character delay before the first visible token, roughly a second at typical decode speeds. That is the trade: one second of latency against never showing a user a corrupted answer.
Setting the abort threshold by hand felt wrong, so we use split-conformal calibration. Take the final fused scores of n held-out clean streams, and set τ to the ⌈(1−α)(n+1)⌉-th smallest. That guarantees Pr[flag | clean] ≤ α
in finite samples under exchangeability.
The practical payoff: the operator's false-alarm budget α becomes the only meaningful knob in the system. You say "I will tolerate 2% false aborts" and the threshold follows.
We ran the finished guard backwards over 121 historical messages in the deployment's database, expecting a clean sweep as a sanity check.
It raised exactly two flags. Both were real corruptions that had shipped to users months earlier and nobody noticed: one desk report written entirely in Chinese, and one English answer that derailed into Chinese plus Python code and stopped mid-fragment.
Zero false alarms on the other 119.
So it is also an audit tool, which was not the plan.
Corruption that stays fluent, same-script, same-topic and statistically unremarkable is invisible to this. Subtly wrong but well-formed analysis needs grounding or factuality methods, and we handle that separately upstream.
Our positives are synthetic, generated by injecting the four corruption classes into clean text at controlled onsets. They are modeled on real derails and they catch every real historical case we have, but production traffic at scale may hold surface forms we have not modeled.
And the perfect scores are a statement about class separation, not difficulty. The observed corruption classes sit far from the clean manifold. I fully expect adversarially subtle corruption, like a slow drift below checkpoint resolution, to erode those margins.
The detector, the benchmark generator and the trained weights are open source: https://github.com/doofzoff/SIMURG
The detector registry takes a three-line protocol — a name, an evaluate(state)
that returns a score, and a set of reasons. You can add a detector without touching the sentinel, and the fusion layer picks up whatever is registered. If you are running a non-Latin-script deployment, the expected-script priors need flipping and I would genuinely like help with that.
If you self-host a model and have watched it derail: what did yours do? I am collecting failure modes, and the taxonomy has four classes right now but I do not believe that is the complete list.