A dev.to tap pulled in a headline made entirely of dots. Not a typo, not a mangled string – an actual title that read . .. . ... . .... . .... . ... .
, like someone had faxed us Morse code and lost the translation key.
Our LLM final-scorer looked at that headline next to a batch of normal ones and ranked it first. Sixty-five points against the mid-40s the real headlines pulled. Auto-resolve did what auto-resolve does: it promoted the top-ranked candidate. A full canonical_blog
run kicked off. GPU time burned. The dots made it all the way to the last QA gate before something finally said no.
That’s the whole story in one paragraph, and it’s the story we want to tell, because it’s the cleanest possible demonstration of a problem that’s easy to miss until it costs you a run: LLM rankers score whatever you hand them. They don’t know what a headline is supposed to look like. They know how to rank the options in front of them relative to each other, and if every option in the batch is garbage, the ranker will still confidently pick a winner.
Why the ranker didn’t blink
This isn’t a knock on the model. Ranking is a relative task – give it five headlines and it tells you which one wins, not whether any of them deserve to be in the race. We wrote about this exact failure mode from a different angle in the metaphor of topics behind glass doors: the pipeline was built to trust what gets handed to the scoring stage, and nobody was checking the glass for cracks before the topic walked through.
The fix wasn’t “make the ranker smarter.” The fix was: stop garbage from reaching the ranker at all. That’s the whole philosophy behind what we now call the topic_sanity_rejected
gate – a pre-scoring checkpoint that looks at a candidate topic and asks dumb, cheap, obvious questions before anything expensive happens. Is this title actually a sentence? Is it truncated? Is it dots?
This lines up with a piece of advice that’s been circulating in alignment research circles: the highest-leverage thing you can do for a pipeline isn’t a clever new metric, it’s a quick sanity check run before you trust the output of anything downstream. The advice is aimed at junior researchers reviewing their own experiments, but it applies just as well to a topic pipeline reviewing its own inputs. Cheap checks, run early, catch expensive mistakes late.
What the gate actually catches
Once we shipped the gate, it started earning its keep immediately – arguably too well. In the seven days after deployment, tap_builtin_topic_source:topic_sanity_rejected
became the single highest-volume alert in the entire system: 294 rejections. And the gate was right every single time. Nothing leaked past it.
The rejection reasons tell you exactly what kind of noise was flowing into the pipeline before this existed:
truncated_title: 'Benchmarking 15 "E-Waste" GPUs with'
– cut off mid-sentence, no idea what happens to those GPUs.truncated_title: 'The Future Of AI Is'
– is what, exactly?truncated_title: 'Reading Anthropic's "When AI Builds Itself" Changed How I Think About AI and'
– a genuinely interesting headline, decapitated right before the payoff.
Every one of these would have gone into a scoring batch, competed against real headlines, and possibly won – the same way the dots won. The gate stops them at ingest, before they cost anything.
The bug behind the volume
294 rejections in a week is a lot, and a number that high told us the gate wasn’t just catching rare edge cases – it was catching a systemic bug upstream. We went looking, and found it in rewrite_as_blog_topic()
, a function shared by all three of our topic sources: dev.to, Hacker News, and web search.
Back in April, someone had added a regex to strip trailing author bylines – the kind of thing where a scraped title comes back as "...New Rendering Pipeline Ships Scott Rose"
and you want to lop off the glued-on name. Reasonable fix, at the time. The problem is that regex only checks one thing: is the character right before the presumed name lowercase? That’s it. That’s the entire signal it uses to decide “this is a byline” versus “this is just the last two or three words of an ordinary Title Case headline.”
Which means a headline like “The Future Of AI Is Unwritten” gets its ending mistaken for an author’s name and stripped, and you’re left with “The Future Of AI Is” – a fragment that reads like a sentence had a stroke. The regex couldn’t tell the difference between “Scott Rose” and “Unwritten,” because it was never looking at the actual words. It was looking at capitalization patterns and letter casing at a word boundary, and that heuristic breaks constantly on ordinary English headlines.
That’s the real lesson from the volume spike: a high rejection rate on a sanity gate isn’t necessarily the gate misbehaving. Sometimes it’s the gate doing exactly its job and surfacing a bug that’s been quietly mangling data for months. The gate didn’t just prevent damage – it pointed a finger at where the damage was coming from.
A second failure mode: reasoning tokens leaking into titles
The truncation bug wasn’t the only thing feeding the sanity gate. We also found tasks getting rejected despite passing every other QA check, and traced it to models like glm-4.7-5090
and gemma4:31b
leaking their internal reasoning tokens straight into generated output. A title that should have been clean prose was showing up with chain-of-thought fragments stitched in – the kind of thing that looks like garbage to a human and gets correctly flagged as garbage by the gate, but for a completely different root cause than the byline regex.
We shipped strip_reasoning_artifacts
in our thinking-models handling code to scrub that leakage before it hits any downstream consumer. Two unrelated bugs, one shared symptom, one shared gate catching both. That’s the value of a sanity check sitting at a chokepoint rather than one bolted onto a specific source: it doesn’t care why the input is broken, only that it is.
The dedup bug nobody asked for
Here’s where it got annoying instead of interesting. With 294 rejections a week, and each rejection correctly firing an alert, our Discord channel turned into a wall of near-identical notifications. The alert emitter was doing the right thing – it generates a stable dedup_key
scoped to topic-sanity-ingest:{niche}:{source}
, and the findings router correctly derives a fingerprint from that key. But the brain dispatcher, further down the chain, threw that fingerprint away and recomputed its own from the raw message body. Since every rejection has slightly different truncated text in the body, every single one looked “new” to the dispatcher. The dedup logic existed. It just never got used.
The tempting fix here is obvious: turn the volume down. Set the alert to log_only
and let it quietly accumulate in the database instead of pinging a channel two hundred times a week. We tried exactly that, and got corrected immediately: don’t suppress useful signal. If something is worth knowing about, silencing it because it’s noisy is the wrong axis to optimize. The right fix is to make the noise correct – repair the dedup bug so one underlying problem produces one alert, not force the alert into silence because the plumbing behind it is broken.
That’s a distinction worth sitting with. A sanity check that’s technically correct but operationally unbearable will get muted by someone eventually, and then it stops being a sanity check at all. It becomes a log line nobody reads. The fix has to happen in the routing, not in the reader’s tolerance for pain.
Sanity checks are a category, not a feature
None of this is a novel idea in isolation. Software teams have run sanity testing after every release for decades, precisely because a full regression suite is expensive and a five-minute smoke test catches the obvious breakage before you waste the expensive cycle. Data teams run the same logic on rows and nulls before a report goes to a stakeholder – row counts, null scans, date validations, the stuff that “catches most issues early” without needing a model or a human in the loop. Even 3D rendering pipelines run a sanity check pass before submitting a job to the farm, flagging fatal errors before you burn render-node hours on a scene that was never going to finish.
Our topic pipeline needed the same discipline applied to text instead of pixels or rows. Is the title complete? Is it actual language? Did a regex eat half of it? Did a model leak its scratch-work into the output? None of these questions require intelligence to answer. They require a checkpoint that runs before the expensive stage, not after.
We’d already been circling this problem from other directions – retiring the Gen-1 TopicDiscovery orchestrator killed a pile of legacy logic that was quietly stalling without saying why, and the echo chamber piece covered a related failure where our own originality checks scored a near-duplicate topic as an advisory pass three times in a row and let a human operator catch what the system should have caught at proposal time. The pattern across all three incidents is the same: cheap, boring, unglamorous checks placed early save you from expensive, glamorous failures placed late.
Where this leaves the pipeline
The dots incident cost us a wasted canonical_blog
run and a moment of “wait, what did we just publish a draft of.” The topic_sanity_rejected
gate now sits between ingest and scoring across all three topic sources, and it’s fully deployed and verified live in production, catching contentless and malformed topics before they ever reach a ranker. The byline regex has a real fix in front of it now that we know exactly which character-casing assumption was wrong. The reasoning-token leakage has its own scrubber. The dedup bug has a name and an owner.
None of these fixes required a smarter model. They required us to stop assuming the inputs were clean and start checking. That’s the whole lesson, and it’s not a new one – it’s the same lesson every discipline that ships things under time pressure eventually relearns: cheap checks up front are how you stop paying for expensive mistakes at the end.