My fact-checker said CONFIRMED about a group that doesn't exist

A developer built a crypto fact-checking tool that queries Nansen's API to label claims about wallet cohorts as CONFIRMED, OVERSTATED, CONTRADICTED, or UNVERIFIABLE, but it returned CONFIRMED for a group of wallets that does not exist. The bug stems from a presence() check that was enforced in the buying/selling branch but omitted from the holding branch, so when the holders call failed and the seven-day flow reported zero wallets, the decision ladder fell through to CONFIRMED. The developer had already documented the sparse Whale label in the repo's JUDGE.md and DX-REPORT.md, noting zero labelled whales on WETH despite 38 holders.

I built a tool that fact-checks crypto claims. You paste a sentence like "Whales are holding $XYZ" and it plans the on-chain calls that claim needs, runs them against Nansen https://nansen.ai 's API, and answers CONFIRMED / OVERSTATED / CONTRADICTED / UNVERIFIABLE with the numbers that decided it. Last week it answered CONFIRMED about a group of wallets that does not exist. Not "returned a weak signal." Not "was slightly off." It printed the word that means this claim is true , next to a cohort the data source tags exactly zero members of. Here is how a tool whose entire purpose is refusing to overstate ended up overstating, and why the README had already warned me. Three limitations ship in the repo's JUDGE.md . The second one says this, verbatim: Nansen's Whale label is sparse. A post's "whale" is often a big wallet Nansen does not tag; when no Whale-labelled wallet exists in the token the tool says UNVERIFIABLE rather than pretending a CONTRADICTED. I wrote that line early, and I wrote it because it is true of the data . My API notes from the build docs/DX-REPORT.md record the measurement it came from: tgm/flow-intelligence 1d on WETH: whale wallet count = 0 . tgm/holders label type=whale on the same token: 38 holders. Zero labelled whales on wrapped Ether. Not an edge case — the label is genuinely sparse, and the flow columns that report it are DEX-only. "No wallets of this class" is an ordinary, frequent answer from this API, not a rare one. So I knew. I documented it. And then I enforced it in exactly one of the two places it mattered. Claims come in three shapes: buying , selling , holding . The first two go down one branch, holding goes down another. Both need the same question answered first — does this class of wallet exist in this token at all? That question has a function: / Does the subject class exist in this token at all any window, any endpoint ? / export function presence e: Evidence, cls: "smart trader" | "whale" : boolean { return e.flow1d?. cls ?.wallets ?? 0 0 || e.flow7d?. cls ?.wallets ?? 0 0 || e.holders?.count ?? 0 0 || cls === "smart trader" && e.table?.traders ?? 0 0 || e.named ? e.named.buyRows + e.named.sellRows 0 : false ; } The flow branch calls it, and does the right thing: if presence e, cls return R "UNVERIFIABLE", "U-NOCLASS", Nansen tags no wallet as ${who} in this token 24 h, 7 d, holders — + the wallet in the post is not one Nansen labels , ; return R "CONTRADICTED", "C-NOBODY", / ... / ; The holding branch never called it. Not once. Here is what it did instead — read it as a ladder, because that is how it fails: js function decideHolding claim, e, rules, T7, R { const who = subjectName claim.subject ; // "Whales" | "Smart Money" const cls = subjectClass claim.subject ; // "whale" | "smart trader" const h = e.holders; const net7 = e.flow7d?. cls ?.net ?? null; if h && net7 == null return R "UNVERIFIABLE", "U-HOLD", / no data at all / ; // ↓ no presence check here ↓ if h && h.count < rules.minHolders return R "OVERSTATED", "O-HOLDERS", / ... / ; if h && net7 = null && net7 <= -T7 && h.delta7d < 0 return R "CONTRADICTED", "C-EXIT", / ... / ; if h && h.delta7d < 0 || net7 = null && net7 <= -T7 return R "OVERSTATED", "O-TRIM", / ... / ; return R "CONFIRMED", "A-HOLD", / "balances are not shrinking" / ; } Zero holders. tgm/holders answers with an empty page, so h.count === 0 . The first rung catches it — 0 < 5 — and returns OVERSTATED , whose UI copy means partly true . Partly true about nobody. Bad, but at least it is hedging. The holders call failed. Now h is null , and the seven-day flow answered with wallets: 0 . Look at the ladder again: every remaining rung is guarded by h && . O-HOLDERS needs h . C-EXIT needs h . O-TRIM needs h or a net flow below the negative threshold — and the net flow is 0 , which is not below anything. So the claim falls all the way through and lands on the last line: { label: "CONFIRMED", rule: "A-HOLD", presence: false, reasons: "7 d Whales net flow $0 threshold $5K ", "Whales balances are not shrinking" } "Whales balances are not shrinking." Technically unfalsifiable and completely true, in the way that "all the unicorns in my garage are healthy" is true. There are no whales. Nothing is shrinking because there is nothing. And the tool rendered that as CONFIRMED , the strongest word it owns, on the one screen a reader actually looks at. The path needs no exotic input. One failed HTTP call to a sparse endpoint, on a token whose class is empty — which, per my own notes, is WETH. if presence e, cls return R "UNVERIFIABLE", "U-NOCLASS", / same message the flow path uses / ; One line, plus a comment, plus 45 lines of tests. The whole commit is +51 . I found the first case because I ran the source past an outside model, which spotted the zero-holders rung. Verifying its claim is what turned up the second: I wrote a throwaway test with holders: null to see what the other branch did, and got CONFIRMED back. The review found the hedge; checking the review found the lie. All 13 recorded fixtures still reproduce with byte-identical sha256 evidence hashes after the change, which is the thing that let me ship a one-line edit to the verdict engine four days before a deadline without flinching. A limitation stated in prose is not a constraint. I had the right belief, written in the right file, in public, in the artifact judges read. It did nothing. Prose cannot fail a build. An invariant enforced in one branch is enforced in zero branches. presence existed. It was correct. It was even called — just not on every path that needed it. A shared helper that callers must remember to call is a convention, and conventions decay the moment someone adds a fourth claim type. The dangerous bug was in the fallback, not the happy path. Every test I had written covered tokens with data, because that is what the demo uses and what the fixtures recorded. The failure mode lived where the data was absent — h === null , wallets: 0 , a call that didn't answer — and absence is exactly what nobody writes fixtures for. If you have a decision function with a default return at the bottom, the question worth asking is not "is this the right default" but "what is the emptiest input that can reach this line?" Sparse-by-design data is a first-class input. I treated "no whales" as degenerate. It is not. It is Wednesday. UNVERIFIABLE , not an investigation of who that wallet is. flow-intelligence vs smart-money/netflow . I surface both rather than reconcile them; docs/DX-REPORT.md has the measurement. For the numbers people ask for: 352 tests, 100% statement/branch/function/line coverage on the engine, 13 fixtures that replay offline with zero network calls, cold p50 4.0 s / p95 5.0 s, ~10 API credits per verdict, 0 of 154 live calls failed in the benchmark. Details in docs/BENCH.md . The tool is at https://rebuttal.edycu.dev https://rebuttal.edycu.dev and the code is at https://github.com/edycutjong/rebuttal https://github.com/edycutjong/rebuttal — the commit in this post is fedb50b , and the test that pins both failure paths is packages/core/test/review4.test.ts . If you have a verdict function with a return at the bottom of a ladder, go and check what the emptiest possible input does to it. That is the whole article.