{"slug": "my-fact-checker-said-confirmed-about-a-group-that-doesn-t-exist", "title": "My fact-checker said CONFIRMED about a group that doesn't exist", "summary": "A developer built a crypto fact-checking tool that queries Nansen's API to label claims about wallet cohorts as CONFIRMED, OVERSTATED, CONTRADICTED, or UNVERIFIABLE, but it returned CONFIRMED for a group of wallets that does not exist. The bug stems from a presence() check that was enforced in the buying/selling branch but omitted from the holding branch, so when the holders call failed and the seven-day flow reported zero wallets, the decision ladder fell through to CONFIRMED. The developer had already documented the sparse Whale label in the repo's JUDGE.md and DX-REPORT.md, noting zero labelled whales on WETH despite 38 holders.", "body_md": "I built a tool that fact-checks crypto claims. You paste a sentence like *\"Whales are holding $XYZ\"* and it plans the on-chain calls that claim needs, runs them against [Nansen](https://nansen.ai)'s API, and answers **CONFIRMED / OVERSTATED / CONTRADICTED / UNVERIFIABLE** with the numbers that decided it.\n\nLast week it answered **CONFIRMED** about a group of wallets that does not exist.\n\nNot \"returned a weak signal.\" Not \"was slightly off.\" It printed the word that means *this claim is true*, next to a cohort the data source tags exactly zero members of. Here is how a tool whose entire purpose is refusing to overstate ended up overstating, and why the README had already warned me.\n\nThree limitations ship in the repo's `JUDGE.md`. The second one says this, verbatim:\n\n**Nansen's Whale label is sparse.** A post's \"whale\" is often a big wallet Nansen does not tag; when no Whale-labelled wallet exists in the token the tool says UNVERIFIABLE rather than pretending a CONTRADICTED.\n\nI wrote that line early, and I wrote it because it is *true of the data*. My API notes from the build (`docs/DX-REPORT.md`) record the measurement it came from:\n\n`tgm/flow-intelligence` 1d on WETH: `whale_wallet_count = 0`. `tgm/holders label_type=whale` on the same token: 38 holders.\n\nZero labelled whales on wrapped Ether. Not an edge case — the label is genuinely sparse, and the flow columns that report it are DEX-only. \"No wallets of this class\" is an ordinary, frequent answer from this API, not a rare one.\n\nSo I knew. I documented it. And then I enforced it in exactly one of the two places it mattered.\n\nClaims come in three shapes: *buying*, *selling*, *holding*. The first two go down one branch, holding goes down another. Both need the same question answered first — **does this class of wallet exist in this token at all?** That question has a function:\n\n```\n/** Does the subject class exist in this token at all (any window, any endpoint)? */\nexport function presence(e: Evidence, cls: \"smart_trader\" | \"whale\"): boolean {\n  return (\n    (e.flow1d?.[cls]?.wallets ?? 0) > 0 ||\n    (e.flow7d?.[cls]?.wallets ?? 0) > 0 ||\n    (e.holders?.count ?? 0) > 0 ||\n    (cls === \"smart_trader\" && (e.table?.traders ?? 0) > 0) ||\n    (e.named ? e.named.buyRows + e.named.sellRows > 0 : false)\n  );\n}\n```\n\nThe flow branch calls it, and does the right thing:\n\n```\nif (!presence(e, cls))\n  return R(\"UNVERIFIABLE\", \"U-NOCLASS\", [\n    `Nansen tags no wallet as ${who} in this token (24 h, 7 d, holders) — ` +\n    `the wallet in the post is not one Nansen labels`,\n  ]);\nreturn R(\"CONTRADICTED\", \"C-NOBODY\", [ /* ... */ ]);\n```\n\nThe holding branch never called it. Not once. Here is what it did instead — read it as a ladder, because that is how it fails:\n\n``` js\nfunction decideHolding(claim, e, rules, T7, R) {\n  const who = subjectName(claim.subject);   // \"Whales\" | \"Smart Money\"\n  const cls = subjectClass(claim.subject);  // \"whale\"  | \"smart_trader\"\n  const h = e.holders;\n  const net7 = e.flow7d?.[cls]?.net ?? null;\n  if (!h && net7 == null) return R(\"UNVERIFIABLE\", \"U-HOLD\", [/* no data at all */]);\n\n  // ↓ no presence() check here ↓\n\n  if (h && h.count < rules.minHolders) return R(\"OVERSTATED\", \"O-HOLDERS\", [/* ... */]);\n  if (h && net7 != null && net7 <= -T7 && h.delta7d < 0) return R(\"CONTRADICTED\", \"C-EXIT\", [/* ... */]);\n  if ((h && h.delta7d < 0) || (net7 != null && net7 <= -T7)) return R(\"OVERSTATED\", \"O-TRIM\", [/* ... */]);\n  return R(\"CONFIRMED\", \"A-HOLD\", [/* \"balances are not shrinking\" */]);\n}\n```\n\n**Zero holders.** `tgm/holders` answers with an empty page, so `h.count === 0`. The first rung catches it — `0 < 5` — and returns `OVERSTATED`, whose UI copy means *partly true*. Partly true about nobody. Bad, but at least it is hedging.\n\n**The holders call failed.** Now `h` is `null`, and the seven-day flow answered with `wallets: 0`. Look at the ladder again: every remaining rung is guarded by `h &&`. `O-HOLDERS` needs `h`. `C-EXIT` needs `h`. `O-TRIM` needs `h` or a net flow below the negative threshold — and the net flow is `0`, which is not below anything. So the claim falls all the way through and lands on the last line:\n\n```\n{ label: \"CONFIRMED\", rule: \"A-HOLD\", presence: false,\n  reasons: [ \"7 d Whales net flow $0 (threshold $5K)\",\n             \"Whales balances are not shrinking\" ] }\n```\n\n\"Whales balances are not shrinking.\" Technically unfalsifiable and completely true, in the way that *\"all the unicorns in my garage are healthy\"* is true. There are no whales. Nothing is shrinking because there is nothing. And the tool rendered that as **CONFIRMED**, the strongest word it owns, on the one screen a reader actually looks at.\n\nThe path needs no exotic input. One failed HTTP call to a sparse endpoint, on a token whose class is empty — which, per my own notes, is WETH.\n\n```\nif (!presence(e, cls))\n  return R(\"UNVERIFIABLE\", \"U-NOCLASS\", [/* same message the flow path uses */]);\n```\n\nOne line, plus a comment, plus 45 lines of tests. The whole commit is `+51`.\n\nI found the first case because I ran the source past an outside model, which spotted the zero-holders rung. Verifying its claim is what turned up the second: I wrote a throwaway test with `holders: null` to see what the other branch did, and got `CONFIRMED` back. The review found the hedge; checking the review found the lie.\n\nAll 13 recorded fixtures still reproduce with byte-identical sha256 evidence hashes after the change, which is the thing that let me ship a one-line edit to the verdict engine four days before a deadline without flinching.\n\n**A limitation stated in prose is not a constraint.** I had the right belief, written in the right file, in public, in the artifact judges read. It did nothing. Prose cannot fail a build.\n\n**An invariant enforced in one branch is enforced in zero branches.** `presence()` existed. It was correct. It was even *called* — just not on every path that needed it. A shared helper that callers must remember to call is a convention, and conventions decay the moment someone adds a fourth claim type.\n\n**The dangerous bug was in the fallback, not the happy path.** Every test I had written covered tokens *with* data, because that is what the demo uses and what the fixtures recorded. The failure mode lived where the data was absent — `h === null`, `wallets: 0`, a call that didn't answer — and absence is exactly what nobody writes fixtures for. If you have a decision function with a default return at the bottom, the question worth asking is not \"is this the right default\" but **\"what is the emptiest input that can reach this line?\"**\n\n**Sparse-by-design data is a first-class input.** I treated \"no whales\" as degenerate. It is not. It is Wednesday.\n\n`UNVERIFIABLE`, not an investigation of who that wallet is.` flow-intelligence` vs `smart-money/netflow`). I surface both rather than reconcile them; `docs/DX-REPORT.md` has the measurement.\nFor the numbers people ask for: 352 tests, 100% statement/branch/function/line coverage on the engine, 13 fixtures that replay offline with zero network calls, cold p50 4.0 s / p95 5.0 s, ~10 API credits per verdict, 0 of 154 live calls failed in the benchmark. Details in `docs/BENCH.md`.\n\nThe tool is at **[https://rebuttal.edycu.dev](https://rebuttal.edycu.dev)** and the code is at **[https://github.com/edycutjong/rebuttal](https://github.com/edycutjong/rebuttal)** — the commit in this post is `fedb50b`, and the test that pins both failure paths is `packages/core/test/review4.test.ts`.\n\nIf you have a verdict function with a `return` at the bottom of a ladder, go and check what the emptiest possible input does to it. That is the whole article.", "url": "https://wpnews.pro/news/my-fact-checker-said-confirmed-about-a-group-that-doesn-t-exist", "canonical_source": "https://dev.to/edycutjong/my-fact-checker-said-confirmed-about-a-group-that-doesnt-exist-2eo8", "published_at": "2026-09-23 00:13:27+00:00", "updated_at": "2026-09-23 00:22:47.509052+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools"], "entities": ["Nansen", "WETH"], "alternates": {"html": "https://wpnews.pro/news/my-fact-checker-said-confirmed-about-a-group-that-doesn-t-exist", "markdown": "https://wpnews.pro/news/my-fact-checker-said-confirmed-about-a-group-that-doesn-t-exist.md", "text": "https://wpnews.pro/news/my-fact-checker-said-confirmed-about-a-group-that-doesn-t-exist.txt", "jsonld": "https://wpnews.pro/news/my-fact-checker-said-confirmed-about-a-group-that-doesn-t-exist.jsonld"}}