{"slug": "show-hn-factlabel-catches-ai-agents-lying-about-the-data-they-re-reporting-on", "title": "Show HN: Factlabel: catches AI agents lying about the data they're reporting on", "summary": "Factlabel launched as a tool that audits AI-written content against source data, recomputing every figure in code and returning typed pass, review, or block verdicts at a cost of about $0.0006 per audit in under a second. The system targets three failure modes — invented figures, false attribution, and spin — and annotates agent-written pages with numbered findings, per-finding confidence, and a Trust Facts badge scoring numbers, sourcing, attribution, framing, and completeness. Factlabel runs a three-tier audit: deterministic code checks, a parallel Jev audit matrix, and routing of below-threshold answers to a human instead of guessing.", "body_md": "**A nutrition label for AI-written content.** factlabel checks what an AI agent writes about\ndata against the data itself, blocks what doesn't hold up, and shows readers exactly what\nwas checked and why.\n\nAI agents now write the updates people read: fundraising reports, financial summaries, dashboards, briefings. The numbers usually come from real records. The story around them often doesn't. Three failure modes show up again and again:\n\n- **Invented figures.** \"More than 2,000 new donors joined.\" The ledger shows 9 donors in all.\n- **False attribution.** \"Driven by an explosion of grassroots momentum.\" 90% of the money came\nfrom one corporate gift.\n- **Spin.** \"Individual giving grew 40%.\" It fell 30.6%.\n\nEach sentence sounds reasonable, and a reader has no way to check it. Asking another large language model to review the text doesn't solve this: it is slow, costly per word, and can be talked into agreeing.\n\n|  | Ask an LLM to review the text | factlabel | \n|---|---|---|\n| **Checks the numbers** | Reads them, doesn't recompute them | Recomputes every figure from the source records in code | \n| **Can be argued out of a finding** | Yes: a confident, well-written draft can talk it round | No: figures are compared by code, not persuaded | \n| **Output** | A paragraph of prose you now have to trust | Typed pass / review / block, with a confidence per finding | \n| **Shows its work** | Sometimes, in more prose | Every finding shows its exact steps, the question asked, and the evidence used | \n| **Cost per check** | Priced per output token, scales with how much it writes | About $0.0006 per audit, output is free (Jev returns typed answers, not text) | \n| **Speed** | Seconds per call, serial | Under a second, ~50 questions answered in parallel | \n| **Uncertain calls** | No native confidence signal | Below-threshold answers are routed to a human instead of guessing | \n\nfactlabel sits between the agent and the reader.\n\n1. **Audits the draft** against your source records and documents: every figure, dashboard\ntile, citation, cause-and-effect claim, and trend.\n2. **Enforces publish gates:** each draft comes out as`pass` ,`review` (a person must look),\nor`block` .\n3. **Annotates the agent's own page** with numbered findings, a \"why this was flagged\"\nexplanation for each, and a Trust Facts badge that summarizes the whole page.\n4. **Fixes it (optional):** code corrects the dashboard numbers from the source data, Claude\nrewrites the text from the findings, and factlabel audits the rewrite from scratch until\nit passes or a person needs to step in.\n\nBehind the badge in the corner is a summary of the whole page, not just a grade: a\nheadline, scores for **numbers, sourcing, attribution, framing, and completeness**, what\nchecked out, and what was checked. Every finding is numbered to match a badge on the page.\n\nEach finding shows the steps that produced it, marked as decided by **code** or by **Jev**,\nwith the exact question Jev was asked and its answer probabilities, the evidence used, and\nthe threshold that applied.\n\nThe same page after the revise loop: every number traces back to the ledger, the real source of the growth is named, and the decline is stated plainly.\n\nThe audit runs in three decoupled tiers:\n\n```\n[agent draft + source data]\n            │\n            ▼\n┌──────────────────────────────────────────────┐\n│ Tier 1: deterministic (code only)            │\n│  - recompute every metric, break every table │\n│    down by its categories                    │\n│  - extract and mask figures, match quotes    │\n│  - spot causal connectors (\"driven by\")      │\n│  - anomaly profile: the material facts an    │\n│    honest draft must disclose                │\n└──────────────────────┬───────────────────────┘\n                       ▼\n┌──────────────────────────────────────────────┐\n│ Tier 2: parallel Jev audit matrix            │\n│  blinded call (the text only, no data):      │\n│   which metric each figure and tile reports, │\n│   which cause a sentence credits, which way  │\n│   it says each metric moved, how loaded its  │\n│   wording is                                 │\n│  grounded call (text + verified facts):      │\n│   is each claim supported, does each         │\n│   citation hold, is each material fact       │\n│   disclosed                                  │\n└──────────────────────┬───────────────────────┘\n                       ▼\n┌──────────────────────────────────────────────┐\n│ Tier 3: supervisory mediation (code)         │\n│  - composite rules over atomic answers       │\n│  - confident violation      -> block (red)   │\n│  - borderline or conflicting -> a person     │\n│    reviews (amber)                           │\n│  - verified in code, matched with confidence │\n│    -> silent pass (green check)              │\n└──────────────────────────────────────────────┘\n```\n\n**The rule that matters most: a model never decides whether a number is right.** Jev only\ntells code *which* source metric a sentence is talking about. Code does the comparison,\nand reads qualifiers literally (\"over 90%\" fails when the truth is 89%).\n\n**Blinded questions.** Anything about what the text *says* is asked without showing Jev the\ndata, so a wrong number in the draft, or the true number in the ledger, can't bias the\nreading. Whether the text *holds up* is asked separately, with the data.\n\n**Attribution in two passes.** Pass A reads the sentence with its numbers masked and asks\nwhich cause it credits and for which outcome. Pass B is computed from the records: the\ncredited cause's real share of that outcome, judged against the change when the sentence\ndescribes a change. Every reading Jev considers is weighted by its probability, so\n\"individual donors (62%) or something not in the data (36%)\" counts as 98% sure the credit\nisn't backed.\n\n**Material omissions.** Tier 1 profiles the data for declines, concentration in one source,\nand growth that came mostly from one place. Each fact is checked with two atomic questions\n(how the draft treats it, and whether it acknowledges it) that must agree. Growth\nconcentration only counts when the draft reports that growth.\n\n[Jev](https://docs.typesafe.ai) from TypeSafe AI is a decision model, not a chat model. It\ndoesn't write text. It answers typed questions about a shared state in one parallel pass:\n\n- **Choice:** pick one option from a list, with a probability for each.\n- **Score:** place something on a rubric you define.\n- **Noul:** the probability that a yes/no statement is true.\n\nThat fits an audit well. Every answer is one of the options you defined, so results are typed and machine-checkable. Each answer comes with a confidence, so uncertain calls can go to a person instead of blocking on their own. And it is fast and cheap enough to run on every draft: a full audit of the example page asks about 50 questions across two parallel calls, takes under a second, and costs about $0.0006.\n\nJev can still be wrong, and TypeSafe's published accuracy figures are its own. That's why numbers are checked by code, uncertain answers go to a person, and every judgment is shown to the reader rather than hidden behind a score.\n\nfactlabel needs Python 3.10 or later. It isn't on PyPI yet, so install it from GitHub:\n\n```\npip install \"factlabel[claude] @ git+https://github.com/generallymatthew/factlabel.git\"\n```\n\nThe `[claude]` extra is only needed for the revise loop. To work on factlabel itself:\n\n```\ngit clone https://github.com/generallymatthew/factlabel.git\ncd factlabel\npython3 -m venv .venv && source .venv/bin/activate\npip install -e \".[dev]\"\npytest\n```\n\nThe tests don't call any API, so no keys are needed to run them.\n\nYou bring your own keys. They are read from environment variables and never stored in the project.\n\n| Variable | Needed for | Get one at | \n|---|---|---|\n| `TYPESAFE_API_KEY` | Audits (Jev) | [console.typesafe.ai/keys](https://console.typesafe.ai/keys) | \n| `ANTHROPIC_API_KEY` | The revise loop only | [console.anthropic.com](https://console.anthropic.com/settings/keys) | \n| `ANTHROPIC_WORKSPACE_ID` | Only if your Anthropic key isn't scoped to a workspace | Anthropic console, Workspaces | \n\nThe easiest way is a `.env` file (already git-ignored). Copy the template and fill it in:\n\n```\ncp .env.example .env\nset -a; source .env; set +a\n```\n\nTry it on the included example: a fundraising agent's Q3 update that gets the story wrong.\n\n```\n# 1. Audit the draft\nfactlabel audit examples/fundraising/spun_draft.json --html out/label.html\n\n# 2. Annotate the agent's own dashboard page\nfactlabel annotate examples/fundraising/spun_draft.json examples/fundraising/dashboard.html \\\n  -o out/annotated.html\n\n# 3. Have Claude fix it, re-auditing each version\nfactlabel revise examples/fundraising/spun_draft.json --out out/revised.json\n\n# 4. Put the fixed draft back into the page, with Trust Facts\nfactlabel rebuild out/revised.json examples/fundraising/dashboard.html -o out/rebuilt.html --annotate\n```\n\nOpen the HTML files in a browser. `examples/fundraising/honest_draft.json` is the same\nupdate written honestly; it passes.\n\nNo API key yet? `--no-jev` runs only the checks code can do on its own (numbers, dashboard\ntiles, quotes), and says clearly which checks were skipped.\n\n| Command | What it does | Exit code | \n|---|---|---|\n| `factlabel audit CASE.json` | Audit a draft. `--html` writes a Trust Facts label,`--json` the full report | `2` block,`1` review,`0` pass (`--fail-on` controls which fail) | \n| `factlabel annotate CASE.json PAGE.html -o OUT.html` | Annotate the agent's page in place. `--correct` swaps wrong numbers for source values | `0` | \n| `factlabel revise CASE.json` | Audit, fix, and re-audit, up to `--max-attempts` (default 3) | `0` passed,`1` review,`2` escalated to a person | \n| `factlabel rebuild CASE.json PAGE.html -o OUT.html` | Write a draft's tiles and text into the page. `--annotate` adds Trust Facts | `1` if anything couldn't be placed | \n| `factlabel eval EVALS_DIR` | Score the audit on paired misleading and honest drafts. `--runs N` measures stability | `0` | \n\nThe exit codes make factlabel easy to drop into CI or an agent's publish step.\n\n``` python\nfrom factlabel import JevJudge, PublishBlocked, enforce, load_case\n\ndraft, sources, metric_specs = load_case(\"examples/fundraising/spun_draft.json\")\ntry:\n    report = enforce(draft, sources, metric_specs=metric_specs, judge=JevJudge())\nexcept PublishBlocked as e:\n    print(e)          # draft blocked (F, 0/100): FALSE_ATTRIBUTION, FIGURE_MISMATCH, ...\n    report = e.report\n```\n\nTo fix drafts with your own agent instead of Claude, pass any function\n`(draft, report, sources) -> draft` to `revise()`. `feedback_for(report, sources)` gives\nyou findings written for an agent to act on:\n\n``` python\nfrom factlabel import JevJudge, feedback_for, load_case, revise\n\ndef my_agent(draft, report, sources):\n    return my_agent_rewrite(draft, feedback_for(report, sources))\n\nresult = revise(draft, sources, my_agent, metric_specs=metric_specs, judge=JevJudge())\nresult.status     # \"passed\", \"review\", or \"escalated\"\n```\n\nA case file holds the draft, the source data, and the metrics to check against.\n\n```\n{\n  \"draft\": {\n    \"narrative\": \"We raised $1.25M in Q3 ...\",\n    \"dashboard\": [{\"label\": \"Total raised (Q3)\", \"value\": 1250000, \"unit\": \"USD\"}],\n    \"citations\": [{\"claim\": \"...\", \"quote\": \"exact words\", \"source\": \"board_memo\"}]\n  },\n  \"sources\": {\n    \"documents\": {\"board_memo\": \"Q3 results were dominated by ...\"},\n    \"tables\": {\n      \"donations\": [{\"donor\": \"Acme Corp\", \"type\": \"corporate\", \"quarter\": \"Q3\", \"amount\": 1125000}]\n    }\n  },\n  \"metrics\": {\n    \"q3_total_raised\": {\n      \"op\": \"sum\", \"table\": \"donations\", \"field\": \"amount\", \"where\": {\"quarter\": \"Q3\"},\n      \"unit\": \"USD\", \"description\": \"Total dollars raised in Q3\"\n    }\n  }\n}\n```\n\nMetric ops: `sum`, `count`, `count_distinct`, `mean`, `min`, `max`, `share` (a percent, with\nan optional `within` scope), `pct_change` (`period_field`, `from`, `to`), and `value` for\nnumbers you compute yourself. Write clear `description` s: Jev uses them to match sentences\nto metrics.\n\nfactlabel also breaks every table down by its categories on its own (by type, by quarter, and so on), so a figure that comes from those breakdowns counts as sourced, and false attribution can be caught without you defining a metric for every slice.\n\n`annotate` and `rebuild` work on any HTML page. These optional attributes make matching exact:\n\n```\n<div data-factlabel-tile=\"Donors\">\n  <span>Donors</span>\n  <b data-factlabel-value>2,140</b>\n</div>\n\n<div data-factlabel-narrative>\n  <p>The agent's text goes here.</p>\n</div>\n```\n\nWithout them, tiles are found by their label text and sentences by text search, including\nsentences that run across tags like `<strong>`. Readers can deep-link to the panel with\n`#trust-facts` or to a finding with `#finding-5`.\n\n| Finding | Severity | Decided by | \n|---|---|---|\n| Invented figure: no metric or breakdown value has this number | block | Jev matches (blinded), code checks | \n| Wrong figure: the number disagrees with its source | block | code | \n| Wrong or unsourced dashboard tile | block | code (Jev matches tiles, without seeing their values) | \n| Fabricated quote: the words aren't in the cited source | block | code | \n| Trend inverted or misframed: said up, went down (or flat) | block | Jev reads the stated direction (blinded), code knows the real one | \n| False / unproven attribution of a cause | block / warn | Pass A by Jev (blinded), Pass B computed from the records | \n| Material fact left out: a decline, a concentration, growth from one source | block / warn | profile in code, two Jev answers must agree | \n| Claim contradicted / unsupported by the data | block / warn | Jev (grounded) | \n| Citation contradicted / unsupported by its source | block / warn | Jev (grounded) | \n| Rhetorical spin (1 to 5), weak sourcing | warn | Jev | \n| Genre, ungrounded claims, no caveats | note | Jev | \n\nSupervisory tiers: a Jev-based violation at 85% confidence or more acts on its own; between\n50% and 85%, or when two atomic answers disagree, a person reviews it; below 50% it isn't\nflagged but stays in the report's audit trace. A check passes silently only when code made\nthe comparison and any matching Jev did was at least 90% sure. All of Jev's questions and\nevery threshold live in one file, [`src/factlabel/questions.py`](https://github.com/generallymatthew/factlabel/blob/main/src/factlabel/questions.py).\n\n`factlabel eval` scores the audit on paired drafts: a misleading draft and an honest draft of\nthe same update, over the same data. Each misleading draft lists the failure types it should\ntrigger, so precision and recall are measured per failure type.\n\n```\nfactlabel eval evals/ --runs 3            # 30 drafts across fundraising, SaaS usage, marketing ROI\nfactlabel eval evals-holdout/ --runs 3    # a retail domain written after the engine was built\n```\n\nResults against live Jev (`jev-1.13`), 3 runs each:\n\n|  | Before the tiered engine | Now | \n|---|---|---|\n| Misleading drafts blocked | 80% | 100% | \n| Honest drafts passed | 87% | 100% | \n| Omission precision | 50% | 100% | \n| Spin recall | 78% | 100% | \n| Every other failure type (precision / recall) | 100% / 100% | 100% / 100% | \n| Jev cost per full run (90 audits) | $0.019 | $0.034 | \n\nRead these numbers with care. The engine was tuned on the main suite, so it measures fit, not\ngeneralization. The retail domain was written after tuning and run once blind: every\nmisleading draft was blocked, but it exposed two real flaws (attribution judged against the\nlevel instead of the change, and a part of a metric read as the whole), and 2 of 3 honest\ndrafts passed. After fixing both, it scores 100% too, so it is no longer truly held out.\n**New example cases are the most useful contribution to this project.**\n\n- **Jev is a paid, closed, hosted service.** factlabel's code is open source, but its judgment\nstep calls TypeSafe's API with your key. The example audit costs about $0.0003. If you\ndepend on factlabel, know that you depend on TypeSafe too. Everything that decides whether\na number is right is plain code in this repository.\n- **Claude is optional.** Only the revise loop uses it, and a rewrite costs a few cents with\nClaude Opus 5. Any other model or agent can be plugged in instead.\n- **Nothing else is sent anywhere.** The annotated pages and labels are self-contained HTML\nfiles with no external requests.\n\n- Jev's answers are probabilities. Treat `review` as \"a person must look\", not \"probably fine\".\nTune the thresholds on your own drafts before relying on them.\n- Figure extraction is regex-based and English-only.\n- Trim long source documents before auditing; accuracy drops as irrelevant context grows.\n- The questions were tuned on four synthetic domains. Real agent drafts will find new failure modes; add them to the benchmark before changing the questions.\n\nContributions are welcome, especially new example cases and better questions. See\n[CONTRIBUTING.md](https://github.com/generallymatthew/factlabel/blob/main/CONTRIBUTING.md). You don't need any API keys to run the tests.\n\n[Apache-2.0](https://github.com/generallymatthew/factlabel/blob/main/LICENSE). Jev and Claude are used through their own APIs under their providers'\nterms; no part of either is included here.", "url": "https://wpnews.pro/news/show-hn-factlabel-catches-ai-agents-lying-about-the-data-they-re-reporting-on", "canonical_source": "https://github.com/generallymatthew/factlabel", "published_at": "2026-09-21 19:07:55+00:00", "updated_at": "2026-09-21 19:24:31.719166+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "ai-products"], "entities": ["Factlabel", "Jev", "Claude"], "alternates": {"html": "https://wpnews.pro/news/show-hn-factlabel-catches-ai-agents-lying-about-the-data-they-re-reporting-on", "markdown": "https://wpnews.pro/news/show-hn-factlabel-catches-ai-agents-lying-about-the-data-they-re-reporting-on.md", "text": "https://wpnews.pro/news/show-hn-factlabel-catches-ai-agents-lying-about-the-data-they-re-reporting-on.txt", "jsonld": "https://wpnews.pro/news/show-hn-factlabel-catches-ai-agents-lying-about-the-data-they-re-reporting-on.jsonld"}}