Show HN: Factlabel: catches AI agents lying about the data they're reporting on Factlabel launched as a tool that audits AI-written content against source data, recomputing every figure in code and returning typed pass, review, or block verdicts at a cost of about $0.0006 per audit in under a second. The system targets three failure modes — invented figures, false attribution, and spin — and annotates agent-written pages with numbered findings, per-finding confidence, and a Trust Facts badge scoring numbers, sourcing, attribution, framing, and completeness. Factlabel runs a three-tier audit: deterministic code checks, a parallel Jev audit matrix, and routing of below-threshold answers to a human instead of guessing. A nutrition label for AI-written content. factlabel checks what an AI agent writes about data against the data itself, blocks what doesn't hold up, and shows readers exactly what was checked and why. AI agents now write the updates people read: fundraising reports, financial summaries, dashboards, briefings. The numbers usually come from real records. The story around them often doesn't. Three failure modes show up again and again: - Invented figures. "More than 2,000 new donors joined." The ledger shows 9 donors in all. - False attribution. "Driven by an explosion of grassroots momentum." 90% of the money came from one corporate gift. - Spin. "Individual giving grew 40%." It fell 30.6%. Each sentence sounds reasonable, and a reader has no way to check it. Asking another large language model to review the text doesn't solve this: it is slow, costly per word, and can be talked into agreeing. | | Ask an LLM to review the text | factlabel | |---|---|---| | Checks the numbers | Reads them, doesn't recompute them | Recomputes every figure from the source records in code | | Can be argued out of a finding | Yes: a confident, well-written draft can talk it round | No: figures are compared by code, not persuaded | | Output | A paragraph of prose you now have to trust | Typed pass / review / block, with a confidence per finding | | Shows its work | Sometimes, in more prose | Every finding shows its exact steps, the question asked, and the evidence used | | Cost per check | Priced per output token, scales with how much it writes | About $0.0006 per audit, output is free Jev returns typed answers, not text | | Speed | Seconds per call, serial | Under a second, ~50 questions answered in parallel | | Uncertain calls | No native confidence signal | Below-threshold answers are routed to a human instead of guessing | factlabel sits between the agent and the reader. 1. Audits the draft against your source records and documents: every figure, dashboard tile, citation, cause-and-effect claim, and trend. 2. Enforces publish gates: each draft comes out as pass , review a person must look , or block . 3. Annotates the agent's own page with numbered findings, a "why this was flagged" explanation for each, and a Trust Facts badge that summarizes the whole page. 4. Fixes it optional : code corrects the dashboard numbers from the source data, Claude rewrites the text from the findings, and factlabel audits the rewrite from scratch until it passes or a person needs to step in. Behind the badge in the corner is a summary of the whole page, not just a grade: a headline, scores for numbers, sourcing, attribution, framing, and completeness , what checked out, and what was checked. Every finding is numbered to match a badge on the page. Each finding shows the steps that produced it, marked as decided by code or by Jev , with the exact question Jev was asked and its answer probabilities, the evidence used, and the threshold that applied. The same page after the revise loop: every number traces back to the ledger, the real source of the growth is named, and the decline is stated plainly. The audit runs in three decoupled tiers: agent draft + source data │ ▼ ┌──────────────────────────────────────────────┐ │ Tier 1: deterministic code only │ │ - recompute every metric, break every table │ │ down by its categories │ │ - extract and mask figures, match quotes │ │ - spot causal connectors "driven by" │ │ - anomaly profile: the material facts an │ │ honest draft must disclose │ └──────────────────────┬───────────────────────┘ ▼ ┌──────────────────────────────────────────────┐ │ Tier 2: parallel Jev audit matrix │ │ blinded call the text only, no data : │ │ which metric each figure and tile reports, │ │ which cause a sentence credits, which way │ │ it says each metric moved, how loaded its │ │ wording is │ │ grounded call text + verified facts : │ │ is each claim supported, does each │ │ citation hold, is each material fact │ │ disclosed │ └──────────────────────┬───────────────────────┘ ▼ ┌──────────────────────────────────────────────┐ │ Tier 3: supervisory mediation code │ │ - composite rules over atomic answers │ │ - confident violation - block red │ │ - borderline or conflicting - a person │ │ reviews amber │ │ - verified in code, matched with confidence │ │ - silent pass green check │ └──────────────────────────────────────────────┘ The rule that matters most: a model never decides whether a number is right. Jev only tells code which source metric a sentence is talking about. Code does the comparison, and reads qualifiers literally "over 90%" fails when the truth is 89% . Blinded questions. Anything about what the text says is asked without showing Jev the data, so a wrong number in the draft, or the true number in the ledger, can't bias the reading. Whether the text holds up is asked separately, with the data. Attribution in two passes. Pass A reads the sentence with its numbers masked and asks which cause it credits and for which outcome. Pass B is computed from the records: the credited cause's real share of that outcome, judged against the change when the sentence describes a change. Every reading Jev considers is weighted by its probability, so "individual donors 62% or something not in the data 36% " counts as 98% sure the credit isn't backed. Material omissions. Tier 1 profiles the data for declines, concentration in one source, and growth that came mostly from one place. Each fact is checked with two atomic questions how the draft treats it, and whether it acknowledges it that must agree. Growth concentration only counts when the draft reports that growth. Jev https://docs.typesafe.ai from TypeSafe AI is a decision model, not a chat model. It doesn't write text. It answers typed questions about a shared state in one parallel pass: - Choice: pick one option from a list, with a probability for each. - Score: place something on a rubric you define. - Noul: the probability that a yes/no statement is true. That fits an audit well. Every answer is one of the options you defined, so results are typed and machine-checkable. Each answer comes with a confidence, so uncertain calls can go to a person instead of blocking on their own. And it is fast and cheap enough to run on every draft: a full audit of the example page asks about 50 questions across two parallel calls, takes under a second, and costs about $0.0006. Jev can still be wrong, and TypeSafe's published accuracy figures are its own. That's why numbers are checked by code, uncertain answers go to a person, and every judgment is shown to the reader rather than hidden behind a score. factlabel needs Python 3.10 or later. It isn't on PyPI yet, so install it from GitHub: pip install "factlabel claude @ git+https://github.com/generallymatthew/factlabel.git" The claude extra is only needed for the revise loop. To work on factlabel itself: git clone https://github.com/generallymatthew/factlabel.git cd factlabel python3 -m venv .venv && source .venv/bin/activate pip install -e ". dev " pytest The tests don't call any API, so no keys are needed to run them. You bring your own keys. They are read from environment variables and never stored in the project. | Variable | Needed for | Get one at | |---|---|---| | TYPESAFE API KEY | Audits Jev | console.typesafe.ai/keys https://console.typesafe.ai/keys | | ANTHROPIC API KEY | The revise loop only | console.anthropic.com https://console.anthropic.com/settings/keys | | ANTHROPIC WORKSPACE ID | Only if your Anthropic key isn't scoped to a workspace | Anthropic console, Workspaces | The easiest way is a .env file already git-ignored . Copy the template and fill it in: cp .env.example .env set -a; source .env; set +a Try it on the included example: a fundraising agent's Q3 update that gets the story wrong. 1. Audit the draft factlabel audit examples/fundraising/spun draft.json --html out/label.html 2. Annotate the agent's own dashboard page factlabel annotate examples/fundraising/spun draft.json examples/fundraising/dashboard.html \ -o out/annotated.html 3. Have Claude fix it, re-auditing each version factlabel revise examples/fundraising/spun draft.json --out out/revised.json 4. Put the fixed draft back into the page, with Trust Facts factlabel rebuild out/revised.json examples/fundraising/dashboard.html -o out/rebuilt.html --annotate Open the HTML files in a browser. examples/fundraising/honest draft.json is the same update written honestly; it passes. No API key yet? --no-jev runs only the checks code can do on its own numbers, dashboard tiles, quotes , and says clearly which checks were skipped. | Command | What it does | Exit code | |---|---|---| | factlabel audit CASE.json | Audit a draft. --html writes a Trust Facts label, --json the full report | 2 block, 1 review, 0 pass --fail-on controls which fail | | factlabel annotate CASE.json PAGE.html -o OUT.html | Annotate the agent's page in place. --correct swaps wrong numbers for source values | 0 | | factlabel revise CASE.json | Audit, fix, and re-audit, up to --max-attempts default 3 | 0 passed, 1 review, 2 escalated to a person | | factlabel rebuild CASE.json PAGE.html -o OUT.html | Write a draft's tiles and text into the page. --annotate adds Trust Facts | 1 if anything couldn't be placed | | factlabel eval EVALS DIR | Score the audit on paired misleading and honest drafts. --runs N measures stability | 0 | The exit codes make factlabel easy to drop into CI or an agent's publish step. python from factlabel import JevJudge, PublishBlocked, enforce, load case draft, sources, metric specs = load case "examples/fundraising/spun draft.json" try: report = enforce draft, sources, metric specs=metric specs, judge=JevJudge except PublishBlocked as e: print e draft blocked F, 0/100 : FALSE ATTRIBUTION, FIGURE MISMATCH, ... report = e.report To fix drafts with your own agent instead of Claude, pass any function draft, report, sources - draft to revise . feedback for report, sources gives you findings written for an agent to act on: python from factlabel import JevJudge, feedback for, load case, revise def my agent draft, report, sources : return my agent rewrite draft, feedback for report, sources result = revise draft, sources, my agent, metric specs=metric specs, judge=JevJudge result.status "passed", "review", or "escalated" A case file holds the draft, the source data, and the metrics to check against. { "draft": { "narrative": "We raised $1.25M in Q3 ...", "dashboard": {"label": "Total raised Q3 ", "value": 1250000, "unit": "USD"} , "citations": {"claim": "...", "quote": "exact words", "source": "board memo"} }, "sources": { "documents": {"board memo": "Q3 results were dominated by ..."}, "tables": { "donations": {"donor": "Acme Corp", "type": "corporate", "quarter": "Q3", "amount": 1125000} } }, "metrics": { "q3 total raised": { "op": "sum", "table": "donations", "field": "amount", "where": {"quarter": "Q3"}, "unit": "USD", "description": "Total dollars raised in Q3" } } } Metric ops: sum , count , count distinct , mean , min , max , share a percent, with an optional within scope , pct change period field , from , to , and value for numbers you compute yourself. Write clear description s: Jev uses them to match sentences to metrics. factlabel also breaks every table down by its categories on its own by type, by quarter, and so on , so a figure that comes from those breakdowns counts as sourced, and false attribution can be caught without you defining a metric for every slice. annotate and rebuild work on any HTML page. These optional attributes make matching exact: