A nutrition label for AI-written content. factlabel checks what an AI agent writes about data against the data itself, blocks what doesn't hold up, and shows readers exactly what was checked and why.
AI agents now write the updates people read: fundraising reports, financial summaries, dashboards, briefings. The numbers usually come from real records. The story around them often doesn't. Three failure modes show up again and again:
- Invented figures. "More than 2,000 new donors joined." The ledger shows 9 donors in all.
- False attribution. "Driven by an explosion of grassroots momentum." 90% of the money came from one corporate gift.
- Spin. "Individual giving grew 40%." It fell 30.6%.
Each sentence sounds reasonable, and a reader has no way to check it. Asking another large language model to review the text doesn't solve this: it is slow, costly per word, and can be talked into agreeing.
| Ask an LLM to review the text | factlabel | |
|---|---|---|
| Checks the numbers | Reads them, doesn't recompute them | Recomputes every figure from the source records in code |
| Can be argued out of a finding | Yes: a confident, well-written draft can talk it round | No: figures are compared by code, not persuaded |
| Output | A paragraph of prose you now have to trust | Typed pass / review / block, with a confidence per finding |
| Shows its work | Sometimes, in more prose | Every finding shows its exact steps, the question asked, and the evidence used |
| Cost per check | Priced per output token, scales with how much it writes | About $0.0006 per audit, output is free (Jev returns typed answers, not text) |
| Speed | Seconds per call, serial | Under a second, ~50 questions answered in parallel |
| Uncertain calls | No native confidence signal | Below-threshold answers are routed to a human instead of guessing |
factlabel sits between the agent and the reader.
- Audits the draft against your source records and documents: every figure, dashboard tile, citation, cause-and-effect claim, and trend.
- Enforces publish gates: each draft comes out as
pass,review(a person must look), orblock. - Annotates the agent's own page with numbered findings, a "why this was flagged" explanation for each, and a Trust Facts badge that summarizes the whole page.
- Fixes it (optional): code corrects the dashboard numbers from the source data, Claude rewrites the text from the findings, and factlabel audits the rewrite from scratch until it passes or a person needs to step in.
Behind the badge in the corner is a summary of the whole page, not just a grade: a headline, scores for numbers, sourcing, attribution, framing, and completeness, what checked out, and what was checked. Every finding is numbered to match a badge on the page.
Each finding shows the steps that produced it, marked as decided by code or by Jev, with the exact question Jev was asked and its answer probabilities, the evidence used, and the threshold that applied.
The same page after the revise loop: every number traces back to the ledger, the real source of the growth is named, and the decline is stated plainly.
The audit runs in three decoupled tiers:
[agent draft + source data]
│
▼
┌──────────────────────────────────────────────┐
│ Tier 1: deterministic (code only) │
│ - recompute every metric, break every table │
│ down by its categories │
│ - extract and mask figures, match quotes │
│ - spot causal connectors ("driven by") │
│ - anomaly profile: the material facts an │
│ honest draft must disclose │
└──────────────────────┬───────────────────────┘
▼
┌──────────────────────────────────────────────┐
│ Tier 2: parallel Jev audit matrix │
│ blinded call (the text only, no data): │
│ which metric each figure and tile reports, │
│ which cause a sentence credits, which way │
│ it says each metric moved, how loaded its │
│ wording is │
│ grounded call (text + verified facts): │
│ is each claim supported, does each │
│ citation hold, is each material fact │
│ disclosed │
└──────────────────────┬───────────────────────┘
▼
┌──────────────────────────────────────────────┐
│ Tier 3: supervisory mediation (code) │
│ - composite rules over atomic answers │
│ - confident violation -> block (red) │
│ - borderline or conflicting -> a person │
│ reviews (amber) │
│ - verified in code, matched with confidence │
│ -> silent pass (green check) │
└──────────────────────────────────────────────┘
The rule that matters most: a model never decides whether a number is right. Jev only tells code which source metric a sentence is talking about. Code does the comparison, and reads qualifiers literally ("over 90%" fails when the truth is 89%).
Blinded questions. Anything about what the text says is asked without showing Jev the data, so a wrong number in the draft, or the true number in the ledger, can't bias the reading. Whether the text holds up is asked separately, with the data.
Attribution in two passes. Pass A reads the sentence with its numbers masked and asks which cause it credits and for which outcome. Pass B is computed from the records: the credited cause's real share of that outcome, judged against the change when the sentence describes a change. Every reading Jev considers is weighted by its probability, so "individual donors (62%) or something not in the data (36%)" counts as 98% sure the credit isn't backed.
Material omissions. Tier 1 profiles the data for declines, concentration in one source, and growth that came mostly from one place. Each fact is checked with two atomic questions (how the draft treats it, and whether it acknowledges it) that must agree. Growth concentration only counts when the draft reports that growth.
Jev from TypeSafe AI is a decision model, not a chat model. It doesn't write text. It answers typed questions about a shared state in one parallel pass:
- Choice: pick one option from a list, with a probability for each.
- Score: place something on a rubric you define.
- Noul: the probability that a yes/no statement is true.
That fits an audit well. Every answer is one of the options you defined, so results are typed and machine-checkable. Each answer comes with a confidence, so uncertain calls can go to a person instead of blocking on their own. And it is fast and cheap enough to run on every draft: a full audit of the example page asks about 50 questions across two parallel calls, takes under a second, and costs about $0.0006.
Jev can still be wrong, and TypeSafe's published accuracy figures are its own. That's why numbers are checked by code, uncertain answers go to a person, and every judgment is shown to the reader rather than hidden behind a score.
factlabel needs Python 3.10 or later. It isn't on PyPI yet, so install it from GitHub:
pip install "factlabel[claude] @ git+https://github.com/generallymatthew/factlabel.git"
The [claude] extra is only needed for the revise loop. To work on factlabel itself:
git clone https://github.com/generallymatthew/factlabel.git
cd factlabel
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
The tests don't call any API, so no keys are needed to run them.
You bring your own keys. They are read from environment variables and never stored in the project.
| Variable | Needed for | Get one at |
|---|---|---|
TYPESAFE_API_KEY |
Audits (Jev) | console.typesafe.ai/keys |
ANTHROPIC_API_KEY |
The revise loop only | console.anthropic.com |
ANTHROPIC_WORKSPACE_ID |
Only if your Anthropic key isn't scoped to a workspace | Anthropic console, Workspaces |
The easiest way is a .env file (already git-ignored). Copy the template and fill it in:
cp .env.example .env
set -a; source .env; set +a
Try it on the included example: a fundraising agent's Q3 update that gets the story wrong.
factlabel audit examples/fundraising/spun_draft.json --html out/label.html
factlabel annotate examples/fundraising/spun_draft.json examples/fundraising/dashboard.html \
-o out/annotated.html
factlabel revise examples/fundraising/spun_draft.json --out out/revised.json
factlabel rebuild out/revised.json examples/fundraising/dashboard.html -o out/rebuilt.html --annotate
Open the HTML files in a browser. examples/fundraising/honest_draft.json is the same
update written honestly; it passes.
No API key yet? --no-jev runs only the checks code can do on its own (numbers, dashboard
tiles, quotes), and says clearly which checks were skipped.
| Command | What it does | Exit code |
|---|---|---|
factlabel audit CASE.json |
Audit a draft. --html writes a Trust Facts label,--json the full report |
2 block,1 review,0 pass (--fail-on controls which fail) |
factlabel annotate CASE.json PAGE.html -o OUT.html |
Annotate the agent's page in place. --correct swaps wrong numbers for source values |
0 |
factlabel revise CASE.json |
Audit, fix, and re-audit, up to --max-attempts (default 3) |
0 passed,1 review,2 escalated to a person |
factlabel rebuild CASE.json PAGE.html -o OUT.html |
Write a draft's tiles and text into the page. --annotate adds Trust Facts |
1 if anything couldn't be placed |
factlabel eval EVALS_DIR |
Score the audit on paired misleading and honest drafts. --runs N measures stability |
0 |
The exit codes make factlabel easy to drop into CI or an agent's publish step.
from factlabel import JevJudge, PublishBlocked, enforce, load_case
draft, sources, metric_specs = load_case("examples/fundraising/spun_draft.json")
try:
report = enforce(draft, sources, metric_specs=metric_specs, judge=JevJudge())
except PublishBlocked as e:
print(e) # draft blocked (F, 0/100): FALSE_ATTRIBUTION, FIGURE_MISMATCH, ...
report = e.report
To fix drafts with your own agent instead of Claude, pass any function
(draft, report, sources) -> draft to revise(). feedback_for(report, sources) gives
you findings written for an agent to act on:
from factlabel import JevJudge, feedback_for, load_case, revise
def my_agent(draft, report, sources):
return my_agent_rewrite(draft, feedback_for(report, sources))
result = revise(draft, sources, my_agent, metric_specs=metric_specs, judge=JevJudge())
result.status # "passed", "review", or "escalated"
A case file holds the draft, the source data, and the metrics to check against.
{
"draft": {
"narrative": "We raised $1.25M in Q3 ...",
"dashboard": [{"label": "Total raised (Q3)", "value": 1250000, "unit": "USD"}],
"citations": [{"claim": "...", "quote": "exact words", "source": "board_memo"}]
},
"sources": {
"documents": {"board_memo": "Q3 results were dominated by ..."},
"tables": {
"donations": [{"donor": "Acme Corp", "type": "corporate", "quarter": "Q3", "amount": 1125000}]
}
},
"metrics": {
"q3_total_raised": {
"op": "sum", "table": "donations", "field": "amount", "where": {"quarter": "Q3"},
"unit": "USD", "description": "Total dollars raised in Q3"
}
}
}
Metric ops: sum, count, count_distinct, mean, min, max, share (a percent, with
an optional within scope), pct_change (period_field, from, to), and value for
numbers you compute yourself. Write clear description s: Jev uses them to match sentences
to metrics.
factlabel also breaks every table down by its categories on its own (by type, by quarter, and so on), so a figure that comes from those breakdowns counts as sourced, and false attribution can be caught without you defining a metric for every slice.
annotate and rebuild work on any HTML page. These optional attributes make matching exact:
<div data-factlabel-tile="Donors">
<span>Donors</span>
<b data-factlabel-value>2,140</b>
</div>
<div data-factlabel-narrative>
<p>The agent's text goes here.</p>
</div>
Without them, tiles are found by their label text and sentences by text search, including
sentences that run across tags like <strong>. Readers can deep-link to the panel with
#trust-facts or to a finding with #finding-5.
| Finding | Severity | Decided by |
|---|---|---|
| Invented figure: no metric or breakdown value has this number | block | Jev matches (blinded), code checks |
| Wrong figure: the number disagrees with its source | block | code |
| Wrong or unsourced dashboard tile | block | code (Jev matches tiles, without seeing their values) |
| Fabricated quote: the words aren't in the cited source | block | code |
| Trend inverted or misframed: said up, went down (or flat) | block | Jev reads the stated direction (blinded), code knows the real one |
| False / unproven attribution of a cause | block / warn | Pass A by Jev (blinded), Pass B computed from the records |
| Material fact left out: a decline, a concentration, growth from one source | block / warn | profile in code, two Jev answers must agree |
| Claim contradicted / unsupported by the data | block / warn | Jev (grounded) |
| Citation contradicted / unsupported by its source | block / warn | Jev (grounded) |
| Rhetorical spin (1 to 5), weak sourcing | warn | Jev |
| Genre, ungrounded claims, no caveats | note | Jev |
Supervisory tiers: a Jev-based violation at 85% confidence or more acts on its own; between
50% and 85%, or when two atomic answers disagree, a person reviews it; below 50% it isn't
flagged but stays in the report's audit trace. A check passes silently only when code made
the comparison and any matching Jev did was at least 90% sure. All of Jev's questions and
every threshold live in one file, src/factlabel/questions.py.
factlabel eval scores the audit on paired drafts: a misleading draft and an honest draft of
the same update, over the same data. Each misleading draft lists the failure types it should
trigger, so precision and recall are measured per failure type.
factlabel eval evals/ --runs 3 # 30 drafts across fundraising, SaaS usage, marketing ROI
factlabel eval evals-holdout/ --runs 3 # a retail domain written after the engine was built
Results against live Jev (jev-1.13), 3 runs each:
| Before the tiered engine | Now | |
|---|---|---|
| Misleading drafts blocked | 80% | 100% |
| Honest drafts passed | 87% | 100% |
| Omission precision | 50% | 100% |
| Spin recall | 78% | 100% |
| Every other failure type (precision / recall) | 100% / 100% | 100% / 100% |
| Jev cost per full run (90 audits) | $0.019 | $0.034 |
Read these numbers with care. The engine was tuned on the main suite, so it measures fit, not generalization. The retail domain was written after tuning and run once blind: every misleading draft was blocked, but it exposed two real flaws (attribution judged against the level instead of the change, and a part of a metric read as the whole), and 2 of 3 honest drafts passed. After fixing both, it scores 100% too, so it is no longer truly held out. New example cases are the most useful contribution to this project.
-
Jev is a paid, closed, hosted service. factlabel's code is open source, but its judgment step calls TypeSafe's API with your key. The example audit costs about $0.0003. If you depend on factlabel, know that you depend on TypeSafe too. Everything that decides whether a number is right is plain code in this repository.
-
Claude is optional. Only the revise loop uses it, and a rewrite costs a few cents with Claude Opus 5. Any other model or agent can be plugged in instead.
-
Nothing else is sent anywhere. The annotated pages and labels are self-contained HTML files with no external requests.
-
Jev's answers are probabilities. Treat
reviewas "a person must look", not "probably fine". Tune the thresholds on your own drafts before relying on them. -
Figure extraction is regex-based and English-only.
-
Trim long source documents before auditing; accuracy drops as irrelevant context grows.
-
The questions were tuned on four synthetic domains. Real agent drafts will find new failure modes; add them to the benchmark before changing the questions.
Contributions are welcome, especially new example cases and better questions. See CONTRIBUTING.md. You don't need any API keys to run the tests.
Apache-2.0. Jev and Claude are used through their own APIs under their providers' terms; no part of either is included here.