# Auditable agents: turn the answer into a claim you can check > Source: > Published: 2026-10-09 23:32:01+00:00 A model tells you *"Q2 cloud spend was $45,000 — a 12.5% variance over budget."* Now two uncomfortable questions: Most agent frameworks can't answer either cleanly. The run is a stream of messages; the "reasoning" is prose in a log; the numbers came from wherever the model felt like. When something goes wrong in production, your audit trail is a transcript you read by eye. This is the second post in a short series about building agents whose answers are **checkable**. The [first one](https://dev.to/bzdvdn/your-ai-agent-re-sends-the-email-on-retry-an-outbox-for-side-effects-13id) was about side effects (an outbox so a replay doesn't re-send). This one is about the other half: making the *state* behind an answer inspectable and reproducible. It's the approach behind [reactifact](https://github.com/bzdvdn/reactifact), but the ideas — typed artifacts, typed edges, a content hash over state — are portable. **TL;DR** — Make the answer a *typed artifact* in a versioned context, not a message in a bag. Link each derivation to its inputs as typed edges. Then "why did it say that?" is a graph walk and "is this the same run?" is a hash — not a transcript you read by eye. Runnable offline in about a minute, no API key (the figures are computed in Python, nothing calls a model): ``` pip install reactifact python -m examples.fintech_audit.main # prints the audit report, re-hashes to prove it's reproducible ``` Code: [github.com/bzdvdn/reactifact](https://github.com/bzdvdn/reactifact) | | Typical agent framework | reactifact | |---|---|---| | The answer is | a string in a message log | a typed artifact in the context | | State | a message list | versioned commits you can diff | | "Why?" | read the transcript by eye | walk typed provenance edges | | "Same run?" | trust the prompt and the model | compare a `context_hash` | The core move is boring and it changes everything: **every meaningful thing an agent produces is a typed artifact in one evolving context**, not a message in a ``` Context v1 Question Context v2 + Document, Document Context v3 + Evidence, Evidence Context v4 + Claim Context v5 + VerifiedClaim Context v6 + Answer ``` An artifact is a pydantic model — `Evidence(text=..., source=...,`, locator="budget.csv")`Variance(pct=0.125)`. It has an id, a version, a content hash, and *who produced it*. The context is versioned like a git history: every step is a commit you can diff, roll back, check out. Already that's more than a transcript. But the interesting part is the edges. When a produce derives something, it doesn't write "used budget.csv" into the prose. It records a **typed relation** — a first-class edge in the artifact graph: ``` source = self.effects.create(SourceRef(locator="transactions.csv"), id="ref:tx") table = self.effects.create(Table(rows=...), id="doc:transactions.csv") spend = self.effects.create(Spend(total=45000.0), id="spend:q2") variance = self.effects.create(Variance(pct=0.125), id="variance:q2") answer = self.effects.create(AuditAnswer(text="...$45,000... (+12.5%)"), id="answer:q2") answer.link("supported_by", variance) # claim ← its calculation variance.link("calculated_from", spend) # calculation ← its inputs spend.link("materialized_from", table) # figure ← the materialized table table.link("materialized_from", source) # table ← the source it was read from ``` The relations are queryable (`context.related(answer.id, "supported_by")`), so "why did it say that?" becomes a graph walk, not a grep. And because the graph is built, you can render it — Mermaid in the CLI/dashboard, or a structured report: ``` python from reactifact.audit import build_report, report_to_markdown report = build_report(context, answer) # walks the whole chain, breadth-first print(report_to_markdown(report)) ``` `build_report` returns every artifact that contributed to the answer — each with its **content hash, version, and producing author** — plus the source locators the answer rests on. That's the "why": a machine-checkable provenance chain, not a paragraph you have to believe. ``` # Audit report - context version: 7 - context sha256: `f38c6a42…` - Answer (sha256 c1d0…, by "finalize") - supported_by → Variance (sha256 9a51…, by "compute_variance") - calculated_from → Spend (sha256 4f2c…, by "compute_spend") - materialized_from → Table (sha256 77b1…, locator "budget.csv") ``` Provenance answers "why". **Reproducibility** answers "is this the same run". Because the context is canonical, you can fingerprint it. `context_hash` is a sha256 over the run's state — each artifact's id, type, version and content hash, plus every relation edge. Timestamps are deliberately **excluded**, so two runs that reach the same state hash identically: ``` python from reactifact.audit import context_hash first = context_hash(await run_pipeline()) second = context_hash(await run_pipeline()) assert first == second # reproducible — or it fails loudly ``` That single string is an audit primitive. Save it next to the answer; later, a reviewer re-runs the pipeline (or replays a saved session) and compares: ``` reactifact replay sessions.sqlite3 --session q2 --verify f38c6a42… # exits non-zero on any mismatch ``` No more "the numbers look about right." The state behind the answer either hashes to the recorded fingerprint or it doesn't. A hash only means something if the inputs are controlled. An agent run has three usual sources of nondeterminism, and each has a handle: ``` python from reactifact.replay import ReplayLLM # pass 1 — record a real run resources = RuntimeResources(llm=ReplayLLM("calls.jsonl", mode="record", inner=real_llm)) # pass 2 — reproduce it exactly; a divergent call raises ReplayMiss, never guesses resources = RuntimeResources(llm=ReplayLLM("calls.jsonl", mode="replay")) ``` **Auto-generated ids (`uuid4`) and wall-clock time.** Pass a deterministic id factory and stop seeding artifact data from `time.time()` / `uuid4()`. A recorded model plus `counter_ids()` is often the whole fix. **Order and set iteration.** Prefer stable, content-derived ids and explicit sorting in your produces. To catch a leak, run the pipeline a few times under a recorded model and strict ids and compare the fingerprints: ``` python from reactifact.replay import verify_run report = await verify_run(build, recording="calls.jsonl") # runs it twice assert report.ok, report.hashes # a diff is real nondeterminism in your code ``` That's the difference between "it usually returns the same thing" and "a second run hashes to the same string." Here's the subtle part, and where this connects to the outbox from the first post. A naive "replay" re-runs the agent — which means it can hit the network again, call tools again, and drift. reactifact's replay instead **rebuilds the state from the commit chain without running any agent**: ``` python from reactifact.replay import replay_context, replay_summary context = await replay_context(store, session_id, version=7) # state at commit 7 print(replay_summary(context)) # counts by artifact type, relations, actions ``` Because the commit chain is deterministic, you can reconstruct the exact context at any point — walk the provenance, render the graph, answer "why" — and, since no agent runs, nothing external fires. Pair it with the outbox and a recorded side effect is read back as state instead of being re-sent. The same versioning makes **alternative states** cheap: `context.branch()` to explore two hypotheses, three-way `merge()` with explicit conflicts (no silent last-write-wins), `context.diff(v4, v9)` to see exactly what changed between two turns, `context.checkout(v7)` to move head back and undo a bad step. Time-travel over one artifact graph, not a checkpoint of a message list. Once the run *is* structured state, evaluation stops being `answer ==`. You can score the layers separately: expected_answer ``` Evidence quality · Claim correctness · Provenance grounding · Calculation correctness · Confidence calibration · Answer quality · Source coverage ``` `reactifact.eval` runs multi-level metrics over the final `Context` — including provenance grounding, i.e. "is the answer actually linked to evidence that supports it?" — which is a *structural* check, not a model's opinion. And because the state is reproducible, a metric that passes today passes on replay. The report answers "why" from the **final state**. The same design makes the **runtime trace** worth keeping: a run isn't a wall of log lines you read by eye, it's a directed record of what actually happened. Each agent span carries the artifact type that triggered the agent, and which `Produce`(s) ran for that event — with how many effect operations each authored and how long it took. So "the model said X" decomposes into "this event woke this agent, and *this* produce did the work", not a black box. That view is built in, not bolted on: the local SQLite dashboard (`create_trace_router`) shows a `Consume → Produce` flow on each span, and the same spans go to Langfuse (one child observation per produce, so the waterfall shows each step) or any OTLP collector through one `Tracer`. Audit becomes two views of one thing — the final provenance graph you can hash, and the causal trace that built it. Auditability here is a property of *your* pipeline, and it only holds as far as you make it hold: `context_hash` excludes timestamps, but if your produce calls `uuid4()` or reads the clock into artifact data, two runs will differ — `verify_run` tells you, it doesn't fix it. That's the wager: an answer should be a claim you can check, and the machinery to check it — typed artifacts, typed edges, a content hash over state, replay that reconstructs — is worth building into the framework rather than bolting onto the logs afterwards. The `fintech_audit` example is exactly the scenario above — a variance over two CSVs and a policy doc — with no API key (nothing calls a model; the figures are computed in Python): ``` .venv/bin/python -m examples.fintech_audit.main ``` It prints the computed figures, the audit report with a content hash per artifact, and then runs the pipeline again and asserts the two hashes match — its exit code is a determinism smoke test. `docs/en/replay.md`, `docs/en/durability.md`, `docs/en/observability.md`, `examples/fintech_audit` If you've shipped agents you had to debug at 2am: **what's your audit trail — logs you read by eye, or state you can query and reproduce?** I'd like to hear