A model tells you "Q2 cloud spend was $45,000 — a 12.5% variance over budget."
Now two uncomfortable questions:
Most agent frameworks can't answer either cleanly. The run is a stream of
messages; the "reasoning" is prose in a log; the numbers came from wherever the
model felt like. When something goes wrong in production, your audit trail is a
transcript you read by eye.
This is the second post in a short series about building agents whose answers are
checkable. The first one was about side effects (an outbox
so a replay doesn't re-send). This one is about the other half: making the state
behind an answer inspectable and reproducible. It's the approach behind
reactifact, but the ideas — typed
artifacts, typed edges, a content hash over state — are portable.
TL;DR — Make the answer a typed artifact in a versioned context, not a
message in a bag. Link each derivation to its inputs as typed edges. Then "why
did it say that?" is a graph walk and "is this the same run?" is a hash — not a
transcript you read by eye.
Runnable offline in about a minute, no API key (the figures are computed in
Python, nothing calls a model):
pip install reactifact
python -m examples.fintech_audit.main # prints the audit report, re-hashes to prove it's reproducible
Code: github.com/bzdvdn/reactifact
| Typical agent framework | reactifact | |
|---|---|---|
| The answer is | a string in a message log | a typed artifact in the context |
| State | a message list | versioned commits you can diff |
| "Why?" | read the transcript by eye | walk typed provenance edges |
| "Same run?" | trust the prompt and the model | compare a context_hash |
The core move is boring and it changes everything: every meaningful thing an agent produces is a typed artifact in one evolving context, not a message in a
Context v1 Question
Context v2 + Document, Document
Context v3 + Evidence, Evidence
Context v4 + Claim
Context v5 + VerifiedClaim
Context v6 + Answer
An artifact is a pydantic model — Evidence(text=..., source=...,,
locator="budget.csv")Variance(pct=0.125). It has an id, a version, a
content hash, and who produced it. The context is versioned like a git history:
every step is a commit you can diff, roll back, check out.
Already that's more than a transcript. But the interesting part is the edges.
When a produce derives something, it doesn't write "used budget.csv" into the
prose. It records a typed relation — a first-class edge in the artifact
graph:
source = self.effects.create(SourceRef(locator="transactions.csv"), id="ref:tx")
table = self.effects.create(Table(rows=...), id="doc:transactions.csv")
spend = self.effects.create(Spend(total=45000.0), id="spend:q2")
variance = self.effects.create(Variance(pct=0.125), id="variance:q2")
answer = self.effects.create(AuditAnswer(text="...$45,000... (+12.5%)"), id="answer:q2")
answer.link("supported_by", variance) # claim ← its calculation
variance.link("calculated_from", spend) # calculation ← its inputs
spend.link("materialized_from", table) # figure ← the materialized table
table.link("materialized_from", source) # table ← the source it was read from
The relations are queryable (context.related(answer.id, "supported_by")), so
"why did it say that?" becomes a graph walk, not a grep. And because the graph is
built, you can render it — Mermaid in the CLI/dashboard, or a structured report:
from reactifact.audit import build_report, report_to_markdown
report = build_report(context, answer) # walks the whole chain, breadth-first
print(report_to_markdown(report))
build_report returns every artifact that contributed to the answer — each with
its content hash, version, and producing author — plus the source locators
the answer rests on. That's the "why": a machine-checkable provenance chain,
not a paragraph you have to believe.
- context version: 7
- context sha256: `f38c6a42…`
- Answer (sha256 c1d0…, by "finalize")
- supported_by → Variance (sha256 9a51…, by "compute_variance")
- calculated_from → Spend (sha256 4f2c…, by "compute_spend")
- materialized_from → Table (sha256 77b1…, locator "budget.csv")
Provenance answers "why". Reproducibility answers "is this the same run".
Because the context is canonical, you can fingerprint it. context_hash is a
sha256 over the run's state — each artifact's id, type, version and content hash,
plus every relation edge. Timestamps are deliberately excluded, so two runs
that reach the same state hash identically:
from reactifact.audit import context_hash
first = context_hash(await run_pipeline())
second = context_hash(await run_pipeline())
assert first == second # reproducible — or it fails loudly
That single string is an audit primitive. Save it next to the answer; later, a
reviewer re-runs the pipeline (or replays a saved session) and compares:
reactifact replay sessions.sqlite3 --session q2 --verify f38c6a42…
No more "the numbers look about right." The state behind the answer either
hashes to the recorded fingerprint or it doesn't.
A hash only means something if the inputs are controlled. An agent run has three
usual sources of nondeterminism, and each has a handle:
from reactifact.replay import ReplayLLM
resources = RuntimeResources(llm=ReplayLLM("calls.jsonl", mode="record", inner=real_llm))
resources = RuntimeResources(llm=ReplayLLM("calls.jsonl", mode="replay"))
Auto-generated ids (uuid4) and wall-clock time. Pass a deterministic id
factory and stop seeding artifact data from time.time() / uuid4(). A
recorded model plus counter_ids() is often the whole fix.
Order and set iteration. Prefer stable, content-derived ids and explicit
sorting in your produces.
To catch a leak, run the pipeline a few times under a recorded model and strict
ids and compare the fingerprints:
from reactifact.replay import verify_run
report = await verify_run(build, recording="calls.jsonl") # runs it twice
assert report.ok, report.hashes # a diff is real nondeterminism in your code
That's the difference between "it usually returns the same thing" and "a second
run hashes to the same string."
Here's the subtle part, and where this connects to the outbox from the first post.
A naive "replay" re-runs the agent — which means it can hit the network again,
call tools again, and drift. reactifact's replay instead rebuilds the state from the commit chain without running any agent:
from reactifact.replay import replay_context, replay_summary
context = await replay_context(store, session_id, version=7) # state at commit 7
print(replay_summary(context)) # counts by artifact type, relations, actions
Because the commit chain is deterministic, you can reconstruct the exact context
at any point — walk the provenance, render the graph, answer "why" — and, since
no agent runs, nothing external fires. Pair it with the outbox and a recorded
side effect is read back as state instead of being re-sent.
The same versioning makes alternative states cheap: context.branch() to
explore two hypotheses, three-way merge() with explicit conflicts (no silent
last-write-wins), context.diff(v4, v9) to see exactly what changed between two
turns, context.checkout(v7) to move head back and undo a bad step. Time-travel over one artifact
graph, not a checkpoint of a message list.
Once the run is structured state, evaluation stops being answer ==. You can score the layers separately:
expected_answer
Evidence quality · Claim correctness · Provenance grounding ·
Calculation correctness · Confidence calibration · Answer quality · Source coverage
reactifact.eval runs multi-level metrics over the final Context — including
provenance grounding, i.e. "is the answer actually linked to evidence that
supports it?" — which is a structural check, not a model's opinion. And because
the state is reproducible, a metric that passes today passes on replay.
The report answers "why" from the final state. The same design makes the
runtime trace worth keeping: a run isn't a wall of log lines you read by
eye, it's a directed record of what actually happened. Each agent span carries
the artifact type that triggered the agent, and which Produce(s) ran for that
event — with how many effect operations each authored and how long it took. So
"the model said X" decomposes into "this event woke this agent, and this
produce did the work", not a black box.
That view is built in, not bolted on: the local SQLite dashboard
(create_trace_router) shows a Consume → Produce flow on each span, and the
same spans go to Langfuse (one child observation per produce, so the waterfall
shows each step) or any OTLP collector through one Tracer. Audit becomes two
views of one thing — the final provenance graph you can hash, and the causal
trace that built it.
Auditability here is a property of your pipeline, and it only holds as far as
you make it hold:
context_hash excludes timestamps, but if your
produce calls uuid4() or reads the clock into artifact data, two runs will
differ — verify_run tells you, it doesn't fix it.
That's the wager: an answer should be a claim you can check, and the machinery to
check it — typed artifacts, typed edges, a content hash over state, replay that
reconstructs — is worth building into the framework rather than bolting onto the
logs afterwards.
The fintech_audit example is exactly the scenario above — a variance over two
CSVs and a policy doc — with no API key (nothing calls a model; the figures are
computed in Python):
.venv/bin/python -m examples.fintech_audit.main
It prints the computed figures, the audit report with a content hash per
artifact, and then runs the pipeline again and asserts the two hashes match — its
exit code is a determinism smoke test.
docs/en/replay.md, docs/en/durability.md, docs/en/observability.md,
examples/fintech_audit
If you've shipped agents you had to debug at 2am: what's your audit trail — logs you read by eye, or state you can query and reproduce? I'd like to hear