This is a crosspost of the canonical version on GitHub.
I run agent-cost, a small open-source CLI that reads local Claude Code / Codex CLI usage logs and estimates token cost. I wanted documentation for it more trustworthy than a hand-written README: not prose claiming "the reader handles the TTL cache-write breakdown correctly," but claims that each point at the exact test or source line backing them, checked against real git history so the pointer can't silently go stale.
To do that I used a second tool I've been building, evidence-docs, to build a
agent-cost
: 17 individual observation
s grouped into 5 topic
s (pricing-catalog validation rules, the cache-write lower-bound behavior, unpriced-fact handling, decimal arithmetic for money, and the measure
command's v1 contract). Each observation is one statement — a behavior, an invariant, a decision record — with a claim_kind
, an epistemic_status
, and one or more provenance
entries naming the exact file, test, or spec section it's backed by, plus a content_digest
of that source at a specific commit.Building that corpus is what surfaced this post's actual finding. To write an honest provenance
entry for a behavior claim, I had to point at the test that exercises it — reading the test, not just the source, for every claim. Doing that for the cache-write TTL breakdown logic and the pricing-status aggregation logic turned up two places where the claim I wanted to make ("the code does X in this case") was true by reading the code, but had no test asserting it. Both went into docs/claims/gaps.yaml
as GAP-01
and GAP-02
, and both became small, scoped pull requests that added regression tests with zero implementation changes.
Neither gap was a live bug — that's the whole point of this post. agent-cost
's existing test suite already covered the two extremes of each piece of logic; what was missing was the middle case, the one that's easy to reason yourself past while staring at the code.
agent_cost/readers/claude.py
, parse_session_facts
): when a Claude Code usage event's cache_creation
field is a dict, the reader computes leftover = cache_creation_input_tokens - (ephemeral_5m + ephemeral_1h)
and emits it as a cache_write_unknown
fact if positive. Existing tests covered a breakdown that exactly accounts for the total, and no breakdown at all — but not a breakdown dict that's agent_cost/aggregate.py
, _STATUS_RANK
/ build_rows
): a row's pricing_status
should be the worst status among its facts, ordered unpriced
(0) < lower_bound
(1) < priced
(2). The existing test only mixed an unpriced
fact with a priced
fact. No test built a row from a lower_bound
fact (e.g. a cache_write_unknown
token, priced as an explicit floor) mixed with a plain priced
fact to confirm the row lands on lower_bound
, not priced
— the half of the ordering that isn't "obviously" covered by the unpriced case.If either had silently broken in a later refactor — reordering _STATUS_RANK
's values, or changing the leftover math — nothing in CI would have caught it. The cost-estimation output would have quietly started under- or over-reporting cost floors, without a single red test.
The obvious response to "I want good docs for this repo" is: write a good README, or write good docstrings, and be careful. I already had both — agent-cost
's README has a "What this measures, and what it doesn't" section, and price_fact()
has a docstring explaining the lower-bound design decision. Careful prose is necessary but not sufficient, for two reasons that showed up directly here:
provenance
entry that must name a specific test does.So the fix isn't "write better docs" in the abstract — it's structural: every behavior claim needs a machine-checkable pointer to what backs it, narrow enough that "no test covers this case" becomes visible while writing it, not something discovered later.
evidence-docs
enforces this as a schema-level invariant on every observation
, not a style guideline. ** epistemic_status** must be one of a fixed vocabulary (from
execution_verified
down to model_inference
/ single_source_observation
/ recorded_decision
) — you have to say how strongly the claim was checked, not just assert it. provenance
source_kind
(source
, test
, spec
, or review_memory
), a repo-relative uri
, a selector
(which test/function/section), a content_digest
, and the repo_commit
the digest was taken against.Neither field is optional, and both are validated structurally: unknown source_kind
values are rejected outright, and content_digest
is checked against the actual git blob at the declared commit, not the working tree. Two of agent-cost
's own observations (OBS-004, the cache-write-TTL claim, and OBS-010, the pricing-status-ranking claim) exist because writing their provenance
forced me to go find "the test that proves this," and in both cases the honest answer was "there isn't one for this specific sub-case" — which is what gaps.yaml
records.
An evidence-docs
corpus is a directory (conventionally docs/claims/
) with: id-registry.yaml
(every topic_id
/observation_id
used anywhere must be pre-registered here, closing typo/duplicate ID bugs at generate time); topics/*.yaml
and observations/*.yaml
(one file per topic by convention, not enforced); and gaps.yaml
, a ledger of gaps found (gap_found: true
) or explicitly searched for and not found (gap_found: false
), each tagged with a taxonomy
— test_missing
for both GAP-01 and GAP-02 here, independent_re_search_no_drift
for a third entry, GAP-03, where I re-checked three README pricing claims against the current rates.json
and found no drift, recorded as a negative result rather than omitted.
evidence-docs validate docs/claims --repo-commit <sha>
runs full structural + provenance verification with no output written (CI-friendly); evidence-docs generate
does the same and then deterministically writes a human-readable site/index.md
and an AI-facing bundle/*.jsonl
manifest.json
. evidence-docs context docs/claims --query '{"seeds": {"paths": [...]}, "token_budget": ...}'
selects a relevant subset of claims from the bundle for a given set of source paths, for feeding into an LLM context window without shipping the whole corpus. --generated-at
and --repo-commit
are always explicit CLI arguments, never derived from datetime.now()
or git rev-parse
at run time, so the same corpus and arguments always produce byte-identical output.
The parts of evidence-docs
that made this finding possible are the parts designed to fail loudly on drift or forgery, not pass silently:
git show <repo_commit>:<uri>
, hashed with sha256), not the current worktree file. This closes a real bypass: checking only the worktree's current hash would still let someone edit the file, recompute the digest, and leave repo_commit
on the old SHA. A worktree/declared-commit mismatch is a warning (repos move on after a snapshot); a mismatch against the source_kind
is a hard rejection, not a skip.--repo-commit
must equal every valid_at_commit
and provenance[].repo_commit
corpus-widegit show
never touches the filesystem and needs its own check separate from worktree path resolution.None of this checks whether a statement is true (see Boundary, below) — but it does mean a fake or lazy provenance
entry gets caught at validate
time rather than trusted forever.
Both gaps followed the same path: recorded in gaps.yaml
, then closed by a small PR that added a test and changed no implementation code.
shiki-yusuke/agent-cost#2
test_cache_creation_partial_ttl_breakdown_leftover_is_unknown
to tests/test_reader_claude.py
: a cache_creation
dict with ephemeral_5m_input_tokens=150
and ephemeral_1h_input_tokens=100
against a cache_creation_input_tokens
total of 300, asserting the reader emits cache_write_5m=150
, cache_write_1h=100
, cache_write_unknown=50
for the 50-token leftover. No implementation changes — the leftover math already behaved this way; the PR's own description says so.shiki-yusuke/agent-cost#3
tests/test_aggregate.py
: test_build_rows_all_priced_facts_mark_row_priced
(baseline), test_build_rows_mixed_priced_and_lower_bound_marks_row_lower_bound
(the previously-untested half — a lower_bound
fact plus a priced
fact should mark the row lower_bound
), test_build_rows_mixed_lower_bound_and_unpriced_marks_row_unpriced
, and test_build_rows_worst_status_ranking_is_order_independent
(same three facts constructed in three different orders, same resulting status). Again, no implementation changes.Both PRs are public and merged; the diffs and CI runs are the actual evidence for this post, not my summary of them.
To be precise about scope, since it's easy to over-read a post like this:
evidence-docs
does not check whether a claim is true.execution_verified
next to a claim backed only by reading the source; the schema's negation_check
field helps catch that but is optional and unenforced.gaps.yaml
completeness is not verified either.rates.json
that found no drift) exists because I chose to record a negative result, not because anything required it.The minimal path that reproduces the "write a claim, discover you can't honestly back it" moment:
pip install evidence-docs
evidence-docs init docs/claims
evidence-docs validate docs/claims --repo-commit "$(git rev-parse HEAD)"
validate
will reject an unregistered ID, an unknown source_kind
, or a digest that doesn't match the git blob at the commit you passed — all useful on their own — but the actual gap-finding step is upstream of the tool: it happens while you're trying to fill in the provenance
field honestly, before you ever run validate
.
shiki-yusuke/agent-cost
shiki-yusuke/evidence-docs
agent-cost#2
agent-cost#3
docs/claims/gaps.yaml
in agent-cost
— the gap ledger (GAP-01/02/03)docs/schema.md
in evidence-docs
— the trust-boundary write-up referenced throughout