{"slug": "writing-evidence-linked-docs-exposed-two-missing-regression-tests", "title": "Writing evidence-linked docs exposed two missing regression tests", "summary": "Shiki Yusuke, the developer behind the open-source CLI agent-cost, used his evidence-docs tool to create evidence-linked documentation for the project. This process exposed two missing regression tests in the codebase, which were added via pull requests without any implementation changes. The gaps were in the cache-write TTL breakdown logic and the pricing-status aggregation logic, where the middle cases were untested.", "body_md": "*This is a crosspost of the canonical version on GitHub.*\n\nI run [ agent-cost](https://github.com/shiki-yusuke/agent-cost), a small open-source CLI that reads local Claude Code / Codex CLI usage logs and estimates token cost. I wanted documentation for it more trustworthy than a hand-written README: not prose claiming \"the reader handles the TTL cache-write breakdown correctly,\" but claims that each point at the exact test or source line backing them, checked against real git history so the pointer can't silently go stale.\n\nTo do that I used a second tool I've been building, [ evidence-docs](https://github.com/shiki-yusuke/evidence-docs), to build a\n\n`agent-cost`\n\n: 17 individual `observation`\n\ns grouped into 5 `topic`\n\ns (pricing-catalog validation rules, the cache-write lower-bound behavior, unpriced-fact handling, decimal arithmetic for money, and the `measure`\n\ncommand's v1 contract). Each observation is one statement — a behavior, an invariant, a decision record — with a `claim_kind`\n\n, an `epistemic_status`\n\n, and one or more `provenance`\n\nentries naming the exact file, test, or spec section it's backed by, plus a `content_digest`\n\nof that source at a specific commit.Building that corpus is what surfaced this post's actual finding. To write an honest `provenance`\n\nentry for a behavior claim, I had to point at the test that exercises it — reading the test, not just the source, for every claim. Doing that for the cache-write TTL breakdown logic and the pricing-status aggregation logic turned up two places where the claim I wanted to make (\"the code does X in this case\") was true by reading the code, but had **no test asserting it**. Both went into `docs/claims/gaps.yaml`\n\nas `GAP-01`\n\nand `GAP-02`\n\n, and both became small, scoped pull requests that added regression tests with zero implementation changes.\n\nNeither gap was a live bug — that's the whole point of this post. `agent-cost`\n\n's existing test suite already covered the two extremes of each piece of logic; what was missing was the *middle* case, the one that's easy to reason yourself past while staring at the code.\n\n`agent_cost/readers/claude.py`\n\n, `parse_session_facts`\n\n): when a Claude Code usage event's `cache_creation`\n\nfield is a dict, the reader computes `leftover = cache_creation_input_tokens - (ephemeral_5m + ephemeral_1h)`\n\nand emits it as a `cache_write_unknown`\n\nfact if positive. Existing tests covered a breakdown that exactly accounts for the total, and no breakdown at all — but not a breakdown dict that's `agent_cost/aggregate.py`\n\n, `_STATUS_RANK`\n\n/ `build_rows`\n\n): a row's `pricing_status`\n\nshould be the worst status among its facts, ordered `unpriced`\n\n(0) < `lower_bound`\n\n(1) < `priced`\n\n(2). The existing test only mixed an `unpriced`\n\nfact with a `priced`\n\nfact. No test built a row from a `lower_bound`\n\nfact (e.g. a `cache_write_unknown`\n\ntoken, priced as an explicit floor) mixed with a plain `priced`\n\nfact to confirm the row lands on `lower_bound`\n\n, not `priced`\n\n— the half of the ordering that isn't \"obviously\" covered by the unpriced case.If either had silently broken in a later refactor — reordering `_STATUS_RANK`\n\n's values, or changing the leftover math — nothing in CI would have caught it. The cost-estimation output would have quietly started under- or over-reporting cost floors, without a single red test.\n\nThe obvious response to \"I want good docs for this repo\" is: write a good README, or write good docstrings, and be careful. I already had both — `agent-cost`\n\n's README has a \"What this measures, and what it doesn't\" section, and `price_fact()`\n\nhas a docstring explaining the lower-bound design decision. Careful prose is necessary but not sufficient, for two reasons that showed up directly here:\n\n`provenance`\n\nentry that must name a specific test does.So the fix isn't \"write better docs\" in the abstract — it's structural: every behavior claim needs a machine-checkable pointer to what backs it, narrow enough that \"no test covers this case\" becomes visible while writing it, not something discovered later.\n\n`evidence-docs`\n\nenforces this as a schema-level invariant on every `observation`\n\n, not a style guideline. ** epistemic_status** must be one of a fixed vocabulary (from\n\n`execution_verified`\n\ndown to `model_inference`\n\n/ `single_source_observation`\n\n/ `recorded_decision`\n\n) — you have to say how strongly the claim was checked, not just assert it. `provenance`\n\n`source_kind`\n\n(`source`\n\n, `test`\n\n, `spec`\n\n, or `review_memory`\n\n), a repo-relative `uri`\n\n, a `selector`\n\n(which test/function/section), a `content_digest`\n\n, and the `repo_commit`\n\nthe digest was taken against.Neither field is optional, and both are validated structurally: unknown `source_kind`\n\nvalues are rejected outright, and `content_digest`\n\nis checked against the actual git blob at the declared commit, not the working tree. Two of `agent-cost`\n\n's own observations (OBS-004, the cache-write-TTL claim, and OBS-010, the pricing-status-ranking claim) exist because writing their `provenance`\n\nforced me to go find \"the test that proves this,\" and in both cases the honest answer was \"there isn't one for this specific sub-case\" — which is what `gaps.yaml`\n\nrecords.\n\nAn `evidence-docs`\n\ncorpus is a directory (conventionally `docs/claims/`\n\n) with: `id-registry.yaml`\n\n(every `topic_id`\n\n/`observation_id`\n\nused anywhere must be pre-registered here, closing typo/duplicate ID bugs at generate time); `topics/*.yaml`\n\nand `observations/*.yaml`\n\n(one file per topic by convention, not enforced); and `gaps.yaml`\n\n, a ledger of gaps found (`gap_found: true`\n\n) *or* explicitly searched for and not found (`gap_found: false`\n\n), each tagged with a `taxonomy`\n\n— `test_missing`\n\nfor both GAP-01 and GAP-02 here, `independent_re_search_no_drift`\n\nfor a third entry, GAP-03, where I re-checked three README pricing claims against the current `rates.json`\n\nand found no drift, recorded as a negative result rather than omitted.\n\n`evidence-docs validate docs/claims --repo-commit <sha>`\n\nruns full structural + provenance verification with no output written (CI-friendly); `evidence-docs generate`\n\ndoes the same and then deterministically writes a human-readable `site/index.md`\n\nand an AI-facing `bundle/*.jsonl`\n\n+ `manifest.json`\n\n. `evidence-docs context docs/claims --query '{\"seeds\": {\"paths\": [...]}, \"token_budget\": ...}'`\n\nselects a relevant subset of claims from the bundle for a given set of source paths, for feeding into an LLM context window without shipping the whole corpus. `--generated-at`\n\nand `--repo-commit`\n\nare always explicit CLI arguments, never derived from `datetime.now()`\n\nor `git rev-parse`\n\nat run time, so the same corpus and arguments always produce byte-identical output.\n\nThe parts of `evidence-docs`\n\nthat made this finding possible are the parts designed to fail loudly on drift or forgery, not pass silently:\n\n`git show <repo_commit>:<uri>`\n\n, hashed with sha256), not the current worktree file. This closes a real bypass: checking only the worktree's current hash would still let someone edit the file, recompute the digest, and leave `repo_commit`\n\non the old SHA. A worktree/declared-commit mismatch is a warning (repos move on after a snapshot); a mismatch against the `source_kind`\n\nis a hard rejection, not a skip.`--repo-commit`\n\nmust equal every `valid_at_commit`\n\nand `provenance[].repo_commit`\n\ncorpus-wide`git show`\n\nnever touches the filesystem and needs its own check separate from worktree path resolution.None of this checks whether a *statement* is true (see Boundary, below) — but it does mean a fake or lazy `provenance`\n\nentry gets caught at `validate`\n\ntime rather than trusted forever.\n\nBoth gaps followed the same path: recorded in `gaps.yaml`\n\n, then closed by a small PR that added a test and changed no implementation code.\n\n`shiki-yusuke/agent-cost#2`\n\n`test_cache_creation_partial_ttl_breakdown_leftover_is_unknown`\n\nto `tests/test_reader_claude.py`\n\n: a `cache_creation`\n\ndict with `ephemeral_5m_input_tokens=150`\n\nand `ephemeral_1h_input_tokens=100`\n\nagainst a `cache_creation_input_tokens`\n\ntotal of 300, asserting the reader emits `cache_write_5m=150`\n\n, `cache_write_1h=100`\n\n, `cache_write_unknown=50`\n\nfor the 50-token leftover. No implementation changes — the leftover math already behaved this way; the PR's own description says so.`shiki-yusuke/agent-cost#3`\n\n`tests/test_aggregate.py`\n\n: `test_build_rows_all_priced_facts_mark_row_priced`\n\n(baseline), `test_build_rows_mixed_priced_and_lower_bound_marks_row_lower_bound`\n\n(the previously-untested half — a `lower_bound`\n\nfact plus a `priced`\n\nfact should mark the row `lower_bound`\n\n), `test_build_rows_mixed_lower_bound_and_unpriced_marks_row_unpriced`\n\n, and `test_build_rows_worst_status_ranking_is_order_independent`\n\n(same three facts constructed in three different orders, same resulting status). Again, no implementation changes.Both PRs are public and merged; the diffs and CI runs are the actual evidence for this post, not my summary of them.\n\nTo be precise about scope, since it's easy to over-read a post like this:\n\n`evidence-docs`\n\ndoes not check whether a claim is true.`execution_verified`\n\nnext to a claim backed only by reading the source; the schema's `negation_check`\n\nfield helps catch that but is optional and unenforced.`gaps.yaml`\n\ncompleteness is not verified either.`rates.json`\n\nthat found no drift) exists because I chose to record a negative result, not because anything required it.The minimal path that reproduces the \"write a claim, discover you can't honestly back it\" moment:\n\n```\npip install evidence-docs\n\nevidence-docs init docs/claims\n# scaffolds topics/, observations/, id-registry.yaml, gaps.yaml, README.md\n\n# ... author one observation for a behavior you believe is true, with a\n#     provenance entry naming the exact test that proves it. If you can't\n#     name one, that's the gap. ...\n\nevidence-docs validate docs/claims --repo-commit \"$(git rev-parse HEAD)\"\n```\n\n`validate`\n\nwill reject an unregistered ID, an unknown `source_kind`\n\n, or a digest that doesn't match the git blob at the commit you passed — all useful on their own — but the actual gap-finding step is upstream of the tool: it happens while you're trying to fill in the `provenance`\n\nfield honestly, before you ever run `validate`\n\n.\n\n`shiki-yusuke/agent-cost`\n\n`shiki-yusuke/evidence-docs`\n\n`agent-cost#2`\n\n`agent-cost#3`\n\n`docs/claims/gaps.yaml`\n\nin `agent-cost`\n\n— the gap ledger (GAP-01/02/03)`docs/schema.md`\n\nin `evidence-docs`\n\n— the trust-boundary write-up referenced throughout", "url": "https://wpnews.pro/news/writing-evidence-linked-docs-exposed-two-missing-regression-tests", "canonical_source": "https://dev.to/shikiyusuke/writing-evidence-linked-docs-exposed-two-missing-regression-tests-3524", "published_at": "2026-08-14 23:15:18+00:00", "updated_at": "2026-08-14 23:40:38.268676+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools"], "entities": ["agent-cost", "evidence-docs", "Shiki Yusuke", "Claude Code", "Codex CLI"], "alternates": {"html": "https://wpnews.pro/news/writing-evidence-linked-docs-exposed-two-missing-regression-tests", "markdown": "https://wpnews.pro/news/writing-evidence-linked-docs-exposed-two-missing-regression-tests.md", "text": "https://wpnews.pro/news/writing-evidence-linked-docs-exposed-two-missing-regression-tests.txt", "jsonld": "https://wpnews.pro/news/writing-evidence-linked-docs-exposed-two-missing-regression-tests.jsonld"}}