An append-only audit log caught two accounting bugs in a 216-star usage tracker

An append-only audit log built by a developer caught two accounting bugs in splitrail, a 216-star Rust usage tracker for agentic CLIs. The log revealed that Claude Code rewrites session files in place, causing usage data to vanish, and that subagent transcripts were being missed. Both bugs were fixed within twelve days of the first report.

Last month I wrote about auditing 34 days of multi-model Claude Code usage https://dev.to/lizhuojunx86/a-missing-model-line-was-half-my-ai-agent-overspend-auditing-34-days-of-multi-model-claude-code-mc7 and finding that a single missing model line was half my overspend. The core of that piece was a boring engineering decision: ingest every assistant message from Claude Code's session files into an append-only log keyed by message.id , and never let the source rewrite history. This is the follow-up I didn't plan to write. The same log just caught two accounting bugs in splitrail https://github.com/Piebald-AI/splitrail , a 216-star Rust usage tracker for agentic CLIs — and within twelve days of the first report, both were fixed in an official release: one through my PR, one through a maintainer-authored patch that adopted the report's analysis. Here's the play-by-play, with the numbers. Claude Code rewrites session JSONL files in place when you resume or compact a conversation. Between two scans of the same file , five of my assistant messages vanished — usage that was already incurred, gone from the record. Any tracker that recomputes totals from live files inherits this drift: yesterday's numbers change while you sleep. I filed it as 200 https://github.com/Piebald-AI/splitrail/issues/200 . The maintainer confirmed the mechanism the same day — splitrail re-read live files, so messages removed by resume/compaction disappeared from historical totals — and shipped a fix in 3.6.0: a local SQLite history store that persists normalized usage and merges it with current session data before deduplication. Fast, clean work. One wrinkle: he suggested validating by comparing 3.5.9 vs 3.6.0 totals, expecting "3.6.0 should be higher." That's subtly wrong, and the distinction matters for anyone testing this class of fix: on a cold start the two versions are equal — the history store has nothing to restore yet. The divergence only appears across drift events. So I built the test to create one. Everything ran against a frozen snapshot APFS clone of ~/.claude/projects , so both versions scanned identical bytes, with every invocation under an isolated $HOME — own config, own history store, no upload path. Then: | | assertion | result | |---|---|---| | A | cold-start parity: fresh 3.5.9 == fresh 3.6.0 on identical input | ✅ identical | | B | 3.5.9 across a simulated resume/compact rewrite | ✅ drops by exactly the removed usage | | C | 3.6.0 across the same rewrite | ✅ totals unchanged — history store restores | | D | 3.6.0 restart stability, 3 consecutive runs | ✅ byte-identical output | The simulated rewrite removes the last five assistant message-groups from the largest transcript — the same shape as the original drift event. The whole thing is now a self-contained regression fixture PR 208 https://github.com/Piebald-AI/splitrail/pull/208 , since merged : it fails red on 3.5.9 and passes green on 3.6.0, which is exactly what you want a regression test to do. The part I care most about: on the transcripts splitrail scans, 3.6.0 agreed with my append-only log token-exact — 18,548,947 output tokens on both sides across 13.5k messages. Two independent implementations, different languages, different dedup strategies, same number to the digit. The tiny message-count delta was zero-usage api-error records splitrail intentionally skips. When two systems reconcile exactly, every remaining discrepancy is a finding , not noise. Because the log records where every message came from, I could classify all of them against the live tree: | class | files | messages | output tokens | |---|---|---|---| | live, main transcript what splitrail scans | 76 | 13,704 | 18,548,947 | live, subagents/ transcript | 1,423 | 16,160 | 13,738,324 | | vanished file exists, message.id gone | 0 | 0 | 0 | | deleted session file gone | 874 | 11,821 | 14,238,006 | Row two is the second bug. Claude Code writes subagent transcripts Task tool: Explore, general-purpose, custom agents under projects/<slug /<sessionId /subagents/ , at directory depth ≥ 4. Splitrail's discovery hard-caps at depth 2 in three places WalkDir…min depth 2 .max depth 2 , a components == 2 path check, and the glob pattern . Those files are structurally invisible — 54% of my live messages, and roughly a third of the dollars, never entered any total . The model mix makes the blind spot vivid: splitrail saw 22 of my 5,059 Sonnet messages and 0 of 1,566 Haiku messages. Those models run almost exclusively inside subagents. If you delegate heavily to cheap models — which is exactly what cost-conscious agent users do — your tracker undercounts the most, on the workflows you optimized hardest. Filed as 207 https://github.com/Piebald-AI/splitrail/issues/207 with the layout, a two-line repro, and a suggested fix. Two days later it was closed — and not by my patch. The maintainer wrote the fix himself: 209 https://github.com/Piebald-AI/splitrail/pull/209 "Include Claude Code subagent transcripts" +428/−100 across seven files , whose description restates the issue's analysis and adopts its central caution — simply lifting the depth cap isn't enough; subagent records have to flow through the same local hash dedup semantics as main transcripts, or you trade an undercount for a double-count. Both fixes shipped together in splitrail 3.6.1 , which hit Homebrew within a day. On my corpus, the newly visible class is row two of the table above: 16,160 messages and 13.7M output tokens finally entering the totals. A fresh 3.6.1-vs-log reconciliation of the live tree is the obvious next check — part 3 starts there. Live mutable files are not an audit trail. If your source can rewrite history, recomputation is not accounting. Append-only ingest with a stable per-record identity message.id , last-write-wins on partials is cheap insurance — mine is a few hundred lines and a SQLite file. Reconcile token-exact or you know nothing. "Close enough" totals hide entire bug classes. It was exact agreement on the scanned subset that turned the remaining gap into two nameable, fixable defects instead of a shrug. Decompose the gap before you blame anyone. "Your numbers are lower than mine" is an accusation; "the gap is exactly deleted-files + an unscanned directory class, here's the table" is a bug report a maintainer can act on in hours. Both turnarounds here — 200→ 204 in eight days, 207→ 209 in two — happened because the mechanism arrived with the report. Frozen snapshot + isolated $HOME is the whole trick for tracker A/B. Clone the data, pin HOME and XDG STATE HOME / XDG DATA HOME — on Linux, state dirs honor XDG and will escape your sandbox; CodeRabbit's review caught that one in my own fixture, fair is fair , and two binaries scanning identical bytes become a controlled experiment.| date 2026 | event | |---|---| | Jul 11 | | Twelve days, two bugs, zero arguments about whose numbers were right — the log settled every question before it became a debate. All of this ran on TraceGuard https://github.com/lizhuojunx86/traceguard 's routing audit module v1.1.0, "audit evidence layer" — the append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store that part 1 was built on. Stable totals are the substrate; the point of the layer above is stated-vs-revealed routing analysis, priced per decision — which model you said you'd route to, which one actually ran, and what the difference cost. That's part 3, and my totals finally sit still long enough to write it. Apparently not a niche itch, either — the splitrail maintainer starred TraceGuard during this exchange and said he'd be interested in the audit part. pip install traceguard — Apache-2.0. The regression fixture is ~600 lines of stdlib Python + bash, merged in PR 208 https://github.com/Piebald-AI/splitrail/pull/208 ; the subagent fix shipped in splitrail 3.6.1 https://github.com/Piebald-AI/splitrail/releases .