A couple of months ago, I gave access to my coding agent to do work on my repos. I was worried about hallucinations so I manually checked each transcript against modifications to see whether what the agent said it did actually happened. I found that in many cases, my coding agent either failed to disclose what it actually did or found workarounds to things like failed tests instead of telling me upfront. That led me to the idea for Rashomon.
Rashomon is an independent execution recorder for coding agents that notifies users whenever there is a discrepancy between what the agent said it did and what we capture as a ground truth.
It aims to capture items like shell commands, exit codes, file writes, tool calls, tests, and subagents. We build a timeline of the events but we do not store any prompts, responses, file contents, or tool outputs. A question we got often has been how is this different than a diff. A diff gives you a look at the final result and what changed while what we aim to provide is visibility into the execution that led to it.
Currently, we do not capture network codes, prompts, responses file contents, and tool outputs. We plan to implement some of those based on feedback recently, but please let us know whether you would find that useful.
To get started, you can install it is as a Claude Code plugin or from the command line.
I would appreciate feedback on the solution, but I also would want to see whether anyone can break a test on purpose and see whether Rashomon can capture everything you would like to see in order to trust the independent record.
Comments URL: [https://news.ycombinator.com/item?id=49998416](https://news.ycombinator.com/item?id=49998416)
Points: 1