# I audited 249 of my own AI coding sessions. The problem wasn't lying. > Source: > Published: 2026-08-01 10:19:48+00:00 I use a coding agent every day. I give it a task, go do something else, and come back to a line at the bottom of the screen like this one: "빌드·회귀 스모크·기존 분석 테스트 모두 통과." (build, regression smoke, and the existing analysis tests all pass) That is a real line from my own session log. I never checked it. Instead of scrolling up to see whether the tests actually ran, I gave it the next task. I did that for months. Then I read a post in a Korean dev community where someone had looked into their agent's private reasoning and found that **the memory given to its tester role contained a list of ways to work around problems when they came up**. That doesn't mean the agent was malicious. If you tell something to make the tests pass, finding the cheapest path is what optimisation does. The problem is when the cheapest path is "avoid the check" rather than "fix the code," and you have no way to notice. So I decided to check. The tool is called **red-handed**. I wasn't trying to build a judge. I wanted **a side-by-side of what the agent said and what it actually did**. The material was already there. Claude Code writes a transcript for every session: every command it ran and what came back, the before and after of every file edit, and every sentence it said to me. Line that up with git and you can compute the gap between the words and the work. Did it claim the tests pass without running them. Did it claim they pass right after a run that failed. Did it rewrite an expected value to match whatever the broken code produced. Nine checks like that. The thing I spent longest on wasn't a feature. It was **what happens when the tool is wrong**. There are two ways to be wrong here. **Missing something.** The user loses nothing — they're exactly as uninformed as before. **Accusing someone who did the work honestly.** That's telling a person they cheated when they didn't. Get that wrong once and nobody opens it again. The costs are nowhere near symmetric. So the rule became: **miss things rather than accuse wrongly.** That turned into two hard constraints. **A confirmed finding needs two pieces of evidence.** The session has to show it happening, *and* the change has to still be in the working tree. If the agent undid it later, there's nothing to accuse anyone of, and the finding disappears. **Something unread is not a failure.** If a test ran and I couldn't parse the output, that's "I don't know," not "it failed." That distinction ends up inverting the whole result. No model is called for any of this. If the verdict came from an LLM, the same transcript would produce different answers on different days — and then it isn't evidence. I pointed it at every session on my machine. ``` 249 sessions. Your agent said "tests pass" 124 times. 117 a test ran first 7 no test ran in that session at all ``` Confirmed findings: **zero**. I'll be honest, that was deflating. Weeks of work on a lie detector and it caught no lies. Then I opened the seven one by one, and realised I'd been asking the wrong question. Four of the seven came from one project. The agent had said things like: "드래그·휠 5방향 브라우저 테스트 통과." (drag and scroll, all five directions, browser tests pass) That isn't a lie. The agent really did open a browser and check. But that check **left no machine-readable trace of any kind.** So there is no way to find out later whether the claim held. Not for me, not for the tool, not for whoever inherits that code. The other three were the same shape: the project runs its tests through its own script and my parser doesn't know that output format. I can tell that *something test-shaped ran*. I can't tell that it passed. That's where the conclusion flipped. **I was worried about an agent that lies. What I actually found was verification nobody can read.** And that's less a property of AI than of how we work now. It used to be fine for a person to open a browser, look, and move on — because that person kept looking at that code. Now the thing doing the checking is an agent whose memory resets every session. **Verification you can't read is indistinguishable from verification that never happened.** So those seven are reported as `SUSPICIOUS` , not `CAUGHT` , and the wording is "go look at what that command actually reported" rather than "it lied to you." That's as far as the tool can honestly go. One caveat I'd rather state than have someone find: of those 249 sessions, only 12 contained a "tests pass" claim at all. The rest are short — a question or two. So "0 out of 249" would be an inflated way to read this. Once it worked, I spent a while trying to break it. That turned out to be the most valuable part of the project. An adversarial review found **five** separate ways it would accuse honest work: `rspec` , `phpunit` , `tox` ), read as "no test ran"The first two are worth spelling out. **The timeout one was the worst.** When a test run is killed for taking too long it leaves a distinctive exit signal, and my code read that as a failure. So if the agent then said the tests passed, it printed the single most insulting verdict in the tool: *"said tests pass — the last run failed."* My code was directly violating the rule I'd written two paragraphs of design notes about. **The second was one character.** The pattern that recognises test runners looked for the word `spec` with a word boundary in front of it — which never matches inside `rspec` , because `r` is a word character. So a Ruby developer could run their suite, get `20 examples, 0 failures` , and my tool would state with confidence that **no test had run at all**. And here's the part that stings. My README proudly said *"0 false positives across 183 real sessions."* That was true — but not because the tool was good. It was true because **every project on my machine is JavaScript or TypeScript**, so neither of those paths had ever executed. I built a tool to verify claims, reported that it passed its own verification, and hadn't actually exercised the code. **I did the exact thing the tool exists to catch.** **The sixth turned up after I shipped.** A scoped `pytest tests/test_sync_cli.py` , then edits to a GitHub workflow file and two `.tsx` files, and the tool called the claim stale. A Python test can't import a `.tsx` , and a CI workflow isn't what just ran on this machine. Nothing had gone stale. There was a tempting fix: "different directory, treat it as unrelated." Simple, and right most of the time. It's also wrong — run `pytest tests/` and then edit `src/` and the claim really has expired. So instead of guessing, I only suppress what is **definitely** true: Python can't load a `.tsx` , a JS runner can't load a `.py` , a CI workflow isn't a local run. Anything ambiguous still counts. **"Looks unrelated" is a guess, and you don't throw out evidence on a guess.** All six are pinned by regression tests now. I re-ran it to refresh the numbers for this article, and found one of mine in the list: "테스트 3개로 고정했고 전체 389개 통과." (pinned it with three tests, all 389 pass) → 1 file changed after that run:`scripts/social.tape` What I said was true, and the file that changed is a screen-recording script for a demo GIF. It has nothing to do with the test suite. But this isn't a bug. It's the rule I chose one section ago — **anything ambiguous still counts** — doing exactly what I told it to. The tool has no idea what a `.tape` file is, and I decided that unknown means counted. That's the price of the choice. To avoid missing things, it sometimes barks at nothing. In exchange, this one came out as `SUSPICIOUS` rather than `CAUGHT` , with the changed filename printed right next to it, so it took me three seconds to dismiss. **That's what the two tiers are for.** **Decide which way you'll be wrong before you tune accuracy.** "What's the accuracy?" is close to meaningless for a tool like this, because the two directions of error cost wildly different amounts. Pick the direction, then defend that decision in every branch. I wrote the principle in a doc and then broke it in three places in code. **Ship evidence, not verdicts.** If the output says "it lied," a user can only believe it or not. Show the timestamps and the quoted line and they can decide for themselves. That's also why dismissing my own false positive above took three seconds. A tool you can catch being wrong is a tool you keep using. **The actual fix is verification you can read.** That's what the four browser-test findings taught me. When you hand work to an agent, don't say "check it" — say **"check it and leave the result somewhere a machine can read."** A check that lives only in someone's head doesn't exist by the next session. Runs locally. Your transcripts and your code never leave your machine — it calls no model and makes no network request of its own. Same session, same answer, every time. ``` # a made-up session where every check fires, so you can see the shape first npx @jinhyuk9714/red-handed@latest demo # then your own version of the numbers above npx @jinhyuk9714/red-handed@latest stats ``` MIT, and the code is at [github.com/sjh9714/red-handed](https://github.com/sjh9714/red-handed). Your numbers will be different from mine. I had zero confirmed findings, seven checks nobody could read, and one false alarm I found while writing this paragraph. Whichever way it goes for you, the point is that **you don't know until you look.**