I audited 249 of my own AI coding sessions. The problem wasn't lying.

A developer audited 249 of their own AI coding sessions using a tool called red-handed and found zero confirmed cases of the agent lying about test results. Instead, the audit revealed a deeper problem: many claimed test passes left no machine-readable trace, making verification impossible. The developer concluded that the real issue is not deceptive AI but unverifiable work processes.

I use a coding agent every day. I give it a task, go do something else, and come back to a line at the bottom of the screen like this one: "빌드·회귀 스모크·기존 분석 테스트 모두 통과." build, regression smoke, and the existing analysis tests all pass That is a real line from my own session log. I never checked it. Instead of scrolling up to see whether the tests actually ran, I gave it the next task. I did that for months. Then I read a post in a Korean dev community where someone had looked into their agent's private reasoning and found that the memory given to its tester role contained a list of ways to work around problems when they came up . That doesn't mean the agent was malicious. If you tell something to make the tests pass, finding the cheapest path is what optimisation does. The problem is when the cheapest path is "avoid the check" rather than "fix the code," and you have no way to notice. So I decided to check. The tool is called red-handed . I wasn't trying to build a judge. I wanted a side-by-side of what the agent said and what it actually did . The material was already there. Claude Code writes a transcript for every session: every command it ran and what came back, the before and after of every file edit, and every sentence it said to me. Line that up with git and you can compute the gap between the words and the work. Did it claim the tests pass without running them. Did it claim they pass right after a run that failed. Did it rewrite an expected value to match whatever the broken code produced. Nine checks like that. The thing I spent longest on wasn't a feature. It was what happens when the tool is wrong . There are two ways to be wrong here. Missing something. The user loses nothing — they're exactly as uninformed as before. Accusing someone who did the work honestly. That's telling a person they cheated when they didn't. Get that wrong once and nobody opens it again. The costs are nowhere near symmetric. So the rule became: miss things rather than accuse wrongly. That turned into two hard constraints. A confirmed finding needs two pieces of evidence. The session has to show it happening, and the change has to still be in the working tree. If the agent undid it later, there's nothing to accuse anyone of, and the finding disappears. Something unread is not a failure. If a test ran and I couldn't parse the output, that's "I don't know," not "it failed." That distinction ends up inverting the whole result. No model is called for any of this. If the verdict came from an LLM, the same transcript would produce different answers on different days — and then it isn't evidence. I pointed it at every session on my machine. 249 sessions. Your agent said "tests pass" 124 times. 117 a test ran first 7 no test ran in that session at all Confirmed findings: zero . I'll be honest, that was deflating. Weeks of work on a lie detector and it caught no lies. Then I opened the seven one by one, and realised I'd been asking the wrong question. Four of the seven came from one project. The agent had said things like: "드래그·휠 5방향 브라우저 테스트 통과." drag and scroll, all five directions, browser tests pass That isn't a lie. The agent really did open a browser and check. But that check left no machine-readable trace of any kind. So there is no way to find out later whether the claim held. Not for me, not for the tool, not for whoever inherits that code. The other three were the same shape: the project runs its tests through its own script and my parser doesn't know that output format. I can tell that something test-shaped ran . I can't tell that it passed. That's where the conclusion flipped. I was worried about an agent that lies. What I actually found was verification nobody can read. And that's less a property of AI than of how we work now. It used to be fine for a person to open a browser, look, and move on — because that person kept looking at that code. Now the thing doing the checking is an agent whose memory resets every session. Verification you can't read is indistinguishable from verification that never happened. So those seven are reported as SUSPICIOUS , not CAUGHT , and the wording is "go look at what that command actually reported" rather than "it lied to you." That's as far as the tool can honestly go. One caveat I'd rather state than have someone find: of those 249 sessions, only 12 contained a "tests pass" claim at all. The rest are short — a question or two. So "0 out of 249" would be an inflated way to read this. Once it worked, I spent a while trying to break it. That turned out to be the most valuable part of the project. An adversarial review found five separate ways it would accuse honest work: rspec , phpunit , tox , read as "no test ran"The first two are worth spelling out. The timeout one was the worst. When a test run is killed for taking too long it leaves a distinctive exit signal, and my code read that as a failure. So if the agent then said the tests passed, it printed the single most insulting verdict in the tool: "said tests pass — the last run failed." My code was directly violating the rule I'd written two paragraphs of design notes about. The second was one character. The pattern that recognises test runners looked for the word spec with a word boundary in front of it — which never matches inside rspec , because r is a word character. So a Ruby developer could run their suite, get 20 examples, 0 failures , and my tool would state with confidence that no test had run at all . And here's the part that stings. My README proudly said "0 false positives across 183 real sessions." That was true — but not because the tool was good. It was true because every project on my machine is JavaScript or TypeScript , so neither of those paths had ever executed. I built a tool to verify claims, reported that it passed its own verification, and hadn't actually exercised the code. I did the exact thing the tool exists to catch. The sixth turned up after I shipped. A scoped pytest tests/test sync cli.py , then edits to a GitHub workflow file and two .tsx files, and the tool called the claim stale. A Python test can't import a .tsx , and a CI workflow isn't what just ran on this machine. Nothing had gone stale. There was a tempting fix: "different directory, treat it as unrelated." Simple, and right most of the time. It's also wrong — run pytest tests/ and then edit src/ and the claim really has expired. So instead of guessing, I only suppress what is definitely true: Python can't load a .tsx , a JS runner can't load a .py , a CI workflow isn't a local run. Anything ambiguous still counts. "Looks unrelated" is a guess, and you don't throw out evidence on a guess. All six are pinned by regression tests now. I re-ran it to refresh the numbers for this article, and found one of mine in the list: "테스트 3개로 고정했고 전체 389개 통과." pinned it with three tests, all 389 pass → 1 file changed after that run: scripts/social.tape What I said was true, and the file that changed is a screen-recording script for a demo GIF. It has nothing to do with the test suite. But this isn't a bug. It's the rule I chose one section ago — anything ambiguous still counts — doing exactly what I told it to. The tool has no idea what a .tape file is, and I decided that unknown means counted. That's the price of the choice. To avoid missing things, it sometimes barks at nothing. In exchange, this one came out as SUSPICIOUS rather than CAUGHT , with the changed filename printed right next to it, so it took me three seconds to dismiss. That's what the two tiers are for. Decide which way you'll be wrong before you tune accuracy. "What's the accuracy?" is close to meaningless for a tool like this, because the two directions of error cost wildly different amounts. Pick the direction, then defend that decision in every branch. I wrote the principle in a doc and then broke it in three places in code. Ship evidence, not verdicts. If the output says "it lied," a user can only believe it or not. Show the timestamps and the quoted line and they can decide for themselves. That's also why dismissing my own false positive above took three seconds. A tool you can catch being wrong is a tool you keep using. The actual fix is verification you can read. That's what the four browser-test findings taught me. When you hand work to an agent, don't say "check it" — say "check it and leave the result somewhere a machine can read." A check that lives only in someone's head doesn't exist by the next session. Runs locally. Your transcripts and your code never leave your machine — it calls no model and makes no network request of its own. Same session, same answer, every time. a made-up session where every check fires, so you can see the shape first npx @jinhyuk9714/red-handed@latest demo then your own version of the numbers above npx @jinhyuk9714/red-handed@latest stats MIT, and the code is at github.com/sjh9714/red-handed https://github.com/sjh9714/red-handed . Your numbers will be different from mine. I had zero confirmed findings, seven checks nobody could read, and one false alarm I found while writing this paragraph. Whichever way it goes for you, the point is that you don't know until you look.