I asked two AI teams to break my bot detector. They found 8 holes, and their fixes opened 3 new ones. A developer ran two AI agent teams, one on Claude and one on Gemini, against their open-source CAPTCHA alternative Synapse Shield, and the agents independently flagged 10 issues, with a red-team agent confirming 8 of 8 claims via working proof-of-concept exploits. Coding agents then produced eleven patches, but three passed their own tests while still being wrong, including a header parser that merged spoofed variants like x_forwarded_for into X-Forwarded-For. All findings are fixed in v0.12.0, and the developer argues every agent-written security fix must be reviewed as if it came from a stranger. Last week I shipped a release where I barely wrote any code myself. AI agents found the bugs, and other agents wrote the fixes. My job was to check every fix as if it had come from a stranger . That turned out to be the most important part. Three of the eleven patches passed their own tests and were still wrong. This post covers what the agents found in my open-source bot detector, how the review process worked, and what I learned about letting AI touch security code. It's a self-hosted replacement for CAPTCHAs. A small browser SDK records how a visitor moves, taps and types. It never records what they type. Your server then scores the session and allows it, blocks it, or asks the browser to solve an invisible proof-of-work. It's written in Python with a Rust core, has SDKs for React, Vue and plain JS, and is MIT licensed. I split the codebase into five areas: detection engine, crypto and tokens, framework adapters and SDKs, Rust core, and monitoring. I gave each area to an AI "owner" with one rule: research only, change nothing, and back every claim with a file and line number . I ran the same assignment twice, once with Claude agents and once with Gemini agents. | Issues found independently by both teams | 10 | | Found only by one team | several each | | Claims later confirmed by a red-team agent writing a working proof of concept | 8 of 8 | The overlap was the useful signal. When two different models independently point at the same line, it's probably real. Then a third agent played attacker. For every claim it wrote a small script that either reproduced the hole or failed to. No reproduction meant no fix. All of these are fixed in v0.12.0. If you run Synapse Shield, please upgrade. 1. Phone-sized screens skipped the behavioural layer critical . If a request said "I support touch" but sent no touch events, the session was supposed to be treated as desktop. That check only ran on screens 1024 px or wider. On a phone-sized screen, a one-click macro scored 0 out of 100 and was allowed. 2. The client could choose its own IP address critical, behind some proxies . Behind a reverse proxy, the server trusted several forwarding headers in a fixed order. If the proxy passed client-sent headers through, an attacker could pick their own IP and dodge bans and rate limits. In development mode, a forged loopback address even unlocked the admin panel. 3. Two framework integrations broke the proof-of-work. They wrapped the challenge response one level deeper than the SDKs expected. Users on those routes got stuck on a 429 they could never clear. 4. The Rust core and the Python fallback disagreed. Leave one field out of the request and the Rust path said "human" while the Python path said "bot". That's a free bypass whenever the fast path is active. 5. A locked database looked like an attack. When SQLite was briefly locked, the nonce check failed. A failed check was counted as a replay, so a real user got a strike against their IP. 6. Very short mouse movements were punished for being short false positives . With 3 or 4 points, some statistics only have one value, so their variance is 0. The rule read that zero as "too perfect, must be a bot". 7. The docs told you to send a field no integration read. 8. The plain-JS SDK didn't solve the proof-of-work. The React and Vue SDKs did. I gave each fix to a coding agent with a strict ticket. It had to: Every agent came back saying it was done and all tests passed. I didn't accept a single "done" without checking. I read every diff, reran every proof of concept myself and wrote my own probes. Three patches were wrong, and every one of them had passed its own tests. The agent's new header parser tried to be helpful. It treated header names written in other ways, such as x forwarded for or http-x-forwarded-for , as the real X-Forwarded-For header and merged their values. My probe added one fake header next to the proxy's real one: php real proxy header only - 203.0.113.5 correct + "http-x-forwarded-for: 9.9.9.9" - 9.9.9.9 attacker wins + "x forwarded for: 8.8.8.8" - 8.8.8.8 attacker wins The agent's tests only covered the attacks from the original report. It closed the reported hole and opened a new one right next to it. In the second round, only the exact canonical header names are read. Rust panics must never kill the Python worker. The agent switched the build from abort to unwind and wrapped every native call so that a PanicException falls back to Python. Its tests injected a fake PanicException class, and they passed. At runtime, though, the real class was never found, because the module the agent looked it up in isn't registered in sys.modules . The code caught every exception except the one it was written for. pyo3 runtime in sys.modules: False resolved PanicException: NeverMatchingPanic In round two the Rust module exports the real type. A test now triggers a real panic and checks that the process survives. The storage fix called three Rust functions "if they exist". None of them did, so the code did nothing at all. It also added a cache with no size limit. Neither problem would ever fail a test. 💡 Lesson: an agent's test suite shows that the agent's idea of the bug is fixed. It doesn't show that the bug is fixed. Someone has to attack the fix with the same suspicion as the original code. Every number below was measured on real people the public SapiMouse dataset, 120 people, and the CMU keystroke dataset , except bot detection, which comes from synthetic generators. | Measurement | v0.11 | v0.12 | |---|---|---| | Real 3–10 point mouse movements blocked | 12.1% | 1.7% | | Real 2 / 6 / 20 s sessions blocked | 2.3 / 0.97 / 0.46% | unchanged | | Real typing sessions blocked CMU, 20,400 | 0.034% | unchanged | | Synthetic bots detected | 100% | 100% | | Rust feature extraction, 300 points | 592 µs | 136 µs | One measurement bug of my own. An agent reported a small regression on 6-second windows. When I re-ran it, I got a third, different number. The cause was the benchmark itself: it reused the same database and IP addresses between runs, so session history from earlier runs leaked into later ones. With one isolated database per run, before and after were identical . The regression never existed, but I would have shipped a wrong number into the docs. What still isn't good enough: on the longest traces 300 points , the 95th-percentile latency is 1.22 ms , over my 1 ms budget. The Rust part is now fast; the remaining time is in Python. That's the next release. The client IP is now read only from the header you name: SYNAPSE PROXY HEADER=x-forwarded-for default SYNAPSE PROXY HEADER=cf-connecting-ip behind Cloudflare: set this If you run behind Cloudflare and don't set this, every visitor will look like a Cloudflare IP. That's why this release is 0.12.0, not a patch, and why both the startup log and synapse-shield doctor warn you about it. pip install -U "synapse-shield server " npm install synapse-shield-react or synapse-shield-vue The React and Vue SDKs are on npm for the first time. 👉 GitHub: 0xStoic-bit/Synapse Shield https://github.com/0xStoic-bit/Synapse Shield If this was useful, a ⭐ on GitHub helps other developers find the project.