# We Pointed an Autonomous AI Engineer at Our Own Repo - It Found a Real Bug > Source: > Published: 2026-11-13 00:00:00+00:00 # We Pointed an Autonomous AI Engineer at Our Own Repo - It Found a Real Bug Field report: a free Devin Cloud (SWE-2) session audited the public agentbadge repo and returned 4 findings. We verified each claim by hand with file:line evidence - zero false positives, one real security bug (unauthenticated domain-verification invalidation), and a fix PR the agent opened itself. We ran a free Devin Cloud session against the public agentbadge repo; it returned 4 findings - an unauthenticated challenge endpoint that lets anyone invalidate domain verification, a missing CI gate for bun run gate, a fresh clone that cannot install (file: dependency outside the repo), and committed build artifacts. We hand-verified every claim with file:line evidence: zero false positives, precise counts unverified (Node-vs-Bun methodology caveat), fix PR for the security bug opened by the agent. We build a scanner that grades how ready a website is for AI agents. So we did the obvious thing to our own codebase: we pointed an autonomous AI engineer — Devin Cloud, on its free SWE-2 tier — at our public repository and told it to find something worth fixing. It came back with four findings. **Then we did the part most "AI found a bug" posts skip: we verified every single claim by hand, file and line, before believing it.** Result: zero false positives, one real security bug, and a fix PR the agent opened itself. This is a field report, not a review of Devin. The interesting question for us is not "is the tool good" — it is "can an agent's audit output be trusted, and how do you check it cheaply". That question is our product thesis applied to ourselves. ## What exactly did we ask the agent to do? **Short answer:** one cloud session, one prompt — clone the repo, review the codebase, report concrete improvement suggestions. No hints, no file pointers, no checklist. The run cost nothing: SWE-2 sessions are free (2 concurrent per user on the Teams plan). The session ran on Devin's VM against [spreadzp/agentbadge](https://github.com/spreadzp/agentbadge) at commit `059eea2f`. Under the hood the agent cloned the repo, tried `bun install` (which itself became finding #3), ran TypeScript and the test suite under Node, and read the routes, stores and CI configuration. Total wall time: under an hour, unattended. The full prompt and transcript live in our docs (`docs/DEVIN/LOGS/cloud/step3-*.log`). ## What did it find? **Short answer:** four findings, ordered by value — one genuine security bug, one missing CI gate, one broken contributor experience, one hygiene issue. Every one of them verified against the code. Zero false positives. **F-1, the real bug:** `GET /api/trust/:domain/challenge` issues domain-ownership challenge tokens. It requires no authentication, and every call *overwrites* the stored token for that domain (`challenge-store.ts` literally documents "Overwrites any existing challenge"). Any crawler, link prefetcher or curious visitor hitting that URL invalidates the DNS TXT record a domain owner already published — their `/verify-ownership` then fails with 403. The tokens also live in a process-memory `Map` with no size limit, die on every deploy despite a documented 90-day lifetime, and are compared with plain `!==` instead of a constant-time comparison (`trust.ts:281`). Four sub-problems, one endpoint. **F-2:** no CI job runs our own `bun run gate` script. Three workflows exist — scanner smoke test, architecture-graph check, weekly board conformance — and none of them run `tsc`, `eslint` or `vitest` on a pull request. **F-3:** a fresh clone does not build. `package.json` depends on `@agentbadge/payments` via `file:../../packages/payments` — a path outside the repo. `bun install` fails on it, ~620 test files then fail on required env vars that `.env.example` leaves blank, and 9–11 spec tests read `../docs/EPICS/…` which is also outside the repo. Bonus: our own blog links to `packages/payments/src` on GitHub — a dead link, the directory does not exist. **F-4:** committed build artifacts: an 854 KB `agentbadge-report.json`, a `coverage-summary.json`, and about ten hackathon-era planning docs sitting in the repo root. ## How did we verify the audit before believing it? **Short answer:** the same way we verify any agent's claim — check it against ground truth. Each finding was re-derived from source: route definitions, the store implementation, workflow files, `package.json`, `git ls-files`. Every claim below carries the file:line evidence we checked; the verification took about fifteen minutes of human attention. This is the part the demo transcripts do not show you. Agent audit output is a *claim set*, not a verdict — exactly the same epistemics as our own scanner's evidence model. A claim that cannot name its evidence is worth nothing; a claim that names `trust.ts:281` costs you thirty seconds to confirm. Devin's report named its evidence well enough that all four findings confirmed on first inspection. That is not a small thing — it is the difference between a useful tool and a confidence generator. We also checked what it got *wrong*, because honest field reports include the caveats. Two numbers in the report are overstated by methodology: the TypeScript error count (310) and the test failure count (272 of 830 files) come from a Node.js run, while our suite is Bun-native — the agent itself flagged that Bun's runtime breaks `vi.mocked` and gives misleading results. The direction is right (there is no CI running the gate), the precise counts are not reproducible as stated. We record that as confirmed-with-asterisk, not confirmed. ## What happened after the audit? **Short answer:** we resumed the same cloud session and told it to fix F-1 — replace the stateful token map with a stateless HMAC challenge (`agentbadge-verify=` + `HMAC(secret, domain+epoch)`), extend the trust tests, and open a PR. Stateless derivation removes all four sub-problems at once: nothing to store, nothing to lose on restart, nothing to grow unbounded, constant-time compare. A second session audited agentbadge.xyz itself (~1,970 URLs, read-only): it found five placeholder pages sitting in the sitemap, 252 broken agent-guide links, JSON-LD emitting `"price": "undefined"` on ~170 pages, and a robots.txt where per-bot groups silently cancel the global Disallows for every listed AI crawler. We ran the same claim-checking pass — five out of five P0 findings verified against source (two already fixed in HEAD, pending deploy); the full verdict list lives in `docs/DEVIN/REVIEW/step5-site-audit-review-*.md` alongside the repo audit review. F-2 and F-3 are ours to fix — the agent found them, but CI wiring and dependency vendoring are decisions, not diffs. ## What does this prove about agent-native workflows? **Short answer:** that the bottleneck moved — an autonomous agent produced a competent audit for free in under an hour, and the expensive part became verification and decision-making, exactly the shape our own stack assumes. The workflow that worked: *agent produces claims → human verifies claims against ground truth → agent executes the approved fix → human reviews the diff*. That is the same loop our supply-chain bricks formalize — self-declared claims with provenance, verification as a separate step, write paths gated by verifiable state. We did not design this session to be dogfooding, but it turned into it: an agent audited the agent-infrastructure, produced evidence-backed claims, and the claim-checking took a human fifteen minutes because the evidence was real. The honest limit, stated plainly: one session, one repo, one audit. We are not claiming autonomous engineers replace reviewers — the security bug it found sat in the code for months precisely because it is the kind of thing neither a test suite nor a casual review catches. What it replaces is the blank page: the audit you never ran because nobody had a free day.