{"slug": "we-pointed-an-autonomous-ai-engineer-at-our-own-repo-it-found-a-real-bug", "title": "We Pointed an Autonomous AI Engineer at Our Own Repo - It Found a Real Bug", "summary": "A free Devin Cloud session running the SWE-2 tier audited the public agentbadge repository at commit 059eea2f and returned 4 findings that the team hand-verified with file:line evidence, reporting zero false positives. The most serious finding was an unauthenticated GET /api/trust/:domain/challenge endpoint that overwrites the stored challenge token for any domain, letting any crawler or prefetcher invalidate a domain owner's published DNS TXT record and cause /verify-ownership to fail with 403; the agent also opened a fix PR for that bug itself. The remaining findings were a missing CI job for bun run gate, a fresh clone that cannot install because package.json depends on @agentbadge/payments via file:../../packages/payments outside the repo, and committed build artifacts.", "body_md": "# We Pointed an Autonomous AI Engineer at Our Own Repo - It Found a Real Bug\n\nField report: a free Devin Cloud (SWE-2) session audited the public agentbadge repo and returned 4 findings. We verified each claim by hand with file:line evidence - zero false positives, one real security bug (unauthenticated domain-verification invalidation), and a fix PR the agent opened itself.\n\nWe ran a free Devin Cloud session against the public agentbadge repo; it returned 4 findings - an unauthenticated challenge endpoint that lets anyone invalidate domain verification, a missing CI gate for bun run gate, a fresh clone that cannot install (file: dependency outside the repo), and committed build artifacts. We hand-verified every claim with file:line evidence: zero false positives, precise counts unverified (Node-vs-Bun methodology caveat), fix PR for the security bug opened by the agent.\n\n We build a scanner that grades how ready a website is for AI agents. So we did the obvious thing to our own codebase: we pointed an autonomous AI engineer — Devin Cloud, on its free SWE-2 tier — at our public repository and told it to find something worth fixing. It came back with four findings. **Then we did the part most \"AI found a bug\" posts skip: we verified every single claim by hand, file and line, before believing it.** Result: zero false positives, one real security bug, and a fix PR the agent opened itself. \n\nThis is a field report, not a review of Devin. The interesting question for us is not \"is the tool good\" — it is \"can an agent's audit output be trusted, and how do you check it cheaply\". That question is our product thesis applied to ourselves.\n\n## What exactly did we ask the agent to do?\n\n **Short answer:** one cloud session, one prompt — clone the repo, review the codebase, report concrete improvement suggestions. No hints, no file pointers, no checklist. The run cost nothing: SWE-2 sessions are free (2 concurrent per user on the Teams plan). \n\n The session ran on Devin's VM against [spreadzp/agentbadge](https://github.com/spreadzp/agentbadge) at commit `059eea2f`. Under the hood the agent cloned the repo, tried `bun install` (which itself became finding #3), ran TypeScript and the test suite under Node, and read the routes, stores and CI configuration. Total wall time: under an hour, unattended. The full prompt and transcript live in our docs (`docs/DEVIN/LOGS/cloud/step3-*.log`). \n\n## What did it find?\n\n **Short answer:** four findings, ordered by value — one genuine security bug, one missing CI gate, one broken contributor experience, one hygiene issue. Every one of them verified against the code. Zero false positives. \n\n **F-1, the real bug:** `GET /api/trust/:domain/challenge` issues domain-ownership challenge tokens. It requires no authentication, and every call *overwrites* the stored token for that domain (`challenge-store.ts` literally documents \"Overwrites any existing challenge\"). Any crawler, link prefetcher or curious visitor hitting that URL invalidates the DNS TXT record a domain owner already published — their `/verify-ownership` then fails with 403. The tokens also live in a process-memory `Map` with no size limit, die on every deploy despite a documented 90-day lifetime, and are compared with plain `!==` instead of a constant-time comparison (`trust.ts:281`). Four sub-problems, one endpoint. \n\n **F-2:** no CI job runs our own `bun run gate` script. Three workflows exist — scanner smoke test, architecture-graph check, weekly board conformance — and none of them run `tsc`, `eslint` or `vitest` on a pull request. \n\n **F-3:** a fresh clone does not build. `package.json` depends on `@agentbadge/payments` via `file:../../packages/payments` — a path outside the repo. `bun install` fails on it, ~620 test files then fail on required env vars that `.env.example` leaves blank, and 9–11 spec tests read `../docs/EPICS/…` which is also outside the repo. Bonus: our own blog links to `packages/payments/src` on GitHub — a dead link, the directory does not exist. \n\n **F-4:** committed build artifacts: an 854 KB `agentbadge-report.json`, a `coverage-summary.json`, and about ten hackathon-era planning docs sitting in the repo root. \n\n## How did we verify the audit before believing it?\n\n **Short answer:** the same way we verify any agent's claim — check it against ground truth. Each finding was re-derived from source: route definitions, the store implementation, workflow files, `package.json`, `git ls-files`. Every claim below carries the file:line evidence we checked; the verification took about fifteen minutes of human attention. \n\n This is the part the demo transcripts do not show you. Agent audit output is a *claim set*, not a verdict — exactly the same epistemics as our own scanner's evidence model. A claim that cannot name its evidence is worth nothing; a claim that names `trust.ts:281` costs you thirty seconds to confirm. Devin's report named its evidence well enough that all four findings confirmed on first inspection. That is not a small thing — it is the difference between a useful tool and a confidence generator. \n\n We also checked what it got *wrong*, because honest field reports include the caveats. Two numbers in the report are overstated by methodology: the TypeScript error count (310) and the test failure count (272 of 830 files) come from a Node.js run, while our suite is Bun-native — the agent itself flagged that Bun's runtime breaks `vi.mocked` and gives misleading results. The direction is right (there is no CI running the gate), the precise counts are not reproducible as stated. We record that as confirmed-with-asterisk, not confirmed. \n\n## What happened after the audit?\n\n **Short answer:** we resumed the same cloud session and told it to fix F-1 — replace the stateful token map with a stateless HMAC challenge (`agentbadge-verify=` + `HMAC(secret, domain+epoch)`), extend the trust tests, and open a PR. Stateless derivation removes all four sub-problems at once: nothing to store, nothing to lose on restart, nothing to grow unbounded, constant-time compare. \n\n A second session audited agentbadge.xyz itself (~1,970 URLs, read-only): it found five placeholder pages sitting in the sitemap, 252 broken agent-guide links, JSON-LD emitting `\"price\": \"undefined\"` on ~170 pages, and a robots.txt where per-bot groups silently cancel the global Disallows for every listed AI crawler. We ran the same claim-checking pass — five out of five P0 findings verified against source (two already fixed in HEAD, pending deploy); the full verdict list lives in `docs/DEVIN/REVIEW/step5-site-audit-review-*.md` alongside the repo audit review. F-2 and F-3 are ours to fix — the agent found them, but CI wiring and dependency vendoring are decisions, not diffs. \n\n## What does this prove about agent-native workflows?\n\n **Short answer:** that the bottleneck moved — an autonomous agent produced a competent audit for free in under an hour, and the expensive part became verification and decision-making, exactly the shape our own stack assumes. \n\n The workflow that worked: *agent produces claims → human verifies claims against ground truth → agent executes the approved fix → human reviews the diff*. That is the same loop our supply-chain bricks formalize — self-declared claims with provenance, verification as a separate step, write paths gated by verifiable state. We did not design this session to be dogfooding, but it turned into it: an agent audited the agent-infrastructure, produced evidence-backed claims, and the claim-checking took a human fifteen minutes because the evidence was real. \n\nThe honest limit, stated plainly: one session, one repo, one audit. We are not claiming autonomous engineers replace reviewers — the security bug it found sat in the code for months precisely because it is the kind of thing neither a test suite nor a casual review catches. What it replaces is the blank page: the audit you never ran because nobody had a free day.", "url": "https://wpnews.pro/news/we-pointed-an-autonomous-ai-engineer-at-our-own-repo-it-found-a-real-bug", "canonical_source": "https://agentbadge.xyz/blog/devin-cloud-audit", "published_at": "2026-11-13 00:00:00+00:00", "updated_at": "2026-11-13 12:21:21.631892+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools"], "entities": ["Devin Cloud", "SWE-2", "agentbadge", "spreadzp/agentbadge", "@agentbadge/payments", "bun", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-pointed-an-autonomous-ai-engineer-at-our-own-repo-it-found-a-real-bug", "markdown": "https://wpnews.pro/news/we-pointed-an-autonomous-ai-engineer-at-our-own-repo-it-found-a-real-bug.md", "text": "https://wpnews.pro/news/we-pointed-an-autonomous-ai-engineer-at-our-own-repo-it-found-a-real-bug.txt", "jsonld": "https://wpnews.pro/news/we-pointed-an-autonomous-ai-engineer-at-our-own-repo-it-found-a-real-bug.jsonld"}}