{"slug": "qa-agents-that-test-the-app-themselves-more-bugs-found-at-2-67-per-bug", "title": "QA Agents That Test the App Themselves: More Bugs Found, at $2.67 per Bug", "summary": "Momentic's agent harness Mo found 269 real bugs across four open-source apps — Gitea, Metabase, Twenty and Saleor — at 19.3% recall, 84% precision and $2.67 per bug, beating Codex (86 bugs, 6.2% recall, $0.80 per bug), Claude Code (59 bugs, 4.2% recall, 80% precision, $4.09 per bug) and Devin (27 bugs, 1.9% recall, 84% precision, $3.89 per bug) in a 60-minute time-boxed run against 1,396 human-labeled bugs with no source-code access. Mo's recall advantage grew with time, rising 52% from 177 bugs at 30 minutes to 269 at 60 minutes, versus 16% for Codex. The post cautions the results come from a single run judged partly by a language model with human review, and recommends a small pilot measuring recall, precision and cost per bug before adopting agent-based QA.", "body_md": "# QA Agents That Test the App Themselves: More Bugs Found, at $2.67 per Bug\n\nThe short answer: letting agents test the app themselves finds more real bugs than writing test scripts — but each bug costs more, and precision is slightly lower. The post introduces Momentic's Mo, a custom agent harness built for catching bugs: as it explores, it builds a self-organizing directed acyclic graph, and large product areas automatically fork into parallel sub-agents, each with its own browser or device for concurrent execution. Planner agents work from screenshots and video, while locator agents inspect page structure to turn plain-English targets into actions. When several agents run at once, they share resources under a lease system: shared use is allowed, but only one agent may change a given resource at a time. For an ai software engineer considering agents for QA, this architecture is the starting point for understanding both the capability and the cost.\n\nThe benchmark gives the comparison something to stand on. Four popular open-source apps — Gitea, Metabase, Twenty, and Saleor — were tested against 1,396 human-labeled bugs, with each agent given the same brief, a fresh sandbox, and no access to source code, in a time-boxed 60-minute run. Mo found 269 bugs, with 19.3% recall, 84% precision, and $2.67 per bug. Codex found 86 bugs, with 6.2% recall, 86% precision, and $0.80 per bug. [Claude Code](https://promptcube3.com/en/tags/claude%20code/) found 59 bugs, with 4.2% recall, 80% precision, and $4.09 per bug. Devin found 27 bugs, with 1.9% recall, 84% precision, and $3.89 per bug. Mo's edge is recall: it surfaces the most real bugs. Its costs are precision that is not the best in the group and a per-bug price above Codex.\n\nTime changes the picture further. From 30 minutes to 60 minutes, Mo went from 177 bugs to 269 bugs, a gain of 52%, while Codex gained only 16%. More time helps Mo more than it helps the others, but time also accumulates cost. Two limits belong in any honest reading. The evaluation used a language model as judge with a human reviewer, and the result comes from one time-boxed run on four apps, so it does not prove the same outcome on every product. The post also notes that language models generating code tend to reinforce their own assumptions, write many useless tests, and struggle with visual understanding — the very problems agent testing aims to fix, and a reason it can still go wrong. For an ai software engineer deciding whether to adopt this approach, the practical move is a small pilot on your own app: measure recall, precision, and cost per bug, then choose full adoption, assisted scanning, or staying with scripted tests.\n\n[Next What Changed in Tech Job Ads After ChatGPT: Counting Tools in 118,545 Indonesian Ads →](https://promptcube3.com/en/threads/10213/)\n\n## All Replies （0）\n\nWant a live back-and-forth? [Join the global AI chat room](https://promptcube3.com/en/chat/) — login to talk.\n\nNo replies yet — be the first!", "url": "https://wpnews.pro/news/qa-agents-that-test-the-app-themselves-more-bugs-found-at-2-67-per-bug", "canonical_source": "https://promptcube3.com/en/threads/10214/", "published_at": "2026-11-03 02:12:20+00:00", "updated_at": "2026-11-03 02:19:20.954522+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-products", "artificial-intelligence"], "entities": ["Momentic", "Mo", "Codex", "Claude Code", "Devin", "Gitea", "Metabase", "Saleor"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/qa-agents-that-test-the-app-themselves-more-bugs-found-at-2-67-per-bug", "markdown": "https://wpnews.pro/news/qa-agents-that-test-the-app-themselves-more-bugs-found-at-2-67-per-bug.md", "text": "https://wpnews.pro/news/qa-agents-that-test-the-app-themselves-more-bugs-found-at-2-67-per-bug.txt", "jsonld": "https://wpnews.pro/news/qa-agents-that-test-the-app-themselves-more-bugs-found-at-2-67-per-bug.jsonld"}}