{"slug": "i-built-an-open-source-qa-agent-because-ai-shouldnt-grade-its-own-homework", "title": "I Built an Open-Source QA Agent Because AI Shouldn’t Grade Its Own Homework", "summary": "A developer built MaruCheck, an open-source QA and verification tool that creates an independent verification layer between AI coding agents and shipped software. The tool uses a 'Quality Contract' to maintain a persistent representation of intended product behavior, detecting semantic changes rather than silently rewriting tests to match new implementations. It also analyzes diffs to assess risk and prioritize verification for critical code changes.", "body_md": "AI coding tools are getting ridiculously good.\n\nYou can describe a feature, let an agent implement it, ask it to write tests, and sometimes have something working in minutes.\n\nBut there’s a problem I kept thinking about:\n\nWhat happens when the same AI writes the code and then validates its own understanding of the requirement?\n\nIf the model misunderstands the requirement once, it can easily carry that same misunderstanding into the tests.\n\nYou end up with something like this:\n\n```\nProduct requirement\n        ↓\nCoding agent\n        ↓\nIncorrect interpretation\n        ↓\nImplementation\n        ↓\nSame agent writes tests\n        ↓\nTests validate the same interpretation\n        ↓\nEverything passes\n```\n\nThe test suite is green.\n\nThe product is still wrong.\n\nThat idea eventually became **MaruCheck**.\n\nMaruCheck is an open-source QA and verification tool designed around AI-generated software.\n\nThe goal is not to replace Playwright, Vitest, Jest, CI, or existing testing infrastructure.\n\nThe goal is to create an **independent verification layer** between the coding agent and the software being shipped.\n\nThe basic idea looks like this:\n\n```\n             PRODUCT INTENT\n                  │\n          ┌───────┴───────┐\n          ↓               ↓\n    CODING AGENT       MARUCHECK\n          ↓               ↓\n        CODE        QUALITY CONTRACT\n          │               │\n          └───────┬───────┘\n                  ↓\n               VERIFY\n                  ↓\n             PASS / BLOCK\n```\n\nThe coding agent builds.\n\nMaruCheck verifies.\n\nOne of the main ideas behind MaruCheck is something I call a **Quality Contract**.\n\nInstead of generating tests entirely from the implementation, the system keeps a persistent representation of what the feature is actually supposed to do.\n\nA simplified example:\n\n```\nfeature: subscription-management\n\nrequirements:\n  - Free users get 10 generations per month\n  - Pro users have unlimited generations\n  - Successful upgrades take effect immediately\n  - Cancellation keeps Pro access until period_end\n\ninvariants:\n  - Failed payments must never activate Pro\n  - Client-controlled data cannot prove payment\n  - Replayed webhooks must be idempotent\n```\n\nNow imagine an AI agent changes the subscription implementation.\n\nMaruCheck can compare that change against the expected behavior instead of assuming that the new implementation defines the truth.\n\nThis is one of the features I care about most.\n\nSuppose the approved behavior says:\n\n```\nFree users can upload 5 files.\n```\n\nThen an AI-generated change does this:\n\n``` js\n- const FREE_LIMIT = 5\n+ const FREE_LIMIT = 10\n```\n\nA test-generation system could potentially see the new implementation, regenerate the test, and now expect 10.\n\nEverything stays green.\n\nMaruCheck should not do that.\n\nInstead, it treats the mismatch as a semantic change:\n\n```\nSEMANTIC CHANGE DETECTED\n\nExpected:\n5\n\nObserved:\n10\n\nWas this intentional?\n```\n\nThe system can propose changing the contract, but it should not silently rewrite product behavior just to make the tests pass.\n\nThat distinction matters.\n\nAnother thing I didn’t want was a tool that blindly runs every possible test after every change.\n\nNot every diff has the same risk.\n\nChanging a marketing page is not the same as changing:\n\n```\nbilling-webhook.ts\nauthentication.ts\npermissions.ts\nsubscription-state.ts\n```\n\nSo MaruCheck analyzes the diff and tries to understand what is actually affected.\n\nFor example:\n\n```\nChanged:\nbilling-webhook.ts\n\nRisk:\nHIGH\n\nWhy:\n+ payment-sensitive code changed\n+ subscription state transition changed\n+ related historical regression exists\n+ critical Quality Contract affected\n```\n\nThat risk assessment can then influence what verification gets run.\n\nThis became another part of the project that I really liked.\n\nImagine six months ago your application had this bug:\n\n```\nBUG #143\n\nA user could access another user's invoice\nby changing the invoice ID.\n```\n\nThe bug gets fixed.\n\nEveryone moves on.\n\nMonths later, a coding agent modifies invoice authorization again.\n\nA normal testing tool mostly sees the current code.\n\nMaruCheck can remember:\n\n```\nThis area previously caused an authorization regression.\n```\n\nand bring the relevant regression checks back into the verification plan:\n\n```\n✓ cross-account invoice checks\n✓ IDOR verification\n✓ ownership tests\n✓ authorization boundary tests\n```\n\nThe idea is that QA knowledge should accumulate over time instead of disappearing into old tickets and forgotten incidents.\n\nSome of the functionality built so far includes:\n\nThe project is designed to work with existing testing tools rather than trying to reinvent all of them.\n\nThe interesting part, to me, is the reasoning and orchestration layer around those tools.\n\nI also wanted MaruCheck to fit directly into the workflow developers already have with coding agents.\n\nSo an agent like Codex, Claude Code, Cursor, or another MCP-compatible tool can eventually follow a workflow like:\n\n```\nImplementation complete.\n\n> maru_verify_feature(\"subscription cancellation\")\n```\n\nMaruCheck performs an independent verification.\n\nIf it finds something:\n\n```\nBLOCKING ISSUE\n\nSUB-004 violated.\n\nExpected:\nPro access remains active until period_end.\n\nActual:\nAccount becomes FREE immediately.\n```\n\nThe coding agent can fix the implementation and ask MaruCheck to verify it again.\n\nThat creates a loop where the builder and verifier have separate responsibilities.\n\nThe name is inspired by the **Kobayashi Maru** idea.\n\nNot because I wanted to make a Star Trek-themed testing tool, but because I liked the underlying concept:\n\nDon’t only test what happens when everything goes according to plan.\n\nTest what happens when assumptions break.\n\nThat maps nicely to the kind of QA I want MaruCheck to do:\n\n```\nWhat if the webhook fires twice?\n\nWhat if the payment succeeds but the database update fails?\n\nWhat if two requests happen at the same time?\n\nWhat if permissions change during the workflow?\n\nWhat if an old regression comes back?\n\nWhat if the implementation passes its tests\nbut still violates the original product requirement?\n```\n\nHence:\n\n**MaruCheck — Test what your AI didn’t.**\n\nI initially thought about keeping the project more closed, but I eventually decided that open source made much more sense.\n\nThis kind of developer infrastructure gets better when developers can:\n\nI’m especially interested in seeing how it behaves against real projects being built heavily with AI coding agents.\n\nWebsite:\n\nCore repository:\n\n[https://github.com/Kidus-M/MaruCheck](https://github.com/Kidus-M/MaruCheck)\n\nWebsite repository:\n\n[https://github.com/Kidus-M/MaruCheck-Web](https://github.com/Kidus-M/MaruCheck-Web)\n\nIf the idea sounds interesting, I’d really appreciate:\n\nBut the feedback I want most is technical criticism.\n\nDoes the independent-verifier model make sense?\n\nWould you actually keep Quality Contracts inside your repository?\n\nWhat would a tool like this have to catch before you trusted it enough to run on every pull request?\n\nAnd most importantly:\n\n**What am I getting wrong?**\n\nI’d love to hear your thoughts.", "url": "https://wpnews.pro/news/i-built-an-open-source-qa-agent-because-ai-shouldnt-grade-its-own-homework", "canonical_source": "https://dev.to/kidus_m/i-built-an-open-source-qa-agent-because-ai-shouldnt-grade-its-own-homework-454f", "published_at": "2026-08-24 17:23:47+00:00", "updated_at": "2026-08-24 17:45:20.012136+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety"], "entities": ["MaruCheck", "Playwright", "Vitest", "Jest"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-open-source-qa-agent-because-ai-shouldnt-grade-its-own-homework", "markdown": "https://wpnews.pro/news/i-built-an-open-source-qa-agent-because-ai-shouldnt-grade-its-own-homework.md", "text": "https://wpnews.pro/news/i-built-an-open-source-qa-agent-because-ai-shouldnt-grade-its-own-homework.txt", "jsonld": "https://wpnews.pro/news/i-built-an-open-source-qa-agent-because-ai-shouldnt-grade-its-own-homework.jsonld"}}