{"slug": "i-built-a-github-app-that-auto-generates-adversarial-tests-for-ai-written-code-s", "title": "I built a GitHub App that auto-generates adversarial tests for AI-written code — here's how it works", "summary": "A developer built Khwand, a GitHub App that automatically generates adversarial tests for AI-written code. The app uses AST extraction, intent analysis via a ReAct agent, and adversarial test generation to catch semantic errors that traditional tests miss. It also computes a stability score and can auto-fix failing tests via a LangGraph pipeline.", "body_md": "Six months ago I kept reading the same story. Developer uses Cursor or Claude Code to ship a feature. CI goes green. Merge lands. Three days later, production breaks in a way no test caught.\n\nThe failure mode isn't the model being wrong. It's that the tests being run were never designed for what an AI agent might do. The agent writes code that's syntactically correct, type-safe, and passes every existing check — but introduces a semantic error nobody scripted a test for.\n\nSo I built Khwand: a GitHub App that generates those tests automatically on every push.\n\nHow it works\n\nWhen a push event hits the webhook, Khwand does four things:\n\n- AST extraction via tree-sitter\nWe parse the changed Python files and extract function signatures, bodies, docstrings, and type annotations as structured data:\n\n`_pythonparser = Parser()`\n\nparser.set_language(PY_LANGUAGE)\n\ntree = parser.parse(bytes(source_code, \"utf8\"))\n\nfor node in traverse(tree.root_node):\n\nif node.type == \"function_definition\":\n\nfunctions.append(extract_function_data(node, source_code))_\n\n- Intent extraction via ReAct agent\nA ReAct agent running on Groq reads each function and produces a structured intent spec: what this function should always do, what it should never do, and what edge cases are most likely to break it.\n- Adversarial test generation\nA Planner Agent generates targeted tests based on that spec — not generic pytest boilerplate, but edge cases specific to this function's risk surface. It cross-references a pgvector database of historical failure patterns to check if similar functions have failed in specific ways before.\n- Stability Score\nResults feed into a 0–100 score across 7 dimensions: test coverage, edge case handling, model compatibility, intent alignment, regression risk, security posture, and heal rate.\nIf tests fail, a LangGraph graph (analyze → fix → validate, max 3 iterations) generates a patch and opens a fix PR automatically.\nThe hardest unsolved problem\nNon-determinism. The same function can get slightly different generated tests across runs, which means a function can \"pass\" on one push and \"fail\" on the next without any code change. I'm currently using semantic caching (sentence-transformers embeddings) to reuse test suites for similar functions, which helps but doesn't fully solve it.\nIf you've dealt with non-determinism in LLM-generated test suites I'd genuinely love to hear how you handled it.\nStack: FastAPI · LangGraph · Groq · Anthropic Claude · tree-sitter · Supabase · Next.js · GitHub App webhooks\nPython-first right now. Free during early access: khwand.com", "url": "https://wpnews.pro/news/i-built-a-github-app-that-auto-generates-adversarial-tests-for-ai-written-code-s", "canonical_source": "https://dev.to/fahadhafeeezofficial/i-built-a-github-app-that-auto-generates-adversarial-tests-for-ai-written-code-heres-how-it-works-3371", "published_at": "2026-06-19 09:56:20+00:00", "updated_at": "2026-06-19 10:07:04.428037+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "large-language-models", "ai-agents", "ai-safety"], "entities": ["Khwand", "GitHub", "Cursor", "Claude Code", "Groq", "Anthropic", "LangGraph", "Supabase"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-github-app-that-auto-generates-adversarial-tests-for-ai-written-code-s", "markdown": "https://wpnews.pro/news/i-built-a-github-app-that-auto-generates-adversarial-tests-for-ai-written-code-s.md", "text": "https://wpnews.pro/news/i-built-a-github-app-that-auto-generates-adversarial-tests-for-ai-written-code-s.txt", "jsonld": "https://wpnews.pro/news/i-built-a-github-app-that-auto-generates-adversarial-tests-for-ai-written-code-s.jsonld"}}