I built a GitHub App that auto-generates adversarial tests for AI-written code — here's how it works

Six months ago I kept reading the same story. Developer uses Cursor or Claude Code to ship a feature. CI goes green. Merge lands. Three days later, production breaks in a way no test caught.

The failure mode isn't the model being wrong. It's that the tests being run were never designed for what an AI agent might do. The agent writes code that's syntactically correct, type-safe, and passes every existing check — but introduces a semantic error nobody scripted a test for.

So I built Khwand: a GitHub App that generates those tests automatically on every push.

How it works

When a push event hits the webhook, Khwand does four things:

• AST extraction via tree-sitter We parse the changed Python files and extract function signatures, bodies, docstrings, and type annotations as structured data:

`_pythonparser = Parser()`

parser.set_language(PY_LANGUAGE)

tree = parser.parse(bytes(source_code, "utf8"))

for node in traverse(tree.root_node):

if node.type == "function_definition":

functions.append(extract_function_data(node, source_code))_

• Intent extraction via ReAct agent A ReAct agent running on Groq reads each function and produces a structured intent spec: what this function should always do, what it should never do, and what edge cases are most likely to break it.
• Adversarial test generation A Planner Agent generates targeted tests based on that spec — not generic pytest boilerplate, but edge cases specific to this function's risk surface. It cross-references a pgvector database of historical failure patterns to check if similar functions have failed in specific ways before.
• Stability Score Results feed into a 0–100 score across 7 dimensions: test coverage, edge case handling, model compatibility, intent alignment, regression risk, security posture, and heal rate. If tests fail, a LangGraph graph (analyze → fix → validate, max 3 iterations) generates a patch and opens a fix PR automatically. The hardest unsolved problem Non-determinism. The same function can get slightly different generated tests across runs, which means a function can "pass" on one push and "fail" on the next without any code change. I'm currently using semantic caching (sentence-transformers embeddings) to reuse test suites for similar functions, which helps but doesn't fully solve it. If you've dealt with non-determinism in LLM-generated test suites I'd genuinely love to hear how you handled it. Stack: FastAPI · LangGraph · Groq · Anthropic Claude · tree-sitter · Supabase · Next.js · GitHub App webhooks Python-first right now. Free during early access: khwand.com