{"slug": "how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon", "title": "How I caught an AI-generated Python regression that the tests missed -hackathon project", "summary": "A developer built Cross-Examine, a verification harness for Python changes that replays observed behavior from a base Git revision against a proposed revision to catch regressions that existing tests miss. The tool, developed during OpenAI Build Week with Codex and GPT-5.6, uses deterministic execution for verdicts and preserves receipts with reproducing inputs and outputs. It is Python-only and runs only on trusted repositories, with an offline demo available on GitHub.", "body_md": "The change was small, the diff looked sensible, and the existing tests passed.\n\nThen I tried an empty list.\n\nThe original function returned []. The revised version returned None. Nothing immediately crashed, but code expecting a list had quietly received a different contract.\n\nThat gap is why I built Cross‑Examine.\n\nCross‑Examine is a verification harness for Python changes. Given a base Git revision and a proposed revision, it captures observed behavior from the base, then replays the same inputs against the changed code.\n\nWhen behavior differs, it preserves a receipt:\n\nInput: []\n\nBase output: []\n\nHead output: None\n\nVerdict: BROKEN\n\nThe receipt includes the reproducing input, the exact command, and captured output from both revisions. It is meant to be something a developer can inspect and rerun, rather than a confidence score from a model.\n\nThe model has a deliberately limited role. GPT‑5.6 proposes behavior worth checking through a constrained schema, but it cannot produce a verdict. Deterministic execution runs the comparison, and a pure aggregation step decides the result. Cross‑Examine also runs bounded, derandomized Hypothesis examples to look for counterexamples beyond the initial proposal.\n\nThe idea is not that a tool can prove a PR correct. It is narrower: when an AI-generated change looks plausible and the tests are green, replay behavior that already existed and see whether the change still keeps its promises.\n\nCross‑Examine is currently Python-only, supports a bounded set of call shapes, and should run only on repositories you trust. It executes target code, so it is not a sandbox. Those limits are intentional. An honest UNVERIFIABLE result is better than an unearned “safe.”\n\nI built it during OpenAI Build Week with Codex and GPT‑5.6. Codex helped implement the system. At runtime, GPT‑5.6 proposes checks, while deterministic code owns the verdict.\n\nThe repository includes an offline demo that recreates the empty-list regression without an API key:\n\n[https://github.com/stefbuilds/cross-examine](https://github.com/stefbuilds/cross-examine)\n\nI’d be interested in where behavior replay would help in your workflow, and where you would be cautious about relying on it.", "url": "https://wpnews.pro/news/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon", "canonical_source": "https://dev.to/stefbuilds/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon-project-hhn", "published_at": "2026-08-12 12:10:40+00:00", "updated_at": "2026-08-12 12:17:38.033242+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "artificial-intelligence"], "entities": ["Cross-Examine", "OpenAI", "Codex", "GPT-5.6", "GitHub", "stefbuilds"], "alternates": {"html": "https://wpnews.pro/news/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon", "markdown": "https://wpnews.pro/news/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon.md", "text": "https://wpnews.pro/news/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon.txt", "jsonld": "https://wpnews.pro/news/how-i-caught-an-ai-generated-python-regression-that-the-tests-missed-hackathon.jsonld"}}