{"slug": "cross-examine-replay-base-behavior-against-an-ai-generated-pr", "title": "Cross-Examine – replay base behavior against an AI-generated PR", "summary": "Cross-Examine, a verification harness for Codex-authored Python changes, detects behavior regressions that standard test suites miss, returning a BROKEN verdict with a reproducing input for a plausible optimization that returns None for an empty list. The tool, developed during OpenAI Build Week 2026, captures base revision behavior and executes head revisions against the same inputs, providing exact command and output evidence for every VERIFIED or REFUTED finding. A live evidence explorer is available at cross-examine-six.vercel.app, and the tool supports macOS, Linux, and Windows.", "body_md": "Codex writes the code. Cross-Examine puts it on the stand.OpenAI Build Week 2026 ·\n\nTrack: Developer tools(testing · agentic workflows)Git worktrees → GPT-5.6 Sol claims → trusted-input base/head execution → pure\n\n`aggregate()`\n\n→ FastAPI/React report.\n\n**The problem.** Agent-authored code passes the tests that exist. Nothing yet checks whether the\nbehavior it replaced still holds. So, the model fixes one bug, introduces another, and the\nsuite stays green throughout. Anyone merging Codex-authored pull requests is reviewing a\ndiff with no evidence about the behavior that diff silently changed.\n\n**The tool.** Cross-Examine is an independent verification harness for Codex-authored Python\nchanges. It captures the base revision's behavior, executes the head revision against the\nsame inputs, and hunts adversarial boundaries. Newly executed reports that pass pipeline\nvalidation show the exact command and captured output behind every `VERIFIED`\n\nor `REFUTED`\n\nfinding. Abstentions show attempted evidence or a deterministic diagnostic instead of\nfabricating a receipt.\n\n**The catch is the product.** A plausible optimization returns `None`\n\nfor an empty list, the\nexisting happy-path test stays green, and Cross-Examine returns `BROKEN`\n\nwith `[]`\n\nas the\nreproducing input — reproducible in 60 seconds from a clean checkout by the command below.\n\n[Judge quickstart: see the catch in 60 seconds](#judge-quickstart-see-the-catch-in-60-seconds)[How Codex and GPT-5.6 were used](#how-codex-and-gpt-56-were-used)[Why this is not a Codex skill](#why-this-is-not-a-codex-skill)[Architecture](#architecture)[Scope and safety](#scope-and-safety)[License](#license)\n\nAlso in this repo: [requirements](#requirements) · [directory map](#directory-map) · [Windows setup](#windows-powershell-setup) · [real repository runs](#real-repository-run) · [tests](#tests) · [video outline](#three-minute-video-outline)\n\nOn macOS or Linux, allocate a fresh workspace, clear ambient model and run-storage variables, and force the checked-in characterization fixture. The findings still come from the real local pipeline:\n\n```\nhero_workspace=$(mktemp -d)\nenv -u OPENAI_API_KEY -u CROSS_EXAMINE_DB -u CROSS_EXAMINE_RUNS CROSS_EXAMINE_DEMO_CHARACTERIZER=fixture \\\n  uv run --isolated --no-editable cross-examine demo --no-open \\\n  --workspace \"$hero_workspace\"\n```\n\nThe first run in that new workspace reports:\n\n```\nCharacterization: deterministic hero fixture\nVerdict: BROKEN\nCorpus: +2 this run · 2 total\nRefuted claim: preserve-empty\nReproducing input: []\n```\n\nRun the same credential-cleared command again with the same `hero_workspace`\n\n. The\nverdict remains `BROKEN`\n\n; corpus output becomes `+0 this run · 2 total`\n\n. A new workspace\nis what makes the advertised first-run `+2`\n\nexact.\n\nTo inspect the same evidence in the product UI:\n\n```\nenv -u OPENAI_API_KEY \\\n  CROSS_EXAMINE_DB=\"$hero_workspace/cross-examine.db\" \\\n  CROSS_EXAMINE_RUNS=\"$hero_workspace/runs\" \\\n  uv run cross-examine serve\n```\n\nOpen the run URL printed by the terminal command, then expand the refuted finding. This server reads the same workspace-local database and run root, so the exact command, base output, head output, expected value, actual value, and reproducing input come from that pipeline-validated persisted report.\n\n**Test it without building anything.** The [live evidence explorer](https://cross-examine-six.vercel.app)\nis a deployed demo instance — no clone, no install, no API key. It serves an explicitly\nlabeled, checked-in evidence fixture so the report UI, the exact-command receipts, and the\nverdict surface can be inspected directly in a browser. Vercel Functions do not provide the\nGit and local-runtime capabilities required to execute repositories, so arbitrary repository\nanalysis is intentionally local-only — the quickstart above runs the real five-stage pipeline.\n\n**Supported platforms.** macOS, Linux, and Windows. The commands above are macOS/Linux; the\nequivalent [Windows PowerShell setup](#windows-powershell-setup) is below. CI exercises\nPython 3.12 on all three.\n\n**Where Codex accelerated the work.** Codex authored and iterated the whole application:\nthe Python pipeline, the schema and validation layer, execution controls, SQLite\npersistence, the FastAPI service, the React evidence explorer, the CLI, packaging, the\ncross-platform verification scripts, and the test suite. It also did the work that is easy\nto underestimate — diagnosing a Windows `cp1252`\n\nchild-encoding failure, a pytest-cache\nrename denial in detached worktrees, and the dependency-shaped false positives documented\nin [docs/trials.md](/stefbuilds/cross-examine/blob/main/docs/trials.md) — each of which changed the execution policy rather\nthan just the code. The dated Git history and the Codex session supplied with the Devpost\nsubmission show that progression.\n\n**Where the key decisions were made.** The human held product authority throughout; Codex\nchose the implementation. The split was deliberate and is the reason the verdict is\ntrustworthy: every doctrine on the left constrains what the code on the right is allowed\nto conclude.\n\n| Human-provided doctrine | Codex-chosen implementation |\n|---|---|\n| Problem selection and Python-only scope | FastAPI / SQLite / React stack |\n| The contract and five-stage structure | Worktree and subprocess mechanics |\n| Abstain-toward-risk policy | Edge catalog and Hypothesis bounds |\n| Layer-A-before-Layer-B sequencing | Persistence and SSE protocol |\n| Trusted-input execution boundary | CLI surface and deterministic hero construction |\n| Build Week deadline | Component selection and adaptation |\n| Interface design requirement | Responsive behavior, tests, packaging |\n| Evidence doctrine and final submission story | Cross-platform diagnosis, release verification |\n\n**How GPT-5.6 is used at run time.** GPT-5.6 Sol (`gpt-5.6-sol`\n\n) reads bounded diff and\nsource context and emits schema-constrained Claims plus optional ProbePlans. It never emits\nan outcome or a verdict. Malformed, duplicate, unknown-target, and forbidden structured\nfields are rejected, and proposal text stays untrusted. The model is a deliberately\nconstrained component rather than the judge — it proposes behavioral claims, while\nmodel-free execution supplies the evidence and a pure deterministic `aggregate()`\n\ndecides\nthe product verdict.\n\nA skill is part of the system being judged. You cannot ask the suspect to be the jury. Cross-Examine is a separate process with a separate state store: it proposes and executes checks, then applies a deterministic verdict function. Corpus v1 persists verified Layer-A fixtures and replays them by repository locator and symbol.\n\nA schema-constrained `Claim`\n\nis an untrusted proposal, not an oracle. Characterization\nmay also propose an optional untrusted `ProbePlan`\n\n; neither can carry an outcome or\nverdict. Executed base behavior and deterministic policy, not claim prose, decide a\npreservation finding. The intended-change abstention rule below follows from that same\nboundary.\n\n```\n---\nconfig:\n  theme: base\n  layout: dagre\n  themeVariables:\n    fontSize: 14px\n    lineColor: '#9ca3af'\n    primaryTextColor: '#111827'\n    edgeLabelBackground: '#ffffff'\n    tertiaryTextColor: '#4b5563'\n  flowchart:\n    curve: basis\n    nodeSpacing: 44\n    rankSpacing: 60\n    padding: 16\n    htmlLabels: true\n---\nflowchart TB\n  PR[\"<b>Python diff</b><br/>base to head\"]:::input\n\n  subgraph U [\"UNTRUSTED PROPOSAL\"]\n    direction LR\n    I[\"<b>1 · Ingest</b><br/>Git worktrees<br/>changed-file candidates\"]:::untrusted\n    C[\"<b>2 · Characterize</b><br/>GPT-5.6 Sol<br/>claims, never a verdict\"]:::untrusted\n    I --> C\n  end\n\n  subgraph EX [\"3 · CROSS-EXAMINE\"]\n    direction LR\n    LA[\"<b>Layer A</b><br/>base capture<br/>head replay\"]:::grounded\n    LB[\"<b>Layer B</b><br/>bounded Hypothesis<br/>and shrink\"]:::grounded\n    RT[\"<b>Repository tests</b><br/>discovered command\"]:::grounded\n    LA --> LB --> RT\n  end\n\n  AG[[\"<b>4 · aggregate()</b><br/>pure · no I/O · no model\"]]:::pure\n\n  CORPUS[\"<b>Corpus v1</b><br/>verified Layer-A fixtures\"]:::corpus\n  R[\"<b>5 · Render</b><br/>SQLite and grounded UI\"]:::report\n\n  PR --> I\n  C -- \"claims, unproven\" --> LA\n  RT ==> AG\n\n  LA -. \"pins eligible fixtures\" .-> CORPUS\n  CORPUS -. \"replays next run\" .-> LA\n\n  AG -- \"preserve-critical refutation\" --> BROKEN([\"<b>BROKEN</b>\"]):::broken\n  AG -- \"other refutation, critical abstain\" --> RISKY([\"<b>RISKY</b>\"]):::risky\n  AG -- \"none of the above\" --> SAFE([\"<b>SAFE · bounded</b>\"]):::safe\n\n  BROKEN --> R\n  RISKY --> R\n  SAFE --> R\n\n  classDef input fill:#ffffff,stroke:#9ca3af,stroke-width:1px,color:#374151\n  classDef untrusted fill:#ffffff,stroke:#b45309,stroke-width:1px,color:#78350f\n  classDef grounded fill:#ffffff,stroke:#475569,stroke-width:1px,color:#1e293b\n  classDef pure fill:#111827,stroke:#111827,stroke-width:1px,color:#ffffff\n  classDef corpus fill:#ffffff,stroke:#9ca3af,stroke-width:1px,color:#4b5563\n  classDef report fill:#ffffff,stroke:#374151,stroke-width:1px,color:#111827\n  classDef broken fill:#ffffff,stroke:#991b1b,stroke-width:2px,color:#7f1d1d\n  classDef risky fill:#ffffff,stroke:#b45309,stroke-width:2px,color:#78350f\n  classDef safe fill:#ffffff,stroke:#15803d,stroke-width:2px,color:#14532d\n\n  style U fill:#fdfcfb,stroke:#d6d3d1,stroke-width:1px,color:#78350f\n  style EX fill:#fafafa,stroke:#d4d4d8,stroke-width:1px,color:#1e293b\n```\n\nThe untrusted zone is model-authored and schema-constrained: it may propose, never conclude. Characterize is the only stage that contacts a model over the network; every execution stage is offline, model-free, bounded, and deterministic. The numbered stages match the five steps below.\n\n**Ingest** resolves base and head into detached Git worktrees and catalogues class, function, async, and nested candidate definitions in changed Python files. This is file-level discovery, not changed-line precision.**Characterize** asks GPT-5.6 Sol for strict Claims and optional ProbePlans. Both are untrusted proposals. In the offline hero, a labeled checked-in Claim fixture replaces the model call.**Cross-examine** replays captured base behavior against head, then runs a bounded Hypothesis search for edge cases. It probes synchronous callables with JSON-compatible inputs; anything outside that set abstains rather than guesses.**Aggregate** is a pure function. A preserve-critical refutation is`BROKEN`\n\n; other refutations, critical abstentions, or missing critical claims are`RISKY`\n\n.**Render** reads the persisted`Report`\n\n.`VERIFIED`\n\nand`REFUTED`\n\nfindings open to an exact command and captured output; abstentions show a deterministic diagnostic.\n\nSee [docs/architecture.md](/stefbuilds/cross-examine/blob/main/docs/architecture.md) for boundaries and failure behavior.\n\nV1 deliberately abstains on intended-change correctness unless the proposal has an\nindependent executable oracle. Since model prose is never an oracle, a represented\nintended-change claim without one keeps the report at least `RISKY`\n\n.\n\nIt reports that nothing was refuted among the checks that actually ran — not that the pull request is correct.`SAFE`\n\nmeans bounded, not proven.\n\nCross-Examine executes the target repository's code, so point it only at repositories you\ntrust. Commands run through argument vectors with `shell=False`\n\n, an executable allowlist,\na minimal child environment that strips secret-shaped variables, deadlines, a 2 MB output\ncap, and receipt redaction. Those controls bound the harness itself, not code the target\nspawns — production use needs real isolation. Serve on `127.0.0.1`\n\n.\n\nThe public [evidence explorer](https://cross-examine-six.vercel.app) is exactly that: it\nserves a labeled, checked-in fixture so the report UI and its receipts can be inspected\nwithout installing anything. Executing a repository needs Git and a local runtime, so\nthat path runs locally by design.\n\n| Requirement | Notes |\n|---|---|\n| Python | 3.12 is tested; package metadata currently permits `>=3.12` |\n| Git | |\n|\n\n`npx playwright install chromium`\n\n)`OPENAI_API_KEY`\n\nCI is configured for Python 3.12 on Windows, macOS, and Ubuntu. Cite an immutable green run before calling that matrix verified. Repository targets are Python-only during Build Week. The local runner executes target code, so use only repositories you trust.\n\nThe package is laid out as the five stages, in order. To read it end to end, start at\n`pipeline.py`\n\n— it orchestrates all five and is the only file you need to follow the whole\nflow.\n\n| Stage | Where it lives | Start here |\n|---|---|---|\n| Orchestration | `src/cross_examine/pipeline.py` |\n`Pipeline.run()` — all five stages in one method |\n| 1 · Ingest | `src/cross_examine/ingest/` |\n`service.py` resolves worktrees, `symbols.py` finds candidates |\n| 2 · Characterize | `src/cross_examine/characterize/` |\n`models.py` is the schema the model must satisfy |\n| 3 · Cross-examine | `src/cross_examine/cross_examine/` |\n`layer_a.py` (base capture / head replay), `layer_b.py` (bounded search) |\n| 4 · Aggregate | `src/cross_examine/schema.py` |\n— 16 lines, pure, no I/O; the verdict lives here`aggregate()` |\n| 5 · Render | `src/cross_examine/api/` , `persistence/` |\n`app.py` serves the report the React app reads |\n| Execution boundary | `src/cross_examine/execution.py` |\nevery subprocess the harness runs goes through here |\n\nIf you only read one function, read `aggregate()`\n\nin `schema.py`\n\n. It is the whole\nargument of the project: the verdict is a pure function of executed findings, so no model\noutput can reach it.\n\n| Path | What's there |\n|---|---|\n`src/cross_examine/` |\nThe Python package: pipeline stages, schemas and validation, execution controls, persistence, CLI, fixtures, and FastAPI application. |\n`frontend/` |\nReact/Vite evidence-explorer source, UI components, frontend tests, and browser end-to-end tests. |\n`api/` |\nVercel entry point that exposes the packaged application. |\n`scripts/` |\nHero-repository builder, real-repository trial runner, and cross-platform verification scripts. |\n`tests/` |\nPython unit, integration, end-to-end, release, and hero-repository fixture tests. |\n`docs/` |\nArchitecture, demo, execution policy, provenance, submission, trial evidence, and probe-plan documentation. |\n\n```\nuv sync --extra dev\nPush-Location frontend\nnpm ci\nnpm run build\nPop-Location\n$env:CROSS_EXAMINE_DEMO_CHARACTERIZER = \"fixture\"\nRemove-Item Env:OPENAI_API_KEY -ErrorAction SilentlyContinue\nRemove-Item Env:CROSS_EXAMINE_DB -ErrorAction SilentlyContinue\nRemove-Item Env:CROSS_EXAMINE_RUNS -ErrorAction SilentlyContinue\n$heroWorkspace = Join-Path ([System.IO.Path]::GetTempPath()) (\"cross-examine-hero-\" + [Guid]::NewGuid())\nuv run --isolated --no-editable cross-examine demo --no-open --workspace $heroWorkspace\n$env:CROSS_EXAMINE_DB = Join-Path $heroWorkspace \"cross-examine.db\"\n$env:CROSS_EXAMINE_RUNS = Join-Path $heroWorkspace \"runs\"\nuv run cross-examine serve\n```\n\nRepeat the same demo command with the same `$heroWorkspace`\n\nto see `+0 this run · 2 total`\n\nafter the fresh run's `+2 this run · 2 total`\n\n.\n\nOpen the printed run URL. The packaged FastAPI server hosts both the API and React\napplication, so direct `/runs/{id}`\n\nlinks work. Completed reports persist; worker queues\nand SSE history are in memory, and stale queued/running work is not resumed after restart.\n\nThe UI's **Run offline hero demo** action creates the stable `hero-base`\n\nand `hero-head`\n\nrepository automatically. Its claim source is visibly labeled `deterministic hero fixture`\n\n; decided findings still require real execution, and deterministic code owns the\nverdict.\n\n```\n$env:OPENAI_API_KEY = \"...\"\nuv run cross-examine run C:\\code\\your-python-repo --base main --head feature/candidate\nuv run cross-examine serve\n```\n\nUse `--no-layer-b`\n\nfor a Layer-A-only compatibility pass. The web form accepts a local path or Git URL and streams stage progress over SSE.\n\nThe repository's current verification entry points are:\n\nOn macOS or Linux:\n\n```\nbash scripts/verify.sh\n```\n\nOn Windows:\n\n```\npowershell -ExecutionPolicy Bypass -File scripts/verify.ps1\n```\n\nBoth entry points remove `OPENAI_API_KEY`\n\n, `CROSS_EXAMINE_DB`\n\n, and\n`CROSS_EXAMINE_RUNS`\n\nfrom child processes, force the fixture, sync locked dependencies,\nand run the current backend/frontend/build checks. Each owns a temporary demo workspace\nand asserts a fresh `BROKEN/+2/2`\n\nrun followed by `BROKEN/+0/2`\n\n. The POSIX script also\nasserts checked-in static-bundle equality; the PowerShell script builds and tests the\nbundle but does not perform the same byte-drift assertion.\n\nCoverage is Python 3.12 across Windows, macOS, and Ubuntu: unit, integration, end-to-end, and release tests on the backend; component, accessibility, and two Chromium flows on the frontend.", "url": "https://wpnews.pro/news/cross-examine-replay-base-behavior-against-an-ai-generated-pr", "canonical_source": "https://github.com/stefbuilds/cross-examine", "published_at": "2026-08-12 11:56:28+00:00", "updated_at": "2026-08-12 12:12:49.077447+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": ["Cross-Examine", "Codex", "OpenAI Build Week 2026", "GPT-5.6", "FastAPI", "React", "Vercel"], "alternates": {"html": "https://wpnews.pro/news/cross-examine-replay-base-behavior-against-an-ai-generated-pr", "markdown": "https://wpnews.pro/news/cross-examine-replay-base-behavior-against-an-ai-generated-pr.md", "text": "https://wpnews.pro/news/cross-examine-replay-base-behavior-against-an-ai-generated-pr.txt", "jsonld": "https://wpnews.pro/news/cross-examine-replay-base-behavior-against-an-ai-generated-pr.jsonld"}}