A/B test whether pre-gathered context helps coding agents complete tasks Unblocked released the Context Engine Simulator, an open-source tool that runs the same coding task twice against a real git repository — once with the agent working from the task description alone and once with a pre-gathered context bundle — then scores both arms 0–100 against user-defined acceptance criteria and reports quality, speed and cost differences. The simulator supports Claude Code, OpenAI Codex, Cursor and Grok as agent CLIs, runs the two arms in parallel in isolated git worktrees, and requires Node.js 18 or later plus MCP servers configured for the target repository so the context arm can pull from issues, PRs, chat and docs. The context arm adds a read-only researcher agent, pattern extraction, a second plan-specific context pass and an attribution analysis showing which gathered context influenced the implementation, with results written to a terminal table, a JSON file and a standalone HTML report. A/B test whether pre-gathered organizational context helps AI coding agents complete tasks faster, cheaper, and better . The simulator runs the same task twice against a real repository — once as a baseline the agent works from the task description alone and once context-enhanced the agent first gathers relevant context from the codebase and connected sources, then plans and implements from that briefing . It then scores both results against your acceptance criteria and reports the quality, speed, and cost difference. It works with multiple coding-agent CLIs — Claude Code , OpenAI Codex , Cursor , and Grok — so you can compare how each one benefits from context. Powered by Unblocked https://getunblocked.com . Each run executes two arms in parallel , each in its own isolated git worktree so they never interfere: Plan ──▶ Review ──▶ Implement ──▶ Evaluate The agent plans, self-reviews the plan, implements it, and the result is scored. Gather Context ──▶ Extract Patterns ──▶ Plan ──▶ Gather Plan Context ──▶ Review ──▶ Implement ──▶ Evaluate + Attribute 1. Gather Context — a read-only researcher agent searches the codebase and any connected MCP sources issues, PRs, chat, docs to produce a focused context bundle. 2. Extract Patterns — distills the bundle into the codebase's coding-style conventions. 3. Plan — plans the task using the bundle as a head start. 4. Gather Plan Context — a targeted second pass for risks and style notes specific to the plan. 5. Review — cross-checks the plan against the gathered context. 6. Implement — writes the code. Both arms are scored 0–100 by an independent evaluator agent against your acceptance criteria, judging primarily on the actual code diff . The context arm additionally produces an attribution analysis showing which gathered context actually influenced the implementation. Results are written as a terminal table , a JSON file , and a standalone HTML report . - Node.js ≥ 18 - git the target must be a git repository - At least one supported agent CLI installed and on your PATH :Agent CLI binary --agent valueClaude Code claude claude default OpenAI Codex codex codex Cursor agent cursor Grok grok grok Each agent must be authenticated per its own CLI. The simulator shells out to whichever one you select. The whole point of the experiment is the context the agent can reach, so the agent under test must have its MCP servers configured for the target repository before you run anything. Worktrees inherit that configuration. Set this up once by launching the agent interactively inside the target repo and confirming its MCP servers connect: cd /path/to/target-repo claude or: codex, cursor, grok — whichever agent you'll test Verify every expected MCP server shows as connected e.g. /mcp in Claude Code, the startup MCP status in the others . Authenticate any that prompt. Once they all connect cleanly here, the simulator's runs will have the same access. Skipping this means the context arm silently gathers less than it should. git clone https://github.com/unblocked/context-engine-simulator.git cd context-engine-simulator npm install No build step — it runs directly via tsx https://github.com/privatenumber/tsx . Create a YAML fixture see fixture.sample.yaml https://github.com/unblocked/context-engine-simulator/blob/main/fixture.sample.yaml : repo: /path/to/target-repo branch: main task: | Add a /health endpoint to the API server that returns the service name and current timestamp as JSON. criteria: | 1. GET /health returns 200 with a JSON body 2. Response includes "service" and "timestamp" fields 3. Endpoint is registered following existing router patterns agent: claude claude | codex | cursor | grok model: sonnet Run it: npm start -- --fixture my-experiment.yaml --verbose Everything in a fixture can be passed as a flag flags override fixture values : npm start -- \ --repo /path/to/repo \ --task "Add a /health endpoint returning service name and timestamp" \ --criteria "GET /health returns 200 JSON with service and timestamp fields" \ --agent claude \ --model opus \ --verbose | Flag | Description | Default | |---|---|---| | --fixture