{"slug": "a-b-test-whether-pre-gathered-context-helps-coding-agents-complete-tasks", "title": "A/B test whether pre-gathered context helps coding agents complete tasks", "summary": "Unblocked released the Context Engine Simulator, an open-source tool that runs the same coding task twice against a real git repository — once with the agent working from the task description alone and once with a pre-gathered context bundle — then scores both arms 0–100 against user-defined acceptance criteria and reports quality, speed and cost differences. The simulator supports Claude Code, OpenAI Codex, Cursor and Grok as agent CLIs, runs the two arms in parallel in isolated git worktrees, and requires Node.js 18 or later plus MCP servers configured for the target repository so the context arm can pull from issues, PRs, chat and docs. The context arm adds a read-only researcher agent, pattern extraction, a second plan-specific context pass and an attribution analysis showing which gathered context influenced the implementation, with results written to a terminal table, a JSON file and a standalone HTML report.", "body_md": "A/B test whether pre-gathered organizational context helps AI coding agents complete tasks **faster, cheaper, and better**.\n\nThe simulator runs the *same task twice* against a real repository — once as a **baseline** (the agent works from the task description alone) and once **context-enhanced** (the agent first gathers relevant context from the codebase and connected sources, then plans and implements from that briefing). It then scores both results against your acceptance criteria and reports the quality, speed, and cost difference.\n\nIt works with multiple coding-agent CLIs — **Claude Code**, **OpenAI Codex**, **Cursor**, and **Grok** — so you can compare how each one benefits from context.\n\nPowered by [Unblocked](https://getunblocked.com).\n\nEach run executes two arms **in parallel**, each in its own isolated git worktree so they never interfere:\n\n```\nPlan ──▶ Review ──▶ Implement ──▶ Evaluate\n```\n\nThe agent plans, self-reviews the plan, implements it, and the result is scored.\n\n```\nGather Context ──▶ Extract Patterns ──▶ Plan ──▶ Gather Plan Context ──▶ Review ──▶ Implement ──▶ Evaluate + Attribute\n```\n\n1. **Gather Context** — a read-only researcher agent searches the codebase and any connected MCP sources (issues, PRs, chat, docs) to produce a focused context bundle.\n2. **Extract Patterns** — distills the bundle into the codebase's coding-style conventions.\n3. **Plan** — plans the task using the bundle as a head start.\n4. **Gather Plan Context** — a targeted second pass for risks and style notes specific to the plan.\n5. **Review** — cross-checks the plan against the gathered context.\n6. **Implement** — writes the code.\n\nBoth arms are scored 0–100 by an independent evaluator agent against your acceptance criteria, judging primarily on the **actual code diff**. The context arm additionally produces an **attribution analysis** showing which gathered context actually influenced the implementation.\n\nResults are written as a **terminal table**, a **JSON file**, and a **standalone HTML report**.\n\n- \n**Node.js** ≥ 18\n- \n**git** (the target must be a git repository)\n- \nAt least one supported agent CLI installed and on your `PATH` :Agent CLI binary `--agent` valueClaude Code `claude`` claude` (default)OpenAI Codex `codex`` codex`Cursor `agent`` cursor`Grok `grok`` grok`\n\nEach agent must be authenticated per its own CLI. The simulator shells out to whichever one you select.\n\nThe whole point of the experiment is the context the agent can reach, so the agent under test must have its MCP servers configured **for the target repository** before you run anything. Worktrees inherit that configuration.\n\nSet this up once by launching the agent interactively inside the target repo and confirming its MCP servers connect:\n\n```\ncd /path/to/target-repo\nclaude        # or: codex, cursor, grok — whichever agent you'll test\n```\n\nVerify every expected MCP server shows as connected (e.g. `/mcp` in Claude Code, the startup MCP status in the others). Authenticate any that prompt. Once they all connect cleanly here, the simulator's runs will have the same access. Skipping this means the context arm silently gathers less than it should.\n\n```\ngit clone https://github.com/unblocked/context-engine-simulator.git\ncd context-engine-simulator\nnpm install\n```\n\nNo build step — it runs directly via [`tsx`](https://github.com/privatenumber/tsx).\n\nCreate a YAML fixture (see [`fixture.sample.yaml`](https://github.com/unblocked/context-engine-simulator/blob/main/fixture.sample.yaml)):\n\n```\nrepo: /path/to/target-repo\nbranch: main\n\ntask: |\n  Add a /health endpoint to the API server that returns the service\n  name and current timestamp as JSON.\n\ncriteria: |\n  1. GET /health returns 200 with a JSON body\n  2. Response includes \"service\" and \"timestamp\" fields\n  3. Endpoint is registered following existing router patterns\n\nagent: claude        # claude | codex | cursor | grok\nmodel: sonnet\n```\n\nRun it:\n\n```\nnpm start -- --fixture my-experiment.yaml --verbose\n```\n\nEverything in a fixture can be passed as a flag (flags override fixture values):\n\n```\nnpm start -- \\\n  --repo /path/to/repo \\\n  --task \"Add a /health endpoint returning service name and timestamp\" \\\n  --criteria \"GET /health returns 200 JSON with service and timestamp fields\" \\\n  --agent claude \\\n  --model opus \\\n  --verbose\n```\n\n| Flag | Description | Default | \n|---|---|---|\n| `--fixture <path>` | YAML fixture file | — | \n| `--repo <path>` | Target git repository | *(required)* | \n| `--task <string>` /`--task-file <path>` | Task description | *(required)* | \n| `--criteria <string>` /`--criteria-file <path>` | Acceptance criteria for scoring | — | \n| `--agent <name>` | `claude` \\|`codex` \\|`cursor` \\|`grok` | `claude` | \n| `--model <model>` | Model for task runs | `sonnet` | \n| `--context-model <model>` | Model for context gathering | same as `--model` | \n| `--eval-model <model>` | Model for evaluation | same as `--model` | \n| `--timeout <seconds>` | Max seconds per task step | `3600` | \n| `--context-timeout <seconds>` | Max seconds per context step | `600` | \n| `--branch <name>` | Branch to base worktrees on | current HEAD | \n| `--disable-mcp <servers...>` | MCP servers to block in both arms | — | \n| `--keep-worktrees` | Keep worktrees after the run for inspection | `false` | \n| `--verbose` | Stream agent activity live | `false` | \n\nIf `--criteria` is omitted, the evaluation and attribution steps are skipped.\n\nA timestamped directory under `results/` containing:\n\n- **`report.html`** — a self-contained visual report (open in a browser)\n- **`result.json`** — full structured results for programmatic analysis\n\nPlus a comparison table printed to the terminal: quality score, wall-clock time, cost, and token counts for each arm and each phase.\n\nTo measure the *isolated* effect of context gathering, the baseline arm must not have backdoor access to the same context sources. Use `--disable-mcp` (or `disableMcp:` in a fixture) to block specific MCP servers in **both** arms' worktrees; the simulator also detects and aborts a run if a blocked server is called, so contamination can't silently skew the comparison.\n\n```\nnpm start -- --fixture my-experiment.yaml --disable-mcp some-context-server\n```\n\n- Each arm runs in a throwaway git worktree (under your system temp dir, or agent-managed for Claude/Cursor). They're cleaned up automatically unless `--keep-worktrees` is set.\n- Cost is computed from each agent's reported token usage where available. Subscription-billed agents (e.g. Grok) report token counts but no per-token price, so their cost shows as `$0` .\n- The tool never commits or pushes; it only reads the target repo and works inside isolated worktrees.\n\n[MIT](https://github.com/unblocked/context-engine-simulator/blob/main/LICENSE) © Unblocked", "url": "https://wpnews.pro/news/a-b-test-whether-pre-gathered-context-helps-coding-agents-complete-tasks", "canonical_source": "https://github.com/unblocked/context-engine-simulator", "published_at": "2026-11-15 04:01:11+00:00", "updated_at": "2026-11-15 04:22:56.813555+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "ai-products"], "entities": ["Unblocked", "Context Engine Simulator", "Claude Code", "OpenAI Codex", "Cursor", "Grok", "Node.js", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-b-test-whether-pre-gathered-context-helps-coding-agents-complete-tasks", "markdown": "https://wpnews.pro/news/a-b-test-whether-pre-gathered-context-helps-coding-agents-complete-tasks.md", "text": "https://wpnews.pro/news/a-b-test-whether-pre-gathered-context-helps-coding-agents-complete-tasks.txt", "jsonld": "https://wpnews.pro/news/a-b-test-whether-pre-gathered-context-helps-coding-agents-complete-tasks.jsonld"}}