# Best AI Coding Assistants: 12 Steps, 90 Min [2026] > Source: > Published: 2026-10-28 01:48:12+00:00 The best AI coding assistants change faster than any review can keep up with, so a ranking you read today may describe a product that has since swapped its model. GitHub’s changelog shows what that looks like in practice. Copilot added GPT-6 Astra on September 4, GPT-6 Sol and GPT-6 Luna on September 22, GPT-6.1 Sol on September 29, Claude Haiku 5.5 on October 7, Gemini 4 Argon on October 13 and GPT-6.1 Terra on October 26, 2026. That is seven models in seven weeks inside one product. Adoption is not the open question anymore. Stack Overflow’s 2025 developer survey press release said, in its own words, [“18% of developers reported using Cursor, while 10% reported using Claude Code among the AI-enabled editors included in the survey.”](https://stackoverflow.co/company/press/archive/stack-overflow-2025-developer-survey/) The harder question for a team lead is which AI coding assistant earns its seat price on your code, with your reviewers, in your CI. Public leaderboards cannot answer that. Task benchmarks like the [Cursor vs GitHub Copilot benchmark](https://shattered.io/benchmark-cursor-vs-github-copilot-2026/) show how tools behave on a fixed set of prompts. A pilot measures something else: what happens to real pull requests when real developers use each tool for a month. A small Python project can run that pilot. It reads your git history, your pull request export and your billing numbers, then ranks four assistants on speed, rework, cost per merged PR and developer sentiment. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. [Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F) ## Why a team pilot beats a benchmark for picking among the best AI coding assistants A benchmark controls everything except the thing you care about. It fixes the repository, the prompt and the grader, which makes results comparable across vendors. It also strips out the friction that decides whether a tool survives in a team: slow approvals, noisy suggestions, awkward review hand-offs and surprise bills. A pilot flips the trade. You give up tight control and gain realism. Each pull request carries a label saying which assistant helped, each commit carries a trailer, and the pilot script turns those marks into four numbers per tool. Those numbers come from your own work, so they already include your test suite, your code style and your reviewers’ patience. The method has limits. With 6 to 20 pull requests per tool, small differences are noise. Developers pick the tool for the task, which biases results. A tool that shipped a model update mid-pilot is a different tool at the end than at the start. The project handles these limits with minimum sample warnings, a rotation rule, a weights stress test and a model log. Treat the output as a decision aid, not a verdict. Three questions drive the design: - Does the assistant shorten the time from opening a pull request to merging it? - Does the merged work come back as a fix within a week? - What does one merged pull request cost once seats and usage are added together? Developer sentiment is the fourth input, kept at the lowest weight, because people rate tools they enjoy higher even when the numbers disagree. For a head-to-head view of two specific contenders before you commit pilot time, the [GitHub Copilot vs Claude Code comparison](https://shattered.io/github-copilot-vs-claude-code-2026/) covers seat pricing and feature differences in more detail. The whole build takes about 90 minutes. The pilot itself runs four weeks. Plan the calendar first, because the tooling is the easy part. ## Prerequisites and versions The project uses only the Python standard library. No pip installs, no virtual environment drama, no lockfile to drift. That choice keeps the pilot reproducible a year from now when you want to rerun it on a new quarter of data. | Component | Version used | Why it matters | |---|---|---| | Python | 3.12.3 (3.11 is the minimum) | tomllib for config parsing, modern type syntax | | Git | 2.43.0 | trailer formatting in git log, interpret-trailers in the hook | | GitHub CLI (gh) | current 2.x release | exports merged pull requests as JSON | | POSIX shell | any | runs the prepare-commit-msg hook | | Test repository | 4+ weeks of merged PRs | the subject of the pilot | | Assistants | latest release of each | install and sign in before day one | You also need a few non-software prerequisites. Get written agreement from the developers in the pilot, since the project reads commit trailers and PR labels that identify who used which tool. Make sure the repository has branch protection and CI, otherwise lead time measures nothing useful. Finally, decide who owns the cost export. Seat charges live in billing consoles, and usage charges sit in a different place for each vendor. Pin the tool versions on day one, whichever AI coding tools you test. Run each assistant’s version command, paste the output into a pilot log and note the default model it shows. Most of these CLIs update themselves, so a version string at the start and another at the end tells you whether the tool changed under you. The Claude Code documentation at [code.claude.com](https://code.claude.com/docs/en) describes the CLI and its settings if you need a refresher on where model selection lives. ## The four assistants in the pilot, as of October 28, 2026 The pilot compares four products that cover the main ways developers use these tools today: a terminal agent from Anthropic, an agent and desktop app from OpenAI, a multi-model platform from GitHub and an agent-first editor from Anysphere. The table lists what the vendors’ own pages and changelogs confirm for this week. | Assistant | Where you use it | Latest confirmed change | Billing notes | |---|---|---|---| | Claude Code | Terminal, IDE extensions | Anthropic’s alignment team published its Auto Mode red-teaming write-up on Oct. 22 | Seats on Anthropic plans, with API credits on some tiers | | OpenAI Codex | CLI, desktop app, Codex Cloud | GPT-6.1 Terra began rolling out in ChatGPT Work and Codex on Oct. 26 | Included with ChatGPT Free through Education plans, Codex Cloud excluded on Free and Go | | GitHub Copilot | IDEs, Copilot CLI, Copilot app | GPT-6.1 Terra generally available Oct. 26, assisted permissions generally available Oct. 26 | Plans: Free, Student, Pro, Pro+, Max, Business, Enterprise, with 1 AI credit equal to $0.01 | | Cursor | Editor, web, iOS, CLI, SDK | Grok 4.5 shipped July 8, trained jointly with SpaceXAI | Composer 2.5 listed at $0.50 per million input tokens and $2.50 output | Two details from that table shape the pilot design. First, Codex usage counts against plan limits, credits or billing even when a cloud environment carries no separate VM charge, so a seat-only cost model understates it. Second, Copilot’s newest models are not offered on every plan. [GitHub’s October changelog](https://github.blog/changelog/label/copilot/) says Gemini 4 Argon is available to Pro+, Max, Business and Enterprise and bills at provider-list pricing under usage-based billing. A developer on a lower plan cannot reproduce a colleague’s results with the same model. Cursor’s pricing is in motion. According to forum posts, Cursor introduced “The X Subscription” on October 12, and users there report that third-party models such as Claude and GPT are billed at API prices under the new tiers. A [community thread on Teams Premium](https://forum.cursor.com/t/switched-from-pro-to-teams-premium-cursor-models-pool-is-smaller-and-more-expensive/174065/7) complains that the Cursor model pool is smaller and costs more. Those are user reports, not a vendor price sheet, so check the pricing page on the day you sign up and write the seat price into the ledger yourself. For the full list of what each plan costs, the [Copilot pricing and AI credits breakdown](https://shattered.io/github-copilot-pricing-ai-credits-claude-codex-agents-2026/) tracks the credit model, and the [Cursor vs Claude Code entry-plan comparison](https://shattered.io/cursor-vs-claude-code-2026/) covers the other side. If your team already leans toward a terminal agent, the usage split in [Codex vs Claude Code](https://shattered.io/codex-vs-claude-code-stack-overflow-65-5-vs-29-5-2026/) gives context on who is using which. ## Steps 1 to 3: Set the rules, scaffold the project, write the config ### Step 1: Define the pilot rules Write these down before anyone installs anything, and get sign-off from the people doing the work. 1. **Duration:** four weeks, starting on a Monday, with no tool changes in the final week. 2. **Subjects:** one repository, or two at most, with active development and required CI. 3. **Assignment:** rotate. Each developer uses each tool for roughly a week, in a different order, so no tool gets only the easiest tickets. 4. **Minimum sample:** 8 merged pull requests per tool. Below that, the report prints a warning and you extend the pilot. 5. **Baseline:** keep a human-only lane. Some pull requests carry no assistant label, and they become the comparison row. 6. **Model log:** record each tool’s version and default model at the start of each week. The rotation rule matters more than it looks. If your strongest developer holds one tool for four weeks, you measure the developer. Rotating spreads the skill effect across all four tools. ### Step 2: Scaffold the project Create a folder and the empty package. Everything lives in one directory so you can zip it and hand it to a colleague. ``` mkdir assistant-pilot && cd assistant-pilot mkdir pilot tests data touch pilot/__init__.py ``` The final layout, once all steps are done, looks like this: ``` assistant-pilot/ pilot.toml make_fixture.py data/ prs.json, costs.csv, survey.csv pilot/ __init__, __main__, bootstrap, cli, config, gitmine, ledger, metrics, min_sample, score tests/ test_pilot.py ``` ### Step 3: Write pilot.toml The config holds everything a reviewer might want to argue about: the date range, the trailer name, the fix window and the scoring weights. Putting weights in a file instead of code means the debate happens in a pull request on the config, not in a hallway. ``` [pilot] repo = "fixture-repo" since = "2026-09-01" trailer = "Assisted-by" baseline = "human" fix_window_days = 7 [weights] speed = 0.25 quality = 0.35 cost = 0.25 survey = 0.15 [files] prs = "data/prs.json" costs = "data/costs.csv" survey = "data/survey.csv" ``` The weights reflect a priority call. Quality counts most because rework is the expensive failure, speed and cost split the next tier, and developer sentiment gets the smallest slice. Change them to match your own priorities, but change them before you see results. Adjusting weights after seeing a ranking is how pilots end up confirming whatever the loudest person already believed. The loader refuses weights that do not add up to 1.0, which catches typos early. This is the config module: ``` python import tomllib from dataclasses import dataclass from pathlib import Path @dataclass(frozen=True) class Config: repo: Path since: str trailer: str baseline: str fix_window_days: int weights: dict prs: Path costs: Path survey: Path def check_weights(w): total = sum(w.values()) if abs(total - 1.0) > 1e-9: raise ValueError(f"weights must sum to 1.0, got {total:.3f}") def load(path="pilot.toml"): raw = tomllib.loads(Path(path).read_text()) p, w, f = raw["pilot"], raw["weights"], raw["files"] check_weights(w) return Config( repo=Path(p["repo"]), since=p["since"], trailer=p["trailer"], baseline=p["baseline"], fix_window_days=int(p["fix_window_days"]), weights=w, prs=Path(f["prs"]), costs=Path(f["costs"]), survey=Path(f["survey"]), ) ``` ## Steps 4 and 5: Tag every change with a trailer and a label The pilot is only as good as its tagging. Two marks identify assistant work: a git trailer on each commit and a label on each pull request. Both must be cheap enough that developers do not skip them. ### Step 4: Install the commit trailer hook A `prepare-commit-msg` hook stamps every commit with `Assisted-by: tool` based on a per-repository git setting. Developers switch tools by changing one setting, and the hook does the rest. Save this as `.git/hooks/prepare-commit-msg` and make it executable. ``` bash #!/bin/sh tool="$(git config --get pilot.tool)" [ -z "$tool" ] && exit 0 [ "$tool" = "human" ] && exit 0 git interpret-trailers --in-place --if-exists replace --trailer "Assisted-by: $tool" "$1" ``` Run `chmod +x .git/hooks/prepare-commit-msg` and then set the tool for the week. The slugs below match what the report prints, so keep them lowercase and consistent. ``` git config pilot.tool claude-code # or: codex, copilot, cursor, human ``` Test it before you trust it. Commit once with a tool set and once with `human`, then read the trailers back: ``` bash $ git log --format='%s | [%(trailers:key=Assisted-by,valueonly,separator=%x2C)]' fix: demo | [] feat: demo (#1) | [codex] ``` Hooks live inside `.git` and do not travel with a clone, so each developer installs their own. Put the script in the repository under `scripts/` and add a one-line install note to your contributing guide. Developers who already follow the [Claude Code tutorial](https://shattered.io/claude-code-tutorial-13-steps-2026/) setup can add the hook to the same onboarding checklist. Commit subjects must also end with the pull request number in the form `(#12)`. GitHub adds that suffix automatically on squash merges, which is how the report maps commits to pull requests. Fix commits link back with a trailer: `Fixes: #12`. Agree on that convention in the rules, because the rework metric depends on it. ### Step 5: Label every pull request Labels carry the same information at the pull request level, where lead time is measured. Create one label per tool, using the `ai:` prefix the script expects. ``` for t in claude-code codex copilot cursor; do gh label create "ai:$t" --color 5319e7 --description "Pilot: PR assisted by $t" done ``` A pull request template with a checkbox list reminds authors to apply the right label. Labels also give you a free audit trail. If a PR has an `ai:codex` label but its commits carry `Assisted-by: cursor`, the author switched tools mid-PR, and you should look at that case by hand. The export you will pull in Step 8 includes labels, so no extra tooling is needed. Teams that also run a headless agent in CI can tag those runs the same way. The [Cursor CLI in GitHub Actions guide](https://shattered.io/cursor-cli-github-actions-headless-agent-2026/) shows how an unattended run fits into a workflow, and you would give that run its own label so automation does not blur the human numbers. ## Steps 6 and 7: Mine git history and measure rework ### Step 6: Read commits with git log The first module shells out to `git log` and asks git itself to extract the trailers. Parsing trailers with regular expressions over raw commit bodies breaks on wrapped lines and mixed-case keys. Git’s `%(trailers:key=...,valueonly)` placeholder does the job correctly. The format string joins five fields with an ASCII unit separator and starts each record with a record separator. Those control characters never appear in commit messages, which avoids the classic bug where a subject line containing a pipe or a tab corrupts the parse. ``` python import re import subprocess from dataclasses import dataclass, field from datetime import datetime RS, US = "\x1e", "\x1f" PR_REF = re.compile(r"\(#(\d+)\)") FIX_REF = re.compile(r"#?(\d+)") @dataclass class Commit: sha: str when: datetime subject: str tool: str pr: int | None = None fixes: list = field(default_factory=list) def read_commits(repo, since, trailer, baseline): fmt = (f"{RS}%H{US}%aI{US}%s{US}%(trailers:key={trailer},valueonly,separator=%x2C)" f"{US}%(trailers:key=Fixes,valueonly,separator=%x2C)") out = subprocess.run( ["git", "-C", str(repo), "log", "--no-merges", f"--since={since}", f"--format={fmt}"], check=True, capture_output=True, text=True, ).stdout commits = [] for record in out.split(RS)[1:]: sha, when, subject, tool, fixes = record.strip("\n").split(US) m = PR_REF.search(subject) commits.append(Commit( sha, datetime.fromisoformat(when), subject, tool.strip().split(",")[0].strip().lower() or baseline, int(m.group(1)) if m else None, [int(x) for x in FIX_REF.findall(fixes)], )) return sorted(commits, key=lambda c: c.when) ``` Three details deserve attention. The code lowercases the tool name, so `Assisted-by: Codex` and `assisted-by: codex` collapse into one row. Commits with no trailer fall into the baseline bucket named in the config. And the PR number comes from the `(#N)` suffix in the subject, which is why Step 4 insisted on that convention. Use `--no-merges` deliberately. Merge commits carry no authorship signal and would double count work already represented by the commits they merge. If your team merges with real merge commits instead of squashing, the trailer needs to live on the PR’s own commits, and the PR number comes from the merge commit message. In that case adjust `PR_REF` to match your merge message format. ### Step 7: Measure rework Rework answers the question reviewers care about most: how often does merged work come back? The definition here is narrow and explicit. A pull request counts as reworked when a later commit carries `Fixes: #N` pointing at it, and that commit lands within the configured window (seven days by default) after the original change. ``` python def rework(commits, window_days): """Share of each tool's PRs that a later commit repairs via a 'Fixes: #N' trailer.""" owner = {c.pr: c for c in commits if c.pr is not None} repaired = set() for fix in commits: for n in fix.fixes: base = owner.get(n) if base and timedelta(0) < fix.when - base.when <= timedelta(days=window_days): repaired.add(n) total, hit = defaultdict(int), defaultdict(int) for n, c in owner.items(): total[c.tool] += 1 hit[c.tool] += n in repaired return {t: hit[t] / k for t, k in total.items()}, dict(total) ``` Save the function inside `pilot/metrics.py` with these imports at the top. The next step adds the second function to the same file. ``` python import json import statistics from collections import defaultdict from datetime import datetime, timedelta ``` The design choice to rely on explicit references rather than file overlap came from a failed first attempt. A first version flagged any later “fix” commit that touched the same files within seven days. On a repository with eight hot files, almost every fix overlapped with someone’s earlier change, and the human-only baseline showed 50% rework. Switching to explicit `Fixes: #N` references dropped the same baseline to 10% and spread the four assistants between 7% and 36%, which follows the failure rates seeded into the synthetic data, small-sample wobble included. The lesson carries over to real repositories. Heuristics that guess at causality inflate every tool’s number and bury the differences you are trying to see. The unit tests include a case that pins this behavior: an unlinked fix commit must not count against anything. If your team will not write the trailer, rework is the one metric you should drop from the score rather than approximate. ## Step 8: Compute lead time from your pull request export Lead time is the gap between opening a pull request and merging it. It captures review waits, CI reruns and back-and-forth that git history alone hides. GitHub’s CLI exports exactly the fields the script needs. ``` gh pr list --state merged --limit 200 \ --search "merged:>=2026-09-01" \ --json number,title,createdAt,mergedAt,labels > data/prs.json ``` Run it from inside the repository, and raise `--limit` if the pilot produces more than 200 merged pull requests. The output is a JSON array. Each element looks like this: ``` { "number": 2, "title": "change 2", "createdAt": "2026-09-22T22:31:30Z", "mergedAt": "2026-09-24T01:53:34Z", "labels": [{"name": "ai:assistant-d"}] } ``` The second function in `pilot/metrics.py` groups pull requests by their `ai:` label and takes the median lead time in hours. A pull request with no `ai:` label lands in the baseline group. ``` python def lead_times(prs_path, baseline): prs = json.loads(open(prs_path).read()) by_tool = defaultdict(list) for pr in prs: tool = next((l["name"].split(":", 1)[1] for l in pr["labels"] if l["name"].startswith("ai:")), baseline) made = datetime.fromisoformat(pr["createdAt"].replace("Z", "+00:00")) merged = datetime.fromisoformat(pr["mergedAt"].replace("Z", "+00:00")) by_tool[tool].append((merged - made).total_seconds() / 3600) return {t: statistics.median(v) for t, v in by_tool.items()}, \ {t: len(v) for t, v in by_tool.items()} ``` The median is the right statistic here. A single pull request that waited over a holiday weekend would drag a mean upward by days, and with a dozen samples per tool that one outlier could flip the ranking. The median shrugs it off. Two cautions apply. Draft pull requests that sat open for a week before anyone asked for review inflate lead time for whoever opened them. If your team uses drafts heavily, export the `reviews` field instead and measure from the first review. And pull requests merged by bots (dependency updates, for instance) belong in the baseline only if humans would otherwise do that work. Filter them out in the export query when in doubt. Lead time rewards small pull requests. A tool that nudges developers into tiny, frequent changes will look fast. That can be a real benefit, but check the size of the changes before crediting the tool for speed. ## Step 9: Build the cost ledger and the developer survey Cost per merged pull request needs two inputs: what you paid and how many PRs merged. The ledger module sums seat and usage charges per tool, and Step 11 divides by the PR count from Step 8. ``` python import csv from collections import defaultdict def totals(costs_path): spend = defaultdict(float) with open(costs_path, newline="") as fh: for row in csv.DictReader(fh): spend[row["tool"]] += float(row["seat_usd"]) + float(row["usage_usd"]) return dict(spend) def survey_means(path): scores = defaultdict(list) with open(path, newline="") as fh: for row in csv.DictReader(fh): scores[row["tool"]].append(float(row["rating"])) return {t: sum(v) / len(v) for t, v in scores.items()} ``` The expected file, `data/costs.csv`, has one row per tool per week. ``` tool,week,seat_usd,usage_usd assistant-a,1,5.0,10.9 assistant-a,2,5.0,14.66 assistant-a,3,5.0,14.94 assistant-a,4,5.0,16.8 ``` Where the numbers come from differs by vendor, and this is the most error-prone chore in the whole pilot. Seat charges appear on the invoice or billing page for each plan. Usage charges hide in other places. Copilot meters usage in GitHub AI Credits, where one credit equals $0.01, and each plan includes a monthly allowance. Codex usage counts against plan limits and credits. Claude Code usage depends on whether your developers sign in with a plan or an API key. Cursor’s billing is changing, so read the current dashboard. Convert everything to dollars per week and write one row per tool. If a vendor bills in credits, multiply by the credit price on the invoice. If a seat is shared across your other work, allocate only the share used on the pilot repository. Honest allocation matters more than precision. Write the allocation rule at the top of your pilot log so the reviewer who reads the report in six months can reproduce it. The survey file is simpler: one row per developer per tool with a rating from 1 to 5, collected in the last week. ``` tool,dev,rating assistant-a,dev1,3 assistant-a,dev2,5 ``` Keep the survey to a single question, such as “How much would you miss this tool if it disappeared tomorrow?” Longer surveys get abandoned, and a single clear question produces cleaner data. The module averages ratings per tool. Because sentiment carries only 15% of the default score, a loved but slow tool cannot win on affection alone. Security reviewers will ask what leaves the building during a pilot. Settle that with your security team before you start, and read the [overview of AI assistant security and data exposure](https://shattered.io/ai-assistant-security-copilots-and-your-data/) to prepare for the questions about prompts, telemetry and retention. ## Step 10: Score and rank the assistants Four metrics live on four different scales: hours, percentages, dollars and a 1-to-5 rating. The scorer converts each one to a 0-to-1 value with min-max normalization across the tools in the pilot, flips the direction where lower is better, and combines them with the weights from the config. | Dimension | Default weight | Metric | Better when | |---|---|---|---| | Speed | 0.25 | Median hours from PR open to merge | Lower | | Quality | 0.35 | Share of PRs repaired by a Fixes: trailer within 7 days | Lower | | Cost | 0.25 | Seat plus usage dollars divided by merged PRs | Lower | | Survey | 0.15 | Mean developer rating, 1 to 5 | Higher | The scorer is short enough to read in one sitting. ``` python def normalise(values, lower_is_better): lo, hi = min(values.values()), max(values.values()) if hi == lo: return {k: 1.0 for k in values} span = hi - lo return {k: (hi - v) / span if lower_is_better else (v - lo) / span for k, v in values.items()} def rank(metrics, weights): parts = { "speed": normalise(metrics["lead_hours"], True), "quality": normalise(metrics["rework"], True), "cost": normalise(metrics["cost_per_pr"], True), "survey": normalise(metrics["survey"], False), } tools = metrics["lead_hours"].keys() scored = {t: round(100 * sum(weights[k] * parts[k][t] for k in weights), 1) for t in tools} return sorted(scored.items(), key=lambda kv: kv[1], reverse=True), parts ``` Min-max scaling has a quirk you should know about before you trust the output. The best tool on each metric gets 1.0 and the worst gets 0.0, no matter how small the gap between them. If two tools differ by 30 minutes of median lead time, the scaled speed score still spreads them across the full range. That is why Step 12 stress-tests the weights and why the report prints the raw numbers beside the score. A tie on any metric returns 1.0 for everyone, which avoids a divide-by-zero error and gives that metric no influence on the ranking. The minimum sample guard sits in its own two-line module. It returns the tools with fewer merged PRs than the floor, and the CLI prints a warning instead of silently ranking a tool on four data points. ``` python def too_small(counts, floor): return sorted(t for t, n in counts.items() if n < floor) ``` ## Step 11: Wire up the CLI, generate test data and run the pilot The command-line module ties the pieces together. The `collect` function gathers every metric into one dictionary, and `main` filters to tools that have both PR data and cost data, applies the optional weights override, prints the table and shows the human-only baseline underneath. ``` python import argparse import sys from dataclasses import replace from . import config, gitmine, ledger, metrics, score from .min_sample import too_small def collect(cfg): commits = gitmine.read_commits(cfg.repo, cfg.since, cfg.trailer, cfg.baseline) rw, commit_counts = metrics.rework(commits, cfg.fix_window_days) lead, pr_counts = metrics.lead_times(cfg.prs, cfg.baseline) spend = ledger.totals(cfg.costs) cost_per_pr = {t: spend[t] / pr_counts[t] for t in spend if t in pr_counts} return { "rework": rw, "commits": commit_counts, "lead_hours": lead, "prs": pr_counts, "spend": spend, "cost_per_pr": cost_per_pr, "survey": ledger.survey_means(cfg.survey), } def main(argv=None): ap = argparse.ArgumentParser(prog="pilot") ap.add_argument("--config", default="pilot.toml") ap.add_argument("--min-prs", type=int, default=8) ap.add_argument("--weights", help="override, e.g. speed=0.1,quality=0.5,cost=0.3,survey=0.1") args = ap.parse_args(argv) cfg = config.load(args.config) if args.weights: override = {k: float(v) for k, v in (kv.split("=") for kv in args.weights.split(","))} config.check_weights(override) cfg = replace(cfg, weights=override) m = collect(cfg) tools = [t for t in m["lead_hours"] if t != cfg.baseline and t in m["cost_per_pr"]] small = too_small({t: m["prs"][t] for t in tools}, args.min_prs) if small: print(f"WARNING: fewer than {args.min_prs} merged PRs for: {', '.join(small)}", file=sys.stderr) view = {k: {t: v[t] for t in tools} for k, v in m.items() if k in ("rework", "lead_hours", "cost_per_pr", "survey")} ranking, _ = score.rank(view, cfg.weights) print(f"{'tool':<14}{'PRs':>5}{'lead h':>9}{'rework':>9}{'$/PR':>8}{'survey':>8}{'score':>8}") for tool, pts in ranking: print(f"{tool:<14}{m['prs'][tool]:>5}{m['lead_hours'][tool]:>9.1f}" f"{m['rework'][tool]:>8.0%} {m['cost_per_pr'][tool]:>8.2f}" f"{m['survey'][tool]:>8.1f}{pts:>8.1f}") b = cfg.baseline print(f"\nbaseline ({b}): {m['prs'].get(b, 0)} PRs, " f"median {m['lead_hours'].get(b, float('nan')):.1f} h, " f"rework {m['rework'].get(b, float('nan')):.0%}") return 0 if __name__ == "__main__": raise SystemExit(main()) ``` Add a two-line `pilot/__main__.py` so `python3 -m pilot` works. ``` python from .cli import main raise SystemExit(main()) ``` You cannot run a four-week pilot to test the tooling, so the project ships a fixture generator. It builds a throwaway git repository with dated commits, trailers and fix references, plus matching PR, cost and survey files. Every number is random but seeded, so reruns produce identical files. The tool names are deliberately generic (assistant-a through assistant-d) so nobody mistakes the output for a real result. ``` """Builds a synthetic repo plus PR, cost and survey exports. All numbers are random but seeded.""" import csv import json import os import random import shutil import subprocess from datetime import datetime, timedelta, timezone from pathlib import Path rng = random.Random(2026) ROOT = Path("fixture-repo") shutil.rmtree(ROOT, ignore_errors=True) Path("data").mkdir(exist_ok=True) ROOT.mkdir() PROFILE = { # tool: (PR count, median lead hours, chance of a follow-up fix, weekly seat, weekly usage) "human": (20, 30.0, 0.25, 0.0, 0.0), "assistant-a": (14, 18.0, 0.20, 5.0, 14.0), "assistant-b": (12, 22.0, 0.12, 5.0, 31.0), "assistant-c": (11, 16.0, 0.35, 2.3, 6.0), "assistant-d": (6, 20.0, 0.18, 9.0, 4.0), } FILES = [f"src/mod{i}.py" for i in range(1, 9)] def git(*args, when=None): env = dict(os.environ, GIT_AUTHOR_NAME="Dev", GIT_AUTHOR_EMAIL="[email protected]", GIT_COMMITTER_NAME="Dev", GIT_COMMITTER_EMAIL="[email protected]") if when: env["GIT_AUTHOR_DATE"] = env["GIT_COMMITTER_DATE"] = when.isoformat() subprocess.run(["git", "-C", str(ROOT), *args], check=True, env=env, capture_output=True) def commit(subject, tool, files, when, n, fixes=None): for f in files: p = ROOT / f p.parent.mkdir(parents=True, exist_ok=True) with p.open("a") as fh: fh.write("".join(f"line {n}.{i}\n" for i in range(rng.randint(3, 40)))) git("add", "-A") trailers = ([f"Assisted-by: {tool}"] if tool != "human" else []) + \ ([f"Fixes: #{fixes}"] if fixes else []) msg = ["commit", "-q", "-m", subject] if trailers: msg += ["-m", "\n".join(trailers)] git(*msg, when=when) git("init", "-q", "-b", "main") start = datetime(2026, 9, 1, 9, tzinfo=timezone.utc) jobs = [t for t, p in PROFILE.items() for _ in range(p[0])] rng.shuffle(jobs) prs, n = [], 0 for i, tool in enumerate(jobs, 1): n += 1 opened = start + timedelta(hours=rng.uniform(0, 24 * 45)) lead = max(2.0, rng.gauss(PROFILE[tool][1], PROFILE[tool][1] * 0.3)) merged = opened + timedelta(hours=lead) files = rng.sample(FILES, rng.randint(1, 3)) commit(f"feat: change {i} (#{i})", tool, files, merged, n) if rng.random() < PROFILE[tool][2]: n += 1 fix_at = merged + timedelta(days=rng.uniform(0.5, 5)) commit(f"fix: follow-up for change {i}", "human", files[:1], fix_at, n, fixes=i) prs.append({"number": i, "title": f"change {i}", "createdAt": opened.strftime("%Y-%m-%dT%H:%M:%SZ"), "mergedAt": merged.strftime("%Y-%m-%dT%H:%M:%SZ"), "labels": [] if tool == "human" else [{"name": f"ai:{tool}"}]}) Path("data/prs.json").write_text(json.dumps(prs, indent=1)) with open("data/costs.csv", "w", newline="") as fh: w = csv.writer(fh) w.writerow(["tool", "week", "seat_usd", "usage_usd"]) for tool, p in PROFILE.items(): if tool == "human": continue for week in range(1, 8): w.writerow([tool, week, p[3], round(p[4] * rng.uniform(0.7, 1.3), 2)]) with open("data/survey.csv", "w", newline="") as fh: w = csv.writer(fh) w.writerow(["tool", "dev", "rating"]) base = {"assistant-a": 4, "assistant-b": 4, "assistant-c": 3, "assistant-d": 3} for tool, b in base.items(): for dev in range(1, 6): w.writerow([tool, f"dev{dev}", min(5, max(1, b + rng.choice([-1, 0, 0, 1])))]) print(f"fixture ready: {len(prs)} PRs") ``` Run the generator and the report: ``` bash $ python3 make_fixture.py fixture ready: 63 PRs $ python3 -m pilot WARNING: fewer than 8 merged PRs for: assistant-d tool PRs lead h rework $/PR survey score assistant-a 14 16.5 7% 9.97 3.6 73.6 assistant-c 11 11.8 36% 5.38 3.2 50.0 assistant-b 12 21.6 8% 21.95 4.0 48.6 assistant-d 6 18.8 17% 15.62 3.4 44.0 baseline (human): 20 PRs, median 31.6 h, rework 10% ``` Read the output in layers. The warning fires first because assistant-d merged only six pull requests, below the floor of eight, so its row is a placeholder and not evidence. The table sorts by score. Assistant-a leads because it combines a fast merge time, low rework and the second-lowest cost per PR. Assistant-c is the cheapest and fastest but fixes 36% of its pull requests afterward, and that quality penalty drags it from first place on speed to second overall. Assistant-b is the best loved and the most expensive. The baseline row shows what the humans did alone, so every assistant can be judged against doing nothing. Now run the tests. They use only `unittest`, so there is nothing to install. ``` bash $ python3 -m unittest discover -s tests -v test_weights_must_sum_to_one (test_pilot.ConfigTests.test_weights_must_sum_to_one) ... ok test_fix_after_window_is_ignored (test_pilot.ReworkTests.test_fix_after_window_is_ignored) ... ok test_fix_inside_window_counts (test_pilot.ReworkTests.test_fix_inside_window_counts) ... ok test_unlinked_fix_is_not_guessed (test_pilot.ReworkTests.test_unlinked_fix_is_not_guessed) ... ok test_normalise_lower_is_better (test_pilot.ScoreTests.test_normalise_lower_is_better) ... ok test_rank_orders_by_weighted_sum (test_pilot.ScoreTests.test_rank_orders_by_weighted_sum) ... ok test_tie_does_not_divide_by_zero (test_pilot.ScoreTests.test_tie_does_not_divide_by_zero) ... ok Ran 7 tests in 0.000s OK ``` The test file covers the three places where a silent error would mislead a decision: the normalization direction, the fix window edges and the refusal to guess at unlinked fixes. ``` python import unittest from datetime import datetime, timedelta from pilot import config, metrics, score from pilot.gitmine import Commit T0 = datetime(2026, 9, 1, 9) def c(sha, days, subject, tool, pr=None, fixes=()): return Commit(sha, T0 + timedelta(days=days), subject, tool, pr, list(fixes)) class ScoreTests(unittest.TestCase): def test_normalise_lower_is_better(self): out = score.normalise({"a": 10, "b": 20, "c": 30}, lower_is_better=True) self.assertEqual(out, {"a": 1.0, "b": 0.5, "c": 0.0}) def test_tie_does_not_divide_by_zero(self): self.assertEqual(score.normalise({"a": 5, "b": 5}, True), {"a": 1.0, "b": 1.0}) def test_rank_orders_by_weighted_sum(self): m = {"lead_hours": {"x": 10, "y": 20}, "rework": {"x": 0.1, "y": 0.1}, "cost_per_pr": {"x": 5, "y": 5}, "survey": {"x": 3, "y": 5}} w = {"speed": 0.6, "quality": 0.0, "cost": 0.0, "survey": 0.4} ranking, _ = score.rank(m, w) self.assertEqual([t for t, _ in ranking], ["x", "y"]) self.assertEqual(dict(ranking), {"x": 60.0, "y": 40.0}) class ReworkTests(unittest.TestCase): def test_fix_inside_window_counts(self): commits = [c("1", 0, "feat (#1)", "a", 1), c("2", 3, "fix thing", "human", None, [1])] rate, total = metrics.rework(commits, 7) self.assertEqual((rate["a"], total["a"]), (1.0, 1)) def test_fix_after_window_is_ignored(self): commits = [c("1", 0, "feat (#1)", "a", 1), c("2", 9, "fix thing", "human", None, [1])] self.assertEqual(metrics.rework(commits, 7)[0]["a"], 0.0) def test_unlinked_fix_is_not_guessed(self): commits = [c("1", 0, "feat (#1)", "a", 1), c("2", 1, "fix thing", "human")] self.assertEqual(metrics.rework(commits, 7)[0]["a"], 0.0) class ConfigTests(unittest.TestCase): def test_weights_must_sum_to_one(self): with self.assertRaises(ValueError): config.check_weights({"speed": 0.5, "quality": 0.6}) if __name__ == "__main__": unittest.main() ``` High rework on a single tool often points at a review gap more than at the tool itself. Teams that see it should consider adding automated review in front of human review, and the [AI code review tools comparison](https://shattered.io/ai-code-review-tools-bugbot-vs-codex-2026/) lists options with their trade-offs. ## Step 12: Stress-test the weights and make the call A ranking that flips when you nudge the weights is not a ranking. Run the same data through several weight profiles and see which conclusions survive. The CLI takes an override for exactly this. ``` bash $ python3 -m pilot --weights speed=0.6,quality=0.1,cost=0.2,survey=0.1 $ python3 -m pilot --weights speed=0.1,quality=0.2,cost=0.6,survey=0.1 $ python3 -m pilot --weights speed=0.1,quality=0.6,cost=0.2,survey=0.1 ``` The four profiles produced this on the synthetic data: | Profile (speed/quality/cost/survey) | 1st | 2nd | 3rd | 4th | |---|---|---|---|---| | Default (0.25/0.35/0.25/0.15) | Not listed | assistant-c 50.0 | Not listed | assistant-d 44.0 | | Speed-heavy (0.6/0.1/0.2/0.1) | assistant-c 80.0 | Not listed | assistant-d 34.0 | Not listed | | Cost-heavy (0.1/0.2/0.6/0.1) | Not listed | assistant-c 70.0 | Not listed | Not listed | | Quality-heavy (0.1/0.6/0.2/0.1) | Not listed | Not listed | Not listed | assistant-c 30.0 | Assistant-a wins three of four profiles and comes second in the fourth, so the conclusion survives. Assistant-c wins only when speed dominates and collapses to last when quality dominates. That pattern tells you to avoid weighing it on one dimension. Assistant-b finishes last in two profiles because it is the slowest to merge and the most expensive per PR, so a negotiated price would help it only on the cost axis. Real pilots produce this kind of structure, and the stress test is how you read it. Add a statistical sanity check before you sign anything. The bootstrap helper in the advanced tips resamples each tool’s lead times and reports a 90% interval for the median. On the fixture, the intervals for assistants a, b, c and d overlap heavily, while all four sit clear of the human baseline. The honest reading: every assistant beat the baseline on speed, and the order among them is not settled by speed alone. Turn the numbers into a decision about which of the best AI coding assistants earns seats, using rules written in advance: | Situation | Action | Reason | |---|---|---| | One tool wins 3 of 4 weight profiles with 8+ PRs | Buy seats for the whole team | The result is stable under reasonable priorities | | Two tools trade first place by profile | Split by role or task type and re-pilot | The weights, not the data, decide the winner | | A tool has fewer PRs than the floor | Extend the pilot two weeks | Too few samples to rank | | A tool is fastest but has the highest rework | Pair it with automated review before adopting | Speed gains vanish if work returns as fixes | | No tool beats the human baseline on rework | Keep the pilot small and revisit next quarter | The assistants add cost without quality gain | Whatever you choose, schedule a re-run. Models change every few weeks, and a tool that lost in October can win in December. Keep the project in the repository, rerun it each quarter on fresh data and compare reports. More coverage of coding tools and the editors they live in sits in the [software and developer tools section](https://shattered.io/category/software/). ## Common pitfalls that skew an AI coding assistant pilot Most failed pilots fail on measurement design, not on tooling. Teams comparing the best AI coding assistants tend to repeat the same mistakes. 1. **Letting developers choose the tool per task.** People reach for the assistant they trust on the easy ticket and the human lane on the hard one. Rotate assignments on a schedule, so each tool sees a mix of work. 2. **Skipping the baseline.** Without a human-only row, a 16-hour median lead time means nothing. The baseline turns “fast” into “faster than we were”. 3. **Changing the weights after seeing results.** Weights encode priorities. Set them in the first week and commit the file, so any later change goes through review. 4. **Ranking on a thin sample.** Six pull requests cannot support a verdict. Keep the minimum-sample warning on, and extend the pilot instead of ranking noise. 5. **Counting seat price only.** Usage charges can match or exceed the seat fee. Copilot’s newest models bill at provider-list pricing, Codex usage counts against plan limits, and Cursor’s third-party model billing is shifting. Put every charge in the ledger. 6. **Ignoring mid-pilot model changes.** A default model swap is a new tool. Log versions weekly and flag any week where the default changed. 7. **Mixing PR size with speed.** Smaller pull requests merge faster. Compare changed lines per PR across tools before you credit a tool for lead time. 8. **Measuring only developers who volunteer.** Enthusiasts produce cheerful surveys and good numbers. Include at least one skeptic per tool. ## Troubleshooting Each item below lists the symptom, the likely cause and the fix. 1. **ValueError: weights must sum to 1.0.** The four weights in the config or the override add up to something else, and the message prints the actual total. Adjust them until they total exactly 1.0. 2. **ModuleNotFoundError: No module named ‘tomllib’.** You are on Python 3.10 or older. The standard library added tomllib in 3.11, so upgrade the interpreter. 3. **Every commit lands in the human bucket.** The hook never ran. Check that it is executable, that it sits at`.git/hooks/prepare-commit-msg` without a file extension, that`git config --get pilot.tool` prints a value, and that`core.hooksPath` does not point somewhere else. 4. **The report is empty or misses early weeks.** A shallow clone drops history. Run`git fetch --unshallow` , and confirm the`since` date in the config predates the pilot start. 5. **Trailers vanish after merge.** Squash merges rewrite messages. Check your repository’s squash message setting on a test PR, and fall back on PR labels, which survive squashing, when commit trailers do not. 6. **gh: command not found or authentication errors.** Install the GitHub CLI and run`gh auth login` , then confirm with`gh auth status` . Enterprise servers need the`--hostname` flag. 7. **The PR export has fewer rows than expected.**`gh pr list` returns 30 results by default. Raise`--limit` , and confirm the`--search` date filter matches your pilot start. 8. **A tool is missing from the table.** The slug in`costs.csv` does not match the label.`claude_code` and`claude-code` are different keys, and the CLI drops tools that lack either cost or PR data. Standardize on lowercase with hyphens. 9. **Rework shows 0% for everyone.** Either commit subjects lack the`(#N)` suffix or developers write`Closes #N` instead of`Fixes: #N` . Pick one convention, and add a CI check that rejects a “fix:” commit with no reference. 10. **Every score reads 100.0.** A tie on each metric normalizes to 1.0. That happens when you test with one tool, or when all tools have identical data. Add at least two tools with different numbers. 11. **fromisoformat raises ValueError on a timestamp.** GitHub returns a trailing Z that older Pythons reject. The code already swaps Z for +00:00, so check whether a custom export uses another format. 12. **Cost per PR shows a huge number.** The tool merged almost no pull requests, so a small count divides a full month’s spend. Treat it as a sample size warning, not a price. ## Advanced tips Once the basic run works, five additions make the pilot harder to argue with. **Add confidence intervals.** Resampling a tool’s lead times shows how much the median could move. Save the helper as `pilot/bootstrap.py`. It is short and uses only the standard library. ``` python import random import statistics def median_ci(samples, rounds=2000, seed=7): rng = random.Random(seed) meds = sorted(statistics.median(rng.choices(samples, k=len(samples))) for _ in range(rounds)) return meds[int(rounds * 0.05)], meds[int(rounds * 0.95)] ``` On the fixture it printed these 90% intervals for median lead time: ``` assistant-a n=14 median 16.5 h 90% interval 14.1 to 20.0 assistant-b n=12 median 21.6 h 90% interval 14.4 to 25.6 assistant-c n=11 median 11.8 h 90% interval 9.0 to 19.7 assistant-d n=6 median 18.8 h 90% interval 14.4 to 23.9 human n=20 median 31.6 h 90% interval 28.1 to 32.8 ``` The assistants overlap one another and sit apart from the baseline, which is exactly the claim the data supports. **Slice by task type.** Add labels such as `bug`, `feature` and `refactor`, then group the metrics by tool and label. A tool that struggles on refactors may shine on bug fixes, and a split purchase is sometimes the right answer. **Log the model.** Create a `models.log` with one line per tool per week holding the version string and the default model. Most CLIs print both. When a result looks odd, the log tells you whether the tool changed that week. **Measure review effort.** Export the `reviews` field from `gh pr list` and count review rounds per pull request. A tool that merges quickly but needs three review cycles moves cost from the author to the reviewer. **Add a zero-cost control.** If seat prices dominate your cost numbers, add a self-hosted lane. The [Aider local assistant setup](https://shattered.io/aider-local-ai-coding-assistant-ollama-git-2026/) shows how to run an open-source tool against a local model. Give it its own trailer and label, and the same report scores it next to the paid tools. ## FAQ **What are the best AI coding assistants in 2026?** The answer depends on your code and your team, which is why a measured pilot beats a ranking. Claude Code, OpenAI Codex, GitHub Copilot and Cursor are the four most common contenders, and each made product or model announcements in October 2026. Run the pilot on your repository and let the numbers decide. **How long should an AI coding assistant pilot run?** Four weeks is a practical minimum. That period usually yields 8 or more merged pull requests per tool when each developer rotates through all of them. Extend the pilot if any tool falls below the minimum sample. **Can I run the pilot without GitHub?** Yes. The git history side works on any host. Replace the `gh` export with a script that writes the same JSON shape (number, title, createdAt, mergedAt, labels) from the GitLab or Bitbucket API, and the rest of the project runs unchanged. **Why does the project use trailers instead of author names?** Developers switch tools during a week, and author names say nothing about which assistant helped. A trailer follows the commit itself, and git can read it natively without parsing the message body. **Does a higher score mean the tool writes better code?** No. The score blends speed, rework, cost and sentiment, which describe how a tool performs inside your workflow. It does not grade code quality directly. Use rework as the closest proxy and read the diffs of a few merged pull requests yourself. **How do I handle tools that change pricing mid-pilot?** Record the new price from the week it takes effect, and note the change in the pilot log. Cursor, for example, reportedly moved to new subscription tiers on October 12, so a pilot spanning that date should split its ledger rows at that point. **Should the survey carry more weight?** Raise it only if adoption is your main risk. A tool nobody wants to use produces no value even when the numbers look good. Otherwise keep it at 15% or lower, since sentiment tracks enjoyment as much as results. **How often should I rerun the pilot?** Quarterly is reasonable, and sooner when a vendor swaps its default model or changes pricing. Rerunning is cheap once the hook and labels are in place, since the project reads the same files each time.