Benchmark Cursor, Aider, and Claude Code on Your Own Repo

A new open-source bash harness lets developers benchmark AI coding agents Claude Code 2.1, Aider 0.86.1, and the Cursor CLI (binary named `agent`) on their own repositories, scoring each attempt with a test script and outputting a CSV scoreboard with pass/fail, wall-clock time, and diff size. The harness, detailed in a guide by Priya Nair on Sourcefeed, uses real closed issues as tasks and requires a throwaway clone because it hard-resets the working tree between runs.

Benchmark Cursor, Aider, and Claude Code on Your Own Repo Build a headless harness that scores AI coding agents on real issues from your codebase. Priya Nair https://sourcefeed.dev/u/priya nair What you'll build A bash harness that runs Claude Code https://code.claude.com/docs/en/overview , Aider https://aider.chat , and the Cursor CLI https://cursor.com/docs/cli/overview headless against real tasks from your own repo, grades each attempt with a test script, and writes a CSV scoreboard — pass/fail, wall-clock time, and diff size — so your agent choice comes from data, not vibes. Prerequisites - macOS or Linux WSL works . bash and git. - Verified against: Claude Code 2.1 native installer , aider 0.86.1, and the current Cursor CLI agent , all per official docs as of August 2026. - Python 3.8–3.13 aider's supported range . - Accounts: a Claude subscription or an ANTHROPIC API KEY ; a Cursor account API key from cursor.com/dashboard/api https://cursor.com/dashboard/api if you'll run in CI . Aider talks to model APIs directly — we'll point it at the same Anthropic key. - A repo with a runnable test suite, in a throwaway clone . The harness hard-resets the working tree between runs. Don't point it at your daily checkout. 1. Install the three agents Claude Code native installer curl -fsSL https://claude.ai/install.sh | bash claude first run opens browser login; then /exit Aider — aider-install puts it in an isolated env in ~/.local/bin python -m pip install aider-install aider-install Cursor CLI — yes, the binary is literally named agent curl https://cursor.com/install -fsS | bash agent login browser auth; in CI use: export CURSOR API KEY=... Aider reads this for Anthropic models export ANTHROPIC API KEY=sk-ant-... Confirm all three respond: claude --version , aider --version , agent --version . 2. Turn real issues into benchmark tasks Inside your throwaway clone, each task is a folder with a prompt and a grader: mkdir -p bench/tasks bench/logs mkdir bench/tasks/issue-142 cat bench/tasks/issue-142/prompt.md <<'EOF' Fix the bug where parse duration "90m" returns 90 seconds instead of 5400. The function lives in src/utils/time.py. Do not add dependencies. EOF cat bench/tasks/issue-142/check.sh <<'EOF' /usr/bin/env bash python -m pytest tests/test time.py -q EOF The gold standard for task selection: closed issues you've already fixed. On a bench branch, git revert your fix but keep or cherry-pick its regression test as check.sh — you get a known-solvable task with an objective grader. Faster alternative: open bugs or small features where you can write the failing test right now. Paste the issue text into prompt.md verbatim; don't add hints you wouldn't give a new teammate. Five to ten tasks is enough to separate the pack. 3. Write the harness Save as bench/bench.sh : bash /usr/bin/env bash Runs each agent against each task from a clean checkout, grades, logs. set -u cd "$ git rev-parse --show-toplevel " BASE=$ git rev-parse HEAD echo "agent,task,passed,seconds,files changed,insertions,deletions" bench/results.csv run one { case "$1" in claude claude -p "$2" --model sonnet \ --dangerously-skip-permissions --max-turns 30 ;; aider aider --message "$2" --model sonnet --yes-always \ --no-auto-commits --no-gitignore --no-stream --no-pretty ;; cursor agent -p "$2" --force --output-format text ;; esac } for task dir in bench/tasks/ /; do task=$ basename "$task dir" prompt=$ <"$task dir/prompt.md" for name in claude aider cursor; do git reset --hard -q "$BASE" && git clean -fdq -e bench echo "=== $name / $task ===" start=$SECONDS run one "$name" "$prompt" "bench/logs/$name-$task.log" 2 &1 elapsed=$ SECONDS - start if bash "$task dir/check.sh" "bench/logs/$name-$task.check.log" 2 &1 then passed=1; else passed=0 fi stat=$ git diff --shortstat files=$ grep -o ' 0-9 file' <<<"$stat" | grep -o ' 0-9 ' ins=$ grep -o ' 0-9 insertion' <<<"$stat" | grep -o ' 0-9 ' del=$ grep -o ' 0-9 deletion' <<<"$stat" | grep -o ' 0-9 ' echo "$name,$task,$passed,$elapsed,${files:-0},${ins:-0},${del:-0}" \ bench/results.csv done done git reset --hard -q "$BASE" && git clean -fdq -e bench column -t -s, bench/results.csv Three design decisions worth knowing: Every run starts from the same commit. git reset --hard plus git clean -fd wipes each agent's changes before the next run; -e bench excludes the harness itself from the wipe. Each agent gets its own "just do it" flag , because headless runs can't answer approval prompts: --dangerously-skip-permissions Claude Code , --yes-always aider , --force Cursor . This is exactly why you're in a disposable clone. Aider also gets --no-auto-commits so its edits stay uncommitted and diffable like the others', and --no-gitignore so it doesn't edit .gitignore and pollute the diff stats. Models are pinned. All three run Anthropic's Sonnet here agent --list-models shows Cursor's exact names — add --model to its arm , so you're benchmarking the agent scaffolding, not different models. Drop the --model flags instead if you want to compare each product as shipped, defaults and all. Diff stats count tracked files only — new files an agent creates won't show — so treat passed as the score and the diff columns as a code-churn tiebreaker. 4. Run the benchmark chmod +x bench/bench.sh bench/tasks/ /check.sh ./bench/bench.sh Budget one to five minutes per agent-task pair; a 3×8 matrix is a coffee break. Follow along in another terminal with tail -f bench/logs/ .log . Verify it works Before the full matrix, smoke-test with a single trivial task e.g., "make tests/test smoke.py pass" with a one-line fix . A healthy run ends with a table like: agent task passed seconds files changed insertions deletions claude issue-142 1 147 2 38 6 aider issue-142 1 63 1 12 4 cursor issue-142 0 201 3 120 41 Every row present, no seconds under ~10 that usually means the agent errored out instantly — check its log , and git status clean afterward except for bench/ . Troubleshooting — you're running Claude Code as root, typically in a Docker CI image. Create a non-root user, or swap the flag for --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons --permission-mode acceptEdits --allowedTools "Bash" .right after installing — both installers drop binaries into aider: command not found or agent: command not found ~/.local/bin , which isn't on PATH in fresh shells. export PATH="$HOME/.local/bin:$PATH" or restart the terminal. Older Cursor CLI installs named the binary cursor-agent ; re-run the installer to get agent . — your bench/tasks/...: No such file or directory on the second task bench/ directory got deleted because git clean -fd ran without -e bench . It removes all untracked files; restore the folder and keep the exclude. Cursor rows always show — print mode won't modify files without passed=0 with zero files changed --force , and an unauthenticated CLI fails silently into the log. Check agent status , and in CI make sure CURSOR API KEY is exported. Next steps Agents are nondeterministic, so run each task 3–5 times and report pass rate, not a single coin flip — wrap the inner loop in for trial in 1 2 3 . Add a cost column: claude -p --output-format json returns structured results including total cost, and aider prints session cost at the end of each run. Extending the field is one case arm per newcomer — OpenAI's Codex CLI and Google's Gemini CLI slot right in. And when you want to see how your private numbers compare to public ones, SWE-bench https://www.swebench.com is the same idea — real issues, test-based grading — at research scale. Sources & further reading - Claude Code CLI reference https://code.claude.com/docs/en/cli-reference — code.claude.com - Claude Code quickstart https://code.claude.com/docs/en/quickstart — code.claude.com - Scripting aider https://aider.chat/docs/scripting.html — aider.chat - Aider options reference https://aider.chat/docs/config/options.html — aider.chat - Cursor CLI parameters https://cursor.com/docs/cli/reference/parameters — cursor.com - Cursor CLI authentication https://cursor.com/docs/cli/reference/authentication — cursor.com Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.