{"slug": "benchmark-cursor-aider-and-claude-code-on-your-own-repo", "title": "Benchmark Cursor, Aider, and Claude Code on Your Own Repo", "summary": "A new open-source bash harness lets developers benchmark AI coding agents Claude Code 2.1, Aider 0.86.1, and the Cursor CLI (binary named `agent`) on their own repositories, scoring each attempt with a test script and outputting a CSV scoreboard with pass/fail, wall-clock time, and diff size. The harness, detailed in a guide by Priya Nair on Sourcefeed, uses real closed issues as tasks and requires a throwaway clone because it hard-resets the working tree between runs.", "body_md": "# Benchmark Cursor, Aider, and Claude Code on Your Own Repo\n\nBuild a headless harness that scores AI coding agents on real issues from your codebase.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\n## What you'll build\n\nA bash harness that runs [Claude Code](https://code.claude.com/docs/en/overview), [Aider](https://aider.chat), and the [Cursor CLI](https://cursor.com/docs/cli/overview) headless against real tasks from your own repo, grades each attempt with a test script, and writes a CSV scoreboard — pass/fail, wall-clock time, and diff size — so your agent choice comes from data, not vibes.\n\n## Prerequisites\n\n- macOS or Linux (WSL works). bash and git.\n- Verified against: Claude Code 2.1 (native installer), aider 0.86.1, and the current Cursor CLI (\n`agent`\n\n), all per official docs as of August 2026. - Python 3.8–3.13 (aider's supported range).\n- Accounts: a Claude subscription or an\n`ANTHROPIC_API_KEY`\n\n; a Cursor account (API key from[cursor.com/dashboard/api](https://cursor.com/dashboard/api)if you'll run in CI). Aider talks to model APIs directly — we'll point it at the same Anthropic key. - A repo with a runnable test suite, in a\n**throwaway clone**. The harness hard-resets the working tree between runs. Don't point it at your daily checkout.\n\n## 1. Install the three agents\n\n```\n# Claude Code (native installer)\ncurl -fsSL https://claude.ai/install.sh | bash\nclaude   # first run opens browser login; then /exit\n\n# Aider — aider-install puts it in an isolated env in ~/.local/bin\npython -m pip install aider-install\naider-install\n\n# Cursor CLI — yes, the binary is literally named `agent`\ncurl https://cursor.com/install -fsS | bash\nagent login   # browser auth; in CI use: export CURSOR_API_KEY=...\n\n# Aider reads this for Anthropic models\nexport ANTHROPIC_API_KEY=sk-ant-...\n```\n\nConfirm all three respond: `claude --version`\n\n, `aider --version`\n\n, `agent --version`\n\n.\n\n## 2. Turn real issues into benchmark tasks\n\nInside your throwaway clone, each task is a folder with a prompt and a grader:\n\n```\nmkdir -p bench/tasks bench/logs\nmkdir bench/tasks/issue-142\ncat > bench/tasks/issue-142/prompt.md <<'EOF'\nFix the bug where parse_duration(\"90m\") returns 90 seconds instead of\n5400. The function lives in src/utils/time.py. Do not add dependencies.\nEOF\ncat > bench/tasks/issue-142/check.sh <<'EOF'\n#!/usr/bin/env bash\npython -m pytest tests/test_time.py -q\nEOF\n```\n\nThe gold standard for task selection: closed issues you've already fixed. On a `bench`\n\nbranch, `git revert`\n\nyour fix but keep (or cherry-pick) its regression test as `check.sh`\n\n— you get a known-solvable task with an objective grader. Faster alternative: open bugs or small features where you can write the failing test right now. Paste the issue text into `prompt.md`\n\nverbatim; don't add hints you wouldn't give a new teammate. Five to ten tasks is enough to separate the pack.\n\n## 3. Write the harness\n\nSave as `bench/bench.sh`\n\n:\n\n``` bash\n#!/usr/bin/env bash\n# Runs each agent against each task from a clean checkout, grades, logs.\nset -u\ncd \"$(git rev-parse --show-toplevel)\"\n\nBASE=$(git rev-parse HEAD)\necho \"agent,task,passed,seconds,files_changed,insertions,deletions\" > bench/results.csv\n\nrun_one() {\n  case \"$1\" in\n    claude) claude -p \"$2\" --model sonnet \\\n              --dangerously-skip-permissions --max-turns 30 ;;\n    aider)  aider --message \"$2\" --model sonnet --yes-always \\\n              --no-auto-commits --no-gitignore --no-stream --no-pretty ;;\n    cursor) agent -p \"$2\" --force --output-format text ;;\n  esac\n}\n\nfor task_dir in bench/tasks/*/; do\n  task=$(basename \"$task_dir\")\n  prompt=$(<\"$task_dir/prompt.md\")\n  for name in claude aider cursor; do\n    git reset --hard -q \"$BASE\" && git clean -fdq -e bench\n    echo \"=== $name / $task ===\"\n    start=$SECONDS\n    run_one \"$name\" \"$prompt\" > \"bench/logs/$name-$task.log\" 2>&1\n    elapsed=$(( SECONDS - start ))\n    if bash \"$task_dir/check.sh\" > \"bench/logs/$name-$task.check.log\" 2>&1\n      then passed=1; else passed=0\n    fi\n    stat=$(git diff --shortstat)\n    files=$(grep -o '[0-9]* file' <<<\"$stat\" | grep -o '[0-9]*')\n    ins=$(grep -o '[0-9]* insertion' <<<\"$stat\" | grep -o '[0-9]*')\n    del=$(grep -o '[0-9]* deletion' <<<\"$stat\" | grep -o '[0-9]*')\n    echo \"$name,$task,$passed,$elapsed,${files:-0},${ins:-0},${del:-0}\" \\\n      >> bench/results.csv\n  done\ndone\n\ngit reset --hard -q \"$BASE\" && git clean -fdq -e bench\ncolumn -t -s, bench/results.csv\n```\n\nThree design decisions worth knowing:\n\n**Every run starts from the same commit.**`git reset --hard`\n\nplus`git clean -fd`\n\nwipes each agent's changes before the next run;`-e bench`\n\nexcludes the harness itself from the wipe.**Each agent gets its own \"just do it\" flag**, because headless runs can't answer approval prompts:`--dangerously-skip-permissions`\n\n(Claude Code),`--yes-always`\n\n(aider),`--force`\n\n(Cursor). This is exactly why you're in a disposable clone. Aider also gets`--no-auto-commits`\n\nso its edits stay uncommitted and diffable like the others', and`--no-gitignore`\n\nso it doesn't edit`.gitignore`\n\nand pollute the diff stats.**Models are pinned.** All three run Anthropic's Sonnet here (`agent --list-models`\n\nshows Cursor's exact names — add`--model`\n\nto its arm), so you're benchmarking the agent scaffolding, not different models. Drop the`--model`\n\nflags instead if you want to compare each product as shipped, defaults and all.\n\nDiff stats count tracked files only — new files an agent creates won't show — so treat `passed`\n\nas the score and the diff columns as a code-churn tiebreaker.\n\n## 4. Run the benchmark\n\n```\nchmod +x bench/bench.sh bench/tasks/*/check.sh\n./bench/bench.sh\n```\n\nBudget one to five minutes per agent-task pair; a 3×8 matrix is a coffee break. Follow along in another terminal with `tail -f bench/logs/*.log`\n\n.\n\n## Verify it works\n\nBefore the full matrix, smoke-test with a single trivial task (e.g., \"make `tests/test_smoke.py`\n\npass\" with a one-line fix). A healthy run ends with a table like:\n\n```\nagent   task       passed  seconds  files_changed  insertions  deletions\nclaude  issue-142  1       147      2              38          6\naider   issue-142  1       63       1              12          4\ncursor  issue-142  0       201      3              120         41\n```\n\nEvery row present, no `seconds`\n\nunder ~10 (that usually means the agent errored out instantly — check its log), and `git status`\n\nclean afterward except for `bench/`\n\n.\n\n## Troubleshooting\n\n— you're running Claude Code as root, typically in a Docker CI image. Create a non-root user, or swap the flag for`--dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons`\n\n`--permission-mode acceptEdits --allowedTools \"Bash\"`\n\n.right after installing — both installers drop binaries into`aider: command not found`\n\nor`agent: command not found`\n\n`~/.local/bin`\n\n, which isn't on PATH in fresh shells.`export PATH=\"$HOME/.local/bin:$PATH\"`\n\nor restart the terminal. (Older Cursor CLI installs named the binary`cursor-agent`\n\n; re-run the installer to get`agent`\n\n.)— your`bench/tasks/...: No such file or directory`\n\non the second task`bench/`\n\ndirectory got deleted because`git clean -fd`\n\nran without`-e bench`\n\n. It removes*all*untracked files; restore the folder and keep the exclude.**Cursor rows always show**— print mode won't modify files without`passed=0`\n\nwith zero files changed`--force`\n\n, and an unauthenticated CLI fails silently into the log. Check`agent status`\n\n, and in CI make sure`CURSOR_API_KEY`\n\nis exported.\n\n## Next steps\n\nAgents are nondeterministic, so run each task 3–5 times and report pass rate, not a single coin flip — wrap the inner loop in `for trial in 1 2 3`\n\n. Add a cost column: `claude -p --output-format json`\n\nreturns structured results including total cost, and aider prints session cost at the end of each run. Extending the field is one `case`\n\narm per newcomer — OpenAI's Codex CLI and Google's Gemini CLI slot right in. And when you want to see how your private numbers compare to public ones, [SWE-bench](https://www.swebench.com) is the same idea — real issues, test-based grading — at research scale.\n\n## Sources & further reading\n\n-\n[Claude Code CLI reference](https://code.claude.com/docs/en/cli-reference)— code.claude.com -\n[Claude Code quickstart](https://code.claude.com/docs/en/quickstart)— code.claude.com -\n[Scripting aider](https://aider.chat/docs/scripting.html)— aider.chat -\n[Aider options reference](https://aider.chat/docs/config/options.html)— aider.chat -\n[Cursor CLI parameters](https://cursor.com/docs/cli/reference/parameters)— cursor.com -\n[Cursor CLI authentication](https://cursor.com/docs/cli/reference/authentication)— cursor.com\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/benchmark-cursor-aider-and-claude-code-on-your-own-repo", "canonical_source": "https://sourcefeed.dev/a/benchmark-cursor-aider-and-claude-code-on-your-own-repo", "published_at": "2026-08-09 11:41:22+00:00", "updated_at": "2026-08-09 12:08:14.357517+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "artificial-intelligence"], "entities": ["Claude Code", "Aider", "Cursor CLI", "Priya Nair", "Sourcefeed", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/benchmark-cursor-aider-and-claude-code-on-your-own-repo", "markdown": "https://wpnews.pro/news/benchmark-cursor-aider-and-claude-code-on-your-own-repo.md", "text": "https://wpnews.pro/news/benchmark-cursor-aider-and-claude-code-on-your-own-repo.txt", "jsonld": "https://wpnews.pro/news/benchmark-cursor-aider-and-claude-code-on-your-own-repo.jsonld"}}