# Benchmark Cursor, Aider, and Claude Code on Your Own Repo

> Source: <https://sourcefeed.dev/a/benchmark-cursor-aider-and-claude-code-on-your-own-repo>
> Published: 2026-08-09 11:41:22+00:00

# Benchmark Cursor, Aider, and Claude Code on Your Own Repo

Build a headless harness that scores AI coding agents on real issues from your codebase.

[Priya Nair](https://sourcefeed.dev/u/priya_nair)

## What you'll build

A bash harness that runs [Claude Code](https://code.claude.com/docs/en/overview), [Aider](https://aider.chat), and the [Cursor CLI](https://cursor.com/docs/cli/overview) headless against real tasks from your own repo, grades each attempt with a test script, and writes a CSV scoreboard — pass/fail, wall-clock time, and diff size — so your agent choice comes from data, not vibes.

## Prerequisites

- macOS or Linux (WSL works). bash and git.
- Verified against: Claude Code 2.1 (native installer), aider 0.86.1, and the current Cursor CLI (
`agent`

), all per official docs as of August 2026. - Python 3.8–3.13 (aider's supported range).
- Accounts: a Claude subscription or an
`ANTHROPIC_API_KEY`

; a Cursor account (API key from[cursor.com/dashboard/api](https://cursor.com/dashboard/api)if you'll run in CI). Aider talks to model APIs directly — we'll point it at the same Anthropic key. - A repo with a runnable test suite, in a
**throwaway clone**. The harness hard-resets the working tree between runs. Don't point it at your daily checkout.

## 1. Install the three agents

```
# Claude Code (native installer)
curl -fsSL https://claude.ai/install.sh | bash
claude   # first run opens browser login; then /exit

# Aider — aider-install puts it in an isolated env in ~/.local/bin
python -m pip install aider-install
aider-install

# Cursor CLI — yes, the binary is literally named `agent`
curl https://cursor.com/install -fsS | bash
agent login   # browser auth; in CI use: export CURSOR_API_KEY=...

# Aider reads this for Anthropic models
export ANTHROPIC_API_KEY=sk-ant-...
```

Confirm all three respond: `claude --version`

, `aider --version`

, `agent --version`

.

## 2. Turn real issues into benchmark tasks

Inside your throwaway clone, each task is a folder with a prompt and a grader:

```
mkdir -p bench/tasks bench/logs
mkdir bench/tasks/issue-142
cat > bench/tasks/issue-142/prompt.md <<'EOF'
Fix the bug where parse_duration("90m") returns 90 seconds instead of
5400. The function lives in src/utils/time.py. Do not add dependencies.
EOF
cat > bench/tasks/issue-142/check.sh <<'EOF'
#!/usr/bin/env bash
python -m pytest tests/test_time.py -q
EOF
```

The gold standard for task selection: closed issues you've already fixed. On a `bench`

branch, `git revert`

your fix but keep (or cherry-pick) its regression test as `check.sh`

— you get a known-solvable task with an objective grader. Faster alternative: open bugs or small features where you can write the failing test right now. Paste the issue text into `prompt.md`

verbatim; don't add hints you wouldn't give a new teammate. Five to ten tasks is enough to separate the pack.

## 3. Write the harness

Save as `bench/bench.sh`

:

``` bash
#!/usr/bin/env bash
# Runs each agent against each task from a clean checkout, grades, logs.
set -u
cd "$(git rev-parse --show-toplevel)"

BASE=$(git rev-parse HEAD)
echo "agent,task,passed,seconds,files_changed,insertions,deletions" > bench/results.csv

run_one() {
  case "$1" in
    claude) claude -p "$2" --model sonnet \
              --dangerously-skip-permissions --max-turns 30 ;;
    aider)  aider --message "$2" --model sonnet --yes-always \
              --no-auto-commits --no-gitignore --no-stream --no-pretty ;;
    cursor) agent -p "$2" --force --output-format text ;;
  esac
}

for task_dir in bench/tasks/*/; do
  task=$(basename "$task_dir")
  prompt=$(<"$task_dir/prompt.md")
  for name in claude aider cursor; do
    git reset --hard -q "$BASE" && git clean -fdq -e bench
    echo "=== $name / $task ==="
    start=$SECONDS
    run_one "$name" "$prompt" > "bench/logs/$name-$task.log" 2>&1
    elapsed=$(( SECONDS - start ))
    if bash "$task_dir/check.sh" > "bench/logs/$name-$task.check.log" 2>&1
      then passed=1; else passed=0
    fi
    stat=$(git diff --shortstat)
    files=$(grep -o '[0-9]* file' <<<"$stat" | grep -o '[0-9]*')
    ins=$(grep -o '[0-9]* insertion' <<<"$stat" | grep -o '[0-9]*')
    del=$(grep -o '[0-9]* deletion' <<<"$stat" | grep -o '[0-9]*')
    echo "$name,$task,$passed,$elapsed,${files:-0},${ins:-0},${del:-0}" \
      >> bench/results.csv
  done
done

git reset --hard -q "$BASE" && git clean -fdq -e bench
column -t -s, bench/results.csv
```

Three design decisions worth knowing:

**Every run starts from the same commit.**`git reset --hard`

plus`git clean -fd`

wipes each agent's changes before the next run;`-e bench`

excludes the harness itself from the wipe.**Each agent gets its own "just do it" flag**, because headless runs can't answer approval prompts:`--dangerously-skip-permissions`

(Claude Code),`--yes-always`

(aider),`--force`

(Cursor). This is exactly why you're in a disposable clone. Aider also gets`--no-auto-commits`

so its edits stay uncommitted and diffable like the others', and`--no-gitignore`

so it doesn't edit`.gitignore`

and pollute the diff stats.**Models are pinned.** All three run Anthropic's Sonnet here (`agent --list-models`

shows Cursor's exact names — add`--model`

to its arm), so you're benchmarking the agent scaffolding, not different models. Drop the`--model`

flags instead if you want to compare each product as shipped, defaults and all.

Diff stats count tracked files only — new files an agent creates won't show — so treat `passed`

as the score and the diff columns as a code-churn tiebreaker.

## 4. Run the benchmark

```
chmod +x bench/bench.sh bench/tasks/*/check.sh
./bench/bench.sh
```

Budget one to five minutes per agent-task pair; a 3×8 matrix is a coffee break. Follow along in another terminal with `tail -f bench/logs/*.log`

.

## Verify it works

Before the full matrix, smoke-test with a single trivial task (e.g., "make `tests/test_smoke.py`

pass" with a one-line fix). A healthy run ends with a table like:

```
agent   task       passed  seconds  files_changed  insertions  deletions
claude  issue-142  1       147      2              38          6
aider   issue-142  1       63       1              12          4
cursor  issue-142  0       201      3              120         41
```

Every row present, no `seconds`

under ~10 (that usually means the agent errored out instantly — check its log), and `git status`

clean afterward except for `bench/`

.

## Troubleshooting

— you're running Claude Code as root, typically in a Docker CI image. Create a non-root user, or swap the flag for`--dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons`

`--permission-mode acceptEdits --allowedTools "Bash"`

.right after installing — both installers drop binaries into`aider: command not found`

or`agent: command not found`

`~/.local/bin`

, which isn't on PATH in fresh shells.`export PATH="$HOME/.local/bin:$PATH"`

or restart the terminal. (Older Cursor CLI installs named the binary`cursor-agent`

; re-run the installer to get`agent`

.)— your`bench/tasks/...: No such file or directory`

on the second task`bench/`

directory got deleted because`git clean -fd`

ran without`-e bench`

. It removes*all*untracked files; restore the folder and keep the exclude.**Cursor rows always show**— print mode won't modify files without`passed=0`

with zero files changed`--force`

, and an unauthenticated CLI fails silently into the log. Check`agent status`

, and in CI make sure`CURSOR_API_KEY`

is exported.

## Next steps

Agents are nondeterministic, so run each task 3–5 times and report pass rate, not a single coin flip — wrap the inner loop in `for trial in 1 2 3`

. Add a cost column: `claude -p --output-format json`

returns structured results including total cost, and aider prints session cost at the end of each run. Extending the field is one `case`

arm per newcomer — OpenAI's Codex CLI and Google's Gemini CLI slot right in. And when you want to see how your private numbers compare to public ones, [SWE-bench](https://www.swebench.com) is the same idea — real issues, test-based grading — at research scale.

## Sources & further reading

-
[Claude Code CLI reference](https://code.claude.com/docs/en/cli-reference)— code.claude.com -
[Claude Code quickstart](https://code.claude.com/docs/en/quickstart)— code.claude.com -
[Scripting aider](https://aider.chat/docs/scripting.html)— aider.chat -
[Aider options reference](https://aider.chat/docs/config/options.html)— aider.chat -
[Cursor CLI parameters](https://cursor.com/docs/cli/reference/parameters)— cursor.com -
[Cursor CLI authentication](https://cursor.com/docs/cli/reference/authentication)— cursor.com

[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer

Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.

## Discussion 0

No comments yet

Be the first to weigh in.