Build a headless harness that scores AI coding agents on real issues from your codebase.
What you'll build #
A bash harness that runs Claude Code, Aider, and the Cursor CLI headless against real tasks from your own repo, grades each attempt with a test script, and writes a CSV scoreboard — pass/fail, wall-clock time, and diff size — so your agent choice comes from data, not vibes.
Prerequisites #
- macOS or Linux (WSL works). bash and git.
- Verified against: Claude Code 2.1 (native installer), aider 0.86.1, and the current Cursor CLI (
agent
), all per official docs as of August 2026. - Python 3.8–3.13 (aider's supported range).
- Accounts: a Claude subscription or an
ANTHROPIC_API_KEY
; a Cursor account (API key fromcursor.com/dashboard/apiif you'll run in CI). Aider talks to model APIs directly — we'll point it at the same Anthropic key. - A repo with a runnable test suite, in a throwaway clone. The harness hard-resets the working tree between runs. Don't point it at your daily checkout.
1. Install the three agents #
curl -fsSL https://claude.ai/install.sh | bash
claude # first run opens browser login; then /exit
python -m pip install aider-install
aider-install
curl https://cursor.com/install -fsS | bash
agent login # browser auth; in CI use: export CURSOR_API_KEY=...
export ANTHROPIC_API_KEY=sk-ant-...
Confirm all three respond: claude --version
, aider --version
, agent --version
.
2. Turn real issues into benchmark tasks #
Inside your throwaway clone, each task is a folder with a prompt and a grader:
mkdir -p bench/tasks bench/logs
mkdir bench/tasks/issue-142
cat > bench/tasks/issue-142/prompt.md <<'EOF'
Fix the bug where parse_duration("90m") returns 90 seconds instead of
5400. The function lives in src/utils/time.py. Do not add dependencies.
EOF
cat > bench/tasks/issue-142/check.sh <<'EOF'
#!/usr/bin/env bash
python -m pytest tests/test_time.py -q
EOF
The gold standard for task selection: closed issues you've already fixed. On a bench
branch, git revert
your fix but keep (or cherry-pick) its regression test as check.sh
— you get a known-solvable task with an objective grader. Faster alternative: open bugs or small features where you can write the failing test right now. Paste the issue text into prompt.md
verbatim; don't add hints you wouldn't give a new teammate. Five to ten tasks is enough to separate the pack.
3. Write the harness #
Save as bench/bench.sh
:
#!/usr/bin/env bash
set -u
cd "$(git rev-parse --show-toplevel)"
BASE=$(git rev-parse HEAD)
echo "agent,task,passed,seconds,files_changed,insertions,deletions" > bench/results.csv
run_one() {
case "$1" in
claude) claude -p "$2" --model sonnet \
--dangerously-skip-permissions --max-turns 30 ;;
aider) aider --message "$2" --model sonnet --yes-always \
--no-auto-commits --no-gitignore --no-stream --no-pretty ;;
cursor) agent -p "$2" --force --output-format text ;;
esac
}
for task_dir in bench/tasks/*/; do
task=$(basename "$task_dir")
prompt=$(<"$task_dir/prompt.md")
for name in claude aider cursor; do
git reset --hard -q "$BASE" && git clean -fdq -e bench
echo "=== $name / $task ==="
start=$SECONDS
run_one "$name" "$prompt" > "bench/logs/$name-$task.log" 2>&1
elapsed=$(( SECONDS - start ))
if bash "$task_dir/check.sh" > "bench/logs/$name-$task.check.log" 2>&1
then passed=1; else passed=0
fi
stat=$(git diff --shortstat)
files=$(grep -o '[0-9]* file' <<<"$stat" | grep -o '[0-9]*')
ins=$(grep -o '[0-9]* insertion' <<<"$stat" | grep -o '[0-9]*')
del=$(grep -o '[0-9]* deletion' <<<"$stat" | grep -o '[0-9]*')
echo "$name,$task,$passed,$elapsed,${files:-0},${ins:-0},${del:-0}" \
>> bench/results.csv
done
done
git reset --hard -q "$BASE" && git clean -fdq -e bench
column -t -s, bench/results.csv
Three design decisions worth knowing:
Every run starts from the same commit.git reset --hard
plusgit clean -fd
wipes each agent's changes before the next run;-e bench
excludes the harness itself from the wipe.Each agent gets its own "just do it" flag, because headless runs can't answer approval prompts:--dangerously-skip-permissions
(Claude Code),--yes-always
(aider),--force
(Cursor). This is exactly why you're in a disposable clone. Aider also gets--no-auto-commits
so its edits stay uncommitted and diffable like the others', and--no-gitignore
so it doesn't edit.gitignore
and pollute the diff stats.Models are pinned. All three run Anthropic's Sonnet here (agent --list-models
shows Cursor's exact names — add--model
to its arm), so you're benchmarking the agent scaffolding, not different models. Drop the--model
flags instead if you want to compare each product as shipped, defaults and all.
Diff stats count tracked files only — new files an agent creates won't show — so treat passed
as the score and the diff columns as a code-churn tiebreaker.
4. Run the benchmark #
chmod +x bench/bench.sh bench/tasks/*/check.sh
./bench/bench.sh
Budget one to five minutes per agent-task pair; a 3×8 matrix is a coffee break. Follow along in another terminal with tail -f bench/logs/*.log
.
Verify it works #
Before the full matrix, smoke-test with a single trivial task (e.g., "make tests/test_smoke.py
pass" with a one-line fix). A healthy run ends with a table like:
agent task passed seconds files_changed insertions deletions
claude issue-142 1 147 2 38 6
aider issue-142 1 63 1 12 4
cursor issue-142 0 201 3 120 41
Every row present, no seconds
under ~10 (that usually means the agent errored out instantly — check its log), and git status
clean afterward except for bench/
.
Troubleshooting #
— you're running Claude Code as root, typically in a Docker CI image. Create a non-root user, or swap the flag for--dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
--permission-mode acceptEdits --allowedTools "Bash"
.right after installing — both installers drop binaries intoaider: command not found
oragent: command not found
~/.local/bin
, which isn't on PATH in fresh shells.export PATH="$HOME/.local/bin:$PATH"
or restart the terminal. (Older Cursor CLI installs named the binarycursor-agent
; re-run the installer to getagent
.)— yourbench/tasks/...: No such file or directory
on the second taskbench/
directory got deleted becausegit clean -fd
ran without-e bench
. It removesalluntracked files; restore the folder and keep the exclude.Cursor rows always show— print mode won't modify files withoutpassed=0
with zero files changed--force
, and an unauthenticated CLI fails silently into the log. Checkagent status
, and in CI make sureCURSOR_API_KEY
is exported.
Next steps #
Agents are nondeterministic, so run each task 3–5 times and report pass rate, not a single coin flip — wrap the inner loop in for trial in 1 2 3
. Add a cost column: claude -p --output-format json
returns structured results including total cost, and aider prints session cost at the end of each run. Extending the field is one case
arm per newcomer — OpenAI's Codex CLI and Google's Gemini CLI slot right in. And when you want to see how your private numbers compare to public ones, SWE-bench is the same idea — real issues, test-based grading — at research scale.
Sources & further reading #
Claude Code CLI reference— code.claude.com - Claude Code quickstart— code.claude.com - Scripting aider— aider.chat - Aider options reference— aider.chat - Cursor CLI parameters— cursor.com - Cursor CLI authentication— cursor.com
Priya Nair· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.