{"slug": "garuda-against-claude-code-with-intervals", "title": "Garuda against Claude Code, with intervals", "summary": "An independent developer released Garuda 0.18, an open-source terminal coding agent, with a rebuilt eval system that reports paired confidence intervals for pass-rate and cost comparisons. Benchmarked against Claude Code 2.1.312 on 30 held-out tasks using Claude Haiku 5.5, Garuda passed 30 of 30 versus 29 of 30 (no clear difference) while costing $0.0034 versus $0.0059 per accepted change, a statistically clear ×1.75 difference. The developer also found that a code-graph MCP server raised cost ×1.28 without improving pass rates, and that an Opus-proposed prompt addendum cut cost to ×0.81 on held-out tasks.", "body_md": "[Garuda](https://github.com/madhu-sv/garuda) is an open-source coding agent for the terminal that I build to learn how coding agents work. Every command runs in an OS sandbox (Seatbelt on macOS, bubblewrap on Linux), edits ask first, and every decision goes to an audit log. It works with Claude, OpenAI-compatible APIs and local models through Ollama, and in editors over ACP.\n\nGaruda had an eval command from its first release, but it gave one run per task and one number. That is enough to catch a regression. It is not enough to say \"Garuda is cheaper than X\" or \"this change helps\". For 0.18 I rebuilt the evals so that a claim comes with an interval, and then I tested Garuda against Claude Code, a code-graph MCP server and a tuned prompt.\n\n`garuda eval compare` takes two reports of the same tasks (it checks a hash of each task) and gives B − A for the pass rate and B / A for the cost, each with a paired interval. A difference whose interval contains 0 (or ×1.0) is \"no clear difference\", never a win.\n`--agent claude-code` runs Claude Code (`claude -p`) on the same tasks, in the same scratch folders, with the same time limit and grading. It runs with every tool allowed, so only inside Garuda's OS sandbox, with a new config folder per run and network only to `api.anthropic.com`. Both agents used Claude Haiku 5.5, 3 runs per held-out task, seed 7.\n\n|  | Garuda | Claude Code 2.1.312 | \n|---|---|---|\n| Passed | 30 of 30 | 29 of 30 | \n| Cost per accepted change | $0.0034 | $0.0059 | \n| Median time | 19 s | 25 s | \n| Unexpected changes | 0 | 0 | \n\nThe pass rates show no clear difference: −3 points, interval −10 to 0. Claude Code's one failure was the refund trap. The cost shows a clear one: Claude Code cost **×1.75** per accepted change (95% interval ×1.54 to ×2.06). Its median run used about 144k tokens, against 46k for Garuda.\n\nThe first live runs failed for reasons that were mine, not the agent's. Each one showed up when I read a failed run in full:\n\n`dueDate`, but the prompt never named it, and the agent chose `dueAt`, which matched the repository's `created_at`. A hidden test may now check only what the prompt or the repository decides.\nAn eval is code. It needs the same care as the code it measures.\n\nI tested [Graphify](https://pypi.org/project/graphifyy/), a code knowledge graph over MCP, with `--mcp graphify --mcp-prepare \"graphify update .\"`. With a note in `GARUDA.md`, the agent used it in 14 of 18 runs on the hard suite (6 tasks, about 110 files). Every run passed with and without it, and it cost **×1.28** per accepted change (×1.19 to ×1.38). The agent already finds code with grep, and the graph's tools and results add tokens to every request. A graph may pay off in a much larger repository. Garuda does not include it; the docs show how to add it and how to measure it.\n\n`garuda eval tune` lets a frontier model (Opus 5.5) propose a short addendum to the system prompt. A version stays only if no task goes from all runs to none and it passes more runs, or as many runs at a clearly lower cost. The held-out set is measured once, at the end, and the proposer never sees it. The first run cost $0.65. The kept version says, in short: batch tool calls that do not depend on each other, read only what the edit needs, verify once, stop when done.\n\n| Held-out set, all runs passing | Cost per accepted change | \n|---|---|\n| Tuned text, Haiku 5.5 (5 runs a task) | ×0.81 (×0.75 to ×0.88) | \n| Tuned text, Sonnet 5.5 (3 runs a task) | ×0.87 (×0.79 to ×0.95) | \n| Shipped rewrite, Sonnet 5.5 | ×0.90 (×0.84 to ×0.96) | \n| Shipped rewrite, Haiku 5.5 (one refund miss) | ×0.88 (×0.79 to ×1.02) | \n\nI removed two rules that served the evals, not users (\"do not call `remember`\", \"a one-sentence answer\") and rewrote the rest in Garuda's style. The rewrite is in 0.18's system prompt. With Haiku its saving is likely but not shown with confidence.\n\n```\nnpm install -g @garuda-agent/garuda\ngaruda eval --split heldout --repeat 3 --shuffle --seed 7 -m claude-haiku-5-5\ngaruda eval --split heldout --repeat 3 --shuffle --seed 7 -m claude-haiku-5-5 --agent claude-code\ngaruda eval compare .garuda/evals/<garuda-run> .garuda/evals/<claude-code-run>\n```\n\nThe method is in the [evals guide](https://madhu-sv.github.io/garuda/docs/guide/evals/). If your numbers differ from mine, I want to know.\n\nGaruda is Apache-2.0, on npm (`npm i -g @garuda-agent/garuda`) and Homebrew. The code and the design documents: [https://github.com/madhu-sv/garuda](https://github.com/madhu-sv/garuda). Feedback on the sandbox model and on the evals is very welcome.", "url": "https://wpnews.pro/news/garuda-against-claude-code-with-intervals", "canonical_source": "https://dev.to/madhusudhan_suv/garuda-against-claude-code-with-intervals-1808", "published_at": "2026-11-04 21:16:43+00:00", "updated_at": "2026-11-04 21:22:53.904949+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-research", "mlops"], "entities": ["Garuda", "Claude Code", "Anthropic", "Graphify", "Ollama", "OpenAI", "Claude Haiku 5.5", "Opus 5.5"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/garuda-against-claude-code-with-intervals", "markdown": "https://wpnews.pro/news/garuda-against-claude-code-with-intervals.md", "text": "https://wpnews.pro/news/garuda-against-claude-code-with-intervals.txt", "jsonld": "https://wpnews.pro/news/garuda-against-claude-code-with-intervals.jsonld"}}