Ponytail vs a good AGENTS.md: What we found on our real codebase

DocSpring evaluated Ponytail 5.3.3, a coding-agent plugin that pushes agents to write less code, against its own agent setup of 42 AGENTS.md files totaling about 65,000 words on a Rails and React monorepo, using Claude Code 2.1 and Opus 5.5. Claude Code analyzed 10,939 messages the author had typed over five months and ran 16 benchmark sessions across four real tasks with a blind judge, finding roughly twice as many complaints that agents built too much as built too little, and that about a third of the code on one audited batch branch came from review loops. Ponytail's own benchmark of 39 tasks run five times each reported 53% fewer lines written, 26% lower cost, 41% faster completion and a 96.7% pass rate on hidden checks versus 95.6% without it, though DocSpring cautions its four-task, two-run test is only a rough direction; the rules it adopted have been in its AGENTS.md since 13 November.

Someone mentioned Ponytail https://github.com/DietrichGebert/ponytail?ref=docspring.com to me the other day. It's a popular plugin for Claude Code, Codex and other coding agents that makes them write less code: it plays the "lazy senior developer" who asks whether a change needs to exist at all before writing the smallest version that works. Its own benchmark shows agents writing about half as many lines, a quarter cheaper, with slightly better results on hidden tests. I wondered if I should be using it, so I asked Claude Code to find out. Claude went through the 10,939 messages I'd typed into Claude Code sessions in our main repository over the previous five months, and sorted every complaint I'd made into "built too much" or "built too little". It read our AGENTS.md files, skills and review records, re-reviewed some of our shipped commits with Ponytail's checklist, and ran 16 benchmark sessions: four real tasks from our history, with and without Ponytail, then a blind judge to score the results. Then it wrote me a report with a decisive recommendation. We evaluated Ponytail 5.3.3 with Claude Code 2.1 and Opus 5.5 on our Rails and React monorepo. The benchmark is small four tasks, two runs each , so you should treat its numbers as a rough direction. The rules we adopted have been in our AGENTS.md since 13 November. What Ponytail does Ponytail is a set of rules about 470 words , a few skills review, audit, a ledger of deliberate shortcuts , and hooks that switch it on for every session and every subagent. The rules boil down to a ladder: does this need to exist? Can existing code do it? The standard library? Then write the least code that works, and leave a shortcut: comment naming any limit you accepted and when to revisit it. Its benchmark ran 39 tasks five times each, with Ponytail and without. Ponytail's runs wrote 53% fewer lines, cost 26% less and finished 41% faster, and passed 96.7% of hidden checks against 95.6%. The biggest wins were on front-end and small app tasks. That benchmark turned off CLAUDE.md files and auto memory. That's fair for measuring the plugin on its own, but my agents never run like that. They start every session with our root AGENTS.md, nested AGENTS.md files for each part of the codebase, memories and a set of workflow skills. So the question for us was whether Ponytail adds anything on top of all that. What we already had DocSpring fills PDFs, generates documents and collects e-signatures. A wrong value in a signed contract is much worse than an error message, so our agent setup leans towards being extremely careful. We have 42 AGENTS.md files, about 65,000 words in all: a root file with the rules every change follows, and a nested file next to each part of the code that has its own pitfalls migrations, background jobs, the public API, CI . On top of that we have skills for the workflow around the code: investigating a ticket before writing anything, collaborative PR review, a second-opinion review from another model, fixing CI, a browser QA pass, and an end-of-session sweep for loose ends. What five months of my messages said Claude Code found about twice as many complaints that an agent had built too much as complaints that it had built too little. Most of the "too much" complaints were about code: options nobody asked for, compatibility shims, backfill tasks, a helper that duplicated one we already had. Ponytail targets exactly that. The "too little" complaints were fewer, but nearly all of them had the same shape: an agent proposed a follow-up ticket for something it should have fixed in the same branch. More pressure to do less would make those worse. Review loops were another big source of extra code. When we audited one of our batch branches the day before, about a third of its code had come from review loops: an AI review found an edge case, the fix added code, the next review found an edge case in the fix, and so on. Many of those cases were inputs no real customer or PDF would ever produce. Our PR review records told a similar story from the other side. About 9 in 10 findings that a person decided on were accepted and fixed. Fewer than 1 in 100 findings were about code that shouldn't exist. Our reviews were good at finding missing things and almost never asked whether something should be there at all. When Claude Code re-reviewed four of our recently shipped commits with Ponytail's checklist, three were already lean. The fourth had hand-built a "run this after the transaction commits" mechanism that Active Record already provides as ActiveRecord.after all transactions commit , and that our billing code uses about ten times. What the benchmark showed Claude Code took four tasks from our git history a dashboard rollup, a sign-up change, an avatar fix, and a welcome modal , checked out the commit before each one, and gave the original task to fresh Claude Code sessions with our AGENTS.md loaded. Half the sessions also had Ponytail's rules. Each combination ran twice. Ponytail's sessions wrote much less: | Ponytail vs our setup alone | Change | |---|---| | Source lines | −56% | | Test lines | −59% | | Cost | −34% | | Time | −38% | | Output tokens | −41% | Then a separate session judged the diffs without knowing which was which. It preferred our setup alone: 6.8 out of 10 overall against 5.9, and 2.4 out of 3 for completeness against 2.1. It also found more code it could remove from our sessions' diffs 253 lines against 94 , so Ponytail's diffs really were leaner. The completeness gap came from real misses. In the sign-up task, both Ponytail sessions changed the sign-up path that uses Google and GitHub but missed the one where people type their name. In the avatar task, both Ponytail sessions left a test that would fail, and their final replies made a claim about the code that wasn't true. And on the rollup task, every session, with and without Ponytail, built a task to repair existing data that nobody had asked for. So on our codebase, Ponytail did what it promises less code, lower cost and cost us some of what matters most to us reaching every place a change has to go . What we adopted We didn't install Ponytail. Instead, we took three ideas from it and wrote them into our root AGENTS.md in our own words: 1. The smallest complete change The scope is complete: every place the change must reach callers, sibling entry points, the other client that applies the same rule, tests, docs and the problems found in the same code, all in the same PR, with no follow-up tickets. The solution is small: use what the codebase already has, and add no option, setting, compatibility path, backfill or repair task that the ticket didn't ask for. When stored data needs repairing, the agent asks first. That one rule answers both kinds of complaint. "Complete" fixes the follow-up tickets. "Small" fixes the shims and the repair tasks. 2. A review finding needs a trigger that can actually happen A finding has to name an input that a customer, a real PDF, an integration or a concurrent job could produce. My first draft of this rule said that cases nobody had ever hit shouldn't be fixed, and that's too blunt for us. New code has no history, so nobody knows yet which edges matter. So the rule is thorough with code the change adds and with anything that could put a wrong value in a document or lose data, and it lists hand-crafted inputs that no real file or system produces as "not fixing", with the reason. When a fix would be bigger than the code it reviews, or a finding is about code an earlier review round added, it goes to me as a decision, which is what stops the review loops. 3. A lean pass in every review Our PR review and second-opinion review now always include a focus area for code that shouldn't be there: dead code, unused options, an abstraction with one implementation, something the codebase already has, or code no line of the ticket asks for. We also wrote down one exception to our "never break a customer's integration" rule. Fixing output that's visibly wrong a value in the wrong cell doesn't need a production count or a per-account gate. It ships to everyone, and we tell the customers we know were affected. Why we didn't install it Ponytail is good, and if your agents run with little project context, I'd try it out. For us it overlapped with rules we already had, and its hooks switch it on for every session and subagent, which makes it harder to reason about what an agent was told. Its codebase map is capped at 2,000 characters, and on our monorepo over 12,000 tracked files it filled up with generated API client templates and skipped 316 folders. And the benchmark suggests that the pressure to write less code doesn't suit a codebase like ours as well. We've shared our setup Writing this up made me realize that most of what helped wasn't specific to DocSpring. So we've packaged up some of our rules and skills and have published them on GitHub: DocSpring/docspring-agent-playbook https://github.com/DocSpring/docspring-agent-playbook?ref=docspring.com MIT licensed . - Principles: the engineering rules our AGENTS.md files are built on, each with its reason and how to enforce it. The smallest complete change and the review-finding rule are there in full, along with rules for errors, API compatibility, data in AI sessions, tests, migrations and untrusted input. - AGENTS.md templates: a short root file to copy and fill in, and example nested files for databases, tests, public APIs, background jobs, UI, config, CI and docs. - Guides for writing AGENTS.md files and skills that agents actually follow. - Skills: investigate a ticket before coding, review a PR with you, get a second-opinion review from another model, fix CI until it's green, QA the changed screens in a browser, and sweep a session for loose ends. They read your tracker, CI and test commands from one section of your AGENTS.md. The scripts that talk to other tools are written for GitHub, GitHub Actions and the Codex CLI; for anything else you write a small adapter. The principles are opinionated. Each one says what we do and why, and the ones that are really policy choices no feature flags, never dropping a column say when they don't fit. Rules that only matter for some products, such as public APIs, generated SDKs and self-hosted installs, sit in their own sections, so you can skip the ones that don't apply to you. You can install the skills with npx skills add DocSpring/docspring-agent-playbook , or as a Claude Code plugin. The templates and principles are meant to be read and cut down, so copy what applies to your codebase and delete the rest. If you're wondering whether a tool like Ponytail is right for your team, you don't have to guess anymore. Try asking your AI agent to find out by reading through your own code and your own history.