# How Do You Prove a Coding Skill Works? A Ponytail Case Study

> Source: <https://julin.ai/2026/09/14/ponytail-benchmark-case-study/>
> Published: 2026-09-13 12:00:00+00:00

# How Do You Prove a Coding Skill Works? A Ponytail Case Study

OpenAI wrote a post called [Testing Agent Skills Systematically with Evals](https://developers.openai.com/blog/eval-skills). The point is simple: if you only judge a skill by feel, you won’t notice when it gets worse. You need a test set, a pass/fail check, and a score you can repeat.

[Ponytail](https://github.com/DietrichGebert/ponytail) is a good example to study. It’s a skill that makes an AI agent write less code. Its claim is easy to test, so its [benchmarks folder](https://github.com/DietrichGebert/ponytail/tree/main/benchmarks) is a good case study of the same idea, but for a coding skill instead of an agent skill.

## What ponytail claims

Ponytail makes the agent ask a few questions before it writes any code:

1. Does this need to exist at all?
2. Is it already in the codebase?
3. Does the standard library already do this?
4. Is there a one-line fix?

Only after those, it writes new code. The claim is not “shorter code is always better.” The claim is “shorter code, but still correct and safe, is better.” That second part is the hard part to test. A skill that just says “write less code” would also make the code shorter. It would just make it wrong too.

## Three groups, same test

The main benchmark runs three groups against the same models (Haiku, Sonnet, Opus) and the same five small tasks: an email checker, a debounce function, a CSV sum, a countdown timer, and a rate limiter.

- **baseline** : no skill at all
- **caveman** : a different skill, used to check that ponytail isn’t just “any skill helps”
- **ponytail** : the skill being tested

Everything else stays the same: same model, same task, same settings. Each combination runs ten times, and the benchmark reports the middle value. This is the part worth copying for your own tests: compare against a plain baseline and a competing option, not just your own past result.

Here’s one real result, on Haiku, for the email-checker task. The baseline wrote 518 lines of code, cost $0.030, and took 37.7 seconds. Ponytail wrote 39 lines, cost $0.011, and took 9.9 seconds. Same task, same model, very different code.

## Check correctness before you check size

Before the benchmark counts lines, `correctness.js` runs the code and checks if it works. For the email task, it runs a Python check against some valid and invalid email strings. For the CSV task, it checks that the sum equals 351. Only code that passes is allowed to be scored on size.

This order matters. If you score size first, you end up rewarding code that is short and broken. The scoring step looks roughly like this:

``` js
function scoreOutput(modelReply, task) {
  const code = extractCodeBlock(modelReply);
  const result = runTaskTest(code, task); // e.g. valid/invalid email checks
  if (!result.pass) return { pass: false, loc: null };
  const loc = countLines(stripCommentsAndBlankLines(code));
  return { pass: true, loc };
}
```

Correctness runs first and can fail the whole thing. Line count runs second and only ever measures. `loc.js` does the stripping and counting; `correctness.js` does the pass/fail gate in front of it.

## The single-shot problem

The headline numbers are big: 80-94% less code, 42-75% lower cost, 3-6x faster. But the authors admit a problem. A single chat reply mixes code with prose. A model that explains its answer in words looks worse on this test than one that just returns code, even if the code itself does the same job.

This is the same trap OpenAI’s post warns about: a small set of tests can make any technique look good if you don’t also test the cases where it might fail. So ponytail’s authors built a second, harder test.

## A more realistic test

This second benchmark, in the `agentic/` folder, runs real Claude Code sessions in a temp project folder, not single chat replies. It checks the actual file changes with `git diff`, against a real FastAPI web app template. It has two parts:

- **12 feature tasks** : things like a date picker or a command palette. This is the kind of task where an agent tends to add a library and a wrapper component nobody asked for.
- **7 safety tasks** : small functions that must survive attacks, like path traversal or SQL injection.

On top of the file diff and the attack tests, a separate model acts as a judge. It checks two things: did the agent over-build this, and did the agent actually finish the feature. Both checks matter together. A judge that only checks “did it over-build” would give a perfect score to an empty file.

On this harder test, the numbers are smaller but easier to trust: 60-94% less code on tasks where over-building is common, about the same on tasks that were already small, and 100% safe on the attack tests versus 95% for the comparison.

## Let other people check it

The benchmarks folder links to two outside groups who ran their own copy of the test. They found the same drop in code size, but pushed back a bit on the harder, agent-style tests. That’s the last idea worth copying: publish your test, not just your score, so someone else can run it and tell you where it breaks.

## What to copy for your own eval

- Test against a plain baseline and a competing option, not just your own past run.
- Check correctness first. A smaller wrong answer is not a win.
- Don’t trust a single chat-reply test if your real use case is a multi-step agent. Build the harder test before you publish the easy one.
- Use one judge to check “did it over-build” and a second judge to check “did it finish.” One without the other can be tricked.
- Publish your test. A number nobody can rerun is a claim, not proof.