I didn’t set out to write a benchmark paper.
I wanted to answer a much dumber, much more practical question:
“For the stuff I actually do at work, when do I really need a frontier model… and when am I just burning money?”
So I did what engineers do when they want to be serious about this:
And then I ran exactly one of them.
It turns out, that one was enough.
At work, someone else pays for LLM tokens.
At home, it’s different:
I still want to understand how these models behave on real workloads, not toy prompts. But in my own projects, I have to be surgical about where the money goes.
That tension is what created the frontier experiments program.
On paper, the program was beautiful.
Each experiment was a slice of a bigger question:
“For a fleet of agents doing devopsy things, which model should handle which task, at what effort level, for how much money?”
I designed 10 experiments around that:
On top of that, I built a reusable evaluation harness, model-compass:
It was a solid plan.
And then I ran one experiment and stopped.
I could have started with something “sexier” like architecture reasoning or multi-agent coordination.
Instead I picked CI diagnostics.
Why?
Because that’s closer to what I actually do all day.
My team ships TypeScript, Swift, Kotlin, C++, Java, Go, and BrightScript across multiple repos. Our CI pipelines glue together a lot of first- and third-party tools. When something goes red, the logs are measured in megabytes, not kilobytes. Most failures are in heavy integration tests, and on some days specific areas see 5–20% failure rates just from tool and test flakiness.
As a leader, I wasn’t the one reading every failure. My team was. I invested personally to understand their pain and started building Claude skills that could actually help. The ugly reality was that “CI red → I know the root cause and next step” used to be an hour or more of down logs and hunting for patterns. End-to-end, getting a fix to production was often a full day.
With the right automation and AI in the loop, we’ve pushed that down to ~15 minutes to understand what’s going on, and under 2 hours to have a fix merged and back in production in many cases.
Most runs in this experiment started the same way: me squinting at a 5,000‑line GitHub Actions log thinking “this is exactly why I want a model to do this instead.”
If I could learn: “For CI diagnostics, which model should I actually pay for?”
that would immediately translate into better decisions at work.
So experiment 01 (a precursor to the bigger program) was simple on paper:
I ran it through the model-compass harness with hard budget caps.
Total cost: $13.53.
$13 looks small, but each experiment has several sub-flavors and with proper sample sizes the full program would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing. That meant cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.
Before I go further, it helps to be clear about scope.
The results:
Haiku:
Sonnet:
In terms of “should I pay for Sonnet for CI diagnostics?” the answer was:
“Not by default.”
If all you read is that, you might think: That’s not how it landed for me.
And this part is important:
You should not take this result as “Haiku is better than Sonnet for everything.”
For my specific slice of work: Haiku is the model that now makes the most sense to me:
At work, I still reach for Sonnet or Opus a lot — especially for harder reasoning, architecture decisions, or more ambiguous problems.
This experiment just moved one chunk of work (CI / DevOps diagnostics) into the “Haiku by default, Sonnet/Opus when needed” bucket in my head.
There is nuance here. In some runs, Sonnet’s answers were crisper and more coherent. Haiku tended to be more verbose. For CI diagnostics, that verbosity is a feature, not a bug. Haiku is summarizing maybe 1/1000 of the raw logs into something an engineer or a downstream agent can work with. I can live with a small quality gap when a human stays in the loop and gets more context to reason about.
I went in expecting frontier models to be expensive.
I wasn’t shocked that:
I also wasn’t shocked that:
Each experiment had several sub-flavors and with proper sample sizes would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing — cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.
This is exactly why I keep my personal experiments constrained and work experiments more open:
The actual lesson was something else:
I didn’t need to run all 10 experiments to learn what I came for.
Designing 10 experiments was still worth it.
That process forced me to think clearly about:
But execution-wise, I realized: That single experiment:
The other 9 experiment designs are still there if I ever need them.
But for now, the design work plus one focused execution was enough.
There’s another layer of this that I think matters more than the experiment result.
I asked an LLM to help me build the evaluation harness itself.
It… worked.
Kind of.
trial/
folder with names like compare_models_v3_final_real_final.py
By the third time I fixed a path bug and re-ran the same batch of calls, I could feel the experiment burning money in the background.
The meta-lesson was brutal and simple: the human has to write the full spec. every field. every output. every edge case.
Once I wrote out:
the LLM was finally useful.
Before that, it was just guessing.
There was another uncomfortable realization buried in this run.
My original harness design was way more generic than the experiments actually needed.
On paper, model-compass looked like a clean, reusable engine:
In practice, the DevOps-style experiments (CI diagnostics, PR review, release-notes generation) all needed their own pipelines:
Large data inputs from public GitHub repos
Because this is a private project, I couldn’t point the harness at internal repos. I had to:
Experiment-specific "automation profiles"
Each experiment needed its own notion of a "profile":
Multi-stage scripts per experiment
The real flow looked more like:
None of that was encoded cleanly in the first version of the harness.
Once the dust settled, I realized we were probably using less than 25% of the original design. Most of the real work lived in experiment-specific scripts.
And that’s… fine.
Nothing is actually throwaway here:
It’s an unglamorous but useful pattern:
design the generic thing, discover it’s wrong, let the specific scripts show you what the generic layer should actually be.
The other big shift was resisting the “one big script” instinct. DeepSeek V4 Pro’s first attempt was a single script that would download logs, run both models, and spit out final comparisons in one shot. It looked clever, but it made iteration expensive. Breaking it into smaller scripts — “download once and cache immutable logs,” “run a single model over profiles,” “summarize saved outputs” — meant I could fix bugs in one stage without burning API calls in the others. When the summarizer broke, I didn’t have to re-run the models; I just reprocessed the saved outputs. When the down broke, I fixed it without touching any LLMs. That separation is what kept this experiment financially and technically sane.
Even with just two models, I needed the outputs to line up so comparison wasn’t vibes-based. For each CI run, I forced the model output into a consistent shape: failure domain (a short label like db-connection
or test-assertion
), a one-to-three sentence root cause summary, evidence pointers (log snippets or line ranges), a coarse confidence bucket (low/medium/high), three to five concrete remediation steps, and a simple verdict on whether this looked like a real issue or something flaky/infra-like.
On top of that, every run captured input/output tokens, cached vs non-cached tokens, total cost in USD, and latency. Scoring and comparison scripts then answered very boring but very important questions: did the failure domain match the labeled truth, was the root cause directionally correct, were the steps actionable and safe, was one model consistently overconfident and wrong, and how much extra did that cost.
From there, I rolled the data up into three layers of summaries: per-flavor (for example, "on DB-connection failures Haiku passed X/Y, Sonnet passed A/B, here’s the cost and latency delta"), per-task across all flavors (overall accuracy, cost to get one "good enough" answer for CI diagnostics per model), and a human-readable decision summary.
This is where TierForge fits. TierForge is an open-source router that turns evidence into YAML routing rules. It wants routing rules backed by real data: for a given use case like ci-diagnostics
, what is the primary model, what is the escalation model, and under what conditions do you switch. The experiment gives it exactly that: a concrete "Haiku-by-default, Sonnet-on-these-conditions" rule, plus actual cost-per-task numbers instead of gut feel.
Looking back, here’s what survived and what I keep using.
1. The harness matters more than the paper
I don’t need to publish “definitive bench results” to get value.
What I needed was:
That’s what model-compass turned into.
2. One well-chosen experiment beats ten generic ones
CI diagnostics:
It’s a better teacher than a grab bag of random tasks I’ll never run in production.
3. My personal budget shapes my learning strategy
Because I can’t justify hundreds of dollars on personal experiments:
At work, I might run more of the matrix. Privately, I only need enough to calibrate my instincts.
The workload-aware profiles I’m building help a lot here. I am not making model choices in a vacuum; I am making them for specific, well-understood slices of work. That lets me stay inside a tight personal budget without taking a meaningful hit on quality.
4. “Cheap vs expensive model” isn’t just a cost problem
The CI experiment answered a cost question.
The hard part is a behavior question:
will I actually reach for Haiku by default for CI work, or will I still click the fancy model because it feels safer?
I’m still catching myself defaulting to the “nice” model sometimes, even with data in hand.
I don’t have clean answers here. Just a few open questions I’m carrying around:
If you’ve navigated this tension — work budget vs personal budget, curiosity vs cost — I’d love to hear how you handle it. Yes, the bill matters. But in the teams I talk to, “this is too expensive” often hides other worries: we’re not sure we trust the outputs yet, we don’t feel fully in control of the workflows, or we’re afraid of looking reckless to leadership. I’m trying to go the other way: tighten costs by getting more workload-aware and evidence-driven, instead of tightening costs because we’re scared.
When you think about containing AI costs in your own team, what is really driving it? Is it just the cloud bill, or something deeper — trust, optics, control, or the feeling that you should understand what’s going on before you let a model touch everything?