{"slug": "i-planned-10-llm-evaluation-experiments-and-only-ran-1-it-was-enough", "title": "I Planned 10 LLM Evaluation Experiments And Only Ran 1. It Was Enough.", "summary": "A developer planned 10 LLM evaluation experiments but ran only one—CI diagnostics—and found it sufficient. The experiment, costing $13.53, showed that Haiku outperformed Sonnet for this specific task, leading the developer to default to Haiku for CI diagnostics while reserving Sonnet/Opus for harder reasoning.", "body_md": "I didn’t set out to write a benchmark paper.\n\nI wanted to answer a much dumber, much more practical question:\n\n“For the stuff I actually do at work, when do I really need a frontier model… and when am I just burning money?”\n\nSo I did what engineers do when they want to be serious about this:\n\nAnd then I ran **exactly one** of them.\n\nIt turns out, that one was enough.\n\nAt work, someone else pays for LLM tokens.\n\nAt home, it’s different:\n\nI still want to understand how these models behave on **real** workloads, not toy prompts. But in my own projects, I have to be surgical about where the money goes.\n\nThat tension is what created the frontier experiments program.\n\nOn paper, the program was beautiful.\n\nEach experiment was a slice of a bigger question:\n\n“For a fleet of agents doing devopsy things, which model should handle which task, at what effort level, for how much money?”\n\nI designed 10 experiments around that:\n\nOn top of that, I built a reusable evaluation harness, **model-compass**:\n\nIt was a solid plan.\n\nAnd then I ran one experiment and stopped.\n\nI could have started with something “sexier” like architecture reasoning or multi-agent coordination.\n\nInstead I picked **CI diagnostics**.\n\nWhy?\n\nBecause that’s closer to what I actually do all day.\n\nMy team ships TypeScript, Swift, Kotlin, C++, Java, Go, and BrightScript across multiple repos. Our CI pipelines glue together a lot of first- and third-party tools. When something goes red, the logs are measured in megabytes, not kilobytes. Most failures are in heavy integration tests, and on some days specific areas see 5–20% failure rates just from tool and test flakiness.\n\nAs a leader, I wasn’t the one reading every failure. My team was. I invested personally to understand their pain and started building Claude skills that could actually help. The ugly reality was that “CI red → I know the root cause and next step” used to be an hour or more of downloading logs and hunting for patterns. End-to-end, getting a fix to production was often a full day.\n\nWith the right automation and AI in the loop, we’ve pushed that down to ~15 minutes to understand what’s going on, and under 2 hours to have a fix merged and back in production in many cases.\n\nMost runs in this experiment started the same way: me squinting at a 5,000‑line GitHub Actions log thinking “this is exactly why I want a model to do this instead.”\n\nIf I could learn:\n\n“For CI diagnostics, which model should I actually pay for?”\n\nthat would immediately translate into better decisions at work.\n\nSo experiment 01 (a precursor to the bigger program) was simple on paper:\n\nI ran it through the model-compass harness with hard budget caps.\n\n**Total cost: $13.53.**\n\n$13 looks small, but each experiment has several sub-flavors and with proper sample sizes the full program would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing. That meant cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.\n\nBefore I go further, it helps to be clear about scope.\n\nThe results:\n\nHaiku:\n\nSonnet:\n\nIn terms of “should I pay for Sonnet for CI diagnostics?” the answer was:\n\n“Not by default.”\n\nIf all you read is that, you might think:\n\nThat’s not how it landed for me.\n\nAnd this part is important:\n\nYou should **not** take this result as “Haiku is better than Sonnet for everything.”\n\nFor my specific slice of work:\n\nHaiku is the model that now makes the most sense to me:\n\nAt work, I still reach for Sonnet or Opus a lot — especially for harder reasoning, architecture decisions, or more ambiguous problems.\n\nThis experiment just moved one chunk of work (CI / DevOps diagnostics) into the “Haiku by default, Sonnet/Opus when needed” bucket in my head.\n\nThere is nuance here. In some runs, Sonnet’s answers were crisper and more coherent. Haiku tended to be more verbose. For CI diagnostics, that verbosity is a feature, not a bug. Haiku is summarizing maybe 1/1000 of the raw logs into something an engineer or a downstream agent can work with. I can live with a small quality gap when a human stays in the loop and gets more context to reason about.\n\nI went in **expecting** frontier models to be expensive.\n\nI wasn’t shocked that:\n\nI also wasn’t shocked that:\n\nEach experiment had several sub-flavors and with proper sample sizes would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing — cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.\n\nThis is exactly why I keep my **personal** experiments constrained and **work** experiments more open:\n\nThe actual lesson was something else:\n\nI didn’t need to run all 10 experiments to learn what I came for.\n\nDesigning 10 experiments was still worth it.\n\nThat process forced me to think clearly about:\n\nBut execution-wise, I realized:\n\nThat single experiment:\n\nThe other 9 experiment designs are still there if I ever need them.\n\nBut for now, the **design work** plus **one focused execution** was enough.\n\nThere’s another layer of this that I think matters more than the experiment result.\n\nI asked an LLM to help me build the evaluation harness itself.\n\nIt… worked.\n\nKind of.\n\n`trial/`\n\nfolder with names like `compare_models_v3_final_real_final.py`\n\nBy the third time I fixed a path bug and re-ran the same batch of calls, I could feel the experiment burning money in the background.\n\nThe meta-lesson was brutal and simple:\n\nthe human has to write the full spec. every field. every output. every edge case.\n\nOnce I wrote out:\n\nthe LLM was finally useful.\n\nBefore that, it was just guessing.\n\nThere was another uncomfortable realization buried in this run.\n\nMy original harness design was **way more generic** than the experiments actually needed.\n\nOn paper, model-compass looked like a clean, reusable engine:\n\nIn practice, the DevOps-style experiments (CI diagnostics, PR review, release-notes generation) all needed their own pipelines:\n\n**Large data inputs from public GitHub repos**\n\nBecause this is a private project, I couldn’t point the harness at internal repos. I had to:\n\n**Experiment-specific \"automation profiles\"**\n\nEach experiment needed its own notion of a \"profile\":\n\n**Multi-stage scripts per experiment**\n\nThe real flow looked more like:\n\nNone of that was encoded cleanly in the first version of the harness.\n\nOnce the dust settled, I realized we were probably using **less than 25%** of the original design. Most of the real work lived in experiment-specific scripts.\n\nAnd that’s… fine.\n\nNothing is actually throwaway here:\n\nIt’s an unglamorous but useful pattern:\n\ndesign the generic thing, discover it’s wrong, let the specific scripts show you what the generic layer should actually be.\n\nThe other big shift was resisting the “one big script” instinct. DeepSeek V4 Pro’s first attempt was a single script that would download logs, run both models, and spit out final comparisons in one shot. It looked clever, but it made iteration expensive. Breaking it into smaller scripts — “download once and cache immutable logs,” “run a single model over profiles,” “summarize saved outputs” — meant I could fix bugs in one stage without burning API calls in the others. When the summarizer broke, I didn’t have to re-run the models; I just reprocessed the saved outputs. When the downloader broke, I fixed it without touching any LLMs. That separation is what kept this experiment financially and technically sane.\n\nEven with just two models, I needed the outputs to line up so comparison wasn’t vibes-based. For each CI run, I forced the model output into a consistent shape: failure domain (a short label like `db-connection`\n\nor `test-assertion`\n\n), a one-to-three sentence root cause summary, evidence pointers (log snippets or line ranges), a coarse confidence bucket (low/medium/high), three to five concrete remediation steps, and a simple verdict on whether this looked like a real issue or something flaky/infra-like.\n\nOn top of that, every run captured input/output tokens, cached vs non-cached tokens, total cost in USD, and latency. Scoring and comparison scripts then answered very boring but very important questions: did the failure domain match the labeled truth, was the root cause directionally correct, were the steps actionable and safe, was one model consistently overconfident and wrong, and how much extra did that cost.\n\nFrom there, I rolled the data up into three layers of summaries: per-flavor (for example, \"on DB-connection failures Haiku passed X/Y, Sonnet passed A/B, here’s the cost and latency delta\"), per-task across all flavors (overall accuracy, cost to get one \"good enough\" answer for CI diagnostics per model), and a human-readable decision summary.\n\nThis is where TierForge fits. [TierForge is an open-source router](https://github.com/deghosal-2026/ai-tierforge) that turns evidence into YAML routing rules. It wants routing rules backed by real data: for a given use case like `ci-diagnostics`\n\n, what is the primary model, what is the escalation model, and under what conditions do you switch. The experiment gives it exactly that: a concrete \"Haiku-by-default, Sonnet-on-these-conditions\" rule, plus actual cost-per-task numbers instead of gut feel.\n\nLooking back, here’s what survived and what I keep using.\n\n**1. The harness matters more than the paper**\n\nI don’t need to publish “definitive bench results” to get value.\n\nWhat I needed was:\n\nThat’s what model-compass turned into.\n\n**2. One well-chosen experiment beats ten generic ones**\n\nCI diagnostics:\n\nIt’s a better teacher than a grab bag of random tasks I’ll never run in production.\n\n**3. My personal budget shapes my learning strategy**\n\nBecause I can’t justify hundreds of dollars on personal experiments:\n\nAt work, I might run more of the matrix. Privately, I only need enough to calibrate my instincts.\n\nThe workload-aware profiles I’m building help a lot here. I am not making model choices in a vacuum; I am making them for specific, well-understood slices of work. That lets me stay inside a tight personal budget without taking a meaningful hit on quality.\n\n**4. “Cheap vs expensive model” isn’t just a cost problem**\n\nThe CI experiment answered a cost question.\n\nThe hard part is a behavior question:\n\nwill I actually reach for Haiku by default for CI work, or will I still click the fancy model because it feels safer?\n\nI’m still catching myself defaulting to the “nice” model sometimes, even with data in hand.\n\nI don’t have clean answers here. Just a few open questions I’m carrying around:\n\nIf you’ve navigated this tension — **work budget vs personal budget, curiosity vs cost** — I’d love to hear how you handle it.\n\nYes, the bill matters. But in the teams I talk to, “this is too expensive” often hides other worries: we’re not sure we trust the outputs yet, we don’t feel fully in control of the workflows, or we’re afraid of looking reckless to leadership. I’m trying to go the other way: tighten costs *by* getting more workload-aware and evidence-driven, instead of tightening costs because we’re scared.\n\nWhen you think about containing AI costs in your own team, what is really driving it? Is it just the cloud bill, or something deeper — trust, optics, control, or the feeling that you should understand what’s going on before you let a model touch everything?", "url": "https://wpnews.pro/news/i-planned-10-llm-evaluation-experiments-and-only-ran-1-it-was-enough", "canonical_source": "https://dev.to/debashish_ghosal/i-planned-10-llm-evaluation-experiments-and-only-ran-1-it-was-enough-2gjf", "published_at": "2026-07-26 23:21:06+00:00", "updated_at": "2026-07-26 23:31:44.842864+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "developer-tools", "ai-products"], "entities": ["Haiku", "Sonnet", "Opus", "Claude", "GitHub Actions", "model-compass"], "alternates": {"html": "https://wpnews.pro/news/i-planned-10-llm-evaluation-experiments-and-only-ran-1-it-was-enough", "markdown": "https://wpnews.pro/news/i-planned-10-llm-evaluation-experiments-and-only-ran-1-it-was-enough.md", "text": "https://wpnews.pro/news/i-planned-10-llm-evaluation-experiments-and-only-ran-1-it-was-enough.txt", "jsonld": "https://wpnews.pro/news/i-planned-10-llm-evaluation-experiments-and-only-ran-1-it-was-enough.jsonld"}}