# LLM Eval: Why I scrapped 9/10 of my experiments

> Source: <https://promptcube3.com/en/threads/3896/>
> Published: 2026-07-26 23:46:28+00:00

# LLM Eval: Why I scrapped 9/10 of my experiments

`model-compass`

to handle everything from CI diagnosis to architecture analysis across multiple providers.The plan was to test things like "Context Rot" (where long-context recall fails) and "Tool Drift" (accuracy degradation with 50+ tools). I wanted a routing table that told me: "For this specific task, use the cheapest model that passes the quality bar."

But I stopped after running exactly one experiment: CI diagnostics.

## Why CI Diagnostics was the only one that mattered

I skipped the "sexy" tests and went straight to CI diagnostics because that's the real-world pain point. My team works across TypeScript, Swift, Kotlin, C++, Java, and Go. When a pipeline goes red, the logs are massive, and a significant chunk of failures are just flaky integration tests.

I wanted to see if an LLM could actually triage these failures meaningfully. The result of that one test gave me more signal than any theoretical benchmark ever could. It highlighted the massive gap between "toy prompts" and the messy reality of megabyte-sized logs and multi-repo dependencies.

## The "Model-Compass" Setup

Even though I only ran one test, the infrastructure I built for this deep dive was useful. For anyone building their own AI workflow or LLM agent, here is how I structured the evaluation:

**Task Rubrics:** I used a mix of binary (pass/fail), exact-match, and LLM-judge scoring.**Adapter Layer:** A unified interface for Anthropic, OpenAI,[DeepSeek](/en/tags/deepseek/), and OpenRouter.**Cost Tracking:** Auto-computed USD estimates per task to see the actual burn rate.

## The Reality Check: Local vs. Frontier

The biggest takeaway from this experience was the tension between a corporate budget and a personal one. At work, running 20-shot experiments on frontier models is just a Jira ticket away. At home, every token comes out of my own pocket, and my local GPU can't handle frontier-level quality at 4K context windows.

This forced me to be surgical. I realized that for most "DevOps-style" agent tasks, you don't need a fleet of 10 experiments to find the breaking point. One high-friction, real-world use case tells you everything you need to know about a model's reasoning ceiling.

[Next London NO2 Forecasting: Why ARIMA and Prophet Failed Me →](/en/threads/3891/)