# I built an LLM eval framework from scratch. Here is what I wish I had bought instead. > Source: > Published: 2026-07-14 21:50:54+00:00 One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever. Six months later that same framework was a mess. Three different judge models with three different parsing hacks. A test dataset nobody had touched since November. A CI gate that kept failing because a vendor nudged their model, not because anyone broke a prompt. And the second engineer on rotation asking me, fairly, "how does this even work?" The framework did not fail. The eighty percent of the work the weekend tutorial skipped is what failed. That gap is the whole story, and this is what I would tell myself before starting again. Here is the split that took me six months to see. Some parts of an eval setup are yours and only yours. The rubric that decides what "good" means for your product. The dataset built from your real failures. The rules for when a change is bad enough to block a release. Nobody else can write these, because they encode your domain. The rest is the same at every company. The thing that calls the judge model, parses its answer, retries, and caches. The machinery to run thousands of checks in parallel. The plumbing that scores live traffic. The system that groups failing calls together. Every team rebuilds these, hits the same bugs, and gains nothing by writing them twice. So build the first list. Do not hand-build the second. I rebuilt the second, and it cost me most of a year. Before writing any part of this, I ask two things: Build the pieces that are specific and compound. Reuse everything else. That single filter would have saved me the whole mess. Four things earned their place, and all four live in the same git repo as the agent code, reviewed in the same pull request as the prompt they score: The judge engine is the one that hurt the most, so start there. Every judge model hands back its score in a slightly different shape. One wraps it in tags, another writes "Score: 4," another buries the number after a paragraph of rambling. My parser caught almost all of it on day one. On day ninety a vendor shipped an update that formatted numbers differently, my parser quietly fell back to a middle score on the cases it missed, and my dashboard showed evals gently trending up. They were not. Middle scores just drag everything toward the average. On top of that came drift, where a judge scores the same answer differently this month than last after a model update, so the gate starts failing on prompts nobody touched. And cost, where running a top-tier judge across a big nightly set becomes a real budget line. Getting all of that right is months of work for an output that looks identical at every company. The same goes for running checks at scale, scoring live traffic, and grouping failing calls into themes. All generic, all a long slog, none of it your edge. I would reuse an existing runner for every one of these now. This was my most expensive small mistake. I set the gate to "fail if the average score drops below a line." The trouble is that a small dataset is noisy. With only thirty test cases, the average wobbles by a few points on its own, just from which cases you picked. So my gate failed on random noise and passed on real regressions, and within a month everyone had learned to ignore it, which is worse than having no gate at all. What actually works is two checks together: The second one is the difference between a gate people trust and a gate people mute. I kept my checks in CI only, and production drifted past my dataset within a quarter. The fix is to run the exact same rubric in both places: against your test set before release, and against live traffic after. When they disagree, that disagreement is useful. If CI is green but production is failing, your dataset has stopped looking like reality. If CI fails on something production never shows, your dataset is carrying dead weight. Either way you learn something, but only if it is the same check on both sides. The weekend version is genuinely a weekend. The real version is not. Once you are shipping a couple of products, keeping all of this alive is close to one to one and a half full-time engineers. And the cost never shows up when you expect it. It shows up the week a judge model updates and your parser breaks. The week a vendor changes their tracing format. The month your dataset quietly stops covering your product's top failure. None of that is on the weekend roadmap, and all of it is the job. To be fair, sometimes it is the correct call: And a few signs you have crossed the line and should stop hand-building: The lesson I keep coming back to is that the hard, valuable part was never the code. It was the rubric and the dataset, the two things only my team could write, and I spent my first six months building everything except those. If you have built one of these, I would love to hear which piece quietly ate the most of your time. For me it will always be that parser.