LLM evals are a parameter sweep — use a parameter sweep tool

An engineer argues that LLM evaluation is fundamentally a parameter sweep and should be treated as such, using tools like pydantic-evals and oryxflow instead of SaaS platforms. The post demonstrates a classification eval for an agent's intent classifier, emphasizing exact-match scoring over LLM-as-judge and reporting precision/recall rather than aggregate accuracy.

The scoring is genuinely new. The matrix underneath it is a solved problem from 2015. Three questions teams actually ask about their LLM systems: They feel like three different projects. They're one shape: a labeled dataset, run under N configurations, compared. Write that out as a grid — cases × scorers × models × prompt variants × repetitions — and you have described a Cartesian product over parameters. A parameter sweep. Data science has had good tooling for parameter sweeps for a decade, built around one idea: cache one output per parameter set, and when a value changes, recompute exactly the cells that value touches. The LLM eval space rebuilt that layer as SaaS, with a meter on it. Some of what those platforms sell is genuinely new and genuinely worth paying for. Traces of an agent's tool calls, a UI for clicking through failures, span-level debugging — that's a hard problem, well solved, and not something you want to rebuild. But the matrix isn't LLM-specific, and neither is the persistence. Those are the parts where treating evals as a normal data-science sweep pays off immediately. Nothing overlaps, which is the whole point. Each layer does one job and none of them needs to know about the others. Every agent has an intent classifier somewhere. Ours decides whether an incoming query needs a live web search: two labels, needs web and no web . Two methodological points before code, because they're where evals usually go wrong. This is classification, not a job for LLM-as-judge. The output is one of two fixed strings. Exact match is the correct scorer. LLMJudge exists for free-text outputs with no single right answer — using it here would add cost, latency and sampling noise to a comparison that was already deterministic. Aggregate accuracy lies. Your two errors aren't symmetric. A false positive — searching unnecessarily — costs a few hundred milliseconds. A false negative — not searching when you should have — means the agent answers a question about this morning's news from training data, confidently. That's the failure users screenshot. If needs web is 30% of traffic and you miss a tenth of it, headline accuracy reads 97% while recall on the class that actually matters is 90%. pydantic-evals handles this without a custom scorer. ConfusionMatrixEvaluator and PrecisionRecallEvaluator are report-level evaluators — they run once over the whole experiment rather than per case — and you pass them via report evaluators on the Dataset : python from pydantic evals import Case, Dataset from pydantic evals.evaluators import EqualsExpected, ConfusionMatrixEvaluator CASES = Case name='weather now', inputs='what is the weather in Lisbon right now', expected output='needs web' , Case name='reverse list', inputs='how do I reverse a list in python', expected output='no web' , Case name='latest release', inputs='what shipped in the newest postgres release', expected output='needs web' , Case name='explain tcp', inputs='explain the TCP handshake', expected output='no web' , dataset = Dataset name='needs web v3', cases=CASES, evaluators= EqualsExpected , report evaluators= ConfusionMatrixEvaluator predicted from='output', expected from='expected output', title='needs web vs no web', , , Everything you vary becomes a task parameter. oryxflow keys the cached output on those values, so each combination gets its own stored result — no filenames to invent, no results sonnet v3 final2.json graveyard. python import pandas as pd import oryxflow class EvalRun oryxflow.tasks.TaskPqPandas : model = oryxflow.Parameter default='claude-sonnet-5-20260501' prompt version = oryxflow.Parameter default='v3' dataset version = oryxflow.Parameter default='needs web v3' def run self : classify = build classifier self.model, self.prompt version report = dataset.evaluate sync classify, name=f'{self.model}/{self.prompt version}' rows = pd.DataFrame { 'case': c.name, 'query': c.inputs, 'expected': c.expected output, 'predicted': c.output, } for c in report.cases rows 'correct' = rows 'expected' == rows 'predicted' self.save rows self.saveMeta { 'accuracy': float rows 'correct' .mean , 'recall needs web': float rows.loc rows 'expected' == 'needs web', 'correct' .mean , } report.print Save per-case rows , not a summary. A stored accuracy number can only answer the question you thought of at the time. Per-case rows let you slice by class, by case, by query type, six months later, without re-running a thing. The grid is a requires that returns a dict. inputLoadConcat row-stacks the dependencies and tags each with the parameters of the task it came from, so model and prompt version arrive as columns: MODELS = 'claude-haiku-4-5', 'claude-sonnet-5-20260501', 'claude-opus-5' PROMPTS = 'v2', 'v3' class EvalMatrix oryxflow.tasks.TaskPqPandas : def requires self : return {f'{m}|{p}': EvalRun model=m, prompt version=p for m in MODELS for p in PROMPTS} def run self : self.save self.inputLoadConcat flow = oryxflow.Workflow EvalMatrix flow.run df = flow.outputLoad df.groupby 'model', 'prompt version' 'correct' .mean the headline df df.expected == 'needs web' .groupby 'model' 'correct' .mean what matters Here's why this framing pays, beyond tidiness. Trustworthy comparison. This is the one people underrate. If the dataset build is itself a cached upstream task, every configuration reads the same output — not because you were careful, but because there is only one output to read. That kills the quiet failure mode where you regenerated the case set midway through a sweep and half your comparison is against a different denominator. The comparison is apples-to-apples by construction — the same reason parameter sweeps over shared features https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/ are trustworthy in ordinary ML work: the shared upstream is computed once, so there's nothing to accidentally desynchronize. Marginal cost. Add a fourth model and only the fourth model runs. The six existing cells load from disk in milliseconds. For ordinary data work that's convenience — you saved some CPU. For LLM evals it's money, and it compounds with every axis: a 300-case set across three models and two prompts is 1,800 calls, and adding a model without caching means re-running all 2,400 instead of the 600 that are new. Under per-score metered billing you re-meter the cached cells too, so you pay twice for the privilege of not learning anything new. A score of 0.91 with no record of which prompt, which model string, which dataset revision is worthless in three months. You'll find it and be unable to defend it — which is how most eval setups actually die, long before retention expires. oryxflow writes an event stream as it runs: plain JSONL you can jq . Every execution records parameters, code version, a fingerprint of the task's source, upstream hashes, git SHA, duration, and the reason it ran. oryxflow.events.runs task family='EvalRun', last=2 diff params, code, hashes Edit your scoring logic and the affected tasks recompute automatically, with a reason naming the changed symbol — code change auto: evals/run.py::EvalRun . You can't compare an old-scorer score against a new-scorer score by accident, because the old one stops being a valid cache entry the moment the code changes. And it's durable in the boring sense: data/ under Git LFS versions your results alongside the code that made them. Use env= to keep blessed baselines env='baseline' apart from Tuesday's throwaway grid env='dev' , commit the scored parquet, and leave the megabyte traces in Logfire where the trace UI lives. Eval platform retention windows run 14 to 30 days on the common plans https://docs.oryxflow.dev/blog/ai-agents/llm-eval-results-deleted-retention/ ; a git repo runs as long as the repo does. A sweep over hyperparameters and a sweep over models aren't identical, and four differences matter. 1. The endpoint is outside the cache . oryxflow hashes your code and inputs. It cannot see a provider updating weights behind a stable alias — so claude-sonnet-5 as a parameter value means your cache will serve July's numbers as current in December. Fix: pin the snapshot — claude-sonnet-5-20260501 . The identity of what you measured is now part of the cache key, and a new snapshot is a new value that runs fresh while old numbers stay readable. To deliberately re-measure the same endpoint and check for drift, invalidate that family alone: flow.reset upstream EvalMatrix, only=EvalRun every cell reruns; dataset build doesn't 2. The function isn't deterministic . In a hyperparameter sweep, re-running a cell reproduces it. Here it doesn't. oryxflow guarantees a result came from the code and inputs it recorded — it is a faithful record of one sampling , not a reproducible constant. Fix: temperature=0 where supported, as a parameter so the record shows it. Then measure the residual variance instead of hoping about it — see point 4. 3. Latency measurement fights the harness. pydantic-evals runs cases concurrently by default, so wall-clock under a sweep measures your own queueing. Fix: a separate task with max concurrency=1 , discard the first call, report p50/p95 rather than the mean — one tail call destroys a mean and tells you nothing about typical experience. 4. Repetitions are a design decision, not a detail. Either the rep index is a parameter or it lives inside pydantic-evals: repeat= in evaluate sync When reps are expensive, take the parameter and let WorkflowMulti fan out: class LatencyRun oryxflow.tasks.TaskPqPandas : model = oryxflow.Parameter default='claude-sonnet-5-20260501' n run = oryxflow.IntParameter default=1 rep index ... flow = oryxflow.WorkflowMulti LatencyRun, {f'rep{i}': {'n run': i} for i in range 1, 4 } flow.run df = flow.outputLoadConcat every rep, tagged, one DataFrame Decide you need five reps? Extend the dict — reps 1–3 come from cache, only 4 and 5 call the API. That's the same marginal-cost property, applied to the axis most likely to grow after you've already run the study. Score with pydantic-evals, because scoring LLM output well is a real problem and it solves it. Debug in Logfire, because trace UIs are worth paying for. But recognize the middle layer for what it is: a Cartesian product over parameters, with one cached result per cell. That's not an LLM problem. It's a parameter sweep, and you can have the marginal-cost economics and the apples-to-apples guarantee that come with treating it like one. pip install oryxflow pydantic-evals Structurally, yes. "Is my classifier right", "did my prompt change help", and "is the cheap model good enough" are all the same shape underneath: a labeled dataset, run under N configurations, compared. Cases, scorers, models, prompt variants and repetitions are the axes of a grid, which makes an eval matrix a Cartesian product over parameters — exactly what parameter-sweep tooling has handled well for a decade. What is genuinely new is the scoring, and pydantic-evals covers that. Share the upstream. If the dataset build is its own cached task that every configuration depends on, all models are scored against byte-for-byte identical cases — not because you were careful, but because there is only one dataset output to read. That removes the quiet failure where you regenerated the case set halfway through a sweep and half your comparison is against a different denominator. You need both, for different jobs. A platform earns its cost for live traces and a UI to click through failures, which is a real problem well solved. The matrix and the persistence are not LLM-specific — one cached output per parameter set is ordinary data-science plumbing, and doing it with a caching workflow engine means adding a model to your grid runs only that model instead of re-metering cells whose results already exist on your disk. Make it a parameter when repetitions are expensive. Each rep then caches separately, so extending a three-rep study to five runs exactly two new evaluations instead of all five. The trade-off is a larger DAG. If repetitions are cheap, use pydantic-evals' repeat argument instead — one cached unit covers every rep, giving a smaller graph but a coarser cache, so changing the count re-runs the whole thing. Then keep going: