Ask HN: What eval harness holds up in practice, and what is still missing?

There’s no shortage of eval frameworks, but once you move past simple single-turn prompt checks into multi-step, tool-calling agents, most of them start to feel either too heavy or too brittle for CI. Testing intermediate trajectories, retries, and non-deterministic state often turns into flaky LLM judges, slow test runs, or snapshot diffs that become painful to maintain.

For people evaluating production LLM apps or agents, what harness or setup has actually held up for you without creating more maintenance than it saves? Are you using a dedicated open-source/commercial framework, or did you eventually fall back to custom runners and curated golden sets? And what’s the biggest thing you still wish current eval tooling handled better?

Comments URL: [https://news.ycombinator.com/item?id=49430207](https://news.ycombinator.com/item?id=49430207)

Points: 1