{"slug": "ask-hn-what-eval-harness-holds-up-in-practice-and-what-is-still-missing", "title": "Ask HN: What eval harness holds up in practice, and what is still missing?", "summary": "A Hacker News user asks which evaluation harnesses remain practical for production LLM agents, noting that most frameworks become brittle or heavy for multi-step, tool-calling scenarios in CI. The user seeks recommendations for setups that avoid maintenance overhead and highlights gaps in current tooling, such as handling intermediate trajectories, retries, and non-deterministic state.", "body_md": "There’s no shortage of eval frameworks, but once you move past simple single-turn prompt checks into multi-step, tool-calling agents, most of them start to feel either too heavy or too brittle for CI. Testing intermediate trajectories, retries, and non-deterministic state often turns into flaky LLM judges, slow test runs, or snapshot diffs that become painful to maintain.\n\nFor people evaluating production LLM apps or agents, what harness or setup has actually held up for you without creating more maintenance than it saves? Are you using a dedicated open-source/commercial framework, or did you eventually fall back to custom runners and curated golden sets? And what’s the biggest thing you still wish current eval tooling handled better?\n\nComments URL: [https://news.ycombinator.com/item?id=49430207](https://news.ycombinator.com/item?id=49430207)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/ask-hn-what-eval-harness-holds-up-in-practice-and-what-is-still-missing", "canonical_source": "https://news.ycombinator.com/item?id=49430207", "published_at": "2026-08-25 07:24:17+00:00", "updated_at": "2026-08-25 07:44:46.474412+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools", "mlops"], "entities": ["Hacker News"], "alternates": {"html": "https://wpnews.pro/news/ask-hn-what-eval-harness-holds-up-in-practice-and-what-is-still-missing", "markdown": "https://wpnews.pro/news/ask-hn-what-eval-harness-holds-up-in-practice-and-what-is-still-missing.md", "text": "https://wpnews.pro/news/ask-hn-what-eval-harness-holds-up-in-practice-and-what-is-still-missing.txt", "jsonld": "https://wpnews.pro/news/ask-hn-what-eval-harness-holds-up-in-practice-and-what-is-still-missing.jsonld"}}