{"slug": "ai-eval-habits-every-agent-team-should-adopt", "title": "AI Eval Habits Every Agent Team Should Adopt", "summary": "A practitioner post on AI agent evaluation argues that teams should grade the full trajectory of tool calls rather than only final answers, using code checks for facts and LLM judges for tone, and converting every failure into a regression test. Commenters added that each case should be run several times with the share of fully passing runs reported, and that recorded tool outputs should be replayed with injected failures such as timeouts and malformed payloads to confirm agents stop within their step budget. The post recommends asserting on tool-call names, argument schema validity, and the absence of repeated side-effecting calls after a retry, and notes that platforms built for continuous AI agent evaluation, such as BotGauge, can turn red-team findings into evals automatically.", "body_md": "If your eval set hasn’t changed since launch, it’s grading the agent you shipped, not the one you’re running.\n\nThree habits that fix that:\n\n→ Grade the full trajectory, not just the final answer\n\n→ Use code checks for facts and LLM judges for tone\n\n→ Turn every failure into a regression test\n\nPlatforms built for [continuous AI agent evaluation](https://www.botgauge.com) can turn red-team findings into evals automatically.\n\nDo you evaluate full trajectories or only outputs? \n\n \n \nI do both, and the trajectory is where most regressions show up. For each case I assert on the sequence of tool calls: names, argument schema validity, and no repeated side-effecting call after a retry. The final answer can look right while the agent took a wrong or duplicated step. Two additions to your three habits: run each case several times and report the share of runs that pass every check, because one clean run says little about reliability; and replay recorded tool outputs with injected failures such as timeouts and malformed payloads, to confirm the agent stops within its step budget. For regression tests built from failures, I keep the exact trace and make the check name the failure mode.", "url": "https://wpnews.pro/news/ai-eval-habits-every-agent-team-should-adopt", "canonical_source": "https://discuss.huggingface.co/t/ai-eval-habits-every-agent-team-should-adopt/190996#post_6", "published_at": "2026-10-30 10:38:02+00:00", "updated_at": "2026-10-30 10:48:28.529354+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools"], "entities": ["BotGauge"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-eval-habits-every-agent-team-should-adopt", "markdown": "https://wpnews.pro/news/ai-eval-habits-every-agent-team-should-adopt.md", "text": "https://wpnews.pro/news/ai-eval-habits-every-agent-team-should-adopt.txt", "jsonld": "https://wpnews.pro/news/ai-eval-habits-every-agent-team-should-adopt.jsonld"}}