If your eval set hasn’t changed since launch, it’s grading the agent you shipped, not the one you’re running. Three habits that fix that:
→ Grade the full trajectory, not just the final answer
→ Use code checks for facts and LLM judges for tone
→ Turn every failure into a regression test
Platforms built for continuous AI agent evaluation can turn red-team findings into evals automatically.
Do you evaluate full trajectories or only outputs?
I do both, and the trajectory is where most regressions show up. For each case I assert on the sequence of tool calls: names, argument schema validity, and no repeated side-effecting call after a retry. The final answer can look right while the agent took a wrong or duplicated step. Two additions to your three habits: run each case several times and report the share of runs that pass every check, because one clean run says little about reliability; and replay recorded tool outputs with injected failures such as timeouts and malformed payloads, to confirm the agent stops within its step budget. For regression tests built from failures, I keep the exact trace and make the check name the failure mode.