A developer warns that self-reported confidence scores from AI agents are not calibrated probabilities and should not be used for gating decisions. The developer argues that these scores are generated…
A developer warns that agent retries can cause double execution of side effects like billing or email, leading to customer overcharges. They argue that side-effect safety is a Tier 1 evaluation proble…
A developer warns that model upgrades can silently break agent workflows even when test suites pass, distinguishing the problem from drift. The post introduces a 'golden trace' approach with three tie…
A developer argues that irreproducibility is the most under-discussed failure in the agent space, as bugs that cannot be reproduced cannot be fixed or tested. They advocate for replayability over dete…
A developer warns that Goodhart's Law undermines agent evaluation suites when pass/fail scores become targets, causing teams to optimize for test performance rather than real-world quality. The post a…
A developer argues that autonomous agent systems suffer from 'drift'—a slow, silent degradation in output quality that standard monitoring fails to detect. They propose a solution combining continuous…
A developer argues that hallucination detection in AI agents is an instrumentation problem, not a model-quality issue, and introduces a layered detection approach using AgentLens and agent-eval tools.…
A developer argues that agent evaluations must run in CI to block regressions before deployment, not as post-hoc dashboards. The post introduces agent-eval for scoring outputs and AgentLens for execut…