AgentLens, a new open-source benchmark for interactive code agents, evaluates not just task completion but also instruction-following, verification, and error recovery by blending formal verification …
AgentLens, a new benchmark for code agents, evaluates user experience and trajectory rather than just pass/fail metrics. Released as an open-source tool on GitHub, it combines formal verification with…
Researchers introduced AgentLens, a benchmark for evaluating interactive code agents that assesses entire trajectories—including instruction following, tool use, error recovery, and communication—rath…
A developer warns that agent retries can cause double execution of side effects like billing or email, leading to customer overcharges. They argue that side-effect safety is a Tier 1 evaluation proble…
A developer argues that irreproducibility is the most under-discussed failure in the agent space, as bugs that cannot be reproduced cannot be fixed or tested. They advocate for replayability over dete…
A developer warns that Goodhart's Law undermines agent evaluation suites when pass/fail scores become targets, causing teams to optimize for test performance rather than real-world quality. The post a…
A developer argues that autonomous agent systems suffer from 'drift'—a slow, silent degradation in output quality that standard monitoring fails to detect. They propose a solution combining continuous…
A developer argues that hallucination detection in AI agents is an instrumentation problem, not a model-quality issue, and introduces a layered detection approach using AgentLens and agent-eval tools.…
A developer argues that agent evaluations must run in CI to block regressions before deployment, not as post-hoc dashboards. The post introduces agent-eval for scoring outputs and AgentLens for execut…
Ramagiri Tharun built AgentLens, a 47-line Python profiler that parses API logs to audit AI tool call efficiency after loading 157 skills in 12 days revealed zero visibility into usage patterns. The t…