Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

A new benchmark called Trial by Trace targets the evaluation of AI agent harnesses for reliability and failure attribution, arguing that existing benchmarks focus on model capabilities and overall agent performance while giving limited attention to the harness layer. The work addresses how agent effectiveness depends on both the underlying models and the harnesses that support their operation.

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the eval