Diagnosing with Insights: Structured Analysis of Agent Failures via Behavioral Abstractions

Researchers introduced AGENTSCOPE, a neuro-symbolic approach that abstracts LLM agent trajectories into structured representations and uses neural invariants with LLM-guided reasoning to pinpoint failure steps and types. On the Who&When dataset and their new AgentErrata dataset, AGENTSCOPE significantly outperformed current state-of-the-art methods in fault localization and attribution accuracy, offering more reliable and interpretable diagnosis of agent failures.

arXiv:2609.02371v1 Announce Type: new Abstract: With the proliferation of LLM agents, the ability to understand and diagnose failures in agents is essential to achieving superior effectiveness and trustworthiness. As agent failures often manifest via long and complex trajectories, manually finding the needles in the haystack is untenable. However, traditional diagnosis techniques for software bugs can hardly address LLM agent failures, while completely relying on LLMs as the judge yields unreliable diagnosis results. To overcome these challenges, this paper presents AGENTSCOPE, a new neuro-symbolic approach for agent failure mode diagnosis. The key principle of AGENTSCOPE is to abstract agent behavior, based on its trajectories, into structured representations. Furthermore, AGENTSCOPE introduces the concept of neural invariants to specify agent behavior properties. AGENTSCOPE leverages LLM-guided reasoning atop the structured representation against neural invariants to pinpoint both the failure step and its type in the trajectory. We show the effectiveness of AGENTSCOPE on publicly available agent failure datasets Who&When and a more comprehensive dataset created by us AgentErrata , where AGENTSCOPE significantly outperforms the current state of the art in fault localization and attribution accuracy. Our work shows that integrating structured abstractions with LLM-guided reasoning enables effective, reliable, and interpretable diagnosis for agent failures.