{"slug": "adamast-adaptive-failure-taxonomies-for-improving-llm-agents", "title": "AdaMAST: Adaptive Failure Taxonomies for Improving LLM Agents", "summary": "Researchers introduced AdaMAST, an adaptive extension of MAST, which converts an LLM agent's execution traces into a taxonomy of named failure codes along three fixed axes, enabling durable and diagnostic failure feedback without human annotation. The taxonomy is induced via eight LLM calls and validated by four LLM annotators with a mean pairwise Cohen's κ of at least 0.75. This approach addresses the limitations of scalar outcomes and free-text critiques by providing a reusable vocabulary of failure modes for improving agent performance.", "body_md": "Research blog\n\n# AdaMAST: Adaptive Failure Taxonomies for Improving LLM Agents\n\n*Fantastic Adaptive\nTaxonomies and How to Use Them*\n\nJuly 2026\n\n## Traces Are Evidence, Not an Interface\n\nImproving an LLM agent without touching its weights usually means one of these loops: sample N trajectories and pick the best one; run the agent, read what went wrong, rewrite the prompt; watch a run in flight and intervene before it commits a mistake. Each of these procedures consumes the same input (execution traces) and extracts feedback in one of two forms.\n\nA **scalar outcome** (a pass rate, a verifier score)\npersists and compares cleanly across runs, but it records only\n*that* a trajectory failed, never *why*. A\n**free-text critique** (Reflexion-style reflection,\nself-refinement) names the why, but it is generated from scratch on\nevery run and discarded with it. The recurring failures of your\nparticular system (a checker that rubber-stamps its solver, a tool\ncalled retried unchanged) get re-diagnosed, in disposable prose,\nforever.\n\nWhat's missing is an artifact that is **durable, like a\nscore, and diagnostic, like a critique**: a compact\nvocabulary of named failure modes, produced once from the system's\nown traces, that downstream procedures reason over instead of long,\nnoisy trajectories.\n\n💡 **The central claim:** the raw trace need not\ndouble as the feedback interface. An agent system should maintain\nan explicit representation of *how it fails*, induced from\nits own behavior and reusable wherever failure feedback is needed.\n\n## AdaMAST: Fixed Axes, Induced Vocabulary\n\nAdaMAST, an adaptive extension of\n[MAST](https://arxiv.org/abs/2503.13657),\nconverts a target system's traces into a taxonomy of named failure\ncodes organized along **three fixed axes**:\n\n-\n**Axis A (system-level):** fix the harness or orchestration (context exhaustion, premature termination). -\n**Axis B (role-specific):** fix a discovered role's behavior (a checker that answers the prompt instead of checking). -\n**Axis C (domain-specific):** inject task knowledge (algorithm mismatch, physical-law violations).\n\nOnly the axes are fixed, so taxonomies stay comparable across\nsystems, and every code maps to an intervention point. The codes\nthemselves (names, definitions, role labels, evidence patterns) are\ninduced from the traces: no code is hand-authored, no trace is\nhuman-annotated. Concretely, induction is eight LLM calls: an\nanalysis pass extracts the domain, the discovered roles, and\nrecurring behavioral signals from the trace pool; a curation pass\nproposes candidate codes per axis, grounded in quoted trace\nevidence (one prompt per discovered role); and a consolidation pass\nmerges duplicates, rejects codes without supporting evidence, and\nenforces coverage. The name carries lineage, not scope: for a\nsingle-agent coding harness, the role axis simply names execution\n*phases* (Edit / Plan / Verify), and for a flat pipeline, it\ncan be empty.\n\nWhy induce the vocabulary rather than fix it in advance? MAST\ndistilled 14 recurring failure modes from hundreds of annotated\nmulti-agent traces and remains a strong baseline throughout this\nwork. But any rubric fixed *before* the target system is\nobserved faces a structural limit, not a quality limit:\nrole-specific failures presuppose roles that exist only once an\narchitecture is instantiated, and domain-specific failures\npresuppose task knowledge that no general rubric enumerates. No\npre-committed list anticipates a competitive-programming agent\ncommitting to a greedy strategy where the problem demands dynamic\nprogramming.\n\nBecause the codes are induced, they are validated before anything trusts them: four LLM annotators label held-out traces independently, then reconcile under a bounded deliberation protocol, and the taxonomy is accepted only when they apply it consistently (mean pairwise Cohen's κ ≥ 0.75: a standard measure of inter-annotator agreement, where 1 is perfect and 0.75 counts as substantial; this gate measures annotation consistency, a different quantity from the agreement-with-human-experts κ in Figure 4). A shortfall triggers merge/add/relabel edits and a re-run. And because systems change, online refinement keeps the vocabulary tracking the current system rather than a snapshot of its past.\n\n## Three Ways AdaMAST Helps Improve Agents\n\nThe point of an interface is reuse. We reuse the same kind of induced vocabulary across three deliberately different improvement procedures.\n\n### 1. Agent-system search: better targets for the mutator\n\nIn evolutionary agent search (we build on\n[AdaEvolve](https://skydiscover-ai.github.io/blog-adaevolve.html)),\nthe mutation step reads the parent's failed traces and decides what\nto change. The default feedback is free-form LLM reflection;\nAdaMAST replaces it with a taxonomy-coded diagnosis: fired codes,\nquoted evidence, cross-problem patterns.\n\nCoded diagnoses win on **all five benchmarks**\n(Frontier-CS competitive programming, OlympiadBench olympiad math,\nMMLU-Pro STEM QA, TheoremQA graduate-level math, and DROP discrete\nreasoning) by +3.5 to +7.5 points. On the largest held-out set (655\nOlympiadBench problems), the taxonomy-guided architecture reaches\n**91.9%** vs. 87.9% for reflection-guided search, and\n89.5% for search guided by the fixed MAST checklist.\n\nThe mechanism is visible in the search trajectory: the vanilla run plateaus at dev score 0.30 and exhausts its search by iteration 26 (the open marker in Figure 3), while the taxonomy-guided run keeps finding productive edits, escaping plateaus at iterations 69 and 89. Each jump coincides with a code-driven architecture edit: first, the addition of a verification gate, then, after the coordinator-aggregation code B.6 fires, the promotion of that gate into a score boost.\n\n### 2. Runtime monitoring: a failure vocabulary as a skill\n\nAt runtime, the taxonomy enters the agent's context, and the agent\nchecks its own recent trace against it at checkpoints: reflection\nanchored to the system's known failure modes, with no additional\nLLM calls (the vocabulary rides in the agent's existing context).\nOn **SWE-bench Verified Mini**, the arms form a clean\nladder:\n\n| Feedback arm | SWE-agent | Claude Code |\n|---|---|---|\n| Base (no reflection) | 50% | 64.0% |\n| Free-text reflection | 60% | n/a |\n| Fixed MAST checklist | 68% | 67.3% |\nAdaMAST | 70% | 70.7% |\n\nClaude Code figures are a three-seed average; SWE-agent is a single seed. The free-text arm belongs to the SWE-agent ladder, where prompted reflection is the main corrective signal; Claude Code natively self-verifies by running tests between edits.\n\nSince the three SWE-agent reflection arms share an identical\nscaffold and differ only in content, the ordering isolates\n*what the reflection should say*: structure accounts for\nmost of the gain, and the induced, harness-specific vocabulary adds\nthe rest. On Claude Code, attaching the taxonomy as a native\nruntime skill still lifts resolution from 64.0% to\n**70.7%**, with the gains concentrated in\nverification-phase failures.\n\n### 3. Trajectory selection: codes as verification criteria\n\nGiven N candidate trajectories and no ground-truth checker, which one actually succeeded? AdaMAST-Judge turns each induced code into a verification criterion (semantic failure modes become LLM scoring prompts, structural ones become cheap heuristic checks) inside an LLM-as-a-Verifier pipeline.\n\n| Method | terminus-2 | claude-code | ForgeCode |\n|---|---|---|---|\n| Pass@1 (no selection) | 61.8% | 57.5% | 81.8% |\n| LLM-as-a-Verifier | 71.2% | 61.2% | 86.5% |\n| Fixed MAST vocabulary | 68.5% | 69.0% | 88.8% |\nAdaMAST-Judge | 73.0% | 72.4% | 89.9% |\n| Best-of-5 oracle | 77.5% | 80.5% | 89.9% |\n\nBest-of-5 selection accuracy on Terminal-Bench 2.0 (five pre-recorded trajectories per task, Claude Opus 4.6 agents, GPT-5.4 as the verifier for all methods). The MAST row runs the identical verifier pipeline with only the vocabulary swapped. On ForgeCode, both vocabularies reach the best-of-5 oracle ceiling.\n\nAcross three Terminal-Bench 2.0 harnesses, it improves best-of-5 accuracy by 8 to 15 points over Pass@1, keeping a +3.4/+4.5 point margin over the fixed MAST vocabulary on the two harnesses that aren't already saturated. The underlying signal is simple: passing trajectories fire fewer codes than failing ones; even a selector that just picks the candidate with the fewest fired codes lifts expected accuracy from 58% to 67% on the tasks where selection can change the outcome.\n\n## Does the Taxonomy Itself Hold Up?\n\nDownstream gains alone do not establish that the induced representation is compact, faithful, or genuinely adaptive, so we also measure the taxonomy directly.\n\n-\n**Compact.** Coding compresses the failure-relevant content of traces by ~18x, yet 89% of traces retain unique code signatures, preserving substantial distinctions across runs. In a functional test on Terminal-Bench, a consumer LLM predicting held-out run success performs as well with the 1.2K-token taxonomy as with the 114K-token verbatim trace pool (statistically indistinguishable balanced accuracies across two consumer models). -\n**Human-faithful.** Under the same annotation panel and prompts, the induced vocabulary matches expert failure labels better than the benchmark's hand-crafted one (κ is 0.682 vs. 0.516; the full protocol reaches 0.725). This is a different question from the ≥ 0.75 gate earlier: that measured whether annotators applied the codes*consistently*; this measures whether the codes align with*human experts*. -\n**Adaptive.** Six domains, mean pairwise code overlap 0.14, backbone projection 0.50. The result supports the need for adaptation: beyond a shared backbone of generic failures, each domain has a substantial vocabulary of its own, role- and task-specific.\n\n## Caveats and FAQ\n\n🔧 **Honest caveats.** First, the Terminal-Bench\nselection gains aren't the vocabulary's alone. AdaMAST-Judge runs\nthe codes through a learned selector that keeps only the most\ndiscriminative criteria, so the codes and that machinery share the\ncredit. Search and runtime are the cleaner tests of the vocabulary\nitself; there, the codes are read directly, with nothing learned in\nbetween. Second, fixed taxonomies are not straw men. On SWE-agent,\nthe fixed 14-code MAST checklist reaches 68%, compared to AdaMAST's\n70%. What the induced taxonomy adds is the long tail: the role- and\ndomain-specific failures that no rubric written before seeing your\nsystem can include.\n\n**Isn't this just better prompting?** A prompt is one\nconsumer of the taxonomy, not the contribution itself. AdaMAST\ninduces and validates a persistent vocabulary from traces,\naggregates failures across runs, and reuses that vocabulary in\nsearch, runtime monitoring, and selection.\n\n**Does it need humans in the loop?** No per-trace\nhuman annotation and no hand-authored codes. Human faithfulness is\n*certified* against expert annotations (that's how we\nvalidate the method), but the pipeline itself runs fully\nautomatically.\n\n**Does it only work for multi-agent systems?** No. The\nrole axis adapts: multi-agent systems get role codes (13 for our\ncompetitive-programming pipeline), single-agent harnesses get phase\ncodes (Edit/Plan/Verify for Claude Code), and flat pipelines get\nnone.\n\n## Closing the Loop\n\nBetween raw traces and scalar outcomes sits a missing representation: named, evidence-grounded, system-specific failure modes. Agents already generate the evidence of their own failures on every run. The cheapest improvement to an agent system may be to first learn, in its own terms, how it fails.\n\n```\n@article{cemri2026adamast,\n  title={Fantastic Adaptive Taxonomies and How to Use Them},\n  author={Cemri, Mert and Cojocaru, Andrei and Pan, Melissa and Liu, Shu and\n          Agarwal, Shubham and Krentsel, Alexander and Tang, Jay and\n          Ramchandran, Kannan and Gonzalez, Joseph E. and Zaharia, Matei and\n          Dimakis, Alex and Stoica, Ion},\n  journal={arXiv preprint arXiv:2607.16387},\n  year={2026}\n}\n```\n\n", "url": "https://wpnews.pro/news/adamast-adaptive-failure-taxonomies-for-improving-llm-agents", "canonical_source": "https://multi-agent-systems-failure-taxonomy.github.io/AdaMAST/blogs/adamast_paper/", "published_at": "2026-07-31 06:22:31+00:00", "updated_at": "2026-07-31 06:52:56.151679+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-research"], "entities": ["AdaMAST", "MAST"], "alternates": {"html": "https://wpnews.pro/news/adamast-adaptive-failure-taxonomies-for-improving-llm-agents", "markdown": "https://wpnews.pro/news/adamast-adaptive-failure-taxonomies-for-improving-llm-agents.md", "text": "https://wpnews.pro/news/adamast-adaptive-failure-taxonomies-for-improving-llm-agents.txt", "jsonld": "https://wpnews.pro/news/adamast-adaptive-failure-taxonomies-for-improving-llm-agents.jsonld"}}