Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers

A new study from researchers testing six large language models across four capability tiers found that the relationship between model capability and optimal harness complexity is non-monotone, contradicting the assumption that higher-capability models require less structural guidance. In a 432-run experiment on the HEAT-24 benchmark, a frontier chat model (Gemini 2.5 Flash) saw its task success rate drop by 29-38 percentage points with increased harness verbosity, while a frontier reasoning model (Qwen3.5-122B) achieved its highest success rate (91.7%) under the strictest harness. The findings, which also showed a 2B model matching the stability of stronger models, highlight that harness sensitivity depends critically on model type and that tier-aware selection guidelines are needed for reliable LLM agent deployment.

Computer Science Artificial Intelligence Submitted on 26 May 2026 Title:It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers View PDF /pdf/2605.26731 HTML experimental https://arxiv.org/html/2605.26731v1 Abstract:A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models need proportionally less structural guidance -- together implying a monotone inverse relationship between model capability tier and optimal harness complexity. We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions light, balanced, strict on HEAT-24, a 24-task synthetic benchmark with git-based workspace verification. Our results refute the monotone inverse relationship on two fronts. First, for the frontier chat model evaluated Gemini 2.5 Flash , increased harness verbosity lowers VTSR by 29-38 percentage points -- a harness-complexity paradox. Second, for the frontier reasoning model evaluated Qwen3.5-122B, extended thinking enabled , strict harness achieves the highest VTSR 91.7% and the lowest latency, the opposite of the prediction. Within the constrained tier, a 2B model Gemma4:e2B matches strong-open-tier stability at 91.7% across all harnesses. Because each tier is represented by a single model in this study, these results should be interpreted as model-specific observations; harness sensitivity appears non-monotone across the models evaluated, and depends critically on model type chat vs. reasoning . We introduce a six-label failure taxonomy showing that format violation dominates capable-model failures while wrong file dominates low-capability failures, and we derive practical tier-aware harness selection guidelines. References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .