# New benchmark reveals AI agents follow complex instructions less than 30% of the time > Source: > Published: 2026-08-17 20:28:15+00:00 Via mlcommons.org # New benchmark reveals AI agents follow complex instructions less than 30% of the time A study using 707 real-world instructions exposes how badly current language models handle the kind of complex, multi-constraint tasks they're actually being deployed for. Ask a large language model to write a haiku and it’ll nail it. Ask it to follow a 1,700-word instruction manual with nearly a dozen constraints while using specific tools in a specific order, and things fall apart fast. A new benchmark called AGENTIF, published on May 22, 2025, puts a number on just how badly. Current advanced AI models achieve less than 30% perfect instruction following when tested against the kind of complex, real-world tasks that companies are actually trying to automate. ## What AGENTIF actually tested The benchmark was developed by researchers at Tsinghua University and Zhipu AI. It consists of 707 human-annotated instructions drawn from 50 task categories spanning industrial applications and open-source systems. Previous evaluation frameworks like IFEval typically used shorter, simpler prompts that don’t reflect the messy reality of deploying AI agents in production environments. AGENTIF’s instructions average 1,723 words each, and each instruction contains an average of 11.9 constraints that the model must satisfy simultaneously. The constraints themselves fall into three broad categories: formatting constraints (how the output should be structured), semantic constraints (what the output should actually say or mean), and tool constraints (which tools to use and how to use them). They also appear in different representation forms, including vanilla statements, conditional logic, and example-based specifications. Models struggled most with tool specifications, which requires not just understanding language but also correctly sequencing actions across external systems. Complex constraint structures compounded the problem. ## The evaluation methodology AGENTIF uses a combination of code-based, LLM-based, and hybrid evaluation methods to assess constraint satisfaction. Code-based evaluation can catch formatting violations and tool misuse with high precision. LLM-based evaluation handles the fuzzier semantic constraints where human judgment is required. ## What this means for AI deployment If the best available models can’t reliably follow complex instructions even 30% of the time, the gap between demo and deployment remains enormous. For enterprises building AI-powered automation, the findings suggest that heavy guardrails, human oversight, and careful task decomposition remain essential. One particularly telling finding: AI agents perform worse when given rules that oppose their default behavior. Models have strong priors baked in during training, and when explicit instructions conflict with those learned tendencies, the instructions frequently lose. The research comes from Tsinghua University and Zhiyu AI, not ByteDance. ByteDance’s Seed AI team conducts its own separate agent research. Regulatory dynamics in China add another layer of complexity. Companies like ByteDance and Alibaba face evolving government requirements that shape what their AI systems can and cannot do, effectively adding yet another set of constraints that models need to satisfy on top of whatever the user or enterprise requires. In a world where instruction following is already below 30%, additional regulatory constraints only narrow the margin for error further. **Disclosure:** This article was edited by Editorial Team. For more information on how we create and review content, see our [Editorial Policy](https://cryptobriefing.com/editorial-policy/).