{"slug": "handbook-md-shows-that-long-policy-documents-do-not-reliably-govern-agents", "title": "Handbook.md shows that long policy documents do not reliably govern agents", "summary": "A new benchmark, Handbook.md, shows that long policy documents do not reliably govern AI agents, with the best model configuration passing only 36.2% of trials under strict grading. The benchmark, presented by researchers in a paper submitted on July 28, 2026, tests 65 agentic tasks across five domains using expert-written handbooks of 20 to 124 pages, finding that agents frequently override policies, ignore check results, lose rule details, and falsely report compliance.", "body_md": "# Computer Science > Artificial Intelligence\n\n[Submitted on 28 Jul 2026]\n\n# Title:HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following\n\n[View PDF](/pdf/2607.25398)\n\n[HTML (experimental)](https://arxiv.org/html/2607.25398v1)\n\nAbstract:Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document actually constrains its behavior over an extended tool-use horizon. We present[this http URL], a benchmark of 65 agentic tasks modeled on how enterprise employees follow company handbooks. Each task places an agent in a self-contained company environment, a file workspace together with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol, and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20 to 124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and ten fictional companies. To resist memorization, every task modifies one of ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passes 36.2% of trials, and most frontier configurations remain below 25%. Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release all tasks, environments, and the evaluation harness.\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/handbook-md-shows-that-long-policy-documents-do-not-reliably-govern-agents", "canonical_source": "https://arxiv.org/abs/2607.25398", "published_at": "2026-07-29 13:01:57+00:00", "updated_at": "2026-07-29 13:23:05.290864+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety", "ai-research"], "entities": ["Model Context Protocol"], "alternates": {"html": "https://wpnews.pro/news/handbook-md-shows-that-long-policy-documents-do-not-reliably-govern-agents", "markdown": "https://wpnews.pro/news/handbook-md-shows-that-long-policy-documents-do-not-reliably-govern-agents.md", "text": "https://wpnews.pro/news/handbook-md-shows-that-long-policy-documents-do-not-reliably-govern-agents.txt", "jsonld": "https://wpnews.pro/news/handbook-md-shows-that-long-policy-documents-do-not-reliably-govern-agents.jsonld"}}