I’ve been talking quite a bit lately about context graphs and the role they might play in building AI agents. The phrase can sound a little abstract, so I have been looking for examples that make the architectural problem concrete. A recent research paper called HANDBOOK.md does exactly that. It tests what happens when you give an agent a company handbook and expect it to follow that handbook while doing real work. The paper accurately describes this as a benchmark for long-context agentic instruction following. It also points to a deeper problem. If the rules that constrain an agent live entirely in a prompt, then their enforcement depends on the agent successfully remembering, and honoring those rules throughout a long sequence of work. That is a lot to ask of a probabilistic system.
If you prefer audio/video, I made a YouTube version of this as well. Moving on!
The researchers created ten fictional companies in finance, medical billing, insurance, logistics, and HR. Each company had its own handbook, ranging from about 20 pages to 124 pages, and the agents had access to realistic business systems for email, chat, calendars, issue tracking, commerce, and files. The tasks were not simple retrieval questions. An average run required roughly 30 tool calls, and the agent might have to inspect information across several systems before it could decide what to do. Across the benchmark, there were 65 tasks and 82 available tools. The grading was strict in a way that feels appropriate for this kind of work: a trial passed only when the agent completed every required action and avoided every prohibited one. The best of the 30 configurations tested passed 36.2% of trials, while most frontier configurations remained below 25%.
That number is more interesting when you look at what happens if the grading standard becomes slightly more forgiving. The researchers found that allowing an agent to miss one grading criterion roughly doubled the success rate for leading models. It would be easy to look at that and conclude that agents are "nearly there," because in many cases they probably did complete the broad shape of the task. But enterprise policy often exists in the details. The single missed criterion may be the condition that determines whether the entire action is safe to take. A system that is mostly compliant in a normal workday can still create a very serious problem. And unless something fundamentally changes about the transformer architecture, I don't buy the notion that success rates will approach 100% in this domain.
The failure modes in HANDBOOK.md are familiar to anyone who has spent time working with agents. In some cases, a plausible request from a user displaced the standing policy in the handbook. In others, the agent completed a required check, received a result that should have stopped the process, and continued anyway. Some agents skipped verification, assumed the condition had been satisfied, and then reported that they had completed the task according to policy. This is the part that should make us cautious about treating a handbook in context as an enforcement mechanism. The model may be able to find the relevant paragraph. It may even explain the policy back to us in convincing detail. None of that means the policy has authority over the tools and systems the model can use.
You can’t prompt your way into policy.
Putting policy inside plain-language instructions gives the model a lot of work to do before it can safely act. Furthermore, other prompts in the context window may be pushing in a different direction. A user may be asking for a refund, an override, an exception, or a shortcut, and the model has to decide that the standing policy is more important than the locally "persuasive" request in front of it. Better prompts and better reasoning models will improve this behavior, but they do not change the fact that the enforcement process is happening inside a system designed to produce plausible continuations.
There are many tasks where that is perfectly fine. An agent can summarize, explain, identify missing information, prepare a request, or help sequence steps in a workflow. Those are valuable uses of language models. The difficulty begins when the agent can also read sensitive data and make consequential changes in business systems. At that point, the policy needs to exist somewhere that can actually prevent an operation from happening. I do not want an agent’s authority to issue a refund or reveal a sensitive field to depend entirely on whether it remembered the right paragraph at tool call number 27.
Tokens are best spent on reasoning, not compliance. That is where a context graph becomes useful.
What is a context graph? #
There are a few products that use the term "context graph" but none of them are convincing IMO. In my view, a context graph starts with a basic problem: an agent cannot do useful work from a pile of disconnected facts. It needs a coherent model of the business domain that tells it what things exist, how they relate, what information is available, and which actions can be requested. A customer has accounts, an account has orders, an order may be eligible for a refund, and a refund may depend on the caller’s identity or a manager’s approval. The “context” is the subset of that domain the agent needs for the task in front of it. The “graph” is the connected, strongly typed model that makes those relationships and capabilities explicit. Governance belongs in the access path so the system can determine which fields and operations are available before their results enter the model’s context or affect the business.
GraphQL and Federation provide a practical foundation for building that layer. GraphQL gives the organization a strongly typed schema for describing its domain and a declarative operation language for requesting a precise shape of data. Federation allows teams to contribute the parts of that schema they own (customers, orders, approvals, refunds) from separate services while presenting consumers with one composed graph. GraphQL does not define an authorization system by itself, so the production graph still needs identity, field- and operation-level controls, persisted operations, demand controls, and observability. Together, those capabilities turn the federated graph into a governed interface through which an agent can retrieve context and request actions without requiring every tool to reconstruct the company’s domain model and policy rules independently. A context graph.
A context graph should give an agent a structured way to ask questions about the business and to request actions, while also providing a place to evaluate the conditions under which those reads and actions are allowed. The word “graph” matters because policy is usually relational. Those relationships are part of the context the agent needs to do useful work, and they are also the facts that determine what the agent is allowed to see or do.
Imagine a policy that allows an agent to issue refunds below $500, while larger refunds require manager approval. If the policy lives only in a handbook, the agent needs to find that rule, inspect the order amount, determine whether the order is eligible, locate an approval if one is required, interpret whether the approval is valid, and then carry all of that reasoning forward when it invokes the refund tool. An agent can certainly help with this process. It can determine that a refund appears appropriate, explain the case to a manager, collect the required information, and prepare the request. The final operation, however, can evaluate the relevant conditions directly. It can check the amount, the current order state, the identity and scope of the caller, and whether a valid approval exists. If the conditions are not met, the refund is rejected. The agent can continue helping the user, but it cannot talk its way around the rule.
<!-- From the handbook (p. 47) -->
Refunds under $500 may be processed without additional
approval. Refunds of $500 or more require sign-off
from an account manager prior to processing.
type Mutation {
issueRefund(
orderId: ID!
amount: Float!
): RefundResult @requiresApproval(above: 500, role: MANAGER)
}
Platform effects #
This approach applies to more than refund thresholds. Field-level authorization can determine which information is available to the agent in the first place. If the agent has no permission to access a customer’s Social Security number, that field never needs to enter the model’s context. Operation-level authorization can determine whether an agent may send a message, change an account, issue a credit, or create a case. Preconditions can connect those operations to business facts such as account status, transaction limits, approvals, contractual terms, or geographic requirements. The model still receives the context it needs to reason about the task, but the control is evaluated at the point where information is retrieved or an action is attempted.
This centralization becomes more important as the number of tools grows. HANDBOOK.md exposes agents to 82 tools; that is not an unusual direction for enterprise agent environments. If every tool independently implements its own partial understanding of an organization’s policy, identity model, and approval requirements, the system will drift. The definition of “manager” may differ from tool to tool. Magic numbers may litter the codebase. Auth rules may be accidentally duplicated and get out-of-sync. A context graph should offer a shared layer where those concepts can have consistent meaning across the agent’s environment, and where teams can observe what an agent requested, what it was allowed to access, what action it attempted, and why a policy check permitted or rejected it.
There are real limits to this argument. A context graph will not convert every sentence in a 100-page handbook into a simple authorization rule. Some policies require judgment, require human review, and/or need to be enforced by an approval service. The graph also has to be an authoritative path to the relevant capabilities. If an agent can bypass it through another tool that writes directly to a database or calls an unconstrained internal API, the control is only as strong as the least governed path available to the agent. These are design requirements, not reasons to resort to prompt-as-policy.
The useful distinction is between rules that inform reasoning and rules that constrain authority. A handbook may contain both. An agent can read a long document to understand how an organization prefers to handle an unusual customer request. It should not be the sole decision-maker about whether it can read or write sensitive information. Access rules, transaction limits, valid operations, field visibility, and scope boundaries should survive a prompt revision, a model upgrade, a context-compression choice, and a user who asks the agent nicely (or insists rudely) to ignore them.
Better models and prompts may incrementally improve the results on benchmarks like HANDBOOK.md. But for a broad set of use cases, you want to put key controls inside of tool calls, not inference calls. A context graph is one way to build that foundation: give the model the governed context it needs to reason about the business, and give the surrounding systems the authority to decide what may happen next.