# You chose the best model. Why is your agent still failing? > Source: > Published: 2026-08-12 15:00:15+00:00 *Co-Authored by Aparna Dhinakaran, Co-founder & Chief Product Officer, Arize AI & Prukalpa Sankar, Co-founder, Atlan.* ** Evals have become core infrastructure faster than almost anything we have seen in AI**. At Arize, we see our customers run more than 100 million evals every month. A typical team operates roughly a dozen [evaluation jobs](https://arize.com/resources/llm-evaluation/), while the most active organizations run thousands. At Atlan, we see the other side of the same shift as enterprises work to give agents governed, current business context. Together, we think context and evaluation are becoming the operating layer for reliable AI. Yet while teams built those [evaluation suites](https://arize.com/resources/llm-evaluation/), the systems being evaluated changed underneath them. Many early production AI applications made a single model call and returned a single answer. Today, agents retrieve data, call tools, execute code, manage permissions, use memory, delegate to [subagents](https://arize.com/resources/ai-agent-evaluation/), and work across long [trajectories](https://arize.com/resources/agent-harness-evaluation-tracing/). Each capability expands what an agent can do, while also creating more places where the system can fail. This is why [context engineering](https://arize.com/resources/what-is-ai-engineering/) and [harness engineering](https://arize.com/resources/agent-harness-evaluation-tracing/) have become two of the most important disciplines in AI development. The context layer determines what the agent knows about the business, including its definitions, policies, trusted sources, and operating norms. [The harness](https://arize.com/resources/agent-harness-evaluation-tracing/) determines how the agent uses that knowledge across retrieval, tools, code, memory, permissions, and handoffs. Together, they shape the experience and reliability of the agent in ways that model choice alone cannot explain. **The three layers of an enterprise AI system** From our work across Atlan and Arize, we see the same three-layer architecture in enterprise AI systems. The model provides reasoning. The context layer supplies the business meaning, policies, and trusted knowledge the model needs. [The harness](https://arize.com/resources/agent-harness-evaluation-tracing/) turns that reasoning and context into actions across retrieval, tools, code, memory, permissions, and handoffs. Evaluation sits across all three layers so teams can locate failures and use what they learn to improve the system. Each layer receives a different level of scrutiny. Frontier labs and the broader research community evaluate models against public benchmarks, while every company remains responsible for the context and harness inside its own environment. Those layers contain your definitions, data, trust signals, workflows, permissions, tools, and edge cases, so their quality can only be measured against your production reality. **Why model benchmarks create incomplete confidence** Public benchmarks answer one question: can this model, in general, do the thing? They say nothing about whether it’s right about your business or whether the system around it executes reliably, and the research shows both move while the model stays fixed. Several research findings illustrate the gap:[ Liu et al. (2024)](https://arxiv.org/abs/2307.03172) showed that answers vary with the position of relevant information in the input. A [ 2025 study](https://arxiv.org/abs/2510.12460) found that conflicting context reduced LLaMA-3.1-8B accuracy by 31 percent.[ SWE-agent](https://arxiv.org/abs/2405.15793) improved resolution rates on real GitHub issues through a purpose-built agent-computer interface, and the [ MAST taxonomy](https://arxiv.org/abs/2503.13657) documented how system design contributes to multi-agent failures. Together, these findings show that performance can move substantially even when the underlying model remains fixed. When a strong model receives stale, incomplete, or conflicting context, it can reason coherently and still produce the wrong answer. The output may be fluent enough to look trustworthy, which makes the failure difficult to catch. We think of this pattern as confident irrelevance: the model applies real intelligence to the wrong information. **The context layer determines what the agent knows** Before a model can reason about a specific problem set, it needs a reliable [ context layer](https://atlan.com/know/enterprise-context-layer/). That layer provides governed definitions, relationships among data assets, operating procedures, approval paths, permissions, and trust signals that tell the agent how the business works. Atlan focuses on making that context machine-readable and portable so every agent can reason from the same version of the business. In our experience, an effective context layer contains three parts, each answering a question the agent has to resolve before it can act: **AI-ready data.** What data and assets can the agent trust? An enriched graph of the estate, with lineage showing where each asset came from, what depends on it, and which sources are certified.**Semantics & ontology.** What do things in this business mean, and how do they connect? The definitions of a customer, product, or metric, and the relationships among those concepts.**Skills.** How does work actually get done here, and what’s allowed? The procedures, decision logic, permissions, and approval paths that determine which actions are acceptable. When any of these parts is incomplete, the model has to resolve ambiguity from whatever information it can find. That can produce an authoritative-sounding answer based on the wrong definition, an outdated rule, or a source that should have been deprecated. **Core capabilities of a context platform** Knowing what belongs in the context layer is different from operating one. Definitions change, policies update, and new agents arrive on new platforms every quarter, so the context layer needs to be produced, governed, and delivered continuously rather than written once. From Atlan’s work as an enterprise context layer, five capabilities do that work: **Context mining.** Most business context already exists in SQL, pipeline code, BI models, documents, and query history.[Atlan’s Context Agents](https://atlan.com/context-agents/), connectors, and column-level lineage reverse-engineer context from those systems, so teams start from what the business already knows instead of writing definitions from scratch.**Context development lifecycle.** Context changes whenever the business does, so it needs the same discipline as code: build, test, review, approve, deploy, learn. In[Atlan’s Context Engineering Studio](https://atlan.com/context-engineering-studio/), AI drafts candidate context and tests it against historical[traces](https://arize.com/resources/ai-agent-tracing-evaluation/), and humans certify what becomes canonical.**Context governance and observability.** Every definition carries an owner, a version, an approval trail, and trust signals such as whether an asset is certified or deprecated. Context drift deserves the same operational discipline as model or data drift, with clear owners, thresholds, and alerts, so a wrong answer can be[trace](https://arize.com/resources/ai-agent-tracing-evaluation/)d back to the context that shaped it.**Context activation and retrieval.** Agents live on many platforms, so context has to reach all of them. Atlan delivers the same governed context through MCP, SQL, APIs, and graph interfaces, so every agent reasons from the same version of the business.**Compounding learning loops.**[Evals](https://arize.com/resources/llm-evaluation/), traces, and corrections from every agent interaction feed back into the shared layer. Each failure a team resolves sharpens the context for every agent that comes after. Together, these capabilities make context an operating asset: versioned, testable, portable, and improving with use. **The harness turns knowledge into action** Once the context layer supplies what the agent knows, [the harness](https://arize.com/resources/agent-harness-evaluation-tracing/) turns that knowledge into action. It includes the iteration loop, context manager, retrieval logic, tool registry, memory, permission checks, code execution, [subagents](https://arize.com/resources/ai-agent-evaluation/), and handoffs that carry the system across a task. Here is where the state space starts to expand. Over the past three years, agents moved from answering prompts to calling functions, browsing the web, operating computers, writing code, and coordinating subagents over long horizons. Each new capability creates more possible [trajectories](https://arize.com/resources/agent-harness-evaluation-tracing/), which means the same application can now fail through many distinct paths. And as the systems have become more complex, their failures did, too. At Arize, we saw this directly with [Alyx](https://arize.com/docs/ax/alyx/arize-copilot) our AI engineering agent built into our platform [Arize AX](https://arize.com/resources/arize-ax-platform/). Each model upgrade introduced new capabilities alongside new failure patterns. Some runs lost important context over time, while others selected the wrong tool, failed during execution, repeated steps, recovered poorly, or reported completion before the work was actually finished. Because Alyx builds custom interfaces and follows a different [trajectory](https://arize.com/resources/agent-harness-evaluation-tracing/) for each user, a fixed output rubric could only cover part of what mattered. This means we need tools to help us and agents debug this new crop of failures faster. And it all starts with [tracing](https://arize.com/resources/opentelemetry-otel-concepts-span-trace-session/). **Tracing makes the trajectory visible** [Tracing](https://arize.com/resources/opentelemetry-otel-concepts-span-trace-session/) is the starting point for harness evaluation because it records every model call, retrieval, [tool invocation](https://arize.com/blog/how-to-evaluate-tool-calling-agents/), handoff, and result. A final answer only tells you where the run ended. A [trace](https://arize.com/resources/ai-agent-tracing-evaluation/) shows the path, which lets teams identify the step where context, routing, permissions, recovery, or execution diverged from the intended behavior. [Human review](https://arize.com/docs/ax/evaluate/human-review) remains valuable, although production volume and the sheer amount of data that’s produced makes it impossible to inspect every [trajectory](https://arize.com/resources/agent-harness-evaluation-tracing/) manually. That means teams need evaluators that can match the structure and complexity of the failure they are trying to find. **Different failures require different evaluators** Different failure modes require different evaluators. A mature [evaluation system](https://arize.com/resources/llm-evaluation/) combines several methods because each one is suited to a different level of ambiguity and complexity: Because unit tests, assertions, and rules are fast, inexpensive, and exact, they work well for checks such as valid JSON, required fields, successful status codes, permission boundaries, and PII leakage.[Code evaluators](https://arize.com/docs/ax/evaluate/create-evaluators#code-evaluators)for deterministic requirements.A calibrated judge can score properties such as faithfulness, relevance, tone, or[LLM-as-a-judge](https://arize.com/guides/llm-as-a-judge/)for known qualitative failures.[hallucination](https://arize.com/resources/llm-hallucination-dataset/)when the team already understands the failure and can describe it with a stable rubric.Another agent can review the full path, inspect evidence from each step, reason about tool choices and recovery, and determine whether the trajectory was effective even when the final answer appears plausible.[Agent-as-a-judge](https://arize.com/blog/agent-as-a-judge-evaluate-agents-with-agents/)for[trajectory](https://arize.com/resources/agent-harness-evaluation-tracing/)and harness failures. For trajectory failures, another agent is often the best evaluator. The judge can follow the journey, compare each action with the task, and explain where the run went off course. That explanation gives engineers a much better starting point than a single score because it points toward [the harness](https://arize.com/resources/agent-harness-evaluation-tracing/) component, context source, or recovery behavior that needs to change. Every evaluator still requires careful design. Code checks need meaningful assertions, [LLM judge](https://arize.com/guides/llm-as-a-judge/)s need calibration and drift monitoring, and agent judges need access to the relevant [trace](https://arize.com/resources/ai-agent-tracing-evaluation/), tools, and evidence along with clear boundaries on what they should inspect. The evaluator should match the failure instead of forcing every problem into one rubric. **Evaluation has to trigger action** A score becomes operationally useful when it starts a clear next step. A poor [trajectory](https://arize.com/resources/agent-harness-evaluation-tracing/) can route to [human review](https://arize.com/docs/ax/evaluate/human-review), become a confirmed failure, enter the [regression dataset](https://arize.com/resources/llm-evaluation/ci-cd-for-llm-apps/), inform an experiment, and shape the next version of the agent. In some workflows, the system may open a ticket or propose a pull request. The result has to move into the engineering process so the team can learn from it. This operating loop is the idea behind [Arize Signal](https://arize.com/docs/ax/evaluate/evaluators), which brings deterministic checks, [LLM judges](https://arize.com/guides/llm-as-a-judge/), and [agent judges](https://arize.com/blog/agent-as-a-judge-evaluate-agents-with-agents/) together across production [trajectories](https://arize.com/resources/agent-harness-evaluation-tracing/). It can surface failures in the path automatically, route them toward action, and help teams build a [regression suite](https://arize.com/resources/llm-evaluation/ci-cd-for-llm-apps/) from real behavior instead of relying only on curated cases from development. **Trace, evaluate, and learn from production** Production introduces edge cases that a launch-time test set cannot anticipate, so [live traces](https://arize.com/resources/online-llm-evaluations/) should expand the evaluation set continuously. Confirmed failures are especially valuable because they preserve the data, user intent, context, and harness behavior that actually occurred. Each example improves coverage for the next release and makes the learning loop more representative of the real system. **How context and evaluation work together** The context layer and the [evaluation system](https://arize.com/resources/llm-evaluation/) answer complementary questions: the context layer determines what the agent knows before it acts, and the [evals](https://arize.com/resources/llm-evaluation/) layer determines whether what it did was right. Together, they help teams distinguish failures in knowledge from failures in execution. A stale definition can send a clean [trajectory](https://arize.com/resources/agent-harness-evaluation-tracing/) in the wrong direction, and without evaluation across the path, that failure surfaces only when a user notices a wrong number. [Evals](https://arize.com/resources/llm-evaluation/) make context quality visible, and the context layer gives it a durable home. [Trace-level evaluation](https://arize.com/resources/ai-agent-tracing-evaluation/) helps teams locate where a trajectory diverged and determine whether the underlying problem came from context, retrieval, tool selection, permissions, or execution. And when the correction ships through a governed context layer, it reaches every agent at once, with lineage showing what else depended on the definition that changed. Together, the two systems close a loop that neither closes alone: [production traces](https://arize.com/resources/llm-evaluation/production-llm-evaluation/) catch the failure, context lineage helps explain it, the context lifecycle ships the fix, and the [regression suite](https://arize.com/resources/llm-evaluation/ci-cd-for-llm-apps/) verifies it stays fixed. **The advantage compounds around the model** Frontier models will continue to improve, and every team will gain access to those advances. Your context, harness, and [evaluation system](https://arize.com/resources/llm-evaluation/) develop through your own production experience. Each corrected definition improves the next query, each [trace](https://arize.com/resources/ai-agent-tracing-evaluation/)d failure expands the [regression suite](https://arize.com/resources/llm-evaluation/ci-cd-for-llm-apps/), and each evaluator makes the next deployment safer. The operating system around a team’s tenth agent should contain lessons from the first nine. That accumulation creates a durable advantage. Context becomes a form of organizational intellectual property because it captures how your business defines entities, makes decisions, and applies policies. Traces and regression cases become operational memory because they preserve what agents tried, where they failed, and how the system improved. Teams that govern context, trace [trajectories](https://arize.com/resources/agent-harness-evaluation-tracing/), evaluate the right layer, and act on failures build systems that become more reliable through use. To see how Arize instruments [the harness](https://arize.com/resources/agent-harness-evaluation-tracing/) and evaluation layers, [get in touch with the Arize team](https://arize.com/request-a-demo/). To see how Atlan delivers the context layer across enterprise agents,[ get in touch with the Atlan team](https://atlan.com/forms/talk-to-sales-contact/).