Read the Trace, Not the Leaderboard A developer argues that agent reliability comes from harness engineering and trace observability rather than swapping to newer models, citing the Agentic Harness Engineering paper, which lifted Terminal-Bench 2 pass@1 from 69.7% to 77.0% by changing only the harness, and LangChain's independent report of a jump from 52.8 to 66.5 on Terminal-Bench 2.0. The piece outlines a minimum span-by-span trace structure aligned with OpenTelemetry GenAI semantic conventions (invoke_agent, chat, execute_tool) and notes the tooling landscape has matured enough that lack of visibility is no longer an excuse. Your agent ran for six hours, made 247 tool calls, and then failed. You have two options. Option A: swap to a newer model and hope it fails less. Option B: read the trace, find the tool call that went sideways at step 183, and fix the harness so it never happens again. Option B is boring. Option B works. 📖 Read the full version with charts and embedded sources on AgentConn → https://agentconn.com/blog/agent-observability-traces-beat-model-swaps The industry spent 2025 and early 2026 saturating the orchestration conversation. Everyone shipped a framework. Everyone argued about which model belonged in the loop. And yet the Agentic Harness Engineering paper https://arxiv.org/abs/2604.25850 demonstrated something that should have ended the debate: holding the base model fixed and iterating only on the harness lifted Terminal-Bench 2 pass@1 from 69.7% to 77.0%. LangChain reported the same pattern https://langchain.com/blog/improving-deep-agents-with-harness-engineering independently — changing only the harness, not the model, lifted their coding agent on Terminal-Bench 2.0 from 52.8 to 66.5. No model swap. No fine-tuning. Just reading traces and tightening the harness. Addy Osmani captured the convergence: the top coding agents today look more like each other than their underlying models do. The differentiator is not which frontier model sits at the center. It is the harness around it — and the observability layer that lets you iterate on that harness with evidence instead of intuition. This piece makes three arguments. First, why traces are the highest-leverage tool for agent reliability — ahead of model selection. Second, what a production trace should actually contain, span by span. Third, how the tooling landscape has matured enough that "we don't have visibility" is no longer an excuse. Here is the uncomfortable truth about model selection for agents: it matters far less than the discourse suggests. The convergence report from our latest intelligence sweep found the same signal across four independent sources — GitHub trending wall-to-wall orchestration repos , AI YouTube, Tech YouTube, and X — all saying the same thing: the gains come from harness engineering, not model shopping. The lesson from teams running agents at scale is not "use a bigger model." It is: instrument everything, read the traces, and fix what breaks. This is not a theoretical position. It is an empirical one: The pattern is consistent: the model is good enough. The harness is not. And you cannot fix the harness without traces. âš ī¸ Contrarian Corner: The counter-argument writes itself — better models make fewer mistakes, so invest in model selection instead of instrumentation overhead. This is the same argument waterfall shops made against unit tests in 2005. The issue is not whether better models help they do . The issue is that you cannot know whether a model swap helped without traces that show where failures actually occur. Model selection without observability is superstition. A trace is a nested tree of spans. The run is the root. Model calls and tool calls sit beneath it. Sub-agents are linked child traces. This is the same conceptual model as distributed tracing in microservices — and that is not a coincidence. Debugging a multi-step agent is closer to debugging a distributed system https://arize.com/blog/ai-agent-observability/ than debugging a single program. The OpenTelemetry GenAI semantic conventions https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/ define three span types for agents: invoke agent at the top, chat spans for model calls, and execute tool for tool calls. These conventions moved to their own repo semantic-conventions-genai with v1.42.0 in June 2026, and they are still marked Development — attribute names may change. But the shape is right: a tree, not a flat log. Here is what each tool-call span should record. This is the minimum viable instrumentation: Per tool call: | Field | Why | |---|---| | Tool name | Identify what ran | | Arguments sensitive fields redacted | Know what was asked | | Result status success / error / timeout | Know what happened | | Duration ms | Spot slow tools before they become blockers | | Retry count | Catch silent retry loops | | Side-effect flag | Distinguish reads from writes | | Approval status auto / human / denied | Audit the control surface | | Output identifier e.g., commit SHA, file path | Confirm what actually changed outside the agent | Per model call: | Field | Why | |---|---| | Model ID and version | Reproduce the exact behavior | | Input tokens / output tokens | Track cost per step | | Prompt hash or version | Link behavior to the prompt that caused it | | Temperature and other params | Identify non-determinism sources | | Latency time to first token, total | Distinguish model slowness from tool slowness | The table is straightforward. The discipline is not. Most teams log the model call and skip the tool call — and then wonder why they cannot explain a failure that happened between steps 40 and 60 of a 200-step run. â„šī¸ The MCP Trace Context spec: As of July 2026, the MCP specification documents W3C Trace Context propagation inside MCP request metadata. This means tool calls routed through MCP servers can carry trace context end-to-end — your agent's spans and your tool server's spans can live in the same trace tree. If you are building MCP tools, propagate the traceparent header. If you are consuming them, verify it arrives. This builds directly on our earlier coverage of agent observability as a governance layer https://agentconn.com/blog/agent-observability-usage-microsoft-claude-budget-2026 . That piece focused on cost visibility — Microsoft pulling Claude Code over budget overruns. This piece focuses on the debugging side: not "how much did it cost?" but "why did it fail?" If you have been running agents in production, you already know that most failures are not model failures. They are tool failures, context failures, and loop failures. The trace is what distinguishes between them. Tool failures are the most common. A tool times out, returns an unexpected format, or silently succeeds while actually doing nothing. Without per-tool-call spans, these look like "the agent got confused" — which is the agent equivalent of "it works on my machine." With spans, you can see: tool X returned a 200 but the response body was empty, the agent treated empty as success, and the next 15 steps built on that phantom result. Context failures happen when the agent's context window fills up and critical information gets evicted. Traces that record input token counts per model call let you see exactly when the context hit the limit and what was lost. This is especially vicious in long-running agents — the kind that run for hours or days. The failure does not announce itself. The agent just starts making worse decisions because it forgot what it was doing. Loop failures are the agent equivalent of infinite loops. The agent retries a tool, the tool fails, the agent retries with slightly different arguments, the tool fails again. Without a retry count in the trace, this burns tokens silently for dozens of iterations before anyone notices. The eval-gaming monitoring piece https://agentconn.com/blog/eval-gaming-agent-monitoring we published in October covers a related pathology: agents that report success while actually failing. Traces are the antidote to self-reported status. Sentry's Agent Tracing https://www.youtube.com/watch?v=2FKg1rJpZk4 product shows the full execution path of an AI agent, including every tool call and token count. This is what the debugging workflow looks like in practice: not reading terminal output for twenty minutes, but filtering a trace tree to the first red span. The observability tooling for agents has matured considerably over the past year. Here is a landscape snapshot — not a ranking, because the right tool depends on your stack. Open-source / self-hosted: Managed / vendor: The right choice depends on two questions: 1 Are you on LangChain? If yes, start with LangSmith. 2 Do you need to self-host? If yes, Langfuse. If neither constraint applies, evaluate based on how well the tool integrates with your existing observability stack — the goal is traces that join the same system your infrastructure already lives in, not a parallel universe for AI. 💡 The OpenTelemetry bet: Every tool on this list either ingests or emits OpenTelemetry traces. The GenAI semantic conventions are still experimental, but they have already become the shared vocabulary. If you instrument once to OTel, you can switch tools later without re-instrumenting. This is the strongest argument against vendor lock-in in the agent observability space. Collecting traces is necessary but not sufficient. The payoff comes from turning trace data into harness changes — and that feedback loop is what separates teams that improve from teams that just watch. The Focused Labs writeup https://dev.to/focused dot io/agent-traces-rewrite-the-harness-focused-labs-24jg captures this directly: "Traces only matter when they change the next harness release." They describe a LangChain deployment where the team held the model fixed and changed only the harness — tool descriptions, middleware around tool calls, retry policies, context management — based on what the traces revealed. Each harness iteration was driven by a specific trace pattern, not by intuition. Martin Fowler and Birgitta Boeckeler's harness engineering guide https://www.martinfowler.com/articles/harness-engineering.html formalizes this as two feedback mechanisms: feedforward controls that prevent known failure modes informed by past trace patterns , and feedback controls that evaluate what happened after the agent acted informed by current traces . The trace is the evidence layer for both. The AHE paper's https://arxiv.org/abs/2604.25850 automated version of this loop is the logical endpoint: the system reads its own multi-million-token trajectories, identifies failure patterns, and proposes harness edits. Ten iterations. No human in the loop. 7.3 percentage points of improvement. The frozen harness even transferred to SWE-bench-verified, using 12% fewer tokens than the seed harness. This is what the harness-as-moat thesis https://agentconn.com/blog/harness-bottleneck-not-model looks like in practice. The moat is not the model. The moat is not even the harness. The moat is the feedback loop from traces to harness changes — the ability to learn from every production run and ship a better harness before the next one starts. If you are building agents today, here is the practical checklist: Instrument tool calls, not just model calls. Most teams log the LLM interaction and skip everything else. The tool call is where the agent meets reality — and where most failures actually originate. Log the full table from Section 2. Use OpenTelemetry's GenAI conventions. They are experimental, but they are the emerging standard. Instrument once; switch vendors later. Record trace context in MCP calls. The W3C traceparent header is now part of the MCP spec. If you are building or consuming MCP tools, propagate it. Build a trace-to-test pipeline. When a production trace shows a failure, capture it as a regression test. LangChain recommends this explicitly https://www.langchain.com/resources/agent-observability : traces that represent real production behavior become the test suite for the next harness version. Stop model-shopping before you have traces. If you cannot point to the specific tool call or context state that caused a failure, switching models is a lottery ticket, not an engineering decision. This echoes our earlier take in Stop Chasing Models, Fix Your Harness https://agentconn.com/blog/stop-chasing-models-fix-your-harness-2026 . Budget for trace storage. Long-running agents generate large traces. Sampling strategies and retention policies are not premature optimization — they are table stakes for agents that run for hours or days. The agent observability stack is no longer a gap — it is a choice. Langfuse, LangSmith, Arize Phoenix, Sentry, Laminar, and Braintrust all ship production-grade tracing. OpenTelemetry gives you a vendor-neutral instrumentation layer. The GenAI semantic conventions give you a shared vocabulary. The papers give you the evidence: harness iteration driven by traces outperforms model swaps by a wide margin. The only question left is whether you are reading your traces or still reading the leaderboard. If you are deploying agents that run for hours or manage real infrastructure, you might also want our coverage of how agents fail on real jobs https://agentconn.com/blog/ai-agents-fail-real-jobs-reliability-2026 , the harness guardrail stack https://agentconn.com/blog/agent-harness-not-model-guardrail-stack-2026 , and fleet orchestration patterns https://agentconn.com/blog/agent-of-agents-fleet-orchestration-background-agents-2026 . Originally published at AgentConn https://agentconn.com/blog/agent-observability-traces-beat-model-swaps