Instrument First, Then Prompt: Finding Real Agentic Pipeline Bugs

A developer argues that most bugs in agentic pipelines stem from data shape mismatches, chunk boundary truncation, or eval dataset blindspots—not from prompt quality. The developer recommends adding lightweight instrumentation around tool calls, context injections, and router branches to catch these issues before rewriting prompts.

The default reaction when an agentic pipeline misbehaves is to open the system prompt and start rewriting it. The instinct makes sense — the prompt is visible, editable, and feels like the thing you control. The problem is that the prompt is almost never where the bug is. Three failure modes come up repeatedly in production agentic systems, and none of them are prompt problems. Chunk boundary truncation. The agent retrieved the right document and extracted the right section. But the chunk ended mid-sentence, cutting off the date field the downstream validation step required. The prompt correctly asked for the date. The agent correctly tried to provide it. The data was not there. Tool output drift. The agent correctly identified which tool to call. The tool returned a response, but the field name had changed — employee count had become headcount in a dependency update three weeks earlier. The prompt expected employee count . The model interpreted the unexpected field and got it wrong roughly half the time. Eval dataset blindspot. The agent handled the first four inputs in the eval set correctly. On the fifth, it received a malformed date string the eval set had never included. The agent guessed wrong. The eval passed. Production failed the first time a real user sent that format. None of these are visible in prompt outputs. You cannot find them by reading model responses. When I wire a new agentic pipeline, the first thing I add is not a system prompt improvement — it is instrumentation. Three pieces. A decorator around every tool call: python import time import json from functools import wraps def traced tool tool fn : @wraps tool fn def wrapper args, kwargs : start = time.time result = tool fn args, kwargs elapsed = time.time - start print json.dumps { "event": "tool call", "tool": tool fn. name , "elapsed ms": round elapsed 1000 , "result keys": list result.keys if isinstance result, dict else type result . name } return result return wrapper The result keys line catches tool output drift. If a field name changes upstream, the log shows the changed shape the next time the tool fires. A log call around every context injection: python def log context injection context items: list, step name: str : print json.dumps { "event": "context inject", "step": step name, "item count": len context items , "item lengths": len str item for item in context items , "total chars": sum len str item for item in context items } The item lengths list shows when a chunk is unusually short — which is often the symptom of a boundary cut. A 12-character chunk where you expected 800 characters tells you something different from an empty result. A counter around every router branch: python route counter: dict str, int = {} def log route branch name: str : route counter branch name = route counter.get branch name, 0 + 1 print json.dumps { "event": "route", "branch": branch name, "count": route counter branch name } None of this requires a tracing backend. Timestamped JSON lines that survive a production deploy are enough to start. grep event=tool call | jq . is sufficient for most initial debugging. Once instrumentation is in place, the loop is: That thing is almost always one of: a data shape mismatch at a tool boundary, a chunk size configuration that cuts content too early, or an eval dataset that does not cover the input format real users actually send. Rewriting the prompt to compensate for a broken interface teaches the model to paper over a structural defect. The next upstream change breaks the workaround and you debug it again. Fixing the interface eliminates the failure mode. For LangGraph pipelines, LangSmith integrates at the graph level and gives you span-level trace visualization without additional instrumentation code. Weave Weights and Biases is the alternative if you are already on the W&B stack. Both capture tool calls, model inputs and outputs, and latency across a full run. For custom pipelines outside any framework, a thin OpenTelemetry layer is roughly an hour of setup and pays for itself the first time you trace a production failure back to a specific tool call. Curious what others are using — LangSmith, Weave, custom spans — and whether prompt-first debugging is actually your default when something breaks.