LLM Observability: Tracing, Logging, Debugging Agent Runs

A developer details how LLM observability—tracing, logging, and evals—is critical for debugging agent runs, citing a logistics customer-support agent that hallucinated tracking numbers for three weeks before detection. The post outlines a three-layer approach: tracing to reconstruct decision chains, logging for full records, and evals to measure improvements, emphasizing that traditional logging is insufficient for non-deterministic LLM systems.

Why your LLM app will fail silently, and how to see it before your customers do. Three weeks. That is how long a customer-support agent shipped confidently wrong answers for a logistics client I work with before anyone noticed. The agent's job was simple: look up a shipment's status and reply to the customer. It did this hundreds of times a day. Every request returned HTTP 200. Latency was fine. The API bill looked normal. And the agent was quietly hallucinating tracking numbers. The first clue came from a phone call. A customer in Jeddah had been told her package was "delivered" — it was not. We pulled the logs. There were logs. They said: request received, model called, response returned, 200 OK. Nothing else. No record of what the retrieval step actually returned, no record of what the prompt looked like that day, no record of which model version answered, no record of how many tokens it burned to be wrong. That is the moment I stopped thinking about LLM observability as a nice-to-have and started treating it as the difference between a working system and an expensive black box. In this article I am going to walk you through everything I now do — and you should too — to trace, log, and debug LLM and agent runs in production. Traditional observability assumes your code is deterministic. You log an error, you see the stack trace, you find the bug, you fix it. An LLM breaks that assumption in four specific ways, and each one changes what "observable" means: This is why "we log everything with our regular logging library" is not enough. You need tracing that reconstructs the chain of decisions , not a flat list of calls. When I design observability for an LLM system now, I build three layers, and they answer three different questions: Tracing answers "what happened, in what order, and how long did each step take?" A trace is a tree of spans. The root span is the request; child spans are retrieval, prompt assembly, each LLM call, each tool call. Every span carries duration, token counts, cost, model name, and the input/output that passed through it. This is what reconstructs the Jeddah incident — you can see the retrieval returned an empty chunk, the prompt went out truncated, and the model answered anyway. Logging answers "what does this record look like for later analysis?" I log the full prompt and completion for every run, the retrieved chunks with their scores and sources, tool arguments and results, and a stable run ID. This is the raw material for audits, for compliance, for reproducing a specific failure after the fact. Evals and metrics answer "is this getting better or worse?" Metrics are counters and gauges: tokens per request, latency percentiles, cost per resolved task, tool-call rate, cache hit rate. Evals are the scored test cases you run against a regression set when you change the prompt or the model. Tracing tells you what broke; evals tell you whether your fix stuck. The eval loop is where most teams quietly drop the ball, so let me be concrete about what it looks like. I keep a regression set of 50–100 real, de-identified interactions — a few per failure mode the team has hit. When I change a prompt, a retrieval strategy, or a model version, I run the set and score it with a mix of exact checks the answer contains the correct tracking number and rubric-based checks did the agent escalate when the tool returned an error . The output is a pass rate and a diff against the previous run. If the change improves the reported incidents but drops the eval score by three points, I do not ship it. That eval set is the only reason I can move fast on prompts without being scared — the trace tells me what changed, and the eval tells me whether it is okay . Let me show you what a real agent trace looks like. Take a question like "What is the status of order SH-991?" The root span splits into children, and each child carries its own numbers: request root span duration 3.1s cost $0.084 ├── auth + routing 0.4ms - ├── retrieval vector store 28ms top k=5 │ └── 1 chunk returned score 0.71 731 tokens ├── prompt assembly 2ms prompt=2,104 tokens ├── llm call 1 1.9s input 2,835 / output 142 │ └── decision: call tool lookup tracking ├── tool: lookup tracking "SH-991" 120ms 400 error: not found ├── llm call 2 860ms input 3,112 / output 89 └── final answer 3,412 tokens total That one trace told me exactly what went wrong in the Jeddah case: retrieval returned a single low-confidence chunk, the tracking tool returned a hard error, and the model answered anyway instead of asking for a corrected reference number. Three spans, three failure points, one trace. Without it I would have spent days guessing. The ecosystem has settled on this shape. OpenTelemetry defined semantic conventions for generative AI the gen ai. attribute family — gen ai.request.model , gen ai.usage.input tokens , gen ai.usage.output tokens so traces from different providers and frameworks can be correlated in one tool. On top of the OTLP transport, you get purpose-built backends — I have used LangSmith and Langfuse, and both will happily ingest OTLP traces now. You do not need to pick between a tracing backend and an APM; they speak the same wire format. Here is the concrete capture contract I use. If a run produces these fields, I can debug any incident in under an hour: I write this to a Postgres table with a JSONB column for the trace plus a few indexed columns for querying, and I ship the same data to a tracing backend for the visual waterfall view. The table is the audit trail; the backend is the debugger. Here is a minimal agent loop with tracing bolted on — no framework, just the OpenTelemetry SDK and a couple of spans. This is the smallest thing I would actually deploy: python import json from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider trace.set tracer provider TracerProvider tracer = trace.get tracer "agent.observability" def run agent goal: str, retriever, llm, tools, max steps: int = 5 : with tracer.start as current span "agent.run" as root: root.set attribute "goal", goal context = for step in range max steps : with tracer.start as current span f"step.{step}" as step span: with tracer.start as current span "retrieve" as ret span: chunks = retriever goal ret span.set attribute "chunks.count", len chunks ret span.set attribute "chunks.scores", json.dumps c.score for c in chunks with tracer.start as current span "llm.call" as llm span: llm span.set attribute "gen ai.request.model", "your-model" decision = llm.decide context, chunks, tools llm span.set attribute "gen ai.usage.input tokens", decision.input tokens llm span.set attribute "gen ai.usage.output tokens", decision.output tokens if decision.is final: return decision.answer with tracer.start as current span "tool.call" as tool span: tool span.set attribute "tool.name", decision.tool name tool span.set attribute "tool.arguments", json.dumps decision.arguments result = tools<a href=" decision.arguments" decision.tool name</a tool span.set attribute "tool.result", json.dumps result context.append decision, result raise RuntimeError "step budget exhausted" The key habit: every span carries the input and output that moved through it . A span that only records duration is a pretty waterfall with no forensic value. When the trace shows chunks.scores = 0.71 and the tool result is a hard error, the debugging is over before it starts. Once tracing was live, the failure modes stopped being mysterious and became a checklist. These are the ones I hit, in order of frequency: model version on every span, I could see the version flip exactly when accuracy dropped.I keep a running rule now: if a production incident takes more than an hour to explain, the tracing is insufficient. Not the model, not the prompt — the tracing. Honest section, because not everything needs a tracing stack. If your use case is a stateless, single-shot LLM call — a summarizer, a classifier, a translation step with no tools and no retrieval — then a plain log of prompt, completion, model, and token count is 90% of the value at 10% of the setup cost. Ship that first. Similarly, if you are prototyping and have fewer than a few hundred calls a day, the tracing backend is overkill. Log to a JSON file. Add the full stack when you hit real users, real money, or a multi-step agent loop. The rule of thumb: one LLM call, no tools, no state → simple logging. A loop, tools, retrieval, or autonomy → tracing, non-negotiable. There is also a spectrum between the two extremes, and most teams are on it. If you are at the "loop with tools" stage but not yet at "needs alerting," start with the logging contract and the eval set, and defer the backend until the trace viewer is actually going to save you time. The mistake I see is the reverse: teams buy the expensive backend first and never build the logging contract, so they have a beautiful waterfall view of runs that do not contain the one field that would explain the incident. The data comes first. The tool is the last mile. The final layer is alerting, and it is the one that turns observability from a postmortem tool into a prevention tool. I set alerts on the metrics that predict incidents before customers feel them: Alert thresholds have to be tuned to each system, but the principle is universal: alert on drift from the system's own baseline , not on absolute numbers. A 500-millisecond latency spike means nothing for a batch summarizer and everything for a chat product. Your historical traces are the baseline; the alert just detects the divergence. Before you call an LLM system production-ready, go through this list: run id correlated to the user sessionAfter the Jeddah incident we rebuilt the agent's tracing from scratch, added a guard that asks the customer to re-confirm the reference number when retrieval returns nothing, and put the whole thing behind the observability stack I described. Two months later, the client asked what the new dashboards cost. When I told him, he laughed and said the alternative — three more weeks of confident wrong answers — would have cost his dispatch team more than that in a single day. Your LLM will fail silently. It is not a question of whether; it is a question of how long you do not know about it. Tracing, logging, and evals are the difference between finding out in hours and finding out from an angry customer. If you are starting today, do not buy a tool. Add the run ID, log the prompt and completion, and put token counts on every call. That is the whole foundation. Everything else — the backends, the dashboards, the alerting — is polish on top of a habit you have to build first. Gulshan Yad