{"slug": "debugging-llm-calls-how-to-trace-what-your-ai-agent-actually-did", "title": "Debugging LLM Calls: How to Trace What Your AI Agent Actually Did", "summary": "A developer building a World Cup chatbot found that when the model gave a wrong answer, there was no way to trace why it responded that way. To solve this, they turned to OpenTelemetry, which automatically logs every LLM call's prompt, response, timing, and cost. The developer notes that adding instrumentation takes under 30 minutes and recommends tools like Arize Phoenix and Enprompta for hands-on exploration.", "body_md": "So I was messing around with a little side project, a chatbot that answers questions about the World Cup. Fun, low stakes, nobody's paycheck depends on it.\n\nSomeone asked it \"has the tournament ended yet?\"\n\nIt said no. Confidently. Wrong. It had actually ended weeks earlier.\n\nFine, models get things wrong sometimes, that's not exactly breaking news. What actually got me was the next question: why? Did it even try to check? Did it just guess from something it remembered? I genuinely had no way to find out. The model doesn't keep a diary. It answers, and that's it, the reasoning just evaporates.\n\nAnd that's true whether you're building a hobby project or something with real users. No logs, no \"here's exactly what I looked at before I answered,\" nothing. You're stuck debugging a black box with screenshots and vibes.\n\nI'm sure a chunk of you already know exactly where this is going: OpenTelemetry. If that's you, feel free to skip ahead.\n\nBut if you're more in the \"yeah I've heard the name, never actually looked into it\" camp, here's the short version: it's basically a black box flight recorder for your AI calls. Every time your code asks a model something, it quietly writes down what was asked, what came back, how long it took, and what it cost.\n\n``` python\nfrom openinference.instrumentation.anthropic import AnthropicInstrumentor\n\nAnthropicInstrumentor().instrument(tracer_provider=provider)\n\n# Nothing else changes below. This call is now traced automatically.\nresponse = client.messages.create(model=model_name, messages=messages)\n```\n\nA full record shows up right after: the prompt, the answer, the timing, the cost. First time you see it, it's a genuinely satisfying \"oh, THAT'S what happened\" moment.\n\n**What you need:** whatever API key you already use for your model, and somewhere to send the data to, several solid free options exist.\n\n**Time to get this working:** under 30 minutes, most of it is pasting two lines into code you've already written.\n\nSome existing tools in this space, worth knowing the names even if you don't need all of them: LangSmith, Langfuse, Helicone, PromptLayer, Braintrust, and Arize Phoenix (free and open source if you want to self-host).\n\nIf you just want to poke around and get a feel for this without committing to anything, [Enprompta](https://enprompta.com) is a solid, beginner-friendly place to start.\n\nThere's even a small public sample project set up specifically for trying this out hands-on:\n\nA lightweight assistant that answers questions about the 2026 FIFA World Cup (USA / Canada / Mexico, June 11 – July 19, 2026). It calls Claude with the built-in web-search tool, so answers can reflect live results and fixtures.\n\nComes in two forms — a web app and a command-line tool — sharing one prompt and one dependency.\n\nThe app is intentionally left uninstrumented; adding instrumentation is a student exercise.\n\nWant to see exactly what instrumentation adds? Open the quickstart notebook in Colab — it clones this app, wires OpenTelemetry tracing, a versioned prompt, and evals around the single LLM call site, then streams the traces to your Enprompta project.\n\nOnce you've traced it, learn to **evaluate** it. This app is the running example in\nEnprompta's free course on agent evaluation — you'll score it across…\n\nClone it, wire up tracing, and go find your own version of \"wait, why did it say that.\"", "url": "https://wpnews.pro/news/debugging-llm-calls-how-to-trace-what-your-ai-agent-actually-did", "canonical_source": "https://dev.to/john_10/debugging-llm-calls-how-to-trace-what-your-ai-agent-actually-did-43j2", "published_at": "2026-07-30 10:07:36+00:00", "updated_at": "2026-07-30 10:33:14.842667+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-agents"], "entities": ["OpenTelemetry", "Anthropic", "LangSmith", "Langfuse", "Helicone", "PromptLayer", "Braintrust", "Arize Phoenix"], "alternates": {"html": "https://wpnews.pro/news/debugging-llm-calls-how-to-trace-what-your-ai-agent-actually-did", "markdown": "https://wpnews.pro/news/debugging-llm-calls-how-to-trace-what-your-ai-agent-actually-did.md", "text": "https://wpnews.pro/news/debugging-llm-calls-how-to-trace-what-your-ai-agent-actually-did.txt", "jsonld": "https://wpnews.pro/news/debugging-llm-calls-how-to-trace-what-your-ai-agent-actually-did.jsonld"}}