How I Traced My AI Agent's Decision Loop With OpenTelemetry and Signoz (and Caught It Calling the Same Tool Twice)

A developer traced an AI agent's decision loop using OpenTelemetry and SigNoz, catching the model calling the same tool with identical input three times in a row. The agent, built with a ReAct-style loop, showed hidden inefficiencies that only appeared in the trace, not in the final JSON response.

From a single JSON response to a full trace of every decision my AI agent made I built a small AI agent, it answers questions by deciding on its own whether to use a calculator, search some notes, or check the time. The answers looked fine, but one query took eight seconds and another needed four tries to answer something a single tool call should've settled. All I had was a JSON response and a stopwatch. So I traced the whole decision loop with OpenTelemetry and SigNoz. Not just the request, every step inside it. In 30 minutes I could see exactly which iteration was slow, and caught the model calling my calculator with the same input three times in a row. Here's what I did. A ReAct-style loop: ask the model what to do, run a tool if it asks for one, feed the result back, repeat. php def run agent query: str - dict: transcript = f"User question: {query}" for iteration in range 1, MAX ITERATIONS + 1 : prompt = SYSTEM INSTRUCTIONS + "\n\nConversation so far:\n" + "\n".join transcript action = parse action call claude api prompt, iteration if action "action" == "final answer": return {"answer": action "answer" } result = TOOLS action "tool" action "input" transcript.append f"Called {action 'tool' } {action 'input' r} - {result}" pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-bootstrap -a install export OTEL RESOURCE ATTRIBUTES="service.name=ai-agent-demo" export OTEL EXPORTER OTLP ENDPOINT="http://localhost:4317" export OTEL EXPORTER OTLP PROTOCOL="grpc" opentelemetry-instrument uvicorn app:app --port 8000 PEP 668. opentelemetry-bootstrap failed with externally-managed-environment . It calls pip install internally, so --break-system-packages on the outer command does nothing. The real fix: pip config set global.break-system-packages true Wrong model name. I'm on OpenRouter, and claude-3-5-sonnet-20241022 404'd, my account only had poolside/laguna-m.1:free . ThinkingBlock has no .text. Some models return a reasoning block before the answer block. Fix: loop through content and grab the first block that actually has .text . SigNoz Traces tab, filtered to service.name = ai-agent-demo — several traces, visibly different durations Auto-instrumentation covers FastAPI and outbound HTTP calls for free. The decision logic, which tool, how many iterations, so I wrapped it manually: with tracer.start as current span "agent.llm call" as span: span.set attribute "agent.iteration", iteration span.set attribute "llm.duration ms", elapsed ms with tracer.start as current span f"agent.tool call.{tool name}" as span: span.set attribute "tool.input", str tool input :300 span.set attribute "tool.output", str result :300 Spans nest by where you open them, so one request produces one root span with however many LLM/tool spans that specific query actually needed. "47 × 12, plus 8" 4 iterations. Called the calculator, got 572 right on the first try, then called it two more times with the identical input before finally answering. Correct the whole time. You'd never see this in the response, only in the trace. Waterfall: three agent.tool call.calculate spans, identical input, between LLM calls "What's our cache TTL?" — the model sent the wrong action shape twice, got nudged, corrected itself on try three, answered on try four. Waterfall: two invalid attempts, then the successful search notes call Click into any span and you get the attributes: iteration number, duration, prompt size, tool input/output. "Iteration 3 took 2.4s, prompt was 1,900 chars, called search notes" that's a diagnosis. "8 seconds total" is just a number. Attributes panel for a single agent.llm call span The payoff is bigger: an agent's interesting behavior lives entirely inside the request, and a trace is the only place it's visible.