From a single JSON response to a full trace of every decision my AI agent made
I built a small AI agent, it answers questions by deciding on its own whether to use a calculator, search some notes, or check the time. The answers looked fine, but one query took eight seconds and another needed four tries to answer something a single tool call should've settled. All I had was a JSON response and a stopwatch. So I traced the whole decision loop with OpenTelemetry and SigNoz. Not just the request, every step inside it. In 30 minutes I could see exactly which iteration was slow, and caught the model calling my calculator with the same input three times in a row. Here's what I did.
A ReAct-style loop: ask the model what to do, run a tool if it asks for one, feed the result back, repeat.
def run_agent(query: str) -> dict:
transcript = [f"User question: {query}"]
for iteration in range(1, MAX_ITERATIONS + 1):
prompt = SYSTEM_INSTRUCTIONS + "\n\nConversation so far:\n" + "\n".join(transcript)
action = _parse_action(_call_claude_api(prompt, iteration))
if action["action"] == "final_answer":
return {"answer": action["answer"]}
result = TOOLS[action["tool"]](action["input"])
transcript.append(f"Called {action['tool']}({action['input']!r}) -> {result}")
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
export OTEL_RESOURCE_ATTRIBUTES="service.name=ai-agent-demo"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
opentelemetry-instrument uvicorn app:app --port 8000
PEP 668. opentelemetry-bootstrap
failed with externally-managed-environment
. It calls pip install
internally, so --break-system-packages
on the outer command does nothing. The real fix:
pip config set global.break-system-packages true
Wrong model name. I'm on OpenRouter, and claude-3-5-sonnet-20241022
404'd, my account only had poolside/laguna-m.1:free
.
** ThinkingBlock has no .text.** Some models return a reasoning block before the answer block. Fix: loop through content and grab the first block that actually has
.text
.
(SigNoz Traces tab, filtered to service.name = ai-agent-demo — several traces, visibly different durations)
Auto-instrumentation covers FastAPI and outbound HTTP calls for free. The decision logic, which tool, how many iterations, so I wrapped it manually:
with tracer.start_as_current_span("agent.llm_call") as span:
span.set_attribute("agent.iteration", iteration)
span.set_attribute("llm.duration_ms", elapsed_ms)
with tracer.start_as_current_span(f"agent.tool_call.{tool_name}") as span:
span.set_attribute("tool.input", str(tool_input)[:300])
span.set_attribute("tool.output", str(result)[:300])
Spans nest by where you open them, so one request produces one root span with however many LLM/tool spans that specific query actually needed.
"47 × 12, plus 8" 4 iterations. Called the calculator, got 572
right on the first try, then called it two more times with the identical input before finally answering. Correct the whole time. You'd never see this in the response, only in the trace.
(Waterfall: three agent.tool_call.calculate spans, identical input, between LLM calls)
"What's our cache TTL?" — the model sent the wrong action shape twice, got nudged, corrected itself on try three, answered on try four.
(Waterfall: two invalid attempts, then the successful search_notes call)
Click into any span and you get the attributes: iteration number, duration, prompt size, tool input/output. "Iteration 3 took 2.4s, prompt was 1,900 chars, called search_notes" that's a diagnosis. "8 seconds total" is just a number.
(Attributes panel for a single agent.llm_call span)
The payoff is bigger: an agent's interesting behavior lives entirely inside the request, and a trace is the only place it's visible.