Tracking LLM Latency & Cost: How I Instrumented an AI Agent Pipeline Using OpenTelemetry and SigNoz

A developer instrumented a Python-based AI agent pipeline using OpenTelemetry and SigNoz to track LLM latency and cost. The setup uses explicit semantic inline tracing to capture domain-specific metrics like token counts and model names, enabling detailed performance monitoring of AI requests.

The Problem: The High Cost of Blind AI Requests When you build a standard CRUD application, performance bottlenecks are predictable: it's almost always a missing database index or a heavy payload. With AI agents and LLMs, performance is wildly unpredictable. A single user prompt can trigger multiple API calls, prompt formatting, vector database lookups RAG , and streaming responses. If a user complains that the app is slow, you can't just check your server CPU usage. You need to know: Which specific LLM call took the longest? Did our vector database search slow down the context retrieval? How many tokens did that request consume so we don't go broke ? To answer this, I instrumented a Python-based AI agent backend using OpenTelemetry and hooked it up to SigNoz. Here is exactly how to do it. Show Your Work: The Setup & Instrumentation Instead of relying on generic auto-instrumentation that only tells you an HTTP request happened, we are going to write explicit, semantic inline tracing. This gives us deep, domain-specific insights like token counts and model names right inside our telemetry. Python from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter from opentelemetry.exporter.otlp.proto.grpc.trace exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource def init tracer service name="ai-agent-service" : Define application metadata resource = Resource.create attributes={"service.name": service name} provider = TracerProvider resource=resource Configure the OTLP exporter to point to SigNoz Default gRPC port is 4317 otlp exporter = OTLPSpanExporter endpoint="http://localhost:4317", insecure=True Add the batch processor to handle spans efficiently without blocking application logic processor = BatchSpanProcessor otlp exporter provider.add span processor processor trace.set tracer provider provider return trace.get tracer service name tracer = init tracer Python import time from tracing import tracer from opentelemetry.trace import StatusCode def fake vector db search query : with tracer.start as current span "vector db search" as span: span.set attribute "db.system", "chromadb" span.set attribute "db.query", query Simulating vector embedding search latency time.sleep 0.4 return "Context: OpenTelemetry is an open-source observability framework." def call llm agent user prompt : Start the top-level parent trace with tracer.start as current span "ai agent pipeline" as parent span: parent span.set attribute "user.prompt length", len user prompt Step 1: Retrieve Context Child Span 1 context = fake vector db search user prompt Step 2: Execute LLM Call Child Span 2 with tracer.start as current span "llm generation" as llm span: Semantic attributes help us filter data in SigNoz later llm span.set attribute "llm.model name", "gpt-4o" llm span.set attribute "llm.temperature", 0.7 try: Simulating external AI API latency start time = time.time time.sleep 1.8 generation time = time.time - start time Mocking a successful API response payload prompt tokens = 120 completion tokens = 250 Capture critical domain/cost metrics llm span.set attribute "llm.prompt tokens", prompt tokens llm span.set attribute "llm.completion tokens", completion tokens llm span.set attribute "llm.total tokens", prompt tokens + completion tokens llm span.set attribute "llm.latency seconds", generation time llm span.set status StatusCode.OK return "AI Response: Steps completed successfully." except Exception as e: llm span.record exception e llm span.set status StatusCode.ERROR, description=str e raise e if name == " main ": print "Running instrumented AI Agent..." call llm agent "How do I properly monitor my pipeline?" Tracking the Output in SigNoz Once you spin up SigNoz via Docker and run the script above, the exact execution flow transfers from your terminal into a visual powerhouse. Analyzing the Flame Graph When you open the Traces tab in SigNoz and click into ai agent pipeline, you instantly see the breakdown of the 2.2-second request: You see the total ai agent pipeline bar spans across the entire duration. Directly underneath it, you see vector db search taking exactly 400ms. Immediately following that, llm generation takes up 1.8 seconds of the timeline. If a request spikes to 5 seconds, you no longer have to guess. The flame graph visually exposes exactly which child span shifted right. Creating a Custom Token & Cost Dashboard Because we explicitly attached attributes like llm.total tokens and llm.model name directly to our spans, we don't just get charts for network speeds—we can build business-critical dashboards. In SigNoz, you can build a custom panel using ClickHouse queries on your span attributes: SQL SELECT sum attributes number value indexOf attributes string key, 'llm.total tokens' as total tokens consumed, attributes string value indexOf attributes string key, 'llm.model name' as model FROM signoz traces.spans WHERE attributes string key = 'llm.model name' GROUP BY model This transforms raw engineering trace telemetry into an executive-level dashboard showing real-time token spend per AI model. Key Gotchas & What I Learned Writing this implementation exposed a few critical engineering details that the generic setup guides completely skip over: 📌 Don't Over-Trace Async Streams: If you are streaming responses token-by-token using stream=True in OpenAI/LangChain , wrapping the entire loop will artifically inflate your generation latency metrics because it includes the client's network read speed. Instead, explicitly measure the time from the initial request to the first byte received to calculate accurate Time-To-First-Token TTFT . 📌 Mind the Batch Processor: Always use BatchSpanProcessor in staging and production environments. Using a synchronous SimpleSpanProcessor will force your backend application to wait for the SigNoz collector to acknowledge receipt of data before responding to your user, completely destroying application performance. Conclusion Observability shouldn't stop at checking if your server container is healthy. By mapping OpenTelemetry custom spans directly to our AI pipelines, we can isolate external vendor API slowdowns, track precise token resource consumption, and debug complex agent architectures with zero guesswork. To learn more about OpenTelemetry semantic conventions for AI architectures, check out the Official OpenTelemetry Documentation. To deploy your own observability stack locally, clone the SigNoz GitHub Repository.