{"slug": "tracking-llm-latency-cost-how-i-instrumented-an-ai-agent-pipeline-using-and", "title": "Tracking LLM Latency & Cost: How I Instrumented an AI Agent Pipeline Using OpenTelemetry and SigNoz", "summary": "A developer instrumented a Python-based AI agent pipeline using OpenTelemetry and SigNoz to track LLM latency and cost. The setup uses explicit semantic inline tracing to capture domain-specific metrics like token counts and model names, enabling detailed performance monitoring of AI requests.", "body_md": "The Problem: The High Cost of Blind AI Requests\n\nWhen you build a standard CRUD application, performance bottlenecks are predictable: it's almost always a missing database index or a heavy payload. With AI agents and LLMs, performance is wildly unpredictable.\n\nA single user prompt can trigger multiple API calls, prompt formatting, vector database lookups (RAG), and streaming responses. If a user complains that the app is slow, you can't just check your server CPU usage. You need to know:\n\nWhich specific LLM call took the longest?\n\nDid our vector database search slow down the context retrieval?\n\nHow many tokens did that request consume (so we don't go broke)?\n\nTo answer this, I instrumented a Python-based AI agent backend using OpenTelemetry and hooked it up to SigNoz. Here is exactly how to do it.\n\nShow Your Work: The Setup & Instrumentation\n\nInstead of relying on generic auto-instrumentation that only tells you an HTTP request happened, we are going to write explicit, semantic inline tracing. This gives us deep, domain-specific insights like token counts and model names right inside our telemetry.\n\nPython\n\n`\n\nfrom opentelemetry import trace\n\nfrom opentelemetry.sdk.trace import TracerProvider\n\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter\n\nfrom opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter\n\nfrom opentelemetry.sdk.resources import Resource\n\ndef init_tracer(service_name=\"ai-agent-service\"):\n\n# Define application metadata\n\nresource = Resource.create(attributes={\"service.name\": service_name})\n\nprovider = TracerProvider(resource=resource)\n\n```\n# Configure the OTLP exporter to point to SigNoz (Default gRPC port is 4317)\notlp_exporter = OTLPSpanExporter(endpoint=\"http://localhost:4317\", insecure=True)\n\n# Add the batch processor to handle spans efficiently without blocking application logic\nprocessor = BatchSpanProcessor(otlp_exporter)\nprovider.add_span_processor(processor)\n\ntrace.set_tracer_provider(provider)\nreturn trace.get_tracer(service_name)\n```\n\ntracer = init_tracer()\n\nPython\n\nimport time\n\nfrom tracing import tracer\n\nfrom opentelemetry.trace import StatusCode\n\ndef fake_vector_db_search(query):\n\nwith tracer.start_as_current_span(\"vector_db_search\") as span:\n\nspan.set_attribute(\"db.system\", \"chromadb\")\n\nspan.set_attribute(\"db.query\", query)\n\n```\n    # Simulating vector embedding search latency\n    time.sleep(0.4) \n    return \"Context: OpenTelemetry is an open-source observability framework.\"\n```\n\ndef call_llm_agent(user_prompt):\n\n# Start the top-level parent trace\n\nwith tracer.start_as_current_span(\"ai_agent_pipeline\") as parent_span:\n\nparent_span.set_attribute(\"user.prompt_length\", len(user_prompt))\n\n```\n    # Step 1: Retrieve Context (Child Span 1)\n    context = fake_vector_db_search(user_prompt)\n\n    # Step 2: Execute LLM Call (Child Span 2)\n    with tracer.start_as_current_span(\"llm_generation\") as llm_span:\n        # Semantic attributes help us filter data in SigNoz later\n        llm_span.set_attribute(\"llm.model_name\", \"gpt-4o\")\n        llm_span.set_attribute(\"llm.temperature\", 0.7)\n\n        try:\n            # Simulating external AI API latency\n            start_time = time.time()\n            time.sleep(1.8) \n            generation_time = time.time() - start_time\n\n            # Mocking a successful API response payload\n            prompt_tokens = 120\n            completion_tokens = 250\n\n            # Capture critical domain/cost metrics\n            llm_span.set_attribute(\"llm.prompt_tokens\", prompt_tokens)\n            llm_span.set_attribute(\"llm.completion_tokens\", completion_tokens)\n            llm_span.set_attribute(\"llm.total_tokens\", prompt_tokens + completion_tokens)\n            llm_span.set_attribute(\"llm.latency_seconds\", generation_time)\n\n            llm_span.set_status(StatusCode.OK)\n            return \"AI Response: Steps completed successfully.\"\n\n        except Exception as e:\n            llm_span.record_exception(e)\n            llm_span.set_status(StatusCode.ERROR, description=str(e))\n            raise e\n```\n\nif **name** == \"**main**\":\n\nprint(\"Running instrumented AI Agent...\")\n\ncall_llm_agent(\"How do I properly monitor my pipeline?\")\n\n`\n\nTracking the Output in SigNoz\n\nOnce you spin up SigNoz via Docker and run the script above, the exact execution flow transfers from your terminal into a visual powerhouse.\n\nAnalyzing the Flame Graph\n\nWhen you open the Traces tab in SigNoz and click into ai_agent_pipeline, you instantly see the breakdown of the 2.2-second request:\n\nYou see the total ai_agent_pipeline bar spans across the entire duration.\n\nDirectly underneath it, you see vector_db_search taking exactly 400ms.\n\nImmediately following that, llm_generation takes up 1.8 seconds of the timeline.\n\nIf a request spikes to 5 seconds, you no longer have to guess. The flame graph visually exposes exactly which child span shifted right.\n\nCreating a Custom Token & Cost Dashboard\n\nBecause we explicitly attached attributes like llm.total_tokens and llm.model_name directly to our spans, we don't just get charts for network speeds—we can build business-critical dashboards.\n\nIn SigNoz, you can build a custom panel using ClickHouse queries on your span attributes:\n\nSQL\n\n`SELECT `\n\nsum(attributes_number_value[indexOf(attributes_string_key, 'llm.total_tokens')]) as total_tokens_consumed,\n\nattributes_string_value[indexOf(attributes_string_key, 'llm.model_name')] as model\n\nFROM signoz_traces.spans\n\nWHERE attributes_string_key = 'llm.model_name'\n\nGROUP BY model\n\nThis transforms raw engineering trace telemetry into an executive-level dashboard showing real-time token spend per AI model.\n\nKey Gotchas & What I Learned\n\nWriting this implementation exposed a few critical engineering details that the generic setup guides completely skip over:\n\n📌 Don't Over-Trace Async Streams: If you are streaming responses token-by-token (using stream=True in OpenAI/LangChain), wrapping the entire loop will artifically inflate your generation latency metrics because it includes the client's network read speed. Instead, explicitly measure the time from the initial request to the first byte received to calculate accurate Time-To-First-Token (TTFT).\n\n📌 Mind the Batch Processor: Always use BatchSpanProcessor in staging and production environments. Using a synchronous SimpleSpanProcessor will force your backend application to wait for the SigNoz collector to acknowledge receipt of data before responding to your user, completely destroying application performance.\n\nConclusion\n\nObservability shouldn't stop at checking if your server container is healthy. By mapping OpenTelemetry custom spans directly to our AI pipelines, we can isolate external vendor API slowdowns, track precise token resource consumption, and debug complex agent architectures with zero guesswork.\n\nTo learn more about OpenTelemetry semantic conventions for AI architectures, check out the Official OpenTelemetry Documentation.\n\nTo deploy your own observability stack locally, clone the SigNoz GitHub Repository.", "url": "https://wpnews.pro/news/tracking-llm-latency-cost-how-i-instrumented-an-ai-agent-pipeline-using-and", "canonical_source": "https://dev.to/vishesh_aggarwal_a44c2c8c/tracking-llm-latency-cost-how-i-instrumented-an-ai-agent-pipeline-using-opentelemetry-and-signoz-3mkc", "published_at": "2026-07-13 16:45:09+00:00", "updated_at": "2026-07-13 17:17:18.941648+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools", "mlops"], "entities": ["OpenTelemetry", "SigNoz", "GPT-4o", "ChromaDB"], "alternates": {"html": "https://wpnews.pro/news/tracking-llm-latency-cost-how-i-instrumented-an-ai-agent-pipeline-using-and", "markdown": "https://wpnews.pro/news/tracking-llm-latency-cost-how-i-instrumented-an-ai-agent-pipeline-using-and.md", "text": "https://wpnews.pro/news/tracking-llm-latency-cost-how-i-instrumented-an-ai-agent-pipeline-using-and.txt", "jsonld": "https://wpnews.pro/news/tracking-llm-latency-cost-how-i-instrumented-an-ai-agent-pipeline-using-and.jsonld"}}