{"slug": "tracing-voice-ai-is-hard-how-i-instrumented-streaming-llms-with-opentelemetry", "title": "Tracing Voice AI is Hard: How I Instrumented Streaming LLMs with OpenTelemetry and SigNoz", "summary": "A developer built Zooid, a custom OpenTelemetry instrumentation layer that traces streaming Voice AI, tracks FinOps token costs, and visualizes everything natively in SigNoz. The project addresses the challenge of tracing streaming LLMs where standard APM tools fail to pinpoint bottlenecks like STT transcription, LLM Time to First Token, or TTS synthesis.", "body_md": "Voice AI agents are magical until they start lagging. Suddenly my Groq-powered voice assistant started taking 3 seconds to respond and standard APM tools were useless. They could tell me that an HTTP request was open, but they couldn’t tell me if the bottleneck was the Speech-to-Text (STT) transcription, the LLM’s Time to First Token (TTFT), or the Text-to-Speech (TTS) synthesis.\n\nHere’s how I did this by building [Zooid](https://github.com/Priyank911/zooid), a custom OpenTelemetry instrumentation layer that traces streaming Voice AI, tracks FinOps token costs, and visualizes everything natively in SigNoz.\n\nA typical CRUD web application is easy to trace. A request comes in, hits a database, and returns a JSON response. You drop in the `opentelemetry-instrumentation-flask`\n\npackage, and you are done.\n\nVoice AI is a whole other beast. A \"Voice Turn\" consists of:\n\nThis is where the normal auto-instrumentation fails, since the LLM returns an async generator and not a static response. If you just trace the function execution, the span ends when the generator is *created*, not when the stream actually *finishes*.\n\nI needed a way to track the lifecycle of a streaming conversation manually, measure Voice specific metrics (like Time To First Audio) and get this data into SigNoz.# Step 1. Instrumenting the async stream\n\nI created a custom Python decorator (@instrument_voice_app) that manually manages the OpenTelemetry tracer to address the streaming problem.\n\nInstead of wrapping a function, it injects a `VoiceTurnContext`\n\nobject into the agent. This enables the inner AI logic to tell you exactly when specific milestones are achieved, such as when the initial audio byte is streamed back to the user.\n\nHere's the basic logic I used to follow the stream:\n\n``` python\nfrom opentelemetry import trace\nimport time\n\ntracer = trace.get_tracer(\"zooid.voice\")\n\ndef instrument_voice_app(func):\n    async def wrapper(*args, **kwargs):\n        with tracer.start_as_current_span(\"voice_turn\") as span:\n            turn_start = time.time()\n            context = VoiceTurnContext()\n\n            try:\n                # Execute the Voice Agent\n                result = await func(context, *args, **kwargs)\n\n                # Record Voice-Specific Metrics\n                if context.first_audio_time:\n                    ttfa = context.first_audio_time - turn_start\n                    span.set_attribute(\"voice.ttfa_ms\", ttfa * 1000)\n\n                if context.stt_confidence:\n                    span.set_attribute(\"stt.confidence\", context.stt_confidence)\n\n                if context.cost_usd:\n                    span.set_attribute(\"voice.turn_cost_usd\", context.cost_usd)\n\n                span.set_status(trace.StatusCode.OK)\n                return result\n\n            except Exception as e:\n                span.set_status(trace.StatusCode.ERROR, str(e))\n                span.record_exception(e)\n                raise\n    return wrapper\n```\n\nThis gave me a beautiful, continuous trace spanning the entire user interaction, but it introduced a new problem: **How do I visualize these custom attributes in SigNoz?**\n\nSending data to the OpenTelemetry Collector (localhost:4317) was quite easy. I could immediately see the traces in the SigNoz \"Traces\" tab.\n\nBut when I went to the \"Dashboards\" tab and tried to create a Time-Series panel for `voice.turn_cost_usd`\n\nand `voice.ttfa_ms`\n\n, the dashboard showed **\"No Data\"**.## The Problem with It\n\nThis is where I found a very important piece of information on how SigNoz stores OpenTelemetry attributes in ClickHouse.\n\nIn Python, when you do span.set_attribute(\"voice.ttfa_ms\", 450.5) it is sent as a Number by OpenTelemetry. This is cleanly stored in the new SigNoz trace schema (`signoz_index_v3`\n\n). However, SigNoz custom dashboard query builder depends on materialized views to map these attributes correctly to aggregate these metrics effectively.\n\n```\n-- The crucial missing lines in the Materialized View\nattributes_number as numberTagMap,\nattributes_bool as boolTagMap\n```\n\nOnce I applied this fix and backfilled the index, the query builder instantly recognized my custom metrics. I was able to query `numberTagMap.voice.turn_cost_usd`\n\nas a Number data type directly in the SigNoz UI.\n\nWith the data flowing correctly, I built a custom JSON Dashboard in SigNoz tailored entirely for Voice AI.\n\n*(If you are following along, you can import this directly into your SigNoz instance by clicking \"Import JSON\" in the Dashboards tab).*\n\nInstead of looking at generic CPU usage, my dashboard tracks:\n\n`voice.turn_cost_usd`\n\nattribute to track Groq/OpenAI token spend in real-time.`voice.interrupted`\n\nboolean attribute to see how often users cut the AI off (a great indicator of whether the bot is talking too much).*(Insert Screenshot of the SigNoz Dashboard showing the TTFA Time Series and FinOps Cost panels here)*\n\nFinally, I utilized SigNoz's Alert Rules to act like a Staff SRE. I set up an alert that triggers if the `P99 TTFA > 1000ms`\n\nfor more than 5 minutes.\n\nBecause we injected the `LLM_PROVIDER`\n\nas a span attribute (e.g., `Groq`\n\nvs `OpenAI`\n\n), the alert payload directly tells me *which* provider is lagging, allowing my team to instantly route traffic to a fallback provider.\n\n`voice.turn_cost_usd`\n\n) directly into the trace span means you can correlate slow performance directly with high inference costs.Standard APM isn't built for the generative AI era, but OpenTelemetry and SigNoz give you the primitives to build exactly what you need. By manually instrumenting our async generators and mapping custom number attributes in ClickHouse, we turned a black-box Voice AI script into a fully observable, enterprise-grade system.\n\nIf you want to try it out for your own Voice AI projects, you can grab the Python SDK from [PyPI](https://pypi.org/project/zooid/), generate a full boilerplate app via `npx create-zooid`\n\n([npm](https://www.npmjs.com/package/create-zooid)), or check out the full code on [GitHub](https://github.com/Priyank911/zooid).\n\n*Built for the WeMakeDevs & SigNoz Observability Hackathon.*", "url": "https://wpnews.pro/news/tracing-voice-ai-is-hard-how-i-instrumented-streaming-llms-with-opentelemetry", "canonical_source": "https://dev.to/jay9122/tracing-voice-ai-is-hard-how-i-instrumented-streaming-llms-with-opentelemetry-and-signoz-5c03", "published_at": "2026-07-26 18:06:00+00:00", "updated_at": "2026-07-26 18:29:17.905677+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Groq", "Zooid", "OpenTelemetry", "SigNoz", "ClickHouse"], "alternates": {"html": "https://wpnews.pro/news/tracing-voice-ai-is-hard-how-i-instrumented-streaming-llms-with-opentelemetry", "markdown": "https://wpnews.pro/news/tracing-voice-ai-is-hard-how-i-instrumented-streaming-llms-with-opentelemetry.md", "text": "https://wpnews.pro/news/tracing-voice-ai-is-hard-how-i-instrumented-streaming-llms-with-opentelemetry.txt", "jsonld": "https://wpnews.pro/news/tracing-voice-ai-is-hard-how-i-instrumented-streaming-llms-with-opentelemetry.jsonld"}}