Tracing Voice AI is Hard: How I Instrumented Streaming LLMs with OpenTelemetry and SigNoz

A developer built Zooid, a custom OpenTelemetry instrumentation layer that traces streaming Voice AI, tracks FinOps token costs, and visualizes everything natively in SigNoz. The project addresses the challenge of tracing streaming LLMs where standard APM tools fail to pinpoint bottlenecks like STT transcription, LLM Time to First Token, or TTS synthesis.

Voice AI agents are magical until they start lagging. Suddenly my Groq-powered voice assistant started taking 3 seconds to respond and standard APM tools were useless. They could tell me that an HTTP request was open, but they couldn’t tell me if the bottleneck was the Speech-to-Text STT transcription, the LLM’s Time to First Token TTFT , or the Text-to-Speech TTS synthesis. Here’s how I did this by building Zooid https://github.com/Priyank911/zooid , a custom OpenTelemetry instrumentation layer that traces streaming Voice AI, tracks FinOps token costs, and visualizes everything natively in SigNoz. A typical CRUD web application is easy to trace. A request comes in, hits a database, and returns a JSON response. You drop in the opentelemetry-instrumentation-flask package, and you are done. Voice AI is a whole other beast. A "Voice Turn" consists of: This is where the normal auto-instrumentation fails, since the LLM returns an async generator and not a static response. If you just trace the function execution, the span ends when the generator is created , not when the stream actually finishes . I needed a way to track the lifecycle of a streaming conversation manually, measure Voice specific metrics like Time To First Audio and get this data into SigNoz. Step 1. Instrumenting the async stream I created a custom Python decorator @instrument voice app that manually manages the OpenTelemetry tracer to address the streaming problem. Instead of wrapping a function, it injects a VoiceTurnContext object into the agent. This enables the inner AI logic to tell you exactly when specific milestones are achieved, such as when the initial audio byte is streamed back to the user. Here's the basic logic I used to follow the stream: python from opentelemetry import trace import time tracer = trace.get tracer "zooid.voice" def instrument voice app func : async def wrapper args, kwargs : with tracer.start as current span "voice turn" as span: turn start = time.time context = VoiceTurnContext try: Execute the Voice Agent result = await func context, args, kwargs Record Voice-Specific Metrics if context.first audio time: ttfa = context.first audio time - turn start span.set attribute "voice.ttfa ms", ttfa 1000 if context.stt confidence: span.set attribute "stt.confidence", context.stt confidence if context.cost usd: span.set attribute "voice.turn cost usd", context.cost usd span.set status trace.StatusCode.OK return result except Exception as e: span.set status trace.StatusCode.ERROR, str e span.record exception e raise return wrapper This gave me a beautiful, continuous trace spanning the entire user interaction, but it introduced a new problem: How do I visualize these custom attributes in SigNoz? Sending data to the OpenTelemetry Collector localhost:4317 was quite easy. I could immediately see the traces in the SigNoz "Traces" tab. But when I went to the "Dashboards" tab and tried to create a Time-Series panel for voice.turn cost usd and voice.ttfa ms , the dashboard showed "No Data" . The Problem with It This is where I found a very important piece of information on how SigNoz stores OpenTelemetry attributes in ClickHouse. In Python, when you do span.set attribute "voice.ttfa ms", 450.5 it is sent as a Number by OpenTelemetry. This is cleanly stored in the new SigNoz trace schema signoz index v3 . However, SigNoz custom dashboard query builder depends on materialized views to map these attributes correctly to aggregate these metrics effectively. -- The crucial missing lines in the Materialized View attributes number as numberTagMap, attributes bool as boolTagMap Once I applied this fix and backfilled the index, the query builder instantly recognized my custom metrics. I was able to query numberTagMap.voice.turn cost usd as a Number data type directly in the SigNoz UI. With the data flowing correctly, I built a custom JSON Dashboard in SigNoz tailored entirely for Voice AI. If you are following along, you can import this directly into your SigNoz instance by clicking "Import JSON" in the Dashboards tab . Instead of looking at generic CPU usage, my dashboard tracks: voice.turn cost usd attribute to track Groq/OpenAI token spend in real-time. voice.interrupted boolean attribute to see how often users cut the AI off a great indicator of whether the bot is talking too much . Insert Screenshot of the SigNoz Dashboard showing the TTFA Time Series and FinOps Cost panels here Finally, I utilized SigNoz's Alert Rules to act like a Staff SRE. I set up an alert that triggers if the P99 TTFA 1000ms for more than 5 minutes. Because we injected the LLM PROVIDER as a span attribute e.g., Groq vs OpenAI , the alert payload directly tells me which provider is lagging, allowing my team to instantly route traffic to a fallback provider. voice.turn cost usd directly into the trace span means you can correlate slow performance directly with high inference costs.Standard APM isn't built for the generative AI era, but OpenTelemetry and SigNoz give you the primitives to build exactly what you need. By manually instrumenting our async generators and mapping custom number attributes in ClickHouse, we turned a black-box Voice AI script into a fully observable, enterprise-grade system. If you want to try it out for your own Voice AI projects, you can grab the Python SDK from PyPI https://pypi.org/project/zooid/ , generate a full boilerplate app via npx create-zooid npm https://www.npmjs.com/package/create-zooid , or check out the full code on GitHub https://github.com/Priyank911/zooid . Built for the WeMakeDevs & SigNoz Observability Hackathon.