Tracing Voice AI is Hard: How I Instrumented Streaming LLMs with OpenTelemetry and SigNoz

Voice AI agents are magical until they start lagging. Suddenly my Groq-powered voice assistant started taking 3 seconds to respond and standard APM tools were useless. They could tell me that an HTTP request was open, but they couldn’t tell me if the bottleneck was the Speech-to-Text (STT) transcription, the LLM’s Time to First Token (TTFT), or the Text-to-Speech (TTS) synthesis.

Here’s how I did this by building Zooid, a custom OpenTelemetry instrumentation layer that traces streaming Voice AI, tracks FinOps token costs, and visualizes everything natively in SigNoz.

A typical CRUD web application is easy to trace. A request comes in, hits a database, and returns a JSON response. You drop in the opentelemetry-instrumentation-flask

package, and you are done.

Voice AI is a whole other beast. A "Voice Turn" consists of:

This is where the normal auto-instrumentation fails, since the LLM returns an async generator and not a static response. If you just trace the function execution, the span ends when the generator is created, not when the stream actually finishes.

I needed a way to track the lifecycle of a streaming conversation manually, measure Voice specific metrics (like Time To First Audio) and get this data into SigNoz.# Step 1. Instrumenting the async stream

I created a custom Python decorator (@instrument_voice_app) that manually manages the OpenTelemetry tracer to address the streaming problem.

Instead of wrapping a function, it injects a VoiceTurnContext

object into the agent. This enables the inner AI logic to tell you exactly when specific milestones are achieved, such as when the initial audio byte is streamed back to the user.

Here's the basic logic I used to follow the stream:

from opentelemetry import trace
import time

tracer = trace.get_tracer("zooid.voice")

def instrument_voice_app(func):
    async def wrapper(*args, **kwargs):
        with tracer.start_as_current_span("voice_turn") as span:
            turn_start = time.time()
            context = VoiceTurnContext()

            try:
                result = await func(context, *args, **kwargs)

                if context.first_audio_time:
                    ttfa = context.first_audio_time - turn_start
                    span.set_attribute("voice.ttfa_ms", ttfa * 1000)

                if context.stt_confidence:
                    span.set_attribute("stt.confidence", context.stt_confidence)

                if context.cost_usd:
                    span.set_attribute("voice.turn_cost_usd", context.cost_usd)

                span.set_status(trace.StatusCode.OK)
                return result

            except Exception as e:
                span.set_status(trace.StatusCode.ERROR, str(e))
                span.record_exception(e)
                raise
    return wrapper

This gave me a beautiful, continuous trace spanning the entire user interaction, but it introduced a new problem: How do I visualize these custom attributes in SigNoz?

Sending data to the OpenTelemetry Collector (localhost:4317) was quite easy. I could immediately see the traces in the SigNoz "Traces" tab.

But when I went to the "Dashboards" tab and tried to create a Time-Series panel for voice.turn_cost_usd

and voice.ttfa_ms

, the dashboard showed "No Data".## The Problem with It

This is where I found a very important piece of information on how SigNoz stores OpenTelemetry attributes in ClickHouse.

In Python, when you do span.set_attribute("voice.ttfa_ms", 450.5) it is sent as a Number by OpenTelemetry. This is cleanly stored in the new SigNoz trace schema (signoz_index_v3

). However, SigNoz custom dashboard query builder depends on materialized views to map these attributes correctly to aggregate these metrics effectively.

-- The crucial missing lines in the Materialized View
attributes_number as numberTagMap,
attributes_bool as boolTagMap

Once I applied this fix and backfilled the index, the query builder instantly recognized my custom metrics. I was able to query numberTagMap.voice.turn_cost_usd

as a Number data type directly in the SigNoz UI.

With the data flowing correctly, I built a custom JSON Dashboard in SigNoz tailored entirely for Voice AI.

(If you are following along, you can import this directly into your SigNoz instance by clicking "Import JSON" in the Dashboards tab).

Instead of looking at generic CPU usage, my dashboard tracks:

voice.turn_cost_usd

attribute to track Groq/OpenAI token spend in real-time.voice.interrupted

boolean attribute to see how often users cut the AI off (a great indicator of whether the bot is talking too much).(Insert Screenshot of the SigNoz Dashboard showing the TTFA Time Series and FinOps Cost panels here)

Finally, I utilized SigNoz's Alert Rules to act like a Staff SRE. I set up an alert that triggers if the P99 TTFA > 1000ms

for more than 5 minutes.

Because we injected the LLM_PROVIDER

as a span attribute (e.g., Groq

vs OpenAI

), the alert payload directly tells me which provider is lagging, allowing my team to instantly route traffic to a fallback provider.

voice.turn_cost_usd

) directly into the trace span means you can correlate slow performance directly with high inference costs.Standard APM isn't built for the generative AI era, but OpenTelemetry and SigNoz give you the primitives to build exactly what you need. By manually instrumenting our async generators and mapping custom number attributes in ClickHouse, we turned a black-box Voice AI script into a fully observable, enterprise-grade system.

If you want to try it out for your own Voice AI projects, you can grab the Python SDK from PyPI, generate a full boilerplate app via npx create-zooid

(npm), or check out the full code on GitHub.

Built for the WeMakeDevs & SigNoz Observability Hackathon.