Your AI Voice Agent Is a Black Box. Here's How to Open It.

A developer created AudioTrace, an open-source library that extracts structured signals from voice agent call recordings. The library combines classical signal processing for acoustic measurements and learned models for semantic analysis, enabling observability into voice interactions that are otherwise opaque. AudioTrace runs locally to preserve data privacy and outputs typed reports suitable for integration into existing monitoring stacks.

When your AI agent types, you can see everything it does. LangChain traces every step, LangSmith replays every run, OpenTelemetry hangs spans off each call. You know what the model saw, what it said, how long it took, and what it cost. The moment that same agent picks up a phone, the lights go out. A voice agent's entire interaction lives inside an .mp3 . The transcript, the customer's mood, the awkward four-second silence, the moment it talked over the caller, the point where the conversation went sideways — all of it is in there. But to your existing observability stack, that file is opaque. LangSmith sees the tokens you fed the LLM; it does not see the audio that reached a human ear. So most teams do the only thing they can: they listen to a handful of calls by hand and hope the sample is representative. That doesn't scale, and it misses the thing that makes voice agents hard — their behavior drifts. You tweak a prompt, swap a model, change a TTS voice, and the agent gets subtly slower, colder, or starts missing intents. No unit test catches it, because the regression lives in the audio. This series is about closing that gap. In this first post I'll lay out the mental model; the next two get hands-on with a tricky signal-extraction problem and with wiring voice signals into CI. Here's what's actually recoverable from a single call recording: That's a lot of signal locked inside one file. The reason teams rebuild this from scratch at every company is that prying it loose means bolting together speech recognition, speaker separation, audio analysis, a sentiment model, and a pricing sheet — and then maintaining all of it. The key insight that makes this tractable: there are really two different kinds of question you can ask of audio, and they want two different tools. 1. Measure it — classical signal processing. Deterministic math run straight on the waveform: energy, pitch, the length of a silence. Cheap, exact, no training data. It shines for physical questions: You measure the answer instead of guessing at it. 2. Estimate it — learned models. Statistical systems like Whisper or a sentiment classifier that have ingested enormous amounts of data and estimate an answer. They own everything that turns on meaning rather than physics: No hand-written rule survives real speech here — you need a model. Most of the craft is knowing which question belongs to which bucket: reach for a model to estimate meaning , for signal processing to measure physics . In the next post you'll see that when a model isn't available, a measurement can sometimes stand in for it — that turns out to be a surprisingly useful trick. I packaged this into a small open-source library called AudioTrace https://github.com/dimastatz/audiotrace . You hand it a recording; it hands back one structured, typed report — split along exactly that measure-vs-estimate line. The acoustic layer silence, pace, pitch is signal processing; the semantic layer transcript, sentiment, intent is models. pip install audiotrace python import audiotrace report = audiotrace.analyze audio="call recording.wav", metadata={"agent version": "v2.1", "provider": "vapi"}, print report.quality.overall score 0.87 print report.quality.speaking pace wpm 168.0 print report.sentiment.caller frustration False print report.latency.total ms 4200 print report.events.drop off False print report.cost.total usd 0.063 The return value is a Pydantic CallReport , so it's typed, validated, and trivial to serialize. You can emit it as OpenTelemetry spans, hang it off your LangChain and LangSmith traces, or assert on it in a CI check — which is exactly where this series is headed. Call recordings are about as sensitive as data gets. So AudioTrace runs entirely on your machine — no audio leaves the box, and the open models download once. Privacy here shouldn't be an upgrade you pay for; it should be the default. The two-layer model sounds tidy, but the interesting part is what happens when the "right" tool isn't available. In the next post I'll walk through a concrete example: labeling who is speaking without the gated model everyone reaches for — and why a few dozen lines of pitch measurement beat it for the common case. If you want to poke at it now: pip install audiotrace ⭐ The repo is at github.com/dimastatz/audiotrace https://github.com/dimastatz/audiotrace . Issues and PRs welcome — it's early, and provider integrations are exactly the kind of contribution that helps most. Keep building