Measure, Don't Estimate: Labeling Speakers Without a Gated Model

A developer building AudioTrace, an audio analysis library, solved speaker diarization without a gated model by measuring voice pitch instead of using a neural network. The pitch-based approach works with zero setup, avoiding the friction of requiring a Hugging Face token for pyannote. The library defaults to the pitch method but allows opting into pyannote for higher quality.

In the first post https://dev.to/dimastatz/your-ai-voice-agent-is-a-black-box-heres-how-to-open-it-41kc I argued there are two ways to pull meaning out of audio: measure it with signal processing, or estimate it with a model. This post is the story of a problem where the obvious move was to estimate — and where measuring turned out to be better. The problem: labeling who is speaking. A transcript that says "Agent: …" and "Customer: …" is far more useful than an undifferentiated wall of text. Splitting a conversation by speaker is called diarization . The strong, well-known tool for diarization is pyannote https://github.com/pyannote/pyannote-audio . It's genuinely good. It is also gated : to run it you need a Hugging Face account, an access token, and to accept a license agreement before the weights will download. That's fine for a production deployment. It's a terrible first impression for someone who just pip install -ed your library and wants to see it work. Without a token, every single turn comes back labeled "unknown" . The newcomer's first run is a wall of unknown: … and they bounce. So I wanted a default path that works with zero setup, and lets you opt into pyannote when you have a token and want the best quality. My first instinct was the dumbest possible heuristic: in a two-party call, the speakers take turns, so just alternate Agent , Customer , Agent , Customer … It fell apart immediately. Speech recognizers like Whisper segment on sentences , not speakers . So the agent's multi-sentence greeting — "Hi there Thanks for calling. How can I help you today?" — gets split into three segments, and the naive alternator flip-flops the label mid-utterance: Agent: Hi there Customer: Thanks for calling. Agent: How can I help you today? Garbage. The structure I assumed one segment per speaker turn simply isn't there. Instead of forcing a model-shaped solution, I asked: what's physically in the audio that distinguishes these two speakers? In a typical support call, the agent and the customer have noticeably different voice pitch . That's a physical property of the waveform — exactly the So the approach becomes: The core of it is just a measurement plus a 2-way split: php import librosa import numpy as np def segment pitch y: np.ndarray, sr: int - float: """Mean fundamental frequency Hz of one transcript segment.""" f0, voiced flag, = librosa.pyin y, fmin=float librosa.note to hz "C2" , fmax=float librosa.note to hz "C7" , sr=sr, voiced = f0 voiced flag return float np.nanmean voiced if voiced.size else 0.0 def assign speakers pitches: list float , labels= "AI Agent", "Customer" : """Split segments into two speakers by a pitch threshold.""" valid = p for p in pitches if p 0 if not valid: return "unknown" len pitches threshold = float np.median valid Lower-pitched cluster - first label, higher - second. return labels 0 if p 0 and p <= threshold else labels 1 if p 0 else "unknown" for p in pitches A few dozen lines. No new dependency. No token. And the labels come out right for the common case — a plain measurement standing in for a model I couldn't assume the user had. Two similar voices two men, two women, a deep-voiced customer can fool the pitch split. With a token, pyannote still does better, and it handles three-plus speakers, overlapping speech, and edge cases this never will. So AudioTrace keeps both paths: python import audiotrace Default: zero-setup, infer speakers by pitch. report = audiotrace.analyze "call.wav", diarize=False, num speakers=2 Best quality: opt into pyannote with a token. report = audiotrace.analyze "call.wav", hf token="hf ..." The lesson I keep relearning: we grab the biggest model out of habit. A careful look at the data often points to something lighter, cheaper, and easier to reason about. "What signal is actually there?" is a more useful question than "which model should I download?" That's also a practical observability principle. The cheap, deterministic measurement runs in milliseconds with no GPU, which means you can run it on every call — and the things you can afford to run on every call are the things that actually catch regressions. We now have a structured CallReport with speakers, quality, sentiment, latency, and cost. In the final post I'll wire it into CI: fail the build when a prompt change makes the agent slower, colder, or less compliant , and emit the signals pip install audiotrace ⭐ Repo: github.com/dimastatz/audiotrace https://github.com/dimastatz/audiotrace Keep building