# Measure, Don't Estimate: Labeling Speakers Without a Gated Model

> Source: <https://dev.to/dimastatz/measure-dont-estimate-labeling-speakers-without-a-gated-model-3pgm>
> Published: 2026-07-11 03:06:02+00:00

In [the first post](https://dev.to/dimastatz/your-ai-voice-agent-is-a-black-box-heres-how-to-open-it-41kc) I argued there are two ways to pull meaning out of audio:

**measure** it with signal processing, or **estimate** it with a model. This post

is the story of a problem where the obvious move was to estimate — and where

measuring turned out to be better.

The problem: **labeling who is speaking.** A transcript that says "Agent: …" and

"Customer: …" is far more useful than an undifferentiated wall of text. Splitting

a conversation by speaker is called *diarization*.

The strong, well-known tool for diarization is

[pyannote](https://github.com/pyannote/pyannote-audio). It's genuinely good. It

is also **gated**: to run it you need a Hugging Face account, an access token, and

to accept a license agreement before the weights will download.

That's fine for a production deployment. It's a terrible first impression for

someone who just `pip install`

-ed your library and wants to see it work. Without a

token, every single turn comes back labeled `"unknown"`

. The newcomer's first run

is a wall of `unknown: …`

and they bounce.

So I wanted a *default* path that works with zero setup, and lets you opt into

pyannote when you have a token and want the best quality.

My first instinct was the dumbest possible heuristic: in a two-party call, the

speakers take turns, so just alternate `Agent`

, `Customer`

, `Agent`

, `Customer`

…

It fell apart immediately. Speech recognizers like Whisper segment on

**sentences**, not **speakers**. So the agent's multi-sentence greeting —

"Hi there! Thanks for calling. How can I help you today?"

— gets split into three segments, and the naive alternator flip-flops the label

mid-utterance:

```
Agent:    Hi there!
Customer: Thanks for calling.
Agent:    How can I help you today?
```

Garbage. The structure I assumed (one segment per speaker turn) simply isn't there.

Instead of forcing a model-shaped solution, I asked: *what's physically in the
audio that distinguishes these two speakers?*

In a typical support call, the agent and the customer have **noticeably
different voice pitch**. That's a physical property of the waveform — exactly the

So the approach becomes:

The core of it is just a measurement plus a 2-way split:

``` php
import librosa
import numpy as np

def segment_pitch(y: np.ndarray, sr: int) -> float:
    """Mean fundamental frequency (Hz) of one transcript segment."""
    f0, voiced_flag, _ = librosa.pyin(
        y,
        fmin=float(librosa.note_to_hz("C2")),
        fmax=float(librosa.note_to_hz("C7")),
        sr=sr,
    )
    voiced = f0[voiced_flag]
    return float(np.nanmean(voiced)) if voiced.size else 0.0

def assign_speakers(pitches: list[float], labels=("AI Agent", "Customer")):
    """Split segments into two speakers by a pitch threshold."""
    valid = [p for p in pitches if p > 0]
    if not valid:
        return ["unknown"] * len(pitches)
    threshold = float(np.median(valid))
    # Lower-pitched cluster -> first label, higher -> second.
    return [
        labels[0] if (p > 0 and p <= threshold) else
        labels[1] if p > 0 else "unknown"
        for p in pitches
    ]
```

A few dozen lines. No new dependency. No token. And the labels come out right for

the common case — a plain *measurement* standing in for a *model* I couldn't

assume the user had.

Two similar voices (two men, two women, a deep-voiced customer) can fool the pitch

split. With a token, pyannote still does better, and it handles three-plus

speakers, overlapping speech, and edge cases this never will. So AudioTrace keeps

both paths:

``` python
import audiotrace

# Default: zero-setup, infer speakers by pitch.
report = audiotrace.analyze("call.wav", diarize=False, num_speakers=2)

# Best quality: opt into pyannote with a token.
report = audiotrace.analyze("call.wav", hf_token="hf_...")
```

The lesson I keep relearning: **we grab the biggest model out of habit.** A

careful look at the data often points to something lighter, cheaper, and easier

to reason about. "What signal is actually there?" is a more useful question than

"which model should I download?"

That's also a practical observability principle. The cheap, deterministic

measurement runs in milliseconds with no GPU, which means you can run it on

*every* call — and the things you can afford to run on every call are the things

that actually catch regressions.

We now have a structured `CallReport`

with speakers, quality, sentiment, latency,

and cost. In the final post I'll wire it into CI: **fail the build when a prompt
change makes the agent slower, colder, or less compliant**, and emit the signals

```
pip install audiotrace
```

⭐ Repo: [github.com/dimastatz/audiotrace](https://github.com/dimastatz/audiotrace)

Keep building!