{"slug": "measure-don-t-estimate-labeling-speakers-without-a-gated-model", "title": "Measure, Don't Estimate: Labeling Speakers Without a Gated Model", "summary": "A developer building AudioTrace, an audio analysis library, solved speaker diarization without a gated model by measuring voice pitch instead of using a neural network. The pitch-based approach works with zero setup, avoiding the friction of requiring a Hugging Face token for pyannote. The library defaults to the pitch method but allows opting into pyannote for higher quality.", "body_md": "In [the first post](https://dev.to/dimastatz/your-ai-voice-agent-is-a-black-box-heres-how-to-open-it-41kc) I argued there are two ways to pull meaning out of audio:\n\n**measure** it with signal processing, or **estimate** it with a model. This post\n\nis the story of a problem where the obvious move was to estimate — and where\n\nmeasuring turned out to be better.\n\nThe problem: **labeling who is speaking.** A transcript that says \"Agent: …\" and\n\n\"Customer: …\" is far more useful than an undifferentiated wall of text. Splitting\n\na conversation by speaker is called *diarization*.\n\nThe strong, well-known tool for diarization is\n\n[pyannote](https://github.com/pyannote/pyannote-audio). It's genuinely good. It\n\nis also **gated**: to run it you need a Hugging Face account, an access token, and\n\nto accept a license agreement before the weights will download.\n\nThat's fine for a production deployment. It's a terrible first impression for\n\nsomeone who just `pip install`\n\n-ed your library and wants to see it work. Without a\n\ntoken, every single turn comes back labeled `\"unknown\"`\n\n. The newcomer's first run\n\nis a wall of `unknown: …`\n\nand they bounce.\n\nSo I wanted a *default* path that works with zero setup, and lets you opt into\n\npyannote when you have a token and want the best quality.\n\nMy first instinct was the dumbest possible heuristic: in a two-party call, the\n\nspeakers take turns, so just alternate `Agent`\n\n, `Customer`\n\n, `Agent`\n\n, `Customer`\n\n…\n\nIt fell apart immediately. Speech recognizers like Whisper segment on\n\n**sentences**, not **speakers**. So the agent's multi-sentence greeting —\n\n\"Hi there! Thanks for calling. How can I help you today?\"\n\n— gets split into three segments, and the naive alternator flip-flops the label\n\nmid-utterance:\n\n```\nAgent:    Hi there!\nCustomer: Thanks for calling.\nAgent:    How can I help you today?\n```\n\nGarbage. The structure I assumed (one segment per speaker turn) simply isn't there.\n\nInstead of forcing a model-shaped solution, I asked: *what's physically in the\naudio that distinguishes these two speakers?*\n\nIn a typical support call, the agent and the customer have **noticeably\ndifferent voice pitch**. That's a physical property of the waveform — exactly the\n\nSo the approach becomes:\n\nThe core of it is just a measurement plus a 2-way split:\n\n``` php\nimport librosa\nimport numpy as np\n\ndef segment_pitch(y: np.ndarray, sr: int) -> float:\n    \"\"\"Mean fundamental frequency (Hz) of one transcript segment.\"\"\"\n    f0, voiced_flag, _ = librosa.pyin(\n        y,\n        fmin=float(librosa.note_to_hz(\"C2\")),\n        fmax=float(librosa.note_to_hz(\"C7\")),\n        sr=sr,\n    )\n    voiced = f0[voiced_flag]\n    return float(np.nanmean(voiced)) if voiced.size else 0.0\n\ndef assign_speakers(pitches: list[float], labels=(\"AI Agent\", \"Customer\")):\n    \"\"\"Split segments into two speakers by a pitch threshold.\"\"\"\n    valid = [p for p in pitches if p > 0]\n    if not valid:\n        return [\"unknown\"] * len(pitches)\n    threshold = float(np.median(valid))\n    # Lower-pitched cluster -> first label, higher -> second.\n    return [\n        labels[0] if (p > 0 and p <= threshold) else\n        labels[1] if p > 0 else \"unknown\"\n        for p in pitches\n    ]\n```\n\nA few dozen lines. No new dependency. No token. And the labels come out right for\n\nthe common case — a plain *measurement* standing in for a *model* I couldn't\n\nassume the user had.\n\nTwo similar voices (two men, two women, a deep-voiced customer) can fool the pitch\n\nsplit. With a token, pyannote still does better, and it handles three-plus\n\nspeakers, overlapping speech, and edge cases this never will. So AudioTrace keeps\n\nboth paths:\n\n``` python\nimport audiotrace\n\n# Default: zero-setup, infer speakers by pitch.\nreport = audiotrace.analyze(\"call.wav\", diarize=False, num_speakers=2)\n\n# Best quality: opt into pyannote with a token.\nreport = audiotrace.analyze(\"call.wav\", hf_token=\"hf_...\")\n```\n\nThe lesson I keep relearning: **we grab the biggest model out of habit.** A\n\ncareful look at the data often points to something lighter, cheaper, and easier\n\nto reason about. \"What signal is actually there?\" is a more useful question than\n\n\"which model should I download?\"\n\nThat's also a practical observability principle. The cheap, deterministic\n\nmeasurement runs in milliseconds with no GPU, which means you can run it on\n\n*every* call — and the things you can afford to run on every call are the things\n\nthat actually catch regressions.\n\nWe now have a structured `CallReport`\n\nwith speakers, quality, sentiment, latency,\n\nand cost. In the final post I'll wire it into CI: **fail the build when a prompt\nchange makes the agent slower, colder, or less compliant**, and emit the signals\n\n```\npip install audiotrace\n```\n\n⭐ Repo: [github.com/dimastatz/audiotrace](https://github.com/dimastatz/audiotrace)\n\nKeep building!", "url": "https://wpnews.pro/news/measure-don-t-estimate-labeling-speakers-without-a-gated-model", "canonical_source": "https://dev.to/dimastatz/measure-dont-estimate-labeling-speakers-without-a-gated-model-3pgm", "published_at": "2026-07-11 03:06:02+00:00", "updated_at": "2026-07-11 03:11:59.921162+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "machine-learning"], "entities": ["AudioTrace", "pyannote", "Whisper", "librosa", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/measure-don-t-estimate-labeling-speakers-without-a-gated-model", "markdown": "https://wpnews.pro/news/measure-don-t-estimate-labeling-speakers-without-a-gated-model.md", "text": "https://wpnews.pro/news/measure-don-t-estimate-labeling-speakers-without-a-gated-model.txt", "jsonld": "https://wpnews.pro/news/measure-don-t-estimate-labeling-speakers-without-a-gated-model.jsonld"}}