{"slug": "ai-voice-analysis", "title": "Ai Voice Analysis", "summary": "A developer built SonicSentinel AI, a dual-model sound-event detection system that pairs a locally trained Python XGBoost classifier with a Google Teachable Machine model, using their disagreements as a signal in the alert pipeline. The Python model, trained on a frozen 2,100/450/450 stratified split of roughly 300 clips per class across ten sound categories, reached 0.693 test accuracy and 0.693 macro-F1, below the project's 0.85/0.80 floors, with Gunshot (F1 0.92) and Person Asking for Help (F1 0.99) strongest and Animal Sound (F1 0.46) weakest.", "body_md": "*How we built a dual-model sound-event detection system with a locally trained Python classifier and a Google Teachable Machine model — and what we learned by forcing them\nto disagree.*\n\n**Word count: ~2,500.**\n\nSonicSentinel AI started as a competition constraint and turned into the most\n\ninteresting design decision of the project. The rules require **two independently trained models**: one we build and train ourselves in Python, and one trained through\n\nThat constraint sounds bureaucratic until you understand what it buys you. A single\n\nmodel is a black box that grades its own homework. Two models trained on the same data\n\nthrough *different* frontends form a natural ensemble: when they agree, confidence is\n\nreal; when they disagree, that disagreement is itself information. Our alert system\n\nuses this directly — a \"Gunshot\" prediction where both models agree with high\n\nconfidence is treated very differently from one where the models split.\n\nThe comparison is deliberately simple, which is what makes it trustworthy:\n\nWe built the dataset from scratch — roughly 300 clips per class across Machinery\n\nFault, Glass Breaking, Alarm or Siren, Vehicle Horn, Animal Sound, Gunshot, Panic\n\nScream, Aggression, Person Asking for Help, and Background Noise. Where real\n\nrecordings were scarce (notably \"Person Asking for Help\", which is entirely synthetic\n\ntext-to-speech, and 50 of 300 Aggression clips) we generated synthetic audio, and we\n\nrecord the real/synthetic ratio per class in the manifest.\n\nThe split is frozen before any training: **2,100 / 450 / 450** (train/val/test),\n\nstratified per class, with sha256 hashes in the manifest. A verifier script runs 28\n\nchecks, including the one that matters most: no recording in validation or test\n\ncontributes a segment to training — neither for the Python model nor for the GTM\n\nmodel. Every derived clip carries its parent's `audio_id` plus a segment marker, so\n\nlineage is provable row by row.\n\nThis sounds pedantic until you consider that a 3-second clip and its 2-second segment\n\nfrom the same recording are nearly the same data. Cutting corners here is the easiest\n\nway to report 95% accuracy that collapses the moment the model meets an unseen clip.\n\nThe Python model does not listen to waveforms directly. A locked feature extractor\n\n(`audiofeat-1.0.0`) converts each clip into a 254-dimensional vector: 128 mel-band\n\nenergies, 20 MFCC means and standard deviations, MFCC deltas, 12 chroma values, plus\n\nspectral centroid, bandwidth, rolloff, flatness, RMS energy, zero-crossing rate,\n\nonset statistics, and tempo. Mel bands approximate human frequency perception; MFCCs\n\ncompress spectral shape; onsets capture percussive attacks.\n\nThe extraction is deterministic: every clip is segmented into 3-second windows\n\n(mode=\"cover\"), features are averaged per segment, tempo is measured over the whole\n\nrecording. Locking this config as a versioned JSON file was important — when we\n\nlater retrained, the feature space could not silently drift under us.\n\nWe trained five model families — SVM (RBF), Random Forest, Extra Trees, Gradient\n\nBoosting, and XGBoost — across a grid of hyperparameters and three seeds, 26\n\ncandidates in total. The harness in `tuning.py` enforces the protocol:\n\nThe winner, `xgboost[max_depth=6, n_estimators=300]`, reached test accuracy of\n\n**0.693** and macro-F1 of **0.693** against our floors of 0.85 / 0.80. Per class:\n\n| Class | F1 | Recall | \n|---|---|---|\n| Gunshot | 0.92 | 0.96 | \n| Person Asking for Help | 0.99 | 1.00 | \n| Glass Breaking | 0.73 | 0.80 | \n| Vehicle Horn | 0.73 | 0.68 | \n| Panic Scream | 0.70 | 0.75 | \n| Background Noise | 0.66 | 0.64 | \n| Aggression | 0.58 | 0.49 | \n| Machinery Fault | 0.58 | 0.58 | \n| Alarm or Siren | 0.56 | 0.56 | \n| Animal Sound | 0.46 | 0.49 | \n\nThe pattern is instructive. The two best classes are the two with the most distinctive\n\nsignatures — a gunshot's broadband transient and the synthetic TTS clips' clean speech\n\nformants. The four weakest classes are exactly the ones that are acoustically\n\nheterogeneous (Animal Sound spans dogs, birds, and roosters) or confusable with\n\nanother class (sirens vs. vehicle horns; industrial hum vs. background noise).\n\nInference latency measured **0.81 ms** per clip on CPU — the SRS allows 8 seconds for\n\na 30-second clip, so the model is four orders of magnitude inside budget. The\n\nbottleneck in practice is feature extraction and decoding, not prediction.\n\nHonest accounting: our first sweep missed the floors. The 254 summary numbers average\n\naway timing detail — precisely the detail that separates a siren's sweep from a horn's\n\nsteady honk, or a scream's rising formant from a bird call. So we built a second\n\ntraining path on **mel-spectrogram tensors** (a CNN over time × mel frames rather\n\nthan averages), with MobileNetV3 transfer weights available for transfer learning.\n\nThis is the current frontier: the classical retrain and the deep run are evaluated\n\nside by side, selection still on validation only. Whatever wins becomes the served\n\nmodel; the loser's metrics stay in the evidence folder. (The final numbers and the\n\nconfusion matrix are in `python_models/metrics/classical_metrics.json` and the\n\ncomparison report.)\n\nThe GTM samples were cut from the **training-split recordings only** — 5,230\n\ntwo-second, 16 kHz segments, 404–619 per class, cut at deterministic positions. The\n\ncutter refuses val/test parents outright, and the manifest records each segment's\n\nparent id. In the browser, GTM trains its own log-mel frontend on these clips; our\n\nserver-side predictor reproduces that frontend exactly (same sample rate, window,\n\nhop, mel bins, normalization), verified by comparing our reproduction against the\n\nbrowser's own predictions clip-by-clip with a 0.05 confidence tolerance and a ≥95%\n\nclass-agreement requirement.\n\nThe result is a model that cannot share our Python model's bias. When they disagree —\n\nfor instance, the Python model says \"Alarm or Siren\" at 0.71 while GTM says \"Vehicle\n\nHorn\" at 0.66 — the comparison layer flags a Model Disagreement and the alert rules\n\nroute the event to manual review instead of trusting either.\n\nModels are the easy part of a detection system. The hard parts are the paths around\n\nthem:\n\n`alert_rules/`. Security operators tune behavior without\ntouching a line of Python.\nBefore writing any model code, we spent a day just looking. The waveform view shows a\n\ngunshot as a single dense vertical burst and a siren as a slow amplitude wave; the\n\nspectrogram view shows a siren's frequency *sweep* (a rising ribbon), a vehicle horn's\n\ntwo steady horizontal harmonics, glass breaking as a spray of vertical broadband\n\nlines, and speech as formant bands that move with the words.\n\nTwo design decisions came directly from staring at these pictures. First, why Animal\n\nSound confuses the model: a dog bark and a rooster crow look nothing alike in a\n\nspectrogram — different fundamental frequencies, different temporal envelope — so one\n\nclass was asking the model to draw one boundary around several unconnected phenomena.\n\nSecond, why sirens and horns collide: in the mel scale's upper region both classes\n\nshow strong harmonic stacks; they differ mainly in *how the harmonics move over time*,\n\nwhich is a property the 254 summary features average away but a CNN over mel frames\n\nretains. That observation drove the deep-model path more than any paper did.\n\nWe also learned to distrust single measurements. A clip can have a perfect RMS level\n\nand still be unusable if it contains thirty seconds of silence followed by one second\n\nof event. So the quality verdict combines silence ratio, clipping fraction, and\n\nsignal-to-noise estimate, and the preprocessing stage reports *why* it flagged a clip\n\n— reasons that end up in the event record and the review UI, not just a boolean.\n\nWe tested robustness the unglamorous way: by degrading our own test clips. Adding\n\nstationary pink noise at 10 dB SNR barely moved Gunshot or Glass Breaking recall —\n\ntheir broadband transients survive — but dropped Alarm or Siren noticeably, because\n\ntonal sweeps sit exactly where stationary noise lives. At 5 dB SNR, Background Noise\n\nrecall *rose* (everything started looking like noise) while Aggression recall fell\n\nthrough the floor.\n\nThe false-positive analysis told a sharper story than the accuracy number. Most false\n\n\"Gunshot\" alerts traced back to fireworks-adjacent transients and door slams inside\n\nBackground Noise recordings — acoustically, a legitimate confusion; a shotgun and a\n\ncar backfire share their first 100 ms. Most false negatives for \"Panic Scream\" were\n\ndistant screams at low SNR, where the spectral fingerprint is real but buried. Both\n\nfindings fed the alert rules: gunshot alerts now require either model agreement or a\n\nrepeat detection within the window, and distant-scream events are routed to manual\n\nreview rather than auto-dismissed.\n\nThe false-negative analysis also changed the *severity* mapping. A missed gunshot\n\nthat later surfaces in event history is worse than a false alarm that a human clears\n\nin two seconds — so the critical-class recall floor (0.85) is deliberately stricter\n\nthan the accuracy floor, and alert thresholds for critical classes sit lower than for\n\ninformational ones. Threshold tuning is a policy decision, and we made it explicit in\n\n`alert_rules.json` rather than hiding it in code.\n\nAn audio monitoring system is a privacy instrument by definition, so the constraints\n\nare part of the design. Live monitoring requires explicit browser consent per session,\n\nrecorded with the session row; sessions are stopped server-side, not abandoned.\n\nRetention is configurable per artifact type (event records, uploaded audio, live\n\nsession audio) with a purge command that respects legal holds — a flagged\n\ninvestigation's audio survives retention expiry. Access is role-gated: a normal user\n\nsees events; only reviewers decide reviews; only administrators touch config, users,\n\nand retention; every privileged action lands in the audit log with actor, action,\n\nand content diff.\n\nOn security: uploads are validated by content (magic bytes and decodability), not by\n\nextension; all API writes re-check authorization server-side rather than trusting the\n\nUI; passwords are stored hashed with per-user salts and a minimum-length policy\n\nenforced server-side; error envelopes return stable machine-readable codes without\n\nstack traces. And per the competition's integrity rules — which we agree with — no\n\nexternal generative-AI API participates in any runtime decision: the final sound\n\nclassification comes only from the two trained models.\n\nThree things stand out. First, we would collect harder data *before* training: the\n\nAnimal Sound class should have been split into sub-classes from the start, since one\n\n\"Animal\" class spanning dogs and songbirds is a taxonomy problem, not a model problem.\n\nSecond, we would build the mel-tensor CNN path on day one instead of treating summary\n\nfeatures as a destination — they were a good baseline, but the timing information they\n\ndiscard turned out to be exactly what the weak classes needed. Third, we would\n\nbenchmark the near-duplicate detector earlier: near-duplicate clips across classes\n\n( background noise captured at the same street corner ) cost us confusion we only\n\ndiagnosed after the first sweep.\n\nThe repository contains the full dataset (manifest + frozen split), both training\n\npipelines, the converted GTM model, the configurable rule files, a 400+ test pytest\n\nsuite, and step-by-step installation and execution instructions in `README.md`. The\n\ndemo video walks through upload → dual prediction → comparison → alert → manual\n\nreview → live monitoring.\n\nSound detection is not a solved problem, and this project is not a finished product —\n\nbut the discipline of two independent models, one frozen split, and a comparison layer\n\nthat treats disagreement as signal rather than noise is a pattern worth reusing.", "url": "https://wpnews.pro/news/ai-voice-analysis", "canonical_source": "https://dev.to/buildwithfun/ai-voice-analysis-13oh", "published_at": "2026-09-27 00:47:06+00:00", "updated_at": "2026-09-27 01:31:00.914852+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools"], "entities": ["SonicSentinel AI", "Google Teachable Machine", "XGBoost", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-voice-analysis", "markdown": "https://wpnews.pro/news/ai-voice-analysis.md", "text": "https://wpnews.pro/news/ai-voice-analysis.txt", "jsonld": "https://wpnews.pro/news/ai-voice-analysis.jsonld"}}