Ai Voice Analysis

A developer built SonicSentinel AI, a dual-model sound-event detection system that pairs a locally trained Python XGBoost classifier with a Google Teachable Machine model, using their disagreements as a signal in the alert pipeline. The Python model, trained on a frozen 2,100/450/450 stratified split of roughly 300 clips per class across ten sound categories, reached 0.693 test accuracy and 0.693 macro-F1, below the project's 0.85/0.80 floors, with Gunshot (F1 0.92) and Person Asking for Help (F1 0.99) strongest and Animal Sound (F1 0.46) weakest.

How we built a dual-model sound-event detection system with a locally trained Python classifier and a Google Teachable Machine model — and what we learned by forcing them to disagree. Word count: ~2,500. SonicSentinel AI started as a competition constraint and turned into the most interesting design decision of the project. The rules require two independently trained models : one we build and train ourselves in Python, and one trained through That constraint sounds bureaucratic until you understand what it buys you. A single model is a black box that grades its own homework. Two models trained on the same data through different frontends form a natural ensemble: when they agree, confidence is real; when they disagree, that disagreement is itself information. Our alert system uses this directly — a "Gunshot" prediction where both models agree with high confidence is treated very differently from one where the models split. The comparison is deliberately simple, which is what makes it trustworthy: We built the dataset from scratch — roughly 300 clips per class across Machinery Fault, Glass Breaking, Alarm or Siren, Vehicle Horn, Animal Sound, Gunshot, Panic Scream, Aggression, Person Asking for Help, and Background Noise. Where real recordings were scarce notably "Person Asking for Help", which is entirely synthetic text-to-speech, and 50 of 300 Aggression clips we generated synthetic audio, and we record the real/synthetic ratio per class in the manifest. The split is frozen before any training: 2,100 / 450 / 450 train/val/test , stratified per class, with sha256 hashes in the manifest. A verifier script runs 28 checks, including the one that matters most: no recording in validation or test contributes a segment to training — neither for the Python model nor for the GTM model. Every derived clip carries its parent's audio id plus a segment marker, so lineage is provable row by row. This sounds pedantic until you consider that a 3-second clip and its 2-second segment from the same recording are nearly the same data. Cutting corners here is the easiest way to report 95% accuracy that collapses the moment the model meets an unseen clip. The Python model does not listen to waveforms directly. A locked feature extractor audiofeat-1.0.0 converts each clip into a 254-dimensional vector: 128 mel-band energies, 20 MFCC means and standard deviations, MFCC deltas, 12 chroma values, plus spectral centroid, bandwidth, rolloff, flatness, RMS energy, zero-crossing rate, onset statistics, and tempo. Mel bands approximate human frequency perception; MFCCs compress spectral shape; onsets capture percussive attacks. The extraction is deterministic: every clip is segmented into 3-second windows mode="cover" , features are averaged per segment, tempo is measured over the whole recording. Locking this config as a versioned JSON file was important — when we later retrained, the feature space could not silently drift under us. We trained five model families — SVM RBF , Random Forest, Extra Trees, Gradient Boosting, and XGBoost — across a grid of hyperparameters and three seeds, 26 candidates in total. The harness in tuning.py enforces the protocol: The winner, xgboost max depth=6, n estimators=300 , reached test accuracy of 0.693 and macro-F1 of 0.693 against our floors of 0.85 / 0.80. Per class: | Class | F1 | Recall | |---|---|---| | Gunshot | 0.92 | 0.96 | | Person Asking for Help | 0.99 | 1.00 | | Glass Breaking | 0.73 | 0.80 | | Vehicle Horn | 0.73 | 0.68 | | Panic Scream | 0.70 | 0.75 | | Background Noise | 0.66 | 0.64 | | Aggression | 0.58 | 0.49 | | Machinery Fault | 0.58 | 0.58 | | Alarm or Siren | 0.56 | 0.56 | | Animal Sound | 0.46 | 0.49 | The pattern is instructive. The two best classes are the two with the most distinctive signatures — a gunshot's broadband transient and the synthetic TTS clips' clean speech formants. The four weakest classes are exactly the ones that are acoustically heterogeneous Animal Sound spans dogs, birds, and roosters or confusable with another class sirens vs. vehicle horns; industrial hum vs. background noise . Inference latency measured 0.81 ms per clip on CPU — the SRS allows 8 seconds for a 30-second clip, so the model is four orders of magnitude inside budget. The bottleneck in practice is feature extraction and decoding, not prediction. Honest accounting: our first sweep missed the floors. The 254 summary numbers average away timing detail — precisely the detail that separates a siren's sweep from a horn's steady honk, or a scream's rising formant from a bird call. So we built a second training path on mel-spectrogram tensors a CNN over time × mel frames rather than averages , with MobileNetV3 transfer weights available for transfer learning. This is the current frontier: the classical retrain and the deep run are evaluated side by side, selection still on validation only. Whatever wins becomes the served model; the loser's metrics stay in the evidence folder. The final numbers and the confusion matrix are in python models/metrics/classical metrics.json and the comparison report. The GTM samples were cut from the training-split recordings only — 5,230 two-second, 16 kHz segments, 404–619 per class, cut at deterministic positions. The cutter refuses val/test parents outright, and the manifest records each segment's parent id. In the browser, GTM trains its own log-mel frontend on these clips; our server-side predictor reproduces that frontend exactly same sample rate, window, hop, mel bins, normalization , verified by comparing our reproduction against the browser's own predictions clip-by-clip with a 0.05 confidence tolerance and a ≥95% class-agreement requirement. The result is a model that cannot share our Python model's bias. When they disagree — for instance, the Python model says "Alarm or Siren" at 0.71 while GTM says "Vehicle Horn" at 0.66 — the comparison layer flags a Model Disagreement and the alert rules route the event to manual review instead of trusting either. Models are the easy part of a detection system. The hard parts are the paths around them: alert rules/ . Security operators tune behavior without touching a line of Python. Before writing any model code, we spent a day just looking. The waveform view shows a gunshot as a single dense vertical burst and a siren as a slow amplitude wave; the spectrogram view shows a siren's frequency sweep a rising ribbon , a vehicle horn's two steady horizontal harmonics, glass breaking as a spray of vertical broadband lines, and speech as formant bands that move with the words. Two design decisions came directly from staring at these pictures. First, why Animal Sound confuses the model: a dog bark and a rooster crow look nothing alike in a spectrogram — different fundamental frequencies, different temporal envelope — so one class was asking the model to draw one boundary around several unconnected phenomena. Second, why sirens and horns collide: in the mel scale's upper region both classes show strong harmonic stacks; they differ mainly in how the harmonics move over time , which is a property the 254 summary features average away but a CNN over mel frames retains. That observation drove the deep-model path more than any paper did. We also learned to distrust single measurements. A clip can have a perfect RMS level and still be unusable if it contains thirty seconds of silence followed by one second of event. So the quality verdict combines silence ratio, clipping fraction, and signal-to-noise estimate, and the preprocessing stage reports why it flagged a clip — reasons that end up in the event record and the review UI, not just a boolean. We tested robustness the unglamorous way: by degrading our own test clips. Adding stationary pink noise at 10 dB SNR barely moved Gunshot or Glass Breaking recall — their broadband transients survive — but dropped Alarm or Siren noticeably, because tonal sweeps sit exactly where stationary noise lives. At 5 dB SNR, Background Noise recall rose everything started looking like noise while Aggression recall fell through the floor. The false-positive analysis told a sharper story than the accuracy number. Most false "Gunshot" alerts traced back to fireworks-adjacent transients and door slams inside Background Noise recordings — acoustically, a legitimate confusion; a shotgun and a car backfire share their first 100 ms. Most false negatives for "Panic Scream" were distant screams at low SNR, where the spectral fingerprint is real but buried. Both findings fed the alert rules: gunshot alerts now require either model agreement or a repeat detection within the window, and distant-scream events are routed to manual review rather than auto-dismissed. The false-negative analysis also changed the severity mapping. A missed gunshot that later surfaces in event history is worse than a false alarm that a human clears in two seconds — so the critical-class recall floor 0.85 is deliberately stricter than the accuracy floor, and alert thresholds for critical classes sit lower than for informational ones. Threshold tuning is a policy decision, and we made it explicit in alert rules.json rather than hiding it in code. An audio monitoring system is a privacy instrument by definition, so the constraints are part of the design. Live monitoring requires explicit browser consent per session, recorded with the session row; sessions are stopped server-side, not abandoned. Retention is configurable per artifact type event records, uploaded audio, live session audio with a purge command that respects legal holds — a flagged investigation's audio survives retention expiry. Access is role-gated: a normal user sees events; only reviewers decide reviews; only administrators touch config, users, and retention; every privileged action lands in the audit log with actor, action, and content diff. On security: uploads are validated by content magic bytes and decodability , not by extension; all API writes re-check authorization server-side rather than trusting the UI; passwords are stored hashed with per-user salts and a minimum-length policy enforced server-side; error envelopes return stable machine-readable codes without stack traces. And per the competition's integrity rules — which we agree with — no external generative-AI API participates in any runtime decision: the final sound classification comes only from the two trained models. Three things stand out. First, we would collect harder data before training: the Animal Sound class should have been split into sub-classes from the start, since one "Animal" class spanning dogs and songbirds is a taxonomy problem, not a model problem. Second, we would build the mel-tensor CNN path on day one instead of treating summary features as a destination — they were a good baseline, but the timing information they discard turned out to be exactly what the weak classes needed. Third, we would benchmark the near-duplicate detector earlier: near-duplicate clips across classes background noise captured at the same street corner cost us confusion we only diagnosed after the first sweep. The repository contains the full dataset manifest + frozen split , both training pipelines, the converted GTM model, the configurable rule files, a 400+ test pytest suite, and step-by-step installation and execution instructions in README.md . The demo video walks through upload → dual prediction → comparison → alert → manual review → live monitoring. Sound detection is not a solved problem, and this project is not a finished product — but the discipline of two independent models, one frozen split, and a comparison layer that treats disagreement as signal rather than noise is a pattern worth reusing.