How we built a dual-model sound-event detection system with a locally trained Python classifier and a Google Teachable Machine model — and what we learned by forcing them to disagree.
Word count: ~2,500.
SonicSentinel AI started as a competition constraint and turned into the most
interesting design decision of the project. The rules require two independently trained models: one we build and train ourselves in Python, and one trained through
That constraint sounds bureaucratic until you understand what it buys you. A single
model is a black box that grades its own homework. Two models trained on the same data
through different frontends form a natural ensemble: when they agree, confidence is
real; when they disagree, that disagreement is itself information. Our alert system
uses this directly — a "Gunshot" prediction where both models agree with high
confidence is treated very differently from one where the models split.
The comparison is deliberately simple, which is what makes it trustworthy:
We built the dataset from scratch — roughly 300 clips per class across Machinery
Fault, Glass Breaking, Alarm or Siren, Vehicle Horn, Animal Sound, Gunshot, Panic
Scream, Aggression, Person Asking for Help, and Background Noise. Where real
recordings were scarce (notably "Person Asking for Help", which is entirely synthetic
text-to-speech, and 50 of 300 Aggression clips) we generated synthetic audio, and we
record the real/synthetic ratio per class in the manifest.
The split is frozen before any training: 2,100 / 450 / 450 (train/val/test),
stratified per class, with sha256 hashes in the manifest. A verifier script runs 28
checks, including the one that matters most: no recording in validation or test
contributes a segment to training — neither for the Python model nor for the GTM
model. Every derived clip carries its parent's audio_id plus a segment marker, so
lineage is provable row by row.
This sounds pedantic until you consider that a 3-second clip and its 2-second segment
from the same recording are nearly the same data. Cutting corners here is the easiest way to report 95% accuracy that collapses the moment the model meets an unseen clip.
The Python model does not listen to waveforms directly. A locked feature extractor
(audiofeat-1.0.0) converts each clip into a 254-dimensional vector: 128 mel-band
energies, 20 MFCC means and standard deviations, MFCC deltas, 12 chroma values, plus
spectral centroid, bandwidth, rolloff, flatness, RMS energy, zero-crossing rate,
onset statistics, and tempo. Mel bands approximate human frequency perception; MFCCs
compress spectral shape; onsets capture percussive attacks.
The extraction is deterministic: every clip is segmented into 3-second windows
(mode="cover"), features are averaged per segment, tempo is measured over the whole
recording. Locking this config as a versioned JSON file was important — when we
later retrained, the feature space could not silently drift under us.
We trained five model families — SVM (RBF), Random Forest, Extra Trees, Gradient
Boosting, and XGBoost — across a grid of hyperparameters and three seeds, 26
candidates in total. The harness in tuning.py enforces the protocol:
The winner, xgboost[max_depth=6, n_estimators=300], reached test accuracy of
0.693 and macro-F1 of 0.693 against our floors of 0.85 / 0.80. Per class:
| Class | F1 | Recall |
|---|---|---|
| Gunshot | 0.92 | 0.96 |
| Person Asking for Help | 0.99 | 1.00 |
| Glass Breaking | 0.73 | 0.80 |
| Vehicle Horn | 0.73 | 0.68 |
| Panic Scream | 0.70 | 0.75 |
| Background Noise | 0.66 | 0.64 |
| Aggression | 0.58 | 0.49 |
| Machinery Fault | 0.58 | 0.58 |
| Alarm or Siren | 0.56 | 0.56 |
| Animal Sound | 0.46 | 0.49 |
The pattern is instructive. The two best classes are the two with the most distinctive
signatures — a gunshot's broadband transient and the synthetic TTS clips' clean speech
formants. The four weakest classes are exactly the ones that are acoustically
heterogeneous (Animal Sound spans dogs, birds, and roosters) or confusable with
another class (sirens vs. vehicle horns; industrial hum vs. background noise).
Inference latency measured 0.81 ms per clip on CPU — the SRS allows 8 seconds for
a 30-second clip, so the model is four orders of magnitude inside budget. The
bottleneck in practice is feature extraction and decoding, not prediction.
Honest accounting: our first sweep missed the floors. The 254 summary numbers average
away timing detail — precisely the detail that separates a siren's sweep from a horn's
steady honk, or a scream's rising formant from a bird call. So we built a second
training path on mel-spectrogram tensors (a CNN over time × mel frames rather
than averages), with MobileNetV3 transfer weights available for transfer learning.
This is the current frontier: the classical retrain and the deep run are evaluated
side by side, selection still on validation only. Whatever wins becomes the served
model; the loser's metrics stay in the evidence folder. (The final numbers and the
confusion matrix are in python_models/metrics/classical_metrics.json and the
comparison report.)
The GTM samples were cut from the training-split recordings only — 5,230
two-second, 16 kHz segments, 404–619 per class, cut at deterministic positions. The
cutter refuses val/test parents outright, and the manifest records each segment's
parent id. In the browser, GTM trains its own log-mel frontend on these clips; our
server-side predictor reproduces that frontend exactly (same sample rate, window,
hop, mel bins, normalization), verified by comparing our reproduction against the
browser's own predictions clip-by-clip with a 0.05 confidence tolerance and a ≥95%
class-agreement requirement.
The result is a model that cannot share our Python model's bias. When they disagree —
for instance, the Python model says "Alarm or Siren" at 0.71 while GTM says "Vehicle Horn" at 0.66 — the comparison layer flags a Model Disagreement and the alert rules
route the event to manual review instead of trusting either.
Models are the easy part of a detection system. The hard parts are the paths around
them:
alert_rules/. Security operators tune behavior without
touching a line of Python.
Before writing any model code, we spent a day just looking. The waveform view shows a
gunshot as a single dense vertical burst and a siren as a slow amplitude wave; the
spectrogram view shows a siren's frequency sweep (a rising ribbon), a vehicle horn's
two steady horizontal harmonics, glass breaking as a spray of vertical broadband
lines, and speech as formant bands that move with the words.
Two design decisions came directly from staring at these pictures. First, why Animal
Sound confuses the model: a dog bark and a rooster crow look nothing alike in a
spectrogram — different fundamental frequencies, different temporal envelope — so one
class was asking the model to draw one boundary around several unconnected phenomena. Second, why sirens and horns collide: in the mel scale's upper region both classes
show strong harmonic stacks; they differ mainly in how the harmonics move over time,
which is a property the 254 summary features average away but a CNN over mel frames
retains. That observation drove the deep-model path more than any paper did.
We also learned to distrust single measurements. A clip can have a perfect RMS level
and still be unusable if it contains thirty seconds of silence followed by one second
of event. So the quality verdict combines silence ratio, clipping fraction, and
signal-to-noise estimate, and the preprocessing stage reports why it flagged a clip
— reasons that end up in the event record and the review UI, not just a boolean.
We tested robustness the unglamorous way: by degrading our own test clips. Adding
stationary pink noise at 10 dB SNR barely moved Gunshot or Glass Breaking recall —
their broadband transients survive — but dropped Alarm or Siren noticeably, because
tonal sweeps sit exactly where stationary noise lives. At 5 dB SNR, Background Noise
recall rose (everything started looking like noise) while Aggression recall fell
through the floor.
The false-positive analysis told a sharper story than the accuracy number. Most false
"Gunshot" alerts traced back to fireworks-adjacent transients and door slams inside
Background Noise recordings — acoustically, a legitimate confusion; a shotgun and a
car backfire share their first 100 ms. Most false negatives for "Panic Scream" were
distant screams at low SNR, where the spectral fingerprint is real but buried. Both
findings fed the alert rules: gunshot alerts now require either model agreement or a
repeat detection within the window, and distant-scream events are routed to manual
review rather than auto-dismissed.
The false-negative analysis also changed the severity mapping. A missed gunshot
that later surfaces in event history is worse than a false alarm that a human clears
in two seconds — so the critical-class recall floor (0.85) is deliberately stricter
than the accuracy floor, and alert thresholds for critical classes sit lower than for
informational ones. Threshold tuning is a policy decision, and we made it explicit in
alert_rules.json rather than hiding it in code.
An audio monitoring system is a privacy instrument by definition, so the constraints
are part of the design. Live monitoring requires explicit browser consent per session,
recorded with the session row; sessions are stopped server-side, not abandoned.
Retention is configurable per artifact type (event records, uploaded audio, live
session audio) with a purge command that respects legal holds — a flagged
investigation's audio survives retention expiry. Access is role-gated: a normal user
sees events; only reviewers decide reviews; only administrators touch config, users,
and retention; every privileged action lands in the audit log with actor, action,
and content diff.
On security: uploads are validated by content (magic bytes and decodability), not by
extension; all API writes re-check authorization server-side rather than trusting the
UI; passwords are stored hashed with per-user salts and a minimum-length policy
enforced server-side; error envelopes return stable machine-readable codes without
stack traces. And per the competition's integrity rules — which we agree with — no
external generative-AI API participates in any runtime decision: the final sound
classification comes only from the two trained models.
Three things stand out. First, we would collect harder data before training: the
Animal Sound class should have been split into sub-classes from the start, since one
"Animal" class spanning dogs and songbirds is a taxonomy problem, not a model problem.
Second, we would build the mel-tensor CNN path on day one instead of treating summary
features as a destination — they were a good baseline, but the timing information they
discard turned out to be exactly what the weak classes needed. Third, we would
benchmark the near-duplicate detector earlier: near-duplicate clips across classes
( background noise captured at the same street corner ) cost us confusion we only
diagnosed after the first sweep.
The repository contains the full dataset (manifest + frozen split), both training
pipelines, the converted GTM model, the configurable rule files, a 400+ test pytest
suite, and step-by-step installation and execution instructions in README.md. The
demo video walks through upload → dual prediction → comparison → alert → manual
review → live monitoring.
Sound detection is not a solved problem, and this project is not a finished product —
but the discipline of two independent models, one frozen split, and a comparison layer
that treats disagreement as signal rather than noise is a pattern worth reusing.