{"slug": "can-yamnet-detect-unseen-sudden-sounds-in-real-time-a-48-stream-evaluation", "title": "Can YAMNet Detect Unseen Sudden Sounds in Real Time? A 48-Stream Evaluation", "summary": "A developer tested whether Google's YAMNet audio classification model can detect unseen sudden sounds in real time without prior specification. The best configuration achieved 45.8% recall, detecting 22 of 48 events, but the results did not support the hypothesis that simple distance over YAMNet embeddings can reliably detect novel sounds. The experiment used YAMNet embeddings and class scores to monitor changes in audio streams, with detectors measuring frame-to-frame differences and distances to a memory of normal sounds.", "body_md": "Hello, everyone.\n\nThere are many situations where we may want to monitor sounds that happen without warning, such as breaking glass or a car horn. Registering every possible sound in advance, however, is not realistic.\n\nToday, I am testing whether YAMNet can detect that a sound stream has changed, without first specifying which sound it should recognize.\n\nThe short answer is that the best configuration detected 22 of 48 events, for **45.8% recall**. Processing was easily fast enough, but the results did not support the hypothesis that a simple distance over YAMNet embeddings can reliably detect unseen sudden sounds.\n\n[YAMNet](https://github.com/tensorflow/models/tree/master/research/audioset/yamnet) is an audio classification model that Google [ added to TensorFlow Models on November 21, 2019](https://github.com/tensorflow/models/commit/dfffd623b6be8d1d9744b8e261fbac370d17c46d). It was trained on AudioSet and predicts 521 acoustic event classes, including speech, rain, vehicles, and animals.\n\nThe network uses MobileNet V1, a convolutional architecture designed to reduce computation on mobile hardware. It reads roughly 0.96 seconds of 16 kHz mono audio at a time and returns:\n\n`scores`\n\n(521 dimensions): confidence for each class`embedding`\n\n(1,024 dimensions): a compact numerical representation of the sound`spectrogram`\n\n(64 bands): frequency content over timeThis experiment needs the embedding as well as the class scores, so I used the [SavedModel version on TensorFlow Hub](https://tfhub.dev/google/yamnet/1).\n\nThe relevant licenses are:\n\nESC-50 includes a noncommercial restriction. The model and dataset have separate licenses, so both need to be considered for redistribution or product use.\n\nThe goal was to determine whether the detectors could notice a new sound mixed into a normal background, without being told the event label.\n\nThe complete code and results are available in the [yamnet-streaming-novelty lab in kiarina/labs](https://github.com/kiarina/labs/tree/main/2026/07/17/yamnet-streaming-novelty).\n\nYou will need `mise`\n\n, `uv`\n\n, FFmpeg, and an internet connection for the first download. The model and data use about 120 MB, while the Python environment uses about 1.3 GB.\n\n```\ngit clone --depth 1 --filter=blob:none --sparse \\\n  https://github.com/kiarina/labs.git\ncd labs\ngit sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \\\n  2026/07/17/yamnet-streaming-novelty\nmise -C 2026/07/17/yamnet-streaming-novelty run\n```\n\nThe first run downloads the pinned YAMNet model and 208 required WAV files from a fixed ESC-50 revision. The task verifies the SHA-256 of the YAMNet archive and writes the results to `output/report.json`\n\n.\n\nYAMNet is the only AI model in this pipeline. Small detectors for frequency change and vector distance operate around it.\n\n```\n16 kHz mono audio stream\n  ├─ frequency change every 32 ms ─────────> spectral flux\n  └─ 0.975 s window sent to YAMNet every 0.48 s\n       ├─ 521 class scores\n       │    ├─ distance from previous frame -> score delta\n       │    └─ distance from normal memory -> score kNN\n       └─ 1,024-dimensional embedding\n            ├─ distance from previous frame -> embedding delta\n            └─ distance from normal memory -> embedding kNN\n```\n\nThe `delta`\n\ndetectors measure change from the immediately previous sound. The `kNN`\n\ndetectors compare each frame with the five nearest examples in a memory of normal sounds. They use cosine distance, which measures the difference in direction between numerical vectors.\n\nI also evaluated `temporal fusion`\n\n, which produces an alert when either score delta or embedding delta fires.\n\nThe normal sounds were `rain`\n\n, `sea_waves`\n\n, `wind`\n\n, and `clock_tick`\n\n. I inserted one of `crying_baby`\n\n, `door_wood_knock`\n\n, `glass_breaking`\n\n, `siren`\n\n, `car_horn`\n\n, and `fireworks`\n\nfrom 2.0 to 3.0 seconds in the same five-second background clip. There were eight examples of each event, for 48 positive streams in total.\n\nEvent-to-background level was tested at -10, -5, 0, and +5 dB. At -10 dB, the event is substantially quieter than the background; at +5 dB, it is louder. The mixed streams existed only in memory during evaluation and were not saved as files.\n\nRecordings were separated by purpose:\n\n| Purpose | ESC-50 fold | Number of clips |\n|---|---|---|\n| Normal-sound memory | 1-3 | 96 |\n| Threshold calibration | 4 | 32 |\n| False-alert evaluation | 5 | 32 |\n| Sudden-event evaluation | 5 | 48 streams |\n\nEach threshold was set high enough to produce no alerts on the normal fold-4 calibration streams. I did not tune it against the final fold-5 evaluation set.\n\nI first compared whether positive streams received higher anomaly scores than negative streams without fixing a threshold. AUROC summarizes this ranking: 1.0 is ideal, while a value around 0.5 is close to chance.\n\n| Detector | AUROC | -10 dB | -5 dB | 0 dB | +5 dB |\n|---|---|---|---|---|---|\n| spectral flux | 0.449 | 0.393 | 0.315 | 0.555 | 0.534 |\n| score delta | 0.717 | 0.622 | 0.672 | 0.776 | 0.797 |\n| embedding delta | 0.734 |\n0.638 | 0.625 | 0.815 | 0.859 |\n| score kNN | 0.661 | 0.484 | 0.698 | 0.789 | 0.672 |\n| embedding kNN | 0.632 | 0.479 | 0.581 | 0.721 | 0.745 |\n\nEmbedding delta ranked first at 0.734, but it was only 0.018 ahead of score delta. These 80 evaluation streams are not enough to conclude that embeddings are generally better.\n\nThe broader pattern is clearer: distance from the previous frame worked better than distance from the normal-sound memory. For short events like these, asking \"did the sound suddenly change?\" was more useful than asking \"is this sound globally unlike the normal set?\"\n\n| Detector | Precision | Recall | F1 | False alerts/hour | Median latency |\n|---|---|---|---|---|---|\n| spectral flux | 0.000 | 0.0% | 0.000 | 0.0 | — |\n| score delta | 0.810 | 35.4% | 0.493 | 22.5 | 0.415 s |\n| embedding delta | 0.765 | 27.1% | 0.400 | 22.5 | 0.415 s |\n| score kNN | 1.000 | 6.2% | 0.118 | 0.0 | 0.895 s |\n| embedding kNN | 0.800 | 16.7% | 0.276 | 0.0 | 0.895 s |\n| score delta + embedding delta | 0.786 |\n45.8% |\n0.579 |\n22.5 | 0.415 s |\n\nPrecision is the proportion of emitted alerts that were correct. Recall is the proportion of the 48 events that were found. The best temporal-fusion detector found 22 and missed 26.\n\nThere was only one false alert in 160 seconds of normal audio. The reported 22.5 false alerts/hour extrapolates that single event to one hour, so it is a highly uncertain estimate rather than a production false-alert rate. Several hours of continuous audio would be needed for a useful measurement.\n\nRecall by event level was:\n\n| Event level | Detected | Recall | Median latency |\n|---|---|---|---|\n| -10 dB | 4/12 | 33.3% | 1.855 s |\n| -5 dB | 3/12 | 25.0% | 0.415 s |\n| 0 dB | 7/12 | 58.3% | 0.415 s |\n| +5 dB | 8/12 | 66.7% | 0.415 s |\n\nBy class, the detector found 6/8 glass-breaking events, 5/8 fireworks, and 4/8 car horns. It found only 2/8 crying-baby and 2/8 siren events. Short, sharp changes were relatively easy, while sounds that blended into the background or were weak in the selected one-second segment were harder.\n\nWith a looser threshold, simple spectral flux reached 29.2% recall but produced 967.5 false alerts/hour. Rain and waves naturally contain many frequency changes, so this score could not isolate unusual events.\n\nCombining spectral flux with embedding kNN also failed to help. Under the strict threshold it matched embedding kNN alone at 16.7% recall. Under the looser threshold it reached 45.8% recall but produced 607.5 false alerts/hour. Adding detectors was not automatically an improvement.\n\nA high novelty score also does not guarantee a correct YAMNet label. `Glass`\n\nwas the top label for only three of the eight glass-breaking events. At low event levels, background labels such as `Water`\n\n, `Rain`\n\n, and `Vehicle`\n\noften remained on top. Detecting that something changed and explaining what changed are separate problems.\n\nI processed 1,040 seconds of audio as sequential windows on a Mac Studio with an Apple M4 Max.\n\n```\nfeature extraction elapsed: 6.262 s\nreal-time factor:           0.0060x\n```\n\nA repeat run while writing this article took 6.921 seconds, for a real-time factor of 0.0067x. The detection counts, AUROC values, thresholds, and latencies matched the original run. Including YAMNet inference and spectral flux, both runs processed audio more than 150 times faster than its duration. Compute throughput was not a problem.\n\nHowever, YAMNet reads about 0.96 seconds at a time, so the shortest observed detection latency was still 0.415 seconds. Fast inference does not remove the wait introduced by the input window.\n\nThe verification environment was:\n\n```\nmachine: Mac Studio (Mac16,9)\nchip: Apple M4 Max\nOS: macOS 26.5.2, arm64\nPython: 3.12.10\nTensorFlow: 2.21.0\nNumPy: 2.3.5\nFFmpeg: 8.1.2\nrandom seed: 20260717\n```\n\nThe results come down to three points.\n\nIt processed 1,040 seconds of audio in 6.262-6.921 seconds across two runs. The compute throughput needed for real-time monitoring was available.\n\nThe best method found only 22 of 48 events. Quiet events were especially likely to disappear into the background. This is not ready for a safety-monitoring application.\n\nComparing the embedding with a memory of normal sounds was weaker than comparing each frame with the previous one. Local change was the more useful signal for these short events.\n\nThis experiment is limited to 48 synthetic five-second streams, 160 seconds of negative audio, ten selected ESC-50 categories, and one Mac. Continuous microphone input, several-hour false-alert measurements, different recording devices, and environments with multiple simultaneous events remain outside its scope.\n\nYAMNet was lightweight and easy to use as a foundation because it exposes both class scores and embeddings. Still, an embedding may contain useful acoustic information without its raw distance being a reliable anomaly score.\n\nIn this experiment, recent change worked more directly than storing a large normal-sound memory. A lightweight first stage that proposes sharp events such as glass breaks or fireworks, followed by YAMNet labels or another decision step, looks like a more promising use of this approach.", "url": "https://wpnews.pro/news/can-yamnet-detect-unseen-sudden-sounds-in-real-time-a-48-stream-evaluation", "canonical_source": "https://dev.to/kiarina/can-yamnet-detect-unseen-sudden-sounds-in-real-time-a-48-stream-evaluation-49d7", "published_at": "2026-07-17 01:53:43+00:00", "updated_at": "2026-07-17 02:28:09.238226+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["Google", "YAMNet", "TensorFlow", "TensorFlow Hub", "AudioSet", "MobileNet V1", "ESC-50", "kiarina"], "alternates": {"html": "https://wpnews.pro/news/can-yamnet-detect-unseen-sudden-sounds-in-real-time-a-48-stream-evaluation", "markdown": "https://wpnews.pro/news/can-yamnet-detect-unseen-sudden-sounds-in-real-time-a-48-stream-evaluation.md", "text": "https://wpnews.pro/news/can-yamnet-detect-unseen-sudden-sounds-in-real-time-a-48-stream-evaluation.txt", "jsonld": "https://wpnews.pro/news/can-yamnet-detect-unseen-sudden-sounds-in-real-time-a-48-stream-evaluation.jsonld"}}