Hello, everyone.
Today is Tanabata in Japan. Events like Tanabata, summer dance festivals, and Christmas all seem to have their own sound images. They also feel like composites of multiple sound categories, such as music, effects, voices, and ambient noise, forming one overall impression.
Today, I am testing how well the YAMNet TFLite model can tag the ESC-50 environmental sound dataset.
YAMNet is an audio tagging model that predicts 521 AudioSet sound event classes. In this experiment, I run the TFLite version published on TensorFlow Hub from Python and map its labels to the 50 categories in ESC-50.
The main questions are:
The complete code and reproducible environment are available in the yamnet-esc50-audio-tagging lab in kiarina/labs.
You will need the following tools, an internet connection for the first run, and a few hundred MB of free space:
The following commands fetch only this lab and run it:
git clone --depth 1 --filter=blob:none --sparse \
https://github.com/kiarina/labs.git
cd labs
git sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \
2026/07/07/yamnet-esc50-audio-tagging
mise -C 2026/07/07/yamnet-esc50-audio-tagging run
On the first run, the task downloads YAMNet TFLite, the class map, and the ESC-50 archive. Later runs verify the downloaded files by SHA-256 and reuse them.
For a short smoke test, run this inside the lab directory:
uv run python main.py --folds 1 --limit-per-class 1
The model is the TFLite version of YAMNet from TensorFlow Hub.
model: https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1?lite-format=tflite
model SHA-256: 10c95ea3eb9a7bb4cb8bddf6feb023250381008177ac162ce169694d05c317de
class map revision: 5c597f85268743140854f0e670f2175e8668553a
class map SHA-256: cdf24d193e196d9e95912a2667051ae203e92a2ba09449218ccb40ef787c6df2
input: 1-D float32 waveform, 16 kHz mono
output: frame scores [frames, 521]
Each clip is converted with FFmpeg to 16 kHz mono float32 PCM and passed to the TFLite model. The frame scores are averaged into clip-level scores.
The input dataset is all 2,000 clips from ESC-50. ESC-50 contains five-second environmental recordings across 50 classes. Those 50 classes are also grouped into five coarse categories:
The dataset is pinned to revision 33c8ce9eb2cf0b1c2f8bcf322eb349b6be34dbb6
. The full ESC-50 dataset is distributed under CC BY-NC, so the noncommercial restriction applies to this experiment, which uses all 50 classes.
YAMNet and ESC-50 do not use the same label set. I therefore manually assigned plausible YAMNet labels to each ESC-50 category.
Some examples:
| ESC-50 category | Mapped YAMNet labels |
|---|---|
dog |
|
Dog , Bark |
|
cat |
|
Cat , Meow , Purr |
|
keyboard_typing |
|
Typing , Computer keyboard |
|
chainsaw |
|
Chainsaw |
|
washing_machine |
|
Mechanisms , Water , Environmental noise |
|
For each clip, the category score is the maximum score among the mapped YAMNet labels for that ESC-50 category. The category with the highest score becomes the prediction. I then report:
There is no training and no cross-validation. This is one pass over all 2,000 clips with a fixed pretrained model.
One run on a Mac Studio without a warm-up produced:
| Scope | count | fine@1 | fine@3 | fine@5 | coarse@1 |
|---|---|---|---|---|---|
| all categories | 2,000 | 60.45% | 79.15% | 85.45% | 78.70% |
| direct label mapping categories | 1,600 | 68.94% | 85.00% | 90.50% | 83.00% |
direct label mapping categories
covers the 40 categories where I judged the ESC-50 category and YAMNet labels to be relatively direct matches. It includes categories such as dog
, cat
, rain
, keyboard_typing
, and chainsaw
, and excludes categories such as insects
, drinking_sipping
, can_opening
, and washing_machine
.
The elapsed time was:
2000 clips elapsed: 57.985 s
seconds per clip: 0.029 s
This includes FFmpeg decoding, TFLite inference, mean aggregation of frame scores, and ESC-50 category scoring for all 2,000 files. It excludes model and dataset downloads.
The strongest fine-grained categories were:
| category | fine@1 | fine@3 | coarse@1 |
|---|---|---|---|
church_bells |
|||
| 100.0% | 100.0% | 100.0% | |
dog |
|||
| 97.5% | 97.5% | 97.5% | |
keyboard_typing |
|||
| 97.5% | 97.5% | 97.5% | |
train |
|||
| 95.0% | 100.0% | 97.5% | |
frog |
|||
| 92.5% | 95.0% | 97.5% |
The hardest categories were:
| category | fine@1 | fine@3 | coarse@1 |
|---|---|---|---|
drinking_sipping |
|||
| 0.0% | 2.5% | 20.0% | |
hen |
|||
| 0.0% | 92.5% | 92.5% | |
mouse_click |
|||
| 2.5% | 37.5% | 82.5% | |
can_opening |
|||
| 5.0% | 20.0% | 57.5% | |
washing_machine |
|||
| 5.0% | 25.0% | 12.5% |
hen
had 0% fine@1, but 92.5% fine@3 and 92.5% coarse@1. Most errors were predicted as rooster
. The exact name was wrong, but the model still placed the clip near a very similar sound.
The most common confusions were:
| actual | predicted | count |
|---|---|---|
hen |
||
rooster |
||
| 35 | ||
snoring |
||
breathing |
||
| 28 | ||
pouring_water |
||
water_drops |
||
| 27 | ||
mouse_click |
||
keyboard_typing |
||
| 22 | ||
siren |
||
clock_alarm |
||
| 20 |
Coarse-category accuracy was:
| coarse category | correct / count | coarse@1 |
|---|---|---|
| animals | 369 / 400 | 92.25% |
| exterior and urban | 349 / 400 | 87.25% |
| human non-speech | 293 / 400 | 73.25% |
| natural soundscapes and water | 284 / 400 | 71.00% |
| interior and domestic | 279 / 400 | 69.75% |
First, YAMNet TFLite ran cleanly from Python. Processing 2,000 five-second clips in just under a minute makes it practical as a lightweight audio tagging step.
The result is not strong enough if you need one exact label among 50 categories every time. Overall fine@1 was 60.45%. However, the correct category appeared in the top five for 85.45% of all clips, and for 90.50% of the direct-mapping subset. That looks useful for generating candidate labels, feeding a downstream classifier, or initializing audio search metadata.
Coarse accuracy rose to 78.70%. The animals and exterior/urban groups were especially stable. The interior/domestic group was weaker. Everyday indoor sounds often share broad labels such as click
, water
, or mechanisms
, so there may not be enough information to separate the fine categories cleanly.
There are important caveats. This evaluation depends on a manual mapping from 521 YAMNet labels to 50 ESC-50 categories. If YAMNet assigns a high score to a nearby label that is not in my mapping, the result can be counted as wrong even when the acoustic interpretation is reasonable. The reverse can also happen: broad labels may accidentally raise a category score.
The ESC-50 repository also documents possible information leakage from class-dependent preprocessing of the original source recordings. These numbers should therefore be read as observations under this fixed setup, not as an absolute model benchmark.
The verification environment was:
YAMNet feels different from a model like CLAP, which is more about semantic distance between sounds. It is easier to understand as a tool that directly attaches labels to what is currently audible. The top-line fine@1 number is in the 60% range, but the top candidates make it much more useful.
Errors such as hen → rooster
, snoring → breathing
, and pouring_water → water_drops
are also close enough that they do not feel random. If an application treats the output as candidates rather than final truth, the result looks like a reasonable coarse understanding of the sound.
Everyday domestic sounds such as washing_machine
and drinking_sipping
look harder. Categories that contain multiple mixed sounds, or categories without a clean YAMNet label, depend heavily on how the mapping table is written.
Still, it is interesting to see sounds that create one impression from multiple combined categories broken back into top candidates. Rather than expecting YAMNet alone to be perfect, I would treat it as a lightweight entry point for audio tagging.