A ten-second recording of a street can contain a car, a dog, speech and rain simultaneously. A classifier that must pick one is answering a question nobody asked. Audio tagging is the multi-label version, and the difference goes deeper than swapping the loss function.
Single-label classification ends in a softmax. A softmax normalises the scores so they sum to one, which encodes an assumption: exactly one class is present, so evidence for a dog is evidence against rain. That coupling is the whole point when the classes really are mutually exclusive, and it is actively harmful when they are not.
Tagging replaces the softmax with an independent sigmoid per class and trains with binary cross-entropy summed over classes. Each output is now its own yes/no question, the outputs do not sum to one, and a clip can score 0.9 for both “Speech” and “Rain” without either suppressing the other. The consequence people find surprising is that you now need a threshold per class, and there is no longer an argmax
that gives you the answer for free.
Sound events have durations that do not match your clip length, so tagging models work on short patches and pool. Google’s YAMNet is a useful concrete case because its geometry is documented. Its model card in the TensorFlow models repository states: 16 kHz mono input, a mel spectrogram of 64 bands covering 125–7500 Hz computed with a 25 ms Hann window and a 10 ms hop, framed into 0.96-second examples of 96 frames each at 50% overlap, fed to a MobileNet v1 depthwise-separable convolutional stack, producing a 1024-dimension embedding and scores over 521 AudioSet classes.
Those numbers explain a detail that otherwise looks like a bug: the card notes you need at least 975 ms of waveform to get the first frame of output. Ninety-six frames at a 10 ms hop span 950 ms of hops, plus the 25 ms window on the final frame, giving 975 ms. A 900 ms clip returns nothing at all.
The training labels are weak: AudioSet says a class occurs somewhere in a ten-second clip, not when. So the model is trained under a multiple-instance assumption — the clip is positive if any patch is positive — and it must discover the localisation itself. This is why tagging models are reliable about whether and much less reliable about when, and why sound event detection with onset and offset times is treated as a harder, separately evaluated task.
Input: 10.0 s, 16 kHz mono, a street recording.
Framing at 0.96 s patches, 0.48 s hop (50% overlap):
floor((10.00 - 0.96) / 0.48) + 1 = floor(18.83) + 1 = 19 patches
Per-patch sigmoid scores (four of 521 classes shown):
patch t(s) Speech Dog Rain Vehicle
1 0.00 0.04 0.02 0.71 0.31
5 1.92 0.06 0.03 0.68 0.88
9 3.84 0.81 0.02 0.66 0.44
13 5.76 0.77 0.05 0.70 0.20
17 7.68 0.09 0.93 0.69 0.12
Clip score by max-pooling over patches:
Speech 0.81 Dog 0.93 Rain 0.71 Vehicle 0.88
Clip score by mean-pooling over all 19 patches:
Speech 0.24 Dog 0.11 Rain 0.69 Vehicle 0.29
At a flat threshold of 0.5, max-pooling emits four tags and
mean-pooling emits one.
The scores above are illustrative inputs chosen to make the arithmetic legible, not measurements. What is real is the shape of the result. Max-pooling finds brief events — the dog barks once, in one patch, and max-pooling keeps it. Mean-pooling finds sustained ones — the rain is present throughout at a middling score and survives averaging while the dog does not. Neither is correct in general. If you care about brief events, max-pool or use an attention pooling layer that learns the weights; if you care about background conditions, mean-pool. Choosing the pooling function is choosing which errors you get.
One more property of the framing is worth stating because it decides how you stream. The patches are independent: nothing in the model carries state from one to the next, so scores for a long recording can be computed in any order, batched arbitrarily, and recomputed for a sub-range without redoing the rest. That makes tagging embarrassingly parallel and cheap to re-run when you change a threshold, which is the opposite of the situation for sequence models and is why the right architecture for a large archive is usually to store the raw per-patch scores once and derive tags from them repeatedly.
Because every class has its own sigmoid, every class has its own operating point, and a flat 0.5 across 521 classes is close to the worst available choice. Rare classes are calibrated pessimistically by the class imbalance in training; common ones fire constantly. Pick a threshold per class on a validation set against whatever you actually care about — usually precision at a fixed recall, or F1.
This is also why the headline metric is mean average precision. AP is the area under a class’s precision–recall curve, so it is computed over all thresholds and is independent of the one you pick; mAP averages that over classes. The PANNs work of Kong et al., published with reference code and checkpoints on GitHub, reports mAP 0.431 on AudioSet against a stated Google baseline of 0.317.
A number like mAP 0.431 sounds low next to image classification accuracies and is not comparable to them. It is an average over 527 classes with wildly different amounts of training data, many of which have a few hundred weakly labelled examples. Check the per-class AP for the handful of classes you care about; it varies over more than an order of magnitude. Published scores also move between checkpoint releases, so treat any figure here as of its publication date.
A tagging model can only tell you about classes in its label set, and the AudioSet ontology’s boundaries were drawn by its authors for their purposes. “Vehicle” and “Car” and “Car passing by” are separate classes with overlapping meaning; something you consider a single event may be spread across three sigmoids that each score 0.4 and none of which crosses your threshold. Summing scores over an ontology subtree before thresholding is often more robust than tuning the leaves.
When your classes are not in the ontology at all, the usual route is not retraining. It is taking the 1024-dimension embedding as a fixed feature and training a small classifier on a few hundred of your own examples, which is the same move described in audio embeddings for similarity search. An alternative is a language-conditioned model where the classes are free text; see CLAP audio embeddings. For the specific case of a fixed set of household events, the deployment concerns differ enough to be worth their own treatment in home sound event detection.