A Coding Guide to Google Research’s MSEB: Writing Sound Encoders to the Benchmark Contract and Scoring Them Across Classification, Clustering, Retrieval and Segmentation

Google Research's MSEB (Massive Sound Embedding Benchmark), version 0.1.0, is structured in three layers — types (Sound, SoundEmbedding, Score, TaskMetadata), the MultiModalEncoder contract that user models implement, and per-task evaluators — with classification, clustering, retrieval, and segmentation evaluators depending only on NumPy and scikit-learn, while reranking and transcription pull in Whisper and the task runner pulls in TensorFlow and apache-beam. A coding walkthrough writes two encoders against the abstract base class, one measuring loudness over time and one measuring timbre, encodes a synthetic corpus generated in the notebook, and drives the four evaluators over those embeddings. The two encoders trade places depending on which evaluator is asked, which the guide presents as the argument for a multi-task benchmark made in numbers rather than prose.

In this tutorial, we work with MSEB https://github.com/google-research/mseb , the Massive Sound Embedding Benchmark from Google Research, and approach it from the perspective of what a leaderboard number actually means: the evaluator surface. We install the package and map its three layers, then write two deliberately different encoders against the framework’s own abstract base class: one that measures loudness over time and one that measures timbre, and encode a small synthetic corpus we generate in the notebook so nothing has to be downloaded. We drive the classification, clustering, retrieval, and segmentation evaluators over those embeddings, call the metric functions directly to see what each one rewards, and finish by assembling the TaskMetadata a real submission carries. The result is a comparison in which the two encoders trade places depending on which evaluator is asked, which is the argument for a multi-task benchmark made in numbers rather than in prose. python import os import sys import json import math import traceback import subprocess import numpy as np RESULTS = {} BENCH = {} def banner title : print "\n" + "=" 78 print title print "=" 78 def section name : def wrap fn : def run a, kw : banner name try: out = fn a, kw RESULTS name = out if isinstance out, str else "ok" return out except Exception as e: RESULTS name = f"SKIPPED / FAILED - {type e . name }: {e}" print f"\n {name} did not complete: {type e . name }: {e}" traceback.print exc limit=3 return None return run return wrap banner "0. Install MSEB and map the three layers we will use" subprocess.run sys.executable, "-m", "pip", "install", "-q", "mseb==0.1.0" , check=True import mseb from mseb import types, encoder as encoder lib, evaluator as evaluator lib, metrics from mseb.evaluators import classification evaluator, clustering evaluator, retrieval evaluator, segmentation evaluator, print f" mseb {mseb. version } | Python {sys.version.split 0 } | numpy {np. version }" print "\n MSEB is three layers, and a benchmark run walks down them:" print " types - Sound, SoundEmbedding, Score, TaskMetadata: the shapes every task speaks" print " encoder - MultiModalEncoder: the contract YOUR model implements" print " evaluators - classification, clustering, retrieval, reranking, transcription, segmentation, ..." print "\n evaluator entry points we will drive:" for module, cls in classification evaluator, "ClassificationEvaluator" , clustering evaluator, "ClusteringEvaluator" , retrieval evaluator, "RetrievalEvaluator" , segmentation evaluator, "SegmentationEvaluator" : print f" {module. name .split '.' -1 :28s} {cls}" print "\n Everything below runs on CPU with no dataset download: we synthesise the audio." We install mseb and import the three layers that a benchmark run walks down. The types module holds the shapes every task speaks, Sound, SoundEmbedding, Score and TaskMetadata; the encoder module holds MultiModalEncoder, the contract our own model implements; and the evaluators package holds one module per task family. We import only the four evaluators this notebook drives, because the classification, clustering, retrieval, and segmentation modules depend on nothing heavier than NumPy and scikit-learn. In contrast, the reranking and transcription evaluators pull in Whisper and the task runner pulls in TensorFlow and apache-beam. Everything below therefore runs on a free CPU runtime with no dataset download and no accelerator. SR = 16000 @section "1. The type contract: Sound, SoundEmbedding, Score" def type contract : t = np.arange SR / SR waveform = 0.5 np.sin 2 np.pi 440 t .astype np.float32 sound = types.Sound waveform=waveform, context=types.SoundContextParams id="demo 000", sample rate=SR, length=len waveform , language="en us", text="a 440 Hz tone" , print f" Sound id={sound.context.id r} {sound.waveform.shape} @ {sound.context.sample rate} Hz" f" - {sound.size bytes:,} bytes" embedding = types.SoundEmbedding embedding=np.zeros 1, 16 , dtype=np.float32 , N, D : one utterance-level vector timestamps=np.array 0.0, 1.0 , dtype=np.float32 , M, 2 : start, end in seconds context=sound.context, encoding stats=types.EncodingStats input size bytes=sound.size bytes, embedding size bytes=16 4 , print f" SoundEmbedding embedding{embedding.embedding.shape} timestamps{embedding.timestamps.shape}" f" - {embedding.size bytes} bytes" print f" compression ratio = {embedding.encoding stats.compression ratio:.5f}" f" {1 / embedding.encoding stats.compression ratio:,.0f}x smaller than the audio " print " N embeddings and M timestamps: M == N is frame-aligned, M == 1 is utterance-level." print " embedding may also hold N strings instead of vectors - step 8 uses exactly that." score = types.Score metric="Accuracy", description="Overall classification accuracy", value=0.875, min=0.0, max=1.0 print f"\n Score {score.metric}={score.value} in {score.min}, {score.max} :: {score.description}" for bad, why in dict metric="", description="d", value=0.5, min=0.0, max=1.0 , "empty metric name" , dict metric="m", description="d", value=0.5, min=1.0, max=0.0 , "min max" : try: types.Score bad except Exception as e: print f" rejected at construction {why} : {type e . name }: {e}" return f"Sound {sound.size bytes:,} B - embedding {embedding.size bytes} B" type contract We start with the type contract, because every other layer is expressed in it. A Sound carries a waveform, along with SoundContextParams, the identifier, sample rate, length, language, and optional transcript, which follow the audio through the whole pipeline. A SoundEmbedding carries an array of N embeddings and an array of M timestamp pairs, and the relation between N and M is the benchmark’s vocabulary: M equal to N means one vector per frame, while M equal to one means a single utterance-level vector, which is what our encoders produce. EncodingStats records the input and embedding sizes and exposes compression ratio, here a thousandfold reduction from audio to vector. A Score is a metric name, a value and its bounds, and it validates itself at construction, rejecting an empty metric name or a minimum above its maximum, so a malformed number cannot reach a leaderboard. The embedding field also accepts N strings instead of N vectors, which is the door that step 8 walks through. class EnergyEnvelopeEncoder encoder lib.MultiModalEncoder : """Baseline: average energy in n bins equal time slices. Loud/quiet, nothing about timbre.""" def init self, n bins: int = 16 : super . init self.n bins = n bins def setup self : self. ready = True a real encoder loads weights here def check input types self, batch : for item in batch: if not isinstance item, types.Sound : raise ValueError f"{type self . name } takes types.Sound, got {type item . name }" def encode self, batch - list types.SoundEmbedding : out = for sound in batch: slices = np.array split sound.waveform.astype np.float32 , self.n bins vec = np.array float np.sqrt np.mean s 2 + 1e-12 for s in slices , dtype=np.float32 vec /= np.linalg.norm vec + 1e-9 out.append types.SoundEmbedding embedding=vec, timestamps=np.array 0.0, sound.context.length / sound.context.sample rate , dtype=np.float32 , context=sound.context return out class SpectralProfileEncoder encoder lib.MultiModalEncoder : """Contender: mean log-magnitude spectrum pooled into n bands bands. Describes timbre.""" def init self, n bands: int = 16, frame: int = 512 : super . init self.n bands, self.frame = n bands, frame def setup self : self. window = np.hanning self.frame .astype np.float32 def check input types self, batch : for item in batch: if not isinstance item, types.Sound : raise ValueError f"{type self . name } takes types.Sound, got {type item . name }" def encode self, batch - list types.SoundEmbedding : out = for sound in batch: w = sound.waveform.astype np.float32 n frames = max 1, len w // self.frame spectra = np.abs np.fft.rfft w i self.frame: i + 1 self.frame self. window for i in range n frames mean spectrum = np.log1p np.mean spectra, axis=0 vec = np.array float b.mean for b in np.array split mean spectrum, self.n bands , dtype=np.float32 vec /= np.linalg.norm vec + 1e-9 out.append types.SoundEmbedding embedding=vec, timestamps=np.array 0.0, sound.context.length / sound.context.sample rate , dtype=np.float32 , context=sound.context return out @section "2. The encoder contract: three methods, and the framework does the rest" def encoder contract : print " MultiModalEncoder abstract methods a subclass must implement:" for name in sorted encoder lib.MultiModalEncoder. abstractmethods : print f" {name}" print " final framework-owned, do not override : setup , encode " t = np.arange SR / SR fade = np.exp -2.5 t .astype np.float32 a decaying note, so the envelope is not flat sound = types.Sound waveform= 0.5 fade np.sin 2 np.pi 440 t .astype np.float32 , context=types.SoundContextParams id="demo 000", sample rate=SR, length=SR for enc in EnergyEnvelopeEncoder , SpectralProfileEncoder : enc.setup emb = enc.encode sound 0 stats = emb.encoding stats attached by encode , not by our code print f"\n {type enc . name :24s} - {emb.embedding.shape} {emb.embedding.dtype}" f" output type={enc.output type . name }" print f" {'':24s} EncodingStats input={stats.input size bytes:,} B, " f"embedding={stats.embedding size bytes} B, flops={stats.flops} " print f" {'':24s} first 6 dims: {np.round emb.embedding 0 :6 , 3 }" print "\n The envelope encoder sees the note decay; the spectral encoder sees one peak at 440 Hz." try: EnergyEnvelopeEncoder .encode "not a Sound" except ValueError as e: print f"\n wrong input type is caught by check input types: {e}" return "two encoders satisfying MultiModalEncoder" encoder contract We write two encoders by subclassing MultiModalEncoder, whose abstract methods are exactly three: setup loads whatever the model needs, check input types rejects anything that is not a Sound, and encode turns a batch into SoundEmbedding objects. The framework owns setup and encode, and encode is what attaches EncodingStats to every result, so our code never fills that in by hand. EnergyEnvelopeEncoder averages energy in sixteen equal time slices and therefore describes only how loudness moves; SpectralProfileEncoder pools the mean log-magnitude spectrum into sixteen bands and therefore describes timbre. Both L2-normalise their output so a dot product is a cosine. Encoding one decaying note through each shows the difference immediately: the envelope encoder sees the decay, and the spectral encoder sees a single peak at 440 Hz. python CLASSES = "tone", "chirp", "noise" N PER CLASS = 12 def synthesize kind: str, index: int, take: int - types.Sound: """One second of audio. take 0 is the document, take 1 is a noisier recording of the SAME clip. Two cues are deliberately separated: the spectrum says which class it is, and the amplitude envelope - drawn per item, independent of class - says which item it is. """ item = np.random.default rng 1000 + CLASSES.index kind 100 + index control = 0.25 + 0.75 item.random 8 envelope = np.interp np.linspace 0, 7, SR , np.arange 8 , control .astype np.float32 t = np.arange SR / SR if kind == "tone": w = np.sin 2 np.pi 380 + 80 item.random t elif kind == "chirp": f0, f1 = 200 + 50 item.random , 3200 + 400 item.random w = np.sin 2 np.pi f0 t + 0.5 f1 - f0 t 2 else: w = item.standard normal SR w /= np.sqrt np.mean w 2 + 1e-9 unit RMS: the envelope is the only loudness cue take rng = np.random.default rng 50 000 + take 10 000 + CLASSES.index kind 100 + index w = 0.4 + 0.2 take rng.random envelope w + 0.02 take rng.standard normal SR return types.Sound waveform=w.astype np.float32 , context=types.SoundContextParams id=f"{kind} {index:02d}" + "" if take == 0 else " take2" , sample rate=SR, length=SR, language="en us", text=kind @section "3. A synthetic corpus, encoded into MSEB embedding caches" def build corpus : corpus = synthesize k, i, 0 for k in CLASSES for i in range N PER CLASS queries = synthesize k, i, 1 for k in CLASSES for i in range N PER CLASS labels = {s.context.id: s.context.text for s in corpus + queries} print f" {len corpus } documents + {len queries } second takes of the same clips," f" {len CLASSES } classes, 1.0s each @ {SR} Hz" caches, query caches = {}, {} for enc in EnergyEnvelopeEncoder , SpectralProfileEncoder : enc.setup embeddings = enc.encode corpus one batched call, like a real runner caches type enc . name = {e.context.id: e for e in embeddings} query caches type enc . name = {e.context.id: e for e in enc.encode queries } matrix = np.vstack e.embedding for e in embeddings within, between = , for i in range len corpus : for j in range i + 1, len corpus : sim = float matrix i @ matrix j within if labels corpus i .context.id == labels corpus j .context.id else between .append sim print f" {type enc . name :24s} cache of {len embeddings } embeddings, dim {matrix.shape 1 }" f" mean cosine: same-class {np.mean within :.3f} vs other-class {np.mean between :.3f}" f" gap {np.mean within - np.mean between :+.3f} " print "\n Read that gap as a prediction: only the spectral encoder separates the classes at all." print " Steps 4-6 check whether the evaluators agree - and whether the gap is the whole story." globals .update CORPUS=corpus, QUERIES=queries, LABELS=labels, CACHES=caches, QCACHES=query caches return f"{len corpus } documents + {len queries } queries encoded by 2 encoders" build corpus We synthesize a corpus in which two cues are deliberately separated. The spectrum says which class a clip belongs to, a tone, a chirp or noise, while the amplitude envelope is drawn per item and is independent of class, so it identifies which clip it is without saying anything about what it is. We normalize every waveform to unit RMS before applying the envelope, leaving the envelope as the only loudness cue. We render each of the thirty-six items twice, once as the document and once as a noisier second take of the same clip, and encode both sets with both encoders into MSEB embedding caches, the plain dictionaries from sound id to SoundEmbedding that every evaluator consumes. The mean same-class and other-class cosine similarities printed here read as a prediction about the next three steps: only the spectral encoder separates the classes at all. python def class prototypes cache, labels : """Class embedding table C, D : the mean unit vector of each class, as the evaluator's weights .""" rows = for name in CLASSES: vecs = np.vstack cache i .embedding for i in cache if labels i == name mean = vecs.mean axis=0 rows.append mean / np.linalg.norm mean + 1e-9 return np.vstack rows .astype np.float32 @section "4. ClassificationEvaluator: prototypes in, Score objects out" def classification : table, example = {}, None for name, cache in CACHES.items : evaluator = classification evaluator.ClassificationEvaluator class labels=CLASSES, weights=class prototypes cache, LABELS , distance fn=evaluator lib.dot product, embeddings are L2-normalised - cosine top k value=2, predictions = evaluator.compute predictions cache {id: per-class score vector} references = classification evaluator.ClassificationReference i, LABELS i for i in cache table name = {s.metric: s.value for s in evaluator.compute metrics predictions, references } if name == "SpectralProfileEncoder": key = next iter predictions example = key, np.round list predictions key , 3 metric names = list next iter table.values :6 print f" {'encoder':26s}" + "".join f"{m :14 : 16s}" for m in metric names for name, row in table.items : print f" {name:26s}" + "".join f"{row m :16.3f}" for m in metric names print f"\n compute predictions returns one raw score per class, e.g. {example 0 r} - {example 1 }" print f" {CLASSES} - the argmax is the prediction, and top k value=2 also scores Top-2 Accuracy. " print " compute metrics turns those into types.Score objects, which is what the leaderboard stores." for name, row in table.items : BENCH.setdefault name, {} "Accuracy" = row "Accuracy" winner = max table, key=lambda k: table k "Accuracy" return "Accuracy: " + ", ".join f"{k} {v 'Accuracy' :.3f}" for k, v in table.items + f" winner {winner} " classification ClassificationEvaluator takes a table of class embeddings as its weights and a distance function, and we build the weights as class prototypes, the mean unit vector of each class. Its two methods separate cleanly: compute predictions returns a raw score per class for every cached embedding, and compute metrics turns those together with ClassificationReference labels into the list of Score objects that a leaderboard stores. Setting top k value to two adds Top-2 Accuracy alongside accuracy, balanced accuracy and the weighted precision, recall and F1. The spectral encoder classifies the corpus perfectly, and the envelope encoder lands well above chance but far below it, which is the ordering the cosine gap predicted. @section "5. ClusteringEvaluator: no labels at encode time, V-measure at score time" def clustering : evaluator = clustering evaluator.ClusteringEvaluator examples = clustering evaluator.ClusteringExample sound id=i, label=LABELS i for i in next iter CACHES.values print f" {len examples } examples, KMeans with k = {len CLASSES } inferred from the labels " for name, cache in CACHES.items : np.random.seed 0 MiniBatchKMeans takes no random state here: scores = evaluator cache, examples it falls back to NumPy's global RNG, so pin that or an unstructured embedding space scores 0.01-0.08 at random. The evaluator is callable. BENCH.setdefault name, {} "VMeasure" = scores 0 .value print f" {name:26s} {scores 0 .metric:12s} {scores 0 .value:6.3f}" f" {scores 0 .min}, {scores 0 .max} :: {scores 0 .description}" print "\n V-measure is the harmonic mean of homogeneity and completeness: 1.0 means the clusters" print " recover the classes exactly, 0.0 means they carry no information about them. Note how much" print " harsher it is on the envelope encoder than accuracy was - clustering gets no labels to lean on." return ", ".join f"{k} V={v 'VMeasure' :.3f}" for k, v in BENCH.items clustering ClusteringEvaluator asks the harder version of the same question, because it never sees a label at encode time: it runs KMeans over the cache. It scores the clusters against the labels with V-measure, the harmonic mean of homogeneity and completeness. The gap between the two encoders widens sharply here compared with classification, because a supervised prototype readout can exploit a faint cue that unsupervised clustering cannot find on its own. One practical detail is worth copying into any reproducible benchmark run: the evaluator constructs MiniBatchKMeans without a random state, so it falls back to NumPy’s global generator, and without seeding that generator an unstructured embedding space scores anywhere between roughly 0.01 and 0.08 from run to run. @section "6. RetrievalEvaluator: index the corpus, query it with a second take, score the ranking" def retrieval : print " Task: each query is a NOISIER RECORDING OF ONE DOCUMENT, and exactly one document is correct." print " This is identity, not category - a different question from steps 4 and 5.\n" out = {} for name, cache in CACHES.items : doc ids = list cache docs = np.vstack cache i .embedding for i in doc ids .astype np.float32 queries = QCACHES name searcher = retrieval evaluator.BruteForceSearcher candidates=docs, num neighbors=10 evaluator = retrieval evaluator.RetrievalEvaluator searcher=searcher, id by index id=doc ids, top k=5 predictions = evaluator.compute predictions queries references = retrieval evaluator.RetrievalReferenceId sound id=q, reference id=q.removesuffix " take2" for q in queries out name = {s.metric: s.value for s in evaluator.compute metrics predictions, references } q0 = next iter queries top = item "id" for item in predictions q0 .items :5 print f" top-5 for query {q0 r} under {name}:" print f" {top}" print f" correct document at rank {top.index q0.removesuffix ' take2' + 1}" f" | neighbours of the same class: {sum LABELS i == LABELS q0 for i in top }/5\n" metric names = "MRR", "EM", "RecallAt5", "NDCG@10" print f" {'encoder':26s}" + "".join f"{m: 14s}" for m in metric names for name, row in out.items : print f" {name:26s}" + "".join f"{row m :14.3f}" for m in metric names BENCH.setdefault name, {} "MRR" = row "MRR" print "\n MRR is 1/rank of the correct document, EM is 'it was rank 1', RecallAt5 is 'it was in the" print " top 5'. NDCG@10 here is graded credit for the same single relevant document." return ", ".join f"{k} MRR={v 'MRR' :.3f}" for k, v in out.items retrieval RetrievalEvaluator answers a different question from the two before it, and we set the task up so that difference is visible. Each query is the noisier second take of exactly one document, so the target is identity rather than category. We index the document embeddings in a BruteForceSearcher, ask for predictions over the query cache, and pass one RetrievalReferenceId per query naming its single correct document. The evaluator returns MRR, exact match, recall at our top k and NDCG at ten. The result inverts the previous two steps: the envelope encoder retrieves every clip at rank one, because the envelope is an item fingerprint. In contrast, the spectral encoder ranks slightly worse because clips of the same class look alike to it. The printed top-five lists make the mechanism plain, one neighbourhood class-random and the other class-pure. @section "7. The metric layer on its own: WER, CER, exact match, MRR, nDCG" def metric layer : truth = "the quick brown fox jumps over the lazy dog" for hypothesis in truth, "the quick brown fox jumped over a lazy dog", "quick brown fox over lazy dog" : werrors, wtotal = metrics.compute word errors truth, hypothesis cerrors, ctotal = metrics.compute character errors truth, hypothesis print f" WER {werrors / wtotal:5.3f} {werrors}/{wtotal} words " f"CER {cerrors / ctotal:5.3f} {cerrors}/{ctotal} chars {hypothesis r}" print "\n ranking metrics take reference, ranked ids :" ranked = "doc b", "doc a", "doc c", "doc d" for reference in "doc b", "doc a", "doc c", "doc z" : rank = ranked.index reference + 1 if reference in ranked else None print f" reference {reference r:8s} rank {str rank :4s}" f" EM {metrics.compute exact match reference, ranked :.1f}" f" MRR {metrics.compute reciprocal rank reference, ranked :.3f}" f" nDCG@4 {metrics.compute ndcg at k reference, ranked, k=4 :.3f}" print " compute ndcg at k assumes ONE relevant document and compares it by equality, so pass a" print " string, not a list - a list reference silently scores 0.0 while MRR still looks fine." print "\n embedding-space distances used by the reconstruction and stability tasks:" a = np.random.default rng 1 .standard normal 8, 4 .astype np.float32 for label, b in "identical", a , "noisy", a + 0.1 np.random.default rng 2 .standard normal a.shape : lp = metrics.compute lp norm a, b, p=2 dtw = metrics.compute dynamic time warping distance a, b print f" {label:10s} L2 {json.dumps {k: round float v , 3 for k, v in lp.items } }" f" DTW {json.dumps {k: round float v , 3 for k, v in dtw.items } }" return "WER/CER, EM/MRR/nDCG, Lp and DTW distances" metric layer We call the metric functions directly, without an evaluator around them, because they are the layer the task families share. compute word errors and compute character errors take two strings and return errors and totals separately, so the caller decides how to aggregate a corpus. The ranking metrics take a reference and a ranked list of identifiers, and comparing exact match, reciprocal rank and nDCG over the same ranking shows what each one pays for position. One sharp edge is worth naming: compute ndcg at k assumes a single relevant document and compares it by equality, so passing a list of relevant ids silently scores zero. In contrast, MRR, which does accept a list, still looks correct. We close with compute lp norm and compute dynamic time warping distance, the embedding-space distances behind the reconstruction and stability tasks. @section "8. SegmentationEvaluator: scoring WHAT was said and WHERE, separately" def segmentation : evaluator = segmentation evaluator.SegmentationEvaluator tau=0.05 print " Here a 'segment' carries a TERM, not a vector: SoundEmbedding.embedding holds N strings" print " and timestamps holds their N start, end spans. tau=0.05 - a boundary may be 50 ms out.\n" TERMS = "weather", 0.00, 0.30 , "in", 0.30, 0.65 , "boston", 0.65, 1.00 truth = segmentation evaluator.Segment embedding=term, start time=s, end time=e, confidence=1.0 for term, s, e in TERMS references = segmentation evaluator.SegmentationReference example id="utt 0", segments=truth def prediction spans : return {"utt 0": types.SoundEmbedding embedding=np.array term for term, , in spans , N strings timestamps=np.array s, e for , s, e in spans , dtype=np.float32 , N start, end context=types.SoundContextParams id="utt 0", sample rate=SR, length=SR , scores=np.ones len spans , dtype=np.float32 } confidences candidates = { "exact": TERMS, "50 ms out": "weather", 0.00, 0.28 , "in", 0.28, 0.67 , "boston", 0.67, 1.00 , "right words, wrong places": "weather", 0.00, 0.45 , "in", 0.45, 0.80 , "boston", 0.80, 1.00 , "right places, wrong words": "weather", 0.00, 0.30 , "on", 0.30, 0.65 , "austin", 0.65, 1.00 , } shown = "TimestampsAccuracy", "EmbeddingsAccuracy", "TimestampsAndEmbeddingsAccuracy", "WordErrorRate", "mAP" print f" {'prediction':28s}" + "".join f"{m :13 : 15s}" for m in shown for label, spans in candidates.items : result = evaluator.compute scores prediction spans , references per-example scores scores = {s.metric: s.value for s in evaluator.compute metrics result } aggregated Scores print f" {label:28s}" + "".join f"{scores m :15.3f}" for m in shown print "\n The last two rows are the point: one metric cannot tell 'knew the words, missed the timing'" print " from 'nailed the timing, heard the wrong words'. Timestamps and embeddings are scored apart," print " and only TimestampsAndEmbeddings credits getting both right at once." return "boundary + term scoring at tau=50 ms" segmentation SegmentationEvaluator scores what was said and where it was said as separate quantities, and it uses the string form of SoundEmbedding that step 1 mentioned: the embedding array holds one term per segment and the timestamps array holds their spans. Its flow is two-stage, compute scores over predictions and references first, then compute metrics over that result. We score four candidate segmentations of the same phrase against one ground truth with a tolerance of fifty milliseconds. Exact and fifty-milliseconds-out both score perfectly, which is what the tolerance is for. The last two rows carry the lesson: right words in the wrong places scores one on embeddings and zero on timestamps, right places with the wrong words does the reverse, and only the combined metric credits getting both right at once. @section "9. TaskMetadata and a leaderboard that disagrees with itself" def task metadata : cache = CACHES "SpectralProfileEncoder" evaluator = classification evaluator.ClassificationEvaluator class labels=CLASSES, weights=class prototypes cache, LABELS , top k value=2 references = classification evaluator.ClassificationReference i, LABELS i for i in cache scores = s for s in evaluator.compute metrics evaluator.compute predictions cache , references if s.metric in "Accuracy", "Weighted F1-Score" metadata = types.TaskMetadata name="SyntheticToneClassification", description="Three-way classification of synthetic tones, chirps and noise", reference="https://github.com/google-research/mseb", type="Classification", category="sound", main score="Accuracy", revision="1", dataset=types.Dataset path="synthetic/in-notebook", revision="1" , scores=scores, eval splits= "test" , eval langs= "en us" , print f" TaskMetadata: {metadata.name} type={metadata.type} main score={metadata.main score r}" print f" dataset={metadata.dataset.path r} rev {metadata.dataset.revision}" f" splits={metadata.eval splits} langs={metadata.eval langs}" print f" scores={ f'{s.metric}={s.value:.3f}' for s in metadata.scores }" try: types.TaskMetadata { {f.name: getattr metadata, f.name for f in metadata. dataclass fields .values }, "scores": } except Exception as e: print f" validated at construction: {type e . name }: {e}" columns = "Accuracy", "VMeasure", "MRR" print f"\n One row per encoder, one column per task family:" print f" {'encoder':26s}" + "".join f"{c: 12s}" for c in columns + " what it measures" for name, row in BENCH.items : print f" {name:26s}" + "".join f"{row c :12.3f}" for c in columns + " timbre - class" if "Spectral" in name else " loudness over time - identity" flips = c for c in columns if max BENCH, key=lambda n: BENCH n c = max BENCH, key=lambda n: BENCH n "Accuracy" print f"\n The winner changes column to column {', '.join flips } goes the other way . That is the whole" print " argument for a MASSIVE benchmark: a single headline number would have hidden it. An encoder" print " that cannot name a sound can still recognise it, and vice versa." print "\n A real submission runs mseb.runner over an mseb.task against a published dataset and writes" print " these same Score objects to JSON; the layers above are exactly what it exercises." return f"TaskMetadata + {len BENCH } encoders x {len columns } task families" task metadata We assemble the TaskMetadata that a real submission carries, the name, type, category, main score, dataset path and revision, evaluation splits and languages, together with the Score objects themselves, and it validates at construction in the same way a Score does, rejecting an empty score list. Then we put all results so far into one table: one row per encoder and one column per task family. The winner changes from column to column: the encoder that cannot name a sound still recognises it, and the encoder that names every sound correctly confuses clips that belong together. A single headline number would have hidden that completely, which is the argument for a benchmark that is massive in tasks rather than only in data. banner "SUMMARY" for name, res in RESULTS.items : print f" {name:<74s} {res}" print """ Where to go next - Swap in a real encoder: mseb/encoders/ ships wav2vec, Whisper, CLAP, EnCodec and SoundStream wrappers, plus CascadeEncoder for speech-to-text-to-embedding chains. Only the three methods from step 2 change; every evaluator above keeps working. - Run a published task: mseb.runner drives mseb.task over a real dataset with apache-beam; the task families live in mseb/tasks/ classification, retrieval, reranking, transcription, segmentation, clustering, reasoning, brain encoding, stability . - Compare against the leaderboard: https://huggingface.co/spaces/google/mseb-leaderboard - Read the contract you implemented: mseb/encoder.py and mseb/evaluator.py are ~500 lines total. """ The summary prints the one-line result each section returned, then points at the three directions this notebook opens: swapping in one of the real encoders shipped in the package, wav2vec, Whisper, CLAP, EnCodec, SoundStream or the cascade wrapper, which changes only the three methods from step 2 and leaves every evaluator working; running a published task through mseb.runner against a real dataset; and comparing the result with the public leaderboard. In conclusion, we treated MSEB as what it is, a contract plus a set of evaluators, and drove it end to end without downloading a dataset or touching an accelerator. Implementing three methods was enough to make our own code a first-class citizen of the benchmark, and the framework handled batching, statistics, and validation from there. The evaluators asked genuinely different questions of the same embeddings: classification and clustering ask what a sound is, retrieval asks which sound it is, and segmentation asks what was said and where, scored apart so a timing failure and a recognition failure never hide inside one average. Our two encoders traded places depending on the question asked, and we carry that result forward, because a single number cannot rank a sound embedding. The next step is to substitute a real encoder for our toy ones and re-run the same evaluators, since none of the scoring code above changes when the embeddings improve. Check out the GitHub Repo with Full Codes https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Voice%20AI/mseb sound embedding benchmark evaluators tutorial Marktechpost.ipynb . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter https://x.com/intent/follow?screen name=marktechpost and don’t forget to join our 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ and Subscribe to our Newsletter https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}} . Wait are you on telegram? now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/MJjjVDPS7whH8Ngs6 Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.