{"slug": "our-research-team-developed-a-more-efficient-audio-embedding-model", "title": "Our Research Team Developed a More Efficient Audio Embedding Model", "summary": "Yan Li, an Applied Research Engineer, developed a more efficient audio embedding model using an AI agent to iteratively explore improvements, addressing limitations in state-of-the-art models like BYOL-A, Audio-JEPA, and EAT, which suffer from limited real-world deployability due to AudioSet's commercial restrictions and massive compute requirements. The new model, still an internal work in progress, aims to overcome these barriers for production systems.", "body_md": "As research projects typically go, where you think you’re headed and where you end up are rarely the same.\n\nYan Li, one of our Applied Research Engineers, set out to simply evaluate an audio anomaly detection model trained on limited data samples at the request of a colleague. The task quickly evolved into something much more ambitious.\n\nAlong the way, Yan used an AI agent running autoresearch sessions to iteratively explore improvements while developing an audio embedding model, intervening with domain expertise when progress stalled. The results? A new model that tackles some of the biggest limitations in today’s state‑of‑the‑art approaches.\n\nWhile his research is still an internal work in progress, the process he took to put it together provides interesting perspective into the current methodologies and tools that we can now access, and how we can leverage them to create new models, features, and more.\n\nLet's take a look at how he did it and the results he got along the way.\n\n## Audio embedding models\n\nAn audio embedding model converts audio into a fixed-length vector that captures the important parts of a sound while discarding unnecessary detail.\n\nInstead of working directly with raw waveforms or spectrograms, downstream systems operate on these compact representations.\n\nIn practice, this makes embeddings incredibly useful:\n\n- A single embedding model can support many tasks, from speech detection to environmental sound classification\n- Downstream models can stay simple, often only a simple classifier or anomaly detector\n- The same embedding can generalize across domains, much like ImageNet features in vision applications\n\nBecause labeled audio data is scarce, most embedding models are trained with self-supervised learning. The model learns by predicting parts of the input or aligning different views of the same sound, rather than relying on explicit labels. This tends to produce representations that are robust and transferable.\n\n### State-of-the-art models\n\nRecent advances in these audio models are dominated by methods like BYOL‑A, Audio‑JEPA, and EAT.\n\nThese models share a few key traits:\n\n- They learn embeddings through prediction in latent space rather than reconstructing raw audio, enabling strong general-purpose representations\n- They rely on large transformer-based architectures, which drive performance but come with significant compute cost\n- They are typically evaluated using embeddings produced by a frozen encoder with simple classifiers downstream, emphasizing transferability across tasks\n\n(If you’re interested in earlier approaches to audio embedding, take a look at VGGish, PANNs, and YAMNet.)\n\n### Critical limitations\n\nDespite strong results, the state-of-the-art models have two major drawbacks that create a gap between what performs well in research benchmarks and what can realistically be deployed in production systems.\n\n**1. Limited real-world deployability**\n\nMost leading models are pretrained on AudioSet. While powerful, AudioSet is not unrestricted for commercial use, which makes these models difficult to deploy without retraining the architecture on alternative datasets.\n\n**2. Massive compute requirements**\n\nThese models are large and expensive to train, often requiring multiple GPUs and thousands of hours of audio. For many teams, reproducing or adapting these models is already a significant barrier.\n\nAfter training, these models remain large and computationally intensive at inference time. Running them on constrained hardware, such as edge devices, is often impractical without substantial modification or downsizing.\n\n## Research objectives\n\nWhile exploring ways to improve the performance of our colleague’s audio anomaly detection model, Yan evaluated both spectrogram-based features and audio embeddings as inputs, instead of relying on raw audio. Embeddings generated by EAT showed promising results. However, due to the limitations outlined earlier, EAT itself was not a viable option for this use case.\n\nThis led to a focused set of research objectives:\n\n- Can a useful audio embedding model be trained without relying on AudioSet?\n- Can the model size and compute requirements be reduced significantly?\n- Can strong performance on real-world downstream tasks be maintained?\n\nIf all three objectives could be satisfied, there would be a practical path toward a deployable solution.\n\n## Research approach\n\nYan started from the Audio-JEPA codebase, and treated it as a baseline for controlled experimentation, for two reasons.\n\nFirst, it already implements the Joint-Embedding Predictive Architecture (JEPA) training objective and eXtensive Audio Representation and Evaluation Suite (X-ARES) flow described in the accompanying paper, so it is a credible starting point.\n\nSecond, the codebase is organized around PyTorch Lightning and Hydra, which made it practical to run controlled experiments: model variants, masking schemes, optimizers, schedulers, and data settings could all be changed through small configuration edits instead of repeatedly rewriting training code.\n\nFrom there, he made a series of targeted changes that are described below.\n\n**Make the model smaller by design**\n\nThe first change was architectural. The JEPA training objective was kept, but the standard ViT patch embedding layer was replaced with an EAT-style convolutional patchifier. The transformer encoder itself was also made smaller than the baseline configuration. The main benefit of this frontend design was that it became feasible to use a much smaller ViT backbone while preserving useful representation quality.\n\n**Train on open data**\n\nAudioSet was replaced with Freesound Dataset 50k (FSD50K), a fully open dataset better suited for commercial use. While much smaller, it allowed iteration and deployment without licensing constraints.\n\nFor scale, see the substantial differences in the two corpora in the following table.\n\nDataset |\nClips |\nLabels |\nTotal audio |\n|---|---|---|---|\n| AudioSet-2M | 2.1M | 527 | 5,800 h |\n| FSD50K | 51K | 200 | 108 h |\n\nTo keep the experimentation loop fast enough for repeated agent-driven runs, mel spectrograms were precomputed and stored as cached shards on S3. Training jobs then streamed these prebuilt tensors instead of recomputing frontend features each time.\n\nThis had a meaningful operational impact. With each train-and-benchmark cycle already taking around 20 minutes, eliminating repeated audio decoding and mel extraction saved enough time to make larger hyperparameter and architecture sweeps practical.\n\n**Focus the model on meaningful signal**\n\nEnvironmental audio often contains long stretches of low-value background noise, while the events of interest are often short and weak. By applying normalization together with a noise floor during preprocessing, the model was biased toward informative parts of the signal.\n\n**Automate iteration with hyperparameter search**\n\nHydra’s Optuna was used to explore model and training configurations efficiently. After each sweep, the agent reviewed train-and-benchmark outputs and promoted strong trials into reproducible experiments for follow-up runs.\n\n**Extend training across datasets**\n\nAfter the initial training on FSD50K, model pretraining continued in stages, resuming from the selected checkpoint and training sequentially on additional datasets, UrbanSound8K and FSDnoisy18k.\n\n**Optimize for transfer, not just training loss**\n\nThe model was evaluated where it mattered most: downstream performance.\n\nInstead of focusing purely on self-supervised training loss, the candidate embedding models were evaluated directly on real-world tasks using X-ARES to measure frozen encoder performance on two benchmark datasets, Environmental Sound Classification 50 (ESC-50) and Domestic Environmental Sound Event Detection (DESED), with optional evaluation on FSD50K.\n\nThis shifted the objective from “lowest training loss” to “best transfer quality,” which is what ultimately matters in practice. In self-supervised setups, improvements in training loss can be made while transfer quality plateaus or regresses.\n\nFor this reason, the models were trained for a fixed number of epochs and evaluation on downstream tasks was prioritized, rather than relying on validation loss for early stopping during model training.\n\n## Agent involvement\n\nAn integral part of the research approach was the use of an AI agent to drive experimentation.\n\nFor this project, the GitHub Copilot CLI in autopilot mode, powered by the GPT-5.4 model with a high reasoning setting, was used as the research agent. In practice, the agent functioned less like an independent researcher and more like a highly efficient optimization engine.\n\nThe agent operated in a tight autoresearch based loop, proposing a change, running a train-and-benchmark cycle, evaluating the results, and keeping or discarding the idea. It explored configurations quickly and systematically, while Yan guided overall direction and stepped in when needed.\n\nA typical session lasted upwards of 9 hours, involved around two dozen experiments, each taking 15–30 minutes for training and benchmarking, used several premium requests, and consumed more than 20 million tokens.\n\nThe agent was given a concrete operating program in a `program.md`\n\nfile, including the instruction that this was \"an experiment to have the research agent improve the (EA-)JEPA model, especially when training on small datasets\" and that \"the goal is simple: get the highest ESC-50 MLP benchmark score.\"\n\nFor additional technical context, the agent was also pointed to a `PAPER.md`\n\nfile that summarized the associated Audio-JEPA paper and helped ground the experiment search in the original JEPA design and evaluation framing.\n\n## Results\n\nThe results demonstrated the main objective of this work: improving embedding quality for downstream use while moving toward a deployment-feasible model size.\n\n**Key outcomes:**\n\n- Model parameter count reduced from 85M to 14M: 71M fewer parameters (83.5% reduction, 6.1x smaller)\n- ESC-50 MLP accuracy increased from 0.338 to 0.6055: +0.2675 absolute (+79.1% relative)\n- ESC-50 k-NN accuracy increased from 0.140 to 0.461: +0.321 absolute (+229.3% relative)\n\nModel |\nParameters |\nESC-50 MLP Accuracy |\nESC-50 k-NN Accuracy |\n|---|---|---|---|\n| Original AS2M Trained Model | 85M | 0.338 | 0.140 |\n| FSD50K Optimized Model | 14M | 0.6055 | 0.461 |\n\nImprovement in both MLP and k-NN accuracy using embeddings from the new model suggests that the encoder is learning more transferable structure in the embedding space rather than simply fitting to a particular downstream classifier.\n\nMLP gains indicate that the embeddings contain information that can be separated more effectively by a small learned nonlinear classifier. k-NN gains indicate that semantically similar samples are naturally grouped together in the embedding space, meaning the representation itself has stronger local structure without requiring an additional learned transformation.\n\n## Observations\n\nA few patterns emerged from working with the agent. It performed best when objectives, constraints, and success metrics were clearly defined, and often benefited from being prompted to ask clarifying questions before acting.\n\nMost of the agent gains came from optimization and tuning rather than novel ideas; larger architectural changes still required human direction. Early iterations tended to deliver the strongest improvements, with diminishing returns later in a session. Overall, the agent was highly effective as an optimization engine but human expertise and domain knowledge remained essential for guiding direction and interpreting experimental results.\n\n## Final thoughts\n\nBy combining a smaller architecture, open data, and a transfer-focused evaluation approach, Yan was able to build a model that is both more efficient and more practical to deploy.\n\nThis work showed that creating quality audio embeddings does not require massive models or restricted datasets. However, this still remains a difficult open problem. The current system should be viewed as a working research platform rather than a finished edge model.\n\nJust as importantly, this project highlighted a new research pattern. Agents can dramatically accelerate iteration but meaningful progress still depends on human insight, especially when navigating trade-offs and defining the right objectives.\n\nThe result was a compelling foundation for building efficient, deployable audio systems, and a glimpse at how human and machine collaboration can reshape the research process.\n\nYan and the research team continue to work on this new model; we'll share more details as that progresses.", "url": "https://wpnews.pro/news/our-research-team-developed-a-more-efficient-audio-embedding-model", "canonical_source": "https://www.edgeimpulse.com/blog/how-our-research-team-developed-a-more-efficient-audio-embedding-model/", "published_at": "2026-07-30 11:08:50+00:00", "updated_at": "2026-07-30 11:22:22.624717+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-agents", "ai-tools"], "entities": ["Yan Li", "BYOL-A", "Audio-JEPA", "EAT", "AudioSet", "VGGish", "PANNs", "YAMNet"], "alternates": {"html": "https://wpnews.pro/news/our-research-team-developed-a-more-efficient-audio-embedding-model", "markdown": "https://wpnews.pro/news/our-research-team-developed-a-more-efficient-audio-embedding-model.md", "text": "https://wpnews.pro/news/our-research-team-developed-a-more-efficient-audio-embedding-model.txt", "jsonld": "https://wpnews.pro/news/our-research-team-developed-a-more-efficient-audio-embedding-model.jsonld"}}