Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment

Researchers proposed a framework that aligns self-supervised respiratory audio encoders with medical terminology using a medical LLM to synthesize structured reports, enabling zero-shot classification. Across 9 tasks on 6 datasets, the method achieved a 61.3% mean zero-shot AUC, surpassing CLAP (51.4%) and Qwen2-Audio (54.9%), and reached the highest linear probing AUC (71.6%) with only 43% of the data used by full-scale baselines.

arXiv:2609.00055v1 Announce Type: new Abstract: Self-supervised respiratory encoders lack semantic grounding in clinical domain needed for zero-shot inference, limiting their utility without task-specific labeled data. We propose a framework that aligns these encoders with medical terminology in a shared latent space turning them into a zero-shot-capable foundation model. To address paired data scarcity, we use a medical LLM to synthesize structured reports from metadata, creating dense semantic anchors for contrastive learning. Our training combines a sigmoid-based contrastive loss with encoder's native SSL objective and similarity-aware negative sampling to sharpen pathological boundaries. Across 9 tasks on 6 datasets, our method achieves a 61.3% mean zero-shot AUC, surpassing CLAP 51.4% and Qwen2-Audio 54.9% while reaching the highest linear probing AUC 71.6% with only 43% of data used by full-scale baselines, showing that structured semantic alignment outperforms large-scale, general-purpose models in clinical diagnostics.