{"slug": "diffusion-gemma-asr-15x-faster-than-whisper", "title": "Diffusion-gemma-asr: 15x Faster Than Whisper", "summary": "A new automatic speech recognition model, Diffusion-gemma-asr, achieves 15x faster inference than OpenAI's Whisper by freezing a Whisper-small encoder and DiffusionGemma weights and adding a 42M parameter adapter (0.16% of the model). The model, trained on 219 hours of audio, reaches a 6.6% word error rate on Librispeech and demonstrates that a frozen diffusion large language model can acquire a new modality with minimal adaptation.", "body_md": "# Diffusion-gemma-asr: 15x Faster Than Whisper\n\nThe technical flex here is actually hilarious: they froze almost everything. They used a frozen Whisper-small encoder and frozen DiffusionGemma weights, then just slapped on a tiny 42M parameter adapter (which is like 0.16% of the model). They used a convolutional projector to squash 1,500 audio frames into 188 tokens, and some LoRA adapters to make sure the model didn't just ignore the audio entirely.\n\nHere is the high-level AI workflow they used for the training:\n\n1. **The Scaffold:** They used CTC loss as a \"training scaffold\" to force the projector to actually learn something before the attention layers decided it was useless.\n\n2. **The Process:** DiffusionGemma doesn't use mask tokens; it starts with a canvas of random vocab tokens and iteratively replaces the \"uncertain\" ones until the text settles.\n\n3. **The Result:** After training on Librispeech and fine-tuning on FLEURS and VoxPopuli, they hit a 6.6% WER on Librispeech.\n\nSure, it's still not beating the massive Whisper models because those were trained on millions of hours of audio while this one only saw about 219 hours. But the fact that a frozen diffusion LLM can pick up a whole new modality with a tiny adapter is a massive win for anyone trying to build lean, fast LLM agents.\n\nIf you want to poke around the implementation, the adapter weights and inference scripts are open source.\n\n```\nhttps://huggingface.co/spaces/interfaze-ai/diffusion-gemma-asr-demo\n```\n\n[Next TypeScript Recursive Types: A Deep Dive →](/en/threads/3125/)", "url": "https://wpnews.pro/news/diffusion-gemma-asr-15x-faster-than-whisper", "canonical_source": "https://promptcube3.com/en/threads/3133/", "published_at": "2026-07-25 09:04:06+00:00", "updated_at": "2026-07-25 09:37:56.223624+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["Diffusion-gemma-asr", "Whisper", "DiffusionGemma", "OpenAI", "Librispeech", "FLEURS", "VoxPopuli", "interfaze-ai"], "alternates": {"html": "https://wpnews.pro/news/diffusion-gemma-asr-15x-faster-than-whisper", "markdown": "https://wpnews.pro/news/diffusion-gemma-asr-15x-faster-than-whisper.md", "text": "https://wpnews.pro/news/diffusion-gemma-asr-15x-faster-than-whisper.txt", "jsonld": "https://wpnews.pro/news/diffusion-gemma-asr-15x-faster-than-whisper.jsonld"}}