The technical flex here is actually hilarious: they froze almost everything. They used a frozen Whisper-small encoder and frozen DiffusionGemma weights, then just slapped on a tiny 42M parameter adapter (which is like 0.16% of the model). They used a convolutional projector to squash 1,500 audio frames into 188 tokens, and some LoRA adapters to make sure the model didn't just ignore the audio entirely.
Here is the high-level AI workflow they used for the training:
-
The Scaffold: They used CTC loss as a "training scaffold" to force the projector to actually learn something before the attention layers decided it was useless.
-
The Process: DiffusionGemma doesn't use mask tokens; it starts with a canvas of random vocab tokens and iteratively replaces the "uncertain" ones until the text settles.
-
The Result: After training on Librispeech and fine-tuning on FLEURS and VoxPopuli, they hit a 6.6% WER on Librispeech.
Sure, it's still not beating the massive Whisper models because those were trained on millions of hours of audio while this one only saw about 219 hours. But the fact that a frozen diffusion LLM can pick up a whole new modality with a tiny adapter is a massive win for anyone trying to build lean, fast LLM agents.
If you want to poke around the implementation, the adapter weights and inference scripts are open source.
https://huggingface.co/spaces/interfaze-ai/diffusion-gemma-asr-demo
Next TypeScript Recursive Types: A Deep Dive →