{"slug": "diffusiongemma-how-google-s-new-open-llm-hits-1000-tokens-sec-and-changes", "title": "DiffusionGemma: How Google's New Open LLM Hits 1,000 Tokens/sec and Changes Inference Economics", "summary": "Google DeepMind released DiffusionGemma, an open-source Apache 2.0-licensed diffusion-based large language model that generates text at over 1,000 tokens per second on a single H100 GPU, achieving up to 4x faster throughput than comparable autoregressive models. The 26B-parameter Mixture of Experts model, which fits in 18 GB of VRAM, iteratively refines entire 256-token blocks of noise rather than predicting one token at a time, trading some accuracy on complex reasoning tasks for significantly lower latency.", "body_md": "TL;DR:Google released DiffusionGemma, an open Apache 2.0 diffusion-based LLM that generates text up to 4x faster than autoregressive models, hitting 1,000+ tokens/sec on a single H100 and fitting in 18 GB VRAM. It trades some accuracy for speed. Here is what that means in practice.\n\nGoogle DeepMind released **DiffusionGemma**, the first production-grade open-weight model that applies discrete diffusion to text generation. The same family of techniques behind image generators like Stable Diffusion, now applied to language.\n\nInstead of predicting one token at a time left-to-right, DiffusionGemma fills a 256-token block with noise and **iteratively refines the entire block across multiple denoising passes** until confidence thresholds are met. It commits roughly 15-20 tokens per forward pass on average, not one.\n\nThis is a fundamentally different compute pattern from everything shipping in production today.\n\n| Metric | Value |\n|---|---|\nTokens/sec (H100, FP8, low batch) |\n1,100+ |\nTokens/sec (RTX 5090) |\n700+ |\nTotal parameters |\n25.2B (marketed as 26B) |\nActive parameters at inference |\n3.8B |\nMoE expert config |\n8 active / 128 total |\nVRAM required (quantized) |\n18 GB |\nCanvas (block) size |\n256 tokens |\nTokens committed per forward pass |\n~15-20 |\nMax denoising steps |\n48 |\nContext window |\n256K tokens |\nLicense |\nApache 2.0 |\n\nFor context: comparable autoregressive models on the same H100 generate roughly 200-250 tokens/sec. DiffusionGemma is up to **4x faster** on throughput. The jump comes from shifting the decode bottleneck from memory bandwidth to compute.\n\nDiffusionGemma is a **26B Mixture of Experts (MoE)** model built on the Gemma 4 backbone, but it replaces the autoregressive decoder with a **diffusion head**.\n\n**How a single generation works:**\n\nThe key difference from GPT-style models: token N can see tokens N+1 through N+256 during generation. This enables genuine **self-correction** across the block. Autoregressive models structurally cannot do this.\n\n| Benchmark | DiffusionGemma | Gemma 4 26B |\n|---|---|---|\n| MMLU Pro | 77.6% | 82.6% |\n| AIME 2026 | 69.1% | 88.3% |\n| GPQA Diamond | 73.2% | 82.3% |\n| MMMU Pro (Vision) | 54.3% | 73.8% |\n\nGoogle describes it as experimental. For reasoning-heavy workloads (complex math, multi-step logic, vision understanding) the autoregressive Gemma 4 is still ahead. DiffusionGemma is the right tool when **latency and throughput matter more than peak accuracy**.\n\nThe model processes interleaved text, images (5 resolution tiers up to 1120 tokens), and video (up to 60 seconds at 1 fps). It supports OCR, chart comprehension, screen understanding, and handwriting recognition across 35+ languages, with training data covering 140+ languages.\n\n```\npip install vllm\n\nvllm serve google/diffusiongemma-26B-A4B-it \\\n  --max-model-len 262144 \\\n  --max-num-seqs 4 \\\n  --gpu-memory-utilization 0.85 \\\n  --attention-backend TRITON_ATTN \\\n  --generation-config vllm \\\n  --hf-overrides '{\"diffusion_sampler\": \"entropy_bound\", \"diffusion_entropy_bound\": 0.1}' \\\n  --diffusion-config '{\"canvas_length\": 256}' \\\n  --enable-chunked-prefill\n```\n\nThe endpoint is OpenAI-compatible. Point your existing client at `http://localhost:8000`\n\nwith no other code changes needed.\n\n**Supported inference runtimes:** vLLM, Hugging Face Transformers, SGLang, MLX (Apple Silicon), NVIDIA NIM containers, Google Cloud Vertex AI Model Garden.\n\nThe ecosystem arrived fast for a day-1 release:\n\nA published case study fine-tuned DiffusionGemma on a Sudoku dataset and improved success rate from approximately 0% to 80%. Fine-tuning can also teach the model to stop denoising early when confidence is already high, reducing inference steps further. Autoregressive models have no equivalent lever.\n\n**This week:**\n\n**Next sprint:**\n\n**Architecture signal:**\n\nThis model is built on the same Gemini Diffusion research that will likely inform future proprietary Gemini releases. If diffusion inference stabilizes at this quality level, it rewrites autoregressive serving assumptions at scale.\n\nDiffusionGemma is not a production replacement for your current LLM stack today. Accuracy trade-offs are real and Google is transparent about the experimental status.\n\nBut the throughput numbers are genuine, the hardware requirements are accessible, and the license is Apache 2.0.\n\n**1,100 tokens per second. 18 GB VRAM. Open weights. From Google.**\n\nThat combination is worth benchmarking on your actual workload this week.\n\n**Resources:**\n\n*Found this useful? Follow for more signal-over-noise breakdowns of AI releases that matter.*", "url": "https://wpnews.pro/news/diffusiongemma-how-google-s-new-open-llm-hits-1000-tokens-sec-and-changes", "canonical_source": "https://dev.to/sayed_ali_alkamel/diffusiongemma-how-googles-new-open-llm-hits-1000-tokenssec-and-changes-inference-economics-4587", "published_at": "2026-06-12 18:30:19+00:00", "updated_at": "2026-06-12 18:43:47.307763+00:00", "lang": "en", "topics": ["large-language-models", "generative-ai", "ai-research", "ai-products", "ai-infrastructure"], "entities": ["Google DeepMind", "DiffusionGemma", "H100", "RTX 5090", "Apache 2.0"], "alternates": {"html": "https://wpnews.pro/news/diffusiongemma-how-google-s-new-open-llm-hits-1000-tokens-sec-and-changes", "markdown": "https://wpnews.pro/news/diffusiongemma-how-google-s-new-open-llm-hits-1000-tokens-sec-and-changes.md", "text": "https://wpnews.pro/news/diffusiongemma-how-google-s-new-open-llm-hits-1000-tokens-sec-and-changes.txt", "jsonld": "https://wpnews.pro/news/diffusiongemma-how-google-s-new-open-llm-hits-1000-tokens-sec-and-changes.jsonld"}}