{"slug": "google-s-4-7x-qwen-3-5-speedup-is-a-sharding-story", "title": "Google's 4.7x Qwen 3.5 Speedup Is a Sharding Story", "summary": "Google engineers reported a 4.7x faster prefill and 3.1x faster decode for Qwen 3.5-397B-A17B on Ironwood TPUs between April and June, achieved by using data parallelism for attention layers and expert parallelism for MoE layers, a topology also recommended by vLLM for 8x H200 GPUs. The model's 2 KV heads and 512 experts made tensor parallelism inefficient, so Google adopted DP=8 and EP=8, with communication overhead reduced via fused FP8 all-gather and hierarchical reduce-scatter.", "body_md": "[AI](https://sourcefeed.dev/c/ai)Article\n\n# Google's 4.7x Qwen 3.5 Speedup Is a Sharding Story\n\nTwo KV heads and 512 experts pushed Ironwood onto DP+EP, the same topology vLLM now recommends for GPUs.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\nGoogle's engineers put out a long, unusually candid write-up on getting [Qwen 3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) to run fast on Ironwood, and the headline number — 4.7x faster prefill, 3.1x faster decode between April and June — is the least interesting thing in it. That's a \"versus our own first bring-up\" figure, measured on a single four-chip host, with no GPU on the other side of the table. What's actually worth your attention is *why* the fast configuration looks the way it does, because the reasoning applies whether or not you'll ever touch a TPU.\n\n## The model dictated the topology\n\nQwen 3.5's big MoE is an odd shape. Sixty layers, three-quarters of them Gated DeltaNet linear attention and one-quarter conventional grouped-query attention. The GQA layers have 32 query heads but only **2 KV heads** at head dimension 256. The MoE layers have **512 experts**, top-10 routing plus one shared expert, each with a skinny 1,024-wide intermediate.\n\nNow try to serve that with the default everyone reaches for: tensor parallelism across 8 devices. Two KV heads over eight devices is a quarter of a head each. You can't do it without duplicating KV heads — which wastes memory bandwidth on exactly the tensor decode is bottlenecked on — or padding, which wastes compute. Sharding 512 tiny experts tensor-wise is similarly bad: each device ends up doing a sliver of every expert's matmul, and the MXUs starve.\n\nSo Google ran the attention layers **data-parallel** (DP=8, weights replicated, each logical device owns whole requests) and the MoE layers **expert-parallel** (EP=8, 64 whole experts per device). Attention weights in a 17B-active model are small enough that replication costs almost nothing; the 380B or so of expert weights get split cleanly. The price is an all-to-all between every attention block and every MoE block — twice per MoE layer, so on the order of 120 per token — and that communication is where most of the reported engineering hours went: fusing the routing indices and gating weights into a single FP8 all-gather instead of three collectives, and a hierarchical reduce-scatter that does the intra-chip half through shared memory before touching the ICI.\n\nHere's the tell that this isn't TPU-specific: the [vLLM recipe](https://docs.vllm.ai/projects/recipes/en/stable/Qwen/Qwen3.5.html) for the same model on 8x H200 is `-dp 8 --enable-expert-parallel`\n\n. Same topology, different silicon. Tensor parallelism is still what the model card suggests (`--tensor-parallel-size 8`\n\n), and it still works, but for a 2-KV-head, 512-expert model it has quietly stopped being the throughput answer on either vendor's hardware.\n\n## What Ironwood adds, and what it doesn't\n\nThe [TPU7x](https://docs.cloud.google.com/tpu/docs/tpu7x) numbers matter for sizing: 192 GiB of HBM per chip at 7.38 TB/s, 2,307 BF16 / 4,614 FP8 TFLOPS, four chips per VM, and JAX exposes each chip as two devices (one per chiplet) — hence \"DP=8\" on a 4-chip host. Four chips is 768 GiB, so a ~400 GB FP8 checkpoint fits with roughly half the memory left for KV cache and GDN recurrent state. That's the entire model on one VM with no multi-host collectives, which is a genuinely nice place to be for a 397B model.\n\nTwo Ironwood-specific tricks are worth knowing about. First, the MoE gather/scatter runs on the SparseCores — Ironwood has four per chip — so the ragged token-to-expert routing doesn't get padded into dense shapes for the TensorCores; Google reports HBM reads for that step dropping from 20 passes to 10 and writes from 15 to 5. Second, the whole GDN block (causal conv1d plus the recurrent delta-rule update) was fused into one Pallas kernel and its recurrent state moved from FP32 to BF16, which they say doubled vector throughput. Whether BF16 state holds up at 260K-token contexts is the kind of thing you'd want to test yourself; the post describes a numerical verification layer watching softmax ranges and expert load balance, which tells you they were worried about it too.\n\nThe results, on the 8K-in/1K-out prefill-heavy shape at concurrency 64: 3,707 tokens/s/chip against a de-rated roofline of 4,500 — 82% of peak. Decode-heavy (1K in / 8K out): 677 tokens/s/chip against 850, about 80%, with a 16.36 ms step time, so roughly 61 tokens/s per stream. Hitting 80% of roofline on a hybrid linear-attention MoE within three months of the chip's April general availability is the real achievement. The 4.7x is mostly a statement about how rough the April baseline was.\n\n## Where the claim is thin\n\nNo GPU comparison. Google frames Ironwood against Blackwell in passing (192 GB vs 288 GB on GB300) and then never benchmarks against it. Every number is tokens-per-chip on one host; there's nothing about multi-host scaling, nothing about disaggregated prefill/decode (which [tpu-inference](https://github.com/vllm-project/tpu-inference) added for Qwen 3.5 in its May release), and nothing about cost per million tokens — Google still hasn't published list pricing for Ironwood, so as of midsummer nobody outside a sales conversation could compute that anyway. Treat this as an engineering report, not a buying guide.\n\nThe other caveat is that tpu-inference is moving fast enough that the post is already a snapshot. The changelog since June shows GDN speculative-decoding modes, context parallelism, and a JAX 0.11 bump; the exact flags and kernel behavior described in July may not match what you install today.\n\n## What to do with it\n\nIf you're on Ironwood: install `vllm-tpu`\n\n, start from the Qwen 3.5 DP+EP configuration rather than TP, and budget `--max-num-batched-tokens=1024 --max-num-seqs=64`\n\nper core as the starting point Google tuned around. Measure tokens/s/chip and compare against their 3,707 / 677 numbers — if you're far below, look at KV page size first (they went from 16 to 256 tokens and cut the attention kernel from 428 µs to 283 µs).\n\nIf you're on GPUs: the same lesson applies. Read the config before picking parallelism. Count KV heads, count experts, look at the expert intermediate width. The generation of open models shipping now — Qwen 3.5, DeepSeek V4, Kimi — are converging on KV-light attention (linear hybrids, MLA, two-head GQA) and hundreds of small experts, and that shape wants DP attention plus EP with a fast all-to-all, not the TP=8 muscle memory left over from dense Llama.\n\nThe thing Google is really selling here is the reusable kernel library — the post frames Qwen 3.5 as one application of a general playbook, and tpu-inference's changelog shows DeepSeek V4 bring-up landing on the same SparseCore MoE kernels a few weeks later. That's the credible part of the pitch. Whether it's cheaper than the GPU you already have is the question they carefully didn't answer.\n\n## Sources & further reading\n\n-\n[Systems Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood (TPU7x)](https://developers.googleblog.com/systems-engineering-playbook-optimizing-qwen-35-397b-moe-on-ironwood-tpu7x/)— developers.googleblog.com -\n[Qwen/Qwen3.5-397B-A17B model card](https://huggingface.co/Qwen/Qwen3.5-397B-A17B)— huggingface.co -\n[TPU7x (Ironwood) documentation](https://docs.cloud.google.com/tpu/docs/tpu7x)— docs.cloud.google.com -\n[Qwen3.5 and Qwen3.6 Usage Guide](https://docs.vllm.ai/projects/recipes/en/stable/Qwen/Qwen3.5.html)— docs.vllm.ai -\n[vllm-project/tpu-inference releases](https://github.com/vllm-project/tpu-inference/releases)— github.com -\n[Qwen3.5: Nobody Agrees on Attention Anymore](https://huggingface.co/blog/mlabonne/qwen35)— huggingface.co -\n[Google Cloud's Ironwood ready for general availability](https://www.constellationr.com/blog-news/insights/google-clouds-ironwood-ready-general-availability)— constellationr.com\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/google-s-4-7x-qwen-3-5-speedup-is-a-sharding-story", "canonical_source": "https://sourcefeed.dev/a/googles-47x-qwen-35-speedup-is-a-sharding-story", "published_at": "2026-08-26 17:09:58+00:00", "updated_at": "2026-08-26 17:13:30.304071+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["Google", "Qwen 3.5-397B-A17B", "Ironwood", "vLLM", "H200", "TPU7x", "JAX", "Pallas"], "alternates": {"html": "https://wpnews.pro/news/google-s-4-7x-qwen-3-5-speedup-is-a-sharding-story", "markdown": "https://wpnews.pro/news/google-s-4-7x-qwen-3-5-speedup-is-a-sharding-story.md", "text": "https://wpnews.pro/news/google-s-4-7x-qwen-3-5-speedup-is-a-sharding-story.txt", "jsonld": "https://wpnews.pro/news/google-s-4-7x-qwen-3-5-speedup-is-a-sharding-story.jsonld"}}