{"slug": "throughputmaxxing-deepseek-v4-flash-on-isambard-ai", "title": "Throughputmaxxing DeepSeek-V4-Flash on Isambard-AI", "summary": "Doubleword, one of six companies in the first wave of UK Sovereign AI investments, achieved up to 3× the throughput of vLLM for DeepSeek-V4-Flash on a single node of Isambard-AI, the UK's national AI supercomputing facility. The node consists of 4xGH200 chips, each equivalent to an H100 with about 96GiB VRAM, connected via NVLink 4. The optimization, detailed in a blog post, involved tweaking vLLM settings for the 4xGH200 node shape, enabling FP8 KV cache, and using NUMA binding to maximize throughput.", "body_md": "# Throughputmaxxing DeepSeek-V4-Flash on Isambard-AI\n\nIt turns out inference optimization is still hard for agents:\n\nAcross 15 frontier agent configurations, agents reliably improve over a naïve PyTorch baseline (up to 8.08×) and often match or exceed serving engines with default settings (4.05× for vLLM), but still fall below a simple hyperparameter search under the same time budget (up to 11.53×) From this recent\n\n[paper].\n\nGiven that novel maths discoveries are\n[not](https://openai.com/index/model-disproves-discrete-geometry-conjecture/)\nso [hard](https://x.com/__alpoge__/status/2079028340955197566), we probably\nshouldn’t assume this will last forever.\n\nThis article then is an offering to our coming AI overlords: one last echo of\nthe age of [centaurs](https://en.wikipedia.org/wiki/Advanced_chess), before the\nhorses gallop off beyond the horizon into fully automated research.\n\nDoubleword was recently named as one of six companies in the first wave of UK\nSovereign AI investments[Our first investments](https://sovereignai.gov.uk/post/our-first-investments), UK Sovereign AI., which has given us access to\n[Isambard-AI](https://blogs.nvidia.com/blog/isambard-ai/), the UK’s national AI\nsupercomputing facility.\n\nThis post is about how to use a single node of that capacity to push DeepSeek-V4-Flash to about 3× what vLLM gives you off the shelf.\n\n## What do we have\n\nEach node in Isambard-AI consists of\n4x[GH200](https://www.nvidia.com/en-gb/data-center/grace-hopper-superchip/)\nchips. Each Hopper chip is equivalent in performance to an H100Not, as you might think, an H200, although there is a GH200 SKU whose GPU is equivalent to an H200., but we have a\nbit more VRAM (about 96GiB). The GPUs are connected with [NVLink\n4](https://blog.doubleword.ai/nvlink-scale-up) P2P connections (no switch),\nwith (on paper) 150GB/s of bandwidth peer to peer. GPUs are connected to the\nLPDDR5x on the host with NVLink C2C, which interestingly on this machine, is\nactually higher bandwidth than the connection between GPUs.\n\n## A first cut\n\nFirst, make it work.\n\nWe’re focused on throughput at all costs. There’s no vLLM\n[recipe](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash) for 4xGH200s,\nso we pick the shape that vLLM suggests for 8xH200s, tweaked for our node\nshape, we’ll change it later.\n\nWe want tool calling and reasoning to be parsed out properly. DeepSeek has a\nnative FP8 KV cacheDeepSeek-V4 [technical report](https://arxiv.org/abs/2606.19348): “Both\npaths use FP8 storage for most KV entries and BF16 only for the RoPE\ndimensions.” We’ll see that KV cache size is a big factor when, as we’re doing\nhere, we’re optimizing for throughput., so enable that. `--numa-bind`\n\nis important on\nmulti-socket chips like the ones on IsambardThe node is 4 GH200 Grace-Hopper pairs, with all 4 Grace chips connected\nvia a CPU-side interconnect and appearing as a single multi-socket CPU. If we\ndon’t bind the engines to their NUMA node, then comms between the host and\ndevice go over the interconnect and we lose throughput..\n\n```\nvllm serve deepseek-ai/DeepSeek-V4-Flash \\\n  --trust-remote-code \\\n  --tensor-parallel-size 4 \\\n  --enable-expert-parallel \\\n  --kv-cache-dtype fp8 \\\n  --block-size 256 \\\n  --gpu-memory-utilization 0.92 \\\n  --max-model-len 128000 \\\n  --max-num-batched-tokens 8192 \\\n  --max-num-seqs 2048 \\\n  --tokenizer-mode deepseek_v4 \\\n  --tool-call-parser deepseek_v4 \\\n  --enable-auto-tool-choice \\\n  --reasoning-parser deepseek_v4 \\\n  --numa-bind\n```\n\nBenchmark is `vllm bench serve`\n\n, completions, random ISL/OSL = 1024/1024,\nrequest rate = inf.\n\n```\nvllm bench serve \\\n  --backend openai \\\n  --model deepseek-ai/DeepSeek-V4-Flash \\\n  --endpoint /v1/completions \\\n  --dataset-name random \\\n  --num-prompts <2 × conc> \\\n  --request-rate inf \\\n  --max-concurrency <conc> \\\n  --random-input-len 1024 \\\n  --random-output-len 1024 \\\n  --ignore-eos\n```\n\nConcurrency is a free parameter: we want it as high as possible. To reliably determine the maximum throughput that the model can serve, we sweep the concurrency up until the KV cache saturates. For this setup, we get to concurrency .\n\nThe run completes 4,096/4,096 with the KV cache pinned at 100%An aside: agents find this terrifying, and think that the KV cache being\nfull will cause the engine to crash. Why?. **5,856\noutput tokens/second**, 1,464 per GPU.\n\n## Changing the shape: Data Parallel Attention\n\nTensor parallelism is the wrong shape for attention on DeepSeek models.\n\nThe reason is Multi-head Latent AttentionDeepSeek-V3 [technical report](https://arxiv.org/abs/2412.19437). For\nsome intuition, Jamie’s post on [tensor network\nattention](https://blog.doubleword.ai/tensor-network-attention#multi-head-latent-attention)., original to DeepSeek, an\nattention variant that reduces the amount of stored KV cache by compressing the\nKV heads into a single shared “latent” vector.\n\nThe problem is that for Tensor Parallel Attention, the head dimension is the useful dimension to shard over — we usually compute different attention heads on different accelerators. With only one shared K/V latent, naive tensor parallelism for MLA has to replicate the KV cache on each accelerator. The KV cache then has to live in N copies, where N is the parallelism degree.\n\nFor DeepSeek, the cleaner shape is *data-parallel attention*. If you replicate\nyour attention layers on each accelerator, then each can have its own KV cache.\nOne nice side effect: every per-token kernel (elementwise ops, FP8 quant,\nsampling) now runs on 1/N of the batch per rank, instead of the full global\nbatch on every rank as it would under TP.\n\nThe drawback is that the weights get replicated, which claws back a little KV cache. But for high-throughput or long-context inference, for models with relatively small attention weights, that’s much better than replicating the KV cache times.\n\nThe diff, and the results:\n\n```\n- --tensor-parallel-size 4\n+ --tensor-parallel-size 1\n+ --data-parallel-size 4\n- --gpu-memory-utilization 0.92\n+ --gpu-memory-utilization 0.95\n- --max-num-seqs 2048\n+ --max-num-seqs 1024   # per DP rank — raise the cap + graphs to reach the KV knee\n+ --compilation-config '{\"max_cudagraph_capture_size\":1024}'\n```\n\n**12,802 output tokens/second at c2752**, 3,201 per GPU — 2.2× the baselineMaximum concurrency (that saturates the KV cache) should always give\nhigher throughput, all else being equal. In practice, this often ends up not\nbeing the case, because all else is not equal..\n\n## A pass through the model\n\nOnce the model’s been hammered into shape, the process of inference optimization looks like a back and forth between reasoning, profiling, optimizing, and repeating.\n\nLet’s take a first shot. We picked a benchmark with ISL=1024 and OSL=1024. Because prefill is so parallelisable, the time taken is going to be dominated by decode: that is, generating 1024 output tokens, one by one. So we ought to focus on that steady decode state: wherein all the sequences have finished prefilling, and we’re just working on sequences in a decode batch.\n\nIf you hit the server with the torch profiler, you see that the model’s forward pass is dominated by the MoE expert GEMMs.\n\nWhich points at an obvious target. The MoE expert weights — most of the model’s 284B parameters — ship in fp4. If we were running Blackwell GPUs, this would be really nice: these would run natively on the fp4 tensor cores. On the Hopper generation, with only fp8, we’ve got to do it in softwareOr just upcast them to fp8 in VRAM, but that would be a waste of VRAM..\n\nvLLM’s default on Hopper is conservative about how it does this: it picks\n[Marlin](https://github.com/IST-DASLab/marlin), a weight-only kernel that\ndequantizes the fp4 weights to bf16 and runs the GEMMs on the GPU’s bf16 tensor\ncores. On Hopper, we’ve got fp8 tensor cores. Can we do fp8?\n\nThe fear we should have is correctness: providers offering models to the public\nought to want the model’s output to match what it was trained to be, not just\nsome vibe-eval hand-wave. DeepSeek’s technical report is pretty specific about\nthe correct arithmetic for those weights: during quantization-aware training,\nthe FP4 weights are dequantized (losslessly) back to FP8 and the forward pass\nruns through DeepSeek’s standard FP8 pipeline — e4m3 activations, scaled per\ntoken per 128 elementsDeepSeek-V4 [technical report](https://arxiv.org/abs/2606.19348), §5.2.1:\nmaster weights are “first quantized to FP4, then dequantized back to FP8 for\ncomputation”, reusing “the existing FP8 training framework” — the DeepSeek-V3\nrecipe of 1×128 activation tiles with FP32 scales..\n\nSo there’s no correctness reason to use Marlin. Is there a performance reason? It depends on the batch size. On the datasheet, the bf16 ridge point on the GH200 arrives at ~250 FLOPs per byte: 990 TF/s of bf16 against 4 TB/s of HBM. But you should always check this stuff. If you run a high intensity kernel on Isambard, you’ll see the clocks come down once the GPU draws ~530–570 W, and sustained bf16 tops out at 583 TF/s — nowhere near the datasheet number. This is a software power cap, not a thermal limit.\n\nSo the roofline for bf16 is actually ~146 FLOP/byte. From some pen & paper, this means we can have a ‘critical’ batch size of . Below that, the bf16 GEMMs are memory bound, and Marlin is as good as anything else. Our batch sizes are larger than . So we have a very rare thing: a MoE model that is compute bound at decode time.\n\nThe [Humming](https://github.com/inclusionAI/humming)One more property worth having: Humming’s FP8 path runs with\nbatch-invariant execution and Marlin’s doesn’t. kernels from the Ant\ngroup are the W4A8 kernels that we’re looking for: they upcast to fp8 in the\nSMs, and then do fp8xfp8 matmuls in the fp8 tensor cores. The weight transfer\nstays the same, but the available FLOPs doubles, and the critical batch size\ndoubles with it. Out of the box, didn’t work here, so we had to do some work on\nthe kernels to bring them up, and then to get the maximum performance out of\nthemTo get these running in vLLM took a kernel bug fix and a tuning fix,\nboth in Humming’s group-scaled path: the WGMMA consumer never applied the\nper-128 input scales on the accumulator, and once we fixed that, the extra\naccumulator it needs halves the per-tile register budget, so the tile\nheuristic’s default BlockM spills catastrophically. Once you fix the scales and\nmake the heuristic better, the performance recovers. Fork:\n[ doublewordai/humming](https://github.com/doublewordai/humming)..\n\n```\n+ --moe-backend humming\n+ VLLM_HUMMING_MOE_GEMM_TYPE=indexed\n+ VLLM_HUMMING_INPUT_QUANT_CONFIG='{\"dtype\":\"float8e4m3\",\"input_scale_group_size\":128}'\n```\n\nThe results:\n\nWe speed up our DP attention baseline by about 40%. Not bad!\n\n## Where does the time go?\n\nThere are many more steps along this path. Before closing, let’s take stock. The profiler breaks down a decode step:\n\nThe breakdown of the forward pass, created by parsing the output of the torch profiler at steady-state decode (per rank per step, at c2048) and mapping the kernels to their inferred roofline ceilings. All on the same scale, grayed out means current, solid means the minimal achievable time.\n\nThere’s lots of work to do! Even with our fixed-up GEMM kernels, the MoE bucket is still at 30% of its roof. Comms costs way more than it ought to: 10.8 ms against ~3ms at the wire rate. We’re wasting time on overhead everywhere: the act+quant bucket is ~520 tiny kernel launches per step, of which ~2 ms is actual data movement.\n\nA lot more exciting stuff comes onto the table as we scale out beyond a single\nnode. Disaggregated inference (where prefill runs on one node and decode on\nanother) makes this kind of kernel optimization much easier, since you can\nhammer each phase in isolation. Isambard has an [HPE\nslingshot](https://www.hpe.com/psnow/doc/a50002546enw) fabric: over which we’ve\nbuilt out [UCCL-P2P](https://github.com/uccl-project/uccl/pull/999) (and\ntherefore [NIXL](https://github.com/ai-dynamo/nixl)) support. And, [wide\nEP](https://www.lmsys.org/blog/2025-05-05-large-scale-ep/) over that fabric\npushes down the weights stored per node, and pushes up the available KV cache:\nsupport for DeepEP style comms over HPE slingshot also lands with a Slingshot\nbackend for [UCCL-EP](https://github.com/uccl-project/uccl/pull/997).\n\nBut that’s a multi-node story. We’ll leave it for the next post.\n\nLast modified: 23 Jul 2026", "url": "https://wpnews.pro/news/throughputmaxxing-deepseek-v4-flash-on-isambard-ai", "canonical_source": "https://fergusfinn.com/blog/throughputmaxxing-v4-flash-single-node/", "published_at": "2026-07-23 00:00:00+00:00", "updated_at": "2026-07-23 14:27:21.730982+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "ai-research"], "entities": ["Doubleword", "UK Sovereign AI", "Isambard-AI", "DeepSeek-V4-Flash", "vLLM", "NVIDIA GH200", "NVIDIA H100", "NVIDIA NVLink 4"], "alternates": {"html": "https://wpnews.pro/news/throughputmaxxing-deepseek-v4-flash-on-isambard-ai", "markdown": "https://wpnews.pro/news/throughputmaxxing-deepseek-v4-flash-on-isambard-ai.md", "text": "https://wpnews.pro/news/throughputmaxxing-deepseek-v4-flash-on-isambard-ai.txt", "jsonld": "https://wpnews.pro/news/throughputmaxxing-deepseek-v4-flash-on-isambard-ai.jsonld"}}