DeepSeek V4 Flash on One AMD GPU Took Nine Patches

A single AMD MI300X GPU with 192 GB of HBM3 now serves DeepSeek's 284B-parameter DeepSeek-V4-Flash-0731 checkpoint in mixed FP4+FP8 format, requiring nine patch overlays against a vLLM ROCm nightly plus custom AITER tuning tables, according to a GitHub repo by Ryan Zhou. The deployment fits 156.67 GiB of weights and a 20 GB FP8 KV cache within the MI300X's memory, with overflow spilling to a 96 GiB CPU tier, highlighting AMD's hardware capacity advantage for memory-bound MoE inference while exposing gaps in ROCm software support, as the official vLLM recipe still does not list the MI300X as supported.

AI https://sourcefeed.dev/c/ai Article DeepSeek V4 Flash on One AMD GPU Took Nine Patches A single 192 GB MI300X now serves the full checkpoint — proof of AMD's hardware edge and its lingering software gap. Priya Nair https://sourcefeed.dev/u/priya nair DeepSeek's official serving recipe for DeepSeek-V4-Flash-0731 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 assumes a four-GPU Nvidia Blackwell node. This week a config landed on GitHub that serves the same checkpoint — as shipped, no further quantization, no layer offload — on a single AMD MI300X https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html . That's a genuinely interesting datapoint for anyone weighing AMD for inference. It's also, if you read the patch list, a fairly damning progress report on how much of ROCm's last mile still gets built by people who don't work at AMD. The capacity math finally lines up V4 Flash is a 284B-parameter mixture-of-experts model with 13B active per token Hugging Face reports 304B because the checkpoint ships with a speculative-decoding draft module attached . Crucially, DeepSeek ships it in a mixed FP4+FP8 format — expert weights in FP4, attention and router in FP8 — so the whole thing is about 149 GiB on disk. That number is the story. It's too big for an H100's 80 GB or an H200's 141 GB, which is why the official vLLM recipe https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash reaches for four H200s or a GB200 NVL4 tray. But it slides neatly under the MI300X's 192 GB of HBM3. The repo https://github.com/ryanzhou/deepseek-v4-flash-mi300x , published by Ryan Zhou, exploits exactly that margin: 156.67 GiB of weights in HBM, a 20 GB FP8 KV cache beside them, and peak utilization within about a gigabyte of the ceiling. KV overflow spills to a 96 GiB CPU tier, which is why the host needs roughly 235 GiB of system RAM. It's a tight fit, but it's a single-GPU fit for a frontier-class open checkpoint — the kind of deployment MoE-era capacity constraints were supposed to make extinct. This is the argument AMD has been making since the MI300X launched: for memory-bound MoE serving, HBM capacity per GPU is the binding constraint, and AMD sells more of it. The argument keeps getting stronger — the MI325X carries 256 GB, the MI355X 288 GB. What keeps undercutting it is everything below the hardware. What "AMD support" meant in practice When V4 landed, vLLM's v0.21.0 release notes claimed DeepSeek V4 AMD/ROCm support. The stock ROCm image then crashed on AMD hardware during startup with "PDL is not supported" — a TileLang kernel referencing Programmatic Dependent Launch, a CUDA-only Hopper feature — and the official recipes repository marked every AMD SKU as unsupported. That contradiction sat in vLLM's issue tracker costing people debugging hours. As of vLLM 0.25, the official recipe validates the MI325X, but only at tensor-parallel 1 with 4K context. The MI300X — the AMD accelerator you can actually rent by the hour almost anywhere, at $1.99/hr on AMD's own Developer Cloud — still isn't in the official support matrix at all. Zhou's repo is what filling that gap actually takes: nine patch overlays against a digest-pinned vLLM ROCm nightly, plus hand-built tuning tables for AITER https://github.com/ROCm/aiter , AMD's kernel library. The individual fixes are a tour of ROCm's sharp edges. The MI300X's gfx942 silicon implements the older FNUZ variant of FP8 E4M3 rather than the OCP standard the checkpoint assumes, so every FP8 path needs format conversion with a 16×16 preshuffle. A bitmatrix padding bug corrupted MoE routing under concurrent load — surfacing, nastily, as garbled tool names in agentic workloads, exactly the workload this model is tuned for. Twenty-one GEMM shapes were untuned for gfx942; adding them was worth a claimed 42–62% on decode. And the CPU-to-GPU KV reload path needed an explicit fence to stop in-flight compute from reading half-loaded cache. None of this is exotic engineering. All of it is table stakes that CUDA users get from the box, and it echoes a pattern: when the Flash preview dropped in the spring, it took a separate community bring-up Fergus Finn's, with a patch from Doubleword to get it running on MI300X at all. Every DeepSeek release so far has needed an outsider to finish the AMD port. The numbers, with asterisks Zhou reports 168.6 tok/s single-stream decode with the bundled speculative decoder, ~542 tok/s aggregate across 8 concurrent streams about 90 tok/s per user , 830 tok/s in a 64-stream burst, and 7–8.5K tok/s prefill, with 256K context validated of the architecture's 1M. Those are author-reported numbers from a brand-new repo; nobody has independently reproduced them yet, so treat them as a ceiling claim, not a benchmark. They're also not competitive with serious Nvidia deployments, and the Hacker News thread was right to say so. DeepSeek's own infrastructure reportedly pushes ~15K tok/s per H800 with the same speculative-decoding scheme, and commenters pegged tuned SGLang/Dynamo setups at multiples of this per GPU. At $1.99/hr, one MI300X sustaining ~500 aggregate tok/s doesn't clear the bar for reselling tokens against DeepSeek's own aggressively priced API. If your business is selling inference, this config is not your stack. Who this is actually for The economics change completely if the tokens aren't for sale. V4 Flash is MIT-licensed, tuned hard for agentic coding 82.7 on Terminal Bench 2.1, up from 61.8 for the preview , and it's the open checkpoint teams most want to self-host right now — for code that can't leave the building, for regulated industries, for agents you want on your own metal. For that buyer, "one GPU, one Docker Compose file, original weights" is the entire pitch. The unit of self-hosting drops from a four-GPU Blackwell node to a single rentable accelerator, and 168 tok/s single-stream is comfortably interactive; 90 tok/s under 8-way load is fine for background agents. The trade-offs are real: you're running a digest-pinned nightly with nine local patches, which means you own every upgrade and every CVE until this work lands upstream. That's the number to watch, and it's a better AMD health metric than any benchmark — whether these fixes merge into vLLM and AITER in weeks, or sit in a third-party repo the way AMD enablement usually has. Call it half a win for AMD. The hardware thesis — capacity-per-GPU decides MoE serving — just got its cleanest demonstration yet. But the day AMD's inference stack is a real Nvidia alternative is the day a DeepSeek checkpoint runs on a MI300X the afternoon it ships, with zero overlay patches. We're visibly closer than a year ago. We're not there. Sources & further reading - DeepSeek V4 Flash on a Single AMD MI300X https://github.com/ryanzhou/deepseek-v4-flash-mi300x — github.com - DeepSeek V4 Flash on a Single AMD MI300X - discussion https://news.ycombinator.com/item?id=49166386 — news.ycombinator.com - deepseek-ai/DeepSeek-V4-Flash-0731 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 — huggingface.co - DeepSeek-V4-Flash official serving recipe https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash — recipes.vllm.ai - DeepSeek-V4-Flash fails on AMD despite claimed ROCm support https://github.com/vllm-project/vllm/issues/42876 — github.com - DeepSeek Releases Official V4-Flash Model as China's AI Race Intensifies https://www.caixinglobal.com/2026-08-01/deepseek-releases-official-v4-flash-model-as-chinas-ai-race-intensifies-102470292.html — caixinglobal.com - Bringing Up DeepSeek-V4-Flash on AMD MI300X https://news.ycombinator.com/item?id=48373675 — news.ycombinator.com Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.