{"slug": "real-world-ish-deepseek-v4-flash-performance-on-a-single-mi300x", "title": "Real world(ish) DeepSeek V4 Flash performance on a single MI300X", "summary": "A single AMD MI300X GPU can serve 32 concurrent coding agents running DeepSeek V4 Flash, delivering 582 generation tokens per second after tuning, according to a benchmark by developer Ryan Zhou. The 30-minute test with real agent workloads on open-source repos showed a 93-99% prefix cache hit rate, with the box renting at $2/hour. The practical ceiling is 32 agents, beyond which performance degrades.", "body_md": "# DeepSeek V4 Flash performance on a single MI300X\n\nWhat a single GPU actually delivers when the clients are\nautonomous coding agents doing real work, not synthetic benchmark load.\nBuilt on the open-source\n[deepseek-v4-flash-mi300x](https://github.com/ryanzhou/deepseek-v4-flash-mi300x)\nserving stack.\n\nMost published LLM serving numbers come from synthetic load: fixed prompt lengths, ShareGPT replays, evenly spaced arrivals. I wanted to know what a single MI300X delivers when the clients are coding agents working on real repositories, because that workload looks nothing like the benchmarks. Every number here comes from a 30-minute measured window tied to a recorded serving-config fingerprint.\n\nThe short version: one MI300X comfortably serves 32 concurrent coding agents on DeepSeek V4 Flash. Past 32 agents this configuration does not degrade gracefully, so I treat 32 as the practical ceiling.\n\n## The workload\n\nThe load generator is a bench harness I built for this: N lanes, each lane\nan [opencode](https://opencode.ai) agent session working on a clone\nof a real open-source repo (req, oban, supabase/realtime, livebook, flask,\npydantic, hono, excalidraw). Each lane picks up a generated task, works it\nlike a contributor would (reads code, edits, runs the test suite, commits),\nand either gets its change merged or times out. All inference goes through\none vLLM instance. Everything is scraped into Postgres every 5 seconds.\n\nWhat makes this workload different from synthetic load, measured during the 32-lane window:\n\n- The average request carries\n**53,194 prompt tokens** and generates just**390**. Agents resend their whole growing conversation every turn. - That only works because of prefix caching: the hit rate under steady load\nis\n**93 to 99%**. When it drops, everything drops. - Arrivals are bursty and phase-correlated. Agents block on their own test suites, then all come back at once.\n\nThe tasks are deliberately hard and most hit their timeout: a typical 30-minute window at 32 lanes completes ~2,600 agent turns, merges 3 or 4 tasks, and abandons ~25 at the timeout. This is a serving benchmark, not a coding-ability one; the tasks exist to generate honest load.\n\n## Hardware and stack\n\nOne AMD Developer Cloud box: 1× MI300X (192 GB HBM3), 20 vCPU, 236 GB RAM. The agents and their test suites run on the same box as the engine, which is realistic and occasionally consequential.\n\nServing is the pinned open-source\n[deepseek-v4-flash-mi300x](https://github.com/ryanzhou/deepseek-v4-flash-mi300x)\nstack: vLLM nightly `0.26.1rc1.dev229`\n\n(ROCm), DeepSeek V4 Flash\nat 262k context, MLA + DSA sparse attention, speculative decoding (~49%\ndraft acceptance, stable across every window), and a 96 GB CPU KV offload\ntier. Plus two sets of overlays from this repo: two upstream vLLM bugfixes\nfor a KV-zeroing kernel that was page-faulting the GPU under load, and the\nAITER GEMM tuning described below.\n\n## Concurrency sweep\n\n30-minute windows per lane count, engine warm, cache hit gated above 90% before each window starts. The first three configurations are the untuned stack; the fourth is the same 32-lane load after tuning, on a freshly provisioned box.\n\n| agents | gen tok/s | gen tok/s per agent | prompt tok/s | TTFT p50 | TTFT p99 | cache hit |\n|---|---|---|---|---|---|---|\n| 8 | 358 | 44.8 | 54,020 | ≤0.75s | ≤5s | 99% |\n| 16 | 448 | 28.0 | 75,517 | ≤1s | ≤5s | 98% |\n| 32 | 533 | 16.7 | 64,441 | ≤2.5s | ≤20s | 94% |\n| 32, tuned | 582 | 18.2 | 78,262 | ≤1s | ≤5s | 93% |\n\nThe GPU runs at 97% utilization drawing ~717 W during the 32-lane window, and the queue stays near empty: the engine is keeping up with demand, not saturating. Scaling from 8 to 32 lanes costs about half a second of median TTFT and buys 63% more generation throughput.\n\n## What it costs\n\nThe box rents for $2/hour. Against the tuned 32-agent window, that works out to:\n\nThe delivered prompt number is only that low because caching absorbs\nnearly everything. Of the 142M prompt tokens in the window, **93.0%**\ncame from the GPU prefix cache, **4.9%** from the CPU KV offload\ntier, and only **2.1%** were actually computed. Keep that in mind\nwhen comparing against per-token API pricing, where cached input is billed\ntoo; the computed-token price is the number that reflects work the GPU\nactually did.\n\n## What tuning bought\n\nThe stack ships AITER GEMM tuning for prefill sizes this workload never produces (the scheduler caps prefill chunks below every tuned shape), so ~70% of engine log lines were config-lookup misses falling back to default kernels. Fixing that was worth 18%: a scheduler config that lets prefill chunks reach a tuned size, an overlay that resolves a lookup miss to the nearest tuned shape, and a grid-tuning session that added 105 tuned shapes so the hop is always short. Result: zero config misses over a full window.\n\n## The ceiling is a cliff\n\nAt 48 lanes this configuration does not queue politely. The working set spills out of GPU KV cache, the offload tier starts churning 12+ GB/min of evictions, prefix cache hit rate falls from 93% to under 30%, and throughput collapses to a small fraction of the 32-lane rate rather than plateauing. Scaling agents past what the KV cache can hold is not a gradual trade-off on this stack, so capacity-plan to the cache, not the queue.\n\n## Reliability\n\nAcross the final sessions: hours of sustained full-utilization load, zero\nGPU page faults, zero engine restarts. Getting there required finding and\noverlaying two upstream vLLM fixes for a KV-block-zeroing kernel that wrote\npast its buffer roughly every 14 minutes under load. That investigation,\nincluding three refuted hypotheses and a 186 GB core dump that turned out to\nbe structurally useless, is\n[written up separately](https://github.com/matthusby/agent_coding_bench/blob/master/docs/investigations/gpu-page-fault.md).\n\n## Reproducing\n\nThe harness, overlays, tuned tables, and provisioning scripts are in\n[the repo](https://github.com/matthusby/agent_coding_bench) under\n`box/`\n\n. Every run's serving config is fingerprinted (image digest,\npatched-file checksums, engine args, model, versions) and stored with the\nmetrics, so any number above can be traced to the exact configuration that\nproduced it. The AITER tuning method and its pitfalls are\n[documented here](https://github.com/matthusby/agent_coding_bench/blob/master/docs/investigations/aiter-gemm-tuning.md).", "url": "https://wpnews.pro/news/real-world-ish-deepseek-v4-flash-performance-on-a-single-mi300x", "canonical_source": "https://matthusby.github.io/agent_coding_bench/", "published_at": "2026-08-21 12:21:09+00:00", "updated_at": "2026-08-21 12:44:31.725458+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["AMD MI300X", "DeepSeek V4 Flash", "vLLM", "Ryan Zhou", "opencode", "AMD Developer Cloud"], "alternates": {"html": "https://wpnews.pro/news/real-world-ish-deepseek-v4-flash-performance-on-a-single-mi300x", "markdown": "https://wpnews.pro/news/real-world-ish-deepseek-v4-flash-performance-on-a-single-mi300x.md", "text": "https://wpnews.pro/news/real-world-ish-deepseek-v4-flash-performance-on-a-single-mi300x.txt", "jsonld": "https://wpnews.pro/news/real-world-ish-deepseek-v4-flash-performance-on-a-single-mi300x.jsonld"}}