{"slug": "parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x", "title": "Parallelism for Long-Context Serving: GLM-5.3-Flash on MI300X", "summary": "A technical benchmark of GLM-5.3-Flash, a 320-billion-parameter sparse mixture-of-experts model with about 18 billion active parameters per token, on a single eight-GPU AMD Instinct MI300X node found that DP8+EP8 parallelism with an FP8 KV cache admits far more long-context sessions than TP8, because each request's cache is stored once rather than copied onto all eight GPUs. The FP8 checkpoint is about 306 GB, roughly a fifth of the node's 1,536 GB of HBM, and FP8 KV cache changed throughput by less than 2% while the BF16 pool could admit the workload, raising it once BF16 could no longer admit the next request. The tests covered three request shapes (1k in/1k out, 8k in/1k out, 96k in/32k out) with requests up to 131,072 tokens, against the model's 1,048,576-token maximum context.", "body_md": "# Parallelism for Long-Context Serving: GLM-5.3-Flash on MI300X[#](#parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x)\n\nGLM-5.3-Flash runs on a single eight-GPU AMD Instinct™ MI300X node. The FP8 checkpoint is about 306 GB, roughly a fifth of the node’s 1,536 GB of HBM. During inference, most of the rest of the memory will be taken by the KV cache. The KV cache holds the attention keys and values of every token a request has processed, so the model does not recompute them at each decoding step. Its footprint is the sum of the tokens held across all active requests, so it grows with request length and with the number of requests in flight. In long-context, high-concurrency serving it can exceed the weights as the largest consumer of HBM. When the pool is exhausted, the scheduler stops admitting requests; new arrivals queue, and throughput and latency degrade.\n\nTwo choices determine the throughput of a long-context, high-concurrency workload:\n\n**Which parallelism strategy?**\n\n- TP8 (tensor parallelism, width 8) keeps one copy of the model, sharded across the eight GPUs. A single request has the lower latency, because each layer’s compute is divided among them. Each GPU holds a full copy of every request’s KV cache, so the node’s cache capacity is that of one GPU.\n- DP8+EP8 (data parallelism and expert parallelism, both width 8) runs eight data-parallel ranks, each serving its own requests, and shards the routed experts across them. A single request is slower than on TP8, but each request’s cache is stored once instead of being copied onto all eight GPUs, so the node holds far more cached tokens and many more long sessions fit.\n\n**Which KV cache precision? This choice applies on top of either parallelism strategy:**\n\n- BF16 is the default.\n- FP8 stores the cached values in one byte instead of two, so the memory pool holds nearly twice as many tokens. In our tests it changed throughput by less than 2% while the BF16 pool could admit the workload, and raised it once the BF16 pool could no longer admit the next request.\n\nTo see how these choices play out for long contexts, this blog does three things:\n\n- It identifies which parts of GLM-5.3-Flash store a KV cache, since they set the per-token cost.\n- It works out how many sessions of a given length each parallelism strategy can hold: its KV-cache ceiling.\n- It measures output throughput against concurrency for TP8 and DP8+EP8, with BF16 and FP8 KV cache, on three request shapes (1k in/1k out, 8k in/1k out, 96k in/32k out), and reads the results against that ceiling.\n\n[The appendix](#appendix-measured-results) lists every measured point.\n\n## GLM-5.3-Flash[#](#glm-5-3-flash)\n\n[GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) is a sparse mixture-of-experts (MoE) model with a hybrid attention stack. It has 320 billion parameters in total and about 18 billion active on each token. The model supports a maximum context length of 1,048,576 tokens (1M). This blog serves requests up to 131,072 tokens (128K). The model used in this blog has weights in FP8 format.\n\nAs shown in Figure 1, GLM-5.3-Flash has forty-five layers. Two separate patterns run through them, and they are easy to confuse because both begin at layer 3:\n\n- **Attention type decides the KV cache.** Thirty-four layers use Kimi Delta Attention (KDA), a linear attention whose state does not grow with the prompt. The other eleven, layers 3, 7, 11 and so on through 43 (every fourth layer), use DeepSeek sparse attention with Multi-head Latent Attention (MLA). Only these eleven store a KV cache that grows with the prompt.\n- **Feed-forward type decides compute and weights.** Layers 0, 1 and 2 use a dense MLP. Layers 3 through 44, all 42 of them, use mixture-of-experts: each token is routed to 8 of the 288 experts, and one shared expert always runs with them.\n\n*Figure 1. Layers 0 through 44. Red layers store the token KV cache. Grey layers are linear attention and keep a fixed recurrent state.*\n\nThe table below summarizes how the 45 layers combine the two patterns.\n\n| Layers | Count | Attention | Stores a growing KV cache? | Feed-forward | \n|---|---|---|---|---|\n| 0, 1, 2 | 3 | KDA | No | Dense MLP | \n| 3, 7, 11, …, 43 | 11 | Sparse MLA | **Yes** | Mixture-of-experts | \n| All other layers from 4 to 44 | 31 | KDA | No | Mixture-of-experts | \n| **Total** | **45** | 34 KDA, 11 sparse MLA | **11 of 45** | 3 dense, 42 mixture-of-experts | \n\nExperts run on 42 layers, but a KV cache is stored on only 11. That is about a quarter of the layers. If all 45 layers stored a cache with entries of the same size, the cache per token would be roughly four times larger. Because so few layers store a growing cache, an eight-GPU node can hold millions of KV tokens once the 306 GB of weights are resident. The cache becomes the scarce resource when many long sessions are in flight together.\n\n**Inside the eleven KV layers.** A lightweight indexer scores the past tokens and selects up to 2,048 of them for attention. This makes each step cheaper, but it does not shrink the cache: any past token may be selected later, so every token’s entry must stay in memory.\n\nEach cached entry is a 512-element latent vector per token per layer (`kv_lora_rank` is 512). DeepSeek-style MLA adds a 64-element RoPE (rotary position) part to that vector, making a 576-element row. GLM-5.3-Flash has none (`qk_rope_head_dim` is 0), so its row stays at 512. The indexer also keeps its own small key cache, with 128-dimensional keys pooled by four (IndexPool).\n\nThe key properties of the GLM-5.3-Flash model and the MI300X server node are summarized in the following table:\n\n| Property | Value | \n|---|---|\n| Checkpoint | FP8, about 306 GB, 62 shards | \n| Parameters | 320B total, about 18B active per token | \n| Layers | 45: 34 linear attention, 11 sparse MLA | \n| Experts | 288 routed, top-8, plus 1 shared | \n| KV row | 512-wide latent, no RoPE half | \n| Maximum context length | 1,048,576 tokens (served at 131,072 in this blog) | \n| MI300X server node | 8× MI300X GPUs, with 192 GB of HBM each | \n\n## Two Parallelism Strategies, One KV Cache Budget[#](#two-parallelism-strategies-one-kv-cache-budget)\n\nBoth parallelism strategies use the same eight GPUs, the same FP8 weights, the same 131,072-token serving window, and the same ROCm sparse-MLA backend. They differ in who owns the cache.\n\n**TP8** slices the attention projections and the experts across the node, so a decode step is quick. In our measurements it had the lowest latency for a single long request. The latent KV is copied onto **every GPU** rather than sliced. With the KV cache in BF16 format, the boot log reports memory available for 9,841,634 tokens in the KV cache for the entire node. A request with 96k-token input length and a 32k-token output length consumes 128,000 of them, so the node can only hold **76** (floor(9,841,634/128000)) such sequences. The 77th request has to wait, because the server stops admitting new requests once the pool is full.\n\n**DP8+EP8** makes each GPU its own data-parallel rank. The rank owns attention for its requests and stores their KV locally. The 288 routed experts are divided across the eight ranks, 36 per GPU, and a token whose expert sits on another GPU crosses the in-node XGMI fabric. There is no tensor-parallel split of attention, so one request does more work on its rank: at a single 96k/32k request, the mean time per output token (TPOT) is **19.7 ms** against **12.6 ms** on TP8, and output throughput is 49.9 tok/s against 78.7. In exchange the cache is no longer a single shared pool. The eight ranks together hold **56,564,848** BF16 KV tokens, **5.75×** the TP8 pool, which translates to **441** concurrent 96k/32k sequences.\n\nThe same division, applied to the three request sizes we measured, results in the capacity plan shown in Figure 2, assuming the KV cache is in BF16 format:\n\n*Figure 2. KV-cache ceiling by request size, from the token pool printed at startup divided by the tokens in one request. The short bar on the right is the limit a long-context service on TP8 actually meets.*\n\nAs illustrated by the capacity plan, a chat service with a few hundred users and prompts in the order of a thousand tokens (1k/1k and 8k/1k) stays under the ceilings for both parallelism strategies. On the other hand, an agent service that keeps over a hundred 100k-token sessions in flight is already past what TP8 can cache. The rest of this blog focuses on that case.\n\nvLLM reports the pool at startup as `GPU KV cache size: N tokens`. On TP8, N covers the whole node. On DP8+EP8, each of the eight ranks reports its own N, so the node total is 8 × 7,070,606 = 56,564,848 tokens. The ceiling is that pool divided by the tokens in one request (input plus output), rounded down.\n\n## Which Parallelism Strategy Carries the Workload[#](#which-parallelism-strategy-carries-the-workload)\n\nWe compared the two strategies on vLLM 0.31.0 across a concurrency ladder, with the serving window, batch limits, and KV precision held fixed so the only change from point to point is how many requests are in flight. Prompts were random tokens at a fixed length, and generations ran to the full output length. Throughput can vary by up to 1.1 percent across repeated runs on the same server and by up to about 3 percent between different nodes, so gaps of about 3 percent or less are within run-to-run noise and should be ignored. The output throughput ratio between the two strategies is shown in Figure 3.\n\n*Figure 3. Output tokens per second for DP8+EP8 divided by the same quantity for TP8, with a BF16 KV cache. Above the dashed line, DP8+EP8 serves more tokens. Markers are measured points.*\n\nWhen the concurrency is one, the throughput ratio is 0.61× to 0.63× on every shape, which means TP8 delivers about 1.6× the throughput of DP8+EP8, and the length of the prompt does not create that lead. This is the latency regime: a single session with no memory bottleneck, and TP8 delivered the higher throughput. However, the curves separate as the server memory fills up with KV cache.\n\nOn 1k/1k the ratio climbs from 0.62× at concurrency 1 to 0.95× at concurrency 256, close to a tie. The TP8 ceiling at that length is 4,805 sequences, so a service of this shape is nowhere near a cache limit, and in our measurements DP8+EP8 did not deliver the higher throughput.\n\nMedium requests cross over while the cache still has room. On 8k/1k, DP8+EP8 is already ahead for concurrency 128 (1.19×) and reaches 1.55× at 512. TP8 could still hold about 1,067 such sequences. The gain here is due to the DP8+EP8 strategy: eight ranks decoding at once, experts sharded, attention left whole on the rank that owns the request. DP8+EP8 delivers the higher throughput in this scenario.\n\nLong requests cross over because TP8 runs out of KV cache memory much sooner. At concurrency 64, still under the ceiling of 76, TP8 leads: **2,105** output tok/s against **1,890** tok/s. At concurrency 128, TP8 delivers **2,085** tok/s and DP8+EP8 delivers **3,218** tok/s. At concurrency 256 the figures are **2,317** and **4,719** tok/s, a gap of 2.04×. The crossover is expected between concurrency 64 and 128, consistent with the TP8 ceiling of 76; the measured data confirms TP8 leads at 64 and DP8+EP8 leads at 128 and above. A long-context deployment sized around TP8’s single-request speed loses that advantage once demand passes this ceiling, which is a realistic load for an internal coding or agent service.\n\n## FP8 KV Cache[#](#fp8-kv-cache)\n\nSince the memory taken up by the KV cache is a key factor that limits the context length and concurrency supported by the MI300X server, one way to mitigate this limitation is to reduce the datatype of the KV cache. FP8 KV cache stores the cached values in one byte instead of two, which reduces memory pressure significantly. The tokens are dequantized when they are read back from the KV cache.\n\nFor cases where the available memory can accommodate the KV cache in BF16 for every in-flight sequence, the extra memory goes unused, so FP8 has nothing to offer. On 96k/32k, FP8 changed throughput by less than 2% at every concurrency below the ceiling that we measured (Figure 4), within run-to-run noise. FP8 KV cache starts to pay off at the concurrency where the BF16 pool can no longer hold another sequence. The table below summarizes the change in throughput from BF16 KV cache to FP8 KV cache for the two parallelism strategies.\n\n|  | TP8 (ceiling 76) | DP8+EP8 (ceiling 441) | \n|---|---|---|\n| Below the ceiling | Within noise (−1.1% to −0.8%) | Within noise (−1.8% to +0.08%) | \n| Above the ceiling | **+42.7%** at concurrency 128 | **+25.8%** at concurrency 512 | \n| Next point | +21.9% at concurrency 256 | +21.3% at concurrency 768 | \n\nBelow the ceiling, FP8 changed throughput by less than 2%, so reading fewer bytes per step made no measurable difference. The gain appears only where the BF16 pool runs out of slots, which points to cache capacity, not memory bandwidth, as the reason.\n\nOnce the FP8 pool also fills, the gain narrows. TP8 in FP8 holds 146 long sequences; the 42.7 percent advantage at concurrency 128 is 21.9 percent at 256. DP8+EP8 in FP8 holds 746; the 25.8 percent at concurrency 512 is 21.3 percent at 768. The pool grows by less than 2×.\n\nFigure 4 shows how output throughput changes when the KV cache is switched from BF16 to FP8, for each strategy at concurrencies below and above the BF16 ceiling. Grey points mark concurrencies where the BF16 cache has room; the change there is within run-to-run noise. Dark bars mark concurrencies where the BF16 cache is full but the FP8 cache still has room, and light bars mark concurrencies where both caches are full. Bar labels give output tokens per second, BF16 to FP8.\n\nNote\n\nFigure 4 compares FP8 with BF16 within each strategy. It does not compare TP8 with DP8+EP8; Figure 3 does.\n\n*Figure 4. Change in output throughput with an FP8 KV cache against a BF16 KV cache, on 96k-in, 32k-out requests. The shaded band is run-to-run noise, about 3%. [The appendix](#appendix-measured-results) lists every measured point.*\n\nNote\n\nFP8 KV cache requires [vLLM #57134](https://github.com/vllm-project/vllm/pull/57134). GLM-5.3-Flash stores 512 values per token per layer, while the ROCm sparse-attention kernels are built for DeepSeek’s 576-value row. The patch routes GLM-5.3-Flash to a kernel that matches its row and handles FP8. It is included in the image built in the Setup section below.\n\n## Set Up the vLLM Server[#](#set-up-the-vllm-server)\n\nThis section provides the instruction on how to reproduce the measurements described above in four steps: download the model, build the patched image, start the server (mounting the AITER patch) in the configuration you want to measure, and run the benchmark script. You need a host with eight MI300X GPUs, Docker, and room for the 306 GB checkpoint.\n\nThe results in this blog were measured with vLLM 0.31.0 (ROCm 7.2.3, PyTorch 2.13.0, AITER 0.1.23) on eight MI300X GPUs. They use the official [`vllm/vllm-openai-rocm:v0.31.0`](https://hub.docker.com/layers/vllm/vllm-openai-rocm/v0.31.0/images/sha256-749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f) image plus three patches. To reproduce the results, you need to apply all three patches.\n\n| Patch | Purpose | How it is applied | \n|---|---|---|\n| [vLLM #59412](https://github.com/vllm-project/vllm/pull/59412) | Correct long-context retrieval on GLM-5.3-Flash | Patch to `vllm/v1/worker/utils.py` , built into the image | \n| [vLLM #57134](https://github.com/vllm-project/vllm/pull/57134) | FP8 KV cache support on GLM-5.3-Flash | Patch to two `rocm_aiter_mla_sparse.py` files, built into the image | \n| [AITER 5979](https://github.com/ROCm/aiter/pull/5979) | MoE output stability at low concurrency | Patched `fused_moe.py` mounted into the container at launch | \n\nThe two vLLM patches ship with this blog as diffs in the [`src/`](https://github.com/ROCm/rocm-blogs/tree/release/blogs/artificial-intelligence/glm53flash-mi300x-vllm/src) folder: `pr59412_runtime.diff` and `pr57134_runtime.diff`. The steps below apply them to the official image `vllm/vllm-openai-rocm:v0.31.0` and build a new image from the result.  Run the commands from a folder that contains `src/`.\n\n```\n# Prerequisite: the Hugging Face client used for the model download\npip install huggingface_hub\n\nexport HF_TOKEN=<your_huggingface_token>\npython3 -c \"\nfrom huggingface_hub import snapshot_download\nimport os\nsnapshot_download(\n    repo_id='zai-org/GLM-5.3-Flash',\n    local_dir='/path/to/glm-5.3-flash-fp8',\n    token=os.environ.get('HF_TOKEN'),\n)\n\"\n\ndocker pull vllm/vllm-openai-rocm:v0.31.0\n\nmkdir -p build\ndocker run --rm --user \"$(id -u):$(id -g)\" --entrypoint bash \\\n  -v \"$(pwd)/build:/out\" vllm/vllm-openai-rocm:v0.31.0 -c '\n  P=/usr/local/lib/python3.12/dist-packages/vllm\n  mkdir -p /out/vllm/v1/worker /out/vllm/v1/attention/backends/mla /out/vllm/v1/attention/ops\n  cp $P/v1/worker/utils.py                                 /out/vllm/v1/worker/utils.py\n  cp $P/v1/attention/backends/mla/rocm_aiter_mla_sparse.py /out/vllm/v1/attention/backends/mla/\n  cp $P/v1/attention/ops/rocm_aiter_mla_sparse.py          /out/vllm/v1/attention/ops/\n'\n\ncd build\npatch -p1 < ../src/pr59412_runtime.diff\npatch -p1 < ../src/pr57134_runtime.diff\ncd ..\n```\n\nHere is the Dockerfile, which checks that both patches are present.  Save it in the same folder that contains `src/`.\n\n```\n# GLM-5.3-Flash on vLLM v0.31.0\n# Patches applied: vllm#59412, vllm#57134.\nFROM vllm/vllm-openai-rocm:v0.31.0\n\nARG VLLM_PKG=/usr/local/lib/python3.12/dist-packages/vllm\n\nRUN cp ${VLLM_PKG}/v1/worker/utils.py \\\n       ${VLLM_PKG}/v1/worker/utils.py.orig && \\\n    cp ${VLLM_PKG}/v1/attention/backends/mla/rocm_aiter_mla_sparse.py \\\n       ${VLLM_PKG}/v1/attention/backends/mla/rocm_aiter_mla_sparse.py.orig && \\\n    cp ${VLLM_PKG}/v1/attention/ops/rocm_aiter_mla_sparse.py \\\n       ${VLLM_PKG}/v1/attention/ops/rocm_aiter_mla_sparse.py.orig\n\nCOPY build/vllm/v1/worker/utils.py ${VLLM_PKG}/v1/worker/utils.py\nCOPY build/vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py \\\n     ${VLLM_PKG}/v1/attention/backends/mla/rocm_aiter_mla_sparse.py\nCOPY build/vllm/v1/attention/ops/rocm_aiter_mla_sparse.py \\\n     ${VLLM_PKG}/v1/attention/ops/rocm_aiter_mla_sparse.py\nCOPY src/ /opt/patches/\n\n# Verify the patches are applied. The build fails if either one is missing.\nRUN python3 -m py_compile ${VLLM_PKG}/v1/worker/utils.py && \\\n    python3 -m py_compile ${VLLM_PKG}/v1/attention/backends/mla/rocm_aiter_mla_sparse.py && \\\n    python3 -m py_compile ${VLLM_PKG}/v1/attention/ops/rocm_aiter_mla_sparse.py && \\\n    grep -q \"Storage-block specs\" ${VLLM_PKG}/v1/worker/utils.py && \\\n    grep -q \"_block_size_is_supported\" ${VLLM_PKG}/v1/worker/utils.py && \\\n    grep -q \"NoPE routing is geometry-based\" \\\n        ${VLLM_PKG}/v1/attention/backends/mla/rocm_aiter_mla_sparse.py && \\\n    grep -q \"KV_IS_FP8\" ${VLLM_PKG}/v1/attention/ops/rocm_aiter_mla_sparse.py && \\\n    find ${VLLM_PKG}/v1 -name '__pycache__' -type d -exec rm -rf {} +\n```\n\nBuild the docker image with the command:\n\n```\nDOCKER_BUILDKIT=1 docker build -f Dockerfile -t glm53flash-vllm-v031:latest .\n```\n\nApply the AITER split-K guard to the `fused_moe.py` from that image and keep the result on the host. The container sees it through a read-only bind mount. The release image bakes in the same patch with an integrity check; the bind-mount approach here avoids a full image rebuild and keeps the patch visible alongside the article.\n\n```\ndocker run --rm --user \"$(id -u):$(id -g)\" --entrypoint cp \\\n  -v \"$(pwd):/out\" glm53flash-vllm-v031:latest \\\n  /usr/local/lib/python3.12/dist-packages/aiter/fused_moe.py \\\n  /out/fused_moe.py\n\npython3 - \"$(pwd)/fused_moe.py\" << 'PYEOF'\nimport sys\nfrom pathlib import Path\n\nf = Path(sys.argv[1])\nsrc = f.read_text()\n\nOLD = \"\"\"    tilek = 256\n    split_max = (cu_num + tg_num - 1) // tg_num\n    # at least split = 2\n    for i in reversed(range(2, split_max + 1)):\n        if (model_dim % i == 0) and ((model_dim // i) % tilek == 0):\n            return i\n    return 0\"\"\"\n\nNEW = \"\"\"    tilek = 256\n    split_max = (cu_num + tg_num - 1) // tg_num\n    # at least split = 2, and at least 2 K-tiles per split (ROCm/aiter #5979)\n    for i in reversed(range(2, split_max + 1)):\n        if (model_dim % i == 0) and ((model_dim // i) % tilek == 0) and (model_dim // i >= 2 * tilek):\n            return i\n    return 0\"\"\"\n\nassert OLD in src, \"Anchor not found — check AITER version\"\nf.write_text(src.replace(OLD, NEW, 1))\nprint(\"ROCm/aiter 5979 applied\")\nPYEOF\n```\n\nThe comparison grid used `--max-model-len 131072` and `--max-num-seqs 1024`, with GPU memory utilization 0.8, prefix caching off, and `--attention-backend ROCM_AITER_MLA_SPARSE`. The release launch script uses `--max-model-len 163840` and `--max-num-seqs 512`; the comparison grid uses the values above to let the KV pool fill at lower concurrency and expose the ceiling cleanly. The sequence cap is above the default of 512 so the KV pool fills before the scheduler does. A server left at 512 can still reach the long-context BF16 ceiling on DP8+EP8 (441 sequences) and the FP8 gain at concurrency 512. KV dtype is the default BF16 cache. Add `--kv-cache-dtype fp8` only for the FP8 half of the comparison.\n\nTo start the vLLM server with DP8+EP8, the strategy behind the larger KV pool, run the following commands from the folder that contains `fused_moe.py`, since the `fused_moe.py` mount uses the current folder.\n\nNote\n\nBefore you run the command, replace `/path/to/glm-5.3-flash-fp8` with the path to the folder containing the downloaded model (the folder with `config.json` and the `.safetensors` files).\n\n```\nVIDEO_GID=$(getent group video  | cut -d: -f3)\nRENDER_GID=$(getent group render | cut -d: -f3)\n\ndocker run -d --name glm53flash-v031 \\\n  --network host --device /dev/kfd --device /dev/dri \\\n  --group-add \"${VIDEO_GID}\" --group-add \"${RENDER_GID}\" \\\n  --ipc host --shm-size 128g \\\n  --cap-add SYS_PTRACE --cap-add IPC_LOCK \\\n  --security-opt seccomp=unconfined \\\n  --ulimit nofile=1048576:1048576 \\\n  `# CHANGE THIS PATH: the folder that contains your downloaded model` \\\n  -v /path/to/glm-5.3-flash-fp8:/mnt/models/GLM-5.3-Flash:ro \\\n  -v \"$(pwd)/fused_moe.py:/usr/local/lib/python3.12/dist-packages/aiter/fused_moe.py:ro\" \\\n  `# Startup can take over 10 minutes; allow up to an hour` \\\n  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n  `# Offline model loading — prevent HuggingFace hub calls inside the container` \\\n  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \\\n  `# ROCm and AITER runtime settings` \\\n  -e AITER_BATON_TIMEOUT=7200 \\\n  -e HSA_NO_SCRATCH_RECLAIM=1 \\\n  -e SAFETENSORS_FAST_GPU=1 \\\n  `# AITER kernel selection — enable AITER, its fused MoE kernels, and the skinny GEMM` \\\n  -e VLLM_ROCM_USE_AITER=1 \\\n  -e VLLM_ROCM_USE_AITER_MOE=1 \\\n  -e VLLM_ROCM_USE_SKINNY_GEMM=1 \\\n  `# Communication quantization — INT4 quantizes AllReduce traffic across GPUs;` \\\n  `# reduces inter-GPU bandwidth at the cost of numeric precision` \\\n  -e VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \\\n  `# FlyDSL FP8 MQA-logits kernel variant` \\\n  -e FLYDSL_FP8_MQA_LOGITS_VARIANT=mfma_r4_w4 \\\n  glm53flash-vllm-v031:latest \\\n  /mnt/models/GLM-5.3-Flash \\\n    --served-model-name glm-5-3-flash-fp8 \\\n    --data-parallel-size 8 --enable-expert-parallel \\\n    --trust-remote-code \\\n    --max-model-len 131072 \\\n    --max-num-batched-tokens 8192 \\\n    --max-num-seqs 1024 \\\n    --gpu-memory-utilization 0.8 \\\n    --no-enable-prefix-caching \\\n    --distributed-timeout-seconds 7200 \\\n    --enable-auto-tool-choice \\\n    --tool-call-parser glm47 \\\n    --reasoning-parser glm47 \\\n    --attention-backend ROCM_AITER_MLA_SPARSE \\\n    --port 8000\n```\n\nTo start the server with TP8, use the same command and replace `--data-parallel-size 8 --enable-expert-parallel` with `--tensor-parallel-size 8`. Everything else stays the same: the mounts, the environment variables, and the remaining flags. Both servers use the container name `glm53flash-v031` and port 8000, so remove the first one with `docker rm -f glm53flash-v031` before you start the second.\n\nBefore sending the comparison traffic, confirm the running container actually contains the two vLLM changes with the following commands:\n\n```\ndocker exec glm53flash-v031 bash -lc '\nP=/usr/local/lib/python3.12/dist-packages/vllm\ngrep -q \"Storage-block specs\" $P/v1/worker/utils.py && echo \"PATCH OK #59412\"\ngrep -q \"NoPE routing is geometry-based\" $P/v1/attention/backends/mla/rocm_aiter_mla_sparse.py && echo \"PATCH OK #57134\"\ngrep -q \"KV_IS_FP8\" $P/v1/attention/ops/rocm_aiter_mla_sparse.py && echo \"PATCH OK #57134 fp8\"\n'\n```\n\nStartup takes several minutes. Wait until the log prints `Application startup complete`. A short reasoning request and a tool call are enough to see that the server is the model you think it is. On this tag the reasoning trace comes back in `content`. The `reasoning_content` field is empty.\n\n### Benchmark the Workload[#](#benchmark-the-workload)\n\nThe script below `bench_grid.sh` measures throughput and latency for the three request shapes used in this blog: 1k in and 1k out, 8k in and 1k out, and 96k in and 32k out. For each concurrency level in the list, it sends that many random, fixed-length prompts at once, lets each one generate its full output, and records the output throughput, time to first token (TTFT), and time per output token (TPOT).\n\n``` bash\n#!/usr/bin/env bash\n# Usage: bash bench_grid.sh <TAG> <SHAPE> \"<concurrency list>\"\n#   TAG   server config label: dp8ep8_bf16 | tp8_bf16 | dp8ep8_fp8 | tp8_fp8\n#   SHAPE 1k1k | 8k1k | 96k32k\n# Example: bash bench_grid.sh dp8ep8_bf16 96k32k \"1 8 32 64 128 256 512 1024\"\n# Re-running skips points that already have a result file.\n# WARMUPS=0 skips the built-in warmup request, which is a full-length request (about 11 minutes at 96k/32k).\nset -uo pipefail\nTAG=${1:?TAG}; SHAPE=${2:?SHAPE}; CONCS=${3:?concurrency list}\nCONTAINER=${CONTAINER:-glm53flash-v031}\nPORT=${PORT:-8000}\nMODEL=${MODEL:-glm-5-3-flash-fp8}\nTOKENIZER=${TOKENIZER:-/mnt/models/GLM-5.3-Flash}\nNP_MULT=${NP_MULT:-1}          # prompts = concurrency x NP_MULT\nIGNORE_EOS=${IGNORE_EOS:-1}    # 1 = force every request to generate its full output length\nOUT=${OUT:-$HOME/bench/$TAG}\n\ncase $SHAPE in\n  1k1k)   ISL=1024;  OSL=1024;  TMO=3600  ;;\n  8k1k)   ISL=8192;  OSL=1024;  TMO=7200  ;;\n  96k32k) ISL=96000; OSL=32000; TMO=21600 ;;\n  *) echo \"unknown SHAPE $SHAPE\"; exit 1 ;;\nesac\nmkdir -p \"$OUT\"\nCSV=\"$OUT/summary_$SHAPE.csv\"\n[ -f \"$CSV\" ] || echo \"shape,isl,osl,concurrency,num_prompts,duration_s,completed,output_tok_s,total_tok_s,mean_ttft_ms,median_ttft_ms,p99_ttft_ms,mean_tpot_ms,median_tpot_ms,p99_tpot_ms\" > \"$CSV\"\n\nfor C in $CONCS; do\n  NP=$((C * NP_MULT)); STEM=\"${SHAPE}_c${C}\"\n  [ -f \"$OUT/$STEM.json\" ] && { echo \"skip $STEM (already done)\"; continue; }\n  echo \"=== $TAG $SHAPE concurrency=$C prompts=$NP  $(date)\"\n  EXTRA=\"\"; [ \"$IGNORE_EOS\" = 1 ] && EXTRA=\"--ignore-eos\"\n  docker exec -e AIOHTTP_CLIENT_TIMEOUT=\"$TMO\" \"$CONTAINER\" vllm bench serve \\\n    --model \"$MODEL\" --base-url \"http://localhost:$PORT\" --tokenizer \"$TOKENIZER\" \\\n    --dataset-name random --random-input-len \"$ISL\" --random-output-len \"$OSL\" --random-range-ratio 0.0 \\\n    --num-prompts \"$NP\" --max-concurrency \"$C\" --num-warmups \"${WARMUPS:-1}\" --temperature 0 $EXTRA \\\n    --save-result --result-dir /tmp/bench_json --result-filename \"$STEM.json\" 2>&1 | tee \"$OUT/$STEM.log\"\n  docker cp \"$CONTAINER:/tmp/bench_json/$STEM.json\" \"$OUT/$STEM.json\"\n  python3 - \"$OUT/$STEM.json\" \"$SHAPE,$ISL,$OSL,$C,$NP\" >> \"$CSV\" <<'PY'\nimport json, sys\nd = json.load(open(sys.argv[1]))\nr = lambda k, n=2: round(float(d.get(k) or 0), n)\nprint(sys.argv[2] + \",\" + \",\".join(str(x) for x in [\n    r(\"duration\"), d.get(\"completed\", \"\"), r(\"output_throughput\"), r(\"total_token_throughput\"),\n    r(\"mean_ttft_ms\", 1), r(\"median_ttft_ms\", 1), r(\"p99_ttft_ms\", 1),\n    r(\"mean_tpot_ms\"), r(\"median_tpot_ms\"), r(\"p99_tpot_ms\")]))\nPY\n  sleep 30\ndone\necho \"Done: $OUT\"; column -s, -t \"$CSV\" 2>/dev/null || cat \"$CSV\"\n```\n\nMake sure the server is already running before running the script. Run the script once per configuration and use the tag to name it. Use the launch command above with these changes:\n\n| Tag | Server change | \n|---|---|\n| `dp8ep8_bf16` | none (the launch command above) | \n| `tp8_bf16` | replace `--data-parallel-size 8 --enable-expert-parallel` with`--tensor-parallel-size 8` | \n| `dp8ep8_fp8` | add `--kv-cache-dtype fp8` | \n| `tp8_fp8` | `--tensor-parallel-size 8` and`--kv-cache-dtype fp8` | \n\nFor example, for the DP8+EP8 server with a BF16 cache, run the script with the three different input/output shapes and corresponding concurrency list:\n\n```\nbash bench_grid.sh dp8ep8_bf16 1k1k   \"1 64 256\"\nbash bench_grid.sh dp8ep8_bf16 8k1k   \"1 8 32 64 128 256 512 1024\"\nbash bench_grid.sh dp8ep8_bf16 96k32k \"1 8 32 64 128 256 512 1024\"\n```\n\nResults are written to `~/bench/<TAG>/`: a summary CSV for each request shape, plus the full log and JSON for every point. The server must allow at least as many sequences as the concurrency you ask for, so keep `--max-num-seqs 1024` for concurrencies up to 1024. The 96k/32k runs at high concurrency take a long time.\n\n## Summary[#](#summary)\n\nIn this blog, you explored how KV-cache capacity shapes GLM-5.3-Flash throughput on an eight-GPU MI300X node. You compared TP8 and DP8+EP8 on three request sizes. TP8 was faster at low concurrency, DP8+EP8 pulled ahead once TP8’s 76-session cache filled, and FP8 KV cache helped only after the BF16 cache was full. The Set Up section gives you the image, launch commands and benchmark script to repeat the measurements on your own workload.\n\nDividing the KV cache pool size that vLLM reports at startup (`GPU KV cache size`) by your request length in tokens, input plus output, shows how many sessions fit, and so which case your workload falls in. We measured throughput, not accuracy, so validate FP8 KV cache on your own workload.\n\n**What’s next:** This post covers serving on a single eight-GPU node. We plan to look at serving long-context workloads across multiple nodes with prefill/decode (PD) disaggregation. Follow the [ROCm Tech Blogs](https://rocm.blogs.amd.com/) for updates.\n\n## Appendix: Measured Results[#](#appendix-measured-results)\n\nAll values are output tokens per second, measured with the benchmark script above on eight MI300X GPUs with vLLM 0.31.0. Each point is a single run: random fixed-length prompts, as many requests as the concurrency, and every request generating its full output. Run-to-run noise is about 3%. The KV cache is BF16 unless a table says otherwise. At 96k/32k each high-concurrency point takes an hour or more, so the ladder stops at 256 on TP8 and at 768 on DP8+EP8.\n\n### 1k in, 1k out[#](#k-in-1k-out)\n\n| Concurrency | TP8 | DP8+EP8 | DP8+EP8 ÷ TP8 | \n|---|---|---|---|\n| 1 | 85.26 | 52.70 | 0.62× | \n| 64 | 2,724.28 | 2,016.18 | 0.74× | \n| 256 | 6,167.32 | 5,886.97 | 0.95× | \n\n### 8k in, 1k out[#](#id2)\n\n| Concurrency | TP8 | DP8+EP8 | DP8+EP8 ÷ TP8 | \n|---|---|---|---|\n| 1 | 78.15 | 47.39 | 0.61× | \n| 8 | 437.07 | 320.45 | 0.73× | \n| 32 | 1,107.82 | 956.04 | 0.86× | \n| 64 | 1,526.32 | 1,466.10 | 0.96× | \n| 128 | 1,957.07 | 2,334.11 | 1.19× | \n| 256 | 2,262.00 | 3,087.30 | 1.36× | \n| 512 | 2,443.62 | 3,787.57 | 1.55× | \n| 1024 | 2,207.27 | 4,250.83 | 1.93× | \n\n### 96k in, 32k out[#](#k-in-32k-out)\n\n| Concurrency | TP8 | DP8+EP8 | DP8+EP8 ÷ TP8 | \n|---|---|---|---|\n| 1 | 78.74 | 49.89 | 0.63× | \n| 8 | 484.76 | 370.92 | 0.77× | \n| 32 | 1,392.09 | 1,118.29 | 0.80× | \n| 64 | 2,104.91 | 1,890.31 | 0.90× | \n| 128 | 2,084.59 | 3,218.28 | 1.54× | \n| 256 | 2,317.08 | 4,719.20 | 2.04× | \n\n### 96k in, 32k out: FP8 against BF16 KV cache[#](#k-in-32k-out-fp8-against-bf16-kv-cache)\n\n| Strategy | Concurrency | BF16 KV cache | FP8 KV cache | Change | \n|---|---|---|---|---|\n| TP8 | 8 | 484.76 | 480.73 | −0.8% | \n| TP8 | 32 | 1,392.09 | 1,376.76 | −1.1% | \n| TP8 | 64 | 2,104.91 | 2,083.36 | −1.0% | \n| TP8 | 128 | 2,084.59 | 2,974.96 | +42.7% | \n| TP8 | 256 | 2,317.08 | 2,824.43 | +21.9% | \n| DP8+EP8 | 64 | 1,890.31 | 1,891.88 | +0.08% | \n| DP8+EP8 | 128 | 3,218.28 | 3,217.03 | −0.04% | \n| DP8+EP8 | 256 | 4,719.20 | 4,718.11 | −0.02% | \n| DP8+EP8 | 384 | 5,851.47 | 5,744.10 | −1.8% | \n| DP8+EP8 | 512 | 5,037.15 | 6,334.99 | +25.8% | \n| DP8+EP8 | 768 | 5,535.57 | 6,711.95 | +21.3% | \n\n768 is the first concurrency we tested above the DP8+EP8 FP8 ceiling of 746 sessions.\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.\n\nPerformance testing conducted by AMD as of November 2026 on 8× AMD Instinct™ MI300X GPUs using vLLM v0.31.0 and ROCm 7.2.3. Performance results are based on testing as of this date and may not reflect all publicly available security updates. Results may vary based on hardware configuration, software versions, and workload characteristics.\n\nAMD, the AMD Arrow logo, AMD Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved.", "url": "https://wpnews.pro/news/parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x", "canonical_source": "https://rocm.blogs.amd.com/artificial-intelligence/glm53flash-mi300x-vllm/README.html", "published_at": "2026-11-11 00:00:00+00:00", "updated_at": "2026-11-12 15:50:32.591598+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-chips", "machine-learning"], "entities": ["GLM-5.3-Flash", "AMD Instinct MI300X", "Kimi Delta Attention", "DeepSeek", "Multi-head Latent Attention", "Z.ai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x", "markdown": "https://wpnews.pro/news/parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x.md", "text": "https://wpnews.pro/news/parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x.txt", "jsonld": "https://wpnews.pro/news/parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x.jsonld"}}