Parallelism for Long-Context Serving: GLM-5.3-Flash on MI300X A technical benchmark of GLM-5.3-Flash, a 320-billion-parameter sparse mixture-of-experts model with about 18 billion active parameters per token, on a single eight-GPU AMD Instinct MI300X node found that DP8+EP8 parallelism with an FP8 KV cache admits far more long-context sessions than TP8, because each request's cache is stored once rather than copied onto all eight GPUs. The FP8 checkpoint is about 306 GB, roughly a fifth of the node's 1,536 GB of HBM, and FP8 KV cache changed throughput by less than 2% while the BF16 pool could admit the workload, raising it once BF16 could no longer admit the next request. The tests covered three request shapes (1k in/1k out, 8k in/1k out, 96k in/32k out) with requests up to 131,072 tokens, against the model's 1,048,576-token maximum context. Parallelism for Long-Context Serving: GLM-5.3-Flash on MI300X parallelism-for-long-context-serving-glm-5-3-flash-on-mi300x GLM-5.3-Flash runs on a single eight-GPU AMD Instinct™ MI300X node. The FP8 checkpoint is about 306 GB, roughly a fifth of the node’s 1,536 GB of HBM. During inference, most of the rest of the memory will be taken by the KV cache. The KV cache holds the attention keys and values of every token a request has processed, so the model does not recompute them at each decoding step. Its footprint is the sum of the tokens held across all active requests, so it grows with request length and with the number of requests in flight. In long-context, high-concurrency serving it can exceed the weights as the largest consumer of HBM. When the pool is exhausted, the scheduler stops admitting requests; new arrivals queue, and throughput and latency degrade. Two choices determine the throughput of a long-context, high-concurrency workload: Which parallelism strategy? - TP8 tensor parallelism, width 8 keeps one copy of the model, sharded across the eight GPUs. A single request has the lower latency, because each layer’s compute is divided among them. Each GPU holds a full copy of every request’s KV cache, so the node’s cache capacity is that of one GPU. - DP8+EP8 data parallelism and expert parallelism, both width 8 runs eight data-parallel ranks, each serving its own requests, and shards the routed experts across them. A single request is slower than on TP8, but each request’s cache is stored once instead of being copied onto all eight GPUs, so the node holds far more cached tokens and many more long sessions fit. Which KV cache precision? This choice applies on top of either parallelism strategy: - BF16 is the default. - FP8 stores the cached values in one byte instead of two, so the memory pool holds nearly twice as many tokens. In our tests it changed throughput by less than 2% while the BF16 pool could admit the workload, and raised it once the BF16 pool could no longer admit the next request. To see how these choices play out for long contexts, this blog does three things: - It identifies which parts of GLM-5.3-Flash store a KV cache, since they set the per-token cost. - It works out how many sessions of a given length each parallelism strategy can hold: its KV-cache ceiling. - It measures output throughput against concurrency for TP8 and DP8+EP8, with BF16 and FP8 KV cache, on three request shapes 1k in/1k out, 8k in/1k out, 96k in/32k out , and reads the results against that ceiling. The appendix appendix-measured-results lists every measured point. GLM-5.3-Flash glm-5-3-flash GLM-5.3-Flash https://huggingface.co/zai-org/GLM-5.3-Flash is a sparse mixture-of-experts MoE model with a hybrid attention stack. It has 320 billion parameters in total and about 18 billion active on each token. The model supports a maximum context length of 1,048,576 tokens 1M . This blog serves requests up to 131,072 tokens 128K . The model used in this blog has weights in FP8 format. As shown in Figure 1, GLM-5.3-Flash has forty-five layers. Two separate patterns run through them, and they are easy to confuse because both begin at layer 3: - Attention type decides the KV cache. Thirty-four layers use Kimi Delta Attention KDA , a linear attention whose state does not grow with the prompt. The other eleven, layers 3, 7, 11 and so on through 43 every fourth layer , use DeepSeek sparse attention with Multi-head Latent Attention MLA . Only these eleven store a KV cache that grows with the prompt. - Feed-forward type decides compute and weights. Layers 0, 1 and 2 use a dense MLP. Layers 3 through 44, all 42 of them, use mixture-of-experts: each token is routed to 8 of the 288 experts, and one shared expert always runs with them. Figure 1. Layers 0 through 44. Red layers store the token KV cache. Grey layers are linear attention and keep a fixed recurrent state. The table below summarizes how the 45 layers combine the two patterns. | Layers | Count | Attention | Stores a growing KV cache? | Feed-forward | |---|---|---|---|---| | 0, 1, 2 | 3 | KDA | No | Dense MLP | | 3, 7, 11, …, 43 | 11 | Sparse MLA | Yes | Mixture-of-experts | | All other layers from 4 to 44 | 31 | KDA | No | Mixture-of-experts | | Total | 45 | 34 KDA, 11 sparse MLA | 11 of 45 | 3 dense, 42 mixture-of-experts | Experts run on 42 layers, but a KV cache is stored on only 11. That is about a quarter of the layers. If all 45 layers stored a cache with entries of the same size, the cache per token would be roughly four times larger. Because so few layers store a growing cache, an eight-GPU node can hold millions of KV tokens once the 306 GB of weights are resident. The cache becomes the scarce resource when many long sessions are in flight together. Inside the eleven KV layers. A lightweight indexer scores the past tokens and selects up to 2,048 of them for attention. This makes each step cheaper, but it does not shrink the cache: any past token may be selected later, so every token’s entry must stay in memory. Each cached entry is a 512-element latent vector per token per layer kv lora rank is 512 . DeepSeek-style MLA adds a 64-element RoPE rotary position part to that vector, making a 576-element row. GLM-5.3-Flash has none qk rope head dim is 0 , so its row stays at 512. The indexer also keeps its own small key cache, with 128-dimensional keys pooled by four IndexPool . The key properties of the GLM-5.3-Flash model and the MI300X server node are summarized in the following table: | Property | Value | |---|---| | Checkpoint | FP8, about 306 GB, 62 shards | | Parameters | 320B total, about 18B active per token | | Layers | 45: 34 linear attention, 11 sparse MLA | | Experts | 288 routed, top-8, plus 1 shared | | KV row | 512-wide latent, no RoPE half | | Maximum context length | 1,048,576 tokens served at 131,072 in this blog | | MI300X server node | 8× MI300X GPUs, with 192 GB of HBM each | Two Parallelism Strategies, One KV Cache Budget two-parallelism-strategies-one-kv-cache-budget Both parallelism strategies use the same eight GPUs, the same FP8 weights, the same 131,072-token serving window, and the same ROCm sparse-MLA backend. They differ in who owns the cache. TP8 slices the attention projections and the experts across the node, so a decode step is quick. In our measurements it had the lowest latency for a single long request. The latent KV is copied onto every GPU rather than sliced. With the KV cache in BF16 format, the boot log reports memory available for 9,841,634 tokens in the KV cache for the entire node. A request with 96k-token input length and a 32k-token output length consumes 128,000 of them, so the node can only hold 76 floor 9,841,634/128000 such sequences. The 77th request has to wait, because the server stops admitting new requests once the pool is full. DP8+EP8 makes each GPU its own data-parallel rank. The rank owns attention for its requests and stores their KV locally. The 288 routed experts are divided across the eight ranks, 36 per GPU, and a token whose expert sits on another GPU crosses the in-node XGMI fabric. There is no tensor-parallel split of attention, so one request does more work on its rank: at a single 96k/32k request, the mean time per output token TPOT is 19.7 ms against 12.6 ms on TP8, and output throughput is 49.9 tok/s against 78.7. In exchange the cache is no longer a single shared pool. The eight ranks together hold 56,564,848 BF16 KV tokens, 5.75× the TP8 pool, which translates to 441 concurrent 96k/32k sequences. The same division, applied to the three request sizes we measured, results in the capacity plan shown in Figure 2, assuming the KV cache is in BF16 format: Figure 2. KV-cache ceiling by request size, from the token pool printed at startup divided by the tokens in one request. The short bar on the right is the limit a long-context service on TP8 actually meets. As illustrated by the capacity plan, a chat service with a few hundred users and prompts in the order of a thousand tokens 1k/1k and 8k/1k stays under the ceilings for both parallelism strategies. On the other hand, an agent service that keeps over a hundred 100k-token sessions in flight is already past what TP8 can cache. The rest of this blog focuses on that case. vLLM reports the pool at startup as GPU KV cache size: N tokens . On TP8, N covers the whole node. On DP8+EP8, each of the eight ranks reports its own N, so the node total is 8 × 7,070,606 = 56,564,848 tokens. The ceiling is that pool divided by the tokens in one request input plus output , rounded down. Which Parallelism Strategy Carries the Workload which-parallelism-strategy-carries-the-workload We compared the two strategies on vLLM 0.31.0 across a concurrency ladder, with the serving window, batch limits, and KV precision held fixed so the only change from point to point is how many requests are in flight. Prompts were random tokens at a fixed length, and generations ran to the full output length. Throughput can vary by up to 1.1 percent across repeated runs on the same server and by up to about 3 percent between different nodes, so gaps of about 3 percent or less are within run-to-run noise and should be ignored. The output throughput ratio between the two strategies is shown in Figure 3. Figure 3. Output tokens per second for DP8+EP8 divided by the same quantity for TP8, with a BF16 KV cache. Above the dashed line, DP8+EP8 serves more tokens. Markers are measured points. When the concurrency is one, the throughput ratio is 0.61× to 0.63× on every shape, which means TP8 delivers about 1.6× the throughput of DP8+EP8, and the length of the prompt does not create that lead. This is the latency regime: a single session with no memory bottleneck, and TP8 delivered the higher throughput. However, the curves separate as the server memory fills up with KV cache. On 1k/1k the ratio climbs from 0.62× at concurrency 1 to 0.95× at concurrency 256, close to a tie. The TP8 ceiling at that length is 4,805 sequences, so a service of this shape is nowhere near a cache limit, and in our measurements DP8+EP8 did not deliver the higher throughput. Medium requests cross over while the cache still has room. On 8k/1k, DP8+EP8 is already ahead for concurrency 128 1.19× and reaches 1.55× at 512. TP8 could still hold about 1,067 such sequences. The gain here is due to the DP8+EP8 strategy: eight ranks decoding at once, experts sharded, attention left whole on the rank that owns the request. DP8+EP8 delivers the higher throughput in this scenario. Long requests cross over because TP8 runs out of KV cache memory much sooner. At concurrency 64, still under the ceiling of 76, TP8 leads: 2,105 output tok/s against 1,890 tok/s. At concurrency 128, TP8 delivers 2,085 tok/s and DP8+EP8 delivers 3,218 tok/s. At concurrency 256 the figures are 2,317 and 4,719 tok/s, a gap of 2.04×. The crossover is expected between concurrency 64 and 128, consistent with the TP8 ceiling of 76; the measured data confirms TP8 leads at 64 and DP8+EP8 leads at 128 and above. A long-context deployment sized around TP8’s single-request speed loses that advantage once demand passes this ceiling, which is a realistic load for an internal coding or agent service. FP8 KV Cache fp8-kv-cache Since the memory taken up by the KV cache is a key factor that limits the context length and concurrency supported by the MI300X server, one way to mitigate this limitation is to reduce the datatype of the KV cache. FP8 KV cache stores the cached values in one byte instead of two, which reduces memory pressure significantly. The tokens are dequantized when they are read back from the KV cache. For cases where the available memory can accommodate the KV cache in BF16 for every in-flight sequence, the extra memory goes unused, so FP8 has nothing to offer. On 96k/32k, FP8 changed throughput by less than 2% at every concurrency below the ceiling that we measured Figure 4 , within run-to-run noise. FP8 KV cache starts to pay off at the concurrency where the BF16 pool can no longer hold another sequence. The table below summarizes the change in throughput from BF16 KV cache to FP8 KV cache for the two parallelism strategies. | | TP8 ceiling 76 | DP8+EP8 ceiling 441 | |---|---|---| | Below the ceiling | Within noise −1.1% to −0.8% | Within noise −1.8% to +0.08% | | Above the ceiling | +42.7% at concurrency 128 | +25.8% at concurrency 512 | | Next point | +21.9% at concurrency 256 | +21.3% at concurrency 768 | Below the ceiling, FP8 changed throughput by less than 2%, so reading fewer bytes per step made no measurable difference. The gain appears only where the BF16 pool runs out of slots, which points to cache capacity, not memory bandwidth, as the reason. Once the FP8 pool also fills, the gain narrows. TP8 in FP8 holds 146 long sequences; the 42.7 percent advantage at concurrency 128 is 21.9 percent at 256. DP8+EP8 in FP8 holds 746; the 25.8 percent at concurrency 512 is 21.3 percent at 768. The pool grows by less than 2×. Figure 4 shows how output throughput changes when the KV cache is switched from BF16 to FP8, for each strategy at concurrencies below and above the BF16 ceiling. Grey points mark concurrencies where the BF16 cache has room; the change there is within run-to-run noise. Dark bars mark concurrencies where the BF16 cache is full but the FP8 cache still has room, and light bars mark concurrencies where both caches are full. Bar labels give output tokens per second, BF16 to FP8. Note Figure 4 compares FP8 with BF16 within each strategy. It does not compare TP8 with DP8+EP8; Figure 3 does. Figure 4. Change in output throughput with an FP8 KV cache against a BF16 KV cache, on 96k-in, 32k-out requests. The shaded band is run-to-run noise, about 3%. The appendix appendix-measured-results lists every measured point. Note FP8 KV cache requires vLLM 57134 https://github.com/vllm-project/vllm/pull/57134 . GLM-5.3-Flash stores 512 values per token per layer, while the ROCm sparse-attention kernels are built for DeepSeek’s 576-value row. The patch routes GLM-5.3-Flash to a kernel that matches its row and handles FP8. It is included in the image built in the Setup section below. Set Up the vLLM Server set-up-the-vllm-server This section provides the instruction on how to reproduce the measurements described above in four steps: download the model, build the patched image, start the server mounting the AITER patch in the configuration you want to measure, and run the benchmark script. You need a host with eight MI300X GPUs, Docker, and room for the 306 GB checkpoint. The results in this blog were measured with vLLM 0.31.0 ROCm 7.2.3, PyTorch 2.13.0, AITER 0.1.23 on eight MI300X GPUs. They use the official vllm/vllm-openai-rocm:v0.31.0 https://hub.docker.com/layers/vllm/vllm-openai-rocm/v0.31.0/images/sha256-749f6f3f944f12af49966ac523c1f8573e4b229594b541954bd5f879b4496b1f image plus three patches. To reproduce the results, you need to apply all three patches. | Patch | Purpose | How it is applied | |---|---|---| | vLLM 59412 https://github.com/vllm-project/vllm/pull/59412 | Correct long-context retrieval on GLM-5.3-Flash | Patch to vllm/v1/worker/utils.py , built into the image | | vLLM 57134 https://github.com/vllm-project/vllm/pull/57134 | FP8 KV cache support on GLM-5.3-Flash | Patch to two rocm aiter mla sparse.py files, built into the image | | AITER 5979 https://github.com/ROCm/aiter/pull/5979 | MoE output stability at low concurrency | Patched fused moe.py mounted into the container at launch | The two vLLM patches ship with this blog as diffs in the src/ https://github.com/ROCm/rocm-blogs/tree/release/blogs/artificial-intelligence/glm53flash-mi300x-vllm/src folder: pr59412 runtime.diff and pr57134 runtime.diff . The steps below apply them to the official image vllm/vllm-openai-rocm:v0.31.0 and build a new image from the result. Run the commands from a folder that contains src/ . Prerequisite: the Hugging Face client used for the model download pip install huggingface hub export HF TOKEN=