Revolut’s David Abitbol says GLM-5.3 fits on 4 GPUs but runs short of memory. The reason is in vLLM’s source: every GPU keeps a full copy of each chat’s memory, the KV cache.
If you self-host GLM-5.3 with vLLM for long coding sessions, give each copy of the model 8 B200 GPUs, not 4, even though it fits on 4. On paper, 4 GPUs leave room for 3.0 full 256K-token sessions per copy; 8 GPUs leave room for 7.6. This article shows why and gives you a 69-line script to check your own setup before you rent anything.
On October 29 the AI Engineer YouTube channel posted “How Revolut Cut AI Coding Costs 90% With Self-Hosted Open Models”, a 12-minute talk by David Abitbol of Revolut.
Paraphrasing his points from the auto-generated captions: serving about 2,000 engineers takes roughly 56 B200s for GLM-5.3. Keep time to first token under 3 seconds. Long context is a scam; 256K is what you need. Use NVFP4 on Blackwell GPUs, about 40% smaller than FP8. And although GLM fits on 4 GPUs (TP4), you don’t get enough KV cache, so 8 (TP8) may work better.
He gave the rules but not the math. Here it is.
The KV cache is the model’s memory of each conversation. GLM-5.3 uses DeepSeek-style latent attention, which stores a small cache per token.
When you split a model across 4 GPUs (tensor parallelism), the weights get split. In vLLM 0.32.0 this cache does not. The attention layer declares its cache with max_tp_shards=1, and the field's docstring in kv_cache_interface.py says GPUs beyond that "hold replicas". So each GPU keeps a full copy of every conversation's cache.
Going from 4 GPUs to 8 halves the weights on each GPU. All the memory that frees up becomes cache.
Two numbers come from the vLLM 0.32.0 source:
vLLM builds an indexer on a layer when max(layer - offset + 1, 0) % freq == 0. With GLM-5.3's freq of 4 and offset of 3, that's 21 of 78 layers. The config's own indexer_types list also marks 21 layers as full.
glmfit.py reads the model's config.json and model.safetensors.index.json. It uses only Python's standard library and needs no GPU.
#!/usr/bin/env python3"""How much KV cache does GLM-5.3 get on B200s at TP4 vs TP8?usage: glmfit.py config.json model.safetensors.index.json [other.index.json]Byte constants are read from vLLM 0.32.0 source (see comments)."""import json, os, sys# MLA row: 576 = fp8_e4m3, FlashInfer sparse (B200 default for an fp8 cache)# 656 = fp8_ds_mla, --attention-backend FLASHMLA_SPARSEIDX = 128 + 128 // 128 * 4 # Indexer k_cache: head_dim + scalesenv = os.environ.getMLA = int(env("MLA", 576))HBM, UTIL = float(env("HBM_GB", 180)), float(env("UTIL", 0.92))OVH, GPUS = float(env("OVERHEAD_GB", 12)), int(env("GPUS", 56))CTX, MTP = int(env("CTX", 262144)), int(env("MTP", 0))DCP = int(env("DCP", 1)) # vLLM -dcp; 1 = offdef load(p): try: return json.load(open(p)) except (OSError, ValueError) as e: sys.exit(f"usage: glmfit.py config.json index.json [index2]\n{e}")def indexer_layers(c): # vLLM models/deepseek_v32/attention.py: Indexer unless skip_topk f, off = c.get("index_topk_freq", 1), c.get("index_skip_topk_offset", 2) pat = c.get("index_topk_pattern") n = c["num_hidden_layers"] if pat: return sum(pat[i] != "S" for i in range(n)) return sum(max(i - off + 1, 0) % f == 0 for i in range(n))if len(sys.argv) < 3: sys.exit(__doc__.splitlines()[1])cfg, idx = load(sys.argv[1]), load(sys.argv[2])wt = idx["metadata"]["total_size"] / 1e9n = cfg["num_hidden_layers"] + MTPni = indexer_layers(cfg) + MTPper_tok = MLA * n + IDX * nisess = CTX * per_tok / 1e9listed = sum(t == "full" for t in cfg.get("indexer_types", []))print(f"weights {wt:7.2f} GB (index.json total_size)")print(f"indexer {ni} of {n} layers (vLLM rule; config lists {listed})")print(f"KV per token {per_tok:,} B = {MLA} x {n} MLA" f" + {IDX} x {ni} indexer")print(f"one {CTX:,}-token session = {sess:.2f} GB;" " at dcp=1 every TP rank holds all of it")print(f"budget {HBM:.0f} GB x {UTIL:.2f} - {OVH:.0f} GB overhead" " (assumed)\n")full = cfg["max_position_embeddings"] * per_tok / 1e9print("TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?")for tp in (4, 8): w = wt / tp # even split: approximation kv = HBM * UTIL - OVH - w reps = GPUS // tp if kv <= 0: print(f"{tp:<2} {'-':>3} {w:6.2f} weights alone exceed the budget") continue d = DCP if tp % DCP == 0 else 1 # vLLM refuses tp % dcp != 0 tok = kv * 1e9 * d / per_tok # dcp=1: every rank holds a full copy boot = "boots" if kv * d >= full else "no" print(f"{tp:<2} {d:3} {w:6.2f} {kv:6.2f} {tok:11,.0f} {tok / CTX:5.2f}" f" {reps:4} {tok * reps / 1e6:8.2f}M {boot}")print(f"\n1M? = can vLLM start with max_model_len" f" {cfg['max_position_embeddings']:,} ({full:.2f} GB)")if len(sys.argv) > 3: o = load(sys.argv[3])["metadata"]["total_size"] / 1e9 print(f"other checkpoint {o:.2f} GB; this one is" f" {100 * (1 - wt / o):.1f}% smaller")
From nvidia/GLM-5.3-NVFP4, download config.json and model.safetensors.index.json (save it as nvfp4.index.json). From zai-org/GLM-5.3, the FP8 release, download the index too (save it as fp8.index.json). Then:
python3 glmfit.py config.json nvfp4.index.json fp8.index.json
weights 464.12 GB (index.json total_size)indexer 21 of 78 layers (vLLM rule; config lists 21)KV per token 47,700 B = 576 x 78 MLA + 132 x 21 indexerone 262,144-token session = 12.50 GB; at dcp=1 every TP rank holds all of itbudget 180 GB x 0.92 - 12 GB overhead (assumed)TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?4 1 116.03 37.57 787,653 3.00 14 11.03M no8 1 58.01 95.59 2,003,889 7.64 7 14.03M boots1M? = can vLLM start with max_model_len 1,048,576 (50.02 GB)other checkpoint 755.62 GB; this one is 38.6% smaller
Read the sess column: full 256K sessions per copy of the model. TP4 gets 3.00 and TP8 gets 7.64. Across 56 GPUs, TP4's 14 copies hold 11.03M cached tokens and TP8's 7 copies hold 14.03M. That's 27% more cache from half as many copies.
The last line compares the two checkpoints’ total_size: 464.12 GB for NVIDIA's NVFP4 against 755.62 GB for Z.ai's FP8. That's 38.6% smaller, close to the talk's "about 40%".
Why not 50%, if 4 bits is half of 8? The Hub lists 27.9 billion BF16 parameters in the NVFP4 repo. Its config puts only the mlp.experts modules in NVFP4 and the first three dense MLPs in FP8. The multi-token-prediction layer, model.layers.78*, is excluded from quantization. The rest stays BF16.
GLM-5.3’s config advertises max_position_embeddings: 1048576. vLLM's check_enough_kv_cache_memory refuses to start if a single request of max_model_len can't fit. At 47,700 bytes a token, one 1M-token request needs 50.02 GB.
On TP4 each GPU has 37.57 GB for cache, and each GPU must hold the whole request. So the 1M? column says no. Set --max-model-len to 256K (anything under about 787K tokens fits) and, by the script's numbers, the startup check passes. "Long context is a scam" is literally true here.
python3 glmfit.py config.json fp8.index.json
TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?4 - 188.90 weights alone exceed the budget8 1 94.45 59.15 1,239,997 4.73 7 8.68M boots
Same config.json; only the weight size changes. At 188.90 GB per GPU, FP8 weights alone are bigger than the 153.6 GB the script leaves for weights and cache. On 8 GPUs it fits but holds 4.73 sessions against NVFP4's 7.64. NVFP4 is what makes 4 GPUs possible at all.
First, vLLM can split the cache across GPUs instead of copying it: --decode-context-parallel-size (-dcp). When vLLM checks whether a max-length request fits, it divides max_model_len by the DCP size. The script assumes the cache splits evenly. With DCP=8:
TP dcp wt/GPU KV/GPU tok/replica sess reps fleet tok 1M?4 1 116.03 37.57 787,653 3.00 14 11.03M no8 8 58.01 95.59 16,031,116 61.15 7 112.22M boots
TP8 goes from 7.64 to 61.15 sessions per copy. vLLM refuses -dcp 8 on 4 GPUs (the GPU count must be divisible by the DCP size), so the script shows that row with DCP off. With DCP=4, TP4 reaches 12.02 sessions and TP8 30.58, so 8 GPUs still hold more.
I read this in source and did not run it on B200s. DCP adds GPU-to-GPU traffic on every decode step, so measure speed before you trust it. The default FlashInfer backend supports DCP with GLM’s a2a default. If you switch to the FlashMLA backend, it only accepts --dcp-comm-backend ag_rs.
Second, the FlashMLA backend (--attention-backend FLASHMLA_SPARSE) stores 656 bytes per layer instead of 576. Run MLA=656 and the numbers drop to 2.66 and 6.76 sessions. 8 GPUs still hold over 2.5 times as many.
Every session, token, GB and percentage in sections 05–10 comes from glmfit.py, either in the output blocks or from the env runs named beside them (DCP=8, DCP=4, MLA=656, OVERHEAD_GB=24, MTP=1). The 27% and 153.6 GB are arithmetic on its output, and the lede rounds 3.00 and 7.64. The talk's figures (56 B200s, 2,000 engineers, 3 seconds, about 40%) come from the captions. The Hub's dtype counts and quantization layout in section 06, and the vLLM file references, come from the Hub pages and the vLLM 0.32.0 wheel. None of this was run on a GPU.
vLLM keeps a full copy of GLM-5.3’s attention cache on every GPU in a tensor-parallel group. On paper that gives 8 GPUs 7.64 full 256K sessions per copy against 3.00 for 4 GPUs, and without DCP, 4 GPUs can’t even start at the advertised 1M context. Revolut’s advice, to cap context at 256K and try 8 GPUs, matches what vLLM’s code implies.
When you size a self-hosted coding model, do you count the weights, or the sessions the cache can hold?
If this saved you a GPU order, a clap helps others find it.
Follow me for daily LLM deep-dives.
Why Self-Hosted GLM-5.3 Needs 8 GPUs, Not 4, for Long Coding Sessions was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.