{"slug": "why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions", "title": "Why Self-Hosted GLM-5.3 Needs 8 GPUs, Not 4, for Long Coding Sessions", "summary": "Self-hosting GLM-5.3 with vLLM for long coding sessions requires 8 B200 GPUs per model copy rather than 4, because vLLM 0.32.0 replicates the full KV cache on every GPU instead of sharding it, according to an analysis of the vLLM source prompted by David Abitbol's October 29 talk on the AI Engineer YouTube channel. Abitbol, of Revolut, said serving about 2,000 engineers takes roughly 56 B200s with time to first token under 3 seconds, and noted that although GLM-5.3 fits on 4 GPUs (TP4), the KV cache is insufficient, so 8 (TP8) may work better. The analysis calculates that 4 GPUs leave room for 3.0 full 256K-token sessions per copy versus 7.6 on 8 GPUs, and publishes a 69-line standard-library Python script, glmfit.py, that reads config.json and model.safetensors.index.json to check memory fit without a GPU.", "body_md": "Revolut’s David Abitbol says GLM-5.3 fits on 4 GPUs but runs short of memory. The reason is in vLLM’s source: every GPU keeps a full copy of each chat’s memory, the KV cache.\n\nIf you self-host GLM-5.3 with vLLM for long coding sessions, give each copy of the model 8 B200 GPUs, not 4, even though it fits on 4. On paper, 4 GPUs leave room for 3.0 full 256K-token sessions per copy; 8 GPUs leave room for 7.6. This article shows why and gives you a 69-line script to check your own setup before you rent anything.\n\nOn October 29 the AI Engineer YouTube channel posted [“How Revolut Cut AI Coding Costs 90% With Self-Hosted Open Models”](https://www.youtube.com/watch?v=Ok6QdZHCcI8), a 12-minute talk by David Abitbol of Revolut.\n\nParaphrasing his points from the auto-generated captions: serving about 2,000 engineers takes roughly 56 B200s for GLM-5.3. Keep time to first token under 3 seconds. Long context is a scam; 256K is what you need. Use NVFP4 on Blackwell GPUs, about 40% smaller than FP8. And although GLM fits on 4 GPUs (TP4), you don’t get enough KV cache, so 8 (TP8) may work better.\n\nHe gave the rules but not the math. Here it is.\n\nThe KV cache is the model’s memory of each conversation. GLM-5.3 uses DeepSeek-style latent attention, which stores a small cache per token.\n\nWhen you split a model across 4 GPUs (tensor parallelism), the weights get split. In vLLM 0.32.0 this cache does not. The attention layer declares its cache with max_tp_shards=1, and the field's docstring in kv_cache_interface.py says GPUs beyond that \"hold replicas\". So each GPU keeps a full copy of every conversation's cache.\n\nGoing from 4 GPUs to 8 halves the weights on each GPU. All the memory that frees up becomes cache.\n\nTwo numbers come from the vLLM 0.32.0 source:\n\nvLLM builds an indexer on a layer when max(layer - offset + 1, 0) % freq == 0. With GLM-5.3's freq of 4 and offset of 3, that's 21 of 78 layers. The config's own indexer_types list also marks 21 layers as full.\n\nglmfit.py reads the model's config.json and model.safetensors.index.json. It uses only Python's standard library and needs no GPU.\n\n``` bash\n#!/usr/bin/env python3\"\"\"How much KV cache does GLM-5.3 get on B200s at TP4 vs TP8?usage: glmfit.py config.json model.safetensors.index.json [other.index.json]Byte constants are read from vLLM 0.32.0 source (see comments).\"\"\"import json, os, sys# MLA row: 576 = fp8_e4m3, FlashInfer sparse (B200 default for an fp8 cache)#          656 = fp8_ds_mla, --attention-backend FLASHMLA_SPARSEIDX = 128 + 128 // 128 * 4         # Indexer k_cache: head_dim + scalesenv = os.environ.getMLA = int(env(\"MLA\", 576))HBM, UTIL = float(env(\"HBM_GB\", 180)), float(env(\"UTIL\", 0.92))OVH, GPUS = float(env(\"OVERHEAD_GB\", 12)), int(env(\"GPUS\", 56))CTX, MTP = int(env(\"CTX\", 262144)), int(env(\"MTP\", 0))DCP = int(env(\"DCP\", 1))            # vLLM -dcp; 1 = offdef load(p):    try:        return json.load(open(p))    except (OSError, ValueError) as e:        sys.exit(f\"usage: glmfit.py config.json index.json [index2]\\n{e}\")def indexer_layers(c):    # vLLM models/deepseek_v32/attention.py: Indexer unless skip_topk    f, off = c.get(\"index_topk_freq\", 1), c.get(\"index_skip_topk_offset\", 2)    pat = c.get(\"index_topk_pattern\")    n = c[\"num_hidden_layers\"]    if pat:        return sum(pat[i] != \"S\" for i in range(n))    return sum(max(i - off + 1, 0) % f == 0 for i in range(n))if len(sys.argv) < 3:    sys.exit(__doc__.splitlines()[1])cfg, idx = load(sys.argv[1]), load(sys.argv[2])wt = idx[\"metadata\"][\"total_size\"] / 1e9n = cfg[\"num_hidden_layers\"] + MTPni = indexer_layers(cfg) + MTPper_tok = MLA * n + IDX * nisess = CTX * per_tok / 1e9listed = sum(t == \"full\" for t in cfg.get(\"indexer_types\", []))print(f\"weights      {wt:7.2f} GB  (index.json total_size)\")print(f\"indexer      {ni} of {n} layers (vLLM rule; config lists {listed})\")print(f\"KV per token {per_tok:,} B = {MLA} x {n} MLA\"      f\" + {IDX} x {ni} indexer\")print(f\"one {CTX:,}-token session = {sess:.2f} GB;\"      \" at dcp=1 every TP rank holds all of it\")print(f\"budget       {HBM:.0f} GB x {UTIL:.2f} - {OVH:.0f} GB overhead\"      \" (assumed)\\n\")full = cfg[\"max_position_embeddings\"] * per_tok / 1e9print(\"TP dcp wt/GPU KV/GPU  tok/replica  sess reps  fleet tok  1M?\")for tp in (4, 8):    w = wt / tp                      # even split: approximation    kv = HBM * UTIL - OVH - w    reps = GPUS // tp    if kv <= 0:        print(f\"{tp:<2} {'-':>3} {w:6.2f}  weights alone exceed the budget\")        continue    d = DCP if tp % DCP == 0 else 1  # vLLM refuses tp % dcp != 0    tok = kv * 1e9 * d / per_tok     # dcp=1: every rank holds a full copy    boot = \"boots\" if kv * d >= full else \"no\"    print(f\"{tp:<2} {d:3} {w:6.2f} {kv:6.2f}  {tok:11,.0f} {tok / CTX:5.2f}\"          f\" {reps:4} {tok * reps / 1e6:8.2f}M  {boot}\")print(f\"\\n1M? = can vLLM start with max_model_len\"      f\" {cfg['max_position_embeddings']:,} ({full:.2f} GB)\")if len(sys.argv) > 3:    o = load(sys.argv[3])[\"metadata\"][\"total_size\"] / 1e9    print(f\"other checkpoint {o:.2f} GB; this one is\"          f\" {100 * (1 - wt / o):.1f}% smaller\")\n```\n\nFrom nvidia/GLM-5.3-NVFP4, download config.json and model.safetensors.index.json (save it as nvfp4.index.json). From zai-org/GLM-5.3, the FP8 release, download the index too (save it as fp8.index.json). Then:\n\n```\npython3 glmfit.py config.json nvfp4.index.json fp8.index.json\nweights       464.12 GB  (index.json total_size)indexer      21 of 78 layers (vLLM rule; config lists 21)KV per token 47,700 B = 576 x 78 MLA + 132 x 21 indexerone 262,144-token session = 12.50 GB; at dcp=1 every TP rank holds all of itbudget       180 GB x 0.92 - 12 GB overhead (assumed)TP dcp wt/GPU KV/GPU  tok/replica  sess reps  fleet tok  1M?4    1 116.03  37.57      787,653  3.00   14    11.03M  no8    1  58.01  95.59    2,003,889  7.64    7    14.03M  boots1M? = can vLLM start with max_model_len 1,048,576 (50.02 GB)other checkpoint 755.62 GB; this one is 38.6% smaller\n```\n\nRead the sess column: full 256K sessions per copy of the model. TP4 gets 3.00 and TP8 gets 7.64. Across 56 GPUs, TP4's 14 copies hold 11.03M cached tokens and TP8's 7 copies hold 14.03M. That's 27% more cache from half as many copies.\n\nThe last line compares the two checkpoints’ total_size: 464.12 GB for NVIDIA's NVFP4 against 755.62 GB for Z.ai's FP8. That's 38.6% smaller, close to the talk's \"about 40%\".\n\nWhy not 50%, if 4 bits is half of 8? The Hub lists 27.9 billion BF16 parameters in the NVFP4 repo. Its config puts only the mlp.experts modules in NVFP4 and the first three dense MLPs in FP8. The multi-token-prediction layer, model.layers.78*, is excluded from quantization. The rest stays BF16.\n\nGLM-5.3’s config advertises max_position_embeddings: 1048576. vLLM's check_enough_kv_cache_memory refuses to start if a single request of max_model_len can't fit. At 47,700 bytes a token, one 1M-token request needs 50.02 GB.\n\nOn TP4 each GPU has 37.57 GB for cache, and each GPU must hold the whole request. So the 1M? column says no. Set --max-model-len to 256K (anything under about 787K tokens fits) and, by the script's numbers, the startup check passes. \"Long context is a scam\" is literally true here.\n\n```\npython3 glmfit.py config.json fp8.index.json\nTP dcp wt/GPU KV/GPU  tok/replica  sess reps  fleet tok  1M?4    - 188.90  weights alone exceed the budget8    1  94.45  59.15    1,239,997  4.73    7     8.68M  boots\n```\n\nSame config.json; only the weight size changes. At 188.90 GB per GPU, FP8 weights alone are bigger than the 153.6 GB the script leaves for weights and cache. On 8 GPUs it fits but holds 4.73 sessions against NVFP4's 7.64. NVFP4 is what makes 4 GPUs possible at all.\n\nFirst, vLLM can split the cache across GPUs instead of copying it: --decode-context-parallel-size (-dcp). When vLLM checks whether a max-length request fits, it divides max_model_len by the DCP size. The script assumes the cache splits evenly. With DCP=8:\n\n```\nTP dcp wt/GPU KV/GPU  tok/replica  sess reps  fleet tok  1M?4    1 116.03  37.57      787,653  3.00   14    11.03M  no8    8  58.01  95.59   16,031,116 61.15    7   112.22M  boots\n```\n\nTP8 goes from 7.64 to 61.15 sessions per copy. vLLM refuses -dcp 8 on 4 GPUs (the GPU count must be divisible by the DCP size), so the script shows that row with DCP off. With DCP=4, TP4 reaches 12.02 sessions and TP8 30.58, so 8 GPUs still hold more.\n\nI read this in source and did not run it on B200s. DCP adds GPU-to-GPU traffic on every decode step, so measure speed before you trust it. The default FlashInfer backend supports DCP with GLM’s a2a default. If you switch to the FlashMLA backend, it only accepts --dcp-comm-backend ag_rs.\n\nSecond, the FlashMLA backend (--attention-backend FLASHMLA_SPARSE) stores 656 bytes per layer instead of 576. Run MLA=656 and the numbers drop to 2.66 and 6.76 sessions. 8 GPUs still hold over 2.5 times as many.\n\nEvery session, token, GB and percentage in sections 05–10 comes from glmfit.py, either in the output blocks or from the env runs named beside them (DCP=8, DCP=4, MLA=656, OVERHEAD_GB=24, MTP=1). The 27% and 153.6 GB are arithmetic on its output, and the lede rounds 3.00 and 7.64. The talk's figures (56 B200s, 2,000 engineers, 3 seconds, about 40%) come from the captions. The Hub's dtype counts and quantization layout in section 06, and the vLLM file references, come from the Hub pages and the vLLM 0.32.0 wheel. None of this was run on a GPU.\n\nvLLM keeps a full copy of GLM-5.3’s attention cache on every GPU in a tensor-parallel group. On paper that gives 8 GPUs 7.64 full 256K sessions per copy against 3.00 for 4 GPUs, and without DCP, 4 GPUs can’t even start at the advertised 1M context. Revolut’s advice, to cap context at 256K and try 8 GPUs, matches what vLLM’s code implies.\n\nWhen you size a self-hosted coding model, do you count the weights, or the sessions the cache can hold?\n\nIf this saved you a GPU order, a clap helps others find it.\n\nFollow me for daily LLM deep-dives.\n\n[Why Self-Hosted GLM-5.3 Needs 8 GPUs, Not 4, for Long Coding Sessions](https://pub.towardsai.net/why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions-b5b38fa3d143) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions", "canonical_source": "https://pub.towardsai.net/why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions-b5b38fa3d143?source=rss----98111c9905da---4", "published_at": "2026-10-30 09:54:34+00:00", "updated_at": "2026-10-30 10:20:41.257312+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["GLM-5.3", "vLLM 0.32.0", "David Abitbol", "Revolut", "Nvidia B200", "AI Engineer YouTube channel", "glmfit.py"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions", "markdown": "https://wpnews.pro/news/why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions.md", "text": "https://wpnews.pro/news/why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions.txt", "jsonld": "https://wpnews.pro/news/why-self-hosted-glm-5-3-needs-8-gpus-not-4-for-long-coding-sessions.jsonld"}}