# GLM-5.3 on Two DGX Stations: NVIDIA Shows 753B Running Locally with vLLM > Source: > Published: 2026-11-13 00:00:00+00:00 NVIDIA has published a step-by-step guide to serving Z.ai's 753-billion-parameter GLM-5.3 on two DGX Stations with vLLM. The model checkpoint is about 464 GB, and a single GB300 GPU holds 252 GB of HBM3e, so no one machine can run it. The fix is tensor parallelism across both Stations. In [NVIDIA's own post](https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/?ref=explainx), one user gets 107.4 tokens per second, and total throughput peaks at 607.8 tokens per second. This is a local-AI recipe, not a product launch. It shows how far "frontier model on your own desk" has moved since we covered [GLM-5.3's vision work](https://explainx.ai/blog/glm-5-3-vision-community-input-zhipu-jie-tang-2026) and NVIDIA's [eight-node DGX Spark cluster for the same model](https://explainx.ai/blog/nvidia-dgx-spark-eight-node-cluster-1tb-glm-5-3-753b-roce-decode-4-43x-switchyard-2026). ## TL;DR: the setup in one table | Question | Answer (from NVIDIA's post) | |---|---| | What model? | `nvidia/GLM-5.3-NVFP4` , a quantized build of Z.ai's mixture-of-experts GLM-5.3 | | How big? | 753B total parameters, 40B active per token, about 464 GB on disk | | Why two machines? | One GB300 GPU has 252 GB of HBM3e, so the weights do not fit | | How are they linked? | Direct ConnectX-8 RDMA (RoCE), expected 400000 Mb/s, MTU 9000 | | Which engine? | `vllm/vllm-openai` v0.30.0 (aarch64), tensor-parallel-size 2 | | Peak throughput? | 607.8 tok/s at 16 concurrent requests | | Single-user speed? | 107.4 tok/s, 8.78 ms per token between tokens | | Is the API protected? | No: it is unauthenticated unless you add a firewall, auth and TLS | ## Why does a 40B-active model still need 464 GB? GLM-5.3 is a mixture-of-experts model. Only 40 billion of its 753 billion parameters run for any one token, which is why it decodes quickly. But a router picks different experts for each token, so every expert must stay resident in GPU memory. NVIDIA states this plainly: sparse activation does not shrink the memory requirement. The NVFP4 format helps. It targets the linear weights and activations of the MoE blocks, which is why the checkpoint reports `MIXED_PRECISION` quantization instead of one uniform format. Even so, the checkpoint is 50 safetensors shards totalling about 464 GB. High-bandwidth memory stack feeding a chip, illustrating why GLM-5.3 weights must fit in GPU memory With tensor parallelism set to 2, each GPU holds roughly half of every weight tensor. NVIDIA's startup logs show about 218 GiB of weights per GPU. After loading and before any request, Station A used 231.35 GiB and Station B used 231.34 GiB. There is no CPU weight offload, so both GPUs compute every request and neither can answer alone. ## How is the two-Station cluster wired? Each DGX Station carries a GB300 GPU and ConnectX-8 networking. NVIDIA connects the two machines directly over two RoCE rails named `cx8r0` and `cx8r1`. The worked example uses `cx8r1`, RDMA device `mlx5_1`, and the addresses 192.168.101.1 (Station A) and 192.168.101.2 (Station B). Before starting, you check that the link reports 400000 Mb/s, that MTU is 9000, and that jumbo-frame pings succeed. Each Station needs its own full copy of the checkpoint, so you need at least 464 GB free on each, plus room for the container image. NVIDIA estimates the download at about an hour on a 1 Gb/s connection and about six minutes at 10 Gb/s. Station A copies the checkpoint to Station B with rsync over passwordless SSH. The roles are asymmetric. Station A is rank 0 and hosts the OpenAI-compatible API on port 8000. Station B is rank 1 and runs a headless worker with no API server. ## Which vLLM flags matter? NVIDIA pins the container by SHA-256 digest so both ranks run identical software. The key choices are: | Flag or setting | Value | Why it matters | |---|---|---| | `--tensor-parallel-size` /`--nnodes` | 2 / 2 | Splits each tensor across the two GPUs | | `--max-model-len` | 98304 | Context ceiling for input plus output | | `--gpu-memory-utilization` | 0.93 | Leaves headroom for KV cache and graphs | | `--kv-cache-dtype` | fp8 | Halves KV-cache memory versus 16-bit | | Speculative decoding | MTP, 4 draft tokens | Raises decode speed per step | | `--enable-prefix-caching` | on | Reuses shared prompt prefixes | | `--max-num-seqs` | 16 (32 for one benchmark group) | Caps concurrent sequences | | MoE backend | `flashinfer_cutlass` | Kernel path for the expert layers | | CUDA graph mode | `FULL_DECODE_ONLY` | Captures decode steps at sizes 5, 10, 20, 40, 80 | | Parsers | `glm47` for tools and reasoning | Correct tool-call and reasoning output | The NCCL detail is the one most likely to save you a day. NVIDIA uses NCCL 2.30.7 from the image, because older versions can deadlock on cross-node CUDA graph replays over RoCE (vLLM issue 52504). It also sets an InfiniBand timeout of 22, a retry count of 7 and GPUDirect RDMA level 5, pinned to the chosen device and GID index. For background on how vLLM-style engines are changing for local hardware, see our coverage of [NVIDIA Flare, a Rust local inference server](https://explainx.ai/blog/nvidia-flare-local-inference-rust-server-sm120-dgx-spark-agents-2026). ## How do you verify it works? NVIDIA gives three checks. `HEALTH_OK` confirms Station A's API answers. `RDMA_CHECK_OK` confirms NCCL logs show GPUDirect RDMA with no socket fallback, warnings or errors. And `nvidia-smi` should show `VLLM::Worker_TP0` on Station A and `VLLM::Worker_TP1` on Station B. The sample request asks a simple sales question (Q1 = 120, Q2 = 180, Q3 = 150) and returns a 50 percent Q1-to-Q2 increase with `finish_reason: stop`. The `reasoning_effort` parameter accepts `low`, `high` and `max`; higher settings allow longer reasoning at the cost of latency. ## What did NVIDIA measure? NVIDIA ran AIPerf 0.12.0 in its own virtual environment so it would not disturb vLLM's dependencies. The workload was 80 Coding conversations (89 turns) from SPEED-Bench, a set of real prompts for evaluating speculative decoding. Each concurrency level sends 3 times C warmup requests and 10 times C measured requests, at default (max) reasoning effort with an 8,192-token output cap. The full sweep takes several hours. | Concurrency | Throughput (tok/s) | Mean latency (s) | Mean TTFT (ms) | Inter-token latency (ms) | |---|---|---|---|---| | 1 | 107.4 | 49.84 | 479.75 | 8.78 | | 8 | 386.0 | 114.17 | 433.79 | 19.01 | | 16 | 607.8 | 136.46 | 482.90 | 23.77 | | 32 | 385.0 | 453.54 | 899.36 | 77.90 | Two readings matter. First, peak total throughput arrives at 16 concurrent requests, and per-user speed drops from 115.3 tok/s at concurrency 1 to 42.4 tok/s at 16. Second, 32 concurrent requests is worse than 16 on every metric, which suggests the cluster is saturated, not scaling. These are NVIDIA's own figures on its own hardware, and the token counts include reasoning tokens. We have not reproduced them. ## How does this compare with smaller local setups? The two-Station recipe sits at the top of the local-AI range. Our [DGX Spark local LLM guide](https://explainx.ai/blog/nvidia-dgx-spark-local-llm-best-setup-2026) covers the entry end, and NVIDIA's [eight-node DGX Spark cluster](https://explainx.ai/blog/nvidia-dgx-spark-eight-node-cluster-1tb-glm-5-3-753b-roce-decode-4-43x-switchyard-2026) showed the full GLM 5.3 at 31 tokens per second on 1 TB of pooled memory. Two Stations are far faster per user, but the hardware costs far more. | Setup | What it shows | Source | |---|---|---| | Two DGX Stations, TP=2 | 107.4 tok/s single user, 607.8 tok/s peak | NVIDIA post, this article | | Eight DGX Sparks | Full GLM 5.3 at 31 tok/s | NVIDIA, covered in our October post | | Microsoft Surface RTX Spark Dev Box | 128 GB unified memory workstation | [Our preorder explainer](https://explainx.ai/blog/surface-rtx-spark-dev-box-5999-preorder-laptop-ultra-local-ai-2026) | For lighter ways to run earlier GLM releases on modest machines, see [Unsloth Studio with GLM-5.2](https://explainx.ai/blog/unsloth-studio-glm-5-2-local-ai-setup-2026) and the [Colibri streaming approach](https://explainx.ai/blog/colibri-glm-5-2-streaming-disk-25gb-ram-july-2026). ## What will readers ask next? **Can I run this on one Station?** Not with this checkpoint. At 464 GB it exceeds the 252 GB of HBM3e on one GB300 GPU, and NVIDIA's recipe has no CPU weight offload. A smaller or more aggressively quantized model would be a different recipe. **Why not just use more concurrency?** The table answers that. Going from 16 to 32 concurrent requests cut total throughput from 607.8 to 385.0 tokens per second and raised mean latency from 136 to 454 seconds. The configuration caps sequences at 16 for most runs for a reason. **Does speculative decoding help here?** NVIDIA enables MTP with 4 draft tokens, which is part of why single-user inter-token latency is only 8.78 ms. The post does not publish a run with it turned off, so we cannot state the gain. **Where is the model card?** NVIDIA publishes the checkpoint on Hugging Face as `nvidia/GLM-5.3-NVFP4`, under the NVIDIA Open Model Agreement plus the original GLM-5.3 license terms. ## What are the risks and gotchas? - **Open API by default.** The recipe binds`--host 0.0.0.0` with host networking, which exposes an unauthenticated API on every interface of Station A. Add a firewall, authentication and TLS. - **Storage and time.** Both machines need the full 464 GB. The benchmark sweep takes hours. - **Gated data.** The benchmark dataset needs you to accept the gated`cais/hle` terms on Hugging Face with the same account. - **Licensing.** The NVIDIA Open Model Agreement and the original GLM-5.3 license terms both apply. - **Memory pressure.** If you hit CUDA out-of-memory, lower`--max-model-len` or concurrency. ## What this means for what you build or pay If you build agents, this is a reference for a private, no-API-bill endpoint with an OpenAI-compatible interface and tool-call parsing. At 107 tok/s for one user, a coding agent is usable. At peak load, the 608 tok/s total is enough for a small team, not a product. If you pay for hosted inference, the cost question is hardware you own against tokens you rent. NVIDIA does not publish a price for this configuration in the post, so we cannot make that comparison. Treat the numbers as an upper bound for tutorial conditions, and test your own prompts before you commit. The main lesson is architectural. Memory capacity, not compute, sets the floor for sparse models, and fast interconnect decides whether splitting across machines works. The same recipe scales down to smaller clusters. ## Related reading *Specifications and benchmark numbers are accurate as of November 13, 2026, and come from NVIDIA's own post.*