When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

NVIDIA published a tutorial showing how to serve its GLM-5.3 NVFP4 model, a quantized version of Z.ai's 753-billion-parameter mixture-of-experts model with 40 billion parameters activated per token, across two GB300 DGX Stations using vLLM with tensor parallelism (TP=2) over ConnectX-8 RDMA. The roughly 464 GB checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU, so the setup keeps all weights in GPU memory with CPU offload disabled, splitting tensors across both GPUs while Station A (rank 0) exposes a single OpenAI-compatible API endpoint and Station B (rank 1) runs only a worker. The tutorial also walks through prerequisites — including 464 GB of free disk per Station and acceptance of the NVIDIA Open Model Agreement — and measures behavior under concurrent load using SPEED-Bench prompts from the gated cais/hle dataset.

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs. The NVIDIA GLM-5.3 NVFP4 https://huggingface.co/nvidia/GLM-5.3-NVFP4 model, a quantized version of Z.ai http://z.ai ’s mixture-of-experts MoE model, has 753 billion parameters, with 40 billion activated per token. Sparse activation reduces computation, but all model weights still need to remain available because each token can be routed to different experts. At about 464 GB, the checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU. This tutorial uses two DGX Stations to keep the weights in GPU memory, with CPU weight offload disabled. Before starting, connect the two Stations with the Connect Two DGX Stations playbook https://build.nvidia.com/station/connect-two-stations . This post shows how to serve GLM-5.3 locally across two NVIDIA DGX Stations with vLLM, send a request, and measure behavior under concurrent load. Tensor parallelism TP=2 splits the model’s tensors across the two GB300 GPUs. Both Stations compute every request, and Station A returns the answer through one OpenAI-compatible API. By the end, you’ll have a working endpoint and a baseline for how it behaves under load. Prerequisites - Two GB300 DGX Stations running Linux, connected and validated with the Connect Two DGX Stations playbook https://build.nvidia.com/station/connect-two-stations , including its GPU-memory RDMA test on both rails. - SSH and sudo access on both Stations with the same Linux username. Docker and the NVIDIA Container Toolkit come preinstalled with the DGX Station software stack https://docs.nvidia.com/dgx/dgx-station-development-guide/porting/software-requirements.html . Allow outbound access to the container registry, Hugging Face, and package repositories. - At least 464 GB of free disk space on each Station for its copy of the checkpoint, plus room for the vLLM container image and temporary files. - Acceptance of the NVIDIA Open Model Agreement and the original GLM-5.3 license terms, listed on the model card https://huggingface.co/nvidia/GLM-5.3-NVFP4 licenseterms-of-use . SPEED-Bench is governed by the NVIDIA Evaluation Dataset License Agreement https://huggingface.co/datasets/nvidia/SPEED-Bench/blob/main/License.pdf . - A Hugging Face account and read access token, with access to the gated cais/hle https://huggingface.co/datasets/cais/hle dataset: sign in, open its page, and select Agree and access repository . Step 5 downloads SPEED-Bench prompts from this dataset. The companion video follows the same workflow. Step 1. Prepare the two Stations This tutorial uses a separate computer, such as your laptop, to manage both Stations. Start by opening an SSH session to each Station from that computer. Alternatively, work from a local terminal on each Station. Run every command in this step on both Stations unless it starts with “On A.” The two Stations run the same container and checkpoint in different roles: Station A rank 0 hosts the API, and Station B rank 1 runs only a worker, with no API server. First, check Docker access and collect the network values you’ll use later: docker version ibdev2netdev ip -br addr | Note: If the docker version output has no Server section, the client cannot reach the Docker daemon. If the error is permission denied , follow the Docker post-installation procedure https://docs.docker.com/engine/install/linux-postinstall/ : add your user to the docker group with sudo usermod -aG docker "$USER" , run exit , then reconnect and rerun docker version . The docker group grants root-level privileges. | |---| The ibdev2netdev output lists the two direct rails, cx8r0 and cx8r1 . Pick one for serving traffic, for example cx8r1 , and note two things for it on each Station: - Its RDMA device , on the ibdev2netdev line that ends with the rail’s name. In mlx5 1 port 1 == cx8r1 Up , the device is mlx5 1 . - Its IPv4 address , on the ip -br addr line that starts with the rail’s name. In cx8r1 UP 192.168.101.1/24 , the address is 192.168.101.1 . The rail name and device are the same on both Stations; only the IPv4 address differs. Next, create ~/glm53.env on each Station. This file stores the values that later commands use: nano ~/glm53.env Paste the following block and replace each <... placeholder, keeping the quotes, with the values you noted. RANK="<NODE RANK " RANK="0" on Station A, RANK="1" on Station B HEAD IP="<STATION A IPV4 ADDRESS " IPv4 address you noted on Station A; same in both files WORKER IP="<STATION B IPV4 ADDRESS " IPv4 address you noted on Station B; same in both files RAIL IFACE="<RAIL NAME " Rail you picked, for example cx8r1; same in both files RAIL HCA="<RDMA DEVICE " RDMA device you noted, for example mlx5 1; same in both files IMG=vllm/vllm-openai:v0.30.0-aarch64 IMG+=@sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56 MODEL REPO=nvidia/GLM-5.3-NVFP4 MODEL REV=e3b8f9f2fad29727dfa95e766c5cac18ea1414bf MODEL HOST="$HOME/models/GLM-5.3-NVFP4" CONTAINER=glm53-full PORT=8000 if "$RANK" = 0 ; then NODE IP=$HEAD IP; PEER IP=$WORKER IP else NODE IP=$WORKER IP; PEER IP=$HEAD IP fi GPU UUID=$ nvidia-smi --query-gpu=uuid,name --format=csv,noheader | awk -F, '$2 ~ /GB300/ {gsub / /,"",$1 ; print $1; exit}' For example, with the default addresses on cx8r1 , Station A’s file starts like this. Station B’s file is identical except for RANK="1" : RANK="0" HEAD IP="192.168.101.1" WORKER IP="192.168.101.2" RAIL IFACE="cx8r1" RAIL HCA="mlx5 1" MODEL HOST is built from your home directory, and Step 2 copies the checkpoint over SSH to the same path on Station B. | Note: Later steps start with source ~/glm53.env to load these values. Run this command again whenever you open a new SSH session. | |---| On both Stations, load the values and check storage, connectivity, and baseline memory: source ~/glm53.env mkdir -p "$MODEL HOST" && df -h "$MODEL HOST" ethtool "$RAIL IFACE" ip link show dev "$RAIL IFACE" ping -I "$RAIL IFACE" -M do -s 8972 -c 3 "$PEER IP" timedatectl show -p NTPSynchronized --value nvidia-smi These checks catch cabling and addressing mistakes before they surface as NCCL timeouts. Expect a 400000Mb/s link, MTU 9000, successful jumbo-frame pings, and synchronized time. nvidia-smi should show no compute processes on the GB300. On a Station with a display GPU, Xorg can appear on both GPUs with a few MiB. | Tip: Note the GPU memory use shown by nvidia-smi ; Step 4 compares it with the loaded model. | |---| Step 2 copies the checkpoint from A to B over SSH, so A needs passwordless access to B. On A, create a key without a passphrase see SSH keys https://ubuntu.com/server/docs/how-to/security/openssh-server/ ssh-keys for details , copy its public key to B after checking B’s host fingerprint, and confirm the login: ssh-keygen -t ed25519 ssh-copy-id "$WORKER IP" ssh -o BatchMode=yes "$WORKER IP" hostname Step 2. Download and copy the model Tensor parallelism requires both ranks to run the same software against the same weights. Start by pulling the pinned image on both Stations: docker pull "$IMG" && docker image inspect "$IMG" --format '{{.Architecture}} {{.RepoDigests}}' The pull can take several minutes. Matching arm64 digests mean both ranks run identical software. On A, download the pinned checkpoint. At about 464 GB, it takes roughly an hour at 1 Gb/s, or about six minutes at 10 Gb/s: mkdir -p "$MODEL HOST" docker run --rm --network host -v "$MODEL HOST:/model" \ --entrypoint bash "$IMG" -lc \ "exec hf download '$MODEL REPO' --revision '$MODEL REV' --local-dir /model" When hf download prints Downloaded , copy the files to B. On A: ssh "$WORKER IP" "mkdir -p '$MODEL HOST'" && rsync -aH --info=progress2 --exclude '.cache/' \ "$MODEL HOST/" "$WORKER IP:$MODEL HOST/" Each disk keeps the full checkpoint; vLLM partitions the weights across the GPUs at load time, with no further quantization. On A, compare both copies the check reads B’s file list over SSH , then validate the checkpoint: set -euo pipefail L=$ find "$MODEL HOST" -path ' /.cache' -prune -o \ -type f -printf '%P %s\n' | LC ALL=C sort | sha256sum R=$ ssh "$WORKER IP" "set -o pipefail; find '$MODEL HOST' \ -path ' /.cache' -prune -o -type f -printf '%P %s\\n' | \ LC ALL=C sort | sha256sum" test "$L" = "$R" test "$ find "$MODEL HOST" -maxdepth 1 \ -name 'model- -of-00050.safetensors' | wc -l " = 50 python3 -c "import json; c=json.load open '$MODEL HOST/config.json' ; \ assert c 'architectures' == 'GlmMoeDsaForCausalLM' ; \ assert c 'quantization config' 'quant algo' =='MIXED PRECISION'" echo MODEL READY MODEL READY means both Stations hold the same file names and sizes in the expected layout. MIXED PRECISION is expected: NVFP4 quantization targets MoE-block linear weights and activations, not every tensor. Step 3. Launch the distributed service This step turns two machines into one service: the same launch script runs on both Stations, reads ~/glm53.env , and starts one vLLM rank on each. Two settings matter most if you adapt it: --max-model-len 98304 , the context ceiling in tokens, which covers input plus output, and --gpu-memory-utilization 0.93 , which gives vLLM 93% of GPU memory. If you change other flags, change one at a time and test again. For the latest recommended configurations, see NVIDIA’s Local AI recipe repository https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station-2gpu/index.html . | Security note: --host 0.0.0.0 with --network host exposes the unauthenticated API on every Station A interface, including management. Firewall the port to intended clients, and add authentication and TLS before wider access. | |---| On both Stations, create a file named ~/launch-glm53.sh : nano ~/launch-glm53.sh The script enables multi-token prediction MTP , a form of speculative decoding: the model’s MTP layer drafts four tokens per step, and vLLM verifies them with the full model before accepting them, so each step can produce several tokens. The gain depends on how often drafts are accepted, which varies with the prompt. FP8 KV-cache storage halves cache memory compared with 16-bit storage, leaving room for more tokens. Prefix caching is enabled to avoid reprocessing prompt prefixes the server has already seen. Paste the script below; it checks your configuration, finds the RoCE GID index for the selected rail, and starts this Station’s vLLM rank in a container: set -euo pipefail source "$HOME/glm53.env" case "$RANK" in 0|1 ;; echo "Invalid RANK" &2; exit 1;; esac test -n "$GPU UUID" test -f "$MODEL HOST/config.json" GID INDEX=$ /usr/sbin/show gids 2 /dev/null | awk -v d="$RAIL HCA" -v ip="$NODE IP" \ '$1==d && $5==ip && $6=="v2" {g=$3} END {if g ="" print g}' test -n "$GID INDEX" HEADLESS= if "$RANK" = 1 ; then HEADLESS=--headless; fi SPEC CONFIG='{"method":"mtp","num speculative tokens":4}' KERNEL CONFIG='{"enable flashinfer autotune":false,"moe backend":"flashinfer cutlass"}' COMPILE CONFIG='{"cudagraph mode":"FULL DECODE ONLY","cudagraph capture sizes": 5,10,20,40,80 ,"pass config":{"fuse allreduce rms":false}}' docker run -d --name "$CONTAINER" \ --gpus device="$GPU UUID" --network host --ipc host --shm-size 32g \ --device /dev/infiniband --cap-add IPC LOCK \ --ulimit memlock=-1 --ulimit stack=67108864 \ -v "$MODEL HOST:/model:ro" \ -e RANK="$RANK" -e HEADLESS="$HEADLESS" \ -e MASTER ADDR="$HEAD IP" -e MODEL REPO="$MODEL REPO" -e PORT="$PORT" \ -e SPEC CONFIG="$SPEC CONFIG" -e KERNEL CONFIG="$KERNEL CONFIG" \ -e COMPILE CONFIG="$COMPILE CONFIG" \ -e NCCL CUMEM ENABLE=1 -e NCCL IB HCA="$RAIL HCA" \ -e NCCL IB GID INDEX="$GID INDEX" \ -e NCCL IB TIMEOUT=22 -e NCCL IB RETRY CNT=7 \ -e VLLM ALLREDUCE USE FLASHINFER=0 -e VLLM ALLREDUCE USE SYMM MEM=0 \ -e NCCL SOCKET IFNAME="$RAIL IFACE" -e GLOO SOCKET IFNAME="$RAIL IFACE" \ -e NCCL NET GDR LEVEL=5 -e NCCL DEBUG=INFO -e NCCL DEBUG SUBSYS=INIT,NET \ -e NCCL DEBUG FILE='/tmp/nccl %h %p.log' -e VLLM HOST IP="$NODE IP" \ -e VLLM EXECUTE MODEL TIMEOUT SECONDS=1800 \ -e PYTORCH CUDA ALLOC CONF=expandable segments:True \ -e VLLM ALLOW LONG MAX MODEL LEN=1 -e FLASHINFER DISABLE VERSION CHECK=1 \ --entrypoint bash "$IMG" -lc ' set -euo pipefail Use NCCL 2.30.7 from the image: older versions can deadlock cross-node CUDA graph replays over RoCE vLLM issue 52504 . export VLLM NCCL SO PATH=$ python3 -c "import nvidia.nccl,os; print os.path.join list nvidia.nccl. path 0 , \"lib\", \"libnccl.so.2\" " test -f "$VLLM NCCL SO PATH" EXTRA= if -n "$HEADLESS" ; then EXTRA+= "$HEADLESS" ; fi exec vllm serve /model --served-model-name "$MODEL REPO" \ --host 0.0.0.0 --port "$PORT" \ --nnodes 2 --node-rank "$RANK" --master-addr "$MASTER ADDR" \ --master-port 29501 --tensor-parallel-size 2 \ --speculative-config "$SPEC CONFIG" \ --max-model-len 98304 --max-num-batched-tokens 8192 \ --block-size 128 --max-num-seqs 16 \ --trust-remote-code --disable-custom-all-reduce \ --kv-cache-dtype fp8 --enable-prefix-caching \ --gpu-memory-utilization 0.93 \ --enable-prompt-tokens-details \ --kernel-config "$KERNEL CONFIG" --compilation-config "$COMPILE CONFIG" \ --distributed-timeout-seconds 1800 \ --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \ "${EXTRA @ }" ' Run the script on B first, then on A: bash ~/launch-glm53.sh Each command prints a container ID and returns immediately; the service is not ready yet. B’s worker waits for A to join, then each Station loads its share of the weights from local storage, prepares kernels, and captures CUDA graphs. This takes several minutes. On A, follow the log to see when the service is ready --since hides output from earlier runs : docker logs --follow --since "$ docker inspect \ --format '{{.State.StartedAt}}' "$CONTAINER" " "$CONTAINER" When the log shows Application startup complete , press Ctrl+C to stop following it; the service keeps running. At this point one model spans two machines. Neither GPU holds a complete copy, and neither can answer a request alone: every token needs both ranks to exchange partial results over the RoCE rail. The remaining GPU memory holds the KV cache, which stores the context of in-flight requests. The startup log reports its capacity https://docs.vllm.ai/en/v0.30.0/serving/parallelism scaling/ GPU KV cache size . All requests in flight share it, so longer prompts or more users need more of it. If you run out, lower --max-model-len or send fewer requests at once. Step 4. Verify serving and send a request On A, check the API 127.0.0.1 is A itself : docker exec "$CONTAINER" curl -fsS --max-time 10 \ "http://127.0.0.1:$PORT/health" /dev/null && echo HEALTH OK HEALTH OK proves that Station A answers. Next, on both Stations, check that the ranks communicate over GPU RDMA rather than falling back to TCP sockets. The check reads the NCCL logs that the launcher writes through NCCL DEBUG=INFO , and the same block shows the loaded memory: docker exec "$CONTAINER" bash -lc ' set -euo pipefail files= /tmp/nccl .log test -e "${files 0 }" grep -hE "NET/IB. GDRDMA|GPU Direct RDMA Enabled" "${files @ }" /dev/null grep -hF "$NCCL IB HCA" "${files @ }" /dev/null if grep -hE "NET/Socket|NCCL WARN|NCCL ERROR" "${files @ }"; then exit 1 fi echo RDMA CHECK OK ' nvidia-smi RDMA CHECK OK means the check passed with no warnings. In nvidia-smi , A shows the VLLM::Worker TP0 process and B shows VLLM::Worker TP1 , one rank per GPU. The startup log reports about 218 GiB of weights per GPU, roughly half the checkpoint, which confirms that the model is split across the two GPUs. | Measurement | Station A | Station B | |---|---|---| | GB300 GPU used GiB | 0.01 → 231.35 | 0.00 → 231.34 | Table 1. GPU memory before and after loading the model, with no requests running. Values include weights, KV cache, and runtime buffers On A, send a chat request through the OpenAI-compatible API: docker exec "$CONTAINER" curl -fsS --max-time 120 \ "http://127.0.0.1:$PORT/v1/chat/completions" \ -H 'Content-Type: application/json' \ -d '{"model":"nvidia/GLM-5.3-NVFP4", "messages": {"role":"user","content":"Quarterly units sold: Q1 = 120; Q2 = 180; Q3 = 150. Summarize the trend in one sentence and calculate the percentage change from Q1 to Q2."} , "temperature":0,"max tokens":4096, "chat template kwargs":{"reasoning effort":"low"}}' | python3 -c 'import json,sys; c=json.load sys.stdin "choices" 0 ; print c "message" "content" ; print "finish reason:",c "finish reason" ' The response should look like this: Trend: Units sold rose sharply in Q2 before partially declining in Q3, remaining above the Q1 level. Q1 → Q2 percentage change: 180 - 120 / 120 × 100 = 50% increase finish reason: stop finish reason: stop means the model finished its answer instead of hitting the token limit. The request sets reasoning effort to low in chat template kwargs . GLM-5.3 also accepts high and max , which let it reason longer before answering, at the cost of latency. Because the launcher enables tool calling --enable-auto-tool-choice , agent frameworks that use the OpenAI tools API can call the same endpoint. Step 5. Concurrency and release resources AIPerf https://docs.nvidia.com/aiperf/latest/welcome-to-ai-perf-documentation , the NVIDIA benchmarking tool for inference endpoints, measures throughput and latency. Installing it inside the serving container can change vLLM’s dependencies, so on A, install it in its own virtual environment: sudo apt update && sudo apt install -y python3.12-venv python3.12-dev build-essential && python3.12 -m venv "$HOME/.venvs/aiperf-0.12.0" && "$HOME/.venvs/aiperf-0.12.0/bin/pip" install 'aiperf==0.12.0' && "$HOME/.venvs/aiperf-0.12.0/bin/pip" check The install should end with No broken requirements found . AIPerf can send synthetic prompts or replay an existing dataset. This tutorial uses the 80 Coding conversations from SPEED-Bench, a set of real prompts built to evaluate speculative decoding, so MTP is measured on realistic text. SPEED-Bench’s preparation script fetches these prompts from several Hugging Face datasets, including the gated cais/hle https://huggingface.co/datasets/cais/hle : if you haven’t already, open its page and select Agree and access repository before you continue. Then, on A, log in to Hugging Face: "$HOME/.venvs/aiperf-0.12.0/bin/hf" auth login Next, download and prepare the dataset. See the official AIPerf guide https://docs.nvidia.com/aiperf/v0.12.0/tutorials/datasets-inputs/profile-with-speed-bench-dataset prepare-the-dataset for details: SPEED BENCH DIR="$HOME/datasets/speed-bench" mkdir -p "$SPEED BENCH DIR" curl -fLsS https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo skills/dataset/speed-bench/prepare.py | \ "$HOME/.venvs/aiperf-0.12.0/bin/python" - --config qualitative --output dir "$SPEED BENCH DIR" On A, check the dataset and service. Continue when the check prints BENCHMARK READY : set -euo pipefail source ~/glm53.env curl -fsS "http://$HEAD IP:$PORT/health" /dev/null "$HOME/.venvs/aiperf-0.12.0/bin/python" - <<'PY CHECK' import json from pathlib import Path root = Path.home path = root / "datasets/speed-bench/qualitative.jsonl" assert path.exists , "Dataset missing: rerun the SPEED-Bench preparation" rows = json.loads line for line in path.read text .splitlines if line.strip coding = row for row in rows if row "category" == "coding" assert len coding == 80, "Expected 80 Coding conversations" assert sum len row "messages" for row in coding == 89 assert all "FULL BENCHMARK DATA SHOULD BE FETCHED" not in message "content" for row in coding for message in row "messages" print "BENCHMARK READY: dataset has 80 Coding conversations, 89 turns in total." PY CHECK The full sweep covers concurrency 1, 8, 16, and 32. At each level, AIPerf sends 3×C warmup requests, then measures 10×C requests, where C is the concurrency. Some conversations are sent more than once at higher concurrency. Benchmark requests use the model’s default reasoning effort max and an 8,192-token output cap; responses can finish before reaching it. Full sweeps can take several hours. A two-hour request timeout leaves room for long responses, --server-metrics collects server counters used to measure MTP acceptance, and --export-level raw keeps each response for inspection. For a quick test, run the block below on A with both containers running and no other API traffic. It tests C1 and C8 with prefix caching enabled, as in the reference measurements in Table 2, below. set -euo pipefail source ~/glm53.env CODING DATASET="$HOME/datasets/speed-bench/qualitative.jsonl" test -f "$CODING DATASET" export CONCURRENCIES=${CONCURRENCIES:-"1 8"} Quick run: C1 and C8. Optional full benchmark: follow the reproduction instructions below. Step 3: --max-num-seqs 16 - export CONCURRENCIES="1 8 16" Step 3: --max-num-seqs 32 - export CONCURRENCIES=32 RUN DIR="$HOME/aiperf-results/blog-coding-$ date +%Y%m%d-%H%M%S " for C in $CONCURRENCIES; do ART="$RUN DIR/c$C" env -u HF HUB OFFLINE -u TRANSFORMERS OFFLINE \ "$HOME/.venvs/aiperf-0.12.0/bin/aiperf" profile \ -m "$MODEL REPO" --url "http://$HEAD IP:$PORT" \ --endpoint-type chat --streaming --tokenizer "$MODEL HOST" \ --custom-dataset-type speed bench coding --input-file "$CODING DATASET" \ --request-count "$ 10 C " --warmup-request-count "$ 3 C " \ --osl 8192 \ --use-server-token-count --concurrency "$C" --random-seed 42 \ --request-timeout-seconds 7200 \ --server-metrics "http://$HEAD IP:$PORT/metrics" \ --export-level raw --artifact-dir "$ART" printf "Results: %s\n" "$ART" done printf "Results directory: %s\n" "$RUN DIR" AIPerf saves metrics, raw responses, and logs in the results directory. Warmup requests are excluded from the reported metrics. Optional: Reproduce Table 2 . In ~/launch-glm53.sh from Step 3 on both Stations, keep --enable-prefix-caching and set --max-num-seqs to 16 or 32 for the corresponding group shown in the comments above. Before each group, stop and remove the current containers on A, then B docker stop --timeout 60 "$CONTAINER" , followed by docker rm "$CONTAINER" . Relaunch the updated script on B, then A with bash ~/launch-glm53.sh and wait for Application startup complete . On A, run the matching export CONCURRENCIES command from the comments, then rerun the AIPerf block. To test without prefix caching, replace --enable-prefix-caching with --no-enable-prefix-caching in the Step 3 script on both Stations, then recreate the containers as above. Performance may differ from Table 2. Figure 2 compares overall output throughput across measured requests with mean per-user throughput. Token counts include reasoning. Table 2, below, shows how throughput and latency change as more requests share the service. | Configured concurrency | Overall throughput tok/s | Mean request latency s | Mean TTFT ms | Mean inter-token latency ms/token | | 1 | 107.4 | 49.84 | 479.75 | 8.78 | | 8 | 386.0 | 114.17 | 433.79 | 19.01 | | 16 | 607.8 | 136.46 | 482.90 | 23.77 | | 32 | 385.0 | 453.54 | 899.36 | 77.90 | Table 2. SPEED-Bench Coding results with prefix caching enabled, four MTP draft tokens per step, and an FP8 KV cache. Latency metrics are means over measured requests; warmup is excluded. TTFT measures time to the first token, which may be reasoning Overall throughput peaks at 607.8 tok/s at C16. It falls to 385.0 tok/s at C32, where mean request latency is 453.5 s. Higher concurrency also reduces mean per-user throughput, from 115.3 tok/s at C1 to 13.0 tok/s at C32. After the requests finish, stop the vLLM container on A, then on B: docker stop --timeout 60 "$CONTAINER" docker inspect --format \ 'Status={{.State.Status}} Running={{.State.Running}}' "$CONTAINER" nvidia-smi The status should read Status=exited Running=false , with no vLLM processes in nvidia-smi ; the checkpoint, containers, and results stay on disk. To resume, run docker start "$CONTAINER" on B, then A, and repeat the Step 4 checks. Troubleshoot common blockers - Missing variables or SSH failure: source ~/glm53.env after reconnecting; verify A-to-B key authentication and B’s fingerprint. - SPEED-Bench preparation fails with a gated-dataset error: sign in to Hugging Face with the same account, open the cais/hle https://huggingface.co/datasets/cais/hle page, select Agree and access repository , then rerun the preparation. - Startup or RDMA check failure: inspect docker logs "$CONTAINER" on both nodes; verify the checkpoint and the GID, and rerun the playbook’s validation script 07 . - CUDA out of memory https://docs.vllm.ai/en/v0.30.0/usage/troubleshooting/ : lower --max-model-len or concurrency, then retest. Put the endpoint to work You now have a 753-billion-parameter model running behind one local API on two NVIDIA DGX Stations, with every weight in GPU memory. That opens up work that’s hard to do on a hosted endpoint: running inference on sensitive or proprietary data, prototyping agents locally, and iterating on prompts without per-token costs. Connect your own application using the vLLM serving documentation https://docs.vllm.ai/en/v0.30.0/serving/online serving/ , then rerun AIPerf with your prompts to find the concurrency that fits your workload. A frontier-scale model is now ready to build on, right from your desk.