# When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

> Source: <https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/>
> Published: 2026-11-13 17:00:00+00:00

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs.

The NVIDIA [GLM-5.3 NVFP4](https://huggingface.co/nvidia/GLM-5.3-NVFP4) model, a quantized version of [Z.ai](http://z.ai)’s mixture-of-experts (MoE) model, has 753 billion parameters, with 40 billion activated per token. Sparse activation reduces computation, but all model weights still need to remain available because each token can be routed to different experts. At about 464 GB, the checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU. This tutorial uses two DGX Stations to keep the weights in GPU memory, with CPU weight offload disabled.

Before starting, connect the two Stations with the [Connect Two DGX Stations playbook](https://build.nvidia.com/station/connect-two-stations).

This post shows how to serve GLM-5.3 locally across two NVIDIA DGX Stations with vLLM, send a request, and measure behavior under concurrent load. Tensor parallelism (TP=2) splits the model’s tensors across the two GB300 GPUs. Both Stations compute every request, and Station A returns the answer through one OpenAI-compatible API. By the end, you’ll have a working endpoint and a baseline for how it behaves under load.

## Prerequisites

- Two GB300 DGX Stations running Linux, connected and validated with the [Connect Two DGX Stations playbook](https://build.nvidia.com/station/connect-two-stations) , including its GPU-memory RDMA test on both rails.
- SSH and sudo access on both Stations with the same Linux username. Docker and the NVIDIA Container Toolkit come preinstalled with the [DGX Station software stack](https://docs.nvidia.com/dgx/dgx-station-development-guide/porting/software-requirements.html) . Allow outbound access to the container registry, Hugging Face, and package repositories.
- At least 464 GB of free disk space on each Station for its copy of the checkpoint, plus room for the vLLM container image and temporary files.
- Acceptance of the NVIDIA Open Model Agreement and the original GLM-5.3 license terms, listed on the [model card](https://huggingface.co/nvidia/GLM-5.3-NVFP4#licenseterms-of-use) . SPEED-Bench is governed by the[NVIDIA Evaluation Dataset License Agreement](https://huggingface.co/datasets/nvidia/SPEED-Bench/blob/main/License.pdf) .
- A Hugging Face account and read access token, with access to the gated [cais/hle](https://huggingface.co/datasets/cais/hle) dataset: sign in, open its page, and select**Agree and access repository** . Step 5 downloads SPEED-Bench prompts from this dataset.

The companion video follows the same workflow.

## Step 1. Prepare the two Stations

This tutorial uses a separate computer, such as your laptop, to manage both Stations. Start by opening an SSH session to each Station from that computer. Alternatively, work from a local terminal on each Station. Run every command in this step on both Stations unless it starts with “On A.”

The two Stations run the same container and checkpoint in different roles: Station A (rank 0) hosts the API, and Station B (rank 1) runs only a worker, with no API server.

First, check Docker access and collect the network values you’ll use later:

```
docker version
ibdev2netdev
ip -br addr
```

| **Note:** If the`docker version` output has no`Server` section, the client cannot reach the Docker daemon. If the error is`permission denied` , follow the[Docker post-installation procedure](https://docs.docker.com/engine/install/linux-postinstall/) : add your user to the`docker` group with`sudo usermod -aG docker "$USER"` , run`exit` , then reconnect and rerun`docker version` . The`docker` group grants root-level privileges. | 
|---|

The `ibdev2netdev` output lists the two direct rails, `cx8r0` and `cx8r1`. Pick one for serving traffic, for example `cx8r1`, and note two things for it on each Station:

- **Its RDMA device** , on the`ibdev2netdev` line that ends with the rail’s name. In`mlx5_1 port 1 ==> cx8r1 (Up)` , the device is`mlx5_1` .
- **Its IPv4 address** , on the`ip -br addr` line that starts with the rail’s name. In`cx8r1 UP 192.168.101.1/24` , the address is`192.168.101.1` .

The rail name and device are the same on both Stations; only the IPv4 address differs.

Next, create `~/glm53.env` on each Station. This file stores the values that later commands use:

```
nano ~/glm53.env
```

Paste the following block and replace each `<...>` placeholder, keeping the quotes, with the values you noted.

```
RANK="<NODE_RANK>"                    # RANK="0" on Station A, RANK="1" on Station B 
HEAD_IP="<STATION_A_IPV4_ADDRESS>"    # IPv4 address you noted on Station A; same in both files
WORKER_IP="<STATION_B_IPV4_ADDRESS>"  # IPv4 address you noted on Station B; same in both files
RAIL_IFACE="<RAIL_NAME>"              # Rail you picked, for example cx8r1; same in both files
RAIL_HCA="<RDMA_DEVICE>"              # RDMA device you noted, for example mlx5_1; same in both files

IMG=vllm/vllm-openai:v0.30.0-aarch64
IMG+=@sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56
MODEL_REPO=nvidia/GLM-5.3-NVFP4
MODEL_REV=e3b8f9f2fad29727dfa95e766c5cac18ea1414bf
MODEL_HOST="$HOME/models/GLM-5.3-NVFP4"
CONTAINER=glm53-full
PORT=8000

if [ "$RANK" = 0 ]; then
  NODE_IP=$HEAD_IP; PEER_IP=$WORKER_IP
else
  NODE_IP=$WORKER_IP; PEER_IP=$HEAD_IP
fi

GPU_UUID=$(nvidia-smi --query-gpu=uuid,name --format=csv,noheader |
  awk -F, '$2 ~ /GB300/ {gsub(/ /,"",$1); print $1; exit}')
```

For example, with the default addresses on `cx8r1`, Station A’s file starts like this. Station B’s file is identical except for `RANK="1"`:

```
RANK="0"
HEAD_IP="192.168.101.1"
WORKER_IP="192.168.101.2"
RAIL_IFACE="cx8r1"
RAIL_HCA="mlx5_1"
```

`MODEL_HOST` is built from your home directory, and Step 2 copies the checkpoint over SSH to the same path on Station B. 

| **Note:** Later steps start with`source ~/glm53.env` to load these values.**Run this command again whenever you open a new SSH session.** | 
|---|

On both Stations, load the values and check storage, connectivity, and baseline memory:

```
source ~/glm53.env
mkdir -p "$MODEL_HOST" && df -h "$MODEL_HOST"
ethtool "$RAIL_IFACE"
ip link show dev "$RAIL_IFACE"
ping -I "$RAIL_IFACE" -M do -s 8972 -c 3 "$PEER_IP"
timedatectl show -p NTPSynchronized --value
nvidia-smi
```

These checks catch cabling and addressing mistakes before they surface as NCCL timeouts. Expect a `400000Mb/s` link, MTU 9000, successful jumbo-frame pings, and synchronized time. `nvidia-smi` should show no compute processes on the GB300. On a Station with a display GPU, Xorg can appear on both GPUs with a few MiB. 

| **Tip:** Note the GPU memory use shown by`nvidia-smi` ; Step 4 compares it with the loaded model. | 
|---|

Step 2 copies the checkpoint from A to B over SSH, so A needs passwordless access to B. On A, create a key without a passphrase (see [SSH keys](https://ubuntu.com/server/docs/how-to/security/openssh-server/#ssh-keys) for details), copy its public key to B after checking B’s host fingerprint, and confirm the login: 

```
ssh-keygen -t ed25519
ssh-copy-id "$WORKER_IP"
ssh -o BatchMode=yes "$WORKER_IP" hostname
```

## Step 2. Download and copy the model

Tensor parallelism requires both ranks to run the same software against the same weights. Start by pulling the pinned image on both Stations:

```
docker pull "$IMG" &&
docker image inspect "$IMG" --format '{{.Architecture}} {{.RepoDigests}}'
```

The pull can take several minutes. Matching arm64 digests mean both ranks run identical software.

On A, download the pinned checkpoint. At about 464 GB, it takes roughly an hour at 1 Gb/s, or about six minutes at 10 Gb/s:

```
mkdir -p "$MODEL_HOST"
docker run --rm --network host -v "$MODEL_HOST:/model" \
  --entrypoint bash "$IMG" -lc \
  "exec hf download '$MODEL_REPO' --revision '$MODEL_REV' --local-dir /model"
```

When `hf download` prints `Downloaded`, copy the files to B. On A:

```
ssh "$WORKER_IP" "mkdir -p '$MODEL_HOST'" &&
rsync -aH --info=progress2 --exclude '.cache/' \
  "$MODEL_HOST/" "$WORKER_IP:$MODEL_HOST/"
```

Each disk keeps the full checkpoint; vLLM partitions the weights across the GPUs at load time, with no further quantization.

On A, compare both copies (the check reads B’s file list over SSH), then validate the checkpoint:

```
(
  set -euo pipefail
  L=$(find "$MODEL_HOST" -path '*/.cache' -prune -o \
    -type f -printf '%P %s\n' | LC_ALL=C sort | sha256sum)
  R=$(ssh "$WORKER_IP" "set -o pipefail; find '$MODEL_HOST' \
    -path '*/.cache' -prune -o -type f -printf '%P %s\\n' | \
    LC_ALL=C sort | sha256sum")
  test "$L" = "$R"
  test "$(find "$MODEL_HOST" -maxdepth 1 \
    -name 'model-*-of-00050.safetensors' | wc -l)" = 50
  python3 -c "import json; c=json.load(open('$MODEL_HOST/config.json')); \
assert c['architectures']==['GlmMoeDsaForCausalLM']; \
assert c['quantization_config']['quant_algo']=='MIXED_PRECISION'"
  echo MODEL_READY
)
```

`MODEL_READY` means both Stations hold the same file names and sizes in the expected layout. `MIXED_PRECISION` is expected: NVFP4 quantization targets MoE-block linear weights and activations, not every tensor.

## Step 3. Launch the distributed service

This step turns two machines into one service: the same launch script runs on both Stations, reads `~/glm53.env`, and starts one vLLM rank on each. Two settings matter most if you adapt it: `--max-model-len 98304`, the context ceiling in tokens, which covers input plus output, and `--gpu-memory-utilization 0.93`, which gives vLLM 93% of GPU memory. If you change other flags, change one at a time and test again.

For the latest recommended configurations, see [NVIDIA’s Local AI recipe repository](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station-2gpu/index.html).

| **Security note:**`--host 0.0.0.0` with`--network host` exposes the unauthenticated API on every Station A interface, including management. Firewall the port to intended clients, and add authentication and TLS before wider access. | 
|---|

On both Stations, create a file named `~/launch-glm53.sh`:

```
nano ~/launch-glm53.sh
```

The script enables multi-token prediction (MTP), a form of speculative decoding: the model’s MTP layer drafts four tokens per step, and vLLM verifies them with the full model before accepting them, so each step can produce several tokens. The gain depends on how often drafts are accepted, which varies with the prompt.

FP8 KV-cache storage halves cache memory compared with 16-bit storage, leaving room for more tokens. Prefix caching is enabled to avoid reprocessing prompt prefixes the server has already seen. Paste the script below; it checks your configuration, finds the RoCE GID index for the selected rail, and starts this Station’s vLLM rank in a container:

```
set -euo pipefail
source "$HOME/glm53.env"
case "$RANK" in 0|1) ;; *) echo "Invalid RANK" >&2; exit 1;; esac
test -n "$GPU_UUID"
test -f "$MODEL_HOST/config.json"
GID_INDEX=$(/usr/sbin/show_gids 2>/dev/null |
  awk -v d="$RAIL_HCA" -v ip="$NODE_IP" \
  '$1==d && $5==ip && $6=="v2" {g=$3} END {if(g!="") print g}')
test -n "$GID_INDEX"
HEADLESS=
if [ "$RANK" = 1 ]; then HEADLESS=--headless; fi
SPEC_CONFIG='{"method":"mtp","num_speculative_tokens":4}'
KERNEL_CONFIG='{"enable_flashinfer_autotune":false,"moe_backend":"flashinfer_cutlass"}'
COMPILE_CONFIG='{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[5,10,20,40,80],"pass_config":{"fuse_allreduce_rms":false}}'

docker run -d --name "$CONTAINER" \
  --gpus device="$GPU_UUID" --network host --ipc host --shm-size 32g \
  --device /dev/infiniband --cap-add IPC_LOCK \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$MODEL_HOST:/model:ro" \
  -e RANK="$RANK" -e HEADLESS="$HEADLESS" \
  -e MASTER_ADDR="$HEAD_IP" -e MODEL_REPO="$MODEL_REPO" -e PORT="$PORT" \
  -e SPEC_CONFIG="$SPEC_CONFIG" -e KERNEL_CONFIG="$KERNEL_CONFIG" \
  -e COMPILE_CONFIG="$COMPILE_CONFIG" \
  -e NCCL_CUMEM_ENABLE=1 -e NCCL_IB_HCA="$RAIL_HCA" \
  -e NCCL_IB_GID_INDEX="$GID_INDEX" \
  -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 \
  -e VLLM_ALLREDUCE_USE_FLASHINFER=0 -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 \
  -e NCCL_SOCKET_IFNAME="$RAIL_IFACE" -e GLOO_SOCKET_IFNAME="$RAIL_IFACE" \
  -e NCCL_NET_GDR_LEVEL=5 -e NCCL_DEBUG=INFO -e NCCL_DEBUG_SUBSYS=INIT,NET \
  -e NCCL_DEBUG_FILE='/tmp/nccl_%h_%p.log' -e VLLM_HOST_IP="$NODE_IP" \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  --entrypoint bash "$IMG" -lc '
    set -euo pipefail  
    # Use NCCL 2.30.7 from the image: older versions can deadlock
    # cross-node CUDA graph replays over RoCE (vLLM issue #52504).
    export VLLM_NCCL_SO_PATH=$(python3 -c "import nvidia.nccl,os;
print(os.path.join(list(nvidia.nccl.__path__)[0], \"lib\", \"libnccl.so.2\"))")
    test -f "$VLLM_NCCL_SO_PATH"
    EXTRA=()
    if [ -n "$HEADLESS" ]; then EXTRA+=("$HEADLESS"); fi
    exec vllm serve /model --served-model-name "$MODEL_REPO" \
      --host 0.0.0.0 --port "$PORT" \
      --nnodes 2 --node-rank "$RANK" --master-addr "$MASTER_ADDR" \
      --master-port 29501 --tensor-parallel-size 2 \
      --speculative-config "$SPEC_CONFIG" \
      --max-model-len 98304  --max-num-batched-tokens 8192 \
      --block-size 128 --max-num-seqs 16 \
      --trust-remote-code --disable-custom-all-reduce \
      --kv-cache-dtype fp8 --enable-prefix-caching \
      --gpu-memory-utilization 0.93 \
      --enable-prompt-tokens-details \
      --kernel-config "$KERNEL_CONFIG" --compilation-config "$COMPILE_CONFIG" \
      --distributed-timeout-seconds 1800 \
      --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \
      "${EXTRA[@]}"
  '
```

Run the script on B first, then on A:

```
bash ~/launch-glm53.sh
```

Each command prints a container ID and returns immediately; the service is not ready yet. B’s worker waits for A to join, then each Station loads its share of the weights from local storage, prepares kernels, and captures CUDA graphs. This takes several minutes.

On A, follow the log to see when the service is ready (`--since` hides output from earlier runs):

```
docker logs --follow --since "$(docker inspect \
  --format '{{.State.StartedAt}}' "$CONTAINER")" "$CONTAINER"
```

When the log shows `Application startup complete`, press Ctrl+C to stop following it; the service keeps running. 

At this point one model spans two machines. Neither GPU holds a complete copy, and neither can answer a request alone: every token needs both ranks to exchange partial results over the RoCE rail.

The remaining GPU memory holds the KV cache, which stores the context of in-flight requests. The startup log reports its [capacity](https://docs.vllm.ai/en/v0.30.0/serving/parallelism_scaling/) (GPU KV cache size). All requests in flight share it, so longer prompts or more users need more of it. If you run out, lower `--max-model-len` or send fewer requests at once.

## Step 4. Verify serving and send a request

On A, check the API (`127.0.0.1` is A itself):

```
docker exec "$CONTAINER" curl -fsS --max-time 10 \
  "http://127.0.0.1:$PORT/health" >/dev/null && echo HEALTH_OK
```

`HEALTH_OK` proves that Station A answers. Next, on both Stations, check that the ranks communicate over GPU RDMA rather than falling back to TCP sockets. The check reads the NCCL logs that the launcher writes through `NCCL_DEBUG=INFO`, and the same block shows the loaded memory:

```
docker exec "$CONTAINER" bash -lc '
  set -euo pipefail
  files=(/tmp/nccl_*.log)
  test -e "${files[0]}"
  grep -hE "NET/IB.*GDRDMA|GPU Direct RDMA Enabled" "${files[@]}" >/dev/null
  grep -hF "$NCCL_IB_HCA" "${files[@]}" >/dev/null
  if grep -hE "NET/Socket|NCCL WARN|NCCL ERROR" "${files[@]}"; then
    exit 1
  fi
  echo RDMA_CHECK_OK
'
nvidia-smi
```

`RDMA_CHECK_OK` means the check passed with no warnings. In `nvidia-smi`, A shows the `VLLM::Worker_TP0` process and B shows `VLLM::Worker_TP1`, one rank per GPU. The startup log reports about 218 GiB of weights per GPU, roughly half the checkpoint, which confirms that the model is split across the two GPUs.

| Measurement | Station A | Station B | 
|---|---|---|
| GB300 GPU used (GiB) | 0.01 → 231.35 | 0.00 → 231.34 | 

*Table 1. GPU memory before and after loading the model, with no requests running. Values include weights, KV cache, and runtime buffers*

On A, send a chat request through the OpenAI-compatible API:

```
docker exec "$CONTAINER" curl -fsS --max-time 120 \
  "http://127.0.0.1:$PORT/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{"model":"nvidia/GLM-5.3-NVFP4",
    "messages":[{"role":"user","content":"Quarterly units sold: Q1 = 120; Q2 = 180; Q3 = 150. Summarize the trend in one sentence and calculate the percentage change from Q1 to Q2."}],
    "temperature":0,"max_tokens":4096,
    "chat_template_kwargs":{"reasoning_effort":"low"}}' |
python3 -c 'import json,sys; c=json.load(sys.stdin)["choices"][0];
print(c["message"]["content"]); print("finish_reason:",c["finish_reason"])'
```

The response should look like this:

```
**Trend:** Units sold rose sharply in Q2 before partially declining in Q3, remaining above the Q1 level.

**Q1 → Q2 percentage change:** (180 - 120) / 120 × 100 = **50% increase**
finish_reason: stop
```

`finish_reason: stop` means the model finished its answer instead of hitting the token limit. The request sets `reasoning_effort` to `low` in `chat_template_kwargs`. GLM-5.3 also accepts `high` and `max`, which let it reason longer before answering, at the cost of latency. Because the launcher enables tool calling (`--enable-auto-tool-choice`), agent frameworks that use the OpenAI tools API can call the same endpoint.

## Step 5.  Concurrency and release resources

[AIPerf](https://docs.nvidia.com/aiperf/latest/welcome-to-ai-perf-documentation), the NVIDIA benchmarking tool for inference endpoints, measures throughput and latency. Installing it inside the serving container can change vLLM’s dependencies, so on A, install it in its own virtual environment:

```
sudo apt update &&
sudo apt install -y python3.12-venv python3.12-dev build-essential &&
python3.12 -m venv "$HOME/.venvs/aiperf-0.12.0" &&
"$HOME/.venvs/aiperf-0.12.0/bin/pip" install 'aiperf==0.12.0' &&
"$HOME/.venvs/aiperf-0.12.0/bin/pip" check
```

The install should end with `No broken requirements found`.

AIPerf can send synthetic prompts or replay an existing dataset. This tutorial uses the 80 Coding conversations from SPEED-Bench, a set of real prompts built to evaluate speculative decoding, so MTP is measured on realistic text. SPEED-Bench’s preparation script fetches these prompts from several Hugging Face datasets, including the gated [cais/hle](https://huggingface.co/datasets/cais/hle): if you haven’t already, open its page and select **Agree and access repository** before you continue. Then, on A, log in to Hugging Face:

```
"$HOME/.venvs/aiperf-0.12.0/bin/hf" auth login
```

Next, download and prepare the dataset. See the [official AIPerf guide](https://docs.nvidia.com/aiperf/v0.12.0/tutorials/datasets-inputs/profile-with-speed-bench-dataset#prepare-the-dataset) for details:

```
SPEED_BENCH_DIR="$HOME/datasets/speed-bench"
mkdir -p "$SPEED_BENCH_DIR"
curl -fLsS https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py | \
  "$HOME/.venvs/aiperf-0.12.0/bin/python" - --config qualitative --output_dir "$SPEED_BENCH_DIR"
```

On A, check the dataset and service. Continue when the check prints `BENCHMARK_READY`:

```
(
  set -euo pipefail
  source ~/glm53.env
  curl -fsS "http://$HEAD_IP:$PORT/health" >/dev/null
  "$HOME/.venvs/aiperf-0.12.0/bin/python" - <<'PY_CHECK'
import json
from pathlib import Path
root = Path.home()
path = root / "datasets/speed-bench/qualitative.jsonl"
assert path.exists(), "Dataset missing: rerun the SPEED-Bench preparation"
rows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
coding = [row for row in rows if row["category"] == "coding"]
assert len(coding) == 80, "Expected 80 Coding conversations"
assert sum(len(row["messages"]) for row in coding) == 89
assert all("FULL BENCHMARK DATA SHOULD BE FETCHED" not in message["content"]
           for row in coding for message in row["messages"])
print("BENCHMARK_READY: dataset has 80 Coding conversations, 89 turns in total.")

PY_CHECK
)
```

The full sweep covers concurrency 1, 8, 16, and 32. At each level, AIPerf sends 3×C warmup requests, then measures 10×C requests, where C is the concurrency. Some conversations are sent more than once at higher concurrency.

Benchmark requests use the model’s default reasoning effort (max) and an 8,192-token output cap; responses can finish before reaching it. Full sweeps can take several hours.

 A two-hour request timeout leaves room for long responses, `--server-metrics` collects server counters used to measure MTP acceptance, and `--export-level raw` keeps each response for inspection. 

For a quick test, run the block below on A with both containers running and no other API traffic. It tests C1 and C8 with prefix caching enabled, as in the reference measurements in Table 2, below.

```
(
  set -euo pipefail
  source ~/glm53.env
  CODING_DATASET="$HOME/datasets/speed-bench/qualitative.jsonl"
  test -f "$CODING_DATASET"
 export CONCURRENCIES=${CONCURRENCIES:-"1 8"} # Quick run: C1 and C8.
  # Optional full benchmark: follow the reproduction instructions below.
  # Step 3: --max-num-seqs 16 -> export CONCURRENCIES="1 8 16"
  # Step 3: --max-num-seqs 32 -> export CONCURRENCIES=32

  RUN_DIR="$HOME/aiperf-results/blog-coding-$(date +%Y%m%d-%H%M%S)"
  for C in $CONCURRENCIES;  do
    ART="$RUN_DIR/c$C"
    env -u HF_HUB_OFFLINE -u TRANSFORMERS_OFFLINE \
      "$HOME/.venvs/aiperf-0.12.0/bin/aiperf" profile \
      -m "$MODEL_REPO" --url "http://$HEAD_IP:$PORT" \
      --endpoint-type chat --streaming --tokenizer "$MODEL_HOST" \
      --custom-dataset-type speed_bench_coding --input-file "$CODING_DATASET" \
      --request-count "$((10 * C))" --warmup-request-count "$((3 * C))" \
      --osl 8192 \
      --use-server-token-count --concurrency "$C" --random-seed 42 \
      --request-timeout-seconds 7200 \
      --server-metrics "http://$HEAD_IP:$PORT/metrics" \
      --export-level raw --artifact-dir "$ART"
    printf "Results: %s\n" "$ART"
  done
  printf "Results directory: %s\n" "$RUN_DIR"
)
```

AIPerf saves metrics, raw responses, and logs in the results directory. Warmup requests are excluded from the reported metrics.

**Optional: Reproduce Table 2**. In `~/launch-glm53.sh` from Step 3 on both Stations, keep `--enable-prefix-caching` and set `--max-num-seqs` to 16 or 32 for the corresponding group shown in the comments above. Before each group, stop and remove the current containers on A, then B (`docker stop --timeout 60 "$CONTAINER"`, followed by `docker rm "$CONTAINER"`). Relaunch the updated script on B, then A with bash `~/launch-glm53.sh` and wait for `Application startup complete`. On A, run the matching `export CONCURRENCIES` command from the comments, then rerun the AIPerf block.

To test without prefix caching, replace `--enable-prefix-caching` with `--no-enable-prefix-caching` in the Step 3 script on both Stations, then recreate the containers as above. Performance may differ from Table 2.

Figure 2 compares overall output throughput across measured requests with mean per-user throughput. Token counts include reasoning.

Table 2, below, shows how throughput and latency change as more requests share the service.

| **Configured** **concurrency** | **Overall throughput** **(tok/s)** | **Mean request latency** **(s)** | **Mean TTFT****(ms)** | **Mean inter-token latency****(ms/token)** | 
| 1 | 107.4 | 49.84 | 479.75 | 8.78 | 
| 8 | 386.0 | 114.17 | 433.79 | 19.01 | 
| 16 | 607.8 | 136.46 | 482.90 | 23.77 | 
| 32 | 385.0 | 453.54 | 899.36 | 77.90 | 

*Table 2. SPEED-Bench Coding results with prefix caching enabled, four MTP draft tokens per step, and an FP8 KV cache. Latency metrics are means over measured requests; warmup is excluded. TTFT measures time to the first token, which may be reasoning*

Overall throughput peaks at 607.8 tok/s at C16. It falls to 385.0 tok/s at C32, where mean request latency is 453.5 s. Higher concurrency also reduces mean per-user throughput, from 115.3 tok/s at C1 to 13.0 tok/s at C32.

After the requests finish, stop the vLLM container on A, then on B:

```
docker stop --timeout 60 "$CONTAINER"
docker inspect --format \
  'Status={{.State.Status}} Running={{.State.Running}}' "$CONTAINER"
nvidia-smi
```

The status should read `Status=exited Running=false`, with no vLLM processes in `nvidia-smi`; the checkpoint, containers, and results stay on disk. To resume, run `docker start "$CONTAINER"` on B, then A, and repeat the Step 4 checks.

## Troubleshoot common blockers

- Missing variables or SSH failure: `source ~/glm53.env` after reconnecting; verify A-to-B key authentication and B’s fingerprint.
- SPEED-Bench preparation fails with a gated-dataset error: sign in to Hugging Face with the same account, open the [cais/hle](https://huggingface.co/datasets/cais/hle) page, select**Agree and access repository** , then rerun the preparation.
- Startup or RDMA check failure: inspect `docker logs "$CONTAINER"` on both nodes; verify the checkpoint and the GID, and rerun the playbook’s validation (script 07).
- [CUDA out of memory](https://docs.vllm.ai/en/v0.30.0/usage/troubleshooting/) : lower`--max-model-len` or concurrency, then retest.

## Put the endpoint to work

You now have a 753-billion-parameter model running behind one local API on two NVIDIA DGX Stations, with every weight in GPU memory. That opens up work that’s hard to do on a hosted endpoint: running inference on sensitive or proprietary data, prototyping agents locally, and iterating on prompts without per-token costs.

Connect your own application using the [vLLM serving documentation](https://docs.vllm.ai/en/v0.30.0/serving/online_serving/), then rerun AIPerf with your prompts to find the concurrency that fits your workload. A frontier-scale model is now ready to build on, right from your desk.