{"slug": "serving-gemma4-with-rust-on-vllm", "title": "Serving Gemma4 with Rust on vLLM 🦀", "summary": "A developer detailed how to build and run vLLM's Rust frontend on an AWS EC2 G5g instance with Graviton2 and an NVIDIA T4G GPU. The tutorial highlights that vLLM now requires a Rust toolchain for source builds, as setuptools_rust is imported at module scope, and provides steps to install the necessary components. The developer also notes that building with --no-build-isolation requires manually supplying all build dependencies, including setuptools_rust and protoc.", "body_md": "This tutorial walks through installing and setting up the **Rust toolchain for vLLM** on an\n\nAWS EC2 **G5g** instance — Graviton2 (aarch64) with an NVIDIA T4G GPU — and getting vLLM's\n\nRust frontend (`vllm-rs`\n\n) built, running, and *verified*.\n\nThis paper is a follow-on to the original G5g Gemma 4 build.\n\nEverything below was run on the box. 🦀\n\nYou betcha. Since [PR #40848](https://github.com/vllm-project/vllm/pull/40848) (merged\n\n2026-05-21), vLLM vendors a **14-crate Rust workspace**:\n\n```\nbench  chat  cmd  engine-core-client  llm  managed-engine  metrics\nmock-engine  parser  parser/python  server  text  tokenizer  tracing\n```\n\nEdition 2024, resolver 3. Straight from the vendored `rust/Cargo.toml`\n\n:\n\n| Crate | Version | Job |\n|---|---|---|\n`axum` |\n0.8.8 | the HTTP server |\n`tokio` |\n1.47.1 | async runtime |\n`zeromq` |\n0.6.0 | talks to the Python engine |\n`rmp-serde` / `rmpv`\n|\n1.3.1 | msgpack on the wire |\n`minijinja` |\n2.22 | chat templates |\n`tonic` / `prost`\n|\n0.14.6 / 0.14.3 | gRPC — remember this one\n|\n\nIt's a drop-in replacement for the Python FastAPI server. Two artifacts get built:\n\n`vllm-rs`\n\n`vllm._rust_tool_parser`\n\nThat's the headline, and it's reason enough on its own: **you cannot build vLLM from source at\nv0.27.2rc0 without Rust in the picture.**\n\n`setup.py`\n\nimports it at module scope, line 21,\n\n``` python\nfrom setuptools_rust.build import build_rust\n```\n\nNo `try`\n\n, no feature flag, no opt-out. Metadata generation doesn't happen without it.\n\nAnd this isn't a quirk of one release. vLLM's Rust surface is **14 crates** covering the HTTP\n\nfrontend, the tool parser, the tokenizer and the benchmark client, and it has been growing\n\nsince it landed. If you build inference infrastructure from source, a Rust toolchain is\n\nbecoming table stakes — so it's worth knowing how to drive it properly rather than working\n\naround it.\n\nThree things do get conflated, though, and they have different scopes:\n\n| Component | Needed to build vLLM? | Needed to serve? |\n|---|---|---|\n`setuptools_rust` (Python pkg) |\nyes, always |\nno |\n`cargo` / `rustc` toolchain |\nfor working Rust artifacts | no |\n`protoc` |\nfor `vllm-rs` specifically |\nno |\n\n`pip install vllm`\n\nneed this?\nBecause normally pip installs it for you. `pyproject.toml`\n\ndeclares it:\n\n```\n[build-system]\nrequires = [\n    \"cmake>=3.26.1\", \"ninja\", \"packaging>=24.2\",\n    \"setuptools>=77.0.3,<81.0.0\", \"setuptools-scm>=8.0\",\n    \"setuptools-rust>=1.9.0\",          # <- pip grabs this automatically\n    \"torch == 2.13.0\",                 # <- ...and this. Which is the problem.\n    \"wheel\", \"jinja2\",\n]\n```\n\nUnder normal **build isolation**, pip creates a clean env, installs that list, and builds.\n\nYou never see `setuptools_rust`\n\nbecause you never had to think about it.\n\nBut look at the `torch`\n\npin. Building in isolation means pip installs **torch 2.13.0 from\nPyPI** — and the PyPI aarch64 wheels are built for sm_80 and up.\n\n`sm_75`\n\n.So on this box you must build against the DLAMI's own torch, and that means:\n\n```\npython use_existing_torch.py\npip install -e . --no-build-isolation\n```\n\n** --no-build-isolation turns off the automatic install of everything in that requires** From that moment on, every build dependency is yours to supply by hand — including\n\n`setuptools_rust`\n\n, which is why it turns up as a bare `ModuleNotFoundError`\n\nminutes into aSo the toolchain was always required; isolation was just hiding it. Building this way means\n\nyou own the dependency list, which is the rest of this walk-through. ⚡\n\nThe AWS Deep Learning ARM64 AMI ships a **runtime**, not a build environment. On a fresh box:\n\n| Thing | Present? |\n|---|---|\nPyTorch 2.12 with `sm_75`\n|\n✅ |\n| NVIDIA driver | ✅ |\n`nvcc` / CUDA toolkit |\n❌ |\n| Rust toolchain | ❌ |\n`setuptools_rust` |\n❌ |\n`protoc` |\n❌ |\n\nFour of those six are on you. Let's install them.\n\nStandard rustup, nothing aarch64-specific about it:\n\n```\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y\n. \"$HOME/.cargo/env\"\nstable-aarch64-unknown-linux-gnu installed - rustc 1.97.1 (8bab26f4f 2026-07-14)\n\nRust is installed now. Great!\n```\n\nNote the triple: `stable-aarch64-unknown-linux-gnu`\n\n. Rust's aarch64 support is a **complete\nnon-event**, which is a lovely change of pace on this hardware. ⚡\n\n```\npython3 -m pip install setuptools_rust\n```\n\nPer the section above: `--no-build-isolation`\n\nmeans pip won't do this for you. Do it **early**\n\n— the failure lands during metadata generation, minutes into a build, as a bare\n\n`ModuleNotFoundError: No module named 'setuptools_rust'`\n\nnowhere near anything that looks\n\nlike Rust.\n\n⚠️ Install it into the **same interpreter you'll build with**. On the DLAMI that's\n\n`/opt/pytorch/bin/python3`\n\n, not the system `python3`\n\n— they're different, and the one that\n\nmatters is whichever owns the torch you're building against.\n\nThis is the one nobody documents:\n\n```\napt-get install -y protobuf-compiler\nprotoc --version\nlibprotoc 3.21.12\n```\n\n**Why:** `vllm-rs`\n\ndepends on the `vllm-server`\n\ncrate, `vllm-server`\n\nbuilds gRPC stubs with\n\n`tonic`\n\n/`prost`\n\n, and `prost-build`\n\nshells out to `protoc`\n\n. Skip it and the frontend binary\n\ndoes not get built — see the summary at the end for how loudly that *doesn't* fail.\n\nThe tool parser has no protobuf dependency, which is why it builds either way.\n\nNot Rust, but the same class of problem, and you need it for vLLM's kernels:\n\n```\n# NVIDIA's **sbsa** repo — not the x86 one, easy reflex to get wrong on Arm\napt-get install -y cuda-toolkit-13-2\ncd /opt/vllm-src\npython tools/build_rust.py --release\n```\n\n⚠️ **Do not omit --release.** setuptools-rust builds inplace targets in debug by default,\n\n`pip install -e .`\n\nis an inplace build. The difference is not subtle:| Artifact | Debug | Release |\n|---|---|---|\n`_rust_tool_parser.abi3.so` |\n100,913,216 B | 1,009,080 B |\n\n**100x.** The debug artifact is four times the size of *every CUDA kernel in vLLM combined*.\n\nTiming on a `g5g.xlarge`\n\n(4 vCPU), cold:\n\n```\nreal    9m1.746s\nuser    25m9.199s\nsys     1m35.023s\n```\n\n**501 crates. Zero warnings. Exit 0.** 🟢\n\nRust's aarch64 support does not put up a fight here — which is a pleasant contrast with the\n\nCUDA side of this box, where SM 7.5 on Graviton needs a custom arch list and a patched\n\nkernel.\n\n```\nls -la vllm/vllm-rs vllm/_rust_tool_parser.abi3.so\n-rwxr-xr-x 1 root root 50039024 vllm/vllm-rs\n-rwxr-xr-x 1 root root  1009080 vllm/_rust_tool_parser.abi3.so\nfile vllm/vllm-rs\nELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV),\ndynamically linked, interpreter /lib/ld-linux-aarch64.so.1, not stripped\nvllm/vllm-rs --help\nRust frontend and managed-engine CLI for vLLM.\n\nCommands:\n  frontend  Run the Rust OpenAI frontend as a Python-supervised worker\n  serve     Launch a managed Python headless engine, then run the Rust OpenAI frontend\n  bench     Run vLLM benchmarks\n  render    Run engine-free request rendering and preprocessing\n```\n\nIf `vllm/vllm-rs`\n\nisn't there, go back to **Step 3**.\n\n```\nVLLM_USE_RUST_FRONTEND=1 vllm serve google/gemma-4-E2B-it \\\n  --dtype float16 \\\n  --kv-cache-dtype auto \\\n  --max-model-len 16384 \\\n  --gpu-memory-utilization 0.90 \\\n  --max-num-seqs 8 \\\n  --tensor-parallel-size 1 \\\n  --host 0.0.0.0 --port 8000\n```\n\n**It must be vllm serve.** If you launch the module directly —\n\n```\n# ❌ VLLM_USE_RUST_FRONTEND is IGNORED here\npython -m vllm.entrypoints.openai.api_server --model … --host 0.0.0.0 --port 8000\n```\n\n— the variable does nothing. No warning, no `Unknown vLLM environment variable`\n\nline. The\n\nserver comes up healthy and serves happily on the Python frontend, and a benchmark run\n\nagainst it looks entirely normal.\n\nThe flag is read in exactly two places:\n\n```\nvllm/entrypoints/cli/serve.py:62        envs.VLLM_RUST_FRONTEND_PATH if envs.VLLM_USE_RUST_FRONTEND else None\nvllm/entrypoints/openai/dp_supervisor.py:261   if envs.VLLM_USE_RUST_FRONTEND and envs.VLLM_RUST_FRONTEND_PATH:\n```\n\n`api_server.py`\n\nnever mentions it.\n\nThree checks. Do all three the first time.\n\n**1. The server: header:**\n\n```\ncurl -si localhost:8000/health | grep -i '^server:'\n```\n\n| Frontend | Response |\n|---|---|\n| 🐍 Python | `server: uvicorn` |\n| 🦀 Rust | (no `server:` header at all) |\n\n**2. The process:**\n\n```\npgrep -af vllm-rs\n26588 /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17\n  --input-address  ipc:///tmp/f60f3962-d45b-4bcd-9026-c0dc32736028\n  --output-address ipc:///tmp/5f75411d-2787-43bb-b4fc-14bf504a1cce\n  --engine-start-index 0 --engine-count 1 --data-parallel-size 1\n```\n\n**3. The log prefix** — `(RustFrontend pid=…)`\n\ninstead of `(APIServer pid=…)`\n\n:\n\n```\nINFO [utils.py:392] Launching Rust frontend: /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17 …\n```\n\nIn **two** places, and they're quite different. One is a separate process; the other is a\n\nshared object loaded *inside* the Python process. Here's the whole VM:\n\n```\n┌─ EC2 g5g.4xlarge ── Graviton2, aarch64 ─────────────────────────────────────┐\n│                                                                             │\n│  Deep Learning ARM64 AMI · Ubuntu 24.04 · NVIDIA driver 595.71.05           │\n│  you add > cuda-toolkit-13-2 (sbsa) · rustup 1.97.1 · protobuf-compiler     │\n│                                                                             │\n│      HTTP :8000                                                             │\n│          |                                                                  │\n│          v                                                                  │\n│  ┌───────────────────────────────┐                                          │\n│  │ [RUST] vllm-rs                │  50 MB aarch64 ELF, its OWN process      │\n│  │        axum 0.8.8 · tokio     │  built from the vendored rust/ workspace │\n│  │        minijinja · fastokens  │  <- Step 5                               │\n│  └────────┬─────────────▲────────┘                                          │\n│           |             |                                                   │\n│  ipc://   | ROUTER      | PULL     msgpack (rmp-serde / rmpv)               │\n│           v             |                                                   │\n│  ┌────────┴─────────────┴────────┐                                          │\n│  │ [PY]   vLLM supervisor        │  `vllm serve` opens the socket, then     │\n│  │                               │  hands listen-fd 17 down to vllm-rs      │\n│  └────────┬──────────────────────┘                                          │\n│           | spawns                                                          │\n│           v                                                                 │\n│  ┌───────────────────────────────┐                                          │\n│  │ [PY]   EngineCore             │  torch 2.12.0+cu132, arch list has sm_75 │\n│  │  ┌─────────────────────────┐  │                                          │\n│  │  │ [RUST] _rust_tool_parser│  │  PyO3 .so LOADED INTO the Python         │\n│  │  │        1.0 MB release   │  │  process — not a process of its own      │\n│  │  └─────────────────────────┘  │                                          │\n│  └────────┬──────────────────────┘                                          │\n│           | CUDA                                                            │\n│           v                                                                 │\n│  ┌───────────────────────────────┐                                          │\n│  │ NVIDIA T4G · SM 7.5           │  15,360 MiB GDDR6 · 277 GB/s measured    │\n│  │ TRITON_ATTN kernels           │  weights 9.94 GiB · KV 2.95 GiB          │\n│  └───────────────────────────────┘                                          │\n└─────────────────────────────────────────────────────────────────────────────┘\n```\n\nTwo things worth pulling out of that picture:\n\n`vllm-rs`\n\nis not a sidecar you point at a port.`_rust_tool_parser`\n\nis Rust living inside Python.`protoc`\n\nneeded), which is why a broken install still leaves Rust on the box — just not\nthe Rust you wanted.And note where the GPU sits relative to all of this: at the bottom, behind everything. That's\n\nthe reason the benchmark below comes out the way it does.\n\n```\nVLLM_USE_RUST_BENCH=1 vllm bench serve …\n```\n\nSame binary, `bench`\n\nsubcommand. Requires `VLLM_RUST_FRONTEND_PATH`\n\nto resolve, so it needs\n\nthe same Step 3 → Step 5 you just did.\n\nThe server came up healthy. These went by in the startup log anyway.\n\n**Gemma 4 defeats the fast tokenizer:**\n\n```\nINFO    [hf.rs:200] loading tokenizer with fastokens\nWARNING [hf.rs:221] failed to load tokenizer with fastokens; falling back to\n        HuggingFace tokenizers\n        error=tokenizer error: normalizer error: unsupported normalizer type: Replace\n```\n\n`fastokens`\n\n0.2.1 doesn't implement the `Replace`\n\nnormalizer that Gemma 4's `tokenizer.json`\n\nuses, so it falls back to the same HuggingFace `tokenizers`\n\nthe Python path uses. Note the\n\nfallback is graceful and correct — you just don't get the fast path on *this* model yet. It's\n\na coverage gap in a young crate, and one normalizer away from closing.\n\n**Multimodal isn't wired up for this model:**\n\n```\nWARNING [multimodal.rs:446] multimodal model spec is not registered; disabling\n        image/video support   model_id=\"google/gemma-4-E2B-it\" model_type=\"gemma4\"\n```\n\nGemma 4 E2B is a **vision** model, and `gemma4`\n\nisn't in the Rust multimodal spec table yet.\n\nText requests behave identically and the endpoint is healthy, so nothing in a normal check\n\nreveals it. Also a registration gap rather than a design problem — but check it for your model\n\nbefore you switch, because a healthy endpoint won't tell you.\n\nOn a T4G, no. Output token throughput, same engine config, client on the box against\n\nlocalhost:\n\n| Concurrency | 🐍 Python | 🦀 Rust |\n|---|---|---|\n| 1 | 28.65 | 29.30 |\n| 4 | 97.48 | 97.26 |\n| 8 | 168.33 | 169.39 |\n| 16 | 169.96 | 170.19 |\n| 32 | 170.99 | 170.34 |\n\nMedian TTFT tracks just as tightly — 14305 ms against 14311 ms at concurrency 32.\n\nThat's the expected result, and worth saying plainly: decode on this card is\n\n**bandwidth-bound** at a measured 277 GB/s, and the engine saturates at `--max-num-seqs 8`\n\n.\n\nA frontend rewrite targets CPU-side per-request overhead. Here that overhead hides behind the\n\nGPU, so swapping it can't move a bottleneck-limited number. **If you want the Rust frontend\nto buy you tokens per second on a small GPU, it won't.**\n\nOne signal does appear, in median inter-token latency at high concurrency:\n\n| Concurrency | 🐍 Python | 🦀 Rust | Δ |\n|---|---|---|---|\n| 16 | 38.55 | 36.18 |\n−6.4% |\n| 32 | 38.41 | 36.23 |\n−5.9% |\n\nMean TPOT barely moves, so this is the middle of the distribution tightening rather than\n\neverything speeding up — the shape you'd expect from a frontend scheduling streaming work\n\nmore evenly once many streams are in flight. Worth knowing if you serve at concurrency; not\n\nworth switching for on its own. 📊\n\n`pip install -e .`\n\nalready ran\nA from-source vLLM install done **without** the steps above succeeds, exits 0, and leaves you\n\nwith a 96 MB debug tool parser and no frontend binary. Four defaults stack up to make that\n\nsilent:\n\n| Symptom | Cause | Fix |\n|---|---|---|\nNo `vllm/vllm-rs` after a clean build |\n`protoc` absent ⇒ `vllm-server` fails with code 101 |\nStep 3 |\n`pip install` exits 0 anyway |\n`optional=not should_require_rust_frontend()` — setuptools-rust swallows it |\n`VLLM_REQUIRE_RUST_FRONTEND=1` |\n`_rust_tool_parser.abi3.so` is ~96 MB |\neditable ⇒ inplace ⇒ debug profile | `--release` |\n`FileNotFoundError: … vllm-rs was not found` |\nthe above, discovered at import time | Steps 3 + 5 |\nHealthy server, but `server: uvicorn`\n|\nflag set on the `api_server` module, which never reads it |\n`vllm serve` |\n\n`VLLM_REQUIRE_RUST_FRONTEND=1`\n\nturns the second row into a hard build failure, which is what\n\nyou want on any machine you plan to serve from.\n\n```\n# toolchain — you supply these by hand because the sm_75 requirement\n# forces --no-build-isolation, which disables pip's automatic build deps\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y\n. \"$HOME/.cargo/env\"\n/opt/pytorch/bin/python3 -m pip install setuptools_rust   # the BUILD interpreter\napt-get install -y protobuf-compiler cuda-toolkit-13-2\n\n# build against the DLAMI's torch, not a PyPI one (PyPI aarch64 has no sm_75)\ncd /path/to/vllm\npython use_existing_torch.py\nTORCH_CUDA_ARCH_LIST=7.5 VLLM_REQUIRE_RUST_FRONTEND=1 \\\n  pip install -e . --no-build-isolation\n\n# Rust artifacts, release profile (editable installs default to debug: 100x bigger)\nVLLM_REQUIRE_RUST_FRONTEND=1 python tools/build_rust.py --release\n\n# confirm\nls -la vllm/vllm-rs && vllm/vllm-rs --help\n\n# run — `vllm serve`, NOT the api_server module\nVLLM_USE_RUST_FRONTEND=1 vllm serve <model> --host 0.0.0.0 --port 8000\n\n# verify it's really Rust\ncurl -si localhost:8000/health | grep -i '^server:'   # Rust sends none\npgrep -af vllm-rs\n```\n\n*Run on EC2 g5g.xlarge and g5g.4xlarge, us-east-1a, NVIDIA T4G (SM 7.5). vLLM*", "url": "https://wpnews.pro/news/serving-gemma4-with-rust-on-vllm", "canonical_source": "https://dev.to/gde/serving-gemma4-with-rust-for-vllm-372l", "published_at": "2026-08-14 20:43:34+00:00", "updated_at": "2026-08-14 21:05:09.174251+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "machine-learning"], "entities": ["vLLM", "AWS EC2 G5g", "Graviton2", "NVIDIA T4G", "Rust", "setuptools_rust", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/serving-gemma4-with-rust-on-vllm", "markdown": "https://wpnews.pro/news/serving-gemma4-with-rust-on-vllm.md", "text": "https://wpnews.pro/news/serving-gemma4-with-rust-on-vllm.txt", "jsonld": "https://wpnews.pro/news/serving-gemma4-with-rust-on-vllm.jsonld"}}