Serving Gemma4 with Rust on vLLM 🦀

A developer detailed how to build and run vLLM's Rust frontend on an AWS EC2 G5g instance with Graviton2 and an NVIDIA T4G GPU. The tutorial highlights that vLLM now requires a Rust toolchain for source builds, as setuptools_rust is imported at module scope, and provides steps to install the necessary components. The developer also notes that building with --no-build-isolation requires manually supplying all build dependencies, including setuptools_rust and protoc.

This tutorial walks through installing and setting up the Rust toolchain for vLLM on an AWS EC2 G5g instance — Graviton2 aarch64 with an NVIDIA T4G GPU — and getting vLLM's Rust frontend vllm-rs built, running, and verified . This paper is a follow-on to the original G5g Gemma 4 build. Everything below was run on the box. 🦀 You betcha. Since PR 40848 https://github.com/vllm-project/vllm/pull/40848 merged 2026-05-21 , vLLM vendors a 14-crate Rust workspace : bench chat cmd engine-core-client llm managed-engine metrics mock-engine parser parser/python server text tokenizer tracing Edition 2024, resolver 3. Straight from the vendored rust/Cargo.toml : | Crate | Version | Job | |---|---|---| axum | 0.8.8 | the HTTP server | tokio | 1.47.1 | async runtime | zeromq | 0.6.0 | talks to the Python engine | rmp-serde / rmpv | 1.3.1 | msgpack on the wire | minijinja | 2.22 | chat templates | tonic / prost | 0.14.6 / 0.14.3 | gRPC — remember this one | It's a drop-in replacement for the Python FastAPI server. Two artifacts get built: vllm-rs vllm. rust tool parser That's the headline, and it's reason enough on its own: you cannot build vLLM from source at v0.27.2rc0 without Rust in the picture. setup.py imports it at module scope, line 21, python from setuptools rust.build import build rust No try , no feature flag, no opt-out. Metadata generation doesn't happen without it. And this isn't a quirk of one release. vLLM's Rust surface is 14 crates covering the HTTP frontend, the tool parser, the tokenizer and the benchmark client, and it has been growing since it landed. If you build inference infrastructure from source, a Rust toolchain is becoming table stakes — so it's worth knowing how to drive it properly rather than working around it. Three things do get conflated, though, and they have different scopes: | Component | Needed to build vLLM? | Needed to serve? | |---|---|---| setuptools rust Python pkg | yes, always | no | cargo / rustc toolchain | for working Rust artifacts | no | protoc | for vllm-rs specifically | no | pip install vllm need this? Because normally pip installs it for you. pyproject.toml declares it: build-system requires = "cmake =3.26.1", "ninja", "packaging =24.2", "setuptools =77.0.3,<81.0.0", "setuptools-scm =8.0", "setuptools-rust =1.9.0", <- pip grabs this automatically "torch == 2.13.0", <- ...and this. Which is the problem. "wheel", "jinja2", Under normal build isolation , pip creates a clean env, installs that list, and builds. You never see setuptools rust because you never had to think about it. But look at the torch pin. Building in isolation means pip installs torch 2.13.0 from PyPI — and the PyPI aarch64 wheels are built for sm 80 and up. sm 75 .So on this box you must build against the DLAMI's own torch, and that means: python use existing torch.py pip install -e . --no-build-isolation --no-build-isolation turns off the automatic install of everything in that requires From that moment on, every build dependency is yours to supply by hand — including setuptools rust , which is why it turns up as a bare ModuleNotFoundError minutes into aSo the toolchain was always required; isolation was just hiding it. Building this way means you own the dependency list, which is the rest of this walk-through. ⚡ The AWS Deep Learning ARM64 AMI ships a runtime , not a build environment. On a fresh box: | Thing | Present? | |---|---| PyTorch 2.12 with sm 75 | ✅ | | NVIDIA driver | ✅ | nvcc / CUDA toolkit | ❌ | | Rust toolchain | ❌ | setuptools rust | ❌ | protoc | ❌ | Four of those six are on you. Let's install them. Standard rustup, nothing aarch64-specific about it: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y . "$HOME/.cargo/env" stable-aarch64-unknown-linux-gnu installed - rustc 1.97.1 8bab26f4f 2026-07-14 Rust is installed now. Great Note the triple: stable-aarch64-unknown-linux-gnu . Rust's aarch64 support is a complete non-event , which is a lovely change of pace on this hardware. ⚡ python3 -m pip install setuptools rust Per the section above: --no-build-isolation means pip won't do this for you. Do it early — the failure lands during metadata generation, minutes into a build, as a bare ModuleNotFoundError: No module named 'setuptools rust' nowhere near anything that looks like Rust. ⚠️ Install it into the same interpreter you'll build with . On the DLAMI that's /opt/pytorch/bin/python3 , not the system python3 — they're different, and the one that matters is whichever owns the torch you're building against. This is the one nobody documents: apt-get install -y protobuf-compiler protoc --version libprotoc 3.21.12 Why: vllm-rs depends on the vllm-server crate, vllm-server builds gRPC stubs with tonic / prost , and prost-build shells out to protoc . Skip it and the frontend binary does not get built — see the summary at the end for how loudly that doesn't fail. The tool parser has no protobuf dependency, which is why it builds either way. Not Rust, but the same class of problem, and you need it for vLLM's kernels: NVIDIA's sbsa repo — not the x86 one, easy reflex to get wrong on Arm apt-get install -y cuda-toolkit-13-2 cd /opt/vllm-src python tools/build rust.py --release ⚠️ Do not omit --release. setuptools-rust builds inplace targets in debug by default, pip install -e . is an inplace build. The difference is not subtle:| Artifact | Debug | Release | |---|---|---| rust tool parser.abi3.so | 100,913,216 B | 1,009,080 B | 100x. The debug artifact is four times the size of every CUDA kernel in vLLM combined . Timing on a g5g.xlarge 4 vCPU , cold: real 9m1.746s user 25m9.199s sys 1m35.023s 501 crates. Zero warnings. Exit 0. 🟢 Rust's aarch64 support does not put up a fight here — which is a pleasant contrast with the CUDA side of this box, where SM 7.5 on Graviton needs a custom arch list and a patched kernel. ls -la vllm/vllm-rs vllm/ rust tool parser.abi3.so -rwxr-xr-x 1 root root 50039024 vllm/vllm-rs -rwxr-xr-x 1 root root 1009080 vllm/ rust tool parser.abi3.so file vllm/vllm-rs ELF 64-bit LSB pie executable, ARM aarch64, version 1 SYSV , dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, not stripped vllm/vllm-rs --help Rust frontend and managed-engine CLI for vLLM. Commands: frontend Run the Rust OpenAI frontend as a Python-supervised worker serve Launch a managed Python headless engine, then run the Rust OpenAI frontend bench Run vLLM benchmarks render Run engine-free request rendering and preprocessing If vllm/vllm-rs isn't there, go back to Step 3 . VLLM USE RUST FRONTEND=1 vllm serve google/gemma-4-E2B-it \ --dtype float16 \ --kv-cache-dtype auto \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --max-num-seqs 8 \ --tensor-parallel-size 1 \ --host 0.0.0.0 --port 8000 It must be vllm serve. If you launch the module directly — ❌ VLLM USE RUST FRONTEND is IGNORED here python -m vllm.entrypoints.openai.api server --model … --host 0.0.0.0 --port 8000 — the variable does nothing. No warning, no Unknown vLLM environment variable line. The server comes up healthy and serves happily on the Python frontend, and a benchmark run against it looks entirely normal. The flag is read in exactly two places: vllm/entrypoints/cli/serve.py:62 envs.VLLM RUST FRONTEND PATH if envs.VLLM USE RUST FRONTEND else None vllm/entrypoints/openai/dp supervisor.py:261 if envs.VLLM USE RUST FRONTEND and envs.VLLM RUST FRONTEND PATH: api server.py never mentions it. Three checks. Do all three the first time. 1. The server: header: curl -si localhost:8000/health | grep -i '^server:' | Frontend | Response | |---|---| | 🐍 Python | server: uvicorn | | 🦀 Rust | no server: header at all | 2. The process: pgrep -af vllm-rs 26588 /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17 --input-address ipc:///tmp/f60f3962-d45b-4bcd-9026-c0dc32736028 --output-address ipc:///tmp/5f75411d-2787-43bb-b4fc-14bf504a1cce --engine-start-index 0 --engine-count 1 --data-parallel-size 1 3. The log prefix — RustFrontend pid=… instead of APIServer pid=… : INFO utils.py:392 Launching Rust frontend: /opt/vllm-src/vllm/vllm-rs frontend --listen-fd 17 … In two places, and they're quite different. One is a separate process; the other is a shared object loaded inside the Python process. Here's the whole VM: ┌─ EC2 g5g.4xlarge ── Graviton2, aarch64 ─────────────────────────────────────┐ │ │ │ Deep Learning ARM64 AMI · Ubuntu 24.04 · NVIDIA driver 595.71.05 │ │ you add cuda-toolkit-13-2 sbsa · rustup 1.97.1 · protobuf-compiler │ │ │ │ HTTP :8000 │ │ | │ │ v │ │ ┌───────────────────────────────┐ │ │ │ RUST vllm-rs │ 50 MB aarch64 ELF, its OWN process │ │ │ axum 0.8.8 · tokio │ built from the vendored rust/ workspace │ │ │ minijinja · fastokens │ <- Step 5 │ │ └────────┬─────────────▲────────┘ │ │ | | │ │ ipc:// | ROUTER | PULL msgpack rmp-serde / rmpv │ │ v | │ │ ┌────────┴─────────────┴────────┐ │ │ │ PY vLLM supervisor │ vllm serve opens the socket, then │ │ │ │ hands listen-fd 17 down to vllm-rs │ │ └────────┬──────────────────────┘ │ │ | spawns │ │ v │ │ ┌───────────────────────────────┐ │ │ │ PY EngineCore │ torch 2.12.0+cu132, arch list has sm 75 │ │ │ ┌─────────────────────────┐ │ │ │ │ │ RUST rust tool parser│ │ PyO3 .so LOADED INTO the Python │ │ │ │ 1.0 MB release │ │ process — not a process of its own │ │ │ └─────────────────────────┘ │ │ │ └────────┬──────────────────────┘ │ │ | CUDA │ │ v │ │ ┌───────────────────────────────┐ │ │ │ NVIDIA T4G · SM 7.5 │ 15,360 MiB GDDR6 · 277 GB/s measured │ │ │ TRITON ATTN kernels │ weights 9.94 GiB · KV 2.95 GiB │ │ └───────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ Two things worth pulling out of that picture: vllm-rs is not a sidecar you point at a port. rust tool parser is Rust living inside Python. protoc needed , which is why a broken install still leaves Rust on the box — just not the Rust you wanted.And note where the GPU sits relative to all of this: at the bottom, behind everything. That's the reason the benchmark below comes out the way it does. VLLM USE RUST BENCH=1 vllm bench serve … Same binary, bench subcommand. Requires VLLM RUST FRONTEND PATH to resolve, so it needs the same Step 3 → Step 5 you just did. The server came up healthy. These went by in the startup log anyway. Gemma 4 defeats the fast tokenizer: INFO hf.rs:200 loading tokenizer with fastokens WARNING hf.rs:221 failed to load tokenizer with fastokens; falling back to HuggingFace tokenizers error=tokenizer error: normalizer error: unsupported normalizer type: Replace fastokens 0.2.1 doesn't implement the Replace normalizer that Gemma 4's tokenizer.json uses, so it falls back to the same HuggingFace tokenizers the Python path uses. Note the fallback is graceful and correct — you just don't get the fast path on this model yet. It's a coverage gap in a young crate, and one normalizer away from closing. Multimodal isn't wired up for this model: WARNING multimodal.rs:446 multimodal model spec is not registered; disabling image/video support model id="google/gemma-4-E2B-it" model type="gemma4" Gemma 4 E2B is a vision model, and gemma4 isn't in the Rust multimodal spec table yet. Text requests behave identically and the endpoint is healthy, so nothing in a normal check reveals it. Also a registration gap rather than a design problem — but check it for your model before you switch, because a healthy endpoint won't tell you. On a T4G, no. Output token throughput, same engine config, client on the box against localhost: | Concurrency | 🐍 Python | 🦀 Rust | |---|---|---| | 1 | 28.65 | 29.30 | | 4 | 97.48 | 97.26 | | 8 | 168.33 | 169.39 | | 16 | 169.96 | 170.19 | | 32 | 170.99 | 170.34 | Median TTFT tracks just as tightly — 14305 ms against 14311 ms at concurrency 32. That's the expected result, and worth saying plainly: decode on this card is bandwidth-bound at a measured 277 GB/s, and the engine saturates at --max-num-seqs 8 . A frontend rewrite targets CPU-side per-request overhead. Here that overhead hides behind the GPU, so swapping it can't move a bottleneck-limited number. If you want the Rust frontend to buy you tokens per second on a small GPU, it won't. One signal does appear, in median inter-token latency at high concurrency: | Concurrency | 🐍 Python | 🦀 Rust | Δ | |---|---|---|---| | 16 | 38.55 | 36.18 | −6.4% | | 32 | 38.41 | 36.23 | −5.9% | Mean TPOT barely moves, so this is the middle of the distribution tightening rather than everything speeding up — the shape you'd expect from a frontend scheduling streaming work more evenly once many streams are in flight. Worth knowing if you serve at concurrency; not worth switching for on its own. 📊 pip install -e . already ran A from-source vLLM install done without the steps above succeeds, exits 0, and leaves you with a 96 MB debug tool parser and no frontend binary. Four defaults stack up to make that silent: | Symptom | Cause | Fix | |---|---|---| No vllm/vllm-rs after a clean build | protoc absent ⇒ vllm-server fails with code 101 | Step 3 | pip install exits 0 anyway | optional=not should require rust frontend — setuptools-rust swallows it | VLLM REQUIRE RUST FRONTEND=1 | rust tool parser.abi3.so is ~96 MB | editable ⇒ inplace ⇒ debug profile | --release | FileNotFoundError: … vllm-rs was not found | the above, discovered at import time | Steps 3 + 5 | Healthy server, but server: uvicorn | flag set on the api server module, which never reads it | vllm serve | VLLM REQUIRE RUST FRONTEND=1 turns the second row into a hard build failure, which is what you want on any machine you plan to serve from. toolchain — you supply these by hand because the sm 75 requirement forces --no-build-isolation, which disables pip's automatic build deps curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y . "$HOME/.cargo/env" /opt/pytorch/bin/python3 -m pip install setuptools rust the BUILD interpreter apt-get install -y protobuf-compiler cuda-toolkit-13-2 build against the DLAMI's torch, not a PyPI one PyPI aarch64 has no sm 75 cd /path/to/vllm python use existing torch.py TORCH CUDA ARCH LIST=7.5 VLLM REQUIRE RUST FRONTEND=1 \ pip install -e . --no-build-isolation Rust artifacts, release profile editable installs default to debug: 100x bigger VLLM REQUIRE RUST FRONTEND=1 python tools/build rust.py --release confirm ls -la vllm/vllm-rs && vllm/vllm-rs --help run — vllm serve , NOT the api server module VLLM USE RUST FRONTEND=1 vllm serve <model --host 0.0.0.0 --port 8000 verify it's really Rust curl -si localhost:8000/health | grep -i '^server:' Rust sends none pgrep -af vllm-rs Run on EC2 g5g.xlarge and g5g.4xlarge, us-east-1a, NVIDIA T4G SM 7.5 . vLLM