Two Rust Clients for Gemma 4: Calling the Endpoint vs. Calling the MCP Server 🦀

A developer published two Rust CLI demos, gemma-rust and gemma-rust-mcp, that query a self-hosted Gemma 4 E2B model through an OpenAI-compatible HTTP endpoint and through an MCP server, respectively. The HTTP client prints the raw model response and metadata, while the MCP client launches the rig's own Python MCP server and returns tool output as markdown, illustrating what an agent sees versus what the model said. Both clients run against llama-server on a GTX 1650 Ti and vLLM on an NVIDIA L4 in Cloud Run.

This article provides a step by step guide to two small Rust CLIs that ask a self-hosted Gemma 4 E2B the same question. The first calls the model's OpenAI-compatible HTTP endpoint directly. The second is an MCP client: it launches the rig's own MCP server and asks through its tools. https://github.com/xbill9/gemma-rust https://github.com/xbill9/gemma-rust https://github.com/xbill9/gemma-rust-mcp https://github.com/xbill9/gemma-rust-mcp Both CLIs are demos, and their output is read by an audience. So neither hides anything behind a --verbose flag: every run prints the target, the health check, the request, the answer, the model's reasoning, token counts, latency, and whatever the server says about itself. They run against two very different deployments of the same model with the same code: llama-server on a 2021-era laptop GPU, a GTX 1650 Ti with 4 GiB, no auth The interesting part is what changes when the same question goes through MCP instead of HTTP. It is not the answer. Because they answer two different questions. gemma-rust shows what the model said. One HTTP call, the raw OpenAI-style response, every field printed. gemma-rust-mcp shows what an agent sees. An MCP client like Claude Code never touches the endpoint. It calls tools, and gets back whatever those tools choose to report. Writing a second client in Rust — one that is not Claude Code and not the Python SDK the servers were built with — is the fastest way to find out what those servers actually return. Neither one starts, stops or deploys anything. The rigs do that. gemma-rust ─────── HTTP reqwest ───────────────────────┐ ├──▶ llama-server GTX 1650 Ti, local │ vLLM NVIDIA L4, Cloud Run gemma-rust-mcp ─── MCP over stdio rmcp ──▶ server.py ──┘ Python, the rig's own The MCP path makes the same HTTP call in the end. It just makes it from inside a Python process that the Rust client launched, and hands back markdown instead of JSON. The strategy for building the two clients is an incremental step by step approach. First, a model server is brought up locally and checked with curl . Then the HTTP client is built and validated against it, including the two ways Gemma 4 returns an empty answer from a healthy server. The same binary is then pointed at Cloud Run. Then the rig's Python MCP server is installed, the MCP client is built, and the same question goes through the same two servers again — which is where the comparison comes from. nvcc — this one is a GTX 1650 Ti, CUDA 13.3 git , cmake , a C++ compiler, and curl hf CLI Use rustup : curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source ~/.cargo/env rustc --version rustc 1.98.1 48a229cea 2026-09-01 Anything recent works. The floors come from the dependencies' own rust-version : | Crate | Needs Rust | Needed by | |---|---|---| | reqwest 0.13.5 | 1.85.0 | gemma-rust | | clap 4.6.6 | 1.85 | both | | rmcp 3.3.0 | 1.88 | gemma-rust-mcp | Both crates are edition 2024. llama-server is the local model server. Build it from source, at the commit the rig runs: git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp cd ~/llama.cpp git checkout 95ef7fc cmake -B build -DGGML CUDA=ON -DCMAKE CUDA ARCHITECTURES=75 -DCMAKE BUILD TYPE=Release cmake --build build --config Release -j --target llama-server ls build/bin/llama-server build/bin/llama-server 75 is Turing, which is what a GTX 1650 Ti is. Set your own card's compute capability there, or leave the flag off and let CMake detect it. The rig serves Google's QAT q4 0 GGUF of Gemma 4 E2B: hf auth login hf download google/gemma-4-E2B-it-qat-q4 0-gguf --local-dir ~/models/gemma-4-E2B-it-qat-q4 0 ls -l ~/models/gemma-4-E2B-it-qat-q4 0/ -rw-rw-r-- 1 xbill xbill 3349516256 Sep 3 13:19 gemma-4-E2B q4 0-it.gguf It fits a 4 GiB card because most of the file never leaves the host — the previous article https://dev.to/gde/gemma-4-on-an-old-4-gb-laptop-gpu-qat-takes-it-from-95-gib-to-16-b5l measures that. Run it in the foreground; Ctrl-C is the whole teardown: ~/llama.cpp/build/bin/llama-server \ -m ~/models/gemma-4-E2B-it-qat-q4 0/gemma-4-E2B q4 0-it.gguf \ --host 127.0.0.1 --port 8080 -ngl 99 -c 8192 From a second terminal: curl -s http://127.0.0.1:8080/health {"status":"ok"} 🟢 That is the whole server side for the local target. The rig wraps this same command as make serve , with its flags in tpu.env . cd ~ git clone https://github.com/xbill9/gemma-rust cd gemma-rust make prod Building release... Finished release profile optimized target s in 0.08s Binary: target/release/gemma-rust That time is an incremental rebuild; a clean one compiles the dependency tree first. The dependencies are few: dependencies anyhow = "1.0.104" clap = { version = "4.6.6", features = "derive", "env" } reqwest = { version = "0.13.5", default-features = false, features = "blocking", "json", "rustls" } rustyline = "18.0.1" serde = { version = "1.0.229", features = "derive" } serde json = "1.0.151" blocking is deliberate. One question, one answer — there is nothing to run concurrently, so there is no async runtime in the client's own code. Lint is the gate: make lint Linting code... Finished dev profile unoptimized + debuginfo target s in 0.09s That is cargo clippy --all-targets -- -D warnings and cargo fmt --check . make test runs and finds 0 tests : both crates are demos, validated by running them against live servers, which is what the rest of this article does. ./target/release/gemma-rust "In one sentence, what is a TPU?" == Target ============================================================ endpoint http://127.0.0.1:8080 target local llama.cpp rig auth none == Health ============================================================ GET /health 200 OK in 0 ms body {"status":"ok"} == Model ============================================================= served /home/xbill/models/gemma-4-E2B-it-qat-q4 0/gemma-4-E2B q4 0-it.gguf context 8192 tokens using /home/xbill/models/gemma-4-E2B-it-qat-q4 0/gemma-4-E2B q4 0-it.gguf first model the server lists == Request =========================================================== POST http://127.0.0.1:8080/v1/chat/completions prompt In one sentence, what is a TPU? max tokens 1024 == Answer ============================================================ A TPU Tensor Processing Unit is a specialized hardware accelerator designed by Google specifically to speed up the computationally intensive matrix operations required for training and running machine learning models. == Reasoning ========================================================= length 1327 chars 1. Identify the core concept: The user wants a one-sentence definition of a TPU Tensor Processing Unit . ... == Stats ============================================================= finish reason stop prompt tokens 25 cached tokens 7 completion tokens 330 total tokens 355 latency client 4786 ms tokens/s client 69.0 completion tokens / latency; includes network and prefill == Server timings llama.cpp ======================================== predicted ms 4615.20 predicted n 330 predicted per second 71.29 prompt ms 147.38 prompt n 18 ... == Response ========================================================== id chatcmpl-rTO4HJkYHsrE4WisG2mileq6Jmxn0R4f model /home/xbill/models/gemma-4-E2B-it-qat-q4 0/gemma-4-E2B q4 0-it.gguf system fingerprint b1-95ef7fc ✅ A one-sentence answer, and 1,327 characters of thinking in front of it. Gemma 4 on llama.cpp reasons by default, and that is where most of the 330 completion tokens went. Three decisions make one code path work on two servers that disagree about almost everything. The model id comes from the server. llama.cpp accepts any model value; vLLM returns 404 unless it is exactly the served id. The only default that works on both is the first id from /v1/models : js let model = served .first .and then |m| m "id" .as str .context "the server listed no models at /v1/models; pass --model" ? .to string ; Both reasoning fields are read. The two servers put Gemma's thinking in different places: derive Deserialize struct Message { content: Option<String , /// Where llama.cpp puts Gemma 4's thinking reasoning content: Option<String , /// Where vLLM puts it reasoning: Option<String , } Server stats are optional and printed generically. llama.cpp returns a timings object; vLLM returns none. Both are Option<Value , and whichever is present gets its own section. Auth follows the host: --auth auto runs gcloud auth print-identity-token for .run.app endpoints only, and prints the token's length, never the token. Use /v1/chat/completions , never /v1/completions . On these instruction-tuned checkpoints the raw completions endpoint returns empty text, which looks exactly like a broken server. Give Gemma room to think. On llama.cpp, content stays empty until the thinking closes. Starve it and see: ./target/release/gemma-rust --max-tokens 32 "In one sentence, what is a TPU?"; echo "exit=$?" == Request =========================================================== POST http://127.0.0.1:8080/v1/chat/completions prompt In one sentence, what is a TPU? max tokens 32 warning: below 512, Gemma 4 may still be reasoning when it hits the limit and return an empty answer == Answer ============================================================ empty The model was still reasoning when it stopped. This is Gemma 4 thinking, not a broken server: raise --max-tokens. == Reasoning ========================================================= length 116 chars 1. Analyze the Request: The user wants a definition of a TPU Tensor Processing Unit in a single sentence . 2 == Stats ============================================================= finish reason length The reply hit max tokens and is cut off. exit=2 finish reason: length , empty content , non-empty reasoning. The CLI says what happened and exits 2 , so a script cannot mistake an empty answer for success. That is why --max-tokens defaults to 1024. --status skips the question and probes every endpoint either server might offer: ./target/release/gemma-rust --status == Server ============================================================ GET /version 404 Not Found not served by this server GET /props 200 OK in 0 ms llama.cpp build b1-95ef7fc model ftype Q4 0 n ctx 8192 total slots 1 modalities text only == Model details ===================================================== GET /v1/models 200 OK in 0 ms n ctx train 131072 n embd 1536 n params 4628569635 4.63 B parameters size 3333699724 3.33 GB on disk == Slots ============================================================= GET /slots 200 OK in 0 ms slots 1 0 busy == Metrics =========================================================== GET /metrics 200 OK in 0 ms predicted tokens seconds 67.523 requests processing 0 ... A 404 is reported, not an error. /version is vLLM's; /props and /slots are llama.cpp's. Each server answers half the probes, and the list of which ones is itself useful. Same binary, different endpoint. If you deployed the Cloud Run rig, its Makefile prints the URL: GEMMA ENDPOINT=$ make -s -C ~/gemma4-dev/gpu-2B-cloudrun-devops-agent endpoint \ ./target/release/gemma-rust "In one sentence, what is a TPU?" == Target ============================================================ endpoint https://<your-service .a.run.app target Cloud Run auth bearer token from gcloud auth print-identity-token 845 chars, not shown == Health ============================================================ Cloud Run scales to zero: the first request can take minutes while a GPU instance starts. GET /health 200 OK in 185 ms == Model ============================================================= served /mnt/models/gemma-4-E2B-it context 16384 tokens using /mnt/models/gemma-4-E2B-it first model the server lists == Answer ============================================================ A TPU Tensor Processing Unit is a specialized integrated circuit designed to accelerate machine learning workloads, particularly those involving large matrix multiplications common in deep learning. == Reasoning ========================================================= none returned == Stats ============================================================= finish reason stop prompt tokens 18 cached tokens - completion tokens 31 total tokens 49 latency client 662 ms tokens/s client 46.8 completion tokens / latency; includes network and prefill == Response ========================================================== id chatcmpl-a870c80a585e2371 model /mnt/models/gemma-4-E2B-it system fingerprint vllm-0.26.0-a3e182ca 🟢 31 tokens and no reasoning — vLLM does not think by default here — so the round trip is 662 ms. The service is --no-allow-unauthenticated ; without a token the CLI explains itself instead of dumping Google's error page: GET /health 403 Forbidden in 304 ms body 299 bytes, not shown The server rejected the request's credentials. Cloud Run needs an identity token from an account with roles/run.invoker gcloud auth print-identity-token . What the one code path had to absorb: | | local llama.cpp | Cloud Run vLLM 0.26 | |---|---|---| | auth | none | identity token 403 without | | model field | any value | exact served id 404 otherwise | | reasoning field | reasoning content | reasoning | | thinks by default | yes | no | | server stats | timings object | none | | context | 8192 | 16384 | The MCP client does not talk to the model. It launches a rig's server.py , so the rigs come next: git clone https://github.com/xbill9/gemma4-dev ~/gemma4-dev make -C ~/gemma4-dev/local-llamacpp-1650ti-2b-q4 0 install python3 -c "import importlib.metadata as m;print 'mcp', m.version 'mcp' " mcp 2.2.0 The rig servers need mcp =2 — the Python SDK line where FastMCP became MCPServer . They install into the system python3 ; if yours refuses system-wide installs, use a virtualenv and point the client at it with --python or GEMMA PYTHON . The rig reads its model path and llama-server location from tpu.env . A real environment variable wins over that file, so set MODEL PATH and LLAMA SERVER BIN if yours live somewhere else. cd ~ git clone https://github.com/xbill9/gemma-rust-mcp cd gemma-rust-mcp make prod Building release... Finished release profile optimized target s in 0.04s Binary: target/release/gemma-rust-mcp The feature flags are the part to get right. rmcp 's defaults are base64 , macros and server — a server's feature set. A client has to ask for client and a transport by name: dependencies anyhow = "1.0.104" clap = { version = "4.6.6", features = "derive", "env" } rmcp = { version = "3.3.0", features = "client", "transport-child-process" } rustyline = "18.0.1" serde = "1.0.229" serde json = "1.0.151" tokio = { version = "1.53.1", features = "macros", "rt-multi-thread", "process", "time" } | Feature | What it brings | |---|---| | client | ServiceExt::serve on the client side, call tool , list all tools | | transport-child-process | TokioChildProcess : spawn a server, speak MCP over its stdin/stdout | make lint is clean here too, and make test again finds 0 tests . Launch the server as a child process. The rig opens its files by relative path, so the working directory matters. Stderr is piped so the server's own log can be printed at the end: js let mut cmd = Command::new &args.python ; cmd.arg "server.py" .current dir &dir ; let transport, stderr = TokioChildProcess::builder cmd .stderr Stdio::piped .spawn ?; Handshake. The client handler is — this client has no callbacks to offer the server: js let client = tokio::time::timeout timeout, .serve transport .await??; List the tools , then call one : js let tools = client.list all tools .await?; let params = CallToolRequestParams::new name .with arguments arguments ; let result = client.call tool params .await?; 💡 CallToolRequestParams is non exhaustive , so a struct literal will not compile. Use the constructor and the builder method. ./target/release/gemma-rust-mcp "In one sentence, what is a TPU?" == MCP server ======================================================== rig local-llamacpp-1650ti-2b-q4 0 command python3 server.py working dir /home/xbill/gemma4-dev/local-llamacpp-1650ti-2b-q4 0 transport stdio child process pid 334541 initialize ok in 793 ms == Server info ======================================================= name local-llamacpp-1650ti-2b-q4 0 version empty protocol 2025-11-25 capabilities tools, resources, prompts == Tools ============================================================= tools/list 7 tools in 2 ms gpu status Report the local GPU: name, compute capability, VRAM total/… model info Report the configured checkpoint, where it is, and the resi… start model server Start llama-server on the local GPU. No-op if it is already… stop model server Stop the running llama-server. Teardown is complete — nothi… model server status Check whether llama-server is up and serving at the known l… query model Send a chat completion to the local endpoint and return the… get help List the tools this rig exposes. = called by this demo, which only calls read-only tools == tools/call gpu status ============================================= arguments {} latency 14 ms isError false result: 📡 GPU — local-llamacpp-1650ti-2b-q4 0 NVIDIA GeForce GTX 1650 Ti with Max-Q Design, 7.5, 4096 MiB, 1632 MiB, 2101 MiB, 615.71.09 == tools/call model server status ==================================== latency 27 ms result: ✅ Serving at http://127.0.0.1:8080 pid 83619 . /health → 200. == tools/call query model ============================================ arguments {"max tokens":1024,"prompt":"In one sentence, what is a TPU?"} latency 4608 ms isError false result: ✅ Reply A TPU Tensor Processing Unit is a specialized integrated circuit developed by Google designed to accelerate machine learning workloads, specifically the complex matrix multiplications required by neural networks, significantly speeding up training and inference. --- plus 1173 chars of reasoning, suppressed prompt 25 tok · completion 326 tok · 71.3 tok/s == Server log stderr =============================================== 2026-09-11 16:27:09,682 INFO HTTP Request: GET http://127.0.0.1:8080/health "HTTP/1.1 200 OK" 2026-09-11 16:27:14,292 INFO HTTP Request: POST http://127.0.0.1:8080/v1/chat/completions "HTTP/1.1 200 OK" ✅ Same model, same kind of answer. What came back around it is completely different: the GPU and its memory, the server's health, and a reply formatted for an agent to read — with the reasoning reduced to its length. The server log at the bottom shows the HTTP call the tool made on the client's behalf. That is the "same servers" arrow in the diagram, visible. The rig servers also offer tools that start and stop the model server, and on Cloud Run, deploy, destroy and rescale a billed GPU service. The client calls a fixed list and nothing else: /// Read-only tools called before the query. Never add a tool that deploys, destroys, /// scales, starts, or stops anything: those are billed or destructive. fn status tools self - &'static &'static str { match self { Rig::Local = & "gpu status", "model server status", "model info" , Rig::Cloudrun = & "cloudrun status", "cloudrun get system status", "cloudrun get model details", , } } There is no "call any tool" option, on the command line or in interactive mode. A demo that can be talked into cloudrun destroy is not a demo anyone should run in front of an audience. MCP has an isError flag on every tool result. These rig tools never set it. They report failure as markdown that starts with ❌ , and a reasoning-only reply as markdown that starts with 📡 : ./target/release/gemma-rust-mcp --no-status --max-tokens 32 "In one sentence, what is a TPU?"; echo "exit=$?" == tools/call query model ============================================ arguments {"max tokens":32,"prompt":"In one sentence, what is a TPU?"} latency 499 ms isError false result: 📡 Reasoning only — no answer yet. finish reason: length after 32 tokens, all of them thinking. This is Gemma 4 reasoning, not a broken server. Re-run with a larger max tokens currently 32 . ... No answer: the model was still reasoning when it stopped. Raise --max-tokens. exit=2 isError false , and still not an answer. A client that trusts the protocol flag reports success here. This one reads the first character of the text, and exits 2 exactly like the HTTP client does. ./target/release/gemma-rust-mcp --rig cloudrun "In one sentence, what is a TPU?" == MCP server ======================================================== rig gpu-2B-cloudrun-devops-agent initialize ok in 3219 ms == Server info ======================================================= name Self-Hosted vLLM DevOps Agent protocol 2025-11-25 == Tools ============================================================= tools/list 27 tools in 2 ms ... == tools/call cloudrun query gemma4 with stats ======================= arguments {"prompt":"In one sentence, what is a TPU?"} latency 1333 ms isError false result: 📊 Performance Stats - Model: /mnt/models/gemma-4-E2B-it - Time to First Token TTFT : 0.086s - Total Generation Time: 0.688s - Tokens per Second: 53.11 tokens/s - Total Tokens approx. : 32 💬 Model Response A TPU Tensor Processing Unit is a specialized type of integrated circuit designed to accelerate machine learning workloads, particularly those involving tensor operations common in deep learning.<turn| == Server log stderr =============================================== ... INFO - 📡 Automatically discovered vLLM at: https://<your-service .a.run.app 2026-09-11 11:41:36,754 - httpx - INFO - HTTP Request: GET https://<your-service .a.run.app/health "HTTP/1.1 200 OK" 2026-09-11 11:41:37,399 - httpx2 - INFO - HTTP Request: GET https://<your-service .a.run.app/v1/models "HTTP/1.1 200 OK" 2026-09-11 11:41:37,479 - httpx2 - INFO - HTTP Request: POST https://<your-service .a.run.app/v1/chat/completions "HTTP/1.1 200 OK" Three things the HTTP client could not have shown: GET /health and a GET /v1/models before its POST . The HTTP client does those once per session; this tool does them on every call. <turn| , Gemma's end-of-turn marker, leaked by the rig's streaming tool. The client prints what the tool returned — which is how the bug was found. The same question, through the same two servers: | | 🦀 gemma-rust HTTP | 🦀 gemma-rust-mcp MCP | |---|---|---| | What you get | 🥇 the raw OpenAI-style response | what the tool chooses to report, as markdown | | Reasoning | 🥇 full text | local: its length only; Cloud Run: none | | Token counts | 🥇 exact, from usage , incl. cached | local: exact; Cloud Run: approximate | | Rig status | HTTP probes of the server | 🥇 GPU, model and deployment, from the rig's tools | | -i keeps the conversation | 🥇 yes | no — the tool takes one prompt | | Runs on its own | 🥇 yes | needs Python 3, mcp =2 and the rig's server.py | And what it cost, measured: | | 🦀 gemma-rust HTTP | 🦀 gemma-rust-mcp MCP | |---|---|---| | Local query | 4786 ms, 330 tokens | 4608 ms, 326 tokens | | Cloud Run query | 🥇 662 ms | 1333 ms | | Before the first query, Cloud Run | 🥇 185 ms GET /health , plus the token fetch | 3219 ms initialize | | Crates in Cargo.lock | 186 | 🥇 121 | | Release binary | 8.5M | 🥇 6.5M | | src/main.rs | 777 lines | 🥇 542 lines | Locally, it is a tie. 4786 ms for 330 tokens against 4608 ms for 326: the model's thinking dominates both, and one sample each cannot separate the MCP layer from the difference between one Gemma reply and the next. On Cloud Run, the hop shows. 1333 ms against 662 ms is 671 ms arithmetic , roughly the two extra round trips the tool makes before it asks. Session start is where MCP really pays: 3219 ms to launch Python, import the SDK and discover the service URL through gcloud , before the first tool call. The smaller crate is the MCP one , because it speaks stdio and never opens a TLS connection. reqwest with rustls brings a TLS stack; the MCP client leaves the HTTPS to the Python server. Use the HTTP client to see the model. Reasoning, exact token counts, cached tokens, server timings, the raw JSON with --raw . When the question is "what did Gemma do", this is the one. Use the MCP client to see the agent's view. It is the fastest way to test an MCP server from outside the SDK it was written with, and it found two things no unit test did: a failure flag that is never set, and an end-of-turn marker leaking into answers. Neither half of that is about Rust being fast. The work is a model thinking on a GPU. Rust earns its place here with one static binary per client, a type for every response field, and a compiler that notices when vLLM and llama.cpp disagree about where the reasoning goes. The goal of this article was to ask Gemma 4 E2B the same question from Rust in two ways — directly over its HTTP endpoint, and through the rig's MCP server — on a local llama.cpp GPU and on Cloud Run. The key to the solution was printing everything each path returns, which turns two small clients into a side-by-side view of what a model says and what an agent sees. The results were: /v1/models and reading both reasoning fields isError false, and the Cloud Run tool leaks Scope: one laptop with a GTX 1650 Ti Max-Q 4 GiB, driver 615.71.09 running llama.cpp b1-95ef7fc with the Gemma 4 E2B QAT q4 0 GGUF, and one Cloud Run service on an NVIDIA L4 running vLLM 0.26.0. One run per client per target, so each figure is a single sample and model replies vary from run to run; the local runs were captured at 16:27 on 2026-09-11 and the Cloud Run runs earlier the same day, and the HTTP and MCP runs are separate requests, not the same one observed twice. Rust 1.98.1, rmcp 3.3.0, reqwest 0.13.5, Python 3.14.7 with mcp 2.2.0. The strategy for using Rust to call Gemma 4 over HTTP and over MCP was validated with an incremental step by step approach.