Running Modern SLMs (Gemma 4, Qwen 3 & Phi-4) on CPU: Pure Rust, Zero Python

A developer built a pure-Rust, Python-free CLI engine using Hugging Face's Candle framework that runs quantized GGUF versions of Google's Gemma 4 (E4B), Alibaba's Qwen 3 (4B), and Microsoft's Phi-4-mini (3.8B) on standard CPUs. The tool switches between model families from the terminal, auto-selecting the right Hugging Face repo, GGUF file, and chat template (ChatML, Gemma, or Phi), and reports benchmarks on RAM usage, cold-start latency, and token generation speed. The developer argues CPU inference in Rust is now a viable production architecture for edge devices, IoT gateways, and low-cost cloud instances.

When I first shared my experiment on running Microsoft’s Phi-3 on a standard consumer CPU using pure Rust and Hugging Face’s Candle framework, the mission was straightforward: prove that we don’t need bloated Python runtimes, complex CUDA setups, or 5GB+ Docker images just to run local AI inference. Fast forward to today, and the Small Language Model SLM landscape has exploded. With the release of edge-optimized powerhouses like Google’s Gemma 4 E4B , Alibaba’s Qwen 3 / 3.6 4B , and Microsoft’s Phi-4-mini 3.8B , running high-reasoning AI directly on standard CPU hardware is no longer just a weekend hack—it is a viable production architecture. In this guide, we will upgrade our pure-Rust inference pipeline to build a multi-model CLI engine capable of running the latest quantized GGUF models on a standard CPU, complete with real-world benchmarks on RAM usage, cold-start latency, and token generation speeds. Why go through the trouble of writing inference code in Rust when Python has transformers and vLLM ? When deploying to edge devices, IoT gateways, or cost-effective CPU cloud instances, runtime overhead matters. Instead of locking our code to a single model, we are testing three of the strongest compact models currently dominating local AI: Let’s write a unified Rust CLI tool that lets you switch between Phi-4 , Qwen , and Gemma models straight from the terminal, leveraging candle-transformers and hf-hub . Cargo.toml Create a new project cargo new rust-slm-bench and configure your Cargo.toml : package name = "rust-slm-bench" version = "0.2.0" edition = "2021" dependencies anyhow = "1.0" candle-core = "0.8" candle-nn = "0.8" candle-transformers = "0.8" hf-hub = { version = "0.3", features = "tokio" } tokenizers = "0.20" clap = { version = "4.5", features = "derive", "ValueEnum" } src/main.rs Below is the clean, streaming implementation. We define an enum for our model families so our engine automatically selects the right Hugging Face repository, GGUF file, and Chat Template ChatML vs. Gemma vs. Phi : use anyhow::{Error as E, Result}; use candle core::quantized::gguf file; use candle core::{Device, Tensor}; use candle transformers::generation::LogitsProcessor; use candle transformers::models::quantized llama::ModelWeights as LlamaStyleModel; use candle transformers::models::quantized phi3::ModelWeights as PhiStyleModel; use candle transformers::models::quantized qwen2::ModelWeights as QwenStyleModel; use clap::{Parser, ValueEnum}; use hf hub::api::sync::Api; use std::io::Write; use tokenizers::Tokenizer; derive Debug, Clone, ValueEnum enum ModelFamily { Phi4, Qwen3, Gemma4, } enum LoadedModel { Phi PhiStyleModel , Qwen QwenStyleModel , Gemma LlamaStyleModel , } impl LoadedModel { fn forward &mut self, x: &Tensor, index pos: usize - candle core::Result<Tensor { match self { Self::Phi m = m.forward x, index pos , Self::Qwen m = m.forward x, index pos , Self::Gemma m = m.forward x, index pos , } } } derive Parser, Debug command author, version, about = "Benchmark Modern SLMs on CPU with Rust & Candle" struct Args { /// Select which SLM family to run arg short, long, value enum, default value t = ModelFamily::Qwen3 model: ModelFamily, /// The prompt to send to the model arg short, long, default value = "Explain why Rust is great for Edge AI in 3 short bullet points." prompt: String, /// Maximum tokens to generate arg short = 'n', long, default value t = 200 max tokens: usize, /// Sampling temperature arg short, long, default value t = 0.6 temperature: f64, } fn main - Result< { let args = Args::parse ; let device = Device::Cpu; let api = Api::new ?; // Map each model family to its GGUF repo, filename, tokenizer, and prompt format let gguf repo, gguf file, tok repo, formatted prompt, eos str = match args.model { ModelFamily::Phi4 = "bartowski/Phi-4-mini-instruct-GGUF", "Phi-4-mini-instruct-Q4 K M.gguf", "microsoft/Phi-4-mini-instruct", format "<|system| You are a concise technical assistant.<|end| <|user| {}<|end| <|assistant| ", args.prompt , "<|end| ", , ModelFamily::Qwen3 = "Qwen/Qwen3-4B-GGUF", "qwen3-4b-q4 k m.gguf", "Qwen/Qwen3-4B", format "<|im start| system\nYou are a concise technical assistant.<|im end| \n<|im start| user\n{}<|im end| \n<|im start| assistant\n", args.prompt , "<|im end| ", , ModelFamily::Gemma4 = "bartowski/gemma-4-E4B-it-GGUF", "gemma-4-E4B-it-Q4 K M.gguf", "google/gemma-4-E4B-it", format "<start of turn user\n{}<end of turn \n<start of turn model\n", args.prompt , "<end of turn ", , }; println "📥 Fetching {:?} weights & tokenizer...", args.model ; let model path = api.model gguf repo.to string .get gguf file ?; let tokenizer path = api.model tok repo.to string .get "tokenizer.json" ?; let tokenizer = Tokenizer::from file tokenizer path .map err E::msg ?; let load start = std::time::Instant::now ; let mut file = std::fs::File::open &model path ?; let content = gguf file::Content::read &mut file ?; let mut model = match args.model { ModelFamily::Phi4 = LoadedModel::Phi PhiStyleModel::from gguf false, content, &mut file, &device ? , ModelFamily::Qwen3 = LoadedModel::Qwen QwenStyleModel::from gguf content, &mut file, &device ? , ModelFamily::Gemma4 = LoadedModel::Gemma LlamaStyleModel::from gguf content, &mut file, &device ? , }; println "⚡ Cold-start model load completed in {:.2}s", load start.elapsed .as secs f64 ; let tokens = tokenizer .encode formatted prompt, true .map err E::msg ? .get ids .to vec ; let mut logits processor = LogitsProcessor::new 42, Some args.temperature , None ; let gen start = std::time::Instant::now ; // Prefill step: process the input prompt let input = Tensor::new tokens.as slice , &device ?.unsqueeze 0 ?; let logits = model.forward &input, 0 ?.squeeze 0 ?; let mut next token = logits processor.sample &logits ?; print "\n🤖 Response:\n" ; if let Ok word = tokenizer.decode & next token , true { print "{word}" ; std::io::stdout .flush ?; } let eos token = tokenizer.get vocab true .get eos str .unwrap or &2 ; let mut generated = 1; // Decode step: autoregressive token streaming for index in 0..args.max tokens { let input = Tensor::new & next token , &device ?.unsqueeze 0 ?; let logits = model.forward &input, tokens.len + index ?.squeeze 0 ?; next token = logits processor.sample &logits ?; if next token == eos token { break; } generated += 1; if let Ok word = tokenizer.decode & next token , true { print "{word}" ; std::io::stdout .flush ?; } } let elapsed = gen start.elapsed .as secs f64 ; println "\n\n📊 Benchmark: {} tokens in {:.2}s {:.2} tokens/sec ", generated, elapsed, generated as f64 / elapsed ; Ok } When running matrix multiplications on a CPU, compiler flags make the difference between sluggish output and blazing-fast generation. Always compile with --release and instruct rustc to utilize native SIMD instructions AVX2 / AVX-512 on x86 64 or NEON on ARM : Run Qwen 3 on CPU RUSTFLAGS="-C target-cpu=native" cargo run --release -- --model qwen3 Run Gemma 4 on CPU RUSTFLAGS="-C target-cpu=native" cargo run --release -- --model gemma4 Run Phi-4-mini on CPU RUSTFLAGS="-C target-cpu=native" cargo run --release -- --model phi4 All tests below were executed on a standard consumer CPU no discrete GPU acceleration used using Q4 K M 4-bit quantized GGUF weights and compiled with target-cpu=native : | Metric | Phi-3-mini 3.8B Baseline | Phi-4-mini 3.8B | Qwen 3 / 3.6 4B | Gemma 4 E4B | |---|---|---|---|---| | Compiled Binary Size | ~18 MB | ~18 MB | ~18 MB | ~18 MB | | GGUF Weight Size | 2.39 GB | 2.49 GB | 2.55 GB | 2.62 GB | | Peak RAM Usage | ~2.7 GB | ~2.8 GB | ~2.9 GB | ~3.0 GB | | Cold Start Latency | 1.15 sec | 1.20 sec | 1.18 sec | 1.25 sec | | Generation Speed CPU | ~24 tok/s | ~23 tok/s | ~28 tok/s | ~29 tok/s | | Best Use Case | General Chat | Logic & Math | Coding & JSON | Edge & Multilingual | LoadedModel inside an Arc<tokio::sync::Mutex<... and serving it via Local AI is no longer limited to developers with expensive multi-GPU rigs. By combining modern 4-bit quantized SLMs Gemma 4, Qwen 3, and Phi-4 with the zero-cost abstractions of Rust and Candle , we can deploy remarkably capable AI systems onto standard CPUs, edge servers, and embedded hardware. Have you benchmarked any of the new 2026 SLMs on CPU yet? Drop your thoughts or performance numbers in the comments