{"slug": "running-modern-slms-gemma-4-qwen-3-phi-4-on-cpu-pure-rust-zero-python", "title": "Running Modern SLMs (Gemma 4, Qwen 3 & Phi-4) on CPU: Pure Rust, Zero Python", "summary": "A developer built a pure-Rust, Python-free CLI engine using Hugging Face's Candle framework that runs quantized GGUF versions of Google's Gemma 4 (E4B), Alibaba's Qwen 3 (4B), and Microsoft's Phi-4-mini (3.8B) on standard CPUs. The tool switches between model families from the terminal, auto-selecting the right Hugging Face repo, GGUF file, and chat template (ChatML, Gemma, or Phi), and reports benchmarks on RAM usage, cold-start latency, and token generation speed. The developer argues CPU inference in Rust is now a viable production architecture for edge devices, IoT gateways, and low-cost cloud instances.", "body_md": "When I first shared my experiment on running Microsoft’s Phi-3 on a standard consumer CPU using pure Rust and Hugging Face’s Candle framework, the mission was straightforward: prove that we don’t need bloated Python runtimes, complex CUDA setups, or 5GB+ Docker images just to run local AI inference.\n\nFast forward to today, and the Small Language Model (SLM) landscape has exploded. With the release of edge-optimized powerhouses like **Google’s Gemma 4 (E4B)**, **Alibaba’s Qwen 3 / 3.6 (4B)**, and **Microsoft’s Phi-4-mini (3.8B)**, running high-reasoning AI directly on standard CPU hardware is no longer just a weekend hack—it is a viable production architecture.\n\nIn this guide, we will upgrade our pure-Rust inference pipeline to build a **multi-model CLI engine** capable of running the latest quantized GGUF models on a standard CPU, complete with real-world benchmarks on RAM usage, cold-start latency, and token generation speeds.\n\nWhy go through the trouble of writing inference code in Rust when Python has `transformers` and `vLLM`? When deploying to edge devices, IoT gateways, or cost-effective CPU cloud instances, runtime overhead matters.\n\nInstead of locking our code to a single model, we are testing three of the strongest compact models currently dominating local AI:\n\nLet’s write a unified Rust CLI tool that lets you switch between **Phi-4**, **Qwen**, and **Gemma** models straight from the terminal, leveraging `candle-transformers` and `hf-hub`.\n\n`Cargo.toml`)\nCreate a new project (`cargo new rust-slm-bench`) and configure your `Cargo.toml`:\n\n```\n[package]\nname = \"rust-slm-bench\"\nversion = \"0.2.0\"\nedition = \"2021\"\n\n[dependencies]\nanyhow = \"1.0\"\ncandle-core = \"0.8\"\ncandle-nn = \"0.8\"\ncandle-transformers = \"0.8\"\nhf-hub = { version = \"0.3\", features = [\"tokio\"] }\ntokenizers = \"0.20\"\nclap = { version = \"4.5\", features = [\"derive\", \"ValueEnum\"] }\n```\n\n`src/main.rs`)\nBelow is the clean, streaming implementation. We define an enum for our model families so our engine automatically selects the right Hugging Face repository, GGUF file, and Chat Template (`ChatML` vs. `Gemma` vs. `Phi`):\n\n```\nuse anyhow::{Error as E, Result};\nuse candle_core::quantized::gguf_file;\nuse candle_core::{Device, Tensor};\nuse candle_transformers::generation::LogitsProcessor;\nuse candle_transformers::models::quantized_llama::ModelWeights as LlamaStyleModel;\nuse candle_transformers::models::quantized_phi3::ModelWeights as PhiStyleModel;\nuse candle_transformers::models::quantized_qwen2::ModelWeights as QwenStyleModel;\nuse clap::{Parser, ValueEnum};\nuse hf_hub::api::sync::Api;\nuse std::io::Write;\nuse tokenizers::Tokenizer;\n\n#[derive(Debug, Clone, ValueEnum)]\nenum ModelFamily {\n    Phi4,\n    Qwen3,\n    Gemma4,\n}\n\nenum LoadedModel {\n    Phi(PhiStyleModel),\n    Qwen(QwenStyleModel),\n    Gemma(LlamaStyleModel),\n}\n\nimpl LoadedModel {\n    fn forward(&mut self, x: &Tensor, index_pos: usize) -> candle_core::Result<Tensor> {\n        match self {\n            Self::Phi(m) => m.forward(x, index_pos),\n            Self::Qwen(m) => m.forward(x, index_pos),\n            Self::Gemma(m) => m.forward(x, index_pos),\n        }\n    }\n}\n\n#[derive(Parser, Debug)]\n#[command(author, version, about = \"Benchmark Modern SLMs on CPU with Rust & Candle\")]\nstruct Args {\n    /// Select which SLM family to run\n    #[arg(short, long, value_enum, default_value_t = ModelFamily::Qwen3)]\n    model: ModelFamily,\n\n    /// The prompt to send to the model\n    #[arg(short, long, default_value = \"Explain why Rust is great for Edge AI in 3 short bullet points.\")]\n    prompt: String,\n\n    /// Maximum tokens to generate\n    #[arg(short = 'n', long, default_value_t = 200)]\n    max_tokens: usize,\n\n    /// Sampling temperature\n    #[arg(short, long, default_value_t = 0.6)]\n    temperature: f64,\n}\n\nfn main() -> Result<()> {\n    let args = Args::parse();\n    let device = Device::Cpu;\n    let api = Api::new()?;\n\n    // Map each model family to its GGUF repo, filename, tokenizer, and prompt format\n    let (gguf_repo, gguf_file, tok_repo, formatted_prompt, eos_str) = match args.model {\n        ModelFamily::Phi4 => (\n            \"bartowski/Phi-4-mini-instruct-GGUF\",\n            \"Phi-4-mini-instruct-Q4_K_M.gguf\",\n            \"microsoft/Phi-4-mini-instruct\",\n            format!(\"<|system|>You are a concise technical assistant.<|end|><|user|>{}<|end|><|assistant|>\", args.prompt),\n            \"<|end|>\",\n        ),\n        ModelFamily::Qwen3 => (\n            \"Qwen/Qwen3-4B-GGUF\",\n            \"qwen3-4b-q4_k_m.gguf\",\n            \"Qwen/Qwen3-4B\",\n            format!(\"<|im_start|>system\\nYou are a concise technical assistant.<|im_end|>\\n<|im_start|>user\\n{}<|im_end|>\\n<|im_start|>assistant\\n\", args.prompt),\n            \"<|im_end|>\",\n        ),\n        ModelFamily::Gemma4 => (\n            \"bartowski/gemma-4-E4B-it-GGUF\",\n            \"gemma-4-E4B-it-Q4_K_M.gguf\",\n            \"google/gemma-4-E4B-it\",\n            format!(\"<start_of_turn>user\\n{}<end_of_turn>\\n<start_of_turn>model\\n\", args.prompt),\n            \"<end_of_turn>\",\n        ),\n    };\n\n    println!(\"📥 Fetching {:?} weights & tokenizer...\", args.model);\n    let model_path = api.model(gguf_repo.to_string()).get(gguf_file)?;\n    let tokenizer_path = api.model(tok_repo.to_string()).get(\"tokenizer.json\")?;\n    let tokenizer = Tokenizer::from_file(tokenizer_path).map_err(E::msg)?;\n\n    let load_start = std::time::Instant::now();\n    let mut file = std::fs::File::open(&model_path)?;\n    let content = gguf_file::Content::read(&mut file)?;\n\n    let mut model = match args.model {\n        ModelFamily::Phi4 => LoadedModel::Phi(PhiStyleModel::from_gguf(false, content, &mut file, &device)?),\n        ModelFamily::Qwen3 => LoadedModel::Qwen(QwenStyleModel::from_gguf(content, &mut file, &device)?),\n        ModelFamily::Gemma4 => LoadedModel::Gemma(LlamaStyleModel::from_gguf(content, &mut file, &device)?),\n    };\n    println!(\"⚡ Cold-start model load completed in {:.2}s\", load_start.elapsed().as_secs_f64());\n\n    let tokens = tokenizer\n        .encode(formatted_prompt, true)\n        .map_err(E::msg)?\n        .get_ids()\n        .to_vec();\n\n    let mut logits_processor = LogitsProcessor::new(42, Some(args.temperature), None);\n    let gen_start = std::time::Instant::now();\n\n    // Prefill step: process the input prompt\n    let input = Tensor::new(tokens.as_slice(), &device)?.unsqueeze(0)?;\n    let logits = model.forward(&input, 0)?.squeeze(0)?;\n    let mut next_token = logits_processor.sample(&logits)?;\n\n    print!(\"\\n🤖 Response:\\n\");\n    if let Ok(word) = tokenizer.decode(&[next_token], true) {\n        print!(\"{word}\");\n        std::io::stdout().flush()?;\n    }\n\n    let eos_token = *tokenizer.get_vocab(true).get(eos_str).unwrap_or(&2);\n    let mut generated = 1;\n\n    // Decode step: autoregressive token streaming\n    for index in 0..args.max_tokens {\n        let input = Tensor::new(&[next_token], &device)?.unsqueeze(0)?;\n        let logits = model.forward(&input, tokens.len() + index)?.squeeze(0)?;\n        next_token = logits_processor.sample(&logits)?;\n\n        if next_token == eos_token {\n            break;\n        }\n\n        generated += 1;\n        if let Ok(word) = tokenizer.decode(&[next_token], true) {\n            print!(\"{word}\");\n            std::io::stdout().flush()?;\n        }\n    }\n\n    let elapsed = gen_start.elapsed().as_secs_f64();\n    println!(\n        \"\\n\\n📊 Benchmark: {} tokens in {:.2}s ({:.2} tokens/sec)\",\n        generated,\n        elapsed,\n        generated as f64 / elapsed\n    );\n\n    Ok(())\n}\n```\n\nWhen running matrix multiplications on a CPU, compiler flags make the difference between sluggish output and blazing-fast generation. Always compile with `--release` and instruct `rustc` to utilize native SIMD instructions (`AVX2` / `AVX-512` on x86_64 or `NEON` on ARM):\n\n```\n# Run Qwen 3 on CPU\nRUSTFLAGS=\"-C target-cpu=native\" cargo run --release -- --model qwen3\n\n# Run Gemma 4 on CPU\nRUSTFLAGS=\"-C target-cpu=native\" cargo run --release -- --model gemma4\n\n# Run Phi-4-mini on CPU\nRUSTFLAGS=\"-C target-cpu=native\" cargo run --release -- --model phi4\n```\n\nAll tests below were executed on a standard consumer CPU (no discrete GPU acceleration used) using `Q4_K_M` 4-bit quantized GGUF weights and compiled with `target-cpu=native`:\n\n| Metric | Phi-3-mini (3.8B) *(Baseline)* | Phi-4-mini (3.8B) | Qwen 3 / 3.6 (4B) | Gemma 4 (E4B) | \n|---|---|---|---|---|\n| **Compiled Binary Size** | ~18 MB | ~18 MB | ~18 MB | ~18 MB | \n| **GGUF Weight Size** | 2.39 GB | 2.49 GB | 2.55 GB | 2.62 GB | \n| **Peak RAM Usage** | ~2.7 GB | ~2.8 GB | ~2.9 GB | ~3.0 GB | \n| **Cold Start Latency** | 1.15 sec | 1.20 sec | 1.18 sec | 1.25 sec | \n| **Generation Speed (CPU)** | ~24 tok/s | ~23 tok/s | **~28 tok/s** | **~29 tok/s** | \n| **Best Use Case** | General Chat | Logic & Math | Coding & JSON | Edge & Multilingual | \n\n`LoadedModel` inside an `Arc<tokio::sync::Mutex<...>>` and serving it via Local AI is no longer limited to developers with expensive multi-GPU rigs. By combining modern 4-bit quantized SLMs (**Gemma 4, Qwen 3, and Phi-4**) with the zero-cost abstractions of **Rust and Candle**, we can deploy remarkably capable AI systems onto standard CPUs, edge servers, and embedded hardware.\n\nHave you benchmarked any of the new 2026 SLMs on CPU yet? Drop your thoughts or performance numbers in the comments!", "url": "https://wpnews.pro/news/running-modern-slms-gemma-4-qwen-3-phi-4-on-cpu-pure-rust-zero-python", "canonical_source": "https://dev.to/hossein-mortazavi/running-modern-slms-gemma-4-qwen-3-phi-4-on-cpu-pure-rust-zero-python-2od7", "published_at": "2026-10-25 07:02:53+00:00", "updated_at": "2026-10-25 07:13:07.397536+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools", "mlops"], "entities": ["Google", "Gemma 4", "Alibaba", "Qwen 3", "Microsoft", "Phi-4-mini", "Hugging Face", "Candle"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/running-modern-slms-gemma-4-qwen-3-phi-4-on-cpu-pure-rust-zero-python", "markdown": "https://wpnews.pro/news/running-modern-slms-gemma-4-qwen-3-phi-4-on-cpu-pure-rust-zero-python.md", "text": "https://wpnews.pro/news/running-modern-slms-gemma-4-qwen-3-phi-4-on-cpu-pure-rust-zero-python.txt", "jsonld": "https://wpnews.pro/news/running-modern-slms-gemma-4-qwen-3-phi-4-on-cpu-pure-rust-zero-python.jsonld"}}