Running Modern SLMs (Gemma 4, Qwen 3 & Phi-4) on CPU: Pure Rust, Zero Python

When I first shared my experiment on running Microsoft’s Phi-3 on a standard consumer CPU using pure Rust and Hugging Face’s Candle framework, the mission was straightforward: prove that we don’t need bloated Python runtimes, complex CUDA setups, or 5GB+ Docker images just to run local AI inference.

Fast forward to today, and the Small Language Model (SLM) landscape has exploded. With the release of edge-optimized powerhouses like Google’s Gemma 4 (E4B), Alibaba’s Qwen 3 / 3.6 (4B), and Microsoft’s Phi-4-mini (3.8B), running high-reasoning AI directly on standard CPU hardware is no longer just a weekend hack—it is a viable production architecture.

In this guide, we will upgrade our pure-Rust inference pipeline to build a multi-model CLI engine capable of running the latest quantized GGUF models on a standard CPU, complete with real-world benchmarks on RAM usage, cold-start latency, and token generation speeds.

Why go through the trouble of writing inference code in Rust when Python has transformers and vLLM? When deploying to edge devices, IoT gateways, or cost-effective CPU cloud instances, runtime overhead matters.

Instead of locking our code to a single model, we are testing three of the strongest compact models currently dominating local AI:

Let’s write a unified Rust CLI tool that lets you switch between Phi-4, Qwen, and Gemma models straight from the terminal, leveraging candle-transformers and hf-hub.

Cargo.toml) Create a new project (cargo new rust-slm-bench) and configure your Cargo.toml:

[package]
name = "rust-slm-bench"
version = "0.2.0"
edition = "2021"

[dependencies]
anyhow = "1.0"
candle-core = "0.8"
candle-nn = "0.8"
candle-transformers = "0.8"
hf-hub = { version = "0.3", features = ["tokio"] }
tokenizers = "0.20"
clap = { version = "4.5", features = ["derive", "ValueEnum"] }

src/main.rs) Below is the clean, streaming implementation. We define an enum for our model families so our engine automatically selects the right Hugging Face repository, GGUF file, and Chat Template (ChatML vs. Gemma vs. Phi):

use anyhow::{Error as E, Result};
use candle_core::quantized::gguf_file;
use candle_core::{Device, Tensor};
use candle_transformers::generation::LogitsProcessor;
use candle_transformers::models::quantized_llama::ModelWeights as LlamaStyleModel;
use candle_transformers::models::quantized_phi3::ModelWeights as PhiStyleModel;
use candle_transformers::models::quantized_qwen2::ModelWeights as QwenStyleModel;
use clap::{Parser, ValueEnum};
use hf_hub::api::sync::Api;
use std::io::Write;
use tokenizers::Tokenizer;

#[derive(Debug, Clone, ValueEnum)]
enum ModelFamily {
    Phi4,
    Qwen3,
    Gemma4,
}

enum LoadedModel {
    Phi(PhiStyleModel),
    Qwen(QwenStyleModel),
    Gemma(LlamaStyleModel),
}

impl LoadedModel {
    fn forward(&mut self, x: &Tensor, index_pos: usize) -> candle_core::Result<Tensor> {
        match self {
            Self::Phi(m) => m.forward(x, index_pos),
            Self::Qwen(m) => m.forward(x, index_pos),
            Self::Gemma(m) => m.forward(x, index_pos),
        }
    }
}

#[derive(Parser, Debug)]
#[command(author, version, about = "Benchmark Modern SLMs on CPU with Rust & Candle")]
struct Args {
    /// Select which SLM family to run
    #[arg(short, long, value_enum, default_value_t = ModelFamily::Qwen3)]
    model: ModelFamily,

    /// The prompt to send to the model
    #[arg(short, long, default_value = "Explain why Rust is great for Edge AI in 3 short bullet points.")]
    prompt: String,

    /// Maximum tokens to generate
    #[arg(short = 'n', long, default_value_t = 200)]
    max_tokens: usize,

    /// Sampling temperature
    #[arg(short, long, default_value_t = 0.6)]
    temperature: f64,
}

fn main() -> Result<()> {
    let args = Args::parse();
    let device = Device::Cpu;
    let api = Api::new()?;

    // Map each model family to its GGUF repo, filename, tokenizer, and prompt format
    let (gguf_repo, gguf_file, tok_repo, formatted_prompt, eos_str) = match args.model {
        ModelFamily::Phi4 => (
            "bartowski/Phi-4-mini-instruct-GGUF",
            "Phi-4-mini-instruct-Q4_K_M.gguf",
            "microsoft/Phi-4-mini-instruct",
            format!("<|system|>You are a concise technical assistant.<|end|><|user|>{}<|end|><|assistant|>", args.prompt),
            "<|end|>",
        ),
        ModelFamily::Qwen3 => (
            "Qwen/Qwen3-4B-GGUF",
            "qwen3-4b-q4_k_m.gguf",
            "Qwen/Qwen3-4B",
            format!("<|im_start|>system\nYou are a concise technical assistant.<|im_end|>\n<|im_start|>user\n{}<|im_end|>\n<|im_start|>assistant\n", args.prompt),
            "<|im_end|>",
        ),
        ModelFamily::Gemma4 => (
            "bartowski/gemma-4-E4B-it-GGUF",
            "gemma-4-E4B-it-Q4_K_M.gguf",
            "google/gemma-4-E4B-it",
            format!("<start_of_turn>user\n{}<end_of_turn>\n<start_of_turn>model\n", args.prompt),
            "<end_of_turn>",
        ),
    };

    println!("📥 Fetching {:?} weights & tokenizer...", args.model);
    let model_path = api.model(gguf_repo.to_string()).get(gguf_file)?;
    let tokenizer_path = api.model(tok_repo.to_string()).get("tokenizer.json")?;
    let tokenizer = Tokenizer::from_file(tokenizer_path).map_err(E::msg)?;

    let load_start = std::time::Instant::now();
    let mut file = std::fs::File::open(&model_path)?;
    let content = gguf_file::Content::read(&mut file)?;

    let mut model = match args.model {
        ModelFamily::Phi4 => LoadedModel::Phi(PhiStyleModel::from_gguf(false, content, &mut file, &device)?),
        ModelFamily::Qwen3 => LoadedModel::Qwen(QwenStyleModel::from_gguf(content, &mut file, &device)?),
        ModelFamily::Gemma4 => LoadedModel::Gemma(LlamaStyleModel::from_gguf(content, &mut file, &device)?),
    };
    println!("⚡ Cold-start model load completed in {:.2}s", load_start.elapsed().as_secs_f64());

    let tokens = tokenizer
        .encode(formatted_prompt, true)
        .map_err(E::msg)?
        .get_ids()
        .to_vec();

    let mut logits_processor = LogitsProcessor::new(42, Some(args.temperature), None);
    let gen_start = std::time::Instant::now();

    // Prefill step: process the input prompt
    let input = Tensor::new(tokens.as_slice(), &device)?.unsqueeze(0)?;
    let logits = model.forward(&input, 0)?.squeeze(0)?;
    let mut next_token = logits_processor.sample(&logits)?;

    print!("\n🤖 Response:\n");
    if let Ok(word) = tokenizer.decode(&[next_token], true) {
        print!("{word}");
        std::io::stdout().flush()?;
    }

    let eos_token = *tokenizer.get_vocab(true).get(eos_str).unwrap_or(&2);
    let mut generated = 1;

    // Decode step: autoregressive token streaming
    for index in 0..args.max_tokens {
        let input = Tensor::new(&[next_token], &device)?.unsqueeze(0)?;
        let logits = model.forward(&input, tokens.len() + index)?.squeeze(0)?;
        next_token = logits_processor.sample(&logits)?;

        if next_token == eos_token {
            break;
        }

        generated += 1;
        if let Ok(word) = tokenizer.decode(&[next_token], true) {
            print!("{word}");
            std::io::stdout().flush()?;
        }
    }

    let elapsed = gen_start.elapsed().as_secs_f64();
    println!(
        "\n\n📊 Benchmark: {} tokens in {:.2}s ({:.2} tokens/sec)",
        generated,
        elapsed,
        generated as f64 / elapsed
    );

    Ok(())
}

When running matrix multiplications on a CPU, compiler flags make the difference between sluggish output and blazing-fast generation. Always compile with --release and instruct rustc to utilize native SIMD instructions (AVX2 / AVX-512 on x86_64 or NEON on ARM):

RUSTFLAGS="-C target-cpu=native" cargo run --release -- --model qwen3

RUSTFLAGS="-C target-cpu=native" cargo run --release -- --model gemma4

RUSTFLAGS="-C target-cpu=native" cargo run --release -- --model phi4

All tests below were executed on a standard consumer CPU (no discrete GPU acceleration used) using Q4_K_M 4-bit quantized GGUF weights and compiled with target-cpu=native:

LoadedModel inside an Arc<tokio::sync::Mutex<...>> and serving it via Local AI is no longer limited to developers with expensive multi-GPU rigs. By combining modern 4-bit quantized SLMs (Gemma 4, Qwen 3, and Phi-4) with the zero-cost abstractions of Rust and Candle, we can deploy remarkably capable AI systems onto standard CPUs, edge servers, and embedded hardware.

Have you benchmarked any of the new 2026 SLMs on CPU yet? Drop your thoughts or performance numbers in the comments!