Originally published on tamiz.pro.
Large language models like Gemma 4 26B typically require powerful GPUs with high VRAM. This tutorial demonstrates how to run the model on a 13-year-old Xeon processor (e.g., Intel Xeon E5 v2 series) using CPU-only optimization techniques like model quantization and memory-efficient execution.
git
, cmake
, and gcc
installedStart by installing core dependencies:
sudo apt-get update
sudo apt-get install -y python3-pip build-essential
pip install torch==2.1.0 transformers optimum
Verify PyTorch's CPU support with:
import torch
print(torch.__version__, torch.cuda.is_available()) # Should return False
Use Hugging Face's from_pretrained
with quantization:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "google/gemma-4-26b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
load_in_4bit=True,
torch_dtype=torch.float16
)
This reduces RAM usage from 120GB (float16) to ~40GB through quantization.
Add CPU-specific optimizations:
from torch._dynamo import optimize_for_cpu
model = optimize_for_cpu(model)
model.tie_weights()
import torch.nn as nn
nn.Linear(model.config.hidden_size, model.config.hidden_size).to(memory_format=torch.channels_last)
Execute with batch size 1 and CPU-optimized pipeline:
input_text = "Explain quantum computing in simple terms"
inputs = tokenizer(input_text, return_tensors="pt")
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
| Metric | Result (Xeon E5 v2) |
|---|---|
| RAM Usage | ~45GB |
| Tokens/Second | ~12 tokens/sec |
| Cold Start Time | 3-5 minutes |
| Power Consumption | ~150W |
load_in_8bit
instead of load_in_4bit
if RAM is constrained--cpu-inference
flag in any training scripts
export MKL_THREADING_LAYER=GNU
export MKL_SERVICE_FORCE_INTEL=1
While modern GPUs provide better throughput (100-300 tokens/sec), this CPU-only approach enables AI inference on legacy hardware at ~15% of GPU costs. Ideal for edge deployments or proof-of-concept work. Consider upgrading to Xeon Scalable (2nd Gen) for production workloads requiring higher throughput.