Red Hat Shrinks Gemma 4 to Fit on a Single 32GB Blackwell GPU Red Hat AI released Gemma 4 26B-A4B NVFP4, a 4-bit checkpoint of Google's mixture-of-experts Gemma 4 that cuts resident weight memory by about 75% — from roughly 57.7 GB at BF16 to about 14.4 GB — so the 25.2B-parameter model (3.8B active per token) runs on a single 32 GB NVIDIA Blackwell GPU. The build retains 97 to 99% of the original's accuracy on most benchmarks, including 99.7% on IFEval with thinking on, but drops to 80.9% recovery on agentic tool calling and 82.8% on AIME 2025 math without thinking. The checkpoint is free under Apache 2.0, served with vLLM, and had 698,393 monthly downloads on Hugging Face at publication time, though early users hit vLLM loading bugs for the MoE expert weights. Red Hat Shrinks Gemma 4 to Fit on a Single 32GB Blackwell GPU Red Hat's 4-bit NVFP4 build of Gemma 4 26B-A4B cuts memory needs by about 75%. It keeps 97 to 99% of the original model's accuracy on most benchmarks, but agentic tool use and hard math take bigger hits. - Red Hat released Gemma 4 26B-A4B NVFP4 https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4?utm source=alphasignal , a 4-bit build that needs about 75% less GPU memory. - Most benchmarks keep 97 to 99% of the original's accuracy; IFEval with thinking on keeps 99.7%. - The weak spots are agentic tool calling 80.9% recovery and AIME 2025 math 82.8% without thinking . - The router, vision tower, and embeddings stay at full precision; the recipe uses LLM Compressor https://github.com/vllm-project/llm-compressor?utm source=alphasignal . - Free under Apache 2.0 and served with vLLM; native FP4 speed needs NVIDIA Blackwell GPUs. - Early users hit vLLM loading bugs for the MoE expert weights; later reports show about 52 tok/s on DGX Spark. Red Hat AI has released a 4-bit checkpoint https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4?utm source=alphasignal of Google’s mixture-of-experts Gemma 4 model. The NVFP4 build compresses 25.2 billion parameters enough to run on a single 32 GB Blackwell GPU, with benchmark losses ranging from negligible on instruction following to substantial on agentic tool use. Hugging Face displayed 698,393 monthly downloads https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4?utm source=alphasignal at publication time, placing the checkpoint among the platform’s most-pulled Gemma 4 variants. Its appeal comes from combining multimodal input, tool calling, long context and reasoning controls in a model that no longer requires a data-center GPU for inference. - Parameters: 25.2B total, with 3.8B active for each token - Context: Up to 256K tokens, subject to available KV-cache memory - Quantization: NVFP4 weights and activations with groups of 16 values - Estimated memory: About 14.4 GB for 4-bit model weights - Best hardware: NVIDIA Blackwell GPUs with native FP4 Tensor Cores - Main trade-off: Larger benchmark losses on competition math and multi-step agent tasks Only 3.8B parameters compute each token Google’s base checkpoint https://huggingface.co/google/gemma-4-26B-A4B-it?utm source=alphasignal uses a mixture-of-experts architecture. A learned router selects the expert subnetworks that process each token, activating 3.8B of the model’s 25.2B parameters during inference. The model also supports a 256K-token context window, multimodal input, function calling, structured output and configurable reasoning. Every expert’s weights must remain available in memory so the router can dispatch tokens without repeatedly loading data. Google’s memory table https://ai.google.dev/gemma/docs/core?utm source=alphasignal estimates that the model consumes about 57.7 GB at BF16 precision, beyond the capacity of common desktop GPUs. Quantization reduces that resident weight memory by representing values with fewer bits. NVFP4 shrinks the resident weights NVFP4 is NVIDIA’s 4-bit floating-point format for Blackwell hardware. Each 4-bit field has 16 possible bit patterns, so the format uses shared scaling factors to preserve a wider numerical range. NVFP4 groups 16 values under an E4M3 FP8 scale and can apply an additional FP32 tensor scale. The smaller groups limit how far an unusually large value can distort neighboring values. This story is for Pro members You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.