Red Hat's 4-bit NVFP4 build of Gemma 4 26B-A4B cuts memory needs by about 75%. It keeps 97 to 99% of the original model's accuracy on most benchmarks, but agentic tool use and hard math take bigger hits.
- Red Hat released Gemma 4 26B-A4B NVFP4 , a 4-bit build that needs about 75% less GPU memory.
- Most benchmarks keep 97 to 99% of the original's accuracy; IFEval with thinking on keeps 99.7%.
- The weak spots are agentic tool calling (80.9% recovery) and AIME 2025 math (82.8% without thinking).
- The router, vision tower, and embeddings stay at full precision; the recipe uses LLM Compressor .
- Free under Apache 2.0 and served with vLLM; native FP4 speed needs NVIDIA Blackwell GPUs.
- Early users hit vLLM bugs for the MoE expert weights; later reports show about 52 tok/s on DGX Spark.
Red Hat AI has released a 4-bit checkpoint of Google’s mixture-of-experts Gemma 4 model. The NVFP4 build compresses 25.2 billion parameters enough to run on a single 32 GB Blackwell GPU, with benchmark losses ranging from negligible on instruction following to substantial on agentic tool use.
Hugging Face displayed 698,393 monthly downloads at publication time, placing the checkpoint among the platform’s most-pulled Gemma 4 variants. Its appeal comes from combining multimodal input, tool calling, long context and reasoning controls in a model that no longer requires a data-center GPU for inference.
- Parameters: 25.2B total, with 3.8B active for each token
- Context: Up to 256K tokens, subject to available KV-cache memory
- Quantization: NVFP4 weights and activations with groups of 16 values
- Estimated memory: About 14.4 GB for 4-bit model weights
- Best hardware: NVIDIA Blackwell GPUs with native FP4 Tensor Cores
- Main trade-off: Larger benchmark losses on competition math and multi-step agent tasks
Only 3.8B parameters compute each token #
Google’s base checkpoint uses a mixture-of-experts architecture. A learned router selects the expert subnetworks that process each token, activating 3.8B of the model’s 25.2B parameters during inference. The model also supports a 256K-token context window, multimodal input, function calling, structured output and configurable reasoning.
Every expert’s weights must remain available in memory so the router can dispatch tokens without repeatedly data. Google’s memory table estimates that the model consumes about 57.7 GB at BF16 precision, beyond the capacity of common desktop GPUs. Quantization reduces that resident weight memory by representing values with fewer bits.
NVFP4 shrinks the resident weights #
NVFP4 is NVIDIA’s 4-bit floating-point format for Blackwell hardware. Each 4-bit field has 16 possible bit patterns, so the format uses shared scaling factors to preserve a wider numerical range. NVFP4 groups 16 values under an E4M3 FP8 scale and can apply an additional FP32 tensor scale. The smaller groups limit how far an unusually large value can distort neighboring values.
This story is for Pro members #
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.