{"slug": "gemma-4-e2b-inference-in-700-lines-of-c", "title": "Gemma 4 E2B inference in 700 lines of C", "summary": "A new educational project, gemma4.c, implements Gemma 4 E2B CPU inference in 700 lines of pure C, achieving 638.86 tok/s prefill and 25.90 tok/s decode on an AMD Ryzen 7 7700, outperforming llama.cpp's Q8_0 build (276.32 tok/s prefill, 23.83 tok/s decode). The project, by developer ryanssenn, validates against Google's unquantized Gemma 4 E2B QAT checkpoint with 96.5% top-1 agreement and a mean KL divergence of 0.005207.", "body_md": "Gemma 4 E2B CPU inference in 700 lines of pure C.\n\nAn educational project made to understand how LLM inference works. The full inference path is implemented in one file without external libraries.\n\nMeasured on an AMD Ryzen 7 7700 using the default native build.\n\n| Implementation | Prefill (pp512) | Decode (tg128) |\n|---|---|---|\n| gemma4.c (int8) | 638.86 ± 2.93 tok/s | 25.90 ± 0.01 tok/s |\n| llama.cpp (Q8_0) | 276.32 ± 1.85 tok/s | 23.83 ± 0.01 tok/s |\n\nResults are the mean ± sample standard deviation over 12 timed runs after one discarded warmup, with each runtime using its fastest tested thread count. Both were built natively for CPU and dynamically quantize matrix inputs to int8.\n\n```\n./run -m ./gemma4-E2B-int8.bin --bench 512 128\n```\n\nYou need a CPU with AVX2, an OpenMP-capable C compiler, and `make`\n\n. The model takes about 5.0 GB of disk space, and 8 GB of RAM is recommended.\n\nClone the repository and download the ready-to-run model:\n\n```\ngit clone https://github.com/ryanssenn/gemma4.c\ncd gemma4.c\npython3 -m pip install -U huggingface_hub\nhf download QmogAI/gemma4-e2b-int8 gemma4-E2B-int8.bin --local-dir .\n```\n\nOn Linux:\n\n```\nmake\n./run -t 0 -n 256 \"Why is the sky blue?\"\n```\n\nOn Windows, use a MinGW-w64 environment that provides `gcc`\n\n, OpenMP, and `make`\n\n:\n\n```\nmake win64 WINCC=gcc\n.\\run.exe -t 0 -n 256 \"Why is the sky blue?\"\n```\n\n`-m`\n\nsets the model path. The default is`gemma4-E2B-int8.bin`\n\n.`-t`\n\nsets the temperature. The default is`1.0`\n\n. Use`0`\n\nfor greedy decoding.`-n`\n\nsets the maximum number of tokens to generate. The default is`1,024`\n\n.`--bench`\n\nmeasures prefill and decode throughput.`--dump-logits`\n\nwrites prompt logits as float32 binary data.\n\nThe C runtime cannot read the original checkpoint directly. `exporter.py`\n\ntakes the tokenizer and language-model weights from the Hugging Face checkpoint and writes them in the exact layout used by `gemma4.c`\n\n.\n\nMatrix weights are stored as int8 with FP16 scales. Inputs to linear layers are dynamically quantized to int8 while the rest of the activations remain float32. The resulting file is about 5.0 GB (4.7 GiB).\n\nTo create it yourself instead:\n\n```\npython3 -m pip install -r requirements.txt\npython3 exporter.py /path/to/gemma-4-E2B-it-qat-q4_0-unquantized -o ./gemma4-E2B-int8.bin\n```\n\nPython is only needed to export the model or run numerical validation. Once the `.bin`\n\nfile exists, inference runs entirely through the C program.\n\nThe implementation is validated against the Hugging Face Transformers reference implementation running Google's unquantized [Gemma 4 E2B QAT checkpoint](https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-unquantized) in BF16. Both implementations process the complete Éva Gauthier article from the WikiText-103 validation split under teacher forcing. The passage contains 2,387 text tokens, and the added BOS token brings the comparison to 2,388 model positions.\n\n| Metric | Result |\n|---|---|\n| Top-1 agreement | 2,305 / 2,388 (96.5%) |\n| Mean KL divergence | 0.005207 |\n\nAn exact match is not expected because gemma4.c uses int8 matrix weights and linear inputs while the reference runs the unquantized checkpoint in BF16. The output distributions nevertheless remain closely aligned.\n\nBuild the C runtime first (`make`\n\nor `make win64`\n\n). The complete validation peaks at about 10 GiB of RAM:\n\n```\npython3 validation.py\n```\n\nPass a smaller token count for a quicker check:\n\n```\npython3 validation.py 64\n```\n\n`validation.py`\n\nuses the runtime's `--dump-logits`\n\nflag to collect float32 logits after each prompt position. The flag can also be used directly when comparing gemma4.c with another implementation:\n\n```\n./run -m ./gemma4-E2B-int8.bin --dump-logits \"Why is the sky blue?\" > logits.bin\n```\n\n`gemma4.c`\n\ncontains the tokenizer, model definitions, kernels, transformer, KV cache, and generation loop.`exporter.py`\n\nconverts the original checkpoint into the binary layout read by the C runtime.`validation.py`\n\ncompares the runtime's logits with Hugging Face Transformers.`validation.txt`\n\ncontains the WikiText-103 passage used for numerical validation.`win.c`\n\nand`win.h`\n\nprovide the small Windows memory-mapping compatibility layer.`Makefile`\n\nbuilds the Linux or Windows executable.", "url": "https://wpnews.pro/news/gemma-4-e2b-inference-in-700-lines-of-c", "canonical_source": "https://github.com/ryanssenn/gemma4.c", "published_at": "2026-08-27 17:30:42+00:00", "updated_at": "2026-08-27 17:48:47.548572+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["Gemma 4 E2B", "gemma4.c", "llama.cpp", "AMD Ryzen 7 7700", "Google", "Hugging Face", "ryanssenn"], "alternates": {"html": "https://wpnews.pro/news/gemma-4-e2b-inference-in-700-lines-of-c", "markdown": "https://wpnews.pro/news/gemma-4-e2b-inference-in-700-lines-of-c.md", "text": "https://wpnews.pro/news/gemma-4-e2b-inference-in-700-lines-of-c.txt", "jsonld": "https://wpnews.pro/news/gemma-4-e2b-inference-in-700-lines-of-c.jsonld"}}