Model: Kolibri-1 78B-A3.5B

Aleph Alpha released Kolibri-1 78B-A3.5B, a 78B-parameter mixture-of-experts model under the Apache 2.0 license that activates about 3.5B parameters per token across 384 experts per layer, using 6 experts plus one shared expert per token. Only 10 of the model's 50 layers grow a KV cache with context while the other 40 use a 513-token window, giving the model a 262K max context, and the weights ship in FP8 with GGUF quantizations from bartowski ranging from IQ1_M at 19.77 GB to Q8_0 at 83.14 GB across 8 variants. No quality loss has been published for any build, and the site's per-quant VRAM figures, from 20.9 GB for IQ1_M to 87.1 GB for Q8_0 at 4K context, are listed as estimated rather than measured.

Kolibri-1 78B-A3.5B 78B-A3.5B Aleph Alpha Kolibri Aleph Alpha's 78B mixture-of-experts model under Apache 2.0, reading about 3.5B parameters per token. It has 384 experts per layer, 6 used per token plus one shared expert. Only 10 of its 50 layers grow a cache with context; the other 40 use a 513-token window, so long context is cheap for its size. The weights ship in FP8; the sizes here come from bartowski's GGUF files. No quality loss has been published for any build. 262K Max Context 8 Quant Variants GGUF Q8 0 Best Quality — Perplexity rise best build Quantization Variants Per-quant VRAM, quality loss, and inference speed on RTX 4090 Measured = run on this site · Estimated = not run here: calculated, or a figure this site has not verified · Community = public reports | Format | Level | BPW | VRAM | PPL Loss | Speed | Source | Actions | |---|---|---|---|---|---|---|---| | GGUF | IQ1 M | 2.03 | 20.9 GB | — | — | Estimated | | | GGUF | IQ2 XXS | 2.33 | 24.0 GB | — | — | Estimated | | | GGUF | IQ2 XS | 2.49 | 25.6 GB | — | — | Estimated | | | GGUF | IQ3 XXS | 3.45 | 35.4 GB | — | — | Estimated | | | GGUF | Q3 K M | 3.85 | 39.5 GB | — | — | Estimated | | | GGUF | IQ4 XS | 4.47 | 45.8 GB | — | — | Estimated | | | GGUF | Q4 K M | 5.1 | 52.2 GB | — | — | Estimated | | | GGUF | Q8 0 | 8.52 | 87.1 GB | — | — | Estimated | | Which file each size is for A file size is not a memory figure: the VRAM column is the calculator's total at 4K context — weights plus KV cache plus a 10% activation buffer. Units differ on purpose: file sizes are decimal GB, as Hugging Face lists them; the VRAM column is GiB 1024³ bytes with 2% added to the weights, so 7.12 GB of file counts as about 6.8 GiB there. A vision projector mmproj , where the repo ships one, is a separate download and is in neither number. - GGUF IQ1 M bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · Kolibri-1-IQ1 M.gguf · 19.77 GB 19,774,226,304 bytes · listing read 2026-11-12 - GGUF IQ2 XXS bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · Kolibri-1-IQ2 XXS.gguf · 22.77 GB 22,774,013,824 bytes · listing read 2026-11-12 - GGUF IQ2 XS bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · Kolibri-1-IQ2 XS.gguf · 24.34 GB 24,339,996,544 bytes · listing read 2026-11-12 - GGUF IQ3 XXS bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · Kolibri-1-IQ3 XXS.gguf · 33.65 GB 33,653,829,504 bytes · listing read 2026-11-12 - GGUF Q3 K M bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · Kolibri-1-Q3 K M.gguf · 37.62 GB 37,620,641,664 bytes · listing read 2026-11-12 - GGUF IQ4 XS bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · Kolibri-1-IQ4 XS.gguf · 43.63 GB 43,632,914,304 bytes · listing read 2026-11-12 - GGUF Q4 K M bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · Kolibri-1-Q4 K M.gguf · 49.76 GB 49,759,219,584 bytes · listing read 2026-11-12 - GGUF Q8 0 bartowski/Kolibri-1-GGUF https://huggingface.co/bartowski/Kolibri-1-GGUF · file name not recorded · 83.14 GB 83,135,393,696 bytes · 3 parts · listing read 2026-11-12 Similar models Compare with Nemotron 3.5 https://quantized.uk/tools/compare/?a=kolibri-1-78b-a3-5b&b=nemotron-3.5-bolt-74b-a9b 74B-A9BConsumer GPUMac / Apple SiliconIQ2 XXS23.1 GB—perplexity rise https://quantized.uk/quant-hub/nemotron-3.5-bolt-74b-a9b/ Nemotron 3.5 Bolt 74B-A9B NVIDIA Nemotron NVIDIA's Nemotron 3.5 Bolt is a 74B mixture-of-experts model with about 9B parameters active per token, built to fit a 24 GB card. It is compressed from Nemotron 3.5 Super VL and ships in one build, NVIDIA's own native 2-bit file of 22.0 GB. Only 8 of its 88 layers keep a growing cache, so context costs little: about 23.1 GB at 4K, 24.1 GB at 128K and 25.2 GB at its full 256K window. That is the 24 GB NVIDIA designed for, but over this site's comfortable line, so a 24 GB card reads as tight; a 32 GB card or a 48 GB Mac runs it comfortably. The image encoder is a separate 1.7 GB file. NVIDIA's benchmark claims are its own, and no quality loss for the 2-bit build has been published. 70BPro GPUMac / Apple SiliconQ4 K M46.1 GB+2.8%perplexity rise https://quantized.uk/quant-hub/llama-3.1-70b/ Llama 3.1 70B Instruct Meta Llama 3.1 Meta's frontier 70B model. Requires 40GB+ VRAM; dual 3090 or M2 Ultra. 70BPro GPUMac / Apple SiliconQ4 K M46.1 GB+2.6%perplexity rise https://quantized.uk/quant-hub/llama-3.3-70b/ Llama 3.3 70B Instruct Meta Llama 3.3 Latest Meta 70B with improved multilingual. Drop-in upgrade from Llama 3.1 70B. 52B-A12BPro GPUMac / Apple SiliconQ4 K M33.0 GB+3.4%perplexity rise https://quantized.uk/quant-hub/jamba-1.5-mini/ Jamba 1.5 Mini AI21 Labs AI21's hybrid Mamba-Transformer MoE: 52B total, about 12B active, 256K context. Only 4 of its 32 layers keep a KV cache, so long context is cheap — but all 52B of weights still have to be resident, about 31 GB at Q4. That is a 32 GB card or a large Mac, not the 16 GB card this entry used to claim. Sizes use the calculator's generic rates; no speed has been measured here. Running Kolibri-1 78B-A3.5B locally At Q4 K M and 4K of context, Kolibri-1 78B-A3.5B needs about 52.2 GB — 47.3 GB of weights, 0.18 GB of KV cache and a 4.8 GB activation buffer. The smallest card in this index that clears that comfortably is the DGX Spark 64G at 64 GB, and 12 of the 78 cards here do. These are calculated figures, not measurements: the estimate stops counting a card as comfortable at 88% of its VRAM, which is roughly the room a desktop session needs. What longer context costs Going from 4K to 32K adds about 0.54 GB, taking the total to 52.8 GB. That is gentler than the parameter count suggests: only 10 of 50 layers keep a KV cache that grows with context, and 40 more cap theirs at a sliding window, so the cache scales at a fraction of the usual rate. The model's native window is 256K; holding all of it at Q4 K M would need about 58 GB. Which build to download This index only tracks Kolibri-1 78B-A3.5B in GGUF, across 8 levels IQ1 M, IQ2 XXS, IQ2 XS, IQ3 XXS, Q3 K M, IQ4 XS, Q4 K M, Q8 0 . No per-level perplexity sweep has been published for this model, so the quality column is empty rather than estimated — within one model, more bits per weight is the only ordering the data supports. No throughput has been measured for this model on this site's hardware. GGUF runs on llama.cpp and Ollama across NVIDIA, AMD and Apple silicon; AWQ and GPTQ target vLLM on CUDA and ROCm; EXL2 is ExLlamaV2 — CUDA only, and now archived, so local use rather than serving. Common questions - How much VRAM does Kolibri-1 78B-A3.5B need? - About 52.2 GB at Q4 K M with 4K of context and batch 1, rising to roughly 52.8 GB at 32K. That figure is weights plus KV cache plus a 10% activation buffer, calculated from the model's architecture rather than measured on a card. - Will Kolibri-1 78B-A3.5B run on a 64GB GPU? - Yes — at Q4 K M and 4K context it needs about 52.2 GB, which leaves 11.8 GB spare on a DGX Spark 64G. That is the smallest card in this index that clears it comfortably; 12 of 78 do. - Which quantization of Kolibri-1 78B-A3.5B should I use? - No published quality comparison exists for this model, so pick by footprint: Q4 K M is the level most people run, and a higher bits-per-weight level is more faithful. All 8 levels in the index are IQ1 M, IQ2 XXS, IQ2 XS, IQ3 XXS, Q3 K M, IQ4 XS, Q4 K M, Q8 0. Note this is a mixture-of-experts model — all parameters must be resident even though only a fraction are active per token, so the memory cost follows the total, not the active count. Where this model fits Sized at Q4 K M with a 4K context window, smallest card first. Comfortable means the estimate uses at most 88% of the memory. - Runs comfortably on - DGX Spark 64G64GB · +11.8 https://quantized.uk/gpu/dgx-spark-64g/ A100 80G80GB · +27.8 https://quantized.uk/gpu/a100-80g/ H100 80G80GB · +27.8 https://quantized.uk/gpu/h100-80g/ Mac M2 Max 96G96GB · +19.8 https://quantized.uk/gpu/mac-m2-max-96g/ DGX Spark 128G128GB · +75.8 https://quantized.uk/gpu/dgx-spark-128g/ - Just misses - A40 48G+4.2GB over https://quantized.uk/gpu/a40-48g/ L40S 48G+4.2GB over https://quantized.uk/gpu/l40s-48g/ - Ships in - GGUF https://quantized.uk/formats/gguf/ - Try it yourself - Size it in the calculator https://quantized.uk/tools/vram-calc/?model=kolibri-1-78b-a3-5b&quant=Q4 K M Generate the run command https://quantized.uk/tools/cli-gen/?model=kolibri-1-78b-a3-5b&quant=Q4 K M How to actually run this Deployment guides for this model and this class of hardware. This page's figures change when the model or the runtime does.Last updated 2026-11-12 RSS → /feed.xml https://quantized.uk/feed.xml