Model: Kolibri-1 78B-A3.5B

wpnews.pro

78B-A3.5B Aleph Alpha Kolibri

Aleph Alpha's 78B mixture-of-experts model under Apache 2.0, reading about 3.5B parameters per token. It has 384 experts per layer, 6 used per token plus one shared expert. Only 10 of its 50 layers grow a cache with context; the other 40 use a 513-token window, so long context is cheap for its size. The weights ship in FP8; the sizes here come from bartowski's GGUF files. No quality loss has been published for any build.

262K

Max Context

8

Quant Variants

GGUF Q8_0

Best Quality

—

Perplexity rise (best build)

Quantization Variants #

Per-quant VRAM, quality loss, and inference speed on RTX 4090

Measured = run on this site · Estimated = not run here: calculated, or a figure this site has not verified · Community = public reports

Format	Level	BPW	VRAM	PPL Loss	Speed	Source
GGUF	IQ1_M	2.03	20.9 GB	—	—	Estimated
GGUF	IQ2_XXS	2.33	24.0 GB	—	—	Estimated
GGUF	IQ2_XS	2.49	25.6 GB	—	—	Estimated
GGUF	IQ3_XXS	3.45	35.4 GB	—	—	Estimated
GGUF	Q3_K_M	3.85	39.5 GB	—	—	Estimated
GGUF	IQ4_XS	4.47	45.8 GB	—	—	Estimated
GGUF	Q4_K_M	5.1	52.2 GB	—	—	Estimated
GGUF	Q8_0	8.52	87.1 GB	—	—	Estimated

Which file each size is for

A file size is not a memory figure: the VRAM column is the calculator's total at 4K context — weights plus KV cache plus a 10% activation buffer. Units differ on purpose: file sizes are decimal GB, as Hugging Face lists them; the VRAM column is GiB (1024³ bytes) with 2% added to the weights, so 7.12 GB of file counts as about 6.8 GiB there. A vision projector (mmproj), where the repo ships one, is a separate download and is in neither number.

- GGUF IQ1_M[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · Kolibri-1-IQ1_M.gguf · 19.77 GB (19,774,226,304 bytes) · listing read 2026-11-12
- GGUF IQ2_XXS[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · Kolibri-1-IQ2_XXS.gguf · 22.77 GB (22,774,013,824 bytes) · listing read 2026-11-12
- GGUF IQ2_XS[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · Kolibri-1-IQ2_XS.gguf · 24.34 GB (24,339,996,544 bytes) · listing read 2026-11-12
- GGUF IQ3_XXS[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · Kolibri-1-IQ3_XXS.gguf · 33.65 GB (33,653,829,504 bytes) · listing read 2026-11-12
- GGUF Q3_K_M[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · Kolibri-1-Q3_K_M.gguf · 37.62 GB (37,620,641,664 bytes) · listing read 2026-11-12
- GGUF IQ4_XS[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · Kolibri-1-IQ4_XS.gguf · 43.63 GB (43,632,914,304 bytes) · listing read 2026-11-12
- GGUF Q4_K_M[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · Kolibri-1-Q4_K_M.gguf · 49.76 GB (49,759,219,584 bytes) · listing read 2026-11-12
- GGUF Q8_0[bartowski/Kolibri-1-GGUF](https://huggingface.co/bartowski/Kolibri-1-GGUF) · file name not recorded · 83.14 GB (83,135,393,696 bytes) · 3 parts · listing read 2026-11-12

Similar models #

[Compare with Nemotron 3.5](https://quantized.uk/tools/compare/?a=kolibri-1-78b-a3-5b&b=nemotron-3.5-bolt-74b-a9b)

[74B-A9BConsumer GPUMac / Apple SiliconIQ2_XXS23.1 GB—perplexity rise](https://quantized.uk/quant-hub/nemotron-3.5-bolt-74b-a9b/)

Nemotron 3.5 Bolt 74B-A9B

NVIDIA Nemotron

NVIDIA's Nemotron 3.5 Bolt is a 74B mixture-of-experts model with about 9B parameters active per token, built to fit a 24 GB card. It is compressed from Nemotron 3.5 Super VL and ships in one build, NVIDIA's own native 2-bit file of 22.0 GB. Only 8 of its 88 layers keep a growing cache, so context costs little: about 23.1 GB at 4K, 24.1 GB at 128K and 25.2 GB at its full 256K window. That is the 24 GB NVIDIA designed for, but over this site's comfortable line, so a 24 GB card reads as tight; a 32 GB card or a 48 GB Mac runs it comfortably. The image encoder is a separate 1.7 GB file. NVIDIA's benchmark claims are its own, and no quality loss for the 2-bit build has been published.

70BPro GPUMac / Apple SiliconQ4_K_M46.1 GB+2.8%perplexity rise

Llama 3.1 70B Instruct

Meta Llama 3.1

Meta's frontier 70B model. Requires 40GB+ VRAM; dual 3090 or M2 Ultra.

70BPro GPUMac / Apple SiliconQ4_K_M46.1 GB+2.6%perplexity rise

Llama 3.3 70B Instruct

Meta Llama 3.3

Latest Meta 70B with improved multilingual. Drop-in upgrade from Llama 3.1 70B.

52B-A12BPro GPUMac / Apple SiliconQ4_K_M33.0 GB+3.4%perplexity rise

Jamba 1.5 Mini

AI21 Labs

AI21's hybrid Mamba-Transformer MoE: 52B total, about 12B active, 256K context. Only 4 of its 32 layers keep a KV cache, so long context is cheap — but all 52B of weights still have to be resident, about 31 GB at Q4. That is a 32 GB card or a large Mac, not the 16 GB card this entry used to claim. Sizes use the calculator's generic rates; no speed has been measured here.

Running Kolibri-1 78B-A3.5B locally #

At Q4_K_M and 4K of context, Kolibri-1 78B-A3.5B needs about 52.2 GB — 47.3 GB of weights, 0.18 GB of KV cache and a 4.8 GB activation buffer. The smallest card in this index that clears that comfortably is the DGX Spark 64G at 64 GB, and 12 of the 78 cards here do. These are calculated figures, not measurements: the estimate stops counting a card as comfortable at 88% of its VRAM, which is roughly the room a desktop session needs.

What longer context costs #

Going from 4K to 32K adds about 0.54 GB, taking the total to 52.8 GB. That is gentler than the parameter count suggests: only 10 of 50 layers keep a KV cache that grows with context, and 40 more cap theirs at a sliding window, so the cache scales at a fraction of the usual rate. The model's native window is 256K; holding all of it at Q4_K_M would need about 58 GB.

Which build to download #

This index only tracks Kolibri-1 78B-A3.5B in GGUF, across 8 levels (IQ1_M, IQ2_XXS, IQ2_XS, IQ3_XXS, Q3_K_M, IQ4_XS, Q4_K_M, Q8_0). No per-level perplexity sweep has been published for this model, so the quality column is empty rather than estimated — within one model, more bits per weight is the only ordering the data supports. No throughput has been measured for this model on this site's hardware. GGUF runs on llama.cpp and Ollama across NVIDIA, AMD and Apple silicon; AWQ and GPTQ target vLLM on CUDA and ROCm; EXL2 is ExLlamaV2 — CUDA only, and now archived, so local use rather than serving.

Common questions #

How much VRAM does Kolibri-1 78B-A3.5B need?
About 52.2 GB at Q4_K_M with 4K of context and batch 1, rising to roughly 52.8 GB at 32K. That figure is weights plus KV cache plus a 10% activation buffer, calculated from the model's architecture rather than measured on a card.
Will Kolibri-1 78B-A3.5B run on a 64GB GPU?
Yes — at Q4_K_M and 4K context it needs about 52.2 GB, which leaves 11.8 GB spare on a DGX Spark 64G. That is the smallest card in this index that clears it comfortably; 12 of 78 do.
Which quantization of Kolibri-1 78B-A3.5B should I use?
No published quality comparison exists for this model, so pick by footprint: Q4_K_M is the level most people run, and a higher bits-per-weight level is more faithful. All 8 levels in the index are IQ1_M, IQ2_XXS, IQ2_XS, IQ3_XXS, Q3_K_M, IQ4_XS, Q4_K_M, Q8_0. Note this is a mixture-of-experts model — all parameters must be resident even though only a fraction are active per token, so the memory cost follows the total, not the active count.

Where this model fits #

Sized at Q4_K_M with a 4K context window, smallest card first. Comfortable means the estimate uses at most 88% of the memory.

Runs comfortably on
DGX Spark 64G64GB · +11.8 A100 80G80GB · +27.8 H100 80G80GB · +27.8 Mac M2 Max 96G96GB · +19.8 DGX Spark 128G128GB · +75.8
Just misses
A40 48G+4.2GB over L40S 48G+4.2GB over
Ships in
GGUF
Try it yourself
Size it in the calculator Generate the run command

How to actually run this #

Deployment guides for this model and this class of hardware.

This page's figures change when the model or the runtime does.Last updated 2026-11-12 RSS → /feed.xml

source & further reading

quantized.uk — original article Model: Xing4.0 29B-A4B Model: Ripple-65B-Preview Model: Metis-1.6 Praxis