GLM-5.2 is a frontier AI model with 744 billion parameters. It normally requires H100 GPUs, hundreds of gigabytes of VRAM, and a cloud bill larger than a car payment. Colibri runs it on a laptop with 25 GB of RAM.
The secret? It never loads the whole model.
Colibri is a pure C inference engine for GLM-5.2, a Mixture-of-Experts (MoE) model released by Z.ai. It streams model experts from disk instead of keeping them in memory, trading disk I/O for RAM.
"This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan."
— Colibri README
GLM-5.2 uses Mixture-of-Experts. For each token, only a small subset of "experts" are active. The model has:
The key insight: only ~11 GB of parameters change token-to-token. Those 11 GB are the routed experts. The remaining 34 GB are dense layers that stay loaded.
Colibri keeps the dense parts in RAM and streams experts from NVMe SSD on demand. It uses per-layer LRU caching, optional pinned hot-store for frequently used experts, and relies on the OS page cache as a free L2 cache.
| Metric | Value |
|---|---|
| Model on disk (int4) | ~370 GB |
| Resident RAM | 9.9 GB |
| Minimum RAM | 25 GB |
| Load time | ~30 seconds |
| Peak RSS during chat | ~20 GB (auto-capped) |
| Cold decode cost | ~11 GB disk reads/token |
| MTP speculation (int8 head) | 2.2–2.8 tok/forward |
maddubs
| Platform | Status | Notes |
|---|---|---|
| Linux | ✅ Full | Primary platform |
| macOS | ✅ Full | Metal backend option |
| Windows 11 | ✅ Native | MinGW-w64 with _WIN32 compatibility layer |
Requirements: gcc with OpenMP, AVX2, ≥16 GB RAM, ~370 GB model on local NVMe.
Quick start:
cd c
./setup.sh # Build and self-tests
COLI_MODEL=/path/to/glm52_i4 ./coli chat
OpenAI-compatible API:
COLI_MODEL=/nvme/glm52_i4 ./coli serve \
--host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
Quality benchmarks:
./coli bench # hellaswag, arc_challenge, mmlu
This is the trade-off. Colibri is not fast. Performance depends heavily on disk speed, RAM capacity, and expert caching:
| Hardware | Tokens/second |
|---|---|
| Apple M5 Max (128 GB RAM, Metal) | 1.06–1.83 tok/s |
| Ryzen AI 9 HX 370 (128 GB RAM) | 0.37 tok/s |
| Ryzen 9 9950X (PCIe 5.0 NVMe) | 0.28 tok/s |
For context, a cloud H100 inference might run at 30–50 tok/s. Colibri is 10–100× slower. But it costs nothing after setup and runs entirely locally.
Frontier AI models are becoming inaccessible. GPT-4, Claude 3.5, GLM-5.2 — these require proprietary APIs, monthly subscriptions, and internet access. Your data leaves your machine. Your conversations are logged.
Colibri proves that with clever engineering, you can run a frontier model on consumer hardware. You need:
The model answers correctly. The quality is there. The trade-off is speed.
Offline coding assistant: Run GLM-5.2 locally without API keys or quotas. Your code never leaves your machine.
Privacy-first knowledge base: Build a RAG system with local retrieval and local generation. Zero data exfiltration.
Benchmarking research: Test frontier model quality without cloud costs. Run hellaswag, arc_challenge, mmlu locally.
Edge deployment: Install on a rugged laptop for field deployment where internet is unreliable.
Colibri is a one-person project built on a 12-core laptop with 25 GB of RAM. The author (JustVugg) exploited the Mixture-of-Experts architecture:
Quantization (int4) reduces the model from ~1.5 TB to ~370 GB on disk. MLA attention compresses KV-cache by 57×. Speculative decoding with MTP head drafts 2–3 tokens per forward pass.
These techniques are not new. Colibri combines them into a single engine optimized for streaming experts.
Colibri does not try to be fast. It does not support LoRA fine-tuning, RLHF alignment , or custom quantization recipes. It is purpose-built for one thing: running GLM-5.2 faithfully on minimal hardware.
For BD developers with cheap VPS or older laptops, this is more theoretical than practical. You need 370 GB of NVMe storage and a modern CPU with AVX2. But it shows the direction: frontier models can run locally if we stop them entirely into RAM.
Colibri is a proof of concept. It proves that streaming experts from disk works. It proves that int4 quantization preserves quality. It proves that MLA attention compression is viable.
The next step is making it fast. Better caching, smarter prefetching, GPU acceleration for matmul, multi-threaded expert . Each improvement inches closer to "fast enough" for real work.
But even at 0.3 tok/s, Colibri is useful. It runs a 744B model on a machine that costs less than a single H100 GPU fan. That is remarkable.
If you have the hardware, here is the path:
git clone https://github.com/JustVugg/colibri
tools/
directorycd c && ./setup.sh
COLI_MODEL=/path/to/glm52_i4 ./coli chat
You will need patience. The first token takes 30–60 seconds. Subsequent tokens are faster. But the answers are correct. The quality is there.
Colibri is not for everyone. It requires 370 GB of NVMe storage, a modern CPU, and patience. But it proves that frontier AI models do not require cloud infrastructure.
A one-person project, a 12-core laptop, and clever engineering can run a 744B model that would otherwise cost tens of thousands in GPU time. That is the story of local AI.
The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. Colibri keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.