{"slug": "colibri-running-a-744b-ai-model-on-your-laptop", "title": "Colibri: Running a 744B AI Model on Your Laptop", "summary": "A developer known as JustVugg created Colibri, a pure C inference engine that runs the 744-billion-parameter GLM-5.2 AI model on a laptop with only 25 GB of RAM by streaming model experts from disk instead of keeping them in memory. The engine exploits the Mixture-of-Experts architecture to load only active experts per token, achieving correct answers at 0.28–1.83 tokens per second on consumer hardware, compared to 30–50 tok/s on cloud H100 GPUs.", "body_md": "GLM-5.2 is a frontier AI model with 744 billion parameters. It normally requires H100 GPUs, hundreds of gigabytes of VRAM, and a cloud bill larger than a car payment. Colibri runs it on a laptop with 25 GB of RAM.\n\nThe secret? It never loads the whole model.\n\nColibri is a pure C inference engine for GLM-5.2, a Mixture-of-Experts (MoE) model released by Z.ai. It streams model experts from disk instead of keeping them in memory, trading disk I/O for RAM.\n\n\"This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan.\"\n\n— Colibri README\n\nGLM-5.2 uses Mixture-of-Experts. For each token, only a small subset of \"experts\" are active. The model has:\n\nThe key insight: only ~11 GB of parameters change token-to-token. Those 11 GB are the routed experts. The remaining 34 GB are dense layers that stay loaded.\n\nColibri keeps the dense parts in RAM and streams experts from NVMe SSD on demand. It uses per-layer LRU caching, optional pinned hot-store for frequently used experts, and relies on the OS page cache as a free L2 cache.\n\n| Metric | Value |\n|---|---|\n| Model on disk (int4) | ~370 GB |\n| Resident RAM | 9.9 GB |\n| Minimum RAM | 25 GB |\n| Load time | ~30 seconds |\n| Peak RSS during chat | ~20 GB (auto-capped) |\n| Cold decode cost | ~11 GB disk reads/token |\n| MTP speculation (int8 head) | 2.2–2.8 tok/forward |\n\n`maddubs`\n\n| Platform | Status | Notes |\n|---|---|---|\n| Linux | ✅ Full | Primary platform |\n| macOS | ✅ Full | Metal backend option |\n| Windows 11 | ✅ Native | MinGW-w64 with `_WIN32` compatibility layer |\n\nRequirements: gcc with OpenMP, AVX2, ≥16 GB RAM, ~370 GB model on local NVMe.\n\nQuick start:\n\n```\ncd c\n./setup.sh  # Build and self-tests\nCOLI_MODEL=/path/to/glm52_i4 ./coli chat\n```\n\nOpenAI-compatible API:\n\n```\nCOLI_MODEL=/nvme/glm52_i4 ./coli serve \\\n  --host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri\n```\n\nQuality benchmarks:\n\n```\n./coli bench  # hellaswag, arc_challenge, mmlu\n```\n\nThis is the trade-off. Colibri is not fast. Performance depends heavily on disk speed, RAM capacity, and expert caching:\n\n| Hardware | Tokens/second |\n|---|---|\n| Apple M5 Max (128 GB RAM, Metal) | 1.06–1.83 tok/s |\n| Ryzen AI 9 HX 370 (128 GB RAM) | 0.37 tok/s |\n| Ryzen 9 9950X (PCIe 5.0 NVMe) | 0.28 tok/s |\n\nFor context, a cloud H100 inference might run at 30–50 tok/s. Colibri is 10–100× slower. But it costs nothing after setup and runs entirely locally.\n\nFrontier AI models are becoming inaccessible. GPT-4, Claude 3.5, GLM-5.2 — these require proprietary APIs, monthly subscriptions, and internet access. Your data leaves your machine. Your conversations are logged.\n\nColibri proves that with clever engineering, you can run a frontier model on consumer hardware. You need:\n\nThe model answers correctly. The quality is there. The trade-off is speed.\n\n**Offline coding assistant:** Run GLM-5.2 locally without API keys or quotas. Your code never leaves your machine.\n\n**Privacy-first knowledge base:** Build a RAG system with local retrieval and local generation. Zero data exfiltration.\n\n**Benchmarking research:** Test frontier model quality without cloud costs. Run hellaswag, arc_challenge, mmlu locally.\n\n**Edge deployment:** Install on a rugged laptop for field deployment where internet is unreliable.\n\nColibri is a one-person project built on a 12-core laptop with 25 GB of RAM. The author (JustVugg) exploited the Mixture-of-Experts architecture:\n\nQuantization (int4) reduces the model from ~1.5 TB to ~370 GB on disk. MLA attention compresses KV-cache by 57×. Speculative decoding with MTP head drafts 2–3 tokens per forward pass.\n\nThese techniques are not new. Colibri combines them into a single engine optimized for streaming experts.\n\nColibri does not try to be fast. It does not support LoRA fine-tuning, RLHF alignment loading, or custom quantization recipes. It is purpose-built for one thing: running GLM-5.2 faithfully on minimal hardware.\n\nFor BD developers with cheap VPS or older laptops, this is more theoretical than practical. You need 370 GB of NVMe storage and a modern CPU with AVX2. But it shows the direction: frontier models can run locally if we stop loading them entirely into RAM.\n\nColibri is a proof of concept. It proves that streaming experts from disk works. It proves that int4 quantization preserves quality. It proves that MLA attention compression is viable.\n\nThe next step is making it fast. Better caching, smarter prefetching, GPU acceleration for matmul, multi-threaded expert loading. Each improvement inches closer to \"fast enough\" for real work.\n\nBut even at 0.3 tok/s, Colibri is useful. It runs a 744B model on a machine that costs less than a single H100 GPU fan. That is remarkable.\n\nIf you have the hardware, here is the path:\n\n`git clone https://github.com/JustVugg/colibri`\n\n`tools/`\n\ndirectory`cd c && ./setup.sh`\n\n`COLI_MODEL=/path/to/glm52_i4 ./coli chat`\n\nYou will need patience. The first token takes 30–60 seconds. Subsequent tokens are faster. But the answers are correct. The quality is there.\n\nColibri is not for everyone. It requires 370 GB of NVMe storage, a modern CPU, and patience. But it proves that frontier AI models do not require cloud infrastructure.\n\nA one-person project, a 12-core laptop, and clever engineering can run a 744B model that would otherwise cost tens of thousands in GPU time. That is the story of local AI.\n\nThe hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. Colibri keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience.", "url": "https://wpnews.pro/news/colibri-running-a-744b-ai-model-on-your-laptop", "canonical_source": "https://dev.to/jamilxt/colibri-running-a-744b-ai-model-on-your-laptop-4l6g", "published_at": "2026-07-17 20:45:21+00:00", "updated_at": "2026-07-17 20:59:54.311416+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools", "ai-research"], "entities": ["JustVugg", "Colibri", "GLM-5.2", "Z.ai", "Apple M5 Max", "Ryzen AI 9 HX 370", "Ryzen 9 9950X"], "alternates": {"html": "https://wpnews.pro/news/colibri-running-a-744b-ai-model-on-your-laptop", "markdown": "https://wpnews.pro/news/colibri-running-a-744b-ai-model-on-your-laptop.md", "text": "https://wpnews.pro/news/colibri-running-a-744b-ai-model-on-your-laptop.txt", "jsonld": "https://wpnews.pro/news/colibri-running-a-744b-ai-model-on-your-laptop.jsonld"}}