{"slug": "colibri-made-me-rethink-how-we-run-llms-and-there-might-be-a-next-step", "title": "Colibri Made Me Rethink How We Run LLMs — And There Might Be a Next Step", "summary": "A developer discovered Colibri, a project that runs massive Mixture of Experts (MoE) models on ordinary CPUs and SSDs by loading only needed experts into memory, challenging the assumption that entire models must reside in RAM. The developer proposes a next step: predictive expert loading, where the system anticipates which experts will be needed and preloads them from SSD to RAM, reducing latency. This proactive approach could transform storage from a passive to an active component in inference.", "body_md": "Over the past few days, I came across a project that made me stop everything to understand how it worked. It wasn't just another framework for running language models, nor another GPU-based optimization. It was something different.\n\nThe project is called **Colibri**.\n\nAt first glance, it almost seems impossible. The idea is simple, yet remarkably bold: run a massive Mixture of Experts (MoE) model on an ordinary computer using nothing more than a CPU and an SSD, loading into memory only the experts that are actually needed at each step.\n\nWhen I first read that, my reaction was probably the same as everyone else's:\n\n*\"How could that possibly work?\"*\n\nAfter studying the architecture, I realized that Colibri's real innovation isn't simply that it can run an enormous model. The real breakthrough is that it completely changes the way we think about inference.\n\nFor years, we've been taught that running a model means loading the entire model into memory.\n\nMore VRAM.\n\nMore RAM.\n\nMore GPUs.\n\nColibri challenges that assumption.\n\nInstead of keeping hundreds of billions of parameters resident in memory, it treats the model as a living system.\n\nThe router selects which experts are needed.\n\nThose experts are loaded from the SSD.\n\nThey perform their computation.\n\nAnd once they're no longer needed, they can be discarded.\n\nIt's an incredibly elegant approach.\n\nIn practice, storage is no longer just a place where model weights are kept—it becomes an active participant in the inference process.\n\nIt's one of those ideas that feels both simple and brilliant.\n\nOnce I understood how it worked, another question came to mind.\n\n**What if the real problem isn't the algorithm?**\n\nBecause the architecture itself makes perfect sense.\n\nThe bottleneck appears somewhere else.\n\nEvery time an expert is needed, there's an unavoidable sequence:\n\n**SSD → RAM → CPU**\n\nThat process happens thousands of times during a single generation.\n\nIn other words, the real limitation is no longer computational power—it's the time spent waiting for data to move from storage into memory.\n\nThat's when I started wondering what the next step for this architecture could be.\n\nModern CPUs already do this.\n\nThey try to predict which instruction will be executed next.\n\nOperating systems do it.\n\nDatabases do it.\n\nWeb browser caches do it.\n\nSo why couldn't an MoE inference engine do the same?\n\nImagine that after processing thousands of tokens, the system discovers patterns like these:\n\n```\nExpert 15\n ↓ 92%\nExpert 48\n\nExpert 48\n ↓ 87%\nExpert 102\n\nExpert 102\n ↓ 95%\nExpert 211\n```\n\nIn this scenario, the engine wouldn't need to wait for the next token before loading the next expert.\n\nWhile the CPU is computing the current expert, another thread could already be loading the most likely next experts.\n\nBy the time they're needed, they're already sitting in RAM.\n\nThe waiting time nearly disappears.\n\nToday, the SSD is used reactively.\n\nFirst, the router makes a decision.\n\nThen, the SSD responds.\n\nBut what if we reversed that logic?\n\nInstead of reacting to requests, the system begins anticipating them.\n\nThat transforms storage from a passive component into a proactive one.\n\nOne possible architecture could consist of four independent modules:\n\nThis separation would allow disk I/O and computation to happen simultaneously, dramatically reducing idle time.\n\nAs I explored the project further, a few additional possibilities came to mind.\n\nExperts that are used hundreds of times shouldn't be evicted from memory as aggressively as experts that are rarely selected.\n\nAs soon as the router makes its decision, immediately begin loading the most likely upcoming experts.\n\nAllow the operating system to manage page caching directly, reducing unnecessary memory copies and making better use of the existing virtual memory subsystem.\n\nExperts that are frequently activated together could be stored physically close to one another on disk, enabling much more efficient sequential reads.\n\nWhile one group of experts is executing...\n\nAnother group is already being loaded.\n\nAnd a third group is waiting in memory.\n\nIdeally, the CPU would spend almost no time waiting for the SSD.\n\nThe most interesting part is that none of these ideas changes Colibri's original philosophy.\n\nThe entire model still isn't loaded into memory.\n\nInference still runs entirely on CPUs.\n\nThe SSD remains an active part of the inference pipeline.\n\nThe difference is that the system stops reacting to events and starts anticipating them.\n\nIt's an evolution of the same idea.\n\nNot a replacement.\n\nColibri demonstrates that there's still plenty of room for innovation in inference engines.\n\nFor years, the conversation has revolved around bigger GPUs, more memory, and greater computational power.\n\nMaybe there's another path.\n\nMaybe we can build increasingly intelligent systems for moving data.\n\nBecause in the end, the hardest problem may not be computing faster.\n\nIt may simply be ensuring that the right data is in the right place at exactly the right time.\n\nThat's what made Colibri so fascinating to me.\n\nAnd perhaps that's exactly where the next major step in this idea lies.\n\nCongratulations to the creator of the project for the creativity and the courage to challenge a paradigm that many considered unquestionable. I hope these ideas inspire further discussion and experimentation, because I believe architectures like this still have tremendous room to evolve.\n\n**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.\n\nColibrì is a lightweight, quality-preserving MoE runtime that treats VRAM RAM, and storage as one managed memory hierarchy. Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics.\n\n```\n$ ./coli chat\n  🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU\n  ✓ ready in 32s · resident 9.9 GB\n  › ciao\n  ◆ Ciao! 😊 Come posso aiutarti oggi?\n```\n\n*The web dashboard ( ./coli web): a 744B model answering at 4+ tok/s end-to-end on 6× RTX 5090 —\nwith live token metrics, the hardware panel, and the VRAM/RAM/disk expert tiers.*\n\n*The Brain page: all 19,456 experts as a living cortex — colour is the storage tier\nbrightness*…", "url": "https://wpnews.pro/news/colibri-made-me-rethink-how-we-run-llms-and-there-might-be-a-next-step", "canonical_source": "https://dev.to/lhabacuc/-colibri-made-me-rethink-how-we-run-llms-and-there-might-be-a-next-step-52bj", "published_at": "2026-07-19 01:24:05+00:00", "updated_at": "2026-07-19 01:57:27.980146+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "developer-tools"], "entities": ["Colibri"], "alternates": {"html": "https://wpnews.pro/news/colibri-made-me-rethink-how-we-run-llms-and-there-might-be-a-next-step", "markdown": "https://wpnews.pro/news/colibri-made-me-rethink-how-we-run-llms-and-there-might-be-a-next-step.md", "text": "https://wpnews.pro/news/colibri-made-me-rethink-how-we-run-llms-and-there-might-be-a-next-step.txt", "jsonld": "https://wpnews.pro/news/colibri-made-me-rethink-how-we-run-llms-and-there-might-be-a-next-step.jsonld"}}