# Run DeepSeek and Qwen Locally on One GPU with Colibri 2.1 > Source: > Published: 2026-11-02 00:00:00+00:00 # Run DeepSeek and Qwen Locally on One GPU with Colibri 2.1 Colibri 2.1 lets one GPU run massive MoE models like DeepSeek and Qwen by splitting experts across VRAM, RAM, and disk storage. ## What is Colibri and why does it matter for local LLMs? Colibri is a local inference tool built to run very large mixture-of-experts (MoE) models on hardware that would normally never fit them. Instead of requiring enough VRAM to hold an entire model, it splits the model across three tiers: GPU memory for the experts used most often, system RAM for the parts that are always needed plus a cache, and disk for everything else. A router inside the model picks a handful of experts for each token, and Colibri fetches whichever expert is needed from wherever it happens to live. The practical result: models with hundreds of experts and tens of billions of parameters can run on a single consumer or prosumer GPU instead of a multi-GPU server. This matters because the biggest open models right now, DeepSeek and Qwen among them, are built as MoE architectures specifically so that only a fraction of parameters activate per token. That design is a natural fit for tiered offloading, but most inference engines still assume the whole model needs to sit in VRAM. Colibri is built around the opposite assumption. ## TL;DR - **Colibri 2.1** adds native support for GGUF files, meaning they now run as published across engines with improved CUDA performance, no conversion or re-quantization required. - The release also ships a **new chat UI** that visualizes which experts fire in real time, alongside token speed, context usage, and a profiling breakdown of where time is spent per turn. - On a single 48GB GPU, a **23GB Qwen 3.6 MoE model** with 256 experts across 40 layers loaded entirely into VRAM and generated at roughly**88.6 tokens per second** . - Colibri scans the host machine (CPU, RAM, disk speed, GPU) automatically and builds a placement plan that decides how many experts go to VRAM, RAM, and disk before serving starts. - A demo running **three large models simultaneously** (Qwen 3.6, a Flint GGUF import, and a roughly 85GB quantized DeepSeek model) kept total GPU usage under 24GB by offloading the rest to RAM and disk. - GGUF models can now be **imported directly** into Colibri with a single import command, with no conversion step, which opens the door to running existing quantized models from the community. - Performance depends heavily on how much of a model fits in VRAM versus how much spills to RAM or disk; heavier models like DeepSeek visibly slow down when more of the computation has to pull from slower tiers. __>__add rate limiting to the API **●** Reading src/server.ts **●** Editing 3 files __+ import__{ limit } __from__'rate' *- app.use(cors())* **✓** Tests passed · ready for PR *● Live* *Approved* *Approved* *Approved* ## How does Colibri’s memory offloading actually work? Every MoE model has a router that decides, per token, which experts to activate. Colibri exploits this by keeping track of where each expert currently lives and loading or evicting them based on usage patterns. The startup process generates a “placement plan” that logs exactly how the model will be split before any serving begins. In the demonstrated Qwen 3.6 run, that plan allocated the full set of roughly 25,127 experts (plus 40 layers of dense weights, about 4.9GB) onto a 44.2GB VRAM budget, while reserving a 74.1GB RAM cache tier. The plan output also shows how aggressively Colibri clamps VRAM usage to avoid overcommitting. When a model’s experts all fit in GPU memory, as happened with Qwen 3.6 on a 48GB card, disk I/O for expert weights drops to zero and the model runs at native GPU speed. When a model is too large to fit, as with the roughly 85GB quantized DeepSeek model, Colibri spreads expert weights across RAM and disk and pulls them on demand, which is slower but still functional. ## What’s new in Colibri 2.1? Two changes define this release. First, GGUF files now run as published across every engine supported by Colibri, with better CUDA support baked in. Previously, using a GGUF model might have required conversion or re-quantization; now it’s a direct import. In the demonstration, a quantized version of the Flint model was downloaded independently and imported into Colibri with a single command, reporting 49 layers, and was immediately available to chat with. Second, there’s a new chat interface that goes well beyond a basic text box. It includes a live “expert map” that visualizes the model’s expert blocks (for Qwen 3.6, 40 blocks of 256 experts) and lights up whichever experts are actively firing for a given prompt. Blocks shown in green sit in GPU memory; blue or gray would indicate spillover to RAM or disk. The UI also tracks token throughput, context window usage, RAM consumption, and a profiling tab that breaks each turn down into I/O wait, expert math, attention, the output layer, and other overhead. ## What performance can you expect on a single GPU? On a 48GB VRAM card with around 94GB of system RAM, the Qwen 3.6 MoE model (256 experts, 40 layers, roughly 23GB on disk) ran with every expert resident in VRAM. That configuration produced about 88.6 tokens per second on a coding prompt, with zero seconds spent on expert read or disk I/O, since nothing needed to be pulled from slower storage. VRAM usage for that model alone sat around 22GB. **Adjuster**✕ Export PHI **#4471** **12,480** users online Things change when multiple large models or models too big for VRAM enter the picture. In one test, Qwen 3.6, the imported Flint model, and an approximately 85GB quantized DeepSeek model were all loaded at once. Total VRAM consumption across all three stayed under 24GB, because Colibri offloaded most of the inactive experts to RAM and disk rather than keeping redundant copies in GPU memory. Running the DeepSeek model under this kind of load was visibly slower than the Qwen-only test, since more computation had to pull from RAM and disk instead of VRAM. The expert map in the chat UI still lit up for DeepSeek, confirming the router was actively selecting experts across tiers, just at a reduced pace. ## Is Colibri worth using for running DeepSeek or Qwen locally? For anyone with a single GPU and a reasonable amount of system RAM and disk space, Colibri removes a major barrier: the need to buy or rent multiple high-VRAM GPUs just to load a massive MoE model. The tradeoff is speed. Models that fit fully in VRAM run at full GPU speed, but models that spill into RAM or disk run slower, proportional to how much of the model lives outside the GPU tier. It’s also worth noting that GGUF import support does not mean every new model works out of the box. Supporting a brand-new model architecture still requires engine-level work from Colibri’s developers, it isn’t a universal drag-and-drop system yet. For established architectures like Qwen’s MoE lineup and DeepSeek, though, the combination of automatic hardware scanning, tiered memory placement, and now direct GGUF import makes it realistic to run frontier-scale open models on hardware that previously couldn’t touch them. ## Frequently Asked Questions ### What hardware do you need to run DeepSeek or Qwen locally with Colibri? The demonstrated setup used a [single GPU with 48GB of VRAM](https://goremy.ai/blog/articles/hardware-requirements-for-local-llm-inference-2026-guide) and about 94GB of system RAM on Ubuntu, plus enough free disk space to hold the model files (the DeepSeek quantized model alone was around 85GB). Colibri scans available CPU, RAM, disk speed, and GPU capacity automatically and builds a placement plan based on what it finds, so requirements scale with the model size you want to run. ### Do I need to convert or re-quantize GGUF models to use them in Colibri? No. As of version 2.1, GGUF files run as published, with no conversion or second quantization step required. Models can be imported with a single import command. ### How much slower is a model when it doesn’t fully fit in VRAM? It depends on how much of the model spills to RAM versus disk. When every expert sits in VRAM, as with the Qwen 3.6 test, I/O wait drops to zero and throughput hits native GPU speed. When a model like the large DeepSeek checkpoint spills heavily into RAM and disk, generation visibly slows, though it remains functional even under the added load of other models running at the same time. ### Can Colibri run multiple large models at once on one GPU? Yes. The demonstration ran Qwen 3.6, an imported Flint GGUF model, and a roughly 85GB quantized DeepSeek model simultaneously, with combined VRAM usage staying under 24GB by offloading inactive experts to RAM and disk. ### What does the expert map in the new chat UI actually show? It’s a live visualization of a model’s expert blocks. Green blocks indicate experts currently sitting in GPU memory; blue or gray would indicate experts offloaded to RAM or disk. As the model generates a response, the blocks corresponding to actively selected experts light up, letting you watch the router’s choices in real time.