I have been working on a small Python stack for local LLM generation on a consumer NVIDIA GPU — specifically the Qwen3.5 hybrid architecture (Gated DeltaNet + attention), when the card does not have enough VRAM to keep every layer resident. The first public release is AirColibri 1.0.0, licensed Apache-2.0:
https://github.com/DIIIUA/air-colibri The recommended checkpoint is the official Hub repo:
What I wanted to avoid was a converter step and a second full copy of the weights. AirColibri reads the standard Hugging Face shards (model-0000X-of-0000N.safetensors plus model.safetensors.index.json) directly. On first start it builds a Q4 cache once (W4A16, group size 64) under data/q4-g64-cache/. There is no PyPI package: on Windows use aircolibri.cmd, on any OS use python aircolibri.py. aircolibri doctor --model Qwen/Qwen3.5-9B downloads the shards (~19 GB) and checks the setup.
VRAM is planned from the free memory at start, not from a fixed card profile:
--vram-mode normal (default) takes almost all free VRAM, but keeps a reserve for the OS, the KV/DeltaNet state of --max-context tokens, and the prefill workspace, so long prompts are less likely to run out of memory.--vram-mode turbo uses all free VRAM with a minimal reserve.
If the quantized model fits, every layer stays resident. If it does not, the remaining budget is split into equal slots that rotate roles — computing, cleaning, and receiving the next layer — so the next layer is already on the GPU when the current one finishes. Prefill is chunked (--prefill-chunk, default 512 tokens), and only the last position is projected onto the 248k-token vocabulary.
Documented hardware: an NVIDIA GPU with CUDA. I tested on an RTX 3060 Ti 8 GB. The README says cards with about 4 GB stream part of the layers. Windows x64 or Linux, Python 3.12, PyTorch 2.6 with CUDA 12.4.
aircolibri serve starts a LAN server with a built-in web chat at / and an OpenAI-compatible POST /v1/chat/completions (JSON or SSE) on port 8787. Generation can be stopped immediately with POST /v1/chat/stop, or by closing the client connection, including during prefill. Only one generation runs at a time. The server is open on the LAN by default; --require-auth enables a Bearer key.
A few caveats worth stating up front:
AirColibri borrows ideas from AirLLM (layer streaming, selective weight reads) and Colibrì (tiered placement, native MTP speculation), with its own kernels. It is an independent project; those names do not imply endorsement. It is also not the same as the Colibrì int4 containers already on the Hub.
I would be glad to answer questions about the VRAM planner, the safetensors reader, or the Q4 cache. If you try this on a card other than a 3060 Ti, I would like to hear how the streaming path behaves.