{"slug": "local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference", "title": "Local LLM generation on a consumer GPU: layer-streamed Qwen3.5 inference", "summary": "AirColibri 1.0.0, an Apache-2.0 Python stack for running the Qwen3.5 hybrid architecture (Gated DeltaNet plus attention) on consumer NVIDIA GPUs with insufficient VRAM, has been released on GitHub, with testing done on an 8 GB RTX 3060 Ti. AirColibri reads standard Hugging Face safetensors shards directly, builds a one-time Q4 cache (W4A16, group size 64), and plans VRAM from free memory at start, splitting the remaining budget into rotating slots that compute, clean and receive the next layer when the quantized model does not fit; the README says cards with about 4 GB stream part of the layers. The tool ships no PyPI package, runs on Windows x64 or Linux with Python 3.12 and PyTorch 2.6 with CUDA 12.4, and its `aircolibri serve` command exposes an OpenAI-compatible `POST /v1/chat/completions` endpoint on port 8787 that is open on the LAN unless `--require-auth` is set.", "body_md": "I have been working on a small Python stack for local LLM generation on a consumer NVIDIA GPU — specifically the Qwen3.5 hybrid architecture (Gated DeltaNet + attention), when the card does not have enough VRAM to keep every layer resident. The first public release is AirColibri 1.0.0, licensed Apache-2.0:\n\n[https://github.com/DIIIUA/air-colibri](https://github.com/DIIIUA/air-colibri)\n\nThe recommended checkpoint is the official Hub repo:\n\nWhat I wanted to avoid was a converter step and a second full copy of the weights. AirColibri reads the standard Hugging Face shards (`model-0000X-of-0000N.safetensors` plus `model.safetensors.index.json`) directly. On first start it builds a Q4 cache once (W4A16, group size 64) under `data/q4-g64-cache/`. There is no PyPI package: on Windows use `aircolibri.cmd`, on any OS use `python aircolibri.py`. `aircolibri doctor --model Qwen/Qwen3.5-9B` downloads the shards (~19 GB) and checks the setup.\n\nVRAM is planned from the free memory at start, not from a fixed card profile:\n\n`--vram-mode normal` (default) takes almost all free VRAM, but keeps a reserve for the OS, the KV/DeltaNet state of `--max-context` tokens, and the prefill workspace, so long prompts are less likely to run out of memory.`--vram-mode turbo` uses all free VRAM with a minimal reserve.\nIf the quantized model fits, every layer stays resident. If it does not, the remaining budget is split into equal slots that rotate roles — computing, cleaning, and receiving the next layer — so the next layer is already on the GPU when the current one finishes. Prefill is chunked (`--prefill-chunk`, default 512 tokens), and only the last position is projected onto the 248k-token vocabulary.\n\nDocumented hardware: an NVIDIA GPU with CUDA. I tested on an RTX 3060 Ti 8 GB. The README says cards with about 4 GB stream part of the layers. Windows x64 or Linux, Python 3.12, PyTorch 2.6 with CUDA 12.4.\n\n`aircolibri serve` starts a LAN server with a built-in web chat at `/` and an OpenAI-compatible `POST /v1/chat/completions` (JSON or SSE) on port 8787. Generation can be stopped immediately with `POST /v1/chat/stop`, or by closing the client connection, including during prefill. Only one generation runs at a time. The server is open on the LAN by default; `--require-auth` enables a Bearer key.\n\nA few caveats worth stating up front:\n\nAirColibri borrows ideas from AirLLM (layer streaming, selective weight reads) and Colibrì (tiered placement, native MTP speculation), with its own kernels. It is an independent project; those names do not imply endorsement. It is also not the same as the Colibrì int4 containers already on the Hub.\n\nI would be glad to answer questions about the VRAM planner, the safetensors reader, or the Q4 cache. If you try this on a card other than a 3060 Ti, I would like to hear how the streaming path behaves.", "url": "https://wpnews.pro/news/local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference", "canonical_source": "https://discuss.huggingface.co/t/local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference/192001#post_1", "published_at": "2026-11-17 23:14:44+00:00", "updated_at": "2026-11-17 23:39:57.515570+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["AirColibri", "Qwen3.5", "Qwen3.5-9B", "NVIDIA", "RTX 3060 Ti", "PyTorch", "Hugging Face", "AirLLM"], "also_reported_by": [], "takeaway": "AirColibri 1.0.0 runs Qwen3.5 on an 8 GB RTX 3060 Ti by reading Hugging Face safetensors shards directly, building a one-time Q4 W4A16 group-64 cache, and rotating equal VRAM slots through compute, clean and receive roles so the next layer is already on the GPU when the current one finishes.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference", "markdown": "https://wpnews.pro/news/local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference.md", "text": "https://wpnews.pro/news/local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference.txt", "jsonld": "https://wpnews.pro/news/local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference.jsonld"}}