# Local LLM generation on a consumer GPU: layer-streamed Qwen3.5 inference

> Source: <https://discuss.huggingface.co/t/local-llm-generation-on-a-consumer-gpu-layer-streamed-qwen3-5-inference/192001#post_1>
> Published: 2026-11-17 23:14:44+00:00

**Key takeaway:** AirColibri 1.0.0 runs Qwen3.5 on an 8 GB RTX 3060 Ti by reading Hugging Face safetensors shards directly, building a one-time Q4 W4A16 group-64 cache, and rotating equal VRAM slots through compute, clean and receive roles so the next layer is already on the GPU when the current one finishes.

I have been working on a small Python stack for local LLM generation on a consumer NVIDIA GPU — specifically the Qwen3.5 hybrid architecture (Gated DeltaNet + attention), when the card does not have enough VRAM to keep every layer resident. The first public release is AirColibri 1.0.0, licensed Apache-2.0:

[https://github.com/DIIIUA/air-colibri](https://github.com/DIIIUA/air-colibri)

The recommended checkpoint is the official Hub repo:

What I wanted to avoid was a converter step and a second full copy of the weights. AirColibri reads the standard Hugging Face shards (`model-0000X-of-0000N.safetensors` plus `model.safetensors.index.json`) directly. On first start it builds a Q4 cache once (W4A16, group size 64) under `data/q4-g64-cache/`. There is no PyPI package: on Windows use `aircolibri.cmd`, on any OS use `python aircolibri.py`. `aircolibri doctor --model Qwen/Qwen3.5-9B` downloads the shards (~19 GB) and checks the setup.

VRAM is planned from the free memory at start, not from a fixed card profile:

`--vram-mode normal` (default) takes almost all free VRAM, but keeps a reserve for the OS, the KV/DeltaNet state of `--max-context` tokens, and the prefill workspace, so long prompts are less likely to run out of memory.`--vram-mode turbo` uses all free VRAM with a minimal reserve.
If the quantized model fits, every layer stays resident. If it does not, the remaining budget is split into equal slots that rotate roles — computing, cleaning, and receiving the next layer — so the next layer is already on the GPU when the current one finishes. Prefill is chunked (`--prefill-chunk`, default 512 tokens), and only the last position is projected onto the 248k-token vocabulary.

Documented hardware: an NVIDIA GPU with CUDA. I tested on an RTX 3060 Ti 8 GB. The README says cards with about 4 GB stream part of the layers. Windows x64 or Linux, Python 3.12, PyTorch 2.6 with CUDA 12.4.

`aircolibri serve` starts a LAN server with a built-in web chat at `/` and an OpenAI-compatible `POST /v1/chat/completions` (JSON or SSE) on port 8787. Generation can be stopped immediately with `POST /v1/chat/stop`, or by closing the client connection, including during prefill. Only one generation runs at a time. The server is open on the LAN by default; `--require-auth` enables a Bearer key.

A few caveats worth stating up front:

AirColibri borrows ideas from AirLLM (layer streaming, selective weight reads) and Colibrì (tiered placement, native MTP speculation), with its own kernels. It is an independent project; those names do not imply endorsement. It is also not the same as the Colibrì int4 containers already on the Hub.

I would be glad to answer questions about the VRAM planner, the safetensors reader, or the Q4 cache. If you try this on a card other than a 3060 Ti, I would like to hear how the streaming path behaves.
