{"slug": "run-qwen3-8-27b-escha-w2-on-a-24gb-gpu-with-sglang", "title": "Run Qwen3.8-27B-Escha-W2 on a 24GB GPU with SGLang", "summary": "Escha Labs released Qwen3.8-27B-Escha-W2, a 2-bit quantized build of Qwen3.8-27B that fits 10.15 GB of weights on a 24 GB consumer GPU, enabling 64k context out of the box and up to 128k with tuning via SGLang. The model uses hybrid attention with only 16 of 64 layers doing full attention, and RTX 3090 users should set ESCHA_ROUTE=blackwell for a 1.72x speedup at batch size 1.", "body_md": "# Run Qwen3.8-27B-Escha-W2 on a 24GB GPU with SGLang\n\nHow to install and tune Escha-W2, a 2-bit quant of Qwen3.8-27B, on a 24GB consumer GPU using SGLang for long context or high throughput.\n\n## What is Qwen3.8-27B-Escha-W2?\n\nEscha-W2 is a 2-bit quantized build of Qwen3.8-27B, published by Escha Labs, that fits the full 27-billion-parameter model into roughly 10.15 GB of weights. That’s small enough to load the model, its KV cache, and a 64k context window on a single 24 GB consumer GPU, with headroom to spare. With a tuned configuration, the same card can push all the way to 128k context. The model is served through a custom SGLang runtime built specifically for this quantization format.\n\n## TL;DR\n\n**Escha-W2 quantizes Qwen3.8-27B to roughly 2.469 bits per weight**(mixed 2/3-bit per projection, int8 embedding and head), bringing total download size to about 10.18 GB.** A 24 GB card runs the shipped defaults out of the box**: 64k context, one concurrent stream, around 18 GB of VRAM used.** The model uses a hybrid attention architecture**where only 16 of 64 layers do full attention and the rest are gated-delta-net, which is why the KV cache is small enough to allow 128k context on consumer hardware.**Three tuning variables (** between context length and concurrent request throughput, and they draw from the same shared memory pool.`MEM`\n\n,`CTXLEN`\n\n,`MAMBA_RATIO`\n\n) control the tradeoff**Prefix caching (**, not growing agent conversations, because the recurrent state can’t be resumed mid-sequence.`RADIX=1`\n\n) only helps exact repeated prompts**RTX 3090 users should set** for a measured 1.72x speedup at batch size 1, and RTX 5090 owners need`ESCHA_ROUTE=blackwell`\n\n`ATTN_BACKEND=triton`\n\ninstead of the default flashinfer backend.**The default reasoning mode (**, so latency-sensitive applications should either switch to`xhigh`\n\n) is a prompt injection, not a hard limit`low`\n\neffort or use a thinking budget to force a cutoff.\n\n## Remy doesn't build the plumbing. It inherits it.\n\nOther agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.\n\nRemy ships with all of it from MindStudio — so every cycle goes into the app you actually want.\n\n## Why does a 27B model fit in 10GB?\n\nStandard 16-bit weights for a 27-billion-parameter model would require on the order of 54 GB, well past what any single consumer GPU can hold. Escha-W2 gets around this with aggressive quantization: most projections run at 2-bit precision, some at 3-bit, averaging out to about 2.469 bits per weight, while the embedding and output head stay at int8 for stability. That shrinks the weight footprint to 10.15 GB, small enough to share a 24 GB card with a meaningful KV cache and still leave room for CUDA graph capture.\n\nThe bigger reason long context is affordable, though, is architectural rather than a quantization trick. Only 16 of the model’s 64 layers perform full attention; the other 48 use gated-delta-net, a linear-attention-style mechanism that holds a fixed amount of recurrent state per stream (about 0.15 GB) regardless of context length. Because most layers don’t scale with sequence length, the KV cache works out to about 64 KiB per token, roughly a quarter of what a conventional dense 27B model with full attention on every layer would need. That’s the mechanical reason 128k context is reachable on a 24 GB card at all.\n\n## How do you install and run it?\n\nThe setup has three moving pieces: a pinned PyTorch build, a runtime wheel from Escha Labs (which bundles its own SGLang fork), and the weights themselves.\n\n```\npython3.12 -m venv .venv && source .venv/bin/activate\npip install -U pip wheel\n\npip install \"torch==2.9.*\" --index-url https://download.pytorch.org/whl/cu128\n\npip install -U \"huggingface_hub[cli]\"\nhf download EschaLabs/escha-runtime-qwen3dense --include \"sglang/*\" --local-dir runtime\npip install ./runtime/sglang/escha-*.whl\n\nhf download EschaLabs/Qwen3.8-27B-Escha-W2 --local-dir Qwen3.8-27B-Escha-W2\n\nMODEL=./Qwen3.8-27B-Escha-W2 bash runtime/sglang/serve.sh\n```\n\nTwo details matter here. First, PyTorch has to be pinned to the 2.9.x line and installed before anything else, because the decode kernels are ABI-linked to that specific build. Installing a loose `torch>=2.9`\n\nconstraint can silently resolve to a newer, incompatible version, and the failure only shows up later as an obscure `undefined symbol`\n\nerror. Second, don’t install `sglang`\n\nfrom PyPI separately. The runtime wheel already includes a modified SGLang build, and the two will conflict.\n\nBefore serving, a sanity check confirms CUDA is available, the custom decode kernel is registered, and SGLang actually imported: all three must print `True`\n\n. Once the server is running, it exposes an OpenAI-compatible endpoint at `http://127.0.0.1:30000/v1`\n\n, and any existing OpenAI client library or tool (including agent frameworks like opencode) can point at it with a placeholder API key, since the server does no authentication by default.\n\n## What do MEM, CTXLEN, and MAMBA_RATIO actually control?\n\nThese three environment variables, passed to `serve.sh`\n\n, govern how the GPU’s memory is split between model weights, KV cache, and the recurrent state used by the gated-delta-net layers.\n\n`MEM`\n\nsets the fraction of total VRAM reserved for the weight and KV pool (default 0.72). Push it too high and CUDA graph capture runs out of memory; push it too low and the server refuses to start with a “not enough memory” error. The fix for an OOM during graph capture is to lower `MEM`\n\n, not raise it.\n\n`CTXLEN`\n\nsets the per-request context ceiling (default 65,536) but doesn’t reserve memory on its own. What actually constrains things is a shared pool the server reports at startup as `max_total_num_tokens`\n\n. That pool needs to be at least as large as the number of concurrent streams multiplied by the context length each stream uses. Raise `CTXLEN`\n\nwithout also raising `MEM`\n\n, and the pool can end up too small, silently truncating long prompts if `TRUNCATE`\n\nis left at its default of 1.\n\n`MAMBA_RATIO`\n\n(the `--mamba-full-memory-ratio`\n\nflag) governs how much memory is set aside for the recurrent state that every concurrent stream needs, independent of context length. The default of 0.3 is deliberately lower than SGLang’s usual 0.9, because this is a hybrid model where recurrent state doesn’t shrink even for short contexts.\n\nBecause context and concurrency draw from the same pool, there’s a real tradeoff: a config tuned for one very long stream (128k tokens) can’t simultaneously serve many short concurrent conversations, and vice versa.\n\n## How do you configure for long context vs. high throughput?\n\nEscha Labs published three verified configurations, each measured on physical hardware:\n\n**Single-user, 24 GB, shipped defaults.** Just `MODEL=./Qwen3.8-27B-Escha-W2 bash sglang/serve.sh`\n\ngets 64k context on an RTX 4090 at about 67 tokens/second at batch size 1, using 18.1 GB of VRAM. On an RTX 3090, the same setup with `ESCHA_ROUTE=blackwell`\n\nadded goes from 23.6 to 40.7 tokens/second.\n\n**High throughput, 24 GB.** Setting `MEM=0.86`\n\n, `CTXLEN=32768`\n\n, `MAXREQ=32`\n\n, `MAXMAMBA=32`\n\n, and expanding `CUDA_GRAPH_BS`\n\nto include larger batch sizes lifts an RTX 4090 to 649 tokens/second across 16 concurrent streams. The key insight is that `MAXREQ`\n\nand `MAXMAMBA`\n\nare what actually raise the stream ceiling; the shipped `MEM`\n\n/`MAMBA_RATIO`\n\ndefaults otherwise cap a 24 GB card at 8 to 9 concurrent streams no matter what batch sizes are listed in `CUDA_GRAPH_BS`\n\n.\n\n**32 GB, RTX 5090.** With `ATTN_BACKEND=triton`\n\nset (required on consumer Blackwell, since the default flashinfer backend isn’t supported for this hybrid architecture on sm_120), `MEM=0.85`\n\n, and similar concurrency settings, a 5090 hits 87.1 tokens/second at batch size 1 and 955 tokens/second at 16 streams.\n\nFor maximum context on a 24 GB card, the documented recipe is `MEM=0.88`\n\n, `CTXLEN=131072`\n\n, `MAXREQ=1`\n\n, `MAXMAMBA=2`\n\n. That configuration handled a 120,000-token prompt in 68 seconds with 22.1 GB peak VRAM out of 24, and represents close to the practical ceiling: `MEM=0.92`\n\n(147,456 tokens) is workable, but `MEM=0.94`\n\noverflows the pool.\n\n## Is prefix caching worth turning on?\n\n- ✕a coding agent\n- ✕no-code\n- ✕vibe coding\n- ✕a faster Cursor\n\nThe one that tells the coding agents what to build.\n\nGenerally, no, unless the workload involves retries or repeated identical prompts. `RADIX=1`\n\nenables prefix caching, but on this hybrid architecture it only reuses an exact, complete match of a previous prompt. Measured at 120k tokens, re-sending the identical prompt dropped wall time from 66.5 seconds to 1.4 seconds. But appending new content to an already-cached prefix, the pattern of a typical agent loop that adds a tool result each turn, reused nothing and took the same 65.7 seconds as a fresh prompt. The likely explanation is that the recurrent state used by the gated-delta-net layers is only valid at the exact point it was captured, so there’s no partial state to resume from. That makes `RADIX=1`\n\nuseful for cache warming or multi-sampling a fixed prompt, but not a latency lever for growing conversations. It also requires `MAXREQ`\n\nof at least 2, since prefix caching consumes a request slot itself.\n\n## Frequently Asked Questions\n\n### How much VRAM does Escha-W2 actually need?\n\nThe weights alone are about 10.15 GB. With the shipped defaults (64k context, single stream), total peak usage on an RTX 4090 is around 18.1 to 18.3 GB, leaving comfortable margin on a 24 GB card. Pushing to 128k context raises peak usage to about 22.1 GB.\n\n### Does this model support multimodal input?\n\nNo. The configuration file references a vision tower inherited from the base architecture, but the quantized checkpoint is text-only.\n\n### What GPUs has this been verified on?\n\nEscha Labs tested on RTX 5090 (32 GB, sm_120), RTX 4090 (24 GB, sm_89), and RTX 3090 (24 GB, sm_86). A 16 GB card should work at reduced context but wasn’t tested.\n\n### Why does generation sometimes produce fluent but wrong answers?\n\nThis usually means the `transformers`\n\nlibrary is below version 5.8, which loads this model’s architecture with a different attention path without raising an error. Upgrading `transformers`\n\nresolves it.\n\n### What does the reasoning_effort setting change?\n\nIt controls a short instruction prepended to the model’s internal reasoning: `xhigh`\n\n(the default) asks it to validate assumptions and prioritize correctness, `medium`\n\nadds no steering at all, and `low`\n\nasks for brief, direct reasoning. It’s a prompt, not an enforced limit, so a genuinely hard problem can still produce a long response even at `low`\n\n. For guaranteed latency, a thinking budget that forces a cutoff is the reliable option.", "url": "https://wpnews.pro/news/run-qwen3-8-27b-escha-w2-on-a-24gb-gpu-with-sglang", "canonical_source": "https://www.mindstudio.ai/blog/run-escha-w2-sglang-consumer-gpu/", "published_at": "2026-08-25 00:00:00+00:00", "updated_at": "2026-08-25 22:13:32.261917+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure"], "entities": ["Escha Labs", "Qwen3.8-27B-Escha-W2", "Qwen3.8-27B", "SGLang", "RTX 3090", "RTX 5090"], "alternates": {"html": "https://wpnews.pro/news/run-qwen3-8-27b-escha-w2-on-a-24gb-gpu-with-sglang", "markdown": "https://wpnews.pro/news/run-qwen3-8-27b-escha-w2-on-a-24gb-gpu-with-sglang.md", "text": "https://wpnews.pro/news/run-qwen3-8-27b-escha-w2-on-a-24gb-gpu-with-sglang.txt", "jsonld": "https://wpnews.pro/news/run-qwen3-8-27b-escha-w2-on-a-24gb-gpu-with-sglang.jsonld"}}