{"slug": "qwen3-8-flash-next-4-05bpw-exl3-solo-launch-tabbyapi-exllamav3-5090-4090", "title": "Qwen3.8-Flash-Next 4.05bpw EXL3 solo launch (TabbyAPI + ExLlamaV3, 5090+4090)", "summary": "A developer published a serving configuration for the turboderp/Qwen3.8-Flash-Next-exl3-4.05bpw EXL3 quant, running the large MoE model split across an RTX 5090 and RTX 4090 with 192 GB of DDR5 system RAM via ExLlamaV3 v1.4.8 and TabbyAPI. The setup offloads 20 MoE layers to CPU and validates at 262K context with roughly 2,750 T/s prefill and 59 T/s decode, with an FP16-KV sibling config reaching about 2,370 T/s prefill and 55 T/s decode.", "body_md": "Serving config for the `turboderp/Qwen3.8-Flash-Next-exl3-4.05bpw` EXL3 quant as a single big-MoE model split across two GPUs + system RAM.\n\n**HW:** RTX 5090 (GPU0) + RTX 4090 (GPU1) + 192 GB DDR5, Ryzen 9 7950X3D (16c/32t). CUDA 13.3.\n**Engine pins:** ExLlamaV3 **v1.4.8** (`d21d38c`, dev branch), TabbyAPI `0.0.1 @ b75fe27`, run via `python main.py` from the tabbyAPI checkout.\n\n```\nexport CUDA_DEVICE_ORDER=PCI_BUS_ID\nexport CUDA_VISIBLE_DEVICES=0,1          # 0=5090, 1=4090\n\n# CPU MoE offload + D2H/compute handoff tuning\nexport EXL3_MOE_ARENA_HUGEPAGE=1         # hugepage-backed CPU expert arena\nexport EXL3_MOE_CPU_THREADS=24           # 24 sweet spot on 16c/32t (32 oversubscribes)\nexport EXL3_MOE_CPU_PIN=1                # pin to distinct physical cores\nexport EXL3_MOE_CPU_SWIZZLE=1            # AVX512-VBMI band-contiguous expert layout\nexport EXL3_MOE_CPU_STAGE_THREADS=8      # staging memcpy threads (default 4)\nexport EXL3_MOE_CPU_SLOT_ROWS=128        # CPU-tail rows/slot (default 64)\nexport EXL3_MOE_CPU_SLOTS=8              # job-ring depth (default 4)\nexport EXL3_MOE_STREAM_BATCH_EXPERTS=32  # experts per weight-staging DMA (default 24)\n\nexport EXL3_NGRAM_STREAM=0               # 39 GB n-gram table stays in RAM\n```\n\nValidated @262K: ~2,750 T/s prefill, ~59 T/s decode.\n\n```\ncd tabbyapi-cwd/          # CWD must hold the model-specific config.yml\npython <tabbyAPI>/main.py \\\n  --port 8002 \\\n  --host 127.0.0.1 \\\n  --disable-auth true \\\n  --model-dir /path/to/llms \\\n  --model-name turboderp--Qwen3.8-Flash-Next-exl3-4.05bpw \\\n  --backend exllamav3 \\\n  --prompt-template qwen3.8.jinja \\\n  --tensor-parallel false \\\n  --gpu-split 27 23 \\\n  --cpu-moe-offload-layers 20 \\\n  --max-seq-len 262144 \\\n  --cache-size 262144 \\\n  --cache-mode 8,8 \\\n  --chunk-size 7168 \\\n  --output-chunking true \\\n  --max-batch-size 1 \\\n  --vision false \\\n  --reasoning true \\\n  --tool-format qwen3_coder \\\n  --draft-mode mtp \\\n  --draft-num-tokens 1 \\\n  --dynamic-draft false\n```\n\n**FP16-KV sibling** (slower, ~2,370/~55): identical except `--cache-mode FP16` and `--cpu-moe-offload-layers 22` (Q8 frees ~2.7 GiB on GPU1 so 2 MoE layers come back to GPU).\n\n```\nnetwork: { host: 127.0.0.1, port: 5000, disable_auth: true }\nlogging: { log_prompt: false, log_generation_params: false }\nmodel:\n  model_dir: /path/to/llms\n  model_name: turboderp--Qwen3.8-Flash-Next-exl3-4.05bpw\n  backend: exllamav3\n  tensor_parallel: false\n  max_batch_size: 1\n  cache_mode: FP16          # fallback only; launcher CLI overrides\n  max_seq_len: -1\n  chunk_size: 2048          # fallback only\n  output_chunking: true\n  cpu_moe_split_experts: 0\n  vision: false\n  reasoning: true\n  tool_format: qwen3_coder\nsampling:\n  override_preset: defaults\n```\n\nPrecedence: `config.yml` < env < CLI — the launcher passes the real split/ctx/chunk/cache as CLI, so file values are fallbacks.\n\n`force: true` → overrides client-supplied sampling params.\n\n```\ntemperature:        { override: 1.0,  force: true }\ntop_k:              { override: 20,   force: true }\ntop_p:              { override: 0.95, force: true }\nmin_p:              { override: 0.0,  force: true }\npresence_penalty:   { override: 0.0,  force: true }\nrepetition_penalty: { override: 1.0,  force: true }\nmodels:\n  qwen38-flash-next-exl3-q8:\n    aliases: [solo]\n    proxy: \"http://127.0.0.1:${PORT}\"\n    checkEndpoint: /v1/model\n    useModelName: turboderp--Qwen3.8-Flash-Next-exl3-4.05bpw\n    ttl: 0\n    unloadTimeout: 30\n    healthCheckTimeout: 600     # ~68 GB load, ~10 min\n    concurrencyLimit: 1\n    env: [ ...same env block above... ]\n    cmd: /path/to/tabbyapi-qwen38-flash-next-q8-launcher.sh ${PORT}\n```\n\n- `--config` ignores CLI`--port` in TabbyAPI (early return in`config._from_args` ) — pass everything as CLI args.\n- Model auto-loads on startup, so `/v1/model` returns 503 until the ~68 GB load completes.\n- Q8 `cache-mode 8,8` quantizes only the 12 QSA layers' gathered K/V; indexer scoring stays fp16.", "url": "https://wpnews.pro/news/qwen3-8-flash-next-4-05bpw-exl3-solo-launch-tabbyapi-exllamav3-5090-4090", "canonical_source": "https://gist.github.com/cosmicnag/edcd2421506b0089bd143f8925b1e79d", "published_at": "2026-09-09 02:00:09+00:00", "updated_at": "2026-09-23 22:29:22.977203+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["Qwen3.8-Flash-Next", "ExLlamaV3", "TabbyAPI", "turboderp", "RTX 5090", "RTX 4090", "Ryzen 9 7950X3D"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/qwen3-8-flash-next-4-05bpw-exl3-solo-launch-tabbyapi-exllamav3-5090-4090", "markdown": "https://wpnews.pro/news/qwen3-8-flash-next-4-05bpw-exl3-solo-launch-tabbyapi-exllamav3-5090-4090.md", "text": "https://wpnews.pro/news/qwen3-8-flash-next-4-05bpw-exl3-solo-launch-tabbyapi-exllamav3-5090-4090.txt", "jsonld": "https://wpnews.pro/news/qwen3-8-flash-next-4-05bpw-exl3-solo-launch-tabbyapi-exllamav3-5090-4090.jsonld"}}