# Qwen3.8-Flash-Next 4.05bpw EXL3 solo launch (TabbyAPI + ExLlamaV3, 5090+4090)

> Source: <https://gist.github.com/cosmicnag/edcd2421506b0089bd143f8925b1e79d>
> Published: 2026-09-09 02:00:09+00:00

Serving config for the `turboderp/Qwen3.8-Flash-Next-exl3-4.05bpw` EXL3 quant as a single big-MoE model split across two GPUs + system RAM.

**HW:** RTX 5090 (GPU0) + RTX 4090 (GPU1) + 192 GB DDR5, Ryzen 9 7950X3D (16c/32t). CUDA 13.3.
**Engine pins:** ExLlamaV3 **v1.4.8** (`d21d38c`, dev branch), TabbyAPI `0.0.1 @ b75fe27`, run via `python main.py` from the tabbyAPI checkout.

```
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1          # 0=5090, 1=4090

# CPU MoE offload + D2H/compute handoff tuning
export EXL3_MOE_ARENA_HUGEPAGE=1         # hugepage-backed CPU expert arena
export EXL3_MOE_CPU_THREADS=24           # 24 sweet spot on 16c/32t (32 oversubscribes)
export EXL3_MOE_CPU_PIN=1                # pin to distinct physical cores
export EXL3_MOE_CPU_SWIZZLE=1            # AVX512-VBMI band-contiguous expert layout
export EXL3_MOE_CPU_STAGE_THREADS=8      # staging memcpy threads (default 4)
export EXL3_MOE_CPU_SLOT_ROWS=128        # CPU-tail rows/slot (default 64)
export EXL3_MOE_CPU_SLOTS=8              # job-ring depth (default 4)
export EXL3_MOE_STREAM_BATCH_EXPERTS=32  # experts per weight-staging DMA (default 24)

export EXL3_NGRAM_STREAM=0               # 39 GB n-gram table stays in RAM
```

Validated @262K: ~2,750 T/s prefill, ~59 T/s decode.

```
cd tabbyapi-cwd/          # CWD must hold the model-specific config.yml
python <tabbyAPI>/main.py \
  --port 8002 \
  --host 127.0.0.1 \
  --disable-auth true \
  --model-dir /path/to/llms \
  --model-name turboderp--Qwen3.8-Flash-Next-exl3-4.05bpw \
  --backend exllamav3 \
  --prompt-template qwen3.8.jinja \
  --tensor-parallel false \
  --gpu-split 27 23 \
  --cpu-moe-offload-layers 20 \
  --max-seq-len 262144 \
  --cache-size 262144 \
  --cache-mode 8,8 \
  --chunk-size 7168 \
  --output-chunking true \
  --max-batch-size 1 \
  --vision false \
  --reasoning true \
  --tool-format qwen3_coder \
  --draft-mode mtp \
  --draft-num-tokens 1 \
  --dynamic-draft false
```

**FP16-KV sibling** (slower, ~2,370/~55): identical except `--cache-mode FP16` and `--cpu-moe-offload-layers 22` (Q8 frees ~2.7 GiB on GPU1 so 2 MoE layers come back to GPU).

```
network: { host: 127.0.0.1, port: 5000, disable_auth: true }
logging: { log_prompt: false, log_generation_params: false }
model:
  model_dir: /path/to/llms
  model_name: turboderp--Qwen3.8-Flash-Next-exl3-4.05bpw
  backend: exllamav3
  tensor_parallel: false
  max_batch_size: 1
  cache_mode: FP16          # fallback only; launcher CLI overrides
  max_seq_len: -1
  chunk_size: 2048          # fallback only
  output_chunking: true
  cpu_moe_split_experts: 0
  vision: false
  reasoning: true
  tool_format: qwen3_coder
sampling:
  override_preset: defaults
```

Precedence: `config.yml` < env < CLI — the launcher passes the real split/ctx/chunk/cache as CLI, so file values are fallbacks.

`force: true` → overrides client-supplied sampling params.

```
temperature:        { override: 1.0,  force: true }
top_k:              { override: 20,   force: true }
top_p:              { override: 0.95, force: true }
min_p:              { override: 0.0,  force: true }
presence_penalty:   { override: 0.0,  force: true }
repetition_penalty: { override: 1.0,  force: true }
models:
  qwen38-flash-next-exl3-q8:
    aliases: [solo]
    proxy: "http://127.0.0.1:${PORT}"
    checkEndpoint: /v1/model
    useModelName: turboderp--Qwen3.8-Flash-Next-exl3-4.05bpw
    ttl: 0
    unloadTimeout: 30
    healthCheckTimeout: 600     # ~68 GB load, ~10 min
    concurrencyLimit: 1
    env: [ ...same env block above... ]
    cmd: /path/to/tabbyapi-qwen38-flash-next-q8-launcher.sh ${PORT}
```

- `--config` ignores CLI`--port` in TabbyAPI (early return in`config._from_args` ) — pass everything as CLI args.
- Model auto-loads on startup, so `/v1/model` returns 503 until the ~68 GB load completes.
- Q8 `cache-mode 8,8` quantizes only the 12 QSA layers' gathered K/V; indexer scoring stays fp16.