{"slug": "qwen-3-6-config-example", "title": "Qwen 3.6 config example", "summary": "A developer shared a configuration example for running Qwen 3.6 with llama-turboquant, a Docker-based CUDA inference container. The setup includes GPU offloading, flash attention, speculative decoding with MTP and ngram-mod, and memory limits of 30GB RAM with 46GB swap.", "body_md": "| #!/usr/bin/env bash | |\n| set -euo pipefail | |\n| cd \"$(dirname \"$0\")\" | |\n| export CUDA_MALLOC_ASYNC_SUPPORTED=1 | |\n| export GGML_CUDA_FORCE_MMQ=1 | |\n| IMAGE=\"${IMAGE:-llama-turboquant:cuda}\" | |\n| HOST_PORT=\"${HOST_PORT:-8080}\" | |\n| NETWORK=\"${NETWORK:-runner-network}\" | |\n| STATIC_IP=\"${STATIC_IP:-172.18.0.10}\" | |\n| # 65 layers | |\n| MODEL_FILE=\"${MODEL_FILE:-Qwen3.6-27B-NVFP4-MTP.gguf}\" | |\n| MMPROJ_FILE=\"${MMPROJ_FILE:-mmproj-Qwen3.6-27B-F16.gguf}\" | |\n| N_GPU_LAYERS=\"${N_GPU_LAYERS:-999}\" | |\n| NAME=\"${NAME:-llama-turboquant}\" | |\n| if docker inspect \"${NAME}\" >/dev/null 2>&1; then | |\n| docker rm -f \"${NAME}\" >/dev/null | |\n| fi | |\n| docker create \\ | |\n| --name \"${NAME}\" \\ | |\n| --restart=unless-stopped \\ | |\n| --gpus all \\ | |\n| -e CUDA_DEVICE_ORDER=PCI_BUS_ID \\ | |\n| -e CUDA_VISIBLE_DEVICES=1,0 \\ | |\n| --memory=30g \\ | |\n| --memory-swap=46g \\ | |\n| --cap-add=IPC_LOCK \\ | |\n| --ulimit memlock=-1:-1 \\ | |\n| --ulimit core=0 \\ | |\n| -e TURBO_AUTO_ASYMMETRIC=0 \\ | |\n| -p \"${HOST_PORT}:8080\" \\ | |\n| --network \"${NETWORK}\" \\ | |\n| --ip \"${STATIC_IP}\" \\ | |\n| -v \"$(pwd)/models:/models:ro\" \\ | |\n| -v \"$(pwd)/scripts:/scripts:ro\" \\ | |\n| --entrypoint /scripts/entrypoint.sh \\ | |\n| \"${IMAGE}\" \\ | |\n| --model \"/models/${MODEL_FILE}\" \\ | |\n| --mmproj \"/models/${MMPROJ_FILE}\" \\ | |\n| --mmproj-offload \\ | |\n| --host 0.0.0.0 \\ | |\n| --port 8080 \\ | |\n| --metrics \\ | |\n| --n-gpu-layers \"${N_GPU_LAYERS}\" \\ | |\n| --main-gpu 0 \\ | |\n| --split-mode layer \\ | |\n| --tensor-split 40,25 \\ | |\n| -fit off \\ | |\n| --flash-attn on \\ | |\n| -c 120000 \\ | |\n| -n -1 \\ | |\n| --parallel 1 \\ | |\n| -ctk q8_0 \\ | |\n| -ctv turbo3 \\ | |\n| -ctkd q8_0 \\ | |\n| -ctvd turbo3 \\ | |\n| --kv-unified \\ | |\n| --no-mmap \\ | |\n| --mlock \\ | |\n| --jinja \\ | |\n| --reasoning off \\ | |\n| --spec-type draft-mtp,ngram-mod \\ | |\n| --spec-draft-n-max 3 \\ | |\n| --spec-ngram-mod-n-match 24 \\ | |\n| --spec-ngram-mod-n-min 4 \\ | |\n| --spec-ngram-mod-n-max 48 \\ | |\n| --temp 0.7 \\ | |\n| --top-p 0.8 \\ | |\n| --top-k 20 \\ | |\n| --min-p 0.0 \\ | |\n| --presence-penalty 1.5 \\ | |\n| --repeat-penalty 1.0 \\ | |\n| -b 1024 \\ | |\n| -ub 512 \\ | |\n| --cache-idle-slots \\ | |\n| --cache-ram 16384 \\ | |\n| --cache-reuse 256 \\ | |\n| --threads 8 \\ | |\n| --cpu-range 0-7 \\ | |\n| --timeout 360 \\ | |\n| \"$@\" >/dev/null | |\n| docker start -a \"${NAME}\" |", "url": "https://wpnews.pro/news/qwen-3-6-config-example", "canonical_source": "https://gist.github.com/mjasnikovs/fe28b4fb87dc7c9f5cfe135829dae672", "published_at": "2026-07-17 10:43:41+00:00", "updated_at": "2026-07-20 19:36:54.286210+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Qwen", "llama-turboquant", "CUDA", "Docker", "GGML"], "alternates": {"html": "https://wpnews.pro/news/qwen-3-6-config-example", "markdown": "https://wpnews.pro/news/qwen-3-6-config-example.md", "text": "https://wpnews.pro/news/qwen-3-6-config-example.txt", "jsonld": "https://wpnews.pro/news/qwen-3-6-config-example.jsonld"}}