{"slug": "run-poolside-laguna-s-2-1-118b-moe-on-a-single-rtx-5090", "title": "Run Poolside Laguna S 2.1 (118B MoE) on a single RTX 5090", "summary": "Poolside's Laguna S 2.1, a 118B MoE coding model, runs on a single RTX 5090 (32 GB) at ~19 tok/s decode and ~60 tok/s prefill using auto-fit layer placement. A developer achieved this by packing full layers into VRAM with the `-ngl auto` and `--fit` flags, avoiding the slower CPU-only expert mode. The setup requires ~30 GB VRAM and 256k context, with a measured 40–45× speedup in prefill and 1.5–2× in decode over the default CPU placement.", "body_md": "**~19 tok/s decode · ~60 tok/s prefill · ~30 GB VRAM · 256k context**\n\nPoolside’s [Laguna S 2.1](https://huggingface.co/poolside/Laguna-S-2.1) is a **118B total / ~8B active** MoE coding model (256 experts, top-10 + shared, hybrid SWA). Official GGUFs: [ poolside/Laguna-S-2.1-GGUF](https://huggingface.co/poolside/Laguna-S-2.1-GGUF).\n\nPeople have been showing hybrid `llama.cpp`\n\nruns on 24 GB cards. That works — but the **default “all experts on CPU” placement** leaves a lot on the table. On a **5090 (32 GB)** the win is packing **full layers (experts included)** into VRAM with auto-fit, not pure `--cpu-moe`\n\n.\n\nThis is a measured recipe from one box, not marketing.\n\n| GPU | NVIDIA GeForce RTX 5090 (32 GB, SM_120 / Blackwell) |\n| RAM | 62 GB system + 128 GB swap |\n| OS | Ubuntu, CUDA 13.1 |\n| Model | `laguna-s-2.1-Q4_K_M.gguf` (~75 GB on disk) |\n| Runtime | Poolside\n`llama.cpp` branch `laguna` |\n\nYou need **~50–64+ GB system RAM** (more is happier). The GGUF alone is ~75 GB; whatever doesn’t fit in VRAM stays on the host.\n\n```\nllama-server -m laguna-s-2.1-Q4_K_M.gguf \\\n  -c 262144 -ngl 999 --cpu-moe \\\n  -fa on --no-mmap -t $(nproc) -b 4096 -ub 4096 \\\n  -ctk q8_0 -ctv q8_0\n```\n\nAttention/non-expert on GPU, **every expert on CPU**. Fits easily (~18 GB VRAM) but decode/prefill are host-bound.\n\n```\nllama-server -m laguna-s-2.1-Q4_K_M.gguf \\\n  -c 262144 \\\n  -ngl auto --fit on --fit-target 2048 \\\n  -fa on --jinja --no-mmap \\\n  -t $(nproc) -b 4096 -ub 4096 \\\n  -ctk q8_0 -ctv q8_0 \\\n  --temp 0.7 --top-p 0.95 --top-k 20 \\\n  --host 0.0.0.0 --port 8095\n```\n\n**Do not** set `-ngl 999`\n\n/ `all`\n\nwithout `--cpu-moe`\n\n. That tries to put the **entire 75 GB** on the GPU and **OOMs**.\n\n`-ngl auto`\n\n+ `--fit`\n\nfills free VRAM with as many **full layers** (including experts) as will fit and leaves the rest on host.\n\n`--fit-target 2048`\n\nleaves ~2 GB free (display / driver headroom on a desktop).\n\nShort coding prompts, thinking off, temp 0.2.\n\n| Mode | VRAM | Swap (peak class) | Prefill (cold) | Decode (cold) | Decode (warm) |\n|---|---|---|---|---|---|\n`--cpu-moe` + `-ngl 999` |\n~18 GB | ~60 GB | 1.3 t/s |\n8.2 t/s | 12.8 t/s |\n`-ngl auto` + `--fit` |\n~30 GB |\n~22 GB |\n~58 t/s |\n~19 t/s |\n~18 t/s |\n\nRough speedups on this machine:\n\n**Prefill: ~40–45×****Decode: ~1.5–2×**→ lands in a** fully usable**interactive range (~18–19 tok/s)\n\nGPU util during decode went from single-digit % to ~15%+ — still hybrid, but no longer “CPU thrashing while the 5090 naps.”\n\nCorrectness spot-checks (merge sorted lists, longest palindrome, binary-search infinite-loop fix) all passed.\n\nOfficial Laguna support is on Poolside’s fork (upstream PR: [ggml-org/llama.cpp#25165](https://github.com/ggml-org/llama.cpp/pull/25165)). Stock mainline may not load `architecture = laguna`\n\nyet.\n\n```\ngit clone --branch laguna --single-branch \\\n  https://github.com/poolsideai/llama.cpp.git\ncd llama.cpp\n\n# One-liner fix seen on GCC 15: std::isfinite needs <cmath>\n# (if build fails in common/speculative.cpp — add #include <cmath>)\n\ncmake -B build \\\n  -DGGML_CUDA=ON \\\n  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc \\\n  -DCMAKE_CUDA_ARCHITECTURES=120 \\\n  -DGGML_NATIVE=ON \\\n  -DCMAKE_BUILD_TYPE=Release\n\ncmake --build build -j$(nproc) --target llama-server\n# binary: build/bin/llama-server  (+ shared libs in same dir)\nexport LD_LIBRARY_PATH=\"$PWD/build/bin${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}\"\n```\n\nDownload weights:\n\n```\nhf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-Q4_K_M.gguf \\\n  --local-dir ./Laguna-S-2.1-GGUF\n```\n\nOptional DFlash draft (~2.2 GB) for speculative decode after the base path is stable:\n\n```\nhf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-DFlash-BF16.gguf \\\n  --local-dir ./Laguna-S-2.1-GGUF\ncurl -sS http://127.0.0.1:8095/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"messages\": [{\"role\":\"user\",\"content\":\"Write merge_sorted(a,b) in Python, O(n+m). Code only.\"}],\n    \"temperature\": 0.2,\n    \"max_tokens\": 400,\n    \"chat_template_kwargs\": {\"enable_thinking\": false}\n  }' | jq -r '.choices[0].message.content, .timings'\n```\n\nWatch:\n\n```\nwatch -n1 'nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv; free -h'\n```\n\nHealthy packing run: **~30 GB VRAM**, moderate swap (not 60+ GB), decode **~15–20 t/s**.\n\n**MoE sparsity**— only ~8B params active per token, but** all**expert weights must be addressable.** Hybrid SWA**— 36/48 layers sliding-window (512) + 8 KV heads →** 256k context is cheap**on KV.** Layer packing > expert exile**— putting*some*full MoE layers on the GPU beats keeping*all*experts on the host when you have 32 GB.— forces resident weight pages (helps once loaded; needs RAM/swap budget).`--no-mmap`\n\n**Large**— prefill batching (4096) matters a lot for prompt throughput.`-b`\n\n/`-ub`\n\nNVFP4 (~71 GB) is great on multi-GPU / 128 GB unified (DGX Spark recipe). It does **not** magically fit a 32 GB card full-GPU. **GGUF hybrid is the single-consumer-GPU path.**\n\n**62 GB RAM is tight.** Expect**some swap**. 96–128 GB host RAM would be much smoother (closer to viral “1k prefill” demos).** Agent UIs (e.g. full tool schemas)**can send** 5–10k+ token**system prompts. At ~60 t/s prefill that’s still** 1–3 minutes**to first token on a cold session — not a hang. Same-session** prefix cache**helps a lot after turn 1.** Thinking mode**can burn hundreds–thousands of tokens before visible answer. For latency tests:`\"chat_template_kwargs\": {\"enable_thinking\": false}`\n\n.- GGUF ships configured for\n**256k**. Native training goes to 1M; Poolside documents YaRN overrides and quality caveats for >256k. - First load of 75 GB can take\n**~1 minute**(disk + page-in) before`/v1/models`\n\nis ready even if`/health`\n\nanswers early. - Desktop safety: leave a couple GB VRAM free (\n`--fit-target 2048`\n\n) so the display doesn’t wedge under memory pressure.\n\nSame recipe applies with less headroom:\n\n- Prefer\nover pure`-ngl auto --fit`\n\n`--cpu-moe`\n\nif anything fits. - Expect\n**lower** layer count on GPU → closer to the slow column, still better than all-experts-host if fit places*any*expert layers on-device. - Stay under VRAM ceiling; don’t force\n`-ngl 999`\n\nwithout fit.\n\n- Model:\n[https://huggingface.co/poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) - GGUF:\n[https://huggingface.co/poolside/Laguna-S-2.1-GGUF](https://huggingface.co/poolside/Laguna-S-2.1-GGUF) - NVFP4 (multi-GPU / Spark):\n[https://huggingface.co/poolside/Laguna-S-2.1-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4) - llama.cpp (Laguna):\n[https://github.com/poolsideai/llama.cpp/tree/laguna](https://github.com/poolsideai/llama.cpp/tree/laguna) - License: OpenMDW-1.1\n\n```\nLaguna S 2.1 Q4_K_M + poolside llama.cpp (laguna branch)\n  -ngl auto --fit on --fit-target 2048\n  -fa on --no-mmap -b 4096 -ub 4096 -ctk q8_0 -ctv q8_0\n  -c 262144\n\nRTX 5090 32GB + 62GB RAM:\n  ~30 GB VRAM · ~18–19 tok/s decode · ~40–60 tok/s prefill\n  vs all-experts-on-CPU: ~45× prefill, ~2× decode\n```\n\n**The VRAM limit isn’t fake — but putting the right tensors in it matters more than the “all MoE on CPU” cheat code.** Pack full layers with auto-fit; keep the rest on host. That’s what made 118B-class agentic coding actually feel usable on one 5090. Big win at any rate.", "url": "https://wpnews.pro/news/run-poolside-laguna-s-2-1-118b-moe-on-a-single-rtx-5090", "canonical_source": "https://gist.github.com/solatticus/8b610a7fc8726786b0fe4d43bb4971ed", "published_at": "2026-07-22 23:05:33+00:00", "updated_at": "2026-07-24 17:30:43.347924+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Poolside", "Laguna S 2.1", "RTX 5090", "NVIDIA", "llama.cpp", "CUDA", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/run-poolside-laguna-s-2-1-118b-moe-on-a-single-rtx-5090", "markdown": "https://wpnews.pro/news/run-poolside-laguna-s-2-1-118b-moe-on-a-single-rtx-5090.md", "text": "https://wpnews.pro/news/run-poolside-laguna-s-2-1-118b-moe-on-a-single-rtx-5090.txt", "jsonld": "https://wpnews.pro/news/run-poolside-laguna-s-2-1-118b-moe-on-a-single-rtx-5090.jsonld"}}