Run Poolside Laguna S 2.1 (118B MoE) on a single RTX 5090

Poolside's Laguna S 2.1, a 118B MoE coding model, runs on a single RTX 5090 (32 GB) at ~19 tok/s decode and ~60 tok/s prefill using auto-fit layer placement. A developer achieved this by packing full layers into VRAM with the `-ngl auto` and `--fit` flags, avoiding the slower CPU-only expert mode. The setup requires ~30 GB VRAM and 256k context, with a measured 40–45× speedup in prefill and 1.5–2× in decode over the default CPU placement.

~19 tok/s decode · ~60 tok/s prefill · ~30 GB VRAM · 256k context Poolside’s Laguna S 2.1 https://huggingface.co/poolside/Laguna-S-2.1 is a 118B total / ~8B active MoE coding model 256 experts, top-10 + shared, hybrid SWA . Official GGUFs: poolside/Laguna-S-2.1-GGUF https://huggingface.co/poolside/Laguna-S-2.1-GGUF . People have been showing hybrid llama.cpp runs on 24 GB cards. That works — but the default “all experts on CPU” placement leaves a lot on the table. On a 5090 32 GB the win is packing full layers experts included into VRAM with auto-fit, not pure --cpu-moe . This is a measured recipe from one box, not marketing. | GPU | NVIDIA GeForce RTX 5090 32 GB, SM 120 / Blackwell | | RAM | 62 GB system + 128 GB swap | | OS | Ubuntu, CUDA 13.1 | | Model | laguna-s-2.1-Q4 K M.gguf ~75 GB on disk | | Runtime | Poolside llama.cpp branch laguna | You need ~50–64+ GB system RAM more is happier . The GGUF alone is ~75 GB; whatever doesn’t fit in VRAM stays on the host. llama-server -m laguna-s-2.1-Q4 K M.gguf \ -c 262144 -ngl 999 --cpu-moe \ -fa on --no-mmap -t $ nproc -b 4096 -ub 4096 \ -ctk q8 0 -ctv q8 0 Attention/non-expert on GPU, every expert on CPU . Fits easily ~18 GB VRAM but decode/prefill are host-bound. llama-server -m laguna-s-2.1-Q4 K M.gguf \ -c 262144 \ -ngl auto --fit on --fit-target 2048 \ -fa on --jinja --no-mmap \ -t $ nproc -b 4096 -ub 4096 \ -ctk q8 0 -ctv q8 0 \ --temp 0.7 --top-p 0.95 --top-k 20 \ --host 0.0.0.0 --port 8095 Do not set -ngl 999 / all without --cpu-moe . That tries to put the entire 75 GB on the GPU and OOMs . -ngl auto + --fit fills free VRAM with as many full layers including experts as will fit and leaves the rest on host. --fit-target 2048 leaves ~2 GB free display / driver headroom on a desktop . Short coding prompts, thinking off, temp 0.2. | Mode | VRAM | Swap peak class | Prefill cold | Decode cold | Decode warm | |---|---|---|---|---|---| --cpu-moe + -ngl 999 | ~18 GB | ~60 GB | 1.3 t/s | 8.2 t/s | 12.8 t/s | -ngl auto + --fit | ~30 GB | ~22 GB | ~58 t/s | ~19 t/s | ~18 t/s | Rough speedups on this machine: Prefill: ~40–45× Decode: ~1.5–2× → lands in a fully usable interactive range ~18–19 tok/s GPU util during decode went from single-digit % to ~15%+ — still hybrid, but no longer “CPU thrashing while the 5090 naps.” Correctness spot-checks merge sorted lists, longest palindrome, binary-search infinite-loop fix all passed. Official Laguna support is on Poolside’s fork upstream PR: ggml-org/llama.cpp 25165 https://github.com/ggml-org/llama.cpp/pull/25165 . Stock mainline may not load architecture = laguna yet. git clone --branch laguna --single-branch \ https://github.com/poolsideai/llama.cpp.git cd llama.cpp One-liner fix seen on GCC 15: std::isfinite needs <cmath if build fails in common/speculative.cpp — add include <cmath cmake -B build \ -DGGML CUDA=ON \ -DCMAKE CUDA COMPILER=/usr/local/cuda-13.1/bin/nvcc \ -DCMAKE CUDA ARCHITECTURES=120 \ -DGGML NATIVE=ON \ -DCMAKE BUILD TYPE=Release cmake --build build -j$ nproc --target llama-server binary: build/bin/llama-server + shared libs in same dir export LD LIBRARY PATH="$PWD/build/bin${LD LIBRARY PATH:+:$LD LIBRARY PATH}" Download weights: hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-Q4 K M.gguf \ --local-dir ./Laguna-S-2.1-GGUF Optional DFlash draft ~2.2 GB for speculative decode after the base path is stable: hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-DFlash-BF16.gguf \ --local-dir ./Laguna-S-2.1-GGUF curl -sS http://127.0.0.1:8095/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "messages": {"role":"user","content":"Write merge sorted a,b in Python, O n+m . Code only."} , "temperature": 0.2, "max tokens": 400, "chat template kwargs": {"enable thinking": false} }' | jq -r '.choices 0 .message.content, .timings' Watch: watch -n1 'nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv; free -h' Healthy packing run: ~30 GB VRAM , moderate swap not 60+ GB , decode ~15–20 t/s . MoE sparsity — only ~8B params active per token, but all expert weights must be addressable. Hybrid SWA — 36/48 layers sliding-window 512 + 8 KV heads → 256k context is cheap on KV. Layer packing expert exile — putting some full MoE layers on the GPU beats keeping all experts on the host when you have 32 GB.— forces resident weight pages helps once loaded; needs RAM/swap budget . --no-mmap Large — prefill batching 4096 matters a lot for prompt throughput. -b / -ub NVFP4 ~71 GB is great on multi-GPU / 128 GB unified DGX Spark recipe . It does not magically fit a 32 GB card full-GPU. GGUF hybrid is the single-consumer-GPU path. 62 GB RAM is tight. Expect some swap . 96–128 GB host RAM would be much smoother closer to viral “1k prefill” demos . Agent UIs e.g. full tool schemas can send 5–10k+ token system prompts. At ~60 t/s prefill that’s still 1–3 minutes to first token on a cold session — not a hang. Same-session prefix cache helps a lot after turn 1. Thinking mode can burn hundreds–thousands of tokens before visible answer. For latency tests: "chat template kwargs": {"enable thinking": false} .- GGUF ships configured for 256k . Native training goes to 1M; Poolside documents YaRN overrides and quality caveats for 256k. - First load of 75 GB can take ~1 minute disk + page-in before /v1/models is ready even if /health answers early. - Desktop safety: leave a couple GB VRAM free --fit-target 2048 so the display doesn’t wedge under memory pressure. Same recipe applies with less headroom: - Prefer over pure -ngl auto --fit --cpu-moe if anything fits. - Expect lower layer count on GPU → closer to the slow column, still better than all-experts-host if fit places any expert layers on-device. - Stay under VRAM ceiling; don’t force -ngl 999 without fit. - Model: https://huggingface.co/poolside/Laguna-S-2.1 https://huggingface.co/poolside/Laguna-S-2.1 - GGUF: https://huggingface.co/poolside/Laguna-S-2.1-GGUF https://huggingface.co/poolside/Laguna-S-2.1-GGUF - NVFP4 multi-GPU / Spark : https://huggingface.co/poolside/Laguna-S-2.1-NVFP4 https://huggingface.co/poolside/Laguna-S-2.1-NVFP4 - llama.cpp Laguna : https://github.com/poolsideai/llama.cpp/tree/laguna https://github.com/poolsideai/llama.cpp/tree/laguna - License: OpenMDW-1.1 Laguna S 2.1 Q4 K M + poolside llama.cpp laguna branch -ngl auto --fit on --fit-target 2048 -fa on --no-mmap -b 4096 -ub 4096 -ctk q8 0 -ctv q8 0 -c 262144 RTX 5090 32GB + 62GB RAM: ~30 GB VRAM · ~18–19 tok/s decode · ~40–60 tok/s prefill vs all-experts-on-CPU: ~45× prefill, ~2× decode The VRAM limit isn’t fake — but putting the right tensors in it matters more than the “all MoE on CPU” cheat code. Pack full layers with auto-fit; keep the rest on host. That’s what made 118B-class agentic coding actually feel usable on one 5090. Big win at any rate.