Run Poolside Laguna S 2.1 (118B MoE) on a single RTX 5090

~19 tok/s decode · ~60 tok/s prefill · ~30 GB VRAM · 256k context

Poolside’s Laguna S 2.1 is a 118B total / ~8B active MoE coding model (256 experts, top-10 + shared, hybrid SWA). Official GGUFs: poolside/Laguna-S-2.1-GGUF.

People have been showing hybrid llama.cpp

runs on 24 GB cards. That works — but the default “all experts on CPU” placement leaves a lot on the table. On a 5090 (32 GB) the win is packing full layers (experts included) into VRAM with auto-fit, not pure --cpu-moe

.

This is a measured recipe from one box, not marketing.

| GPU | NVIDIA GeForce RTX 5090 (32 GB, SM_120 / Blackwell) | | RAM | 62 GB system + 128 GB swap | | OS | Ubuntu, CUDA 13.1 | | Model | laguna-s-2.1-Q4_K_M.gguf (~75 GB on disk) | | Runtime | Poolside llama.cpp branch laguna |

You need ~50–64+ GB system RAM (more is happier). The GGUF alone is ~75 GB; whatever doesn’t fit in VRAM stays on the host.

llama-server -m laguna-s-2.1-Q4_K_M.gguf \
  -c 262144 -ngl 999 --cpu-moe \
  -fa on --no-mmap -t $(nproc) -b 4096 -ub 4096 \
  -ctk q8_0 -ctv q8_0

Attention/non-expert on GPU, every expert on CPU. Fits easily (~18 GB VRAM) but decode/prefill are host-bound.

llama-server -m laguna-s-2.1-Q4_K_M.gguf \
  -c 262144 \
  -ngl auto --fit on --fit-target 2048 \
  -fa on --jinja --no-mmap \
  -t $(nproc) -b 4096 -ub 4096 \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.7 --top-p 0.95 --top-k 20 \
  --host 0.0.0.0 --port 8095

Do not set -ngl 999

/ all

without --cpu-moe

. That tries to put the entire 75 GB on the GPU and OOMs.

-ngl auto

• --fit

fills free VRAM with as many full layers (including experts) as will fit and leaves the rest on host.

--fit-target 2048

leaves ~2 GB free (display / driver headroom on a desktop).

Short coding prompts, thinking off, temp 0.2.

Rough speedups on this machine:

Prefill: ~40–45×****Decode: ~1.5–2×→ lands in a** fully usable**interactive range (~18–19 tok/s)

GPU util during decode went from single-digit % to ~15%+ — still hybrid, but no longer “CPU thrashing while the 5090 naps.”

Correctness spot-checks (merge sorted lists, longest palindrome, binary-search infinite-loop fix) all passed.

Official Laguna support is on Poolside’s fork (upstream PR: ggml-org/llama.cpp#25165). Stock mainline may not load architecture = laguna

yet.

git clone --branch laguna --single-branch \
  https://github.com/poolsideai/llama.cpp.git
cd llama.cpp


cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=120 \
  -DGGML_NATIVE=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build -j$(nproc) --target llama-server
export LD_LIBRARY_PATH="$PWD/build/bin${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

Download weights:

hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-Q4_K_M.gguf \
  --local-dir ./Laguna-S-2.1-GGUF

Optional DFlash draft (~2.2 GB) for speculative decode after the base path is stable:

hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-DFlash-BF16.gguf \
  --local-dir ./Laguna-S-2.1-GGUF
curl -sS http://127.0.0.1:8095/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role":"user","content":"Write merge_sorted(a,b) in Python, O(n+m). Code only."}],
    "temperature": 0.2,
    "max_tokens": 400,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq -r '.choices[0].message.content, .timings'

Watch:

watch -n1 'nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv; free -h'

Healthy packing run: ~30 GB VRAM, moderate swap (not 60+ GB), decode ~15–20 t/s.

MoE sparsity— only ~8B params active per token, but** allexpert weights must be addressable. Hybrid SWA**— 36/48 layers sliding-window (512) + 8 KV heads →** 256k context is cheapon KV. Layer packing > expert exile**— puttingsomefull MoE layers on the GPU beats keepingallexperts on the host when you have 32 GB.— forces resident weight pages (helps once loaded; needs RAM/swap budget).--no-mmap

Large— prefill batching (4096) matters a lot for prompt throughput.-b

/-ub

NVFP4 (~71 GB) is great on multi-GPU / 128 GB unified (DGX Spark recipe). It does not magically fit a 32 GB card full-GPU. GGUF hybrid is the single-consumer-GPU path.

62 GB RAM is tight. Expectsome swap. 96–128 GB host RAM would be much smoother (closer to viral “1k prefill” demos).** Agent UIs (e.g. full tool schemas)can send 5–10k+ tokensystem prompts. At ~60 t/s prefill that’s still 1–3 minutesto first token on a cold session — not a hang. Same-session prefix cachehelps a lot after turn 1. Thinking mode**can burn hundreds–thousands of tokens before visible answer. For latency tests:"chat_template_kwargs": {"enable_thinking": false}

.- GGUF ships configured for 256k. Native training goes to 1M; Poolside documents YaRN overrides and quality caveats for >256k. - First load of 75 GB can take ~1 minute(disk + page-in) before/v1/models

is ready even if/health

answers early. - Desktop safety: leave a couple GB VRAM free ( --fit-target 2048

) so the display doesn’t wedge under memory pressure.

Same recipe applies with less headroom:

• Prefer over pure-ngl auto --fit

--cpu-moe

if anything fits. - Expect lower layer count on GPU → closer to the slow column, still better than all-experts-host if fit placesanyexpert layers on-device. - Stay under VRAM ceiling; don’t force -ngl 999

without fit.

• Model: https://huggingface.co/poolside/Laguna-S-2.1 - GGUF: https://huggingface.co/poolside/Laguna-S-2.1-GGUF - NVFP4 (multi-GPU / Spark): https://huggingface.co/poolside/Laguna-S-2.1-NVFP4 - llama.cpp (Laguna): https://github.com/poolsideai/llama.cpp/tree/laguna - License: OpenMDW-1.1

Laguna S 2.1 Q4_K_M + poolside llama.cpp (laguna branch)
  -ngl auto --fit on --fit-target 2048
  -fa on --no-mmap -b 4096 -ub 4096 -ctk q8_0 -ctv q8_0
  -c 262144

RTX 5090 32GB + 62GB RAM:
  ~30 GB VRAM · ~18–19 tok/s decode · ~40–60 tok/s prefill
  vs all-experts-on-CPU: ~45× prefill, ~2× decode

The VRAM limit isn’t fake — but putting the right tensors in it matters more than the “all MoE on CPU” cheat code. Pack full layers with auto-fit; keep the rest on host. That’s what made 118B-class agentic coding actually feel usable on one 5090. Big win at any rate.