# Run Poolside Laguna S 2.1 (118B MoE) on a single RTX 5090

> Source: <https://gist.github.com/solatticus/8b610a7fc8726786b0fe4d43bb4971ed>
> Published: 2026-07-22 23:05:33+00:00

**~19 tok/s decode · ~60 tok/s prefill · ~30 GB VRAM · 256k context**

Poolside’s [Laguna S 2.1](https://huggingface.co/poolside/Laguna-S-2.1) is a **118B total / ~8B active** MoE coding model (256 experts, top-10 + shared, hybrid SWA). Official GGUFs: [ poolside/Laguna-S-2.1-GGUF](https://huggingface.co/poolside/Laguna-S-2.1-GGUF).

People have been showing hybrid `llama.cpp`

runs on 24 GB cards. That works — but the **default “all experts on CPU” placement** leaves a lot on the table. On a **5090 (32 GB)** the win is packing **full layers (experts included)** into VRAM with auto-fit, not pure `--cpu-moe`

.

This is a measured recipe from one box, not marketing.

| GPU | NVIDIA GeForce RTX 5090 (32 GB, SM_120 / Blackwell) |
| RAM | 62 GB system + 128 GB swap |
| OS | Ubuntu, CUDA 13.1 |
| Model | `laguna-s-2.1-Q4_K_M.gguf` (~75 GB on disk) |
| Runtime | Poolside
`llama.cpp` branch `laguna` |

You need **~50–64+ GB system RAM** (more is happier). The GGUF alone is ~75 GB; whatever doesn’t fit in VRAM stays on the host.

```
llama-server -m laguna-s-2.1-Q4_K_M.gguf \
  -c 262144 -ngl 999 --cpu-moe \
  -fa on --no-mmap -t $(nproc) -b 4096 -ub 4096 \
  -ctk q8_0 -ctv q8_0
```

Attention/non-expert on GPU, **every expert on CPU**. Fits easily (~18 GB VRAM) but decode/prefill are host-bound.

```
llama-server -m laguna-s-2.1-Q4_K_M.gguf \
  -c 262144 \
  -ngl auto --fit on --fit-target 2048 \
  -fa on --jinja --no-mmap \
  -t $(nproc) -b 4096 -ub 4096 \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.7 --top-p 0.95 --top-k 20 \
  --host 0.0.0.0 --port 8095
```

**Do not** set `-ngl 999`

/ `all`

without `--cpu-moe`

. That tries to put the **entire 75 GB** on the GPU and **OOMs**.

`-ngl auto`

+ `--fit`

fills free VRAM with as many **full layers** (including experts) as will fit and leaves the rest on host.

`--fit-target 2048`

leaves ~2 GB free (display / driver headroom on a desktop).

Short coding prompts, thinking off, temp 0.2.

| Mode | VRAM | Swap (peak class) | Prefill (cold) | Decode (cold) | Decode (warm) |
|---|---|---|---|---|---|
`--cpu-moe` + `-ngl 999` |
~18 GB | ~60 GB | 1.3 t/s |
8.2 t/s | 12.8 t/s |
`-ngl auto` + `--fit` |
~30 GB |
~22 GB |
~58 t/s |
~19 t/s |
~18 t/s |

Rough speedups on this machine:

**Prefill: ~40–45×****Decode: ~1.5–2×**→ lands in a** fully usable**interactive range (~18–19 tok/s)

GPU util during decode went from single-digit % to ~15%+ — still hybrid, but no longer “CPU thrashing while the 5090 naps.”

Correctness spot-checks (merge sorted lists, longest palindrome, binary-search infinite-loop fix) all passed.

Official Laguna support is on Poolside’s fork (upstream PR: [ggml-org/llama.cpp#25165](https://github.com/ggml-org/llama.cpp/pull/25165)). Stock mainline may not load `architecture = laguna`

yet.

```
git clone --branch laguna --single-branch \
  https://github.com/poolsideai/llama.cpp.git
cd llama.cpp

# One-liner fix seen on GCC 15: std::isfinite needs <cmath>
# (if build fails in common/speculative.cpp — add #include <cmath>)

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=120 \
  -DGGML_NATIVE=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build -j$(nproc) --target llama-server
# binary: build/bin/llama-server  (+ shared libs in same dir)
export LD_LIBRARY_PATH="$PWD/build/bin${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
```

Download weights:

```
hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-Q4_K_M.gguf \
  --local-dir ./Laguna-S-2.1-GGUF
```

Optional DFlash draft (~2.2 GB) for speculative decode after the base path is stable:

```
hf download poolside/Laguna-S-2.1-GGUF laguna-s-2.1-DFlash-BF16.gguf \
  --local-dir ./Laguna-S-2.1-GGUF
curl -sS http://127.0.0.1:8095/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role":"user","content":"Write merge_sorted(a,b) in Python, O(n+m). Code only."}],
    "temperature": 0.2,
    "max_tokens": 400,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq -r '.choices[0].message.content, .timings'
```

Watch:

```
watch -n1 'nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv; free -h'
```

Healthy packing run: **~30 GB VRAM**, moderate swap (not 60+ GB), decode **~15–20 t/s**.

**MoE sparsity**— only ~8B params active per token, but** all**expert weights must be addressable.** Hybrid SWA**— 36/48 layers sliding-window (512) + 8 KV heads →** 256k context is cheap**on KV.** Layer packing > expert exile**— putting*some*full MoE layers on the GPU beats keeping*all*experts on the host when you have 32 GB.— forces resident weight pages (helps once loaded; needs RAM/swap budget).`--no-mmap`

**Large**— prefill batching (4096) matters a lot for prompt throughput.`-b`

/`-ub`

NVFP4 (~71 GB) is great on multi-GPU / 128 GB unified (DGX Spark recipe). It does **not** magically fit a 32 GB card full-GPU. **GGUF hybrid is the single-consumer-GPU path.**

**62 GB RAM is tight.** Expect**some swap**. 96–128 GB host RAM would be much smoother (closer to viral “1k prefill” demos).** Agent UIs (e.g. full tool schemas)**can send** 5–10k+ token**system prompts. At ~60 t/s prefill that’s still** 1–3 minutes**to first token on a cold session — not a hang. Same-session** prefix cache**helps a lot after turn 1.** Thinking mode**can burn hundreds–thousands of tokens before visible answer. For latency tests:`"chat_template_kwargs": {"enable_thinking": false}`

.- GGUF ships configured for
**256k**. Native training goes to 1M; Poolside documents YaRN overrides and quality caveats for >256k. - First load of 75 GB can take
**~1 minute**(disk + page-in) before`/v1/models`

is ready even if`/health`

answers early. - Desktop safety: leave a couple GB VRAM free (
`--fit-target 2048`

) so the display doesn’t wedge under memory pressure.

Same recipe applies with less headroom:

- Prefer
over pure`-ngl auto --fit`

`--cpu-moe`

if anything fits. - Expect
**lower** layer count on GPU → closer to the slow column, still better than all-experts-host if fit places*any*expert layers on-device. - Stay under VRAM ceiling; don’t force
`-ngl 999`

without fit.

- Model:
[https://huggingface.co/poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) - GGUF:
[https://huggingface.co/poolside/Laguna-S-2.1-GGUF](https://huggingface.co/poolside/Laguna-S-2.1-GGUF) - NVFP4 (multi-GPU / Spark):
[https://huggingface.co/poolside/Laguna-S-2.1-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4) - llama.cpp (Laguna):
[https://github.com/poolsideai/llama.cpp/tree/laguna](https://github.com/poolsideai/llama.cpp/tree/laguna) - License: OpenMDW-1.1

```
Laguna S 2.1 Q4_K_M + poolside llama.cpp (laguna branch)
  -ngl auto --fit on --fit-target 2048
  -fa on --no-mmap -b 4096 -ub 4096 -ctk q8_0 -ctv q8_0
  -c 262144

RTX 5090 32GB + 62GB RAM:
  ~30 GB VRAM · ~18–19 tok/s decode · ~40–60 tok/s prefill
  vs all-experts-on-CPU: ~45× prefill, ~2× decode
```

**The VRAM limit isn’t fake — but putting the right tensors in it matters more than the “all MoE on CPU” cheat code.** Pack full layers with auto-fit; keep the rest on host. That’s what made 118B-class agentic coding actually feel usable on one 5090. Big win at any rate.