# Moe expert offloading on a 2-core Celeron with 2.7GB RAM

> Source: <https://github.com/dhishwasher/moe-offload-bench>
> Published: 2026-09-01 22:37:12+00:00

Real, on-hardware measurements of MoE expert-weight offloading on a resource-constrained box: does prefetching mixture-of-experts weights off disk, ahead of the matmuls that need them, actually help — and if so, which mechanism does the work?

Model: [OLMoE-1B-7B-0924](https://huggingface.co/allenai/OLMoE-1B-7B-0924-GGUF) (GGUF,
64 experts/layer, top-8 routing, 16 layers), run through a custom
[llama.cpp](https://github.com/ggml-org/llama.cpp) example harness
(`harness/expert-log.cpp`

) that hooks `ggml_backend_sched`

's eval callback to see expert
routing decisions in real time and prefetch the selected experts' weight slabs before the
matmuls that consume them run.

Everything here is a real process run on real hardware — no simulation, no synthetic
timing model. All logs and CSVs backing every number below are committed under `results/`

.

| Environment | ChromeOS Crostini Linux VM (`penguin` ) |
| CPU | Intel Celeron N4000 @ 1.10GHz, 2 cores (no hyperthreading) |
| RAM | 2.7 GiB total, no swap |
| Root disk | `/dev/vdc` , 34G, ~98% full during this project |
| Kernel | Linux 6.6.135 x86_64 |

This is not a datacenter box. It's the kind of machine MoE weight-offloading would need to target for it to matter: RAM well under the model size, a single-digit-core CPU, and a disk that is the actual bottleneck for most of this workload.

**Measured hardware ceilings** (both regenerated fresh for this write-up; see
`results/io_ceiling/`

and `results/compute_ceiling/`

):

-
**I/O ceiling: ~150 MB/s.**`O_DIRECT`

sequential`dd`

reads off the root disk, three 1024/512 MiB samples at different offsets: 145–157 MB/s. This bypasses the page cache entirely, so it's the raw disk's ceiling, not anything our harness does. -
**Compute ceiling: ~2.0 tok/s.** A dense, fully-RAM-resident 630M-param model (Qwen2.5-0.5B-Instruct, Q8_0, 638.74 MiB) run twice back-to-back via`llama-bench`

(`-p 0 -n 32 -r 1 --no-warmup`

, 2 threads): 2.10 tok/s cold, 2.03 tok/s warm. Cold and warm being nearly identical (not a page-cache warm-up effect) confirms this is a genuine compute measurement, not one still partly gated on disk.An earlier, unlogged estimate mid-project had put this figure closer to ~1.0 tok/s; that number was never saved to a committed log and could not be reproduced when re-measured for this write-up, most likely because this VM's actual CPU allocation varies with host load (see Caveats). The ~2.0 tok/s figure is the one with a log backing it and is what the ratios below use.

`harness/expert-log.cpp`

is a `llama.cpp`

example binary (`llama-expert-log`

) built against
upstream `llama.cpp`

with a tiny patch (`harness/llama.cpp.patch`

, +9 lines: one exported
`llama_model_get_tensor()`

accessor). It does four things:

-
**Reads GGUF tensor metadata directly**, independent of loading the model, via the low-level`gguf.h`

API — tensor name,`ne[]`

shape, byte size, file offset. Expert tensors are stored as merged 3D tensors (`[n_embd, n_ff, n_expert]`

for gate/up,`[n_ff, n_embd, n_expert]`

for down) with expert as the**outermost** dimension, so each expert's slab is a contiguous byte range —`base_offset + expert_id * bytes_per_expert`

. This is what makes byte-accounting exact rather than estimated:`bytes_per_token`

in every table below comes straight out of this metadata, and it matched the harness's own`pread()`

totals exactly in every run. -
**Hooks** on tensors named`ggml_backend_sched_eval_callback`

`"ffn_moe_topk-<layer>"`

— the top-k expert-selection output that`llama.cpp`

's`build_moe_ffn()`

already tags. This fires with the real routing decision, strictly before the gate/up/down matmul nodes for that layer execute, and is the only per-token, per-layer hook point GGML exposes (there is no per-expert hook — gate/up/down each run as one batched node over all`n_used`

experts, not one node per expert). -
**Prefetches each routed expert's gate/up/down slabs**: a buffered`pread()`

(no`O_DIRECT`

) into a reused scratch buffer, which warms the kernel page cache for that byte range, followed by`madvise(MADV_WILLNEED)`

on the same range in the tensor's real mmap'd address. No userspace cache, no custom eviction policy — the kernel's own page cache is the only cache. This is the design that survived (see Finding 1 for why). -
(embeddings, attention, norms, output head — 297.2 MiB, 147 tensors) unconditionally at startup, so the kernel never evicts the always-needed working set and the full page-cache budget goes to expert slabs, the only thing re-fetched every token (Finding 7).`mlock()`

s every non-expert tensor

Per-token instrumentation: wall-clock decode time, `getrusage().ru_majflt`

(major page
faults — pages that required a real fetch from backing store, not served from cache) taken
as a delta around each `llama_decode()`

call, and `pread()`

/`madvise()`

timing buckets.
`/proc/self/io`

's `read_bytes`

field was tried as an independent check on real disk bytes
but reads back `0`

in this container environment the whole project (an LXC/Crostini
limitation, not a code bug) — `ru_majflt`

was the working substitute.

Each finding names the exact config and points at the log/CSV in `results/`

it came from.
Full tables are in [ RESULTS.md](/dhishwasher/moe-offload-bench/blob/master/RESULTS.md).

First design: `O_DIRECT`

reads + a userspace LRU slab cache + `mprotect()`

-ing expert
tensors writable so cache hits/misses could `memcpy()`

straight into the mmap'd weight
memory. Swept the LRU budget 0/256/512/800/1200 MB on Q4_K_M, 32 tokens
(`results/sweep/`

):

| Budget | hit rate | tok/s | MiB/token from disk |
|---|---|---|---|
| 0 MB | 0.0% | 0.045 | 466.50 |
| 256 MB | 0.0% | 0.043 | 466.50 |
| 512 MB | 33.6% | 0.043 | 310.00 |
| 800 MB | 37.0% | 0.037 |
293.96 |
| 1200 MB | — | crashed |
— |

tok/s got *worse* as the cache budget (and hit rate) went *up* — LRU bookkeeping and
`O_DIRECT`

overhead outweighed the disk-read savings at every budget tried. At 1200 MB the
process didn't just get slow, it died: RSS climbed 1091→1469→1619→1733 MiB over four
tokens before a fifth took 2269 seconds and the box hit `virtio_balloon: Out of puff`

(genuine host memory pressure). Root cause: every cache hit *or* miss wrote into the
`mmap`

'd weight memory, and on a `MAP_PRIVATE`

mapping that write is copy-on-write —
a permanent, unreclaimable anonymous page. With no swap on this box, that RSS growth had
nowhere to go but OOM. This whole design was scrapped. The eventual replacement (pure
`pread()`

+ `madvise(WILLNEED)`

, no userspace cache at all, Finding 7) reached **0.114
tok/s** — 2.5x the *best* number this design ever produced, safely.

64-token generation, all 16 layers (`results/expert_skew.csv`

, via `scripts/analyze_skew.py`

):
covering 80% of a layer's activations takes 20–33 of its 64 experts (median ~25), and
44–59 distinct experts get touched per layer within just 64 tokens. Routing is close to
uniform. There's no small "hot" subset of experts worth special-casing a cache around —
any useful cache has to be able to hold most of the expert population, which on this
hardware it can't.

Same 64-token run (`results/locality.csv`

, via `scripts/analyze_locality.py`

): adjacent
tokens (distance 1) share on average only **35.7%** of their 8 routed experts per layer;
at distance 8 that drops to **20.8%**. A "cache what the last token used" policy would
miss roughly two-thirds of the time even one token later. This and Finding 2 together are
why no expert-frequency or recency cache was pursued further — the workload doesn't have
the locality such a cache needs.

Q4_K_M vs Q2_K, identical prompt, 32 tokens, pread+madvise design (`results/final/`

):

| Quant | bytes/token | tok/s | bytes ratio (Q4/Q2) | tok/s ratio (Q2/Q4) |
|---|---|---|---|---|
| Q4_K_M | 487,587,840 (465.0 MiB) | 0.091 | 1.67x | 1.92x |
| Q2_K | 291,504,128 (278.0 MiB) | 0.175 |

If tok/s were purely `1/bytes`

, Q2 should be 1.67x faster than Q4. It's actually 1.92x
faster — Q4 pays a bigger-than-proportional penalty for its larger footprint. Finding 5 is
why.

Same two runs, major faults per token:

| Quant | majflt/token | ratio |
|---|---|---|
| Q4_K_M | 385.56 | 9.3x |
| Q2_K | 41.47 |

9.3x more major faults for only 1.67x more bytes. The harness's synchronous `pread()`

timer only measures its *own* reads; this confirms — rather than just implies — that Q4's
larger per-token working set causes disproportionate page-cache eviction-and-refetch churn
during compute itself, on top of what the harness explicitly prefetches. (`/proc/self/io`

would have measured this directly but reads `0`

in this container, per the harness design
notes above — `ru_majflt`

is the substitute, and it's unambiguous here.)

Q2_K's 0.175–0.201 tok/s vs the ~2.0 tok/s dense-model compute ceiling (Hardware section
above) is roughly a **10x** gap. This workload is I/O-bound, not compute-bound, at every
quant level tested — which is the whole reason prefetch/caching work on the I/O side
(Findings 1, 7) is where the effort in this project went.

All four configs below are Q4_K_M, 32 tokens, pread+madvise prefetch, same prompt:

| Config | tok/s | majflt/token | vs. no-mlock baseline |
|---|---|---|---|
| No mlock (baseline) | 0.091 | 385.56 | — |
+ mlock non-expert tensors |
0.115 |
134.28 |
+26% tok/s, 2.9x fewer majflt |
mlock + pread directly into the tensor's mmap (no scratch copy) |
0.041 | 347.09 | -55% tok/s — reverted |
| mlock + background thread pool prefetching (1 worker, overlap pread with compute) | 0.081 | 1029.16 | -29% tok/s, 7.4x more majflt — reverted |

**mlock (win):** locking the 297.2 MiB non-expert working set (embeddings, attention, norms, output head) so it's never evicted leaves the full page-cache budget for expert slabs — the only thing re-fetched every token. +26% tok/s, majflt cut 2.9x. This is the only optimization attempt in this project that improved on the plain pread+madvise baseline.**Write-in-place (reverted):** eliminating the scratch-buffer copy by`pread()`

-ing straight into the tensor's own mmap'd address required making that mapping`MAP_SHARED`

+`PROT_WRITE`

(it was read-only). Every prefetched page came back dirty and needed writeback to disk — that write traffic contended with read traffic for the same disk queue and made things over 2x slower despite a genuine ~10% drop in major faults.**Background thread pool (reverted):** issuing prefetch`pread()`

s from a worker thread so compute for one layer could overlap disk reads for the next was meant to hide I/O behind compute. It did the opposite — one background`pread()`

thread and the compute thread's own page faults contending for the same disk queue cost far more than any overlap saved, and majflt/token rose 7.4x. There's no idle core on a 2-core box to hide that contention behind, and this disk didn't show the concurrent-queue-depth throughput gain a network download earlier in the project had (6 parallel HTTP range requests: 1.3 → 23 MB/s) — local disk and remote CDN throttling are different bottlenecks.

**Time breakdown at the best config** (mlock + pread + madvise, Q4_K_M,
`results/step4/q4_32tok_buckets.log`

): of 8.767s/token, pread is 5.091s (58%), real
compute is 3.658s (42%), and madvise + the harness's own CSV-logging bookkeeping are both
noise (<0.2% combined, confirmed by re-running with logging fully disabled: 0.109 tok/s,
statistically indistinguishable from 0.114 tok/s with logging on). pread, gated by real
disk throughput, is the floor — and per Finding 7's other two rows, this box has no
cheap way to shrink it further.

```
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 9a286ac98d2cab74231bd3f1fc3f2b8bdf05422e   # commit this was built against
git apply /path/to/moe-offload-bench/harness/llama.cpp.patch
cp /path/to/moe-offload-bench/harness/expert-log.cpp examples/expert-log/
cp /path/to/moe-offload-bench/harness/CMakeLists.txt examples/expert-log/
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-expert-log -j2

./build/bin/llama-expert-log -m /path/to/olmoe-1b-7b-0924-q4_k_m.gguf \
    -o results.csv -p "The history of the Roman Empire begins with" -n 32
```

CLI flags: `-q`

disables per-expert CSV logging (measures instrumentation cost, Finding
7). mlock of non-expert tensors is unconditional.

- This is a
**single shared/variable VM**, not an isolated benchmark rig —`dmesg`

shows`virtio_balloon: Out of puff`

events during the heaviest runs, and repeated measurements of the same config vary by 10-20% run to run (occasionally more; the compute-ceiling re-measurement for this write-up came back ~2x higher than an earlier unlogged estimate). Every number in this repo is real and reproducible in direction, but treat absolute figures as "this box, this hour," not a portable hardware spec. - All findings use
`n=32`

generated tokens per run (single run per config, not averaged across repeats) — enough to see clear, consistent, order-of-magnitude effects, not enough to quote a confidence interval on a ±5% difference. `results/`

also contains earlier one-off debugging runs (`expert_activations_meta_test*`

,`expert_activations_naive2*`

,`validate_cache.csv`

, etc.) from before the harness reached its current form. They're kept for the record but aren't referenced above.