Real, on-hardware measurements of MoE expert-weight off on a resource-constrained box: does prefetching mixture-of-experts weights off disk, ahead of the matmuls that need them, actually help — and if so, which mechanism does the work?
Model: OLMoE-1B-7B-0924 (GGUF,
64 experts/layer, top-8 routing, 16 layers), run through a custom
llama.cpp example harness
(harness/expert-log.cpp
) that hooks ggml_backend_sched
's eval callback to see expert routing decisions in real time and prefetch the selected experts' weight slabs before the matmuls that consume them run.
Everything here is a real process run on real hardware — no simulation, no synthetic
timing model. All logs and CSVs backing every number below are committed under results/
.
| Environment | ChromeOS Crostini Linux VM (penguin ) |
| CPU | Intel Celeron N4000 @ 1.10GHz, 2 cores (no hyperthreading) |
| RAM | 2.7 GiB total, no swap |
| Root disk | /dev/vdc , 34G, ~98% full during this project |
| Kernel | Linux 6.6.135 x86_64 |
This is not a datacenter box. It's the kind of machine MoE weight-off would need to target for it to matter: RAM well under the model size, a single-digit-core CPU, and a disk that is the actual bottleneck for most of this workload.
Measured hardware ceilings (both regenerated fresh for this write-up; see
results/io_ceiling/
and results/compute_ceiling/
):
I/O ceiling: ~150 MB/s.O_DIRECT
sequentialdd
reads off the root disk, three 1024/512 MiB samples at different offsets: 145–157 MB/s. This bypasses the page cache entirely, so it's the raw disk's ceiling, not anything our harness does. -
Compute ceiling: ~2.0 tok/s. A dense, fully-RAM-resident 630M-param model (Qwen2.5-0.5B-Instruct, Q8_0, 638.74 MiB) run twice back-to-back viallama-bench
(-p 0 -n 32 -r 1 --no-warmup
, 2 threads): 2.10 tok/s cold, 2.03 tok/s warm. Cold and warm being nearly identical (not a page-cache warm-up effect) confirms this is a genuine compute measurement, not one still partly gated on disk.An earlier, unlogged estimate mid-project had put this figure closer to ~1.0 tok/s; that number was never saved to a committed log and could not be reproduced when re-measured for this write-up, most likely because this VM's actual CPU allocation varies with host load (see Caveats). The ~2.0 tok/s figure is the one with a log backing it and is what the ratios below use.
harness/expert-log.cpp
is a llama.cpp
example binary (llama-expert-log
) built against
upstream llama.cpp
with a tiny patch (harness/llama.cpp.patch
, +9 lines: one exported
llama_model_get_tensor()
accessor). It does four things:
Reads GGUF tensor metadata directly, independent of the model, via the low-levelgguf.h
API — tensor name,ne[]
shape, byte size, file offset. Expert tensors are stored as merged 3D tensors ([n_embd, n_ff, n_expert]
for gate/up,[n_ff, n_embd, n_expert]
for down) with expert as theoutermost dimension, so each expert's slab is a contiguous byte range —base_offset + expert_id * bytes_per_expert
. This is what makes byte-accounting exact rather than estimated:bytes_per_token
in every table below comes straight out of this metadata, and it matched the harness's ownpread()
totals exactly in every run. -
Hooks on tensors namedggml_backend_sched_eval_callback
"ffn_moe_topk-<layer>"
— the top-k expert-selection output thatllama.cpp
'sbuild_moe_ffn()
already tags. This fires with the real routing decision, strictly before the gate/up/down matmul nodes for that layer execute, and is the only per-token, per-layer hook point GGML exposes (there is no per-expert hook — gate/up/down each run as one batched node over alln_used
experts, not one node per expert). -
Prefetches each routed expert's gate/up/down slabs: a bufferedpread()
(noO_DIRECT
) into a reused scratch buffer, which warms the kernel page cache for that byte range, followed bymadvise(MADV_WILLNEED)
on the same range in the tensor's real mmap'd address. No userspace cache, no custom eviction policy — the kernel's own page cache is the only cache. This is the design that survived (see Finding 1 for why). -
(embeddings, attention, norms, output head — 297.2 MiB, 147 tensors) unconditionally at startup, so the kernel never evicts the always-needed working set and the full page-cache budget goes to expert slabs, the only thing re-fetched every token (Finding 7).mlock()
s every non-expert tensor
Per-token instrumentation: wall-clock decode time, getrusage().ru_majflt
(major page
faults — pages that required a real fetch from backing store, not served from cache) taken
as a delta around each llama_decode()
call, and pread()
/madvise()
timing buckets.
/proc/self/io
's read_bytes
field was tried as an independent check on real disk bytes
but reads back 0
in this container environment the whole project (an LXC/Crostini
limitation, not a code bug) — ru_majflt
was the working substitute.
Each finding names the exact config and points at the log/CSV in results/
it came from. Full tables are in RESULTS.md.
First design: O_DIRECT
reads + a userspace LRU slab cache + mprotect()
-ing expert
tensors writable so cache hits/misses could memcpy()
straight into the mmap'd weight
memory. Swept the LRU budget 0/256/512/800/1200 MB on Q4_K_M, 32 tokens
(results/sweep/
):
| Budget | hit rate | tok/s | MiB/token from disk |
|---|---|---|---|
| 0 MB | 0.0% | 0.045 | 466.50 |
| 256 MB | 0.0% | 0.043 | 466.50 |
| 512 MB | 33.6% | 0.043 | 310.00 |
| 800 MB | 37.0% | 0.037 | |
| 293.96 | |||
| 1200 MB | — | crashed | |
| — |
tok/s got worse as the cache budget (and hit rate) went up — LRU bookkeeping and
O_DIRECT
overhead outweighed the disk-read savings at every budget tried. At 1200 MB the
process didn't just get slow, it died: RSS climbed 1091→1469→1619→1733 MiB over four
tokens before a fifth took 2269 seconds and the box hit virtio_balloon: Out of puff
(genuine host memory pressure). Root cause: every cache hit or miss wrote into the
mmap
'd weight memory, and on a MAP_PRIVATE
mapping that write is copy-on-write —
a permanent, unreclaimable anonymous page. With no swap on this box, that RSS growth had
nowhere to go but OOM. This whole design was scrapped. The eventual replacement (pure
pread()
madvise(WILLNEED)
, no userspace cache at all, Finding 7) reached 0.114 tok/s — 2.5x the best number this design ever produced, safely.
64-token generation, all 16 layers (results/expert_skew.csv
, via scripts/analyze_skew.py
): covering 80% of a layer's activations takes 20–33 of its 64 experts (median ~25), and 44–59 distinct experts get touched per layer within just 64 tokens. Routing is close to uniform. There's no small "hot" subset of experts worth special-casing a cache around — any useful cache has to be able to hold most of the expert population, which on this hardware it can't.
Same 64-token run (results/locality.csv
, via scripts/analyze_locality.py
): adjacent tokens (distance 1) share on average only 35.7% of their 8 routed experts per layer; at distance 8 that drops to 20.8%. A "cache what the last token used" policy would miss roughly two-thirds of the time even one token later. This and Finding 2 together are why no expert-frequency or recency cache was pursued further — the workload doesn't have the locality such a cache needs.
Q4_K_M vs Q2_K, identical prompt, 32 tokens, pread+madvise design (results/final/
):
| Quant | bytes/token | tok/s | bytes ratio (Q4/Q2) | tok/s ratio (Q2/Q4) |
|---|---|---|---|---|
| Q4_K_M | 487,587,840 (465.0 MiB) | 0.091 | 1.67x | 1.92x |
| Q2_K | 291,504,128 (278.0 MiB) | 0.175 |
If tok/s were purely 1/bytes
, Q2 should be 1.67x faster than Q4. It's actually 1.92x faster — Q4 pays a bigger-than-proportional penalty for its larger footprint. Finding 5 is why.
Same two runs, major faults per token:
| Quant | majflt/token | ratio |
|---|---|---|
| Q4_K_M | 385.56 | 9.3x |
| Q2_K | 41.47 |
9.3x more major faults for only 1.67x more bytes. The harness's synchronous pread()
timer only measures its own reads; this confirms — rather than just implies — that Q4's
larger per-token working set causes disproportionate page-cache eviction-and-refetch churn
during compute itself, on top of what the harness explicitly prefetches. (/proc/self/io
would have measured this directly but reads 0
in this container, per the harness design
notes above — ru_majflt
is the substitute, and it's unambiguous here.)
Q2_K's 0.175–0.201 tok/s vs the ~2.0 tok/s dense-model compute ceiling (Hardware section above) is roughly a 10x gap. This workload is I/O-bound, not compute-bound, at every quant level tested — which is the whole reason prefetch/caching work on the I/O side (Findings 1, 7) is where the effort in this project went.
All four configs below are Q4_K_M, 32 tokens, pread+madvise prefetch, same prompt:
| Config | tok/s | majflt/token | vs. no-mlock baseline |
|---|---|---|---|
| No mlock (baseline) | 0.091 | 385.56 | — |
- mlock non-expert tensors | 0.115 | 134.28 | +26% tok/s, 2.9x fewer majflt | mlock + pread directly into the tensor's mmap (no scratch copy) | 0.041 | 347.09 | -55% tok/s — reverted | | mlock + background thread pool prefetching (1 worker, overlap pread with compute) | 0.081 | 1029.16 | -29% tok/s, 7.4x more majflt — reverted |
mlock (win): locking the 297.2 MiB non-expert working set (embeddings, attention, norms, output head) so it's never evicted leaves the full page-cache budget for expert slabs — the only thing re-fetched every token. +26% tok/s, majflt cut 2.9x. This is the only optimization attempt in this project that improved on the plain pread+madvise baseline.Write-in-place (reverted): eliminating the scratch-buffer copy bypread()
-ing straight into the tensor's own mmap'd address required making that mappingMAP_SHARED
+PROT_WRITE
(it was read-only). Every prefetched page came back dirty and needed writeback to disk — that write traffic contended with read traffic for the same disk queue and made things over 2x slower despite a genuine ~10% drop in major faults.Background thread pool (reverted): issuing prefetchpread()
s from a worker thread so compute for one layer could overlap disk reads for the next was meant to hide I/O behind compute. It did the opposite — one backgroundpread()
thread and the compute thread's own page faults contending for the same disk queue cost far more than any overlap saved, and majflt/token rose 7.4x. There's no idle core on a 2-core box to hide that contention behind, and this disk didn't show the concurrent-queue-depth throughput gain a network download earlier in the project had (6 parallel HTTP range requests: 1.3 → 23 MB/s) — local disk and remote CDN throttling are different bottlenecks.
Time breakdown at the best config (mlock + pread + madvise, Q4_K_M,
results/step4/q4_32tok_buckets.log
): of 8.767s/token, pread is 5.091s (58%), real compute is 3.658s (42%), and madvise + the harness's own CSV-logging bookkeeping are both noise (<0.2% combined, confirmed by re-running with logging fully disabled: 0.109 tok/s, statistically indistinguishable from 0.114 tok/s with logging on). pread, gated by real disk throughput, is the floor — and per Finding 7's other two rows, this box has no cheap way to shrink it further.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 9a286ac98d2cab74231bd3f1fc3f2b8bdf05422e # commit this was built against
git apply /path/to/moe-offload-bench/harness/llama.cpp.patch
cp /path/to/moe-offload-bench/harness/expert-log.cpp examples/expert-log/
cp /path/to/moe-offload-bench/harness/CMakeLists.txt examples/expert-log/
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-expert-log -j2
./build/bin/llama-expert-log -m /path/to/olmoe-1b-7b-0924-q4_k_m.gguf \
-o results.csv -p "The history of the Roman Empire begins with" -n 32
CLI flags: -q
disables per-expert CSV logging (measures instrumentation cost, Finding 7). mlock of non-expert tensors is unconditional.
- This is a
single shared/variable VM, not an isolated benchmark rig —
dmesg
showsvirtio_balloon: Out of puff
events during the heaviest runs, and repeated measurements of the same config vary by 10-20% run to run (occasionally more; the compute-ceiling re-measurement for this write-up came back ~2x higher than an earlier unlogged estimate). Every number in this repo is real and reproducible in direction, but treat absolute figures as "this box, this hour," not a portable hardware spec. - All findings use
n=32
generated tokens per run (single run per config, not averaged across repeats) — enough to see clear, consistent, order-of-magnitude effects, not enough to quote a confidence interval on a ±5% difference. results/
also contains earlier one-off debugging runs (expert_activations_meta_test*
,expert_activations_naive2*
,validate_cache.csv
, etc.) from before the harness reached its current form. They're kept for the record but aren't referenced above.