Moe expert offloading on a 2-core Celeron with 2.7GB RAM

A developer's on-hardware test shows that prefetching mixture-of-experts (MoE) expert weights from disk ahead of matmuls can improve token generation speed on a severely resource-constrained machine, with the Intel Celeron N4000 (2 cores, 2.7GB RAM) achieving a compute ceiling of ~2.0 tok/s and an I/O ceiling of ~150 MB/s. The test used the OLMoE-1B-7B-0924 model (64 experts/layer, top-8 routing, 16 layers) via a custom llama.cpp harness, and all measurements were logged in the project's results directory.

Real, on-hardware measurements of MoE expert-weight offloading on a resource-constrained box: does prefetching mixture-of-experts weights off disk, ahead of the matmuls that need them, actually help — and if so, which mechanism does the work? Model: OLMoE-1B-7B-0924 https://huggingface.co/allenai/OLMoE-1B-7B-0924-GGUF GGUF, 64 experts/layer, top-8 routing, 16 layers , run through a custom llama.cpp https://github.com/ggml-org/llama.cpp example harness harness/expert-log.cpp that hooks ggml backend sched 's eval callback to see expert routing decisions in real time and prefetch the selected experts' weight slabs before the matmuls that consume them run. Everything here is a real process run on real hardware — no simulation, no synthetic timing model. All logs and CSVs backing every number below are committed under results/ . | Environment | ChromeOS Crostini Linux VM penguin | | CPU | Intel Celeron N4000 @ 1.10GHz, 2 cores no hyperthreading | | RAM | 2.7 GiB total, no swap | | Root disk | /dev/vdc , 34G, ~98% full during this project | | Kernel | Linux 6.6.135 x86 64 | This is not a datacenter box. It's the kind of machine MoE weight-offloading would need to target for it to matter: RAM well under the model size, a single-digit-core CPU, and a disk that is the actual bottleneck for most of this workload. Measured hardware ceilings both regenerated fresh for this write-up; see results/io ceiling/ and results/compute ceiling/ : - I/O ceiling: ~150 MB/s. O DIRECT sequential dd reads off the root disk, three 1024/512 MiB samples at different offsets: 145–157 MB/s. This bypasses the page cache entirely, so it's the raw disk's ceiling, not anything our harness does. - Compute ceiling: ~2.0 tok/s. A dense, fully-RAM-resident 630M-param model Qwen2.5-0.5B-Instruct, Q8 0, 638.74 MiB run twice back-to-back via llama-bench -p 0 -n 32 -r 1 --no-warmup , 2 threads : 2.10 tok/s cold, 2.03 tok/s warm. Cold and warm being nearly identical not a page-cache warm-up effect confirms this is a genuine compute measurement, not one still partly gated on disk.An earlier, unlogged estimate mid-project had put this figure closer to ~1.0 tok/s; that number was never saved to a committed log and could not be reproduced when re-measured for this write-up, most likely because this VM's actual CPU allocation varies with host load see Caveats . The ~2.0 tok/s figure is the one with a log backing it and is what the ratios below use. harness/expert-log.cpp is a llama.cpp example binary llama-expert-log built against upstream llama.cpp with a tiny patch harness/llama.cpp.patch , +9 lines: one exported llama model get tensor accessor . It does four things: - Reads GGUF tensor metadata directly , independent of loading the model, via the low-level gguf.h API — tensor name, ne shape, byte size, file offset. Expert tensors are stored as merged 3D tensors n embd, n ff, n expert for gate/up, n ff, n embd, n expert for down with expert as the outermost dimension, so each expert's slab is a contiguous byte range — base offset + expert id bytes per expert . This is what makes byte-accounting exact rather than estimated: bytes per token in every table below comes straight out of this metadata, and it matched the harness's own pread totals exactly in every run. - Hooks on tensors named ggml backend sched eval callback "ffn moe topk-<layer " — the top-k expert-selection output that llama.cpp 's build moe ffn already tags. This fires with the real routing decision, strictly before the gate/up/down matmul nodes for that layer execute, and is the only per-token, per-layer hook point GGML exposes there is no per-expert hook — gate/up/down each run as one batched node over all n used experts, not one node per expert . - Prefetches each routed expert's gate/up/down slabs : a buffered pread no O DIRECT into a reused scratch buffer, which warms the kernel page cache for that byte range, followed by madvise MADV WILLNEED on the same range in the tensor's real mmap'd address. No userspace cache, no custom eviction policy — the kernel's own page cache is the only cache. This is the design that survived see Finding 1 for why . - embeddings, attention, norms, output head — 297.2 MiB, 147 tensors unconditionally at startup, so the kernel never evicts the always-needed working set and the full page-cache budget goes to expert slabs, the only thing re-fetched every token Finding 7 . mlock s every non-expert tensor Per-token instrumentation: wall-clock decode time, getrusage .ru majflt major page faults — pages that required a real fetch from backing store, not served from cache taken as a delta around each llama decode call, and pread / madvise timing buckets. /proc/self/io 's read bytes field was tried as an independent check on real disk bytes but reads back 0 in this container environment the whole project an LXC/Crostini limitation, not a code bug — ru majflt was the working substitute. Each finding names the exact config and points at the log/CSV in results/ it came from. Full tables are in RESULTS.md /dhishwasher/moe-offload-bench/blob/master/RESULTS.md . First design: O DIRECT reads + a userspace LRU slab cache + mprotect -ing expert tensors writable so cache hits/misses could memcpy straight into the mmap'd weight memory. Swept the LRU budget 0/256/512/800/1200 MB on Q4 K M, 32 tokens results/sweep/ : | Budget | hit rate | tok/s | MiB/token from disk | |---|---|---|---| | 0 MB | 0.0% | 0.045 | 466.50 | | 256 MB | 0.0% | 0.043 | 466.50 | | 512 MB | 33.6% | 0.043 | 310.00 | | 800 MB | 37.0% | 0.037 | 293.96 | | 1200 MB | — | crashed | — | tok/s got worse as the cache budget and hit rate went up — LRU bookkeeping and O DIRECT overhead outweighed the disk-read savings at every budget tried. At 1200 MB the process didn't just get slow, it died: RSS climbed 1091→1469→1619→1733 MiB over four tokens before a fifth took 2269 seconds and the box hit virtio balloon: Out of puff genuine host memory pressure . Root cause: every cache hit or miss wrote into the mmap 'd weight memory, and on a MAP PRIVATE mapping that write is copy-on-write — a permanent, unreclaimable anonymous page. With no swap on this box, that RSS growth had nowhere to go but OOM. This whole design was scrapped. The eventual replacement pure pread + madvise WILLNEED , no userspace cache at all, Finding 7 reached 0.114 tok/s — 2.5x the best number this design ever produced, safely. 64-token generation, all 16 layers results/expert skew.csv , via scripts/analyze skew.py : covering 80% of a layer's activations takes 20–33 of its 64 experts median ~25 , and 44–59 distinct experts get touched per layer within just 64 tokens. Routing is close to uniform. There's no small "hot" subset of experts worth special-casing a cache around — any useful cache has to be able to hold most of the expert population, which on this hardware it can't. Same 64-token run results/locality.csv , via scripts/analyze locality.py : adjacent tokens distance 1 share on average only 35.7% of their 8 routed experts per layer; at distance 8 that drops to 20.8% . A "cache what the last token used" policy would miss roughly two-thirds of the time even one token later. This and Finding 2 together are why no expert-frequency or recency cache was pursued further — the workload doesn't have the locality such a cache needs. Q4 K M vs Q2 K, identical prompt, 32 tokens, pread+madvise design results/final/ : | Quant | bytes/token | tok/s | bytes ratio Q4/Q2 | tok/s ratio Q2/Q4 | |---|---|---|---|---| | Q4 K M | 487,587,840 465.0 MiB | 0.091 | 1.67x | 1.92x | | Q2 K | 291,504,128 278.0 MiB | 0.175 | If tok/s were purely 1/bytes , Q2 should be 1.67x faster than Q4. It's actually 1.92x faster — Q4 pays a bigger-than-proportional penalty for its larger footprint. Finding 5 is why. Same two runs, major faults per token: | Quant | majflt/token | ratio | |---|---|---| | Q4 K M | 385.56 | 9.3x | | Q2 K | 41.47 | 9.3x more major faults for only 1.67x more bytes. The harness's synchronous pread timer only measures its own reads; this confirms — rather than just implies — that Q4's larger per-token working set causes disproportionate page-cache eviction-and-refetch churn during compute itself, on top of what the harness explicitly prefetches. /proc/self/io would have measured this directly but reads 0 in this container, per the harness design notes above — ru majflt is the substitute, and it's unambiguous here. Q2 K's 0.175–0.201 tok/s vs the ~2.0 tok/s dense-model compute ceiling Hardware section above is roughly a 10x gap. This workload is I/O-bound, not compute-bound, at every quant level tested — which is the whole reason prefetch/caching work on the I/O side Findings 1, 7 is where the effort in this project went. All four configs below are Q4 K M, 32 tokens, pread+madvise prefetch, same prompt: | Config | tok/s | majflt/token | vs. no-mlock baseline | |---|---|---|---| | No mlock baseline | 0.091 | 385.56 | — | + mlock non-expert tensors | 0.115 | 134.28 | +26% tok/s, 2.9x fewer majflt | mlock + pread directly into the tensor's mmap no scratch copy | 0.041 | 347.09 | -55% tok/s — reverted | | mlock + background thread pool prefetching 1 worker, overlap pread with compute | 0.081 | 1029.16 | -29% tok/s, 7.4x more majflt — reverted | mlock win : locking the 297.2 MiB non-expert working set embeddings, attention, norms, output head so it's never evicted leaves the full page-cache budget for expert slabs — the only thing re-fetched every token. +26% tok/s, majflt cut 2.9x. This is the only optimization attempt in this project that improved on the plain pread+madvise baseline. Write-in-place reverted : eliminating the scratch-buffer copy by pread -ing straight into the tensor's own mmap'd address required making that mapping MAP SHARED + PROT WRITE it was read-only . Every prefetched page came back dirty and needed writeback to disk — that write traffic contended with read traffic for the same disk queue and made things over 2x slower despite a genuine ~10% drop in major faults. Background thread pool reverted : issuing prefetch pread s from a worker thread so compute for one layer could overlap disk reads for the next was meant to hide I/O behind compute. It did the opposite — one background pread thread and the compute thread's own page faults contending for the same disk queue cost far more than any overlap saved, and majflt/token rose 7.4x. There's no idle core on a 2-core box to hide that contention behind, and this disk didn't show the concurrent-queue-depth throughput gain a network download earlier in the project had 6 parallel HTTP range requests: 1.3 → 23 MB/s — local disk and remote CDN throttling are different bottlenecks. Time breakdown at the best config mlock + pread + madvise, Q4 K M, results/step4/q4 32tok buckets.log : of 8.767s/token, pread is 5.091s 58% , real compute is 3.658s 42% , and madvise + the harness's own CSV-logging bookkeeping are both noise <0.2% combined, confirmed by re-running with logging fully disabled: 0.109 tok/s, statistically indistinguishable from 0.114 tok/s with logging on . pread, gated by real disk throughput, is the floor — and per Finding 7's other two rows, this box has no cheap way to shrink it further. git clone https://github.com/ggml-org/llama.cpp cd llama.cpp git checkout 9a286ac98d2cab74231bd3f1fc3f2b8bdf05422e commit this was built against git apply /path/to/moe-offload-bench/harness/llama.cpp.patch cp /path/to/moe-offload-bench/harness/expert-log.cpp examples/expert-log/ cp /path/to/moe-offload-bench/harness/CMakeLists.txt examples/expert-log/ cmake -B build -DCMAKE BUILD TYPE=Release cmake --build build --target llama-expert-log -j2 ./build/bin/llama-expert-log -m /path/to/olmoe-1b-7b-0924-q4 k m.gguf \ -o results.csv -p "The history of the Roman Empire begins with" -n 32 CLI flags: -q disables per-expert CSV logging measures instrumentation cost, Finding 7 . mlock of non-expert tensors is unconditional. - This is a single shared/variable VM , not an isolated benchmark rig — dmesg shows virtio balloon: Out of puff events during the heaviest runs, and repeated measurements of the same config vary by 10-20% run to run occasionally more; the compute-ceiling re-measurement for this write-up came back ~2x higher than an earlier unlogged estimate . Every number in this repo is real and reproducible in direction, but treat absolute figures as "this box, this hour," not a portable hardware spec. - All findings use n=32 generated tokens per run single run per config, not averaged across repeats — enough to see clear, consistent, order-of-magnitude effects, not enough to quote a confidence interval on a ±5% difference. results/ also contains earlier one-off debugging runs expert activations meta test , expert activations naive2 , validate cache.csv , etc. from before the harness reached its current form. They're kept for the record but aren't referenced above.