{"slug": "moe-expert-offloading-on-a-2-core-celeron-with-2-7gb-ram", "title": "Moe expert offloading on a 2-core Celeron with 2.7GB RAM", "summary": "A developer's on-hardware test shows that prefetching mixture-of-experts (MoE) expert weights from disk ahead of matmuls can improve token generation speed on a severely resource-constrained machine, with the Intel Celeron N4000 (2 cores, 2.7GB RAM) achieving a compute ceiling of ~2.0 tok/s and an I/O ceiling of ~150 MB/s. The test used the OLMoE-1B-7B-0924 model (64 experts/layer, top-8 routing, 16 layers) via a custom llama.cpp harness, and all measurements were logged in the project's results directory.", "body_md": "Real, on-hardware measurements of MoE expert-weight offloading on a resource-constrained box: does prefetching mixture-of-experts weights off disk, ahead of the matmuls that need them, actually help — and if so, which mechanism does the work?\n\nModel: [OLMoE-1B-7B-0924](https://huggingface.co/allenai/OLMoE-1B-7B-0924-GGUF) (GGUF,\n64 experts/layer, top-8 routing, 16 layers), run through a custom\n[llama.cpp](https://github.com/ggml-org/llama.cpp) example harness\n(`harness/expert-log.cpp`\n\n) that hooks `ggml_backend_sched`\n\n's eval callback to see expert\nrouting decisions in real time and prefetch the selected experts' weight slabs before the\nmatmuls that consume them run.\n\nEverything here is a real process run on real hardware — no simulation, no synthetic\ntiming model. All logs and CSVs backing every number below are committed under `results/`\n\n.\n\n| Environment | ChromeOS Crostini Linux VM (`penguin` ) |\n| CPU | Intel Celeron N4000 @ 1.10GHz, 2 cores (no hyperthreading) |\n| RAM | 2.7 GiB total, no swap |\n| Root disk | `/dev/vdc` , 34G, ~98% full during this project |\n| Kernel | Linux 6.6.135 x86_64 |\n\nThis is not a datacenter box. It's the kind of machine MoE weight-offloading would need to target for it to matter: RAM well under the model size, a single-digit-core CPU, and a disk that is the actual bottleneck for most of this workload.\n\n**Measured hardware ceilings** (both regenerated fresh for this write-up; see\n`results/io_ceiling/`\n\nand `results/compute_ceiling/`\n\n):\n\n-\n**I/O ceiling: ~150 MB/s.**`O_DIRECT`\n\nsequential`dd`\n\nreads off the root disk, three 1024/512 MiB samples at different offsets: 145–157 MB/s. This bypasses the page cache entirely, so it's the raw disk's ceiling, not anything our harness does. -\n**Compute ceiling: ~2.0 tok/s.** A dense, fully-RAM-resident 630M-param model (Qwen2.5-0.5B-Instruct, Q8_0, 638.74 MiB) run twice back-to-back via`llama-bench`\n\n(`-p 0 -n 32 -r 1 --no-warmup`\n\n, 2 threads): 2.10 tok/s cold, 2.03 tok/s warm. Cold and warm being nearly identical (not a page-cache warm-up effect) confirms this is a genuine compute measurement, not one still partly gated on disk.An earlier, unlogged estimate mid-project had put this figure closer to ~1.0 tok/s; that number was never saved to a committed log and could not be reproduced when re-measured for this write-up, most likely because this VM's actual CPU allocation varies with host load (see Caveats). The ~2.0 tok/s figure is the one with a log backing it and is what the ratios below use.\n\n`harness/expert-log.cpp`\n\nis a `llama.cpp`\n\nexample binary (`llama-expert-log`\n\n) built against\nupstream `llama.cpp`\n\nwith a tiny patch (`harness/llama.cpp.patch`\n\n, +9 lines: one exported\n`llama_model_get_tensor()`\n\naccessor). It does four things:\n\n-\n**Reads GGUF tensor metadata directly**, independent of loading the model, via the low-level`gguf.h`\n\nAPI — tensor name,`ne[]`\n\nshape, byte size, file offset. Expert tensors are stored as merged 3D tensors (`[n_embd, n_ff, n_expert]`\n\nfor gate/up,`[n_ff, n_embd, n_expert]`\n\nfor down) with expert as the**outermost** dimension, so each expert's slab is a contiguous byte range —`base_offset + expert_id * bytes_per_expert`\n\n. This is what makes byte-accounting exact rather than estimated:`bytes_per_token`\n\nin every table below comes straight out of this metadata, and it matched the harness's own`pread()`\n\ntotals exactly in every run. -\n**Hooks** on tensors named`ggml_backend_sched_eval_callback`\n\n`\"ffn_moe_topk-<layer>\"`\n\n— the top-k expert-selection output that`llama.cpp`\n\n's`build_moe_ffn()`\n\nalready tags. This fires with the real routing decision, strictly before the gate/up/down matmul nodes for that layer execute, and is the only per-token, per-layer hook point GGML exposes (there is no per-expert hook — gate/up/down each run as one batched node over all`n_used`\n\nexperts, not one node per expert). -\n**Prefetches each routed expert's gate/up/down slabs**: a buffered`pread()`\n\n(no`O_DIRECT`\n\n) into a reused scratch buffer, which warms the kernel page cache for that byte range, followed by`madvise(MADV_WILLNEED)`\n\non the same range in the tensor's real mmap'd address. No userspace cache, no custom eviction policy — the kernel's own page cache is the only cache. This is the design that survived (see Finding 1 for why). -\n(embeddings, attention, norms, output head — 297.2 MiB, 147 tensors) unconditionally at startup, so the kernel never evicts the always-needed working set and the full page-cache budget goes to expert slabs, the only thing re-fetched every token (Finding 7).`mlock()`\n\ns every non-expert tensor\n\nPer-token instrumentation: wall-clock decode time, `getrusage().ru_majflt`\n\n(major page\nfaults — pages that required a real fetch from backing store, not served from cache) taken\nas a delta around each `llama_decode()`\n\ncall, and `pread()`\n\n/`madvise()`\n\ntiming buckets.\n`/proc/self/io`\n\n's `read_bytes`\n\nfield was tried as an independent check on real disk bytes\nbut reads back `0`\n\nin this container environment the whole project (an LXC/Crostini\nlimitation, not a code bug) — `ru_majflt`\n\nwas the working substitute.\n\nEach finding names the exact config and points at the log/CSV in `results/`\n\nit came from.\nFull tables are in [ RESULTS.md](/dhishwasher/moe-offload-bench/blob/master/RESULTS.md).\n\nFirst design: `O_DIRECT`\n\nreads + a userspace LRU slab cache + `mprotect()`\n\n-ing expert\ntensors writable so cache hits/misses could `memcpy()`\n\nstraight into the mmap'd weight\nmemory. Swept the LRU budget 0/256/512/800/1200 MB on Q4_K_M, 32 tokens\n(`results/sweep/`\n\n):\n\n| Budget | hit rate | tok/s | MiB/token from disk |\n|---|---|---|---|\n| 0 MB | 0.0% | 0.045 | 466.50 |\n| 256 MB | 0.0% | 0.043 | 466.50 |\n| 512 MB | 33.6% | 0.043 | 310.00 |\n| 800 MB | 37.0% | 0.037 |\n293.96 |\n| 1200 MB | — | crashed |\n— |\n\ntok/s got *worse* as the cache budget (and hit rate) went *up* — LRU bookkeeping and\n`O_DIRECT`\n\noverhead outweighed the disk-read savings at every budget tried. At 1200 MB the\nprocess didn't just get slow, it died: RSS climbed 1091→1469→1619→1733 MiB over four\ntokens before a fifth took 2269 seconds and the box hit `virtio_balloon: Out of puff`\n\n(genuine host memory pressure). Root cause: every cache hit *or* miss wrote into the\n`mmap`\n\n'd weight memory, and on a `MAP_PRIVATE`\n\nmapping that write is copy-on-write —\na permanent, unreclaimable anonymous page. With no swap on this box, that RSS growth had\nnowhere to go but OOM. This whole design was scrapped. The eventual replacement (pure\n`pread()`\n\n+ `madvise(WILLNEED)`\n\n, no userspace cache at all, Finding 7) reached **0.114\ntok/s** — 2.5x the *best* number this design ever produced, safely.\n\n64-token generation, all 16 layers (`results/expert_skew.csv`\n\n, via `scripts/analyze_skew.py`\n\n):\ncovering 80% of a layer's activations takes 20–33 of its 64 experts (median ~25), and\n44–59 distinct experts get touched per layer within just 64 tokens. Routing is close to\nuniform. There's no small \"hot\" subset of experts worth special-casing a cache around —\nany useful cache has to be able to hold most of the expert population, which on this\nhardware it can't.\n\nSame 64-token run (`results/locality.csv`\n\n, via `scripts/analyze_locality.py`\n\n): adjacent\ntokens (distance 1) share on average only **35.7%** of their 8 routed experts per layer;\nat distance 8 that drops to **20.8%**. A \"cache what the last token used\" policy would\nmiss roughly two-thirds of the time even one token later. This and Finding 2 together are\nwhy no expert-frequency or recency cache was pursued further — the workload doesn't have\nthe locality such a cache needs.\n\nQ4_K_M vs Q2_K, identical prompt, 32 tokens, pread+madvise design (`results/final/`\n\n):\n\n| Quant | bytes/token | tok/s | bytes ratio (Q4/Q2) | tok/s ratio (Q2/Q4) |\n|---|---|---|---|---|\n| Q4_K_M | 487,587,840 (465.0 MiB) | 0.091 | 1.67x | 1.92x |\n| Q2_K | 291,504,128 (278.0 MiB) | 0.175 |\n\nIf tok/s were purely `1/bytes`\n\n, Q2 should be 1.67x faster than Q4. It's actually 1.92x\nfaster — Q4 pays a bigger-than-proportional penalty for its larger footprint. Finding 5 is\nwhy.\n\nSame two runs, major faults per token:\n\n| Quant | majflt/token | ratio |\n|---|---|---|\n| Q4_K_M | 385.56 | 9.3x |\n| Q2_K | 41.47 |\n\n9.3x more major faults for only 1.67x more bytes. The harness's synchronous `pread()`\n\ntimer only measures its *own* reads; this confirms — rather than just implies — that Q4's\nlarger per-token working set causes disproportionate page-cache eviction-and-refetch churn\nduring compute itself, on top of what the harness explicitly prefetches. (`/proc/self/io`\n\nwould have measured this directly but reads `0`\n\nin this container, per the harness design\nnotes above — `ru_majflt`\n\nis the substitute, and it's unambiguous here.)\n\nQ2_K's 0.175–0.201 tok/s vs the ~2.0 tok/s dense-model compute ceiling (Hardware section\nabove) is roughly a **10x** gap. This workload is I/O-bound, not compute-bound, at every\nquant level tested — which is the whole reason prefetch/caching work on the I/O side\n(Findings 1, 7) is where the effort in this project went.\n\nAll four configs below are Q4_K_M, 32 tokens, pread+madvise prefetch, same prompt:\n\n| Config | tok/s | majflt/token | vs. no-mlock baseline |\n|---|---|---|---|\n| No mlock (baseline) | 0.091 | 385.56 | — |\n+ mlock non-expert tensors |\n0.115 |\n134.28 |\n+26% tok/s, 2.9x fewer majflt |\nmlock + pread directly into the tensor's mmap (no scratch copy) |\n0.041 | 347.09 | -55% tok/s — reverted |\n| mlock + background thread pool prefetching (1 worker, overlap pread with compute) | 0.081 | 1029.16 | -29% tok/s, 7.4x more majflt — reverted |\n\n**mlock (win):** locking the 297.2 MiB non-expert working set (embeddings, attention, norms, output head) so it's never evicted leaves the full page-cache budget for expert slabs — the only thing re-fetched every token. +26% tok/s, majflt cut 2.9x. This is the only optimization attempt in this project that improved on the plain pread+madvise baseline.**Write-in-place (reverted):** eliminating the scratch-buffer copy by`pread()`\n\n-ing straight into the tensor's own mmap'd address required making that mapping`MAP_SHARED`\n\n+`PROT_WRITE`\n\n(it was read-only). Every prefetched page came back dirty and needed writeback to disk — that write traffic contended with read traffic for the same disk queue and made things over 2x slower despite a genuine ~10% drop in major faults.**Background thread pool (reverted):** issuing prefetch`pread()`\n\ns from a worker thread so compute for one layer could overlap disk reads for the next was meant to hide I/O behind compute. It did the opposite — one background`pread()`\n\nthread and the compute thread's own page faults contending for the same disk queue cost far more than any overlap saved, and majflt/token rose 7.4x. There's no idle core on a 2-core box to hide that contention behind, and this disk didn't show the concurrent-queue-depth throughput gain a network download earlier in the project had (6 parallel HTTP range requests: 1.3 → 23 MB/s) — local disk and remote CDN throttling are different bottlenecks.\n\n**Time breakdown at the best config** (mlock + pread + madvise, Q4_K_M,\n`results/step4/q4_32tok_buckets.log`\n\n): of 8.767s/token, pread is 5.091s (58%), real\ncompute is 3.658s (42%), and madvise + the harness's own CSV-logging bookkeeping are both\nnoise (<0.2% combined, confirmed by re-running with logging fully disabled: 0.109 tok/s,\nstatistically indistinguishable from 0.114 tok/s with logging on). pread, gated by real\ndisk throughput, is the floor — and per Finding 7's other two rows, this box has no\ncheap way to shrink it further.\n\n```\ngit clone https://github.com/ggml-org/llama.cpp\ncd llama.cpp\ngit checkout 9a286ac98d2cab74231bd3f1fc3f2b8bdf05422e   # commit this was built against\ngit apply /path/to/moe-offload-bench/harness/llama.cpp.patch\ncp /path/to/moe-offload-bench/harness/expert-log.cpp examples/expert-log/\ncp /path/to/moe-offload-bench/harness/CMakeLists.txt examples/expert-log/\ncmake -B build -DCMAKE_BUILD_TYPE=Release\ncmake --build build --target llama-expert-log -j2\n\n./build/bin/llama-expert-log -m /path/to/olmoe-1b-7b-0924-q4_k_m.gguf \\\n    -o results.csv -p \"The history of the Roman Empire begins with\" -n 32\n```\n\nCLI flags: `-q`\n\ndisables per-expert CSV logging (measures instrumentation cost, Finding\n7). mlock of non-expert tensors is unconditional.\n\n- This is a\n**single shared/variable VM**, not an isolated benchmark rig —`dmesg`\n\nshows`virtio_balloon: Out of puff`\n\nevents during the heaviest runs, and repeated measurements of the same config vary by 10-20% run to run (occasionally more; the compute-ceiling re-measurement for this write-up came back ~2x higher than an earlier unlogged estimate). Every number in this repo is real and reproducible in direction, but treat absolute figures as \"this box, this hour,\" not a portable hardware spec. - All findings use\n`n=32`\n\ngenerated tokens per run (single run per config, not averaged across repeats) — enough to see clear, consistent, order-of-magnitude effects, not enough to quote a confidence interval on a ±5% difference. `results/`\n\nalso contains earlier one-off debugging runs (`expert_activations_meta_test*`\n\n,`expert_activations_naive2*`\n\n,`validate_cache.csv`\n\n, etc.) from before the harness reached its current form. They're kept for the record but aren't referenced above.", "url": "https://wpnews.pro/news/moe-expert-offloading-on-a-2-core-celeron-with-2-7gb-ram", "canonical_source": "https://github.com/dhishwasher/moe-offload-bench", "published_at": "2026-09-01 22:37:12+00:00", "updated_at": "2026-09-01 22:52:11.556824+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure"], "entities": ["Intel Celeron N4000", "OLMoE-1B-7B-0924", "llama.cpp", "Qwen2.5-0.5B-Instruct", "ChromeOS Crostini"], "alternates": {"html": "https://wpnews.pro/news/moe-expert-offloading-on-a-2-core-celeron-with-2-7gb-ram", "markdown": "https://wpnews.pro/news/moe-expert-offloading-on-a-2-core-celeron-with-2-7gb-ram.md", "text": "https://wpnews.pro/news/moe-expert-offloading-on-a-2-core-celeron-with-2-7gb-ram.txt", "jsonld": "https://wpnews.pro/news/moe-expert-offloading-on-a-2-core-celeron-with-2-7gb-ram.jsonld"}}