The core logic here is applying BOLT/PGO concepts to model binaries. In MoE routing, experts tend to fire in cliques rather than randomly. If an inference engine streams experts from an SSD, the physical file layout determines whether a token's misses result in a few sequential reads or thousands of scattered, random reads. By tracing routing and clustering co-activation, we can rewrite the file to optimize this layout while keeping the weights byte-exact.
The Technical Workflow #
This is essentially a layout pass for model weights to minimize NVMe seek overhead. The process follows these steps:
-
Trace Routing: Analyze which experts are activated together across a representative dataset.
-
Cluster Co-activation: Group experts that frequently fire in the same sequence.
-
Rewrite Binary: Reorder the tensors in the model file to match these clusters.
-
Verification: Ensure weights remain byte-identical and routing mappings are 100% preserved.
Real-World Performance & Constraints #
The impact of this optimization depends entirely on the inference engine's memory management.
mmap-based engines (e.g., stock llama.cpp): These are generally layout-blind. Page-fault paths don't see the benefit of this reordering, resulting in parity.Explicit-read streamers (e.g., MLX expert-offload or Colibri): These see massive gains because they issue explicit reads from disk.
In testing, I initially hypothesized that "pinning" hot experts in RAM or prefetching across layers would help. Data proved me wrong:
Heat-pinning: Actually loses to a plain per-layer LRU cache because frozen slots prevent the workload's natural drift from being cached.Cross-layer prefetch: Fails because expert overlap between adjacent layers is negligible (0.017 vs 0.016 for shuffled controls), meaning there's no signal to predict.
The layout optimization remains the only viable lever. For those building LLM agents or deployment pipelines for massive MoE models on consumer hardware, focusing on the physical disk layout of weights is a high-leverage move.
For those interested in the implementation, the logic is handled via the mbolt project:
https://github.com/doramirdor/mbolt
Next GPT-5.5 Hidden Token Leak: 1,447 Tokens per Request →