A Mixture-of-Experts model keeps roughly 97 percent of itself asleep at any given moment. That one fact means models far bigger than your memory can still run on your machine, slowly but genuinely, if you let the sleeping parts stay on disk and stream them in when they are called. This explains exactly how that works, using DeepSeek V4’s published architecture for the math, and includes a complete streaming engine in C you can compile and run in about ten seconds. Running it, I watched a model 24 times larger than the memory it occupied sit there generating tokens.
The first time you look at the numbers on a model like DeepSeek V4, it reads like a closed door. 1.6 trillion parameters. Around 800 gigabytes on disk even after squeezing every weight down to four bits. And every serving guide says the same thing, all of those weights have to sit in GPU memory at once, because the router might pick any expert on any token. If you own a laptop rather than a rack of accelerators, the obvious conclusion is that this model simply is not for you.
It can be, though. Not quickly, but genuinely, and the reason is hiding in how these models are built.
That serving advice is correct for the job it was written for, which is running many users at high throughput on rented hardware. But it quietly turns a speed choice into a hardware requirement, and those aren’t the same thing. The router can pick any expert, true. It doesn’t pick all of them, and it doesn’t pick them evenly. The gap between what might be needed and what actually gets touched on any given token is enormous, and that gap is the opening.
So this piece does two things. It works through the sparsity math using DeepSeek V4’s published architecture, so you can see where the room comes from. Then it builds a complete miniature streaming engine in C, the same shape as V4, small enough that you can compile it and watch the effect yourself in about ten seconds. Every number I quote from that engine came out of running it.
Start with what DeepSeek published about V4, because the numbers do the arguing.
V4-Pro has 1.6 trillion total parameters and activates about 49 billion per token. That’s 3.1 percent. V4-Flash has 284 billion total and activates about 13 billion, which is 4.6 percent. Both use 61 transformer layers with a hidden dimension of 7168, and each MoE layer holds one shared expert plus a large pool of routed ones, 384 for Pro and 256 for Flash, with the router picking the top 6 per token. Six of 384 is 1.6 percent of the expert pool firing at any moment. The other 98.4 percent sits idle.
Now put a size on one expert, because the expert is the unit of work in a streaming design. An expert is three matrices, gate, up, and down, and with an intermediate dimension of 2048 against a hidden dimension of 7168 that is three times 2048 times 7168, or about 44 million parameters. At four bits per parameter, packing two values into every byte, one expert is 21 mebibytes.
That single figure sets everything else. Per token, V4-Pro reads 6 experts across 61 layers, which is 366 expert reads, and if every single one of those missed a cache it would be 7.5 gibibytes of reading per token. That’s the worst case, and the entire art of a streaming engine is making sure you almost never pay it.
The other side of the ledger is the disk footprint. At four bits, V4-Pro’s 1.6 trillion parameters occupy roughly 800 gigabytes, and V4-Flash’s 284 billion occupy about 142. Those are large numbers for a disk and impossible ones for consumer VRAM. But the part that must be resident isn’t the whole model, it’s the always-on machinery, the attention weights, the shared expert, the embeddings, the routers, and the norms. The routed experts, which are the overwhelming majority of the bytes, are the part that can live on disk and come in when called.
So the design writes itself. Keep the always-on part in memory. Keep a modest cache of recently used experts in memory too. Read the rest from disk on demand, and drop the pages afterward so your resident set doesn’t creep toward the model size. Whether that actually works depends on one empirical question, which is whether routing is skewed enough that a small cache catches most requests. Let’s find out by building it.
I wrote a miniature streaming MoE engine in C, in one file, with no libraries beyond the math library. It mirrors DeepSeek V4’s shape, 256 routed experts per layer with top-6 routing, but with small dimensions so the whole thing builds and runs in seconds. Every mechanism in it is the real one. It packs weights to int4 with per-row scales, it reads experts with pread and drops the pages with posix_fadvise, it keeps a per-layer least recently used cache, and it does its inner products in integer arithmetic.
The weights are synthetic random values, and I want to be plain about that up front. This demonstrates the memory mechanism, not model quality. What it can prove is exactly what it claims to prove, that resident memory stays small while the model on disk doesn’t, and that the caching works because routing is skewed.
The first piece is quantization, because four bits per parameter is what turns an impossible disk footprint into a merely large one. The scheme is symmetric and per row. For each output row you find the largest absolute value, divide by 7 to get a scale, and store each weight as a rounded nibble in the range from negative 8 to positive 7. Two nibbles pack into one byte, stored plus 8 so the byte stays unsigned.
static void pack_int4(const float *w, unsigned char *q4, float *scale, int rows, int cols) { int rb = (cols + 1) / 2; for (int o = 0; o < rows; o++) { const float *wr = w + (size_t)o * cols; float amax = 0.f; for (int i = 0; i < cols; i++) { float a = fabsf(wr[i]); if (a > amax) amax = a; } float s = amax / 7.f; /* 4-bit signed max magnitude is 7 */ if (s < 1e-8f) s = 1e-8f; scale[o] = s; unsigned char *qr = q4 + (size_t)o * rb; for (int i = 0; i < cols; i += 2) { int v0 = (int)lrintf(wr[i] / s); if (v0 > 7) v0 = 7; if (v0 < -8) v0 = -8; int v1 = 0; if (i + 1 < cols) { v1 = (int)lrintf(wr[i + 1] / s); if (v1 > 7) v1 = 7; if (v1 < -8) v1 = -8; } qr[i >> 1] = (unsigned char)((v0 + 8) | ((v1 + 8) << 4)); } }}
Reading those weights back is where it gets interesting, because you don’t have to convert them to floats to use them. If you also quantize the activation to int8 with its own scale, then the entire inner product is integer arithmetic and the two scales get applied once at the end.
static float dot_int4_row(const unsigned char *qr, const signed char *xq, int cols, float w_scale, float x_scale) { int acc = 0; int i = 0; for (; i + 1 < cols; i += 2) { unsigned char b = qr[i >> 1]; acc += ((int)(b & 0x0F) - 8) * (int)xq[i]; acc += ((int)(b >> 4) - 8) * (int)xq[i + 1]; } if (i < cols) { unsigned char b = qr[i >> 1]; acc += ((int)(b & 0x0F) - 8) * (int)xq[i]; } return (float)acc * w_scale * x_scale;}
Does it actually work? The engine self-tests the packing against a plain float reference on startup, and this is what it printed.
int4 self-test : 32 rows x 128 cols relative RMS error 5.90% cosine similarity 0.99827 packing 128 cols -> 64 bytes/row (2.0 values/byte)
Two values per byte confirms the packing is genuinely four bits. A relative RMS error near 6 percent with a cosine similarity of 0.998 is what four-bit weights against random data should give, and it matches the theoretical estimate closely enough that I trust the implementation.
One honest note about that metric. My first version of this test reported a mean per-row relative error of 16 percent with a worst case over 200 percent, which looked alarming. It was a bad metric, not a bad kernel. A dot product of two random uncorrelated vectors often lands near zero, and dividing by a near-zero reference makes the ratio explode for reasons that have nothing to do with quantization. Relative RMS error across all rows is the honest measure, and it tells a completely different and much calmer story. It’s worth knowing that the scary version of that number is an artifact.
The second piece is the , and this is the heart of the whole design. When the router asks for an expert, we look in a small per-layer cache. A hit costs nothing. A miss reads the expert’s bytes straight from the file at a computed offset, into the least recently used slot.
static const unsigned char *expert_get(int layer, int eid) { LayerCache *lc = &g_cache[layer]; for (int i = 0; i < lc->n; i++) { if (lc->slots[i].expert_id == eid) { g_hits++; lc->slots[i].used = ++g_clock; return lc->slots[i].slab; } } g_misses++; Slot *s; if (lc->n < g_cache_slots) { s = &lc->slots[lc->n++]; s->slab = malloc(EXPERT_BYTES); } else { int lru = 0; for (int i = 1; i < lc->n; i++) if (lc->slots[i].used < lc->slots[lru].used) lru = i; s = &lc->slots[lru]; } off_t off = ((off_t)layer * N_EXPERTS + eid) * (off_t)EXPERT_BYTES; pread(g_fd, s->slab, EXPERT_BYTES, off); /* Drop the file pages we just read. Without this the page cache grows * inside our resident set and the memory saving disappears. */ posix_fadvise(g_fd, off, EXPERT_BYTES, POSIX_FADV_DONTNEED); s->expert_id = eid; s->used = ++g_clock; return s->slab;}
Two details in there carry the whole memory argument. The first is that an expert’s three matrices are contiguous in the file, so fetching one is a single read into a single allocation rather than three scattered ones. The second, and the one people miss, is the posix_fadvise call. If you simply memory-map the checkpoint and let the kernel page it in, every page you touch stays resident, and your memory use climbs steadily toward the size of the model, which defeats the entire purpose. Telling the kernel to forget those pages right after you have copied the bytes into your own buffer is what pins resident memory to the size of your cache instead of the size of your model.
Here’s the engine running. The model is 8 layers of 256 experts, 222 megabytes on disk, and the cache is swept from 2 slots per layer up to 64.
=== miniature streaming MoE ===shape: 8 layers, 256 experts/layer, top-6, d_model 512, d_ff 144expert size: 111.1 KB model on disk: 222.2 MB
baseline RSS before any expert is touched: 2.1 MB
--- cache sweep, skewed (realistic) routing --- 2 slots/layer hit 2.1% disk read 407.9 MB RSS 4.2 MB 4 slots/layer hit 7.2% disk read 386.6 MB RSS 5.9 MB 8 slots/layer hit 24.8% disk read 313.2 MB RSS 9.4 MB16 slots/layer hit 42.3% disk read 240.3 MB RSS 16.4 MB32 slots/layer hit 57.9% disk read 175.5 MB RSS 30.3 MB64 slots/layer hit 69.9% disk read 125.2 MB RSS 58.0 MB
The headline is the last two columns read together. A 222 megabyte model generated tokens with 9.4 megabytes resident, and even at the most generous cache setting it never went past 58. Resident memory tracks the cache you chose, not the model you loaded, which is precisely the property that makes a 142 gigabyte model tractable on a machine with 16 gigabytes of RAM.
The sweep also shows the shape of the trade. Every doubling of the cache buys a real jump in hit rate, from 2 percent at the smallest setting to 70 percent at the largest, and disk traffic falls by more than a factor of three across that range. More memory genuinely helps, in a smooth and predictable way, which is a comfortable thing to know when you’re deciding how much RAM to give the process.
None of this works if the router spreads its choices evenly, because then a small cache catches almost nothing. Real MoE routing is heavily skewed, a small head of experts fires far more often than the long tail, so I modeled that with a Zipf draw and then measured what happens if you take the skew away.
--- what the skew is worth (8 slots/layer) ---skewed routing hit 24.8% disk read 313.2 MB RSS 9.4 MBuniform routing hit 2.2% disk read 407.6 MB RSS 9.4 MB
Same cache size, same everything else, and the hit rate goes from 24.8 percent down to 2.2 percent when routing becomes uniform. That’s more than a factor of eleven, and it converts directly into a third more bytes pulled off the disk. The 2.2 percent figure isn’t really a cache working, it’s a cache of 8 slots holding 8 of 256 experts and getting lucky at exactly the rate chance predicts.
So the honest statement of why streaming MoE works isn’t “because MoE models are sparse.” Sparsity alone would give you the uniform row. It works because MoE models are sparse and their routing concentrates, so a cache that holds a small fraction of the experts catches a large fraction of the requests. If a future architecture deliberately flattened its routing distribution, this whole approach would degrade toward that second row, and the technique would stop paying.
Put the measured behavior next to V4’s published numbers and you can see the shape of a real deployment.
V4-Flash at four bits is about 142 gigabytes on disk. The always-on part, attention plus the shared expert plus embeddings and routers, is a small fraction of that and is what has to stay resident. The routed experts are the bulk, and they stream. With a cache of a few gigabytes and the routing skew doing its work, most expert requests are served from memory and only the tail reaches the disk. That’s a machine you can buy rather than rent.
The wall you hit isn’t memory, it’s time. Each expert read is 21 mebibytes, and a fast consumer NVMe drive delivers a few gigabytes per second under this kind of random access pattern. A token that misses on most of its 366 expert lookups is reading gigabytes, and gigabytes at a few gigabytes per second is a second or more per token. Published reports of people running very large MoE models this way land in the range of a fraction of a token per second on laptops, which is the honest speed you should expect. It’s fast enough to ask a question and read the answer. It isn’t fast enough for anything interactive.
That trade, memory for time, is the whole deal. You aren’t getting a datacenter deployment for free. You’re converting a hardware problem you can’t solve into a patience problem you can.
I’d rather draw this line clearly than let the demo imply more than it showed.
What the engine genuinely demonstrates is the memory mechanism. The int4 packing is correct and I measured its error. The streaming keeps resident memory at the size of the cache rather than the size of the model, and I measured that across six cache sizes. The caching works because routing is skewed, and I measured the difference against uniform routing. Those results are real, reproducible, and came from the code printed here.
What it doesn’t demonstrate is anything about DeepSeek V4 specifically. I didn’t download V4’s weights, I didn’t run V4, and the quality cost of quantizing a real model to four bits is a separate question I didn’t measure. The weights in my engine are random, which is fine for testing memory behavior and useless for testing output quality. Prior work at full scale has found that the sensitive parts of these models, the attention path and the embeddings, degrade much faster under aggressive quantization than the routed experts do, which is why serious implementations keep experts at four bits while holding the dense path at eight. That distinction matters and my synthetic test can’t see it.
The per-token speed estimate above is arithmetic from disk bandwidth, not a measurement of a real model on real hardware. Treat it as an order of magnitude.
I should also credit where this approach has been demonstrated properly. A recent write-up walked through building a full inference engine in C for a 744 billion parameter MoE model, streaming the experts from disk, with the whole pipeline from quantization to an OpenAI-compatible server and every number captured from real runs. That’s the full-scale version of what I’ve shown in miniature here, and it’s worth reading if this idea interests you.
Open weights are only meaningfully open if you can actually run them. A model published under a permissive license that requires 800 gigabytes of VRAM is open in a legal sense and closed in a practical one for almost everybody. The streaming approach changes that calculation, not by making the model fast on modest hardware, but by making it possible at all, which is a different and more important threshold.
There’s also a mental model worth taking away, and it generalizes past this one trick. The instinct to treat VRAM, RAM, and disk as three separate categories, where a model either fits or it doesn’t, is what makes a trillion-parameter model look impossible. Treating them instead as one memory hierarchy with different speeds, and asking which bytes actually need to be fast, turns the same problem into an engineering exercise about working sets and cache hit rates. That reframing is what the whole design rests on, and it’s the part that’ll still be useful when the specific models here are long obsolete.
Compile the code below, run it, and watch a model many times larger than the memory it occupies produce tokens. The mechanism isn’t complicated. It just requires being willing to let most of the model sleep.
One file, no dependencies beyond the math library. Build with gcc -O3 -march=native -o moe moe.c -lm and run ./moe. It writes a synthetic checkpoint, streams it back under several cache sizes, and reports what it measured.
/* * moe.c * A miniature streaming Mixture-of-Experts engine in pure C. * * It demonstrates the one idea that lets a model far larger than your RAM run * on your machine: in an MoE, only a few experts fire per token, so keep the * always-on part resident and stream the sleeping experts from disk on demand. * * The shape here mirrors DeepSeek V4: 256 routed experts per layer, top-6 per * token. The dimensions are scaled down so the whole thing builds and runs in * seconds, but every mechanism is the real one: int4 packing with per-row * scales, pread with posix_fadvise to keep pages from accumulating, a per-layer * LRU cache, and integer dot products. * * Build: gcc -O3 -march=native -o moe moe.c -lm * Run: ./moe (writes a checkpoint, then streams it back) */#define _GNU_SOURCE#include <stdio.h>#include <stdlib.h>#include <string.h>#include <math.h>#include <fcntl.h>#include <unistd.h>#include <time.h>/* ---- Model shape. Same expert ratio as DeepSeek V4, smaller dimensions. ---- */#define N_LAYERS 8#define N_EXPERTS 256 /* routed experts per layer, as in V4-Flash */#define TOP_K 6 /* experts that fire per token, as in V4 */#define D_MODEL 512 /* hidden size */#define D_FF 144 /* expert intermediate size *//* Cache: how many experts per layer we are willing to hold in RAM. Deliberately * tiny compared to N_EXPERTS, which is the whole point. */#define MAX_CACHE_SLOTS 64static int g_cache_slots = 8; /* runtime configurable */#define CKPT "experts.bin"/* One expert is three matrices: gate [D_FF,D_MODEL], up [D_FF,D_MODEL], * down [D_MODEL,D_FF]. Packed int4 is two values per byte, plus one f32 scale * per output row. */#define GATE_ROWS D_FF#define UP_ROWS D_FF#define DOWN_ROWS D_MODEL#define GATE_COLS D_MODEL#define UP_COLS D_MODEL#define DOWN_COLS D_FF#define PACKED(rows, cols) ((size_t)(rows) * (((cols) + 1) / 2))#define SCALES(rows) ((size_t)(rows) * sizeof(float))static const size_t EXPERT_BYTES = PACKED(GATE_ROWS, GATE_COLS) + SCALES(GATE_ROWS) + PACKED(UP_ROWS, UP_COLS) + SCALES(UP_ROWS) + PACKED(DOWN_ROWS, DOWN_COLS) + SCALES(DOWN_ROWS);/* ------------------------------------------------------------------------- *//* int4 packing: symmetric, per row. Each value becomes a nibble in [-8,7], * stored as v+8 so the byte is unsigned, two values per byte. *//* ------------------------------------------------------------------------- */static void pack_int4(const float *w, unsigned char *q4, float *scale, int rows, int cols) { int rb = (cols + 1) / 2; for (int o = 0; o < rows; o++) { const float *wr = w + (size_t)o * cols; float amax = 0.f; for (int i = 0; i < cols; i++) { float a = fabsf(wr[i]); if (a > amax) amax = a; } float s = amax / 7.f; /* 4-bit signed max magnitude is 7 */ if (s < 1e-8f) s = 1e-8f; scale[o] = s; unsigned char *qr = q4 + (size_t)o * rb; for (int i = 0; i < cols; i += 2) { int v0 = (int)lrintf(wr[i] / s); if (v0 > 7) v0 = 7; if (v0 < -8) v0 = -8; int v1 = 0; if (i + 1 < cols) { v1 = (int)lrintf(wr[i + 1] / s); if (v1 > 7) v1 = 7; if (v1 < -8) v1 = -8; } qr[i >> 1] = (unsigned char)((v0 + 8) | ((v1 + 8) << 4)); } }}/* Quantize an activation vector to int8 with one shared scale (absmax / 127). * Doing this lets the inner product below run entirely in integers. */static float quantize_activation(const float *x, signed char *q, int n) { float amax = 0.f; for (int i = 0; i < n; i++) { float a = fabsf(x[i]); if (a > amax) amax = a; } float s = amax / 127.f; if (s < 1e-12f) s = 1e-12f; for (int i = 0; i < n; i++) { int v = (int)lrintf(x[i] / s); if (v > 127) v = 127; if (v < -128) v = -128; q[i] = (signed char)v; } return s;}/* One int4-packed weight row against an int8 activation. Both sides are small * integers, so the whole inner product is integer arithmetic and the two scales * are applied once at the very end. */static float dot_int4_row(const unsigned char *qr, const signed char *xq, int cols, float w_scale, float x_scale) { int acc = 0; int i = 0; for (; i + 1 < cols; i += 2) { unsigned char b = qr[i >> 1]; acc += ((int)(b & 0x0F) - 8) * (int)xq[i]; acc += ((int)(b >> 4) - 8) * (int)xq[i + 1]; } if (i < cols) { unsigned char b = qr[i >> 1]; acc += ((int)(b & 0x0F) - 8) * (int)xq[i]; } return (float)acc * w_scale * x_scale;}/* ------------------------------------------------------------------------- *//* An expert slot in the cache. The three matrices live in one contiguous slab * so an expert is a single read. *//* ------------------------------------------------------------------------- */typedef struct { int expert_id; /* which expert is resident here, -1 if empty */ unsigned long used; /* recency stamp for LRU eviction */ unsigned char *slab; /* the expert's bytes, one allocation */} Slot;typedef struct { Slot slots[MAX_CACHE_SLOTS]; int n;} LayerCache;static LayerCache g_cache[N_LAYERS];static unsigned long g_clock = 0;static long g_hits = 0, g_misses = 0;static long long g_disk_bytes = 0;static int g_fd = -1;/* Views into a slab. */typedef struct { const unsigned char *gate_q, *up_q, *down_q; const float *gate_s, *up_s, *down_s;} Expert;static void expert_views(const unsigned char *slab, Expert *e) { size_t off = 0; e->gate_q = slab + off; off += PACKED(GATE_ROWS, GATE_COLS); e->gate_s = (const float *)(slab + off); off += SCALES(GATE_ROWS); e->up_q = slab + off; off += PACKED(UP_ROWS, UP_COLS); e->up_s = (const float *)(slab + off); off += SCALES(UP_ROWS); e->down_q = slab + off; off += PACKED(DOWN_ROWS, DOWN_COLS); e->down_s = (const float *)(slab + off);}/* The heart of it. Return an expert's weights, from cache if resident, else * read them from disk into the least-recently-used slot. After the read we tell * the kernel to drop those pages, so our resident memory stays at the size of * the cache rather than creeping toward the size of the model. */static const unsigned char *expert_get(int layer, int eid) { LayerCache *lc = &g_cache[layer]; for (int i = 0; i < lc->n; i++) { if (lc->slots[i].expert_id == eid) { g_hits++; lc->slots[i].used = ++g_clock; return lc->slots[i].slab; } } g_misses++; Slot *s; if (lc->n < g_cache_slots) { s = &lc->slots[lc->n++]; s->slab = malloc(EXPERT_BYTES); if (!s->slab) { perror("malloc slab"); exit(1); } } else { int lru = 0; for (int i = 1; i < lc->n; i++) if (lc->slots[i].used < lc->slots[lru].used) lru = i; s = &lc->slots[lru]; } off_t off = ((off_t)layer * N_EXPERTS + eid) * (off_t)EXPERT_BYTES; ssize_t got = pread(g_fd, s->slab, EXPERT_BYTES, off); if (got != (ssize_t)EXPERT_BYTES) { perror("pread expert"); exit(1); } g_disk_bytes += got; /* Drop the file pages we just read. Without this the page cache grows * inside our resident set and the memory saving disappears. */ posix_fadvise(g_fd, off, EXPERT_BYTES, POSIX_FADV_DONTNEED); s->expert_id = eid; s->used = ++g_clock; return s->slab;}/* ------------------------------------------------------------------------- *//* Routing. Real MoE routing is heavily skewed: a small head of experts fires * far more often than the tail. We model that with a Zipf-like draw, and can * also switch to uniform to show what caching is worth when there is no skew. *//* ------------------------------------------------------------------------- */static double g_zipf_cdf[N_EXPERTS];static void routing_init(double s) { double sum = 0.0; for (int i = 0; i < N_EXPERTS; i++) sum += 1.0 / pow(i + 1, s); double acc = 0.0; for (int i = 0; i < N_EXPERTS; i++) { acc += (1.0 / pow(i + 1, s)) / sum; g_zipf_cdf[i] = acc; }}static int draw_expert(int uniform) { if (uniform) return rand() % N_EXPERTS; double u = (double)rand() / ((double)RAND_MAX + 1.0); int lo = 0, hi = N_EXPERTS - 1; while (lo < hi) { int mid = (lo + hi) / 2; if (u <= g_zipf_cdf[mid]) hi = mid; else lo = mid + 1; } return lo;}/* Pick TOP_K distinct experts for one token. */static void route(int *out, int uniform) { int n = 0; while (n < TOP_K) { int e = draw_expert(uniform); int dup = 0; for (int i = 0; i < n; i++) if (out[i] == e) { dup = 1; break; } if (!dup) out[n++] = e; }}/* ------------------------------------------------------------------------- *//* One expert's forward pass: SwiGLU. silu(gate(x)) * up(x), then down(). *//* ------------------------------------------------------------------------- */static inline float silu(float v) { return v / (1.f + expf(-v)); }static void expert_forward(const Expert *e, const float *x, float *out) { signed char xq[D_MODEL], hq[D_FF]; float h[D_FF]; float xs = quantize_activation(x, xq, D_MODEL); for (int r = 0; r < D_FF; r++) { const unsigned char *gr = e->gate_q + (size_t)r * ((GATE_COLS + 1) / 2); const unsigned char *ur = e->up_q + (size_t)r * ((UP_COLS + 1) / 2); float g = dot_int4_row(gr, xq, GATE_COLS, e->gate_s[r], xs); float u = dot_int4_row(ur, xq, UP_COLS, e->up_s[r], xs); h[r] = silu(g) * u; } float hs = quantize_activation(h, hq, D_FF); for (int r = 0; r < D_MODEL; r++) { const unsigned char *dr = e->down_q + (size_t)r * ((DOWN_COLS + 1) / 2); out[r] = dot_int4_row(dr, hq, DOWN_COLS, e->down_s[r], hs); }}/* ------------------------------------------------------------------------- *//* Resident memory, straight from the kernel. *//* ------------------------------------------------------------------------- */static double rss_mb(void) { FILE *f = fopen("/proc/self/statm", "r"); if (!f) return -1; long size = 0, resident = 0; if (fscanf(f, "%ld %ld", &size, &resident) != 2) { fclose(f); return -1; } fclose(f); return (double)resident * (double)sysconf(_SC_PAGESIZE) / (1024.0 * 1024.0);}static double now_sec(void) { struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); return ts.tv_sec + ts.tv_nsec * 1e-9;}/* ------------------------------------------------------------------------- *//* Write a synthetic checkpoint: every expert of every layer, int4 packed. *//* ------------------------------------------------------------------------- */static void write_checkpoint(void) { printf("writing checkpoint: %d layers x %d experts, %.1f KB per expert\n", N_LAYERS, N_EXPERTS, EXPERT_BYTES / 1024.0); int fd = open(CKPT, O_WRONLY | O_CREAT | O_TRUNC, 0644); if (fd < 0) { perror("open ckpt"); exit(1); } float *w = malloc(sizeof(float) * (size_t)D_FF * D_MODEL); unsigned char *slab = malloc(EXPERT_BYTES); if (!w || !slab) { perror("malloc"); exit(1); } for (int l = 0; l < N_LAYERS; l++) { for (int e = 0; e < N_EXPERTS; e++) { size_t off = 0; /* gate */ for (size_t i = 0; i < (size_t)GATE_ROWS * GATE_COLS; i++) w[i] = ((float)rand() / RAND_MAX - 0.5f) * 0.1f; pack_int4(w, slab + off, (float *)(slab + off + PACKED(GATE_ROWS, GATE_COLS)), GATE_ROWS, GATE_COLS); off += PACKED(GATE_ROWS, GATE_COLS) + SCALES(GATE_ROWS); /* up */ for (size_t i = 0; i < (size_t)UP_ROWS * UP_COLS; i++) w[i] = ((float)rand() / RAND_MAX - 0.5f) * 0.1f; pack_int4(w, slab + off, (float *)(slab + off + PACKED(UP_ROWS, UP_COLS)), UP_ROWS, UP_COLS); off += PACKED(UP_ROWS, UP_COLS) + SCALES(UP_ROWS); /* down */ for (size_t i = 0; i < (size_t)DOWN_ROWS * DOWN_COLS; i++) w[i] = ((float)rand() / RAND_MAX - 0.5f) * 0.1f; pack_int4(w, slab + off, (float *)(slab + off + PACKED(DOWN_ROWS, DOWN_COLS)), DOWN_ROWS, DOWN_COLS); if (write(fd, slab, EXPERT_BYTES) != (ssize_t)EXPERT_BYTES) { perror("write expert"); exit(1); } } } free(w); free(slab); close(fd);}/* ------------------------------------------------------------------------- */static void run(int n_tokens, int uniform, const char *label) { for (int l = 0; l < N_LAYERS; l++) { for (int i = 0; i < MAX_CACHE_SLOTS; i++) { g_cache[l].slots[i].expert_id = -1; g_cache[l].slots[i].used = 0; if (g_cache[l].slots[i].slab) { free(g_cache[l].slots[i].slab); g_cache[l].slots[i].slab = NULL; } } g_cache[l].n = 0; } g_hits = g_misses = 0; g_disk_bytes = 0; srand(1234); float x[D_MODEL], acc[D_MODEL], tmp[D_MODEL]; for (int i = 0; i < D_MODEL; i++) x[i] = ((float)rand() / RAND_MAX - 0.5f); double t0 = now_sec(); for (int t = 0; t < n_tokens; t++) { for (int l = 0; l < N_LAYERS; l++) { int chosen[TOP_K]; route(chosen, uniform); memset(acc, 0, sizeof(acc)); for (int k = 0; k < TOP_K; k++) { const unsigned char *slab = expert_get(l, chosen[k]); Expert e; expert_views(slab, &e); expert_forward(&e, x, tmp); for (int i = 0; i < D_MODEL; i++) acc[i] += tmp[i] / TOP_K; } for (int i = 0; i < D_MODEL; i++) x[i] = 0.9f * x[i] + 0.1f * acc[i]; } } double dt = now_sec() - t0; long total = g_hits + g_misses; printf("%-22s hit %5.1f%% (%ld hit / %ld miss) disk read %6.1f MB " "RSS %6.1f MB %.2f tok/s\n", label, total ? 100.0 * g_hits / total : 0.0, g_hits, g_misses, g_disk_bytes / (1024.0 * 1024.0), rss_mb(), n_tokens / dt);}/* ------------------------------------------------------------------------- *//* Correctness: pack a known matrix, then check the int4 dot product against a * plain float reference. int4 is lossy, so we report the relative error rather * than demand an exact match. *//* ------------------------------------------------------------------------- */static void selftest_int4(void) { enum { R = 32, C = 128 }; float *w = malloc(sizeof(float) * R * C); float *x = malloc(sizeof(float) * C); unsigned char *q = malloc(PACKED(R, C)); float *sc = malloc(SCALES(R)); srand(7); for (int i = 0; i < R * C; i++) w[i] = ((float)rand() / RAND_MAX - 0.5f) * 2.0f; for (int i = 0; i < C; i++) x[i] = ((float)rand() / RAND_MAX - 0.5f) * 2.0f; pack_int4(w, q, sc, R, C); signed char *xq = malloc(C); float xs = quantize_activation(x, xq, C); /* Relative RMS error is the honest metric here. Per-row relative error is * misleading because a random dot product often lands near zero, which * makes the ratio explode for reasons that have nothing to do with int4. */ double se = 0.0, sref = 0.0, dotp = 0.0, nref = 0.0, ngot = 0.0; for (int r = 0; r < R; r++) { double ref = 0.0; for (int i = 0; i < C; i++) ref += (double)w[(size_t)r * C + i] * x[i]; float got = dot_int4_row(q + (size_t)r * ((C + 1) / 2), xq, C, sc[r], xs); double d = got - ref; se += d * d; sref += ref * ref; dotp += (double)got * ref; nref += ref * ref; ngot += (double)got * got; } printf("int4 self-test : %d rows x %d cols\n", R, C); printf(" relative RMS error %.2f%%\n", 100.0 * sqrt(se / sref)); printf(" cosine similarity %.5f\n", dotp / (sqrt(nref) * sqrt(ngot))); printf(" packing %d cols -> %zu bytes/row (%.1f values/byte)\n", C, PACKED(1, C), (double)C / PACKED(1, C)); free(w); free(x); free(q); free(sc); free(xq);}int main(void) { printf("=== miniature streaming MoE ===\n"); printf("shape: %d layers, %d experts/layer, top-%d, d_model %d, d_ff %d\n", N_LAYERS, N_EXPERTS, TOP_K, D_MODEL, D_FF); double model_mb = (double)EXPERT_BYTES * N_EXPERTS * N_LAYERS / (1024.0 * 1024.0); printf("expert size: %.1f KB model on disk: %.1f MB\n\n", EXPERT_BYTES / 1024.0, model_mb); selftest_int4(); srand(1); write_checkpoint(); g_fd = open(CKPT, O_RDONLY); if (g_fd < 0) { perror("open ckpt for read"); exit(1); } printf("\nbaseline RSS before any expert is touched: %.1f MB\n\n", rss_mb()); routing_init(1.2); printf("--- cache sweep, skewed (realistic) routing ---\n"); int sizes[] = {2, 4, 8, 16, 32, 64}; for (unsigned i = 0; i < sizeof(sizes) / sizeof(sizes[0]); i++) { g_cache_slots = sizes[i]; char label[64]; snprintf(label, sizeof(label), "%2d slots/layer", sizes[i]); run(80, 0, label); } printf("\n--- what the skew is worth (8 slots/layer) ---\n"); g_cache_slots = 8; run(80, 0, "skewed routing"); run(80, 1, "uniform routing"); printf("\nmodel on disk %.1f MB. Resident never approached it.\n", model_mb); close(g_fd); unlink(CKPT); return 0;}
The engine here uses synthetic weights and is built to demonstrate the memory mechanism, not model quality or real-world speed. Architecture figures for DeepSeek V4 come from published configuration and model documentation, and the per-token speed estimate is arithmetic rather than a measurement. Model details and tooling change quickly, so check current documentation before building on any of it.
How a 1.6 Trillion Parameter Model Fits on a Laptop, With C Code You Can Run was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.