This article was written with the assistance of an AI, Claude (Anthropic), from my notes, ideas and measurements. I reviewed and edited the text, and every measurement was run on my own machines.
Last week I wrote about running Qwen3.8-Flash-Next, a 125-billion-parameter mixture-of-experts model, on a 2013 workstation with Strata, an open-source (MIT) engine built for this one model. Strata’s trick is to keep the most-used experts on the GPUs and every expert in RAM, so the CPU computes the few the GPUs don’t hold. That works while RAM is big enough. When it isn’t, Strata reads the missing experts from the SSD, and every token waits for the drive.
Strata can already split the model across several GPUs inside one PC. When someone asked to split it across machines, the author closed the request: “Several GPUs inside one PC are supported with — layer-split (see docs/MULTI_GPU.md); splitting a model across machines is not something we plan to build” (#1256). The code is MIT, so I built it myself as a small modification, and spent two days measuring it on three PCs in my house.
The short version: it works, and the network costs little. With a 4-bit model that did not fit in my best PC’s RAM, a second PC made generation 87% faster at the same quality, and 51% faster than one PC running Strata’s defaults. With a model that already fit, the second PC added nothing but power draw. Long prompts got slower. The rest of this article is how I got those numbers, and where they stop.
The model has 48 layers, each with 512 small experts (24,576 in all). Every token goes through 10 of them per layer, plus one shared expert.
My first idea was to keep the “cold” experts on the second PC and compute them there. I recorded which experts a real conversation used, and the record ruled it out on gigabit. The cold experts are spread over most layers: in every pass, 19 to 48 layers would have needed a trip to the other PC, adding 6–20 ms. Moving the experts themselves is worse. Strata’s adaptive cache swaps about 200 experts per second, and gigabit can carry 40 to 60 (an expert is about 3 MB in the 4-bit file, 2 MB in the 3.5-bit one).
Strata’s own multi-GPU mode points to the better design. Each GPU runs a block of consecutive layers, and between blocks only the hidden state moves: the model’s working memory, about 51 KB per token. My modification sends that hand-off over TCP instead of through RAM:
On my network a round trip took 158 µs for 2 KB and 1.36 ms for 64 KB, with a ceiling of 116 MB/s. In the runs below, A sent about 150 KB per verify window, roughly 1.3 ms of transfer.
Cancelling a request. If a user cancels mid-answer, both engines must stop after the same window, or the next request finds them out of step. My first version got this wrong. Now the cancel travels inside the messages the PCs already exchange.
Losing the link. If the link drops, both engines exit and restart: the second PC’s at once, under systemd, and the first PC’s at the latest when the next request arrives, started by its own server. A closed connection is noticed at once. A silent failure, such as a pulled cable or a frozen PC, sends no signal at all, so the link uses the kernel’s keepalive probes and a 10-second limit on unacknowledged data. My first version had neither, and a reviewer pointed out that it would have waited forever.
The modification adds about 1,060 lines to 11 files; 467 of them are the network link and its test. The new code runs only when the new --net-role option is given.
soft-2 also has an RTX 3060 on a PCIe x4 slot, and soft-27 an RTX 2070 SUPER on an x1 slot. With those slow slots, four cards were slower than one per PC, so unless stated each PC used the card on its fast slot.
Last week’s X79 built the binaries and hosted the first two-process tests and one of the correctness checks below. I left it out of the measured pairs: its Pascal cards and its CPU without AVX2 need a separate build, and it would have set the pace.
The 3.5-bit file (ISTA-DASLab’s IQ3_S, 83.6 GB) has 46.8 GiB of experts, too much for a 32 GB PC. On soft-2 alone, Strata reads part of them from the drive. Split between soft-2 and soft-27, every expert fits in RAM.
That is 29% faster. Both rows ran with Strata’s default shortcut that skips rarely chosen experts (more on it under day 2). It probably helps the single PC more, because there a skipped expert is also a drive read saved, so the gain at the same quality could be somewhat larger. I did not measure it.
Three of those rows need a word.
The 0 of 8 looks like a network bug, and it is not. Strata’s documentation says that at temperature 0 the same prompt can end in a different, equally good answer, because the expert cache and some timing-based choices change the arithmetic in the last bits. With those features switched off, the pair gave identical answers run after run.
The killed connection (ss -K) is the easy case: the socket closes cleanly. I tried it three times, but only once did the cut land inside an answer; the other two times the answer had already finished, so that row rests on a single reproduction.
The silent failure is the hard case. I ran it on day 2, on this same pair and file, once the link had its timeouts. In both failures the pair was back about three minutes later, the time the PCs need to reload the model. The silent case only looks faster in the table because its next request came later, while the second PC was already re.
Identical answers across two runs show that the pair is consistent with itself, not that it computes what one PC computes. Strata holds its own GPU split to a stricter standard: the hand-off between cards is bit-exact. So I ran the same check across the network, on two machines with two GPUs each:
On both machines the eight answers at temperature 0 were identical: 880 tokens on soft-14, with the same binary as the speed numbers, and 889 on the X79. The soft-14 run is the one that counts. The X79’s Titan Xp cards are Pascal, which Strata runs only through its experimental engine for older cards (OLDER_GPUS.md), so there even the one-process reference runs on a path the authors do not validate.
My first attempt matched only 3 of 8. The second card’s cache had 458 slots in one run and 567 in the other, and an expert computed on the GPU in one run and on the CPU in the other rounds differently, as Strata’s documentation warns.
Two limits remain. The answers were short (at most 128 tokens, about 110 on average), and a longer answer is more likely to meet a near-tie between two tokens that would expose a small difference. And identical text is strong evidence, but it is not a comparison of probabilities. My network version does not return them through the API yet. Strata’s engine can write teacher-forced log-probabilities to a file (STRATA_LOGPOS, what its quantization tables are measured with), and on the second PC, which holds the output head, that should work as it is. I have not tried it yet.
The second day was the real test. Unsloth’s UD-Q4_K_XL file is 111.3 GB, with 71.7 GiB of experts: more than soft-14’s 60 GiB of usable RAM. So on soft-14 alone, Strata’s setup does what it is designed to do:
In the pair, soft-14 holds all of its experts in RAM. soft-2 is about 3 GiB short. It keeps 16.7 GiB of its experts in RAM, and the 4.2 GiB its GPU caches live only there, so an expert that leaves the GPU comes back from its SSD: 0.3–1.4 GB per answer in these runs. I turned the shortcut off in the pair and checked in both engines’ logs that no expert was skipped, so the pair and soft-14 with the shortcut off run at the same quality.
Why bother with 4-bit at all? Strata’s authors compare each file with the 8-bit model (QUANTS.md): The 0.056 was measured with every expert in RAM on an AMD card; the 0.053 above comes from the RAM-budget test on an NVIDIA card. Both are against the same 8-bit reference.
The pair was soft-14 (first 32 layers, on its two 5060 Ti) and soft-2 (last 16 layers, on its 4070). soft-14 alone split its 48 layers between its two cards at layer 24. Every row uses the same binary, built from the same source, with the same settings apart from the shortcut, as marked.
Taking the median of the repeated runs, at the same quality the pair generates 87% faster (30.5 against 16.3 tokens per second) and decodes twice as fast. Against soft-14 alone with Strata’s default shortcut, which gives up some quality, it is 51% faster.
The last row is the control. The 3.5-bit file fits in soft-14 alone, and adding soft-2 left the speed where it was (56.9 against 57.8 tokens per second; decode 70.8 against 67.8) while nearly doubling the power at the wall. The pair pays off only when it changes where the experts live.
The link keeps its own counters. In every verify window the second PC waited about 39 ms for the first, and the first about 13 ms for the second, network included. The two PCs take turns, so a window lasts the sum: about 52 ms.
Per layer, soft-2 was the cheaper machine: about 0.75 ms against 1.2 ms on soft-14. Part of that is probably its higher hit rate. Its GPU held only 18% of its experts (1,437 of 8,192), yet found 99% of the ones it needed while decoding, against 88% on soft-14.
That looks too good, but it has nothing to do with the last layers or the network. On day 1 soft-2 alone found 99.3% with 25% of its experts on its cards, and each PC of the day-1 pair about 99% with about 20%. The counter does not count as misses the experts a GPU fetched from RAM over PCIe, but those were only 0–1.6% of routed experts in the logs I kept. soft-14 found fewer in every 4-bit run (80–88%), and 95% alone with the 3.5-bit file. I have not worked out why, and Strata does not log hit rates per layer.
With a single stream of requests, the fastest split puts as many layers as possible on the cheaper machine. RAM stops that at 16 layers on soft-2. I tried two ways around it, and both were worse:
So the smaller PC’s RAM sets the limit. The other limit is structural: each PC waits while the other works. Inside one PC, Strata can overlap consecutive windows between GPUs; on NVIDIA that is opt-in, and it was off in my single-PC runs too. My network version cannot do it yet. Overlapping windows, or serving two requests at once, would put that idle time to use.
soft-14 alone read the prompts at 578 and 790 tokens per second. On one PC, Strata reads a long prompt layer by layer: each layer’s experts are loaded once per block of up to 32K tokens, and the loads are planned around what is in RAM and on the drive.
The logs of the 8,192-chunk runs show where the pair lost its time:
So the slow long prompts are mostly a RAM shortage on the second PC. The network adds something too: 28K tokens are about 1.1 GB on the cable, about 10 s at gigabit speed, and because each chunk waits for the previous one to be acknowledged, that transfer is not fully hidden behind the computation. I did not measure how much of it is.
A second PC with enough RAM for its whole half should bring the pair level with one PC, not ahead of it. Even with all its experts in RAM, soft-14 needed 24 and 41 s for its 32 layers, while on its own it read all 48 in 20.5 and 35.2 s, with part of its experts on the drive. Some of the 24 and 41 s is waiting for soft-2, but the gap per layer is large: one PC reads a long prompt layer by layer, the pair chunk by chunk, each layer’s experts once per chunk. To win on long prompts, the pair would need the layer-by-layer path across the network, chunks that travel while the previous one is being computed, and perhaps Strata’s optional faster prompt kernels for 4-bit K-quants.
In the seven-prompt runs (prompts included), the pair used 11.9 joules per token, measured in the run where I recorded both PCs, against 10.7 for soft-14 alone at the same quality: about 11% more. soft-2’s estimate carries ±10%, about ±17 W or ±5% of the pair’s 341 W, so the difference is larger than that uncertainty. Part of it may still come from the estimate, which if anything runs high.
soft-2 drew about 169 W while it waited three-quarters of each window, and there is a clue to why. In the 3.5-bit pair, soft-2’s engine kept 3.6 CPU cores busy on average, although 99.4% of its experts were on its GPU and it sat idle about half of each window. That looks like worker threads spinning while they wait, which would be the first thing to fix for energy. I have not confirmed it with a profiler.
For this model, the 3.5-bit file on one PC is in a different class: 3.7 joules per token, at twice the speed of the 4-bit pair. The advantage is real, but narrow.
Strata’s authors decided not to build this, and that is their call. The modification is small because Strata’s GPU split already defines the layer blocks, the hand-off and the chunked prompt. What I added is a TCP transport for that hand-off, a RAM arena that holds only one PC’s layers, and handling for a lost connection.
Their contribution guide asks that a declined proposal not be resubmitted under another title, and that new evidence go to the original discussion. So I will post these numbers under #1256 and leave the decision to them. The modification is on my fork, strata-cluster, with setup and details in its docs/STRATA_CLUSTER.md. It is based on Strata 0.2.7, and it is not an official version.
Two Home PCs, One 125B Model: Splitting Strata’s Layers Over Gigabit Ethernet was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.