{"slug": "a-763b-moe-on-two-96-gb-gpus-put-the-idle-experts-in-host-ram-then-measure-per", "title": "A 763B MoE on two 96 GB GPUs: put the idle experts in host RAM, then measure per stream", "summary": "InferenceKey assembled a stack that runs the 763B-parameter DeepSeek-V4.1-Flash mixture-of-experts model on two 96 GB RTX PRO 6000 Blackwell GPUs by moving 4,608 of 15,360 routed experts plus lookup tables into host RAM and reading them over PCIe via UVA, dropping each card to 72.4 GiB. The team found stock vLLM's own UVA offload crashed during initialization or on the first request, and measured 186 median tokens/s for a single agent stream, 59 at four concurrent agents and 40 at six, with GPUs on different NUMA nodes running 1.17x to 1.51x slower.", "body_md": "The model has 763B parameters, the server has two graphics cards, and the gap was about six gigabytes per card. DeepSeek-V4.1-Flash, even compressed to 3 bits, wanted 102.1 GiB of VRAM (the memory on the GPU itself) on each of two RTX PRO 6000 Blackwell cards, and each card offered 95.6 GiB. Two launches in a row died with out-of-memory errors, before any room for the KV cache, the memory that holds each conversation's context.\n\nFor anyone running a model like this on their own hardware, the useful part is which setting got it in, which one looked equivalent and failed, and what to measure afterwards. The [full write-up of DeepSeek-V4.1-Flash on two 96 GB GPUs](https://inferencekey.com/research/deepseek-two-gpus-19-seats) has every number, the method and the limits.\n\nIn BF16 (two bytes per parameter, the uncompressed format) the weights would be about 1.5 TB. DeepSeek ships its checkpoint in FP4 and FP8 at 475.2 GiB. coolbho3k's 3-bit EXL3 pack is 309.3 GiB (EXL3 is a quantization format: fewer bits per weight, a smaller file). The cards add up to 192 GB. Shrinking the weights got close, not close enough.\n\nThe way out is that it is a mixture of experts: each of its 40 layers holds 384 routed experts, small sub-networks, and every token (a chunk of a word) passes through only 6 of them per layer. The idle ones do not need to live on the card. Diffbot's per-expert plan moves 4,608 of the 15,360 routed experts to host RAM, the server's ordinary memory, plus lookup tables (both Engram tables in int4 and the embedding). The GPU reads them where they sit, across the PCIe bus, through UVA (unified virtual addressing, which lets the card address host memory directly). Each card dropped to 72.4 GiB, which leaves space for context.\n\nYour first instinct will be vLLM's own UVA offload (vLLM is the open source model server underneath). It looks like the same thing, but with an identical set of bytes on the host, it crashed during initialization with a KV block size of 64, and with 128 it started and then crashed on the first request. A stock vLLM install cannot serve this model on these two cards.\n\nWhat works is a stack, and every layer has an author. DeepSeek published the weights. coolbho3k quantized them to 3 bits. turboderp wrote the EXL3 format and ExLlamaV3, the library that runs it fast. Victor Cruz built vllm-exl3, the bridge into vLLM. Diffbot designed the per-expert plan, the split launch and the kernels, on top of vLLM. InferenceKey assembled those pieces, got them running together and measured the result.\n\nWith experts in host RAM, the path from each GPU to that memory sets part of the speed. On one rented machine the GPUs sat on different NUMA nodes (groups of CPU cores, each with its own local memory), and it ran between 1.17 and 1.51 times slower. Check before measuring:\n\n```\nnvidia-smi topo -m\n# Shows how each GPU connects to the other GPU and to the CPUs.\n# GPUs on different NUMA nodes ran 1.17x to 1.51x slower here.\n```\n\nOtherwise you may blame the offload for a slow machine.\n\nA developer feels the speed of their own session, not the box's total, so measure tokens per second per stream (one stream is one agent's session). The article sets the bar at 40 tokens per second for each person. These are the median and the 5th percentile (the slowest 5% of samples) on agent turns:\n\n| Concurrent agents | Median tok/s | p5 tok/s | \n|---|---|---|\n| 1 | 186 | 134 | \n| 4 | 59 | 52 | \n| 6 | 40 | 34 | \n\nFour streams at once, with a 10% margin, is the capacity. Long context hardly slowed generation: the median time per output token was 7.6 ms after 4K tokens of context and 8.3 ms after 256K. Reading a long prompt the first time is the expensive step, 18.33 s for 128K tokens, which is why prefix caching (keeping an already processed prompt so it is not read again) matters. With a cached 256K prefix and 4 users, 95% of requests got their first token within 1.66 s.\n\nOne developer's real week with Claude Code had the model generating for roughly 2.8 hours, 1.7% of the week. Treat each person as active with a 7.1% chance in an 8 hour window, independently of the others, and a team of 19 goes over 4 simultaneous streams no more than 1% of the time. Colleagues on a shared schedule are not independent, so the estimate leans generous, and it rests on one person's usage. The [seat model behind the 19 developers per server](https://inferencekey.com/research/deepseek-two-gpus-19-seats) also sets the hardware against 180 euro subscriptions.\n\nGoing to 3 bits changes the model, so it was compared with the reference: fixed subsets of three benchmarks, thinking turned off, against DeepSeek's own API. GSM8K (math) came out at 0.960 on both sides. IFEval (instruction following) gave +0.013, interval [-0.013, +0.047]. HumanEval (code) gave -0.018, interval [-0.049, +0.012]. Both intervals include zero, and on roughly 150 questions a point or two is chance. The LiveBench agentic coding score (77.3, first of 69) is for the unquantized model at max effort.\n\nEvery experiment ran on one boot only, and the agent turns were synthetic rather than captured from real sessions. There is also no baseline yet with the whole model in VRAM on a larger machine. That run is already written up as experiment E9. The method is in the [paper (PDF)](https://assets.revoluciontech.org/inferencekey/papers/deepseek-v4-1-flash-two-96gb-gpus.pdf).\n\nSizing a box for a team of coding agents? Start from the [complete DeepSeek-V4.1-Flash measurements on inferencekey.com](https://inferencekey.com/research/deepseek-two-gpus-19-seats): the memory budget, per-stream speeds across six scenarios, the seat model and its limits.", "url": "https://wpnews.pro/news/a-763b-moe-on-two-96-gb-gpus-put-the-idle-experts-in-host-ram-then-measure-per", "canonical_source": "https://dev.to/inferencekey/a-763b-moe-on-two-96-gb-gpus-put-the-idle-experts-in-host-ram-then-measure-per-stream-40hm", "published_at": "2026-10-16 05:33:04+00:00", "updated_at": "2026-10-16 05:34:58.383079+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["DeepSeek", "DeepSeek-V4.1-Flash", "InferenceKey", "Diffbot", "vLLM", "ExLlamaV3", "turboderp", "coolbho3k"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-763b-moe-on-two-96-gb-gpus-put-the-idle-experts-in-host-ram-then-measure-per", "markdown": "https://wpnews.pro/news/a-763b-moe-on-two-96-gb-gpus-put-the-idle-experts-in-host-ram-then-measure-per.md", "text": "https://wpnews.pro/news/a-763b-moe-on-two-96-gb-gpus-put-the-idle-experts-in-host-ram-then-measure-per.txt", "jsonld": "https://wpnews.pro/news/a-763b-moe-on-two-96-gb-gpus-put-the-idle-experts-in-host-ram-then-measure-per.jsonld"}}