# Want to Run Massive AI Models on a Collection of Home PCs? > Source: > Published: 2026-10-12 19:07:53+00:00 If you've tried running models like **Qwen2-57B** or **Mixtral-8x22B (141B parameters)** locally, you've probably hit the hard physical ceiling of consumer hardware: **VRAM exhaustion**. A single consumer GPU tops out around 8GB, 12GB, or 16GB of VRAM. Squeezing a 141B Mixture-of-Experts (MoE) model into one box usually demands an enterprise-grade 80GB datacenter accelerator costing thousands. Meanwhile, look around your desk or home lab: What if you could turn that collection of mismatched, heterogeneous hardware into a single pipeline cluster over a standard home gigabit network? That is why I built **[Glacier.Inference.Distributed](https://github.com/ian-cowley/Glacier.Inference.Distributed)**. Instead of requiring uniform cluster hardware or complex distributed training frameworks, `Glacier.Inference.Distributed` partitions GGUF transformer models across $N$ machines using **pipeline parallelism**. Model layers are sliced across available nodes. A single token's hidden state activation flows sequentially across the cluster: ``` [Token IDs] │ ▼ ┌──────────────────────────────────────────────┐ │ Node 0: Host Laptop (RTX 4060) │ │ Embeddings + Root Attention Blocks 0..N1 │ └──────────────────────┬───────────────────────┘ │ Fused Wire Frame (INT8 / FP16) ▼ ┌──────────────────────────────────────────────┐ │ Node 1: Desktop (RTX 3060 12GB) │ │ Transformer Blocks N1+1..N2 │ └──────────────────────┬───────────────────────┘ │ Streaming Socket Frame ▼ ┌──────────────────────────────────────────────┐ │ Node 2: Mini PC (AMD Radeon 680M APU) │ │ Transformer Blocks N2+1..N3 │ └──────────────────────┬───────────────────────┘ │ Low-Latency Socket Frame ▼ ┌──────────────────────────────────────────────┐ │ Node 3: Linux Server (Ryzen Zen 5 AVX-512) │ │ Final Blocks N3+1..End + LM Head / Logits │ └──────────────────────┬───────────────────────┘ │ Next Token ID ▼ [Next Token] ``` The most common assumption with distributed inference is that standard 1000BASE-T Gigabit Ethernet will choke the token generation speed. In practice, the math proves otherwise. During the sequential decode phase of an LLM, the cluster does **not** exchange model weights over the wire—each node keeps its assigned layers resident in local VRAM or system RAM. The only data crossing the network per token is the **activation tensor** (the hidden state vector). For a model with hidden dimension $d = 3584$: Over standard Gigabit Ethernet: When running heavy models, computing the multi-head attention and MoE feed-forward networks across active parameters takes hundreds of milliseconds to seconds. A ~8 ms wire delay accounts for **less than 0.2% of total token execution time**, leaving **99.8% of the run spent on raw compute**. Most distributed AI stacks drag along dozens of heavy dependencies: CUDA toolkit installations, Python runtime environments, Torch binaries, and native C++ orchestration wrappers. `Glacier.Inference.Distributed` is built from the metal up in **pure C# targeting .NET 10** (and is Native AOT-ready): `System.IO.Pipelines` utilizing 64-byte aligned socket framing (`ClusterFrameHeader` + `FusedActivationHeader`). Because the coordinator auto-inspects standard GGUF metadata, it automatically calculates layer boundaries and hidden dimensions based on the nodes you provide: | Model Architecture | Parameters | Active per Token | Slicing Example (4 Nodes) | Disk Footprint | |---|---|---|---|---| | **Qwen2-57B-A14B** | 57B | 14B (8/64 MoE) | 7 layers / node (28 total) | ~25.6 GB ( `Q3_K_M` ) | | **Mixtral-8x22B** | 141B | 39B (2/8 MoE) | 14 layers / node (56 total) | ~35.3 GB ( `IQ2_XXS` ) | | **Mixtral-8x7B** | 47B | 13B (2/8 MoE) | 8 layers / node (32 total) | ~26.4 GB ( `Q4_K_M` ) | | **DeepSeek-R1-Distill-14B** | 14.7B | 14.7B (Dense) | 12 layers / node (48 total) | ~9.0 GB ( `Q4_K_M` ) | Install the .NET 10 SDK on your machines and clone the repo: ``` git clone https://github.com/ian-cowley/Glacier.Inference.Distributed.git cd Glacier.Inference.Distributed ``` Rather than using `huggingface-cli` or external Python tools, Glacier includes a pure C# multi-stream HTTP Range downloader that opens concurrent sockets to saturate your bandwidth: ``` # Download Qwen2-57B dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- download qwen2-57b # Or download Mixtral-8x22B (141B) dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- download mixtral-8x22b ``` On your secondary machines, start the worker daemon. Each worker auto-detects whether to initialize D3D12 GPU shaders or CPU SIMD: ``` # On Machine 1 (e.g. 192.168.1.108 with RTX 3060): dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5055 # On Machine 2 (e.g. 192.168.1.101 with Radeon 680M): dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5056 # On Machine 3 (e.g. 192.168.1.200 Linux CPU box): dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5057 ``` From your primary machine (e.g., host laptop), point the coordinator to the worker endpoints: ``` dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- \ --role coordinator \ --stages 192.168.1.108:5055,192.168.1.101:5056,192.168.1.200:5057 \ --model ./models/qwen2-57b-a14b-instruct-q3_k_m.gguf \ --tokens 50 \ --prompt "Mixture-of-Experts enables scaling parameter capacity across distributed home clusters by" ``` The coordinator connects over TCP, verifies the hardware state, assigns layer boundaries, and streams tokens directly to the console. If you prefer building applications directly in .NET rather than invoking a CLI tool, the coordinator can be embedded using a simple async streaming API: ``` js using Glacier.Inference.Distributed; var endpoints = new[] { new StageEndpoint("192.168.1.108", 5055, "RTX 3060 Node"), new StageEndpoint("192.168.1.101", 5056, "APU Node"), new StageEndpoint("192.168.1.200", 5057, "AVX-512 Server") }; using var coordinator = new DistributedPipelineCoordinator( stages: endpoints, modelPath: "models/Mixtral-8x22B-Instruct-v0.1.i1-IQ2_XXS.gguf"); await foreach (var token in coordinator.GenerateStreamAsync("Explain distributed tensor pipelines in simple terms.")) { Console.Write(token); } ``` Running empirical tests across 4 physical machines over a basic 1 Gbps switch with `DeepSeek-R1-Distill-Qwen-14B` yielded these verified metrics: Large AI models don't have to be restricted to cloud APIs or ultra-high-end enterprise workstations. By pairing lean, zero-allocation network framing with modern SIMD and graphics APIs, you can pool whatever spare hardware you already have sitting on your local network to run models that would otherwise never fit on a single desk machine. Check out the full source, benchmark harness, and documentation on GitHub: 👉 [github.com/ian-cowley/Glacier.Inference.Distributed](https://github.com/ian-cowley/Glacier.Inference.Distributed)