Want to Run Massive AI Models on a Collection of Home PCs?

If you've tried running models like Qwen2-57B or Mixtral-8x22B (141B parameters) locally, you've probably hit the hard physical ceiling of consumer hardware: VRAM exhaustion.

A single consumer GPU tops out around 8GB, 12GB, or 16GB of VRAM. Squeezing a 141B Mixture-of-Experts (MoE) model into one box usually demands an enterprise-grade 80GB datacenter accelerator costing thousands.

Meanwhile, look around your desk or home lab:

What if you could turn that collection of mismatched, heterogeneous hardware into a single pipeline cluster over a standard home gigabit network?

That is why I built Glacier.Inference.Distributed.

Instead of requiring uniform cluster hardware or complex distributed training frameworks, Glacier.Inference.Distributed partitions GGUF transformer models across $N$ machines using pipeline parallelism.

Model layers are sliced across available nodes. A single token's hidden state activation flows sequentially across the cluster:

[Token IDs]
     │
     ▼
┌──────────────────────────────────────────────┐
│ Node 0: Host Laptop (RTX 4060)               │
│ Embeddings + Root Attention Blocks 0..N1     │
└──────────────────────┬───────────────────────┘
                       │ Fused Wire Frame (INT8 / FP16)
                       ▼
┌──────────────────────────────────────────────┐
│ Node 1: Desktop (RTX 3060 12GB)              │
│ Transformer Blocks N1+1..N2                  │
└──────────────────────┬───────────────────────┘
                       │ Streaming Socket Frame
                       ▼
┌──────────────────────────────────────────────┐
│ Node 2: Mini PC (AMD Radeon 680M APU)        │
│ Transformer Blocks N2+1..N3                  │
└──────────────────────┬───────────────────────┘
                       │ Low-Latency Socket Frame
                       ▼
┌──────────────────────────────────────────────┐
│ Node 3: Linux Server (Ryzen Zen 5 AVX-512)   │
│ Final Blocks N3+1..End + LM Head / Logits    │
└──────────────────────┬───────────────────────┘
                       │ Next Token ID
                       ▼
                  [Next Token]

The most common assumption with distributed inference is that standard 1000BASE-T Gigabit Ethernet will choke the token generation speed.

In practice, the math proves otherwise.

During the sequential decode phase of an LLM, the cluster does not exchange model weights over the wire—each node keeps its assigned layers resident in local VRAM or system RAM. The only data crossing the network per token is the activation tensor (the hidden state vector).

For a model with hidden dimension $d = 3584$:

Over standard Gigabit Ethernet:

When running heavy models, computing the multi-head attention and MoE feed-forward networks across active parameters takes hundreds of milliseconds to seconds. A ~8 ms wire delay accounts for less than 0.2% of total token execution time, leaving 99.8% of the run spent on raw compute.

Most distributed AI stacks drag along dozens of heavy dependencies: CUDA toolkit installations, Python runtime environments, Torch binaries, and native C++ orchestration wrappers.

Glacier.Inference.Distributed is built from the metal up in pure C# targeting .NET 10 (and is Native AOT-ready):

System.IO.Pipelines utilizing 64-byte aligned socket framing (ClusterFrameHeader + FusedActivationHeader). Because the coordinator auto-inspects standard GGUF metadata, it automatically calculates layer boundaries and hidden dimensions based on the nodes you provide:

Install the .NET 10 SDK on your machines and clone the repo:

git clone https://github.com/ian-cowley/Glacier.Inference.Distributed.git
cd Glacier.Inference.Distributed

Rather than using huggingface-cli or external Python tools, Glacier includes a pure C# multi-stream HTTP Range down that opens concurrent sockets to saturate your bandwidth:

dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- download qwen2-57b

dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- download mixtral-8x22b

On your secondary machines, start the worker daemon. Each worker auto-detects whether to initialize D3D12 GPU shaders or CPU SIMD:

dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5055

dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5056

dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- --role worker --port 5057

From your primary machine (e.g., host laptop), point the coordinator to the worker endpoints:

dotnet run --project src/Glacier.Inference.Distributed.Cli -c Release -- \
  --role coordinator \
  --stages 192.168.1.108:5055,192.168.1.101:5056,192.168.1.200:5057 \
  --model ./models/qwen2-57b-a14b-instruct-q3_k_m.gguf \
  --tokens 50 \
  --prompt "Mixture-of-Experts enables scaling parameter capacity across distributed home clusters by"

The coordinator connects over TCP, verifies the hardware state, assigns layer boundaries, and streams tokens directly to the console.

If you prefer building applications directly in .NET rather than invoking a CLI tool, the coordinator can be embedded using a simple async streaming API:

using Glacier.Inference.Distributed;

var endpoints = new[]
{
    new StageEndpoint("192.168.1.108", 5055, "RTX 3060 Node"),
    new StageEndpoint("192.168.1.101", 5056, "APU Node"),
    new StageEndpoint("192.168.1.200", 5057, "AVX-512 Server")
};

using var coordinator = new DistributedPipelineCoordinator(
    stages: endpoints,
    modelPath: "models/Mixtral-8x22B-Instruct-v0.1.i1-IQ2_XXS.gguf");

await foreach (var token in coordinator.GenerateStreamAsync("Explain distributed tensor pipelines in simple terms."))
{
    Console.Write(token);
}

Running empirical tests across 4 physical machines over a basic 1 Gbps switch with DeepSeek-R1-Distill-Qwen-14B yielded these verified metrics:

Large AI models don't have to be restricted to cloud APIs or ultra-high-end enterprise workstations. By pairing lean, zero-allocation network framing with modern SIMD and graphics APIs, you can pool whatever spare hardware you already have sitting on your local network to run models that would otherwise never fit on a single desk machine.

Check out the full source, benchmark harness, and documentation on GitHub:

👉 github.com/ian-cowley/Glacier.Inference.Distributed