Peer-to-peer LLM inference in browser tabs, Qwen 3.8 27B

Every device brings a slice. Together they run the whole model.

Site · Start a swarm · Architecture · Benchmarks · Roadmap · Threat model · Contributing

swarmllm-demo-2026-09-07.mp4 #

_{Demo, recorded September 7, 2026: Qwen 3.8 27B across a MacBook and an iPhone in browser tabs, same Wi‑Fi, 400 tokens at 10.7 tok/s. Download.}

SwarmLLM runs large language models across the devices in a room, in their browser tabs. Each device holds a slice of the model; a 10 KB activation vector passes between them over direct WebRTC connections. Nothing to install, no accounts, no server does any thinking.

• 27B in browser tabs. Qwen 3.8 27B (15 GB of Q4_0 weights) across laptops, phones and PCs that individually can't hold it.
• Native-competitive decode. A from-scratch WebGPU engine (~50 WGSL kernels) at the memory roofline: 9.0 tok/s plain and 16 tok/s with speculative decoding on a GB10, where native llama.cpp measures 8.0 on the same file and GPU. (bench log )
• Bit-exact by construction. Every optimization is gated on golden tests; the speculative path produces the same stream as plain decoding.
• Cross-network. Rooms span networks via WebRTC; prefill sends 16 tokens per round trip and decode chains speculative drafts so a slow link still moves several tokens per lap.
• Nothing leaves the room. A room is a shared conversation: everyone in it sees the questions and answers, by design. No server ever sees them, and the devices running layers work on mid-model activations, which arenot private against a determined peer either (seeSECURITY.md ).

Use it: open swarmllm.ai/room, create a room, share the code, pick a model, start. Every device downloads only its layers (cached for next time).

Run it locally:

git clone https://github.com/Nehanth/swarmllm && cd swarmllm
npx -y serve -l 8080 .        # any static server works; then open http://localhost:8080/room

Hack on the engine (needs Deno 2.x and a WebGPU-capable GPU; model files go under models/, see docs/models.md):

npm test              # unit tests, no GPU
npm run test:gpu      # golden tests on Qwen3 0.6B
npm run test:q38      # 27B suites incl. speculative-vs-plain equality
npm run bench:q38     # decode / prefill tok/s
host      embed the last token → hidden state (5,120 floats)
   ↓ 10 KB over WebRTC
peer A    layers 0–21           ─┐
peer B    layers 22–42           ├─ each device runs its slice on its own GPU
peer C    layers 43–63          ─┘
   ↓ back to the host
host      final norm → LM head → sample → next token (and the draft head guesses the one after)

Generating a token is memory-bound: every token reads all of the weights once. So the engine's job is reading fewer bytes (4-bit blocks with f16 scales) and reading them well (64 threads sweep each row together, dequantize in registers, reduce in shared memory), with one command submit per token. The runtime's job is making network laps carry more: batched prefill, and multi-token-prediction speculation verified in a single batched pass with exact rollback of the recurrent state. Details: docs/architecture.md, docs/kernels.md, docs/protocol.md.

Other projects split or share models across machines. The differences are what has to be installed and where the model runs.

The engine underneath is our own WGSL, not WebLLM, MLC or llama.cpp; the model weights and tokenizer come from Qwen, hosting from Hugging Face, and signaling only from PeerJS.

Browsers: Chrome on macOS is the tested host. Safari on an iPhone joins a room and holds a small slice; Safari on a Mac reloads the tab under memory pressure when it holds the 27B's large slice, so do not host from it. Firefox and Linux Chromium need WebGPU enabled and are untested by us. Headless: Deno 2 (wgpu). See docs/models.md.

Qwen 3.8 27B Q4_0, greedy, bit-identical output at every row. Full history with commits in docs/bench-log.md.

Decode on the GB10 runs at the memory bandwidth a WebGPU buffer read can reach on that machine (183 of 184 GB/s measured), which is why it is ahead of the native build there. Prefill is the known gap: the DeltaNet recurrence is serial and the prefill GEMM is young. One token's hidden state on the wire is 10 KB (5,120 × f16).

engine/            the runtime (ES modules; engine.js re-exports the public API)
  dense.js         DenseEngine: dense Llama-architecture models (Qwen3, SmolLM)
  qwen35.js        Qwen35Engine: hybrid Gated-DeltaNet + attention, batched paths, MTP speculation
  gguf.js          GGUF parsing, quantization repacking, streaming upload
  wgsl/            base.js (shared kernels) · coop.js (generated GEMV family) · qwen35.js (DeltaNet kernels)
  tokenizer.js · sampling.js · quant.js · autotune.js · selftest.js · safetensors.js
room.js + room/    the room: signaling, mesh, weight streaming, generation loop; wire/markdown/sampling/models helpers
index.html         landing page          p2p.html   the room's markup (served at /room)
tests/             GPU golden tests · unit/ (no GPU) · golden/ · reference/ · run.sh
benchmarks/        tok/s harnesses, kernel-family profiler, GEMM prototype
docs/              architecture · tech-stack · kernels (every trick, measured) · protocol · models · bench log · research · agents (rules for automated contributors)
roadmap/           one file per planned item with status, design and done-criteria

See CONTRIBUTING.md and GOVERNANCE.md. Benchmark reports from hardware we don't have are especially welcome (there's an issue template).

@software{swarmllm2026,
  author = {Narendrula, Nehanth},
  title  = {SwarmLLM: peer-to-peer LLM inference across browser tabs},
  year   = {2026},
  url    = {https://github.com/Nehanth/swarmllm}
}

Model weights and the GGUF format come from the Qwen team and llama.cpp / ggml, whose speculative-decoding graph for Qwen 3.5/3.8 was the reference for ours. Prior work that shaped this: Petals, exo, WebLLM, LlamaWeb, and the Gated DeltaNet and PipeInfer papers.

MIT.