{"slug": "peer-to-peer-llm-inference-in-browser-tabs-qwen-3-8-27b", "title": "Peer-to-peer LLM inference in browser tabs, Qwen 3.8 27B", "summary": "SwarmLLM, an open-source project by developer Nehanth, enables a Qwen 3.8 27B model to run across browser tabs on multiple devices, with each device holding a slice of the model and passing 10 KB activation vectors over WebRTC. The system achieves 9.0 tok/s plain and 16 tok/s with speculative decoding on a GB10, outperforming native llama.cpp's 8.0 tok/s on the same hardware, and is bit-exact by construction. The project is available at swarmllm.ai and on GitHub, requiring no installation or accounts.", "body_md": "**Every device brings a slice. Together they run the whole model.**\n\n[Site](https://swarmllm.ai) ·\n  [Start a swarm](https://swarmllm.ai/room) ·\n  [Architecture](/Nehanth/swarmllm/blob/main/docs/architecture.md) ·\n  [Benchmarks](/Nehanth/swarmllm/blob/main/docs/bench-log.md) ·\n  [Roadmap](/Nehanth/swarmllm/blob/main/roadmap) ·\n  [Threat model](/Nehanth/swarmllm/blob/main/SECURITY.md) ·\n  [Contributing](/Nehanth/swarmllm/blob/main/CONTRIBUTING.md)\n\n## swarmllm-demo-2026-09-07.mp4\n\n<sub>Demo, recorded September 7, 2026: Qwen 3.8 27B across a MacBook and an iPhone in browser tabs, same Wi‑Fi, 400 tokens at 10.7 tok/s. [Download](https://github.com/Nehanth/swarmllm/releases/download/v0.2.0/swarmllm-demo-2026-09-07.mp4).</sub>\n\nSwarmLLM runs large language models across the devices in a room, in their browser tabs. Each device holds a slice of the model; a 10 KB activation vector passes between them over direct WebRTC connections. Nothing to install, no accounts, no server does any thinking.\n\n- **27B in browser tabs.** Qwen 3.8 27B (15 GB of Q4_0 weights) across laptops, phones and PCs that individually can't hold it.\n- **Native-competitive decode.** A from-scratch WebGPU engine (~50 WGSL kernels) at the memory roofline: 9.0 tok/s plain and 16 tok/s with speculative decoding on a GB10, where native llama.cpp measures 8.0 on the same file and GPU. ([bench log](/Nehanth/swarmllm/blob/main/docs/bench-log.md) )\n- **Bit-exact by construction.** Every optimization is gated on golden tests; the speculative path produces the same stream as plain decoding.\n- **Cross-network.** Rooms span networks via WebRTC; prefill sends 16 tokens per round trip and decode chains speculative drafts so a slow link still moves several tokens per lap.\n- **Nothing leaves the room.** A room is a shared conversation: everyone in it sees the questions and answers, by design. No server ever sees them, and the devices running layers work on mid-model activations, which are*not* private against a determined peer either (see[SECURITY.md](/Nehanth/swarmllm/blob/main/SECURITY.md) ).\n\n**Use it:** open [swarmllm.ai/room](https://swarmllm.ai/room), create a room, share the code, pick a model, start. Every device downloads only its layers (cached for next time).\n\n**Run it locally:**\n\n```\ngit clone https://github.com/Nehanth/swarmllm && cd swarmllm\nnpx -y serve -l 8080 .        # any static server works; then open http://localhost:8080/room\n```\n\n**Hack on the engine** (needs [Deno](https://deno.com) 2.x and a WebGPU-capable GPU; model files go under `models/`, see [docs/models.md](/Nehanth/swarmllm/blob/main/docs/models.md)):\n\n```\nnpm test              # unit tests, no GPU\nnpm run test:gpu      # golden tests on Qwen3 0.6B\nnpm run test:q38      # 27B suites incl. speculative-vs-plain equality\nnpm run bench:q38     # decode / prefill tok/s\nhost      embed the last token → hidden state (5,120 floats)\n   ↓ 10 KB over WebRTC\npeer A    layers 0–21           ─┐\npeer B    layers 22–42           ├─ each device runs its slice on its own GPU\npeer C    layers 43–63          ─┘\n   ↓ back to the host\nhost      final norm → LM head → sample → next token (and the draft head guesses the one after)\n```\n\nGenerating a token is memory-bound: every token reads all of the weights once. So the engine's job is reading fewer bytes (4-bit blocks with f16 scales) and reading them well (64 threads sweep each row together, dequantize in registers, reduce in shared memory), with one command submit per token. The runtime's job is making network laps carry more: batched prefill, and multi-token-prediction speculation verified in a single batched pass with exact rollback of the recurrent state. Details: [docs/architecture.md](/Nehanth/swarmllm/blob/main/docs/architecture.md), [docs/kernels.md](/Nehanth/swarmllm/blob/main/docs/kernels.md), [docs/protocol.md](/Nehanth/swarmllm/blob/main/docs/protocol.md).\n\nOther projects split or share models across machines. The differences are what has to be installed and where the model runs.\n\n|  | Model per device | Devices | Install | Network | \n|---|---|---|---|---|\n| **SwarmLLM** | a slice of layers | laptops and phones, any OS with a WebGPU browser | none, open a URL | same Wi‑Fi or across the internet (WebRTC) | \n| exo | a slice of layers | machines that run Python and MLX or tinygrad | Python package per node | one network | \n| llama.cpp `rpc-server` | a slice of layers | machines that run the binary | binary and an open port per node; the docs say not for untrusted networks | LAN in practice | \n| Petals | a slice of layers | server GPUs in a public swarm | Python client and server | internet, public swarm | \n| distributed-llama | a slice of layers | Linux boxes and Raspberry Pis | binary per node | LAN | \n| WebLLM / MLC, transformers.js | the whole model | one browser tab | none | none needed | \n| Ollama, llmman | the whole model | one machine per request | native app | routing between machines, no splitting | \n\nThe engine underneath is our own WGSL, not WebLLM, MLC or llama.cpp; the model weights and tokenizer come from Qwen, hosting from Hugging Face, and signaling only from PeerJS.\n\n| Model | Format | Notes | \n|---|---|---|\n| Qwen 3.8 27B | GGUF Q4_0 | hybrid Gated-DeltaNet + attention; MTP speculation | \n| Qwen3 0.6B / 1.7B / 4B | GGUF Q8_0 / Q4_0 | dense; used for golden tests | \n| SmolLM2 135M | safetensors f32 | smallest demo | \n\nBrowsers: Chrome on macOS is the tested host. Safari on an iPhone joins a room and holds a small slice; Safari on a Mac reloads the tab under memory pressure when it holds the 27B's large slice, so do not host from it. Firefox and Linux Chromium need WebGPU enabled and are untested by us. Headless: Deno 2 (wgpu). See [docs/models.md](/Nehanth/swarmllm/blob/main/docs/models.md).\n\nQwen 3.8 27B Q4_0, greedy, bit-identical output at every row. Full history with commits in [docs/bench-log.md](/Nehanth/swarmllm/blob/main/docs/bench-log.md).\n\n| Device | Decode plain | Decode speculative | Prefill | Native llama.cpp, same GGUF, same machine | \n|---|---|---|---|---|\n| NVIDIA GB10 (Deno / Vulkan, headless) | 9.0 tok/s | 16.1 tok/s | 44 tok/s | 8.0 decode (tg32), 377 prefill (pp86), CUDA build 749f688 | \n| MacBook Pro (Chrome / Metal), solo | 6.7 tok/s | 10.8 tok/s | ~20 tok/s | — | \n| MacBook Pro + iPhone, same Wi‑Fi, 62 + 2 layers | — | 7.7 tok/s | 8.5 s for a 169-token prompt | — | \n| Cross-internet room (host + peer) | — | 3.5–6 tok/s | — | — | \n\nDecode on the GB10 runs at the memory bandwidth a WebGPU buffer read can reach on that machine (183 of 184 GB/s measured), which is why it is ahead of the native build there. Prefill is the known gap: the DeltaNet recurrence is serial and the prefill GEMM is young. One token's hidden state on the wire is 10 KB (5,120 × f16).\n\n```\nengine/            the runtime (ES modules; engine.js re-exports the public API)\n  dense.js         DenseEngine: dense Llama-architecture models (Qwen3, SmolLM)\n  qwen35.js        Qwen35Engine: hybrid Gated-DeltaNet + attention, batched paths, MTP speculation\n  gguf.js          GGUF parsing, quantization repacking, streaming upload\n  wgsl/            base.js (shared kernels) · coop.js (generated GEMV family) · qwen35.js (DeltaNet kernels)\n  tokenizer.js · sampling.js · quant.js · autotune.js · selftest.js · safetensors.js\nroom.js + room/    the room: signaling, mesh, weight streaming, generation loop; wire/markdown/sampling/models helpers\nindex.html         landing page          p2p.html   the room's markup (served at /room)\ntests/             GPU golden tests · unit/ (no GPU) · golden/ · reference/ · run.sh\nbenchmarks/        tok/s harnesses, kernel-family profiler, GEMM prototype\ndocs/              architecture · tech-stack · kernels (every trick, measured) · protocol · models · bench log · research · agents (rules for automated contributors)\nroadmap/           one file per planned item with status, design and done-criteria\n```\n\nSee [CONTRIBUTING.md](/Nehanth/swarmllm/blob/main/CONTRIBUTING.md) and [GOVERNANCE.md](/Nehanth/swarmllm/blob/main/GOVERNANCE.md). Benchmark reports from hardware we don't have are especially welcome (there's an issue template).\n\n```\n@software{swarmllm2026,\n  author = {Narendrula, Nehanth},\n  title  = {SwarmLLM: peer-to-peer LLM inference across browser tabs},\n  year   = {2026},\n  url    = {https://github.com/Nehanth/swarmllm}\n}\n```\n\nModel weights and the GGUF format come from the [Qwen](https://huggingface.co/Qwen) team and [llama.cpp / ggml](https://github.com/ggml-org/llama.cpp), whose speculative-decoding graph for Qwen 3.5/3.8 was the reference for ours. Prior work that shaped this: [Petals](https://github.com/bigscience-workshop/petals), [exo](https://github.com/exo-explore/exo), [WebLLM](https://github.com/mlc-ai/web-llm), [LlamaWeb](https://arxiv.org/abs/2605.20706), and the Gated DeltaNet and PipeInfer papers.\n\n[MIT](/Nehanth/swarmllm/blob/main/LICENSE).", "url": "https://wpnews.pro/news/peer-to-peer-llm-inference-in-browser-tabs-qwen-3-8-27b", "canonical_source": "https://github.com/Nehanth/swarmllm", "published_at": "2026-09-07 20:08:22+00:00", "updated_at": "2026-09-07 20:32:11.100913+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "ai-tools"], "entities": ["SwarmLLM", "Nehanth", "Qwen 3.8 27B", "WebGPU", "WebRTC", "llama.cpp", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/peer-to-peer-llm-inference-in-browser-tabs-qwen-3-8-27b", "markdown": "https://wpnews.pro/news/peer-to-peer-llm-inference-in-browser-tabs-qwen-3-8-27b.md", "text": "https://wpnews.pro/news/peer-to-peer-llm-inference-in-browser-tabs-qwen-3-8-27b.txt", "jsonld": "https://wpnews.pro/news/peer-to-peer-llm-inference-in-browser-tabs-qwen-3-8-27b.jsonld"}}