{"slug": "llmlet-p2p-distributed-llm-inference-on-browsers", "title": "LLMlet: P2P distributed LLM inference on browsers", "summary": "LLMlet, an experimental open-source tool, enables distributed large language model inference across web browsers using WebRTC and Wasm-compiled llama.cpp. Users can connect multiple browser tabs as peers to collaboratively run models like SmolLM2-1.7B, with one client distributing chunks to server peers for processing.", "body_md": "LLMlet is an on-browser LLM runner based on [Wasm-compiled llama.cpp](https://github.com/ggml-org/llama.cpp/blob/10e9780154365b191fb43ca4830659ef12def80f/docs/build.md#webgpu-in-progress).\n\n- Distributed LLM inference among browsers connected via WebRTC using\n[PeerJS](https://peerjs.com/). - Calling functions written in JavaScript\n\nDemo page: [https://ktock.github.io/llmlet-demo/](https://ktock.github.io/llmlet-demo/)\n\nThis is an experimental software.\nSee [Known limitations](#known-limitations).\n\nEvery browser tab acts as a peer. On a peer, you can load a model, distribute the chunks to other peers and run inference on them.\n\nThis section describes an example of running 3 peers, using one of them as a client and two as servers.\n\n### 1. Open the [demo page](https://ktock.github.io/llmlet-demo/) in 3 separated browsers or tabs\n\nOn each tab, a peer starts automatically using [PeerJS's public server](https://peerjs.com/peerserver) and displays its assigned ID.\n\nIn this example, the peers received the following IDs:\n\n`3bd154c7-0f32-4675-888c-6d0f7fa1496f`\n\n`d40d0182-c874-4cb0-8bff-7a57daa3eb92`\n\n`cb296f1f-00cc-456c-94c4-30f9aeb7f408`\n\nOn every peer, write the full list of IDs to the `Peers to connect to / accept (comma or newline separated)`\n\nfield.\nThis makes each peer ready to handle requests from other peers.\n\nYou can use any peer as the client.\nOn the client peer, specify a model to use via an HTTP URL (e.g. GGUF file on Hugging Face) or by selecting a local file.\nThis example uses `https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF/resolve/main/smollm2-1.7b-instruct-q4_k_m.gguf`\n\n.\n\nFinally, enter a prompt in the `Ask something`\n\ntextarea on the client peer to start inference.\n\nNOTE: It might take a while to distribute the model and run inference on each peer.\n\n`system log`\n\nwindow shows logs from llama.cpp and is convenient to see the progress. On each server peer, message such as`Handling requests`\n\nwill appear during processing the client requests.\n\nYou can build LLMlet using make or Docker then serve them to browsers using HTTP servers.\n\n```\nmake EMDAWNWEBGPU_DIR=/path/to/emdawnwebgpu_pkg\n```\n\nThis step creates the following files in the `build`\n\ndirectory.\n\n`llmlet-mod.js`\n\n`llmlet-mod.wasm`\n\n- Prerequisites\n`docker build`\n\n```\ndocker build --output type=local,dest=build .\n```\n\nThis step creates the following files in the `build`\n\ndirectory.\n\n`llmlet-mod.js`\n\n`llmlet-mod.wasm`\n\nThis section describes an example to run LLMlet using a localhost HTTP server.\n\nStore the assets compiled in the previous steps to `/tmp/test/htdocs/`\n\n.\n\nThen copy the following files from this repo to the same document root.\n\n`llmlet.js`\n\n: A JS file to configure and start a peer in the browser.`examples/simple/`\n\n: Contains an example index.html file for local testing.\n\n```\ncp llmlet.js examples/simple/* /tmp/test/htdocs/\n```\n\n`examples/simple/index.html`\n\nis intended for local testing and requires a localhost PeerJS signaling server.\n\n```\ndocker run --rm -d --name peerjs -p 127.0.0.1:9000:9000 peerjs/peerjs-server\n```\n\nYou can serve the assets using an HTTP server.\n\n```\ndocker run --rm -d -p 127.0.0.1:8888:80 --name peer \\\n           -v /tmp/test/htdocs/:/usr/local/apache2/htdocs/:ro \\\n           httpd httpd-foreground \\\n           -c \"Header set Cross-Origin-Opener-Policy: same-origin\" \\\n           -c \"Header set Cross-Origin-Embedder-Policy: require-corp\"\n```\n\nNOTE: This project relies on Emscripten's pthread support which requires additional headers as shown in the above command example. For more details, see\n\n[https://emscripten.org/docs/porting/pthreads.html]\n\nThen the page is accesible via `localhost:8888`\n\n.\n\nLLMlet uses [PeerJS](https://peerjs.com/) to connect browsers with WebRTC and uses [llama.cpp's RPC feature](https://github.com/ggml-org/llama.cpp/blob/99c53d6558e1882fdddd1077495b618f3210cc02/tools/rpc/README.md) to run distributed LLM inferences.\nAlthough llama.cpp already supports [Wasm compilation and WebGPU](https://github.com/ggml-org/llama.cpp/blob/10e9780154365b191fb43ca4830659ef12def80f/docs/build.md#webgpu-in-progress), patches were needed to function over WebRTC.\n\nPeers use the [PeerJS's server](https://github.com/peers/peerjs-server) for setting up WebRTC connections among them.\nDistributing model chunks and facilitating inference are done by the peer itself so additional servers aren't needed for those purposes.\nThe example is hosted on GitHub Pages and uses [PeerJS's public server](https://peerjs.com/peerserver).\n\n- Parallelism is not yet supported. Peers sequentially evaluates the model chunks and communication each other adds overhead to the inference speed. (related:\n[ggml-org/llama.cpp#8500](https://github.com/ggml-org/llama.cpp/discussions/8500),[ggml-org/llama.cpp#16020](https://github.com/ggml-org/llama.cpp/pull/16020)) - Models are split in layer granularity so each peer must have enough resources to host at least one layer. (related:\n[ggml-org/llama.cpp#16020](https://github.com/ggml-org/llama.cpp/pull/16020)) - A server can't serve multiple clients simultaneously. When starting a second client in the network, the existing client must be stopped first.\n- Demo page relies on PeerJS's public server which\n[doesn't offer](https://github.com/orgs/peers/discussions/1172)the[TURN](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Protocols#turn)service for relaying packets among peers. So peers can't connect each other if they are in restricted network environments which don't allow P2P communication.\n\nLLMlet uses llama.cpp's [Wasm and WebGPU support](https://github.com/ggml-org/llama.cpp/blob/10e9780154365b191fb43ca4830659ef12def80f/docs/build.md#webgpu-in-progress) with the following configuration:\n\n- The CPU backend is enabled when WebGPU is not supported in the browser. Available heap size is up to 2GB.\n- The WebGPU backend is used when the browser supports WebGPU (see\n[https://caniuse.com/webgpu](https://caniuse.com/webgpu)). Note that WebGPU isn't enabled by default on some platforms (e.g. Chrome on Linux requires flags such as`enable-unsafe-webgpu`\n\n). - The RPC backend is patched to recognize browser peers connected over WebRTC via PeerJS.\n\nBrowsers offer portable execution environments such as Wasm and WebGPU, making it possible to run LLMs inside browser. However, a single browser can't run models that exceed its memory capacity. This project addresses this limitation by directly linking multiple browsers and using them as a single larger computing environment.\n\n**WebGPU backend is not enabled**- Ensure that the browser supports WebGPU and that any required browser flags are enabled (c.f.\n[https://caniuse.com/webgpu](https://caniuse.com/webgpu)).\n\n- Ensure that the browser supports WebGPU and that any required browser flags are enabled (c.f.\n**Inference fails with an allocation failure**- The maximum heap size of the Wasm module is configured as 2GB. To increase total available memory, open another browser tab and join it to the network as an additional peer.\n\n**Inference failes with**`Remote RPC server crashed or returned malformed response`\n\n- If it is a temporary connection failure, you can restart inference using\n`/restart`\n\ncommand on the client peer. If it fails frequently, connecting to peers from a different network (e.g. different WiFi network) might improve the connection.\n\n- If it is a temporary connection failure, you can restart inference using\n**LLM prints corrupted string**- Try reloading and restarting the page. Stale data might be cached in the IndexedDB named\n`ChunkCache`\n\nso remove them ([a guide to delete IndexedDB data on chrome](https://developer.chrome.com/docs/devtools/storage/indexeddb?hl=en#deletedatabase)).\n\n- Try reloading and restarting the page. Stale data might be cached in the IndexedDB named\n\n: Examples of function calling, etc.`./examples/`\n\n- Frameworks such as\n[llama.cpp](https://github.com/ggml-org/llama.cpp)and[vLLM](https://github.com/vllm-project/vllm)support distributed inference over multiple machines but they don't work across browsers. [web-llm](https://github.com/mlc-ai/web-llm)and[wllama](https://github.com/ngxson/wllama)enable LLM inference inside browser but they don't support distributed inference.[exo](https://github.com/exo-explore/exo)enables to connect multiple devices to create an AI cluster. However it doesn't support browsers.[DistML.js](https://github.com/mil-tokyo/distmljs)([paper](https://arxiv.org/abs/2407.01023)) supports distributed training on browsers with data parallelism. However it isn't designed as peer-to-peer communication among browsers and uses a server to distribute weights and facilitate inference.\n\nThis project relies on third party softwares including the following.", "url": "https://wpnews.pro/news/llmlet-p2p-distributed-llm-inference-on-browsers", "canonical_source": "https://github.com/ktock/llmlet", "published_at": "2026-07-12 06:30:45+00:00", "updated_at": "2026-07-12 07:05:30.120989+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["LLMlet", "llama.cpp", "PeerJS", "WebRTC", "HuggingFaceTB", "SmolLM2-1.7B-Instruct-GGUF", "Emscripten"], "alternates": {"html": "https://wpnews.pro/news/llmlet-p2p-distributed-llm-inference-on-browsers", "markdown": "https://wpnews.pro/news/llmlet-p2p-distributed-llm-inference-on-browsers.md", "text": "https://wpnews.pro/news/llmlet-p2p-distributed-llm-inference-on-browsers.txt", "jsonld": "https://wpnews.pro/news/llmlet-p2p-distributed-llm-inference-on-browsers.jsonld"}}