# Mesh LLM Lets You Run 235B AI Models Without the Cloud > Source: > Published: 2026-07-12 17:12:16+00:00 A new open source tool called Mesh LLM launched yesterday and immediately hit the Hacker News front page with 72 comments. The pitch is simple: pool the GPUs sitting in your machines, expose them as one **OpenAI-compatible API**, and stop paying cloud inference bills for workloads you could run on hardware you already own. It works on Linux, macOS, and Windows. The install is a single curl command. ## Three Commands to a Distributed GPU Cluster Getting started with Mesh LLM requires almost no setup. Install the 18 MB binary, run setup, and start serving: ``` curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash mesh-llm setup mesh-llm serve --auto ``` That exposes an OpenAI-compatible API at `http://localhost:9337/v1` . Your existing OpenAI SDK code works without modification — change the base URL and you’re done. The `--auto` flag handles routing automatically: the mesh tracks which nodes have which models loaded and their current latency, then routes your request to the best available machine. You can check what’s available with a standard models endpoint call: ``` curl -s http://localhost:9337/v1/models | jq '.data[].id' ``` The [Mesh LLM catalog](https://github.com/Mesh-LLM/mesh-llm) covers 72 model families — Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, and more — from 500M-parameter models that fit on a laptop to 235B mixture-of-experts variants. Multimodal inference is certified for Qwen2-VL and Qwen3-VL. One developer testing it on a MacBook Pro put it plainly: “I can’t overstate how easy it was — it worked first try.” ## How the Mesh Works: iroh and the Skippy Pipeline The networking layer under Mesh LLM is [iroh](https://www.iroh.computer/blog/mesh-llm), a QUIC-based P2P library from n0.computer that handles NAT traversal and hole-punching automatically. Rather than dialing IP addresses and configuring port forwarding, every node has an Ed25519 public key that stays constant regardless of which network it’s on. Two nodes that have never connected before find each other via a relay, attempt a direct QUIC connection, and succeed about nine times out of ten. No VPN, no manual firewall rules. For models that exceed the VRAM of any single machine, Mesh LLM’s “Skippy” pipeline splits the model by layer ranges across nodes — layers 0-15 on one machine, 16-31 on the next — and passes activations sequentially through the chain. The protocol uses three dedicated QUIC sub-protocols to keep gossip traffic, control messages, and activation transport from interfering with each other. It’s genuinely clever engineering, and it’s why setup is a single command instead of a week of networking configuration. Related:[Ollama Raises $65M Series B: What Changes for Developers] ## Performance: Great on LAN, Slow on WAN The core team reports 16 tokens per second running Qwen 235B across two LAN-connected nodes — close enough to interactive use for most tasks. A developer in the [Hacker News thread](https://news.ycombinator.com/item?id=48876505) hit roughly 10 tok/s with GLM 5.2 on a home lab setup with 5ms simulated latency. Those numbers are respectable for a 235B model on consumer hardware you already own. WAN is a different story. The Skippy pipeline passes activations between nodes for every layer, so inter-node latency compounds. Developers testing across internet nodes with 100ms or more of latency described the result as potentially “unbearably slow” for chat applications. The July 2026 update improved QUIC congestion control for high-latency Wi-Fi links, which helped with mid-inference stalls, but WAN distributed inference is still better treated as an experiment than a production tool. If your nodes are on the same local network, Mesh LLM is genuinely useful. If they’re spread across the internet, manage your expectations. ## What to Watch Out For The public mesh is not private. When you join the public mesh, your requests route through nodes you don’t control, and those nodes process your activations. The team acknowledges that malicious activation poisoning and data visibility on public nodes are open problems without current solutions. For sensitive workloads, configure a private mesh — the tool explicitly supports it, and it’s the right call for anything involving proprietary code, user data, or confidential information. Don’t use `mesh-llm serve --auto` and assume your data stays on your machines. A few other limitations worth knowing: AMD GPU support is functional but less tested than Nvidia and Apple Silicon paths. The project is still at v0.72 — not production-stable, and the API surface could change. There’s no speculative decoding yet, which is a performance gap developers are actively requesting. The project has 1,500 GitHub stars as of launch day — it’s gaining momentum, but the ecosystem is young. The natural comparison is [Exo and other distributed inference tools](https://sharedllm.org/blog/sharedllm-vs-petals-vs-exo.html), which do similar layer sharding with strong Apple Silicon support. If your cluster is all Apple Silicon, Exo might still win on raw performance. Mesh LLM’s advantage is cross-platform support and the OpenAI-compatible API out of the box — no custom client needed. For solo developers on a single machine, [GLM 5.2 and similar open-weight models](https://byteiota.com/glm-52-open-weight-coding-model/) via Ollama remain simpler. Mesh LLM’s sweet spot is two or more machines on a LAN where you want a unified API endpoint with zero cloud dependency. ## Key Takeaways - Mesh LLM pools GPUs across machines via iroh P2P and exposes one OpenAI-compatible API — existing SDK code works unchanged with a base URL swap - LAN performance is genuinely competitive: 16 tok/s on Qwen 235B across two nodes is close to interactive speed; WAN is too slow for most chat use cases - Never send confidential data through the public mesh — nodes you don’t control process your activations; use private mesh for anything sensitive - The tool is at v0.72 and best suited for home labs or small team clusters; single-machine users are better served by Ollama for now