How to Install Mesh LLM and Pool GPUs Into One Local API

Mesh LLM, a tool that pools GPU memory across multiple machines and exposes the combined capacity as a single OpenAI-compatible API, installs via a one-line shell script on Linux, with Windows support through WSL2 and a Mac path also available, though the native install can throw missing-library and dependency errors. The Docker image is the more reliable install path because it isolates the environment, and a built-in web dashboard on port 3131 shows pooled VRAM, active models and node status; nodes auto-discover peers and models, loading a model onto the GPU only when it is used, and a special "mesh" model name sends a prompt to every loaded model at once and returns a consolidated response.

How to Install Mesh LLM and Pool GPUs Into One Local API Mesh LLM pools GPU memory across machines into one OpenAI-compatible API. Here's how installation, Docker setup, and model routing work. What is Mesh LLM? Mesh LLM is a tool that pools GPU memory across multiple machines and exposes the combined capacity as a single OpenAI-compatible API. Instead of running one model on one machine, you start a node, optionally add more machines to the mesh, and the system decides whether a given model runs locally, gets routed to a peer with spare capacity, or gets split across machines when it’s too large to fit on a single GPU. The result behaves like one API endpoint, even though the actual compute may be scattered across a home lab, a team’s workstations, or rented GPU instances. TL;DR - Mesh LLM pools VRAM across machines and presents it as one OpenAI-compatible API, so agents and apps talk to a single endpoint regardless of how many physical GPUs sit behind it. - Installation is a one-line shell script on Linux, with Windows support through WSL2 and Mac support also available, though the native install can throw dependency and library errors. - Docker is the more reliable install path because it isolates the environment and avoids the prerequisite issues seen in the native install. - Nodes auto-discover peers and models , downloading and loading models on demand rather than preloading everything into VRAM at startup. - A built-in web dashboard on port 3131 shows pooled VRAM, active models, and node status, and lets you reconfigure runtime settings without touching files directly. - A special “mesh” model name triggers parallel querying , sending a prompt to every loaded model at once and returning a consolidated response instead of a single model’s output. - Loading a model doesn’t mean it occupies GPU memory immediately , it only gets pulled onto the GPU when it’s actually used, which keeps VRAM consumption lower than the model catalog suggests. Everyone else built a construction worker. We built the contractor. One file at a time. UI, API, database, deploy. How do you install Mesh LLM? The fastest path is a single shell command that works on Ubuntu and other Linux distributions. On Windows, Mesh LLM runs through WSL2 Windows Subsystem for Linux , and a Mac installation path is also supported. After the one-liner finishes, you set the tool’s path in your shell and source it so the change takes effect immediately. Running a version check confirms the install worked. That native install path isn’t fully polished. Depending on the platform, you can hit missing library errors and other dependency complaints right after installation. These errors don’t necessarily block the tool from running, but they’re a sign the native install isn’t bulletproof across every environment. Why use Docker instead of a native install? Docker sidesteps the dependency problems entirely. Pulling the Mesh LLM Docker image gives you a self-contained environment where the prerequisites are already handled, so you don’t have to chase down missing libraries on your specific OS. This matters on Windows and other non-Linux systems too, where native dependency management is even messier. Once the image is pulled, you can verify that the container has GPU access and confirm the Mesh LLM version from inside the container. From there, the setup is the same regardless of host OS: create folders on the host machine for config and model storage so they persist across container restarts, then start the node with GPU access enabled, host networking configured, those folders mounted, and a starting model specified a Gemma model was used in testing . The node then auto-discovers available hardware, downloads the specified model, and loads it, a process that takes a few minutes depending on model size and connection speed. Once the status shows “ready,” the node is live. How does GPU pooling and routing actually work? Once a node is running, it exposes an OpenAI-compatible chat API, so you can send requests the same way you would to any hosted LLM provider, just pointed at your local endpoint instead. If you have multiple machines, Mesh LLM automatically discovers the other hosts and factors their GPU memory into the pool. With a single machine, as in a basic setup, the mesh just uses the local GPUs it finds. A key design choice is that loading a model into the catalog doesn’t immediately consume VRAM. A model only gets pulled onto the GPU when an actual request needs it. This means you can have several models registered and visible in the dashboard, but idle models aren’t eating into your memory budget. For anyone running on a single GPU with limited headroom, that distinction matters a lot: you can stage multiple models without constantly hitting out-of-memory errors. What does the web dashboard show? regulatory compliance and security. Mesh LLM includes a web interface, by default on port 3131, that visualizes the mesh in real time. A live mesh can show details like node count, total pooled VRAM across all machines, and the number of active models, alongside a model catalog you can browse and manage. From the dashboard you can trigger a chat session directly, switch between models, and adjust runtime configuration, including which models are loaded and how many “slots” and what context window size each gets. Configuration changes can also be made from the terminal by editing the node’s config file directly and then running a validation command to confirm the config is well-formed before restarting the node. This terminal-first workflow was described as the more reliable method for making changes, compared to doing everything through the UI. By default, a freshly started node is reachable as a “public mesh” on your local network interface, but it can be reconfigured into a private mesh if you don’t want it exposed beyond your own machine. What is the “mesh” model, and why does it matter? The most distinct feature of Mesh LLM isn’t pooling GPUs for a single model, it’s the virtual “mesh” model name. When you send a chat request with the model field set to “mesh” instead of naming a specific model, the system queries every model currently loaded in the pool in parallel, compares their answers, and returns a single consolidated response rather than one model’s raw output. This requires at least two models loaded at once. In testing, two different model families were loaded simultaneously a Gemma model and a Qwen model , and asking a question through the “mesh” endpoint caused both to be consulted before a combined answer came back. This is effectively a lightweight, built-in form of model ensembling, useful if you want a sanity check across models without manually querying each one and comparing outputs yourself. Is Mesh LLM worth setting up? For anyone experimenting with self-hosted inference across more than one machine, Mesh LLM addresses a real gap: most local LLM tools assume a single box with a single GPU, and scaling beyond that usually means hand-rolling your own load balancing or model-sharding logic. Mesh LLM automates node discovery, memory pooling, and request routing, and wraps it all in an API that’s compatible with existing OpenAI-client tooling, meaning agent frameworks and chat clients built for OpenAI’s API can point at a Mesh LLM endpoint with minimal changes. The rough edges are real, though. The native installation isn’t consistently stable across platforms, with dependency and library errors showing up depending on the OS. Docker avoids most of that friction and is the more dependable way to run it, whether on Linux, Windows, or Mac. If you’re comfortable with Docker and have more than one machine with a GPU sitting around, or just one machine you want to expose as an API for a team, it’s a reasonable tool to test. If you need rock-solid production stability out of the box, treat it as early-stage software for now. Frequently Asked Questions What operating systems does Mesh LLM support? It runs natively on Linux demonstrated on Ubuntu , and on Windows via WSL2. Mac is also supported. Docker is available as an installation method across all of these, and is the more stable option given dependency issues seen in the native install. Do I need multiple machines to use Mesh LLM? Rapidly build enterprise-grade, production-ready software. Already in: ✓ SOC 2✓ HIPAA✓ Audit log No. You can run it on a single machine with one or more GPUs. Multi-machine pooling is a feature you can add later by starting additional nodes that join the same mesh, but a single-node setup already gives you the OpenAI-compatible API and model management dashboard. Does loading more models use more GPU memory immediately? No. Models are downloaded and registered in the catalog, but they only get loaded onto the GPU when a request actually needs them. This keeps idle VRAM usage lower than the total size of all catalog models combined. What does querying the “mesh” model do differently? Setting the model parameter to “mesh” instead of a specific model name sends the prompt to every currently loaded model in parallel, compares the responses, and returns one consolidated answer. It requires at least two models to be loaded simultaneously. Can I control which models load and their settings? Yes. You can manage models and runtime settings through the web dashboard on port 3131, or edit the node’s configuration file directly from the terminal and validate it before restarting the node, which was the preferred workflow in testing.