Self-hosted inference orchestrators compared: LocalAI, exo, GPUStack, vLLM

This is the reference version of a survey I did for my own project, written so it is useful to someone who is not running CoderAI. If you have one or more machines with GPUs and want an OpenAI-compatible endpoint in front of them, these are the self-hosted orchestrators that exist in September 2026, what each one actually does across machines, and which one to

                    pick for which situation. Star counts are from the GitHub API on 2026-09-20; feature cells are
                    from the projects' own README and docs, linked at the end. Where I could not confirm something
                    the cell says so instead of guessing.

The table #

Modalities: what the server itself generates, not what a plugin can reach. Multi-machine:

                    whether one model, or one request, can use more than one box. *Cache-aware routing*: whether a
                    follow-up turn is sent where its KV/prefix cache already is. *Cloud burst*: whether the
                    server can obtain a rented GPU by itself. *Non-LLM fan-out*: whether one image/audio/embedding
                    request is split over machines. Petals (last commit 2024) and Hugging Face TGI (archived March
                    2026) are omitted.

What each one is for #

Ollama and Open WebUI — the default, for one machine

One binary, ollama pull, an answer. A library of models with sane defaults, a Modelfile format, and, with Open WebUI in front, a chat product a non-engineer can use. It is the right answer for a laptop or one desktop. It is one machine, one model at a time, and there is

                    no cluster story at all beyond Open WebUI round-robining several Ollama URLs; if that is what you
                    need, stop reading here.

llama.cpp and vLLM — the engines everyone else wraps

Both are orchestrators only in the narrow sense that they can span machines: llama.cpp with

                    `rpc-server` on each box and `--rpc` on the server (layer split by default, row
                    split for tensor parallel on a fast link), vLLM with Ray for tensor and pipeline parallelism.
                    Neither manages models, users, placement or anything beyond one model per process. They are
                    what LocalAI, GPUStack, Xinference and CoderAI run underneath, and if you only ever need one
                    model, running the engine directly is simplest.

LiteLLM — a router, not a runtime

It sits in front of a hundred providers and your own endpoints with keys, budgets and spend tracking, and never runs a model. It is the piece you put in front of any of the others when several teams share them, and it is often mistaken for a self-hosting solution; it is not one.

LocalAI — the closest thing to "all of it"

OpenAI-compatible API over text, image, video, audio and embeddings, each backend a gRPC service in its own OCI image, no GPU required, Helm charts, and — since June 2026 — a real

                    distributed mode: `--p2p` generates a shared token, instances discover each other
                    over libp2p/EdgeVPN, requests federate to the least loaded node or a llama.cpp model is sharded
                    across workers, and a NATS-based "v3" router is aware of VRAM and of prefix caches. Backend
                    images are cosign-signed. What it does not do: rent a GPU, split a non-LLM request over
                    machines, train. Its raw LLM throughput trails a dedicated engine by some tens of percent
                    because the generality costs. If you want breadth with a large community and do not need
                    escalation, this is the mainstream pick.

exo — a stack of Macs as one computer

Zero-config discovery, ring/pipeline/tensor partitioning proportional to each device's memory, MLX underneath, and RDMA over Thunderbolt 5 on recent macOS for tensor parallelism that genuinely scales (3.2× on four devices is their number). It is also, in September 2026, CPU-only on Linux with NVIDIA and AMD "under development", and it serves language models (image generation behind a feature flag). If your hardware is Apple Silicon, nothing else comes close; if it is not, exo is not for you yet.

GPUStack and Xinference — the enterprise consoles

Both are supervisor-plus-workers with a web console, both are backed by companies, and both are what I see actually deployed as clusters in Asia. GPUStack is the more operational: users and roles, API keys with metering, Prometheus and Grafana, automatic recovery of failed models, Ray worker logs in the UI, llama-box (llama.cpp RPC) and vLLM/SGLang/TensorRT-LLM with tensor and pipeline parallelism, and support for nine accelerator vendors including Ascend, Hygon and MThreads. Workers are Linux only. Xinference covers more model types (speech, image, rerank) with a built-in registry, shared KV across vLLM replicas, and an enterprise edition with the multi-tenant features. Neither discovers nodes, bursts to a cloud, splits non-LLM work, or trains. If you run a department's GPUs and need to show someone a dashboard, pick one of these.

NVIDIA Dynamo and llm-d — the datacenter fabrics

Disaggregated prefill and decode, KV-cache-aware routing, multi-tier KV storage, on Kubernetes, on NVIDIA. They solve problems that begin at a rack and are the wrong tool below it. Included because "KV-aware routing" is the idea the smaller projects are now borrowing.

SkyPilot and dstack — bursting jobs, not requests

Both schedule jobs across on-prem Kubernetes and every cloud, and burst to the cloud when the on-prem cluster is full, finding the cheapest capacity. They are excellent at that and are

                    not inference servers: you bring vLLM, they place it. If you need an endpoint that decides
                    per request whether to run locally or rent, neither is that.

CoderAI — escalation, and every modality distributed

My own, so read with that in mind. One OpenAI-compatible endpoint over text, image, video,

                    TTS, STT, diarization, embeddings, rerank and OCR, with the engine chosen per model (llama.cpp
                    on CUDA *and* Vulkan in one box, transformers, vLLM, ktransformers and three pure-C
                    engines). The multi-machine layer looks like GPUStack's (nodes as engines, llama.cpp RPC with
                    layer or row split, vLLM on Ray, SGLang multi-node) plus LocalAI's zero-config (one shared
                    token, mDNS discovery, prefix-cache routing) and ops (Prometheus, per-key usage, node logs,
                    cosign-signed images). The parts nobody else has are the three-tier escalation — a model runs on
                    your card, then on a machine you own, then on a RunPod GPU rented by the second with a price cap
                    and a budget, chosen per model and switchable to "only when busy" — the fan-out of image, video,
                    embedding, speech, transcription and OCR requests over every machine that has the model, a
                    video pipeline relayed part by part, and LoRA/QLoRA training data-parallel across nodes from the
                    same server. What it lacks: Kubernetes, Apple Silicon, a model catalogue, and a community; and
                    its multi-machine paths are tested against fakes and localhost, not yet across a real cable.

Pick by situation #

• One machine, want it to just work : Ollama (+ Open WebUI).
• One model, maximum throughput, many users : vLLM directly, LiteLLM in front if teams share it.
• A stack of Macs : exo. Nothing else is close on Apple Silicon.
• Everything (image, audio, video too) on Linux, big community, no cloud : LocalAI.
• A department's GPUs with users, keys and dashboards : GPUStack; Xinference if you need its wider model registry.
• A rack on Kubernetes : Dynamo or llm-d, with SkyPilot or dstack for the bursting of jobs.
• A few machines you own plus a rented card when they are not enough, every modality, one endpoint : CoderAI. Also the only one here that trains a LoRA across your boxes.

Sources #

[ollama/ollama](https://github.com/ollama/ollama) ·
                    [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md) ·
                    [vLLM distributed serving](https://docs.vllm.ai/en/latest/serving/distributed_serving.html) ·
                    [BerriAI/litellm](https://github.com/BerriAI/litellm) ·
                    [LocalAI P2P / federated inference](https://localai.io/docs/features/distribute/) ·
                    [mudler/LocalAI](https://github.com/mudler/LocalAI) ·
                    [exo-explore/exo](https://github.com/exo-explore/exo) ·
                    [xorbitsai/inference](https://github.com/xorbitsai/inference) ·
                    [gpustack/gpustack](https://github.com/gpustack/gpustack) ·
                    [GPUStack inference backends](https://docs.gpustack.ai/2.0/user-guide/built-in-inference-backends/) ·
                    [ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo) ·
                    [llm-d introduction](https://developers.redhat.com/articles/2025/11/21/introduction-distributed-inference-llm-d) ·
                    [SkyPilot on Kubernetes](https://blog.skypilot.co/ai-on-kubernetes/) ·
                    [dstack](https://dstack.ai/) ·
                    [CoderAI docs/cluster.md](https://github.com/nextime/coderai/blob/master/docs/cluster.md).
                    Star counts: GitHub API, 2026-09-20. Corrections welcome — open an issue on the CoderAI repo and I will fix the table.