# LocalAI – Run any model – LLMs, vision, voice, image, video – on any hardware

> Source: <https://github.com/mudler/LocalAI>
> Published: 2026-08-28 22:12:49+00:00

[Deutsch](https://zdoc.app/de/mudler/LocalAI) |
[Español](https://zdoc.app/es/mudler/LocalAI) |
[français](https://zdoc.app/fr/mudler/LocalAI) |
[日本語](https://zdoc.app/ja/mudler/LocalAI) |
[한국어](https://zdoc.app/ko/mudler/LocalAI) |
[Português](https://zdoc.app/pt/mudler/LocalAI) |
[Русский](https://zdoc.app/ru/mudler/LocalAI) |
[中文](https://zdoc.app/zh/mudler/LocalAI)

**LocalAI** is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

**A small core, not a bundle.** Each backend wraps a best-in-class engine (llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX...) in its own image, pulled only when a model needs it. You install nothing you don't use.

**Composable by design**: backends are separate and pulled on demand, so you install only what your model needs** Open and extensible**: load any model, or build your own backend in any language against an open interface** Drop-in API compatibility**: OpenAI, Anthropic, and ElevenLabs APIs across every backend** Any model, any modality**: LLMs, vision, voice, image, and video behind one API** Any hardware**: NVIDIA, AMD, Intel, Apple Silicon, Vulkan, or CPU-only** Multi-user ready**: API key auth, user quotas, role-based access** Built-in AI agents**: autonomous agents with tool use, RAG, MCP, and skills** Privacy-first**: your data never leaves your infrastructure

Created by [Ettore Di Giacinto](https://github.com/mudler) and maintained by the [LocalAI team](#team).

[📖 Documentation]|[💬 Discord]|[💻 Quickstart]|[🖼️ Models]|[❓FAQ]

## model-fit-canvas-mode.mp4

## Click to see more!

## usersquota-1775167475876.mp4

## agents.mp4

## usage.mp4

## quantize-fine-tune.mp4

## talk.mp4

Note:The DMG is not signed by Apple. After installing, run:`sudo xattr -d com.apple.quarantine /Applications/LocalAI.app`

. See[#6268]for details.

Already ran LocalAI before? Use

`docker start -i local-ai`

to restart an existing container.

```
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
# CUDA 13
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13

# CUDA 12
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12

# NVIDIA Jetson ARM64 (CUDA 12, for AGX Orin and similar)
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64

# NVIDIA Jetson ARM64 (CUDA 13, for DGX Spark)
docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13
docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intel
docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan
# From the model gallery (see available models with `local-ai models list` or at https://models.localai.io)
local-ai run llama-3.2-1b-instruct:q4_k_m
# From Huggingface
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
# From the Ollama OCI registry
local-ai run ollama://gemma:2b
# From a YAML config
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
# From a standard OCI registry (e.g., Docker Hub)
local-ai run oci://localai/phi-2:latest
```

To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models`

lists installed models and `/model <name>`

switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs.

```
# Terminal 1
local-ai run llama-3.2-1b-instruct:q4_k_m

# Terminal 2
local-ai chat --model llama-3.2-1b-instruct:q4_k_m
```

Automatic Backend Detection: LocalAI automatically detects your GPU capabilities and downloads the appropriate backend. For advanced options, see[GPU Acceleration].

For more details, see the [Getting Started guide](https://localai.io/basics/getting_started/).

**June 2026**: New native biometric backends from the LocalAI team:[voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp)for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and[face-detect.cpp](https://github.com/mudler/face-detect.cpp)for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavier Python`insightface`

and`speaker-recognition`

backends ([PR #10441](https://github.com/mudler/LocalAI/pull/10441)).**June 2026**: New[realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo)(a tiny Go client for the Realtime API with a full talk-back voice loop and tool calling), plus[streaming of the realtime LLM / TTS / transcription pipeline stages](https://github.com/mudler/LocalAI/pull/10176)and[configurable WebRTC ICE candidates](https://github.com/mudler/LocalAI/pull/10231).**June 2026**: Big speech push: the[parakeet.cpp](https://github.com/mudler/parakeet.cpp)ASR engine gains[NeMo-faithful segment timestamps](https://github.com/mudler/LocalAI/pull/10207), a[multilingual streaming Nemotron-3.5 model](https://github.com/mudler/LocalAI/pull/10199),[dynamic batching for concurrent transcription](https://github.com/mudler/LocalAI/pull/10112)and[CUDA graphs](https://github.com/mudler/LocalAI/pull/10273); the new[CrispASR backend](https://github.com/mudler/LocalAI/pull/10099)adds multi-architecture ASR + TTS, and[60 Piper TTS voices across 42 languages](https://github.com/mudler/LocalAI/pull/10296)land in the gallery (plus[per-request TTS instructions and params](https://github.com/mudler/LocalAI/pull/10172)).**June 2026**: New backends and models:[locate-anything.cpp](https://github.com/mudler/LocalAI/pull/10264)for open-vocabulary object detection via ggml,[Ideogram4 image generation](https://github.com/mudler/LocalAI/pull/10201)in stablediffusion-ggml,[llama.cpp video input](https://github.com/mudler/LocalAI/pull/10216), and the[Gemma 4 QAT family with MTP speculative-decoding pairs](https://github.com/mudler/LocalAI/pull/10215). Plus an[interactive CLI chat mode](https://github.com/mudler/LocalAI/pull/10226)and[RAG source citations in agent responses](https://github.com/mudler/LocalAI/pull/10228).**June 2026**: Distributed mode hardening:[prefix-cache-aware routing](https://github.com/mudler/LocalAI/pull/10071), a[production-ready request router with auto-sized embedding/rerank batches](https://github.com/mudler/LocalAI/pull/10104),[ds4 layer-split distributed inference](https://github.com/mudler/LocalAI/pull/10098),[NATS JWT auth + TLS/mTLS](https://github.com/mudler/LocalAI/pull/10159), and[resumable file uploads](https://github.com/mudler/LocalAI/pull/10109).**May 2026**:** LocalAI 4.3.0**-`llama.cpp`

[prompt cache on by default](https://github.com/mudler/LocalAI/pull/9925)(repeated system prompts collapse from minutes to seconds),[keyless cosign signing of backend OCI images](https://github.com/mudler/LocalAI/pull/9823),[per-API-key + per-user usage attribution](https://github.com/mudler/LocalAI/pull/9920), Distributed v3 with[per-request replica routing](https://github.com/mudler/LocalAI/pull/9968).[Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.3.0)**May 2026**:** LocalAI 4.2.0**- LocalAI sees and hears:[voice recognition](https://github.com/mudler/LocalAI/pull/9500),[face recognition + antispoofing liveness](https://github.com/mudler/LocalAI/pull/9480), speaker diarization. Plus[drop-in Ollama API](https://github.com/mudler/LocalAI/pull/9284),[video generation](https://github.com/mudler/LocalAI/pull/9420), redesigned UI with i18n + admin-configurable branding, vLLM at feature parity with llama.cpp, and 11 new backends.[Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.2.0)**April 2026**:** LocalAI 4.1.0**- LocalAI becomes a control tower: distributed cluster mode with VRAM-aware smart routing + autoscaling, multi-user platform with OIDC and API keys, per-user quotas with predictive analytics, in-UI fine-tuning with TRL (auto-export to GGUF), on-the-fly quantization backend, visual pipeline editor.[Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.1.0)**March 2026**:** LocalAI 4.0.0**- native agentic orchestration with the new[Agenthub](https://agenthub.localai.io)community hub, full React UI rewrite with Canvas mode,[MCP Apps + client-side](https://github.com/mudler/LocalAI/pull/8947)with tool streaming,[WebRTC realtime audio](https://github.com/mudler/LocalAI/pull/8790),[MLX-distributed](https://github.com/mudler/LocalAI/pull/8801).[Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.0.0)**February 2026**:[Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245),[ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396)**January 2026**:** LocalAI 3.10.0**— Anthropic API support, Open Responses API, video & image generation (LTX-2), unified GPU backends, tool streaming, Moonshine, Pocket-TTS.[Release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0)**December 2025**:[Dynamic Memory Resource reclaimer](https://github.com/mudler/LocalAI/pull/7583),[Automatic multi-GPU model fitting (llama.cpp)](https://github.com/mudler/LocalAI/pull/7584),[Vibevoice backend](https://github.com/mudler/LocalAI/pull/7494)**November 2025**:[Import models via URL](https://github.com/mudler/LocalAI/pull/7245),[Multiple chats and history](https://github.com/mudler/LocalAI/pull/7325)**October 2025**:[Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/)support for agentic capabilities** September 2025**: New Launcher for macOS and Linux, extended backend support for Mac and Nvidia L4T, MLX-Audio, WAN 2.2** August 2025**: MLX, MLX-VLM, Diffusers, llama.cpp now supported on Apple Silicon** July 2025**: All backends migrated outside the main binary —[lightweight, modular architecture](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)

For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [blog](https://localai.io/blog/).

[Text generation](https://localai.io/features/text-generation/)(`llama.cpp`

,`transformers`

,`vllm`

...[and more](https://localai.io/model-compatibility/))[Text to Audio](https://localai.io/features/text-to-audio/)[Audio to Text](https://localai.io/features/audio-to-text/)[Image generation](https://localai.io/features/image-generation)[OpenAI-compatible tools API](https://localai.io/features/openai-functions/)[Realtime API](https://localai.io/features/openai-realtime/)(Speech-to-speech)[Embeddings generation](https://localai.io/features/embeddings/)[Constrained grammars](https://localai.io/features/constrained_grammars/)[Download models from Huggingface](https://localai.io/models/)[Vision API](https://localai.io/features/gpt-vision/)[Object Detection](https://localai.io/features/object-detection/)[Reranker API](https://localai.io/features/reranker/)[P2P Inferencing](https://localai.io/features/distribute/)[Distributed Mode](https://localai.io/features/distributed-mode/)— Horizontal scaling with PostgreSQL + NATS[Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/)[Built-in Agents](https://localai.io/features/agents/)— Autonomous AI agents with tool use, RAG, skills, SSE streaming, and[Agent Hub](https://agenthub.localai.io)[Backend Gallery](https://localai.io/backends/)— Install/remove backends on the fly via OCI images- Voice Activity Detection (Silero-VAD)
- Integrated WebUI

LocalAI supports **60+ backends** including llama.cpp, vLLM, SGLang, transformers, whisper.cpp, diffusers, MLX, MLX-VLM, and many more. Hardware acceleration is available for **NVIDIA** (CUDA 12/13), **AMD** (ROCm), **Intel** (oneAPI/SYCL), **Apple Silicon** (Metal), **Vulkan**, and **NVIDIA Jetson** (L4T). All backends can be installed on-the-fly from the [Backend Gallery](https://localai.io/backends/).

See the full [Backend & Model Compatibility Table](https://localai.io/model-compatibility/) and [GPU Acceleration guide](https://localai.io/features/gpu-acceleration/).

Most backends wrap a best-in-class upstream engine. A handful of them are native C/C++/GGML engines (no Python at inference) developed and maintained by the LocalAI project itself:

| Backend | What it does |
|---|---|
|

[parakeet.cpp](https://github.com/mudler/parakeet.cpp)[moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp)[moss-tts.cpp](https://github.com/mudler/moss-tts.cpp)[magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp)[ced.cpp](https://github.com/localai-org/ced.cpp)[voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp)[voxtral-tts.c](https://github.com/mudler/voxtral-tts.c)[vibevoice.cpp](https://github.com/mudler/vibevoice.cpp)[rf-detr.cpp](https://github.com/localai-org/rf-detr.cpp)[locate-anything.cpp](https://github.com/mudler/locate-anything.cpp)[depth-anything.cpp](https://github.com/mudler/depth-anything.cpp)[face-detect.cpp](https://github.com/mudler/face-detect.cpp)[free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp)[trellis2.cpp](https://github.com/localai-org/trellis2cpp)[privacy-filter.cpp](https://github.com/localai-org/privacy-filter.cpp)[LocalVQE](https://github.com/localai-org/LocalVQE)[local-store](https://github.com/mudler/LocalAI)We also maintain [apex-quant](https://github.com/localai-org/apex-quant), a per-tensor, per-layer quantization recipe for Mixture-of-Experts models that exploits their structural sparsity to produce GGUFs matching or beating Q8_0 quality - and they run out of the box on stock llama.cpp.

[Documentation](https://localai.io/)[LLM fine-tuning guide](https://localai.io/docs/advanced/fine-tuning/)[Build from source](https://localai.io/basics/build/)[Kubernetes installation](https://localai.io/basics/getting_started/#run-localai-in-kubernetes)[Integrations & community projects](https://localai.io/docs/integrations/)[Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)[Blog: release write-ups, benchmarks and engineering notes](https://localai.io/blog/)[Examples](https://github.com/mudler/LocalAI-examples)— including the[realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo)(Go client for the Realtime API with tool calling)

LocalAI is maintained by a small team of humans, together with the wider community of contributors.

— original author and project lead[Ettore Di Giacinto](https://github.com/mudler)— maintainer[Richard Palethorpe](https://github.com/richiejp)

A huge thank you to everyone who contributes code, reviews PRs, files issues, and helps users in [Discord](https://discord.gg/uJAeKSAGDy) — LocalAI is a community-driven project and wouldn't exist without you. See the full [contributors list](https://github.com/mudler/LocalAI/graphs/contributors).

If you utilize this repository, data in a downstream project, please consider citing it with:

```
@misc{localai,
  author = {Ettore Di Giacinto},
  title = {LocalAI: The free, Open source OpenAI alternative},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/go-skynet/LocalAI}},
```

Do you find LocalAI useful?

Support the project by becoming [a backer or sponsor](https://github.com/sponsors/mudler). Your logo will show up here with a link to your website.

A huge thank you to our generous sponsors who support this project covering CI expenses, and our [Sponsor list](https://github.com/sponsors/mudler):

A special thanks to individual sponsors, a full list is on [GitHub](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler). Special shout out to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!

LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/) and maintained by the [LocalAI team](#team).

MIT - Author Ettore Di Giacinto [mudler@localai.io](mailto:mudler@localai.io)

LocalAI couldn't have been built without the help of great software already available from the community. Thank you!

[llama.cpp](https://github.com/ggerganov/llama.cpp)[https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca)[https://github.com/cornelk/llama-go](https://github.com/cornelk/llama-go)for the initial ideas[https://github.com/antimatter15/alpaca.cpp](https://github.com/antimatter15/alpaca.cpp)[https://github.com/EdVince/Stable-Diffusion-NCNN](https://github.com/EdVince/Stable-Diffusion-NCNN)[https://github.com/ggerganov/whisper.cpp](https://github.com/ggerganov/whisper.cpp)[https://github.com/rhasspy/piper](https://github.com/rhasspy/piper)[exo](https://github.com/exo-explore/exo)for the MLX distributed auto-parallel sharding implementation

This is a community project, a special thanks to our contributors!
[
](https://github.com/go-skynet/LocalAI/graphs/contributors)