llmman, the unified interface for language models

Eric Curtin's llmman, a Rust command-line tool billed as "Run any agent on any model," manages models as standard OCI Image Layouts pulled from Docker Hub, Hugging Face or private registries and serves them behind a single API on port 17434 that speaks OpenAI, Ollama and Anthropic formats. The tool can also launch code agents including Claude Code, Codex, OpenCode and Gemini against a local model or remote endpoint, and works with llama.cpp as its inference runtime, which llmman downloads and caches automatically. Installation is available via a curl script or Homebrew, and models are fetched with commands such as "llmman pull hf.co/unsloth/Qwen3.5-0.8B-GGUF.

llmman in two minutes llmman-in-two-minutes llmman https://github.com/llmmanorg/llmman is a command-line tool written in Rust with the slogan "Run any agent on any model" . llmman is a project initiated by Eric Curtin https://github.com/ericcurtin 👏 Its 3 main functions are as follows: 1. It manages models like OCI images. A model is retrieved from Docker Hub, Hugging Face, ... or your private registry, and is stored locally in a standard OCI Image Layout . So, no proprietary format, we stay standard. 2. The part that interests me the most: It serves these models behind a single API llmman serve , port 17434 that knows how to speak OpenAI, Ollama and Anthropic . 3. It can launch code agents Claude Code, Codex, OpenCode, Gemini, ... pointed at a local model or a remote endpoint, in a single command. And llmman knows how to work with llamacpp https://github.com/ggml-org/llama.cpp as an inference engine otherwise known as: as a runtime . This is convenient, because my preferred engines are llamacpp and Docker Model Runner https://docs.docker.com/ai/model-runner/ which itself uses llamacpp . So, today let's see how to use llmman with llamacpp to serve my favorite model. llmman comes with many features, I encourage you to consult the excellent documentation. Installation installation I tested this installation on both macOS and Linux . I used the following options: Official script curl -fsSL https://llmmanorg.github.io/install.sh | sh or via Homebrew brew install llmmanorg/tap/llmman For Linux , I used the curl option, and for macOS , I used the curl option on one Mac and the brew option on another Mac. Choosing the llamacpp runtime choosing-the-llamacpp-runtime llamacpp runtime There are several solutions here as well. I chose to let llmman handle it, which will download and cache the official llamacpp binary once , and then use it at every startup. It is also possible to specify the path to your existing llamacpp installation . Starting up starting-up It's simple, just run this command: llmman serve To verify that everything is working correctly, you can use the following commands: curl -s http://127.0.0.1:17434/api/version curl -s http://127.0.0.1:17434/v1/models {"data": ,"object":"list"} if you haven't installed any model We need a model we-need-a-model To retrieve models, it's also simple, and afterwards llmman will allow you to easily switch from one to another. For example, if you want to install Qwen3.5-0.8B-GGUF , the version provided by Unsloth https://unsloth.ai/ on the Hugging Face https://huggingface.co/ platform, type the following command: llmman pull hf.co/unsloth/Qwen3.5-0.8B-GGUF Wait a little while for the download, and once it's finished, you can verify that everything is fine and that your model is indeed there with these commands: llmman list llmman show hf.co/unsloth/Qwen3.5-0.8B-GGUF There, you are now ready to query your model. Testing the three API versions Ollama, Anthropic, OpenAI testing-the-three-api-versions-ollama-anthropic-openai You can query the various APIs using curl : OpenAI curl -s http://127.0.0.1:17434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"hf.co/unsloth/Qwen3.5-0.8B-GGUF","messages": {"role":"user","content":"Explain why the hawaiian pizza is the best pizza in the world"} ,"max tokens":300,"chat template kwargs":{"enable thinking":false}}' Anthropic curl -s http://127.0.0.1:17434/v1/messages \ -H "Content-Type: application/json" \ -d '{"model":"hf.co/unsloth/Qwen3.5-0.8B-GGUF","max tokens":200,"messages": {"role":"user","content":"Explain why the hawaiian pizza is the best pizza in the world"} ,"thinking":{"type":"disabled"}}' Ollama curl -s http://127.0.0.1:17434/api/chat -d '{\n "model": "hf.co/unsloth/Qwen3.5-0.8B-GGUF",\n "messages": {"role": "user", "content": "Explain why the hawaiian pizza is the best pizza in the world"} ,\n "stream": false\n}' As you can see, it's easy to implement llmman and then use it in generative AI applications or with your favorite agents. Go Example with the OpenAI Go SDK go-example-with-the-openai-go-sdk You can therefore use the API exposed by llmman with the usual frameworks, as here with Go and the OpenAI API: package main import "context" "fmt" openai "github.com/openai/openai-go/v3" "github.com/openai/openai-go/v3/option" func main { ctx := context.Background model := "hf.co/unsloth/Qwen3.5-0.8B-GGUF" baseURL := "http://127.0.0.1:17434/v1" client := openai.NewClient option.WithBaseURL baseURL , option.WithAPIKey "not-needed" , completion, err := client.Chat.Completions.New ctx, openai.ChatCompletionNewParams{ Model: model, Messages: openai.ChatCompletionMessageParamUnion{ openai.SystemMessage "You are a pizza expert." , openai.UserMessage "Tell me more about Hawaiian pizza." , }, Temperature: openai.Float 0.0 , TopP: openai.Float 0.9 , MaxTokens: openai.Int 2048 , }, option.WithJSONSet "chat template kwargs", map string any{ "enable thinking": false, }, , if err = nil { fmt.Println " error:", err, " " return } if len completion.Choices == 0 { fmt.Println " error: empty completion " return } fmt.Println completion.Choices 0 .Message.Content } One last thing before you go one-last-thing-before-you-go llmman provides a particularly well-made web UI. To access it, it's simple: open this URL in your browser: http://127.0.0.1:17434/ http://127.0.0.1:17434/ and you will be able to search for and download models, and "chat with your models": That's all for today. I'll leave you to play with llmman . Feel free to ask questions 🙂 Written by Keep reading Docker Agent + llama.cpp: a local code agent in 5 minutes 🤖💙🦙 Serve a GGUF model with llama.cpp, point Docker Agent at it with a short agent.yaml, and get a fully local code agent running in minutes. Sep 15, 2026 Zed Editor + Docker Agent + ACP: coding with a local agent plugged into llmman Plug Zed's agent panel into Docker Agent over ACP, then point it at llmman to chat with a fully local, OpenAI-compatible coding agent. Oct 3, 2026 Docker Model Runner, standalone edition dmr Install and use dmr, the standalone Docker Model Runner: one binary with the inference daemon and the full model CLI, no Docker Desktop needed. Jul 31, 2026