A lightweight, high-performance LLM proxy for caching, automatic failover, cost tracking, and seamless integration between local and cloud AI providers.
llmproxy is a lightweight Flask server that emulates the HTTP APIs of several
popular local LLM runtimes (Ollama, the
OpenAI /v1
API, and
llama.cpp's llama-server
) and
transparently forwards every request to NVIDIA's OpenAI-compatible API
(https://integrate.api.nvidia.com/v1
).
This lets any tool that already speaks Ollama, OpenAI, or llama.cpp talk to a NVIDIA-hosted model without any client-side changes — you simply point the client at llmproxy instead of at a real local runtime. It covers chat, completions, and embeddings, supports streaming, multi-model discovery, optional inbound authentication, automatic retries on transient upstream errors, and a live ** /stats** metrics & process dashboard.
The proxy starts, exposes the models, and answers both an OpenAI-compatible
/v1/chat/completions
call and a native Ollama streaming/api/chat
call — every request forwarded to NVIDIA. The recording is scripted in[(source cast:]scripts/demo.sh
[).]assets/demo.cast
flowchart LR
client["Your client<br/>(Open WebUI, curl, SDK)"]
proxy["llmproxy<br/>(Flask)"]
nvidia["NVIDIA API<br/>integrate.api.nvidia.com/v1"]
client -->|"Ollama / OpenAI / llama.cpp<br/>HTTP request"| proxy
proxy -->|"OpenAI request"| nvidia
nvidia -->|"streaming / JSON"| proxy
proxy -->|"streaming / JSON response"| client
| Document | Description |
|---|---|
InstallationConfigurationLogging & TelemetryAPI ReferenceUsage ExamplesTestingscripts/tests.sh
runner (bash + optional TUI)DeploymentTroubleshooting
cp .env.example .env
docker compose up -d
curl http://localhost:11434/
The prebuilt image is published on Docker Hub as lordraw/llmproxy; see
Deploymentfor building and publishing with the
Makefile
.Released under the MIT License — see the LICENSE file for the full text. In short: free to use, copy, modify, and distribute, with attribution and no warranty.