# Show HN: LLM Proxy - Python with SSE stream aggregation and timeout prevention

> Source: <https://github.com/lordraw77/llmproxy>
> Published: 2026-07-24 10:38:26+00:00

**A lightweight, high-performance LLM proxy for caching, automatic failover, cost tracking, and seamless integration between local and cloud AI providers.**

**llmproxy** is a lightweight Flask server that emulates the HTTP APIs of several
popular local LLM runtimes ([Ollama](https://ollama.com), the
[OpenAI](https://platform.openai.com) `/v1`

API, and
[llama.cpp](https://github.com/ggerganov/llama.cpp)'s `llama-server`

) and
transparently **forwards every request to NVIDIA's OpenAI-compatible API**
(`https://integrate.api.nvidia.com/v1`

).

This lets any tool that already speaks Ollama, OpenAI, or llama.cpp talk to a
NVIDIA-hosted model **without any client-side changes** — you simply point the
client at llmproxy instead of at a real local runtime. It covers chat,
completions, and **embeddings**, supports streaming, multi-model discovery,
optional inbound authentication, automatic retries on transient upstream errors,
and a live ** /stats** metrics & process dashboard.

The proxy starts, exposes the models, and answers both an OpenAI-compatible

`/v1/chat/completions`

call and a native Ollama streaming`/api/chat`

call — every request forwarded to NVIDIA. The recording is scripted in[(source cast:]`scripts/demo.sh`

[).]`assets/demo.cast`

```
flowchart LR
    client["Your client<br/>(Open WebUI, curl, SDK)"]
    proxy["llmproxy<br/>(Flask)"]
    nvidia["NVIDIA API<br/>integrate.api.nvidia.com/v1"]

    client -->|"Ollama / OpenAI / llama.cpp<br/>HTTP request"| proxy
    proxy -->|"OpenAI request"| nvidia
    nvidia -->|"streaming / JSON"| proxy
    proxy -->|"streaming / JSON response"| client
```

| Document | Description |
|---|---|
|

[Installation](/lordraw77/llmproxy/blob/main/docs/installation.md)[Configuration](/lordraw77/llmproxy/blob/main/docs/configuration.md)[Logging & Telemetry](/lordraw77/llmproxy/blob/main/docs/logging.md)[API Reference](/lordraw77/llmproxy/blob/main/docs/api-reference.md)[Usage Examples](/lordraw77/llmproxy/blob/main/docs/usage.md)[Testing](/lordraw77/llmproxy/blob/main/docs/testing.md)`scripts/tests.sh`

runner (bash + optional TUI)[Deployment](/lordraw77/llmproxy/blob/main/docs/deployment.md)[Troubleshooting](/lordraw77/llmproxy/blob/main/docs/troubleshooting.md)

```
# 1. Configure your NVIDIA API key
cp .env.example .env
# edit .env and set NVIDIA_API_KEY

# 2. Run with Docker Compose (or the prebuilt image)
docker compose up -d
# or: docker run -d -p 11434:11434 --env-file .env lordraw/llmproxy:latest

# 3. Test it
curl http://localhost:11434/
# → "Ollama is running"
```

The prebuilt image is published on Docker Hub as
[ lordraw/llmproxy](https://hub.docker.com/r/lordraw/llmproxy); see

[Deployment](/lordraw77/llmproxy/blob/main/docs/deployment.md)for building and publishing with the

`Makefile`

.Released under the **MIT License** — see the [ LICENSE](/lordraw77/llmproxy/blob/main/LICENSE) file for the
full text. In short: free to use, copy, modify, and distribute, with attribution
and no warranty.