{"slug": "show-hn-llm-proxy-python-with-sse-stream-aggregation-and-timeout-prevention", "title": "Show HN: LLM Proxy - Python with SSE stream aggregation and timeout prevention", "summary": "Developer lordraw77 released llmproxy, a lightweight Flask server that emulates Ollama, OpenAI, and llama.cpp APIs and forwards every request to NVIDIA's OpenAI-compatible API, enabling any existing tool to use NVIDIA-hosted models without client-side changes. The proxy supports streaming, multi-model discovery, automatic retries, and a live /stats dashboard, and is available under the MIT License on Docker Hub as lordraw/llmproxy.", "body_md": "**A lightweight, high-performance LLM proxy for caching, automatic failover, cost tracking, and seamless integration between local and cloud AI providers.**\n\n**llmproxy** is a lightweight Flask server that emulates the HTTP APIs of several\npopular local LLM runtimes ([Ollama](https://ollama.com), the\n[OpenAI](https://platform.openai.com) `/v1`\n\nAPI, and\n[llama.cpp](https://github.com/ggerganov/llama.cpp)'s `llama-server`\n\n) and\ntransparently **forwards every request to NVIDIA's OpenAI-compatible API**\n(`https://integrate.api.nvidia.com/v1`\n\n).\n\nThis lets any tool that already speaks Ollama, OpenAI, or llama.cpp talk to a\nNVIDIA-hosted model **without any client-side changes** — you simply point the\nclient at llmproxy instead of at a real local runtime. It covers chat,\ncompletions, and **embeddings**, supports streaming, multi-model discovery,\noptional inbound authentication, automatic retries on transient upstream errors,\nand a live ** /stats** metrics & process dashboard.\n\nThe proxy starts, exposes the models, and answers both an OpenAI-compatible\n\n`/v1/chat/completions`\n\ncall and a native Ollama streaming`/api/chat`\n\ncall — every request forwarded to NVIDIA. The recording is scripted in[(source cast:]`scripts/demo.sh`\n\n[).]`assets/demo.cast`\n\n```\nflowchart LR\n    client[\"Your client<br/>(Open WebUI, curl, SDK)\"]\n    proxy[\"llmproxy<br/>(Flask)\"]\n    nvidia[\"NVIDIA API<br/>integrate.api.nvidia.com/v1\"]\n\n    client -->|\"Ollama / OpenAI / llama.cpp<br/>HTTP request\"| proxy\n    proxy -->|\"OpenAI request\"| nvidia\n    nvidia -->|\"streaming / JSON\"| proxy\n    proxy -->|\"streaming / JSON response\"| client\n```\n\n| Document | Description |\n|---|---|\n|\n\n[Installation](/lordraw77/llmproxy/blob/main/docs/installation.md)[Configuration](/lordraw77/llmproxy/blob/main/docs/configuration.md)[Logging & Telemetry](/lordraw77/llmproxy/blob/main/docs/logging.md)[API Reference](/lordraw77/llmproxy/blob/main/docs/api-reference.md)[Usage Examples](/lordraw77/llmproxy/blob/main/docs/usage.md)[Testing](/lordraw77/llmproxy/blob/main/docs/testing.md)`scripts/tests.sh`\n\nrunner (bash + optional TUI)[Deployment](/lordraw77/llmproxy/blob/main/docs/deployment.md)[Troubleshooting](/lordraw77/llmproxy/blob/main/docs/troubleshooting.md)\n\n```\n# 1. Configure your NVIDIA API key\ncp .env.example .env\n# edit .env and set NVIDIA_API_KEY\n\n# 2. Run with Docker Compose (or the prebuilt image)\ndocker compose up -d\n# or: docker run -d -p 11434:11434 --env-file .env lordraw/llmproxy:latest\n\n# 3. Test it\ncurl http://localhost:11434/\n# → \"Ollama is running\"\n```\n\nThe prebuilt image is published on Docker Hub as\n[ lordraw/llmproxy](https://hub.docker.com/r/lordraw/llmproxy); see\n\n[Deployment](/lordraw77/llmproxy/blob/main/docs/deployment.md)for building and publishing with the\n\n`Makefile`\n\n.Released under the **MIT License** — see the [ LICENSE](/lordraw77/llmproxy/blob/main/LICENSE) file for the\nfull text. In short: free to use, copy, modify, and distribute, with attribution\nand no warranty.", "url": "https://wpnews.pro/news/show-hn-llm-proxy-python-with-sse-stream-aggregation-and-timeout-prevention", "canonical_source": "https://github.com/lordraw77/llmproxy", "published_at": "2026-07-24 10:38:26+00:00", "updated_at": "2026-07-24 10:52:17.003601+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["lordraw77", "llmproxy", "Flask", "Ollama", "OpenAI", "llama.cpp", "NVIDIA", "Docker Hub"], "alternates": {"html": "https://wpnews.pro/news/show-hn-llm-proxy-python-with-sse-stream-aggregation-and-timeout-prevention", "markdown": "https://wpnews.pro/news/show-hn-llm-proxy-python-with-sse-stream-aggregation-and-timeout-prevention.md", "text": "https://wpnews.pro/news/show-hn-llm-proxy-python-with-sse-stream-aggregation-and-timeout-prevention.txt", "jsonld": "https://wpnews.pro/news/show-hn-llm-proxy-python-with-sse-stream-aggregation-and-timeout-prevention.jsonld"}}