{"slug": "show-hn-inference-stack-everything-you-need-to-own-your-inference", "title": "Show HN: Inference Stack – Everything you need to own your inference", "summary": "BerriAI released Inference Stack v0.1.2, an open-source package that lets teams run their own model inference by bundling vLLM for GPU serving, LiteLLM for a single API gateway with keys, teams and budgets, and Prometheus-backed dashboards. The stack installs with one curl command on a single machine running Docker Compose, curl and OpenSSL, then serves a launcher at localhost:8080 where an admin signs in with a generated password, picks a model in vLLM, deploys it and sends prompts through the LiteLLM Playground. Real models require NVIDIA GPUs; without one, users can enable simulation in a model's Settings for canned responses, and the LiteLLM router selects a vLLM replica per request using cache and queue data.", "body_md": "### vLLM\n\nSearch Hugging Face, check GPU requirements, and deploy from the launcher. vLLM runs the model on your GPUs.\n\n[vLLM docs](https://docs.vllm.ai)\n\nEverything you need to own your inference.\n\nDeploy with one command.\n\nFor one machine. Requires Docker with Compose, curl, and OpenSSL. [Setup details](https://www.inferencestack.io/docs#docker-compose)\n\n```\ncurl -fsSL https://github.com/BerriAI/inference-stack/releases/download/v0.1.2/install.sh | bash -s -- docker\n```\n\nOpen [localhost:8080](http://localhost:8080) and sign in as `admin` with the generated password. Choose a model in **vLLM**, deploy it, then open **LiteLLM → Playground** to send a prompt.\n\nReal models need NVIDIA GPUs. No GPU? Enable simulation in the model’s Settings for canned responses.\n\nSend requests through one API. Create keys, manage teams and budgets, or try a model in the Playground.\n\nSelects a vLLM replica for each request using cache and queue data. Routing works by default; adjust the weights when you need to.\n\nSee model health, request traffic, and response times in the included dashboards. Metrics come from Prometheus.", "url": "https://wpnews.pro/news/show-hn-inference-stack-everything-you-need-to-own-your-inference", "canonical_source": "https://www.inferencestack.io/", "published_at": "2026-11-22 02:02:13+00:00", "updated_at": "2026-11-22 02:41:50.809595+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "large-language-models", "mlops", "developer-tools"], "entities": ["BerriAI", "Inference Stack", "vLLM", "LiteLLM", "Prometheus", "Hugging Face", "Docker Compose", "NVIDIA"], "also_reported_by": [], "takeaway": "Inference Stack v0.1.2 bundles vLLM serving, a LiteLLM gateway with per-request replica routing based on cache and queue data, and Prometheus dashboards into a single Docker Compose install that runs on one machine with NVIDIA GPUs.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/show-hn-inference-stack-everything-you-need-to-own-your-inference", "markdown": "https://wpnews.pro/news/show-hn-inference-stack-everything-you-need-to-own-your-inference.md", "text": "https://wpnews.pro/news/show-hn-inference-stack-everything-you-need-to-own-your-inference.txt", "jsonld": "https://wpnews.pro/news/show-hn-inference-stack-everything-you-need-to-own-your-inference.jsonld"}}