{"slug": "open-source-control-panel-for-llm-deployment", "title": "Open source Control Panel for LLM deployment", "summary": "An open-source project called LLMPanel released a self-hosted control panel that deploys LLM inference on a GPU server or a user's own cloud GPU account and returns an OpenAI-compatible endpoint at /v1 without requiring a Dockerfile or nginx config. The panel provisions deployments in the background with a health check, runs vLLM by default (SGLang via API) with fp16, int8, int4 and GGUF weights, and adds per-deployment API keys, usage analytics charting p50/p95/p99 latency over 24 hours to 90 days, and a chat playground reporting tokens per second and time to first token. It supports a built-in Local Docker provider plus user-connected RunPod and Vast.ai accounts, and ships beta knowledge bases, agents served at /v1/agents/<id>/chat/completions, and Telegram, Discord, Slack and WhatsApp channels.", "body_md": "**An open-source control panel for self-hosted LLM inference.** Point it at a GPU\nserver or your own cloud GPU account, pick a model, and get an OpenAI-compatible\nendpoint — with API keys, usage analytics and a chat playground — without writing\na Dockerfile or an nginx config.\n\nThink Coolify, but for model inference.\n\n```\ncurl -fsSL https://raw.githubusercontent.com/LLMPanel/llmpanel-oss/master/install.sh | sudo bash\n```\n\n<sub>Screenshots are from a demo install with sample data.</sub>\n\nChoose where it runs, pick a model from the registry, choose a quantization and a GPU, and deploy. The panel returns immediately and provisions in the background: weights download, the container starts, and a health check gates it live, with progress streamed to the dashboard.\n\nEach deployment is a project with its own endpoint, keys, logs and analytics. The overview shows what is running, deploying, failed or stopped.\n\nThe chat playground streams from the deployment and reports tokens per second, time to first token and total latency for every reply. System prompt, temperature, max tokens and top-p are adjustable per chat.\n\nEvery request through the gateway is logged. The Usage page charts requests, tokens, throughput, error rate and p50/p95/p99 latency over 24 hours to 90 days, broken down per model and per API key, with CSV export.\n\nEvery deployment gets an API reference with ready-to-copy cURL, Python and JavaScript snippets, and keys you can scope, rate-limit and revoke.\n\n- **Providers.** The built-in**Local Docker** provider runs models on the host's\nNVIDIA GPUs and is added automatically on a self-hosted install. You can also\nconnect your own**RunPod** or**Vast.ai** account. Lambda Cloud, TensorDock,\nDataCrunch, Hyperstack, Hot Aisle and Thunder Compute are listed as coming soon.\n- **Engines.** vLLM by default (SGLang is available through the API), with\nfp16, int8, int4 and GGUF weights, multi-GPU tensor parallelism, prefix caching\nand speculative decoding.\n- **OpenAI-compatible gateway** at`/v1` : chat completions, completions,\nembeddings, image generations and model listing. Call a deployment by path\n(`/v1/deployments/<id>/…` ), let the gateway pick it from the request's`model` ,\nor give each deployment its own hostname on your domain.\n- **API keys** shown once and stored hashed, scoped to one deployment or all of\nthem, with per-key scopes (chat, embeddings, images), requests-per-minute and\nper-day limits, expiry, and usage per key.\n- **Per-GPU metrics** — utilisation, VRAM, temperature, power and clocks — from`nvidia-smi` on Local Docker and RunPod, plus token throughput from gateway\ntraffic.\n- **Image generation** with ten diffusion models (FLUX.1, SDXL, Stable Diffusion\n3.5, Qwen-Image and more) on Local Docker, behind`/v1/images/generations` ,\nwith an image playground.\n- **Knowledge bases**<sup>beta</sup> : upload PDF, Markdown, text, CSV, JSON or\nHTML; chunk and embed it with a built-in model, one of your deployments, or any\nOpenAI-compatible embeddings API; store vectors locally or in Qdrant.\n- **Agents**<sup>beta</sup> : a system prompt plus a`knowledge_search` tool over\nyour knowledge bases, served at`/v1/agents/<id>/chat/completions` .\n- **Channels**<sup>beta</sup> : put a deployment or an agent behind a Telegram,\nDiscord, Slack or WhatsApp bot, with per-channel memory, allowlists and rate\nlimits.\n- **Alerts** to Telegram, Discord or Slack when a deployment starts, becomes\nready, fails or stops.\n- **Spend tracking** for cloud providers: burn rate, provider balances and cost\nhistory.\n- **Backups** : per-deployment snapshots of agents, knowledge bases, keys, channels\nand alerts that you can download and restore, plus S3 backup for knowledge-base\ndata.\n- **Workspaces** with owner, member and viewer roles, email invitations, TOTP\ntwo-factor sign-in, optional GitHub and Google sign-in, and admin tools for\nmanaging users and sessions.\n\n|  <sub>**Knowledge base** — upload documents; they are chunked and embedded with a built-in model.</sub> |  <sub>**Agent** — a system prompt plus knowledge search, served at its own `/v1/agents/…` endpoint.</sub> | \n|  <sub>**API keys** — scopes, per-minute and per-day limits, expiry and usage per key.</sub> |  <sub>**Request logs** — every gateway call with latency, tokens, the key that made it, and the prompt and reply.</sub> | \n|  <sub>**Providers** — Local Docker is built in; connect RunPod or Vast.ai with your own key.</sub> |  <sub>**Model registry** — LLMs and image models with VRAM needs per quantization.</sub> | \n|  <sub>**Channels** — put a deployment or agent behind a Telegram, Discord, Slack or WhatsApp bot.</sub> |  <sub>**Alerts** — notify Slack, Discord or Telegram when a deployment starts, goes live, fails or stops.</sub> | \n\n| Host | Linux, with root. Docker is installed for you if it is missing. | \n| GPU (local) | An NVIDIA GPU with drivers. The installer sets up the NVIDIA Container Toolkit on apt-based systems (Ubuntu, Debian). | \n| GPU (cloud) | A RunPod or Vast.ai API key instead — the panel itself then runs fine on a small CPU-only VPS. | \n| Ports | 3000 and 8000, or 80 and 443 when you install with a domain. | \n\n```\ncurl -fsSL https://raw.githubusercontent.com/LLMPanel/llmpanel-oss/master/install.sh | sudo bash\n```\n\nThe installer:\n\n1. installs Docker and the Compose plugin if they are missing;\n2. probes for an NVIDIA GPU and, if it finds one, enables GPU support;\n3. writes `/data/llmpanel/.env` with freshly generated secrets (re-running keeps them);\n4. pulls `ghcr.io/llmpanel/llmpanel-api` and`ghcr.io/llmpanel/llmpanel-web` and starts them.\n\nThe dashboard is then on `http://<your-server>:3000` and the API on\n`http://<your-server>:8000`. For HTTPS on your own domain, see\n[Install with a domain](#install-with-a-domain).\n\nOpen the dashboard and register. **The first account becomes the instance\nadmin.** If an email provider is configured, you confirm the address with a\nsix-digit code.\n\nOn a self-hosted install the **Local Docker** provider is already there. To use\ncloud GPUs instead, open **Providers** and add a RunPod or Vast.ai API key. Keys\nare encrypted at rest.\n\nClick **New Project** and follow the four steps: provider, model, GPU, review. The\nGPU step shows how much VRAM the model needs and whether the GPU you picked fits\nit. The card on the dashboard follows provisioning live and turns green when the\nhealth check passes.\n\nFor gated models such as Llama, set `HF_TOKEN` in `/data/llmpanel/.env` first.\n\nOpen the deployment, go to **API Keys** and create one. The key (`gpup_sk_…`) is\nshown once — copy it. Keys get the `chat` scope by default; add `embeddings` or\n`images` if you need them.\n\nAny OpenAI client works. Point it at your API and use the model's Hugging Face ID\n(or the deployment ID) as `model`:\n\n```\nexport LLMPANEL_API=http://your-server:8000   # or https://api.example.com\nexport LLMPANEL_KEY=gpup_sk_your_key_here\n\ncurl \"$LLMPANEL_API/v1/chat/completions\" \\\n  -H \"Authorization: Bearer $LLMPANEL_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"Qwen/Qwen3-8B\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]}'\npython\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://your-server:8000/v1\", api_key=\"gpup_sk_your_key_here\")\nreply = client.chat.completions.create(\n    model=\"Qwen/Qwen3-8B\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n)\nprint(reply.choices[0].message.content)\n```\n\nThe deployment's **API Reference** page has these snippets prefilled with its\nexact URL, plus JavaScript and the vLLM extras (tokenize, health, Prometheus\nmetrics).\n\n**Usage** shows traffic by model and by key, **Request Logs** shows individual\nrequests, **Container Logs** streams the engine's output, and **Metrics** charts\nGPU telemetry. Requests sent from the chat playground go straight to the model\nand are not counted in Usage.\n\nWith no environment variables you get the plain-HTTP install above. For HTTPS through Traefik and Let's Encrypt, pass your domains:\n\n```\nPANEL_DOMAIN=panel.example.com \\\nAPI_DOMAIN=api.example.com \\\nACME_EMAIL=you@example.com \\\nBASE_DOMAIN=example.com \\\nCF_DNS_API_TOKEN=your-cloudflare-dns-token \\\n  curl -fsSL https://raw.githubusercontent.com/LLMPanel/llmpanel-oss/master/install.sh | sudo -E bash\n```\n\n- `PANEL_DOMAIN` and`ACME_EMAIL` are enough for HTTPS on the dashboard and API.`API_DOMAIN` defaults to`api.` plus the panel's parent domain.\n- `BASE_DOMAIN` and`CF_DNS_API_TOKEN` are optional and go together: they give\neach deployment its own hostname (`qwen3-8b-ab12cd34.example.com` ). That needs a`*.example.com` DNS record pointing at the host and a Cloudflare token with\nZone:DNS:Edit, because Let's Encrypt issues wildcard certificates only over the\nDNS-01 challenge.\n\n```\ncd /data/llmpanel && docker compose pull && docker compose up -d\n```\n\nDatabase migrations run automatically when the API starts.\n\nThe installer writes `/data/llmpanel/.env`; [`.env.example`](https://github.com/LLMPanel/llmpanel-oss/blob/master/.env.example)\ndocuments most options. The ones you are most likely to touch:\n\n| Variable | Purpose | \n|---|---|\n| `LLMPANEL_VERSION` | Image tag to run: `latest` (default), a release such as`0.1.0` , or`edge` for the newest`master` build. | \n| `HF_TOKEN` | Hugging Face token for gated models. | \n| `PANEL_DOMAIN` ,`API_DOMAIN` ,`ACME_EMAIL` | HTTPS on your own domain. | \n| `BASE_DOMAIN` ,`CF_DNS_API_TOKEN` | Per-deployment hostnames with a wildcard certificate. | \n| `KMS_URL` ,`KMS_TOKEN` | Fetch the encryption key from a remote service instead of storing it on the host — see [Security](#security) . | \n| `GITHUB_CLIENT_ID` /`_SECRET` ,`GOOGLE_CLIENT_ID` /`_SECRET` | Optional social sign-in. | \n| `EMAIL_PROVIDER` ,`RESEND_API_KEY` ,`EMAIL_FROM` | Email for sign-up codes, password resets and invitations (Resend; Cloudflare Email is also supported). Without one, those flows stay off. | \n| `DATABASE_URL` ,`DATABASE_AUTH_TOKEN` | Use a hosted libSQL database (such as Turso) instead of the local SQLite file. | \n| `MODEL_REGISTRY_URL` | Where the model catalogue comes from (default `https://registry.llmpanel.io` ). | \n\nAfter changing `.env`, apply it with `cd /data/llmpanel && docker compose up -d`.\n\n- **Registration stays open after setup.** Anyone who can reach the dashboard\ncan create an account, and on a self-hosted install every account can deploy\nto the host's GPUs. Keep the dashboard on a private network or VPN, or behind\nan authenticating proxy, unless you mean to offer it to others.\n- **Local Docker model containers publish their own ports** on the host and\nserve without an API key — the key check happens in the LLMPanel gateway.\nFirewall those ports so that only the API can reach them, and send clients\nthrough`/v1` .\n\nRequires Node 20.9+, pnpm 10 and Docker.\n\n```\npnpm install\n\ncat > apps/api/.env <<EOF\nBETTER_AUTH_SECRET=$(openssl rand -hex 32)\nENCRYPTION_KEY=$(openssl rand -hex 32)\nNEXT_PUBLIC_APP_URL=http://localhost:3000\nBETTER_AUTH_BASE_URL=http://localhost:8000\nEOF\necho \"HONO_API_URL=http://localhost:8000\" > apps/web/.env.local\n\npnpm dev        # dashboard on :3000, API on :8000\n```\n\nThe API applies database migrations on every start, using `apps/api/local.db`\nunless `DATABASE_URL` says otherwise. You don't need a GPU for most work: deploys\nto Local Docker fail without one, but the dashboard, gateway and chat paths work\nagainst any OpenAI-compatible server you already run.\n\nAfter editing anything under `packages/db/src/schema/`, generate a migration:\n\n```\npnpm --filter @llmpanel/db db:generate\n```\n\nNever run `drizzle-kit push` against a database the API will boot against. Push\nchanges the schema without recording it in the `__drizzle_migrations` ledger, so\nthe next start replays migrations that were already applied and fails with\n`table ... already exists`.\n\nBefore opening a pull request, run `pnpm typecheck`, `pnpm lint` and `pnpm build` —\nthe same checks CI runs.\n\n```\napps/\n  web/        Next.js 16 dashboard (React 19, shadcn/ui, TanStack Query, Better Auth)\n  api/        Hono API — Drizzle ORM, the OpenAI-compatible gateway, SSE for live status\npackages/\n  db/         Drizzle schema, migrations and connection (SQLite / libSQL)\n  providers/  GPU provider abstraction — Local Docker, RunPod, Vast.ai and more\n  shared/     Constants shared by both apps (roles, app name)\ninfra/\n  kms-worker/ Optional Cloudflare Worker that holds the encryption key off-host\n```\n\nThe API returns as soon as a deployment is created and provisions it in the\nbackground; the dashboard follows progress over server-sent events. Deeper\ndocumentation lives in [`docs/`](https://github.com/LLMPanel/llmpanel-oss/blob/master/docs), starting with\n[`docs/01-overview.md`](https://github.com/LLMPanel/llmpanel-oss/blob/master/docs/01-overview.md).\n\nProvider credentials, bot tokens, alert-channel settings and other secrets are\nencrypted at rest with AES-256-GCM, keyed by HKDF-SHA256 from `ENCRYPTION_KEY`,\nand each ciphertext is bound to its database row. The API refuses to start in\nproduction without a valid key. API keys are stored as SHA-256 hashes.\n\nIf you would rather the key never sat on the host, deploy\n[`infra/kms-worker`](https://github.com/LLMPanel/llmpanel-oss/blob/master/infra/kms-worker) and set `KMS_URL` and `KMS_TOKEN`: the key\nis then fetched at boot, held only in memory, and can be revoked.\n\nTo report a vulnerability, see [SECURITY.md](https://github.com/LLMPanel/llmpanel-oss/blob/master/SECURITY.md). Please do not open a\npublic issue for one.\n\nContributions are welcome. See [CONTRIBUTING.md](https://github.com/LLMPanel/llmpanel-oss/blob/master/CONTRIBUTING.md) for how this\nrepository is maintained — it is a mirror of a private monorepo, which changes the\nworkflow slightly — and what a good pull request looks like.\n\nYou can run, modify and self-host LLMPanel freely, including commercially. If you\nmodify it and offer it to others over a network, the AGPL requires you to publish\nyour modified source. A managed service is available at\n[llmpanel.io](https://llmpanel.io) for teams who would rather not host it.", "url": "https://wpnews.pro/news/open-source-control-panel-for-llm-deployment", "canonical_source": "https://github.com/LLMPanel/llmpanel-oss", "published_at": "2026-10-21 14:21:38+00:00", "updated_at": "2026-10-21 14:43:15.570196+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-agents", "mlops"], "entities": ["LLMPanel", "vLLM", "SGLang", "RunPod", "Vast.ai", "Qdrant", "FLUX.1", "SDXL"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/open-source-control-panel-for-llm-deployment", "markdown": "https://wpnews.pro/news/open-source-control-panel-for-llm-deployment.md", "text": "https://wpnews.pro/news/open-source-control-panel-for-llm-deployment.txt", "jsonld": "https://wpnews.pro/news/open-source-control-panel-for-llm-deployment.jsonld"}}