{"slug": "fifty-api-keys-one-queue-ten-gpu-workers-the-parts-of-a-shared-llm-that-are-not", "title": "Fifty API keys, one queue, ten GPU workers: the parts of a shared LLM that are not the model", "summary": "An engineer demonstrated a self-hosted shared LLM setup that issues fifty per-team API keys against a single OpenAI-compatible endpoint, backed by a queue feeding ten GPU workers with 8 concurrent requests each (80 slots). The demo, which used test-mode processes returning filler text rather than real model inference, showed per-key cost accounting splitting a $326.97 reserved-worker bill across teams by token share, with finance-board-reporting consuming 18.7% ($61.03). The write-up notes that worker scaling and worker-failure scenarios were not exercised during the run.", "body_md": "If you are the engineer asked to put one self-hosted model behind a whole company, the model choice is the short meeting. The long tickets are about keys, queues, alerts and the invoice. We recorded a [demo of a private ChatGPT shared by fifty teams](https://inferencekey.com/news/private-chatgpt-fifty-teams-one-bill) with exactly that setup: fifty keys, one endpoint that speaks the OpenAI API format, ten GPU servers, and a cost screen that splits an hourly bill per key.\n\nOne disclosure first, because it shapes everything below: the ten \"H100\" machines (an H100 is a high-end Nvidia GPU) were test-mode processes on one ordinary host, returning filler text instead of running a model. The panel, keys, queue and accounting were the real product. So this run says a lot about the management side and nothing about how fast a real model answers.\n\nThe demo issues a key per team, often per use case: roadmap, competitive intel, security policy Q&A, board reporting. Fifty in total, and each one is allowed on a single endpoint, `nexora-private-gpt/enterprise-chat`.\n\nTwo reasons for that. A leaked key can only reach the company chat, not anything else the project runs. And the key is the unit of accounting, so a key per use case is what lets you tell finance that board reporting, not \"finance\", is the heavy user. Offboarding a team is unticking its key. How keys, scopes and environment variables work is in the [Tokens & API keys guide](https://docs.inferencekey.com/en/quickstart/tokens/).\n\nBecause the endpoint accepts the same request format as OpenAI, existing clients only change a base URL and a key.\n\nIn the panel, the served model is a workload and each machine serving it is a worker. Ten workers serve one workload, and each takes up to 8 requests at the same time (its concurrency), so the fleet has 80 slots. All fifty keys feed one queue and whichever worker has a free slot claims the next job. You decide which machines serve the model, and the queue decides which one answers a given request.\n\nThat pays off when you scale: you pick a registered machine or a new cloud GPU, set its concurrency, and no team changes its key or URL. Nobody clicked \"Add worker\" while the traffic was running, and the [write-up of the fifty-team demo](https://inferencekey.com/news/private-chatgpt-fifty-teams-one-bill) lists that, and a worker dying mid-load, among the scenes it has not filmed yet.\n\nThe backlog is the number of requests waiting for a free slot, and throughput is how many get finished per minute. A backlog of 20 says little alone. One that keeps climbing is the signal to add a machine. The drawer turns that into a label. It stays on \"Healthy flow\" as long as throughput is above zero and the backlog, compared over the last complete minutes (five at most), has grown by no more than 25% or 2 jobs, whichever is larger. A falling backlog with throughput going up shows \"Recovering\". A rising one shows \"At risk\", and \"Critical\" when throughput falls at the same time.\n\nA trap on the same screen: the jobs-per-minute counter in the header only counts completions your open tab has seen in the past minute, so it restarts at zero on every page load. In the take it climbed from 0 to about 3,700. The real load is the drawer's five-minute average, 4,252 finished per minute, from a script pushing about 70 requests a second.\n\nNothing is priced per word. The panel prices reserved worker time and only uses tokens (the chunks of text a model reads and writes, roughly three quarters of a word each) to split the total:\n\n```\nproject_cost = reserved_hours * hourly_price          # 134.53 h * $2.4306/h = $326.97\nkey_cost     = key_tokens / project_tokens * project_cost\n               # 17.37M / 93.07M = 18.7% -> $61.03 for finance-board-reporting\n```\n\nThe hourly price is $1,750 a month over 720 hours. Ten workers cost $24.31 an hour, about $17,500 a month if they never stop. A worker accrues hours for the whole time it stays assigned to the project, busy or not. The total moves by about a cent every 1.5 seconds, so screenshots taken seconds apart differ by a few cents. Do not write a test that expects them to match.\n\nThe panel shows $3.51 per million tokens. Do not put that next to a public API's price. Each worker was reserved for about 13.5 hours, and at its steady rate the test traffic would have filled only around 72 minutes of that, so the figure mostly measures machines waiting. When you pay by the hour, that unit cost drops the busier you keep the fleet, and on real GPUs it comes down to how many tokens per second each machine can produce.\n\nWhat you would still have to measure on real GPUs:\n\nLike a soundcheck with the PA on and no band, it tells you the cables and the desk work. It does not tell you how loud the room gets.\n\nThe [full article on one private assistant for fifty teams](https://inferencekey.com/news/private-chatgpt-fifty-teams-one-bill) has the video, every panel screen and the full cost table per key.", "url": "https://wpnews.pro/news/fifty-api-keys-one-queue-ten-gpu-workers-the-parts-of-a-shared-llm-that-are-not", "canonical_source": "https://dev.to/inferencekey/fifty-api-keys-one-queue-ten-gpu-workers-the-parts-of-a-shared-llm-that-are-not-the-model-4h7g", "published_at": "2026-10-21 05:40:36+00:00", "updated_at": "2026-10-21 05:47:31.390701+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["InferenceKey", "Nvidia", "OpenAI", "Nexora"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fifty-api-keys-one-queue-ten-gpu-workers-the-parts-of-a-shared-llm-that-are-not", "markdown": "https://wpnews.pro/news/fifty-api-keys-one-queue-ten-gpu-workers-the-parts-of-a-shared-llm-that-are-not.md", "text": "https://wpnews.pro/news/fifty-api-keys-one-queue-ten-gpu-workers-the-parts-of-a-shared-llm-that-are-not.txt", "jsonld": "https://wpnews.pro/news/fifty-api-keys-one-queue-ten-gpu-workers-the-parts-of-a-shared-llm-that-are-not.jsonld"}}