{"slug": "show-hn-gpuworks-a-40-mb-gpu-cluster-manager-for-vllm-sglang-no-k8s-needed", "title": "Show HN: GPUWorks – A 40 MB GPU cluster manager for vLLM/SGLang, no K8s needed", "summary": "GPUWorks, released by flitai-io as a 40 MB control-plane download, is a GPU cluster manager and Model-as-a-Service platform that deploys vLLM, SGLang or template-defined inference engines across NVIDIA, AMD Instinct and Ryzen AI Max, Huawei Ascend, T-Head PPU, Hygon DCU and Apple Silicon nodes without Kubernetes. The built-in catalog ships 1,235 launch specs for 392 models, and in an Alibaba Cloud drill one server ran 1,003 rented GPU machines across 7 regions, placing over 500 replicas within 4 minutes and handling 2,000 concurrent requests through its OpenAI-compatible gateway while peaking at 9 CPU cores and 1.6 GB of memory. The project also syncs instances from RunPod, CompShare, AutoDL, Vast.ai and Alibaba Cloud plus ZStack Cloud VMs into one pool, and exposes a read-only MCP endpoint for clients such as Claude Code.", "body_md": "**A 40 MB MaaS platform: one cluster across public and private clouds, on any mix of accelerators.**\n\n  English | [简体中文](https://github.com/flitai-io/gpuworks/blob/master/README.zh-CN.md)\n\nGPUWorks is an ultra-lightweight GPU management and Model-as-a-Service (MaaS) platform. The whole control plane, web UI included, is a 40 MB download. Join your own GPU servers, rented cloud instances or containers, or a Mac with one command, pick a model, and GPUWorks estimates the memory it needs, picks the GPUs, starts vLLM, SGLang or any engine an inference template defines (Strata, ninfer, …), and serves it behind one OpenAI-compatible endpoint. NVIDIA, AMD Instinct and Ryzen AI Max, Huawei Ascend, T-Head PPU, Hygon DCU and Apple Silicon run side by side in the same cluster.\n\nThe pool of capacity is the core: bare-metal servers, public cloud instances and private cloud VMs today, with Kubernetes clusters and Slurm partitions joining next. On top of it, GPUWorks serves models now and will run fine-tuning and training jobs on the same accelerators.\n\n- **Verified model × accelerator combinations deploy in one click.** The built-in catalog ships 1,235 launch specs\nfor 392 models: image, engine, flags and pinned weight revisions for each GPU type. The combinations we have run\non real NVIDIA, AMD, Ascend, PPU and Apple hardware come with launch settings that are known to work, so you pick a\nmodel and press deploy instead of hunting for flags. Qwen, DeepSeek, GLM, Kimi, MiniMax, Llama, Gemma, gpt-oss,\nplus embedding, rerank, speech, image and video models.\n- **Heterogeneous accelerators, one scheduler.** NVIDIA, AMD, Ascend, PPU, Hygon and Apple Silicon nodes join the same\ncluster; the scheduler matches each deployment to hardware its launch spec supports.\n- **Public and private cloud in one pool.** Sync instances from RunPod, CompShare, AutoDL, Vast.ai and Alibaba Cloud,\nor VMs from ZStack Cloud, install the agent in one click, and schedule them together with your own servers.\n- **Ultra-lightweight.** The server and the agent are each one static binary (`CGO_ENABLED=0` ); the server embeds the\nweb UI (English / 中文) and the agents for every platform, and keeps its state in one SQLite file. The agent dials\nout over WebSocket, so nodes need no inbound ports (or the server dials in, for nodes that cannot reach out). In a\ndrill on Alibaba Cloud, one server ran 1,003 rented GPU machines across 7 regions: over 500 replicas placed and\nrunning within 4 minutes, and 2,000 concurrent requests through the gateway with the server peaking at 9 CPU cores\nand 1.6 GB of memory.\n- **Memory estimate and explained placement.** Before anything starts, you see how much GPU memory weights, KV\ncache and engine overhead need, which GPUs the scheduler picked and why the others were rejected.\n- **Share a GPU between models.** dGPU puts a hard memory quota on each instance so several models fit on one card\n(NVIDIA via HAMi-core, Ascend via vCANN, PPU via HGGC, Hygon via hy-smi virtual devices). Sleep/wake lets vLLM models take turns\non the same GPUs on NVIDIA, Ascend, PPU and Hygon.\n- **Scale out.** Replicas across nodes, multi-node tensor/pipeline parallel, and prefill/decode disaggregation\n(vLLM + NIXL on NVIDIA, vLLM-Ascend + Mooncake on Ascend, SGLang or vLLM + Mooncake on PPU, within one host or over RDMA).\n- **OpenAI-compatible gateway.** Chat, completions, embeddings, rerank, audio, images, video and realtime under`/v1` ;`/v1/messages` and`/v1/responses` pass through. Least-request, prefix-affinity or failover routing, API\nkeys, per-key token usage.\n- **See what is going on.** GPU and engine metrics, searchable logs, per-request traces with a timeline (OTLP export),\na web terminal into hosts and containers, and a built-in benchmark that sweeps concurrency and finds the SLA limit.\n- **AI assistant and MCP.** Ask questions about the cluster in the UI, or connect Claude Code / any MCP client to the\nread-only MCP endpoint.\n- **Air-gapped ready.** Import images and models from archives or an HTTP file server, and sync models between nodes.\n\n**1. Install the server** on a Linux host (x86_64 or arm64) or an Apple Silicon Mac:\n\n```\nbash <(curl -fsSL https://dl.gpuworks.ai/get.sh)\n```\n\nThe script picks the fastest download source (including mirrors in mainland China), verifies the checksum and runs\nthe installer, which sets up a service on port 8800 and prints the URL and the admin password. Add `--quiet` to\naccept every default. To check each step yourself, follow the [manual install](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/install-server.md).\n\n**2. Add a GPU node.** Open the web UI, go to **Compute nodes → Add node**, and run the generated command on the GPU\nhost:\n\n```\ncurl -fsSL http://<server>:8800/install.sh | sh -s -- --server http://<server>:8800 --token <node-token>\n```\n\n**3. Deploy a model.** Open **Deploy**, pick a model such as `Qwen/Qwen3-1.7B` and a recommended inference template,\nreview the configuration evaluation (memory estimate and placement), and deploy. Then create an inference token under\n**API Tokens** and call the model by its public model name:\n\n```\ncurl http://<server>:8800/v1/chat/completions \\\n  -H \"Authorization: Bearer gw-...\" -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"qwen3-1.7b\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]}'\n```\n\nThe full walkthrough is in [Getting started](https://github.com/flitai-io/gpuworks/blob/master/docs-en/tutorials/getting-started.md). For a Mac, see\n[Run your first local model on a Mac](https://github.com/flitai-io/gpuworks/blob/master/docs-en/tutorials/mac-local-model.md); for air-gapped sites, see\n[Offline deployment](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/deploy-offline.md).\n\nPaste this into Claude Code, Codex, Cursor or a similar agent that has shell access to your server:\n\n```\nInstall the GPUWorks control plane on this host and verify it.\n1. Run `bash <(curl -fsSL https://dl.gpuworks.ai/get.sh) --quiet`. It downloads the latest build, verifies its\n   checksum and installs a service on port 8800 (use sudo when asked). If I have no sudo or no systemd, append\n   `--dir ~/gpuworks-server --no-systemd` instead.\n2. Verify with `curl -fsS http://<listen-address>/healthz` (expect \"status\":\"ok\") and show me the install summary:\n   URL, install directory, data directory and the initial admin password location.\n3. Do not open firewall ports, change other services, or join GPU nodes. Tell me which address GPU nodes should\n   use to reach the server; I will add nodes from the web UI.\n```\n\n| Accelerator | Whole-GPU deploy | dGPU (shared GPU) | Sleep/wake | Multi-node | PD disaggregation | \n|---|---|---|---|---|---|\n| **NVIDIA** (CUDA) | ✅ Docker, containerd or runveil (built-in rootless runtime) | ✅ memory quota at MiB granularity | ✅ vLLM | ✅ | ✅ vLLM + NIXL | \n| **AMD** (ROCm: Instinct MI300 series, Ryzen AI Max) | ✅ Docker or containerd | — | — | ✅ vLLM | — | \n| **Huawei Ascend** (CANN) | ✅ Docker, containerd or runveil (built-in rootless runtime) | ✅ memory quota at MiB granularity | ✅ vLLM-Ascend | ✅ | ✅ vLLM-Ascend + Mooncake | \n| **T-Head PPU** (HGGC) | ✅ Docker, containerd or runveil (built-in rootless runtime) | ✅ memory quota at MiB granularity | ✅ vLLM | ✅ | ✅ SGLang + Mooncake (same node or over RDMA), vLLM + Mooncake (RDMA) | \n| **Hygon DCU** (DTK) | ✅ Docker, containerd or runveil (built-in rootless runtime) | ✅ memory quota at MiB granularity | ✅ vLLM | ✅ | — | \n| **Apple Silicon** (M1–M5, macOS 15+) | ✅ vllm-metal, MLX, llama.cpp as native processes | — | — | — | — | \n\n✅ verified on real hardware · — not supported or not validated. Sleep/wake beyond NVIDIA is\nverified on Ascend 910C, PPU ZW810E and Hygon K100; the tested versions and limits are in\n[choosing features](https://github.com/flitai-io/gpuworks/blob/master/docs-en/explanation/choosing-features.md#sleepwake-models-take-turns-on-one-gpu).\nGPUs are detected with the vendor CLIs (`nvidia-smi`, `npu-smi`, `ppu-smi`, `hy-smi`) or, for AMD, amdgpu sysfs and KFD, so there is no NVML or cgo\ndependency. Per-combination status lives in the [support matrix](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/support-matrix.md) and the\n[Ascend matrix](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/ascend-support-matrix.md).\n\n**Engines:** vLLM (incl. vLLM-Ascend and vLLM-Omni), SGLang, and on Mac vllm-metal, MLX-VLM, MLX-Audio and\nllama.cpp. Built-in images are published as `gpuworks/gpuworks-runner` on Docker Hub, Quay and Huawei SWR\n([image families](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/image-families.md)); you can also bring your own image.\n\nThe built-in catalog pins every model to fixed Hugging Face and ModelScope revisions and ships launch specs per engine and accelerator. The deploy page recommends the specs that match the model and the target accelerator.\n\n| Type | Examples | \n|---|---|\n| Chat / reasoning / code | Qwen3 / 3.5 / 3.6 / 3.8, Qwen3-Coder, DeepSeek R1 / V3.2 / V4, GLM-4.5 – 5.3, Kimi K2 – K3, MiniMax M2 – M3, Llama 3.x / 4, Gemma 3 / 4, gpt-oss, Mistral Large 3, ERNIE 4.5, Step-3.5, MiMo, Seed-OSS, Nemotron | \n| Vision-language / OCR | Qwen3-VL, Qwen3-Omni, GLM-4.xV, Kimi-VL, DeepSeek-OCR, PaddleOCR-VL, GLM-OCR | \n| Embedding / rerank | Qwen3-Embedding, Qwen3-Reranker, bge-m3, bge-reranker-v2-m3, Jina v5, EmbeddingGemma | \n| Speech | Qwen3-ASR, Whisper large-v3, Qwen3-TTS, GLM-TTS, Voxtral, FishAudio S2-Pro | \n| Image / video | Qwen-Image, GLM-Image, Stable Diffusion 3.5, Wan 2.1 / 2.2, LTX-2.5, Cosmos3 | \n\nCatalog totals: 392 models from 77 organizations, 1,235 launch specs (745 NVIDIA, 216 PPU, 209 Ascend, 41 Hygon, 15 AMD, 9 Apple). Models outside the catalog can be registered from Hugging Face, ModelScope, a local directory, NFS or an HTTP file server and deployed with your own arguments.\n\n|  **Overview** of a production cluster with NVIDIA, PPU, Ascend and Hygon GPUs |  **Scheduling analysis** : models taking turns on the same GPUs, card by card | \n|  **Monitoring** in production: latency, request shapes, concurrency and KV cache over 7 days |  **Usage analytics** : tokens by hour, API key and model | \n|  **PD disaggregation** on Ascend 910C: KV hand-off, latency breakdown, requests in flight |  **GPU health checks** , diagnosed by the AI assistant | \n|  **Network test** : HCCS card-to-card reachability matrix |  **Benchmark** report: throughput, TTFT, latency | \n|  **Model catalog** with launch specs per vendor and engine |  **Playground** against any deployment | \n\n``` php\nflowchart LR\n  user[\"Browser / CLI / MCP client\"] -->|\"Web UI, REST API\"| server\n  app[\"Apps (OpenAI SDK)\"] -->|\"/v1/* + API key\"| server\n  subgraph server[\"gpuworks server (one binary)\"]\n    direction TB\n    gw[\"OpenAI gateway\"]\n    ctl[\"Controller + scheduler<br/>memory estimate\"]\n    db[(\"SQLite<br/>metrics · logs · traces\")]\n  end\n  subgraph node1[\"GPU node\"]\n    a1[\"gpuworks-agent\"] --> e1[\"vLLM / SGLang<br/>Docker · nerdctl · runveil\"]\n  end\n  subgraph node2[\"Mac\"]\n    a2[\"gpuworks-agent\"] --> e2[\"vllm-metal / MLX / llama.cpp\"]\n  end\n  a1 -->|\"outbound WebSocket + yamux<br/>ctrl lane / data lane\"| server\n  a2 -->|\"outbound WebSocket\"| server\n```\n\n- The **server** is the control plane: REST API and web UI, the OpenAI-compatible gateway, a controller that\nreconciles desired state, the scheduler, and storage for metrics, logs and request traces.\n- An **agent** runs on each GPU node. It probes the accelerators, runs engines through Docker, nerdctl, the built-in\nrootless runtime (runveil) or as host processes on macOS, and reports metrics and logs.\n- Agents connect out to the server and multiplex everything over two lanes: `ctrl` for commands, metrics and\nterminals,`data` for inference traffic, logs and model/image transfer. Inference requests reach the engine\nthrough that tunnel, so nodes never expose ports.\n\nMore in [Architecture](https://github.com/flitai-io/gpuworks/blob/master/docs-en/explanation/architecture.md) and the [ADRs](https://github.com/flitai-io/gpuworks/blob/master/docs-en/adr/README.md).\n\n| I want to… | Go to | \n|---|---|\n| Get it running | [Getting started](https://github.com/flitai-io/gpuworks/blob/master/docs-en/tutorials/getting-started.md) ·[Install the server](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/install-server.md) ·[Connect nodes](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/connect-nodes.md) | \n| Fit more on my GPUs | [Choosing features](https://github.com/flitai-io/gpuworks/blob/master/docs-en/explanation/choosing-features.md) ·[dGPU](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/dgpu.md) ·[Sleep/wake](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/sleep-wake.md) ·[Two models on one GPU](https://github.com/flitai-io/gpuworks/blob/master/docs-en/tutorials/one-gpu-two-models.md) | \n| Serve bigger models or more traffic | [Multi-node inference](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/multi-node-inference.md) ·[PD disaggregation](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/pd-disaggregation.md) ·[Scale replicas](https://github.com/flitai-io/gpuworks/blob/master/docs-en/tutorials/scale-model-replicas.md) | \n| Measure and debug | [Benchmark](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/benchmark.md) ·[Monitoring and logs](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/monitor-and-logs.md) ·[Trace a slow request](https://github.com/flitai-io/gpuworks/blob/master/docs-en/tutorials/trace-slow-request.md) ·[Troubleshooting](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/troubleshooting.md) | \n| Integrate | [OpenAI gateway](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/openai-gateway.md) ·[REST API](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/api.md) ·[MCP](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/use-mcp.md) ·[CLI](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/cli.md) | \n| Run it in production | [Production checklist](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/deploy-production.md) ·[Offline](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/deploy-offline.md) ·[Upgrades](https://github.com/flitai-io/gpuworks/blob/master/docs-en/how-to/upgrade-and-compatibility.md) ·[Security](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/security.md) | \n\nEverything else: [documentation index](https://github.com/flitai-io/gpuworks/blob/master/docs-en/README.md).\n\nRequires Go 1.26 and Node.js.\n\n```\nmake build      # web UI + server + agents into ./bin\nmake demo       # throwaway server + a GPU-less agent: onboarding, heartbeat, reconnect\nmake test       # static checks + Go unit tests + UI tests\nmake test-l2    # multi-process tests with real binaries and fake GPUs\n```\n\nSee the [developer docs](https://github.com/flitai-io/gpuworks/blob/master/docs-en/dev/README.md) and [testing guide](https://github.com/flitai-io/gpuworks/blob/master/docs-en/dev/testing.md). How to file a\nreport, send a patch, or get a hardware combination into the support matrix is in\n[CONTRIBUTING.md](https://github.com/flitai-io/gpuworks/blob/master/CONTRIBUTING.md); contributor conventions are in [CLAUDE.md](https://github.com/flitai-io/gpuworks/blob/master/CLAUDE.md) (Chinese) and\ndomain terms in [CONTEXT.md](https://github.com/flitai-io/gpuworks/blob/master/CONTEXT.md). Security issues go to [SECURITY.md](https://github.com/flitai-io/gpuworks/blob/master/SECURITY.md) — never a\npublic issue.\n\nProduction installations send one anonymous usage report per day (hardware and engine mix, task counts, error\ncategories; no hostnames, model inputs or outputs, or any other sensitive information). Turn it off in\n**Settings → General** or with `usage_report: false` in `server.yaml`. Every field is listed in\n[User Experience Improvement Program](https://github.com/flitai-io/gpuworks/blob/master/docs-en/reference/usage-report.md).\n\nGPUWorks is licensed under the [Apache License 2.0](https://github.com/flitai-io/gpuworks/blob/master/LICENSE). Bundled third-party components keep their own\nlicenses; see [NOTICE](https://github.com/flitai-io/gpuworks/blob/master/NOTICE).", "url": "https://wpnews.pro/news/show-hn-gpuworks-a-40-mb-gpu-cluster-manager-for-vllm-sglang-no-k8s-needed", "canonical_source": "https://github.com/flitai-io/gpuworks", "published_at": "2026-11-04 16:09:54+00:00", "updated_at": "2026-11-04 16:20:46.400806+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools", "ai-agents", "developer-tools"], "entities": ["GPUWorks", "flitai-io", "vLLM", "SGLang", "NVIDIA", "AMD", "Huawei Ascend", "Alibaba Cloud"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-gpuworks-a-40-mb-gpu-cluster-manager-for-vllm-sglang-no-k8s-needed", "markdown": "https://wpnews.pro/news/show-hn-gpuworks-a-40-mb-gpu-cluster-manager-for-vllm-sglang-no-k8s-needed.md", "text": "https://wpnews.pro/news/show-hn-gpuworks-a-40-mb-gpu-cluster-manager-for-vllm-sglang-no-k8s-needed.txt", "jsonld": "https://wpnews.pro/news/show-hn-gpuworks-a-40-mb-gpu-cluster-manager-for-vllm-sglang-no-k8s-needed.jsonld"}}