{"slug": "dgx-model-manager", "title": "DGX Model Manager", "summary": "Nik Ghodki released DGX Model Manager (DGXMM), a self-hosted web app that deploys local LLMs on an NVIDIA DGX Spark (GB10, 128 GB unified memory) or similar single-GPU Linux machine as vLLM or llama.cpp containers behind one OpenAI-compatible LiteLLM endpoint with virtual keys. DGXMM previews every deploy against a unified-memory budget and does nothing until the user confirms a plan when a model does not fit, and it installs as a single container with one command (./install.sh), with optional flags --with-litellm and --with-decider. The project also adds an optional local decision model for 16 advisory features, on-demand models that start on first request and stop after idle minutes, and an upgrade note that clones made before the 2026-10-18 repository history rewrite must be moved with git fetch && git reset --hard origin/main.", "body_md": "DGXMM is a self-hosted web app for serving local LLMs on an NVIDIA DGX Spark (GB10, 128 GB unified\nmemory) or a similar single-GPU Linux box (see [supported hardware](https://nikghodki.github.io/dgxmodelmanager/getting-started/requirements/#supported-hardware)). It deploys the models on your disk as vLLM or llama.cpp\ncontainers, checks before every start that a model fits next to everything already running, and\nserves all of them through one OpenAI-compatible LiteLLM endpoint with virtual keys. An optional small\nlocal decision model adds advice (why a start failed, which config, engine, parser or draft model to use,\nwhether a deployment is healthy, when it could sleep), always after DGXMM's own rules and never without\nyour confirmation. It runs as a single container and installs with one command.\n\n**Documentation: [https://nikghodki.github.io/dgxmodelmanager/](https://nikghodki.github.io/dgxmodelmanager/)**\n\n- **Memory-safe deploys:** every deploy is previewed against a unified-memory budget; when a model does not\nfit, DGXMM offers plans and does nothing until you confirm one.\n- **vLLM and llama.cpp** , with switch and reconfigure that roll back automatically, and engines that start\none at a time.\n- **One LiteLLM endpoint** for every deployment, with virtual keys (your LiteLLM or a bundled one).\n- **On-demand models:** a model can start on its first request and stop again after a few idle minutes,\nmaking room or queueing when memory is short. It works through LiteLLM on DGXMM's Docker network, so new\ndeployments are always on unless you choose it (the deploy wizard preselects it when LiteLLM is set up).\nSee[on-demand models](https://nikghodki.github.io/dgxmodelmanager/user-guide/on-demand/) .\n- **An optional decision model** for 16 advisory features,**insights** from background checks,**schedules** proposed from traffic, and**⌘K Ask** in the command palette.\n- **Observability and administration:** live metrics, tuning advice, a benchmark, the process list,`admin` and`viewer` roles and an audit log.\n\nSee [all features](https://nikghodki.github.io/dgxmodelmanager/getting-started/features/) and the\n[screenshots](https://nikghodki.github.io/dgxmodelmanager/getting-started/features/#screenshots).\n\nOn Linux with an NVIDIA GPU, Docker with the Compose plugin and the NVIDIA runtime\n([requirements](https://nikghodki.github.io/dgxmodelmanager/getting-started/requirements/)):\n\n```\ngit clone https://github.com/nikghodki/dgxmodelmanager.git && cd dgxmodelmanager && ./install.sh\n```\n\nOpen the URL it prints and sign in as `admin` with the printed password. Add `--with-litellm` for a\nbundled LiteLLM endpoint, `--with-decider` for the decision model, or `--dry-run` to see first what it\nwould do. Next: [install options](https://nikghodki.github.io/dgxmodelmanager/getting-started/install/),\n[first login](https://nikghodki.github.io/dgxmodelmanager/getting-started/first-login/) and\n[upgrade](https://nikghodki.github.io/dgxmodelmanager/getting-started/upgrade/).\n\n**Cloned before 2026-10-18?** The repository history was rewritten, so `./install.sh upgrade` refuses\nto fast-forward an older clone. Move it once with `git fetch && git reset --hard origin/main`, then run\n`./install.sh` (your `.env`, `config.yaml`, `secrets/` and `data/` stay as they are; see\n[clones from before the history rewrite](https://nikghodki.github.io/dgxmodelmanager/getting-started/upgrade/#clones-from-before-the-2026-10-18-history-rewrite)).\n\n| Section | What it covers | \n|---|---|\n| [Getting started](https://nikghodki.github.io/dgxmodelmanager/getting-started/features/) | Features, requirements and supported hardware, install, first login, upgrade, uninstall | \n| [User guide](https://nikghodki.github.io/dgxmodelmanager/user-guide/dashboard/) | The UI page by page: dashboard, models and catalog, deploying, deployments, insights, schedules, ⌘K, the decision model, LiteLLM, processes, users, settings | \n| [Architecture](https://nikghodki.github.io/dgxmodelmanager/architecture/) | Components, request and data flow, memory budget, deployer, decision model, job engine, schedule runner, data model, frontend | \n| [Configuration](https://nikghodki.github.io/dgxmodelmanager/configuration/) | Every setting with its default | \n| [Operations](https://nikghodki.github.io/dgxmodelmanager/operations/) | Backups, logs, admin command line, troubleshooting | \n| [Security](https://nikghodki.github.io/dgxmodelmanager/security/overview/) | Threat model, hardening, security policy | \n| [Evaluation](https://nikghodki.github.io/dgxmodelmanager/decider-evaluation/) | The decision model's 16 features measured on real data | \n\nThe site is built from [`docs/`](https://github.com/nikghodki/dgxmodelmanager/blob/main/docs) (see [` docs/README.md`](https://github.com/nikghodki/dgxmodelmanager/blob/main/docs/README.md)).\n\nDGXMM mounts the Docker socket so it can start engine containers: **access to it is equivalent to root\non the host**, so only give the `admin` role to people you would trust with root, and keep the UI off\nthe public internet. See the [security overview](https://nikghodki.github.io/dgxmodelmanager/security/overview/).\n\n**Reporting a vulnerability:** please do not open a public issue. Use GitHub's private vulnerability\nreporting as described in [SECURITY.md](https://github.com/nikghodki/dgxmodelmanager/blob/main/SECURITY.md), which also lists what is in scope.\n\nBug reports and feature requests are welcome as\n[issues](https://github.com/nikghodki/dgxmodelmanager/issues/new/choose): the forms ask for what is needed\nto reproduce a problem (version, host and GPU, `./install.sh status`, logs). The development setup, the\ntest suites and the code layout are in [CONTRIBUTING.md](https://github.com/nikghodki/dgxmodelmanager/blob/main/CONTRIBUTING.md); the changelog is in\n[CHANGELOG.md](https://github.com/nikghodki/dgxmodelmanager/blob/main/CHANGELOG.md).\n\nMIT. See [LICENSE](https://github.com/nikghodki/dgxmodelmanager/blob/main/LICENSE).", "url": "https://wpnews.pro/news/dgx-model-manager", "canonical_source": "https://github.com/nikghodki/dgxmodelmanager", "published_at": "2026-11-02 21:34:25+00:00", "updated_at": "2026-11-02 21:51:15.387564+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "developer-tools"], "entities": ["DGX Model Manager", "Nik Ghodki", "NVIDIA DGX Spark", "GB10", "vLLM", "llama.cpp", "LiteLLM", "Docker"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/dgx-model-manager", "markdown": "https://wpnews.pro/news/dgx-model-manager.md", "text": "https://wpnews.pro/news/dgx-model-manager.txt", "jsonld": "https://wpnews.pro/news/dgx-model-manager.jsonld"}}