{"slug": "unload-all-llama-cpp-router-models-without-restarting", "title": "Unload All llama.cpp Router Models Without Restarting", "summary": "While llama.cpp's router mode allows loading and unloading individual models via HTTP API calls to `/models/unload`, there is no built-in \"unload all\" endpoint. The recommended approach for unloading all models is to iterate through the list of loaded models from the `/models` endpoint and send a separate unload request for each one, which provides safer and more explicit operational control.", "body_md": "[llama.cpp router mode](https://www.glukhov.org/llm-hosting/llama-cpp/llama-server-router-mode/) is one of the most useful changes to `llama-server`\n\nin years. It finally gives local LLM operators something close to the model management experience people expect from Ollama, while keeping the raw performance and low-level control that make [llama.cpp](https://www.glukhov.org/llm-hosting/llama-cpp/) worth using in the first place.\n\nBut there is one sharp edge: unloading everything is not a single magic button in the HTTP API.\n\nThe router can list models. It can load a model. It can unload a model. It can evict the least recently used model when `--models-max`\n\nis reached. What it does not currently document as a first-class endpoint is a universal `unload all models now`\n\ncall.\n\nThat is not a real blocker. The correct pattern is simple, explicit, and scriptable:\n\n- Ask the router which models exist.\n- Filter the models whose status is\n`loaded`\n\n. - Call\n`/models/unload`\n\nonce per loaded model.\n\nThis is the approach I recommend for serious [local LLM workflows](https://www.glukhov.org/llm-hosting/). It is boring, visible, and easy to debug. That is exactly what you want when your goal is to free VRAM without restarting the whole inference service.\n\n## What llama.cpp router mode actually does\n\nIn classic `llama-server`\n\nusage, you start one server with one model:\n\n```\nllama-server \\\n  --model ./models/qwen3-8b.gguf \\\n  --port 8080\n```\n\nRouter mode changes that model. Instead of binding the server to one GGUF file, the router becomes a coordinator for multiple models. It can discover models from a cache or from a models directory, load them on demand, route requests to the correct model, and unload models when needed.\n\nA typical router-mode startup looks like this:\n\n```\nllama-server \\\n  --models-dir ./models \\\n  --models-max 4 \\\n  --port 8080\n```\n\nThe important option here is `--models-max`\n\n. It controls how many models may be loaded at the same time. If the limit is reached, llama.cpp can evict the least recently used model. That is useful, but it is not a substitute for a deliberate unload operation. LRU eviction is reactive. An unload script is operational control.\n\nMy opinionated take: if you run local models for real work, you should treat router mode like an inference process manager, not like a toy chat server. Explicit lifecycle operations matter.\n\n## The model management endpoints you need\n\nThe main endpoint for discovery is:\n\n```\ncurl -s http://localhost:8080/models | jq\n```\n\nThat endpoint returns the models known to the router and their current lifecycle status. The exact JSON shape can vary slightly between builds, so inspect your own response before writing automation.\n\nA common response shape looks like this:\n\n```\n{\n  \"data\": [\n    {\n      \"id\": \"qwen3-8b\",\n      \"status\": \"loaded\"\n    },\n    {\n      \"id\": \"llama-3.2-3b\",\n      \"status\": \"unloaded\"\n    }\n  ]\n}\n```\n\nTo unload one model, call:\n\n```\ncurl -s -X POST http://localhost:8080/models/unload \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"qwen3-8b\"}' \\\n  | jq\n```\n\nThat is the primitive operation. Everything else in this article builds on that.\n\n## There is no documented unload all endpoint\n\nThis is the part that trips people up.\n\nYou might expect something like this:\n\n```\ncurl -X POST http://localhost:8080/models/unload-all\n```\n\nDo not build around that assumption. The documented operation is per model. You pass a model identifier to `/models/unload`\n\n, and llama.cpp unloads that one model.\n\nThis is not necessarily bad API design. A per-model operation is safer. It makes the caller decide what should be unloaded. It also avoids surprising production behavior where one admin request accidentally kills every warm model being used by other clients.\n\nFor a workstation, an unload-all shortcut would be convenient. For a multi-user inference box, explicit loops are better.\n\n## Unload one model first\n\nBefore automating anything, test the exact model identifier your router expects.\n\nFirst list models:\n\n```\ncurl -s http://localhost:8080/models | jq\n```\n\nPick one loaded model from the output, then unload it:\n\n```\ncurl -s -X POST http://localhost:8080/models/unload \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"qwen3-8b\"}' \\\n  | jq\n```\n\nCheck the model list again:\n\n```\ncurl -s http://localhost:8080/models | jq\n```\n\nIf the model status changes to `unloaded`\n\n, your endpoint, port, and model identifier are correct.\n\nIf it does not work, do not guess. Inspect the JSON. Router aliases, GGUF filenames, and model IDs are often not the same string.\n\n## Unload all loaded models with curl and jq\n\nOnce the single-model unload works, the unload-all pattern is just a shell loop.\n\nUse this when your `/models`\n\nresponse has `.data[].id`\n\nand `.data[].status`\n\n:\n\n```\ncurl -s http://localhost:8080/models \\\n| jq -r '.data[] | select(.status == \"loaded\") | .id' \\\n| while IFS= read -r model; do\n    echo \"Unloading: $model\"\n    curl -s -X POST http://localhost:8080/models/unload \\\n      -H \"Content-Type: application/json\" \\\n      -d \"{\\\"model\\\":\\\"$model\\\"}\" \\\n      | jq\n  done\n```\n\nThis is the whole trick. It is not glamorous, but it is the right shape for an admin operation:\n\n- It only unloads models that are actually loaded.\n- It prints what it is doing.\n- It fails model by model instead of hiding everything behind one opaque action.\n- It works from cron, systemd hooks, SSH, or CI jobs.\n\n## A reusable script for production use\n\nFor anything you run more than twice, stop pasting one-liners. Save a script.\n\nCreate `llama-router-unload-all.sh`\n\n:\n\n``` bash\n#!/usr/bin/env bash\nset -euo pipefail\n\nLLAMA_SERVER_URL=\"${LLAMA_SERVER_URL:-http://localhost:8080}\"\n\nmodels_json=\"$(curl -fsS \"$LLAMA_SERVER_URL/models\")\"\n\nloaded_models=\"$(printf '%s' \"$models_json\" \\\n  | jq -r '.data[] | select(.status == \"loaded\") | .id')\"\n\nif [ -z \"$loaded_models\" ]; then\n  echo \"No loaded models found.\"\n  exit 0\nfi\n\nprintf '%s\\n' \"$loaded_models\" | while IFS= read -r model; do\n  [ -z \"$model\" ] && continue\n\n  echo \"Unloading: $model\"\n\n  curl -fsS -X POST \"$LLAMA_SERVER_URL/models/unload\" \\\n    -H \"Content-Type: application/json\" \\\n    -d \"{\\\"model\\\":\\\"$model\\\"}\" \\\n    | jq\n\ndone\n\necho \"Done. Current model state:\"\ncurl -fsS \"$LLAMA_SERVER_URL/models\" | jq\n```\n\nMake it executable:\n\n```\nchmod +x llama-router-unload-all.sh\n```\n\nRun it against the default local server:\n\n```\n./llama-router-unload-all.sh\n```\n\nRun it against another host:\n\n```\nLLAMA_SERVER_URL=http://192.168.1.50:8080 ./llama-router-unload-all.sh\n```\n\nThis is the version I would actually keep in a tools directory. It uses `curl -f`\n\nso HTTP errors fail the script, and it lets you override the server URL without editing the file.\n\n## Adapting the script to your JSON shape\n\nDo not blindly assume every llama.cpp build returns the exact same fields forever. Router mode is still evolving, and your build may expose a slightly different JSON shape.\n\nStart by inspecting the response:\n\n```\ncurl -s http://localhost:8080/models | jq\n```\n\nThe script uses this filter:\n\n```\njq -r '.data[] | select(.status == \"loaded\") | .id'\n```\n\nIf your model identifier is in `.name`\n\n, change it to:\n\n```\njq -r '.data[] | select(.status == \"loaded\") | .name'\n```\n\nIf your status field uses another value, adjust the filter accordingly. The principle is what matters: select loaded models, extract the identifier accepted by `/models/unload`\n\n, then call unload for each one.\n\n## Why models may load again after you unload them\n\nThis is the most common source of confusion.\n\nRouter mode supports on-demand loading. If a client sends a chat completion request for a model that is currently unloaded, the router may load it again automatically.\n\nThat means this sequence is possible:\n\n- You unload every model.\n- Open WebUI, a test script, or an agent sends a request.\n- llama.cpp loads the requested model again.\n- You think unload failed, but it did not.\n\nThe fix is operational, not technical. Stop client traffic first if your goal is to keep VRAM free.\n\nFor example:\n\n- Stop benchmark scripts.\n- Pause agents and cron jobs.\n- Close or disconnect Open WebUI sessions.\n- Disable health checks that accidentally perform real model requests.\n\nUnloading is not a firewall. If clients keep asking for models, router mode is doing its job by serving them.\n\n## Open WebUI and the Eject button\n\n[Open WebUI](https://www.glukhov.org/llm-hosting/llm-frontends/open-webui-overview-quickstart-and-alternatives/) can integrate with llama.cpp model unload support. When the provider is configured as `llama.cpp`\n\n, Open WebUI can show loaded-model state and expose an Eject action for admins.\n\nUnder the hood, that action calls Open WebUI's own unload API, which then calls llama.cpp's `/models/unload`\n\nendpoint on the configured connection.\n\nThat is nice for manual operation, but I would still keep the shell script. A UI button is convenient. A script is auditable, repeatable, and usable on a headless box at 2 AM.\n\n## When to use unload all\n\nUnloading every loaded model is useful when you want to:\n\n- Free GPU memory before starting a larger model.\n- Reset a development box without restarting\n`llama-server`\n\n. - Prepare for a benchmark run with a clean memory state.\n- Drain local inference workloads before maintenance.\n- Recover from a messy session where too many models were warmed.\n\nIt is not the right tool when active users are depending on warm models. In that case, tune `--models-max`\n\n, use deliberate routing, and let LRU eviction do part of the work. If you need smarter timeout-based unloading with per-model lifecycle control, [llama-swap](https://www.glukhov.org/llm-hosting/llama-swap/) is a purpose-built proxy that layers exactly that on top of any `llama-server`\n\nsetup.\n\nMy rule is simple: use LRU for normal pressure, use explicit unload for operator intent.\n\n## Troubleshooting\n\n### The models endpoint returns 404\n\nYou may not be running a router-capable build, or you may be calling the wrong port.\n\nCheck the server process and available options:\n\n```\nllama-server --help | grep -i models\n```\n\nThen test both endpoints:\n\n```\ncurl -s http://localhost:8080/models | jq\ncurl -s http://localhost:8080/v1/models | jq\n```\n\nThe `/v1/models`\n\nendpoint is the OpenAI-compatible model list. The `/models`\n\nendpoint is the router model-management endpoint. They are related, but they are not the same thing.\n\n### jq is not installed\n\nInstall it before scripting JSON parsing.\n\nOn Ubuntu or Debian:\n\n```\nsudo apt-get update\nsudo apt-get install jq\n```\n\nOn macOS with Homebrew:\n\n```\nbrew install jq\n```\n\n### The unload call returns an error\n\nMost failures come from passing the wrong model identifier. Use the exact identifier returned by `/models`\n\n, not the filename you think should work.\n\nAlso check whether your model name contains quotes, slashes, or spaces. The script above handles normal strings well, but unusual names may require more careful JSON construction.\n\nFor maximum safety, you can build the POST body with `jq`\n\n:\n\n```\njq -n --arg model \"$model\" '{model: $model}'\n```\n\nA more defensive unload loop would use that body instead of hand-escaped JSON.\n\n### VRAM is not freed immediately\n\nFirst confirm the model status changed. Then check whether another request reloaded it. Also remember that GPU memory tools can lag or report allocator behavior rather than instant application-level intent.\n\nThe practical test is simple: stop traffic, unload models, list model status, then inspect GPU memory. For measured VRAM usage across model sizes and context windows on llama.cpp, the [16 GB VRAM llama.cpp benchmarks](https://www.glukhov.org/llm-performance/benchmarks/best-llm-on-16gb-vram-gpu/) give concrete figures to sanity-check against.\n\n## A safer JSON body version\n\nIf your model identifiers contain unusual characters, use `jq`\n\nto generate the JSON request body:\n\n```\ncurl -s http://localhost:8080/models \\\n| jq -r '.data[] | select(.status == \"loaded\") | .id' \\\n| while IFS= read -r model; do\n    echo \"Unloading: $model\"\n    body=\"$(jq -n --arg model \"$model\" '{model: $model}')\"\n    curl -s -X POST http://localhost:8080/models/unload \\\n      -H \"Content-Type: application/json\" \\\n      -d \"$body\" \\\n      | jq\n  done\n```\n\nThis is the version to use if your models are named with repository-style identifiers, custom aliases, or paths.\n\n## Final take\n\nllama.cpp router mode is a big step forward for local LLM operations. It gives you dynamic loading, model switching, and memory-aware eviction without giving up the directness of `llama-server`\n\n.\n\nBut do not wait for a perfect unload-all endpoint. The clean solution already exists: list loaded models and unload them one by one.\n\nThat pattern is explicit. It is scriptable. It works over SSH. It plays nicely with Open WebUI. And most importantly, it frees VRAM without restarting the router.\n\nFor local AI infrastructure, that is exactly the kind of boring control surface you want.", "url": "https://wpnews.pro/news/unload-all-llama-cpp-router-models-without-restarting", "canonical_source": "https://dev.to/rosgluk/unload-all-llamacpp-router-models-without-restarting-8dj", "published_at": "2026-05-20 01:00:03+00:00", "updated_at": "2026-05-20 01:34:36.803357+00:00", "lang": "en", "topics": ["large-language-models", "open-source", "developer-tools"], "entities": ["llama.cpp", "Ollama", "llama-server"], "alternates": {"html": "https://wpnews.pro/news/unload-all-llama-cpp-router-models-without-restarting", "markdown": "https://wpnews.pro/news/unload-all-llama-cpp-router-models-without-restarting.md", "text": "https://wpnews.pro/news/unload-all-llama-cpp-router-models-without-restarting.txt", "jsonld": "https://wpnews.pro/news/unload-all-llama-cpp-router-models-without-restarting.jsonld"}}