Run Hermes Fully Locally with QVAC

A developer guide explains how to run the Hermes agent fully locally using QVAC, an OpenAI-compatible server, enabling offline inference, persistent memory, and tool execution without cloud dependencies. The setup requires a Unix-like environment, Node.js, Python, and a local model, with an acceptance test that verifies generation, tool calling, durable memory, and offline operation.

Run Hermes Fully Locally with QVAC | Agent Lab Journal AL Agent Lab Journal Guides Glossary LOCAL AI AGENTS · PRACTICAL DEPLOYMENT Level: advanced Reading time: 45 minutes Result: local Hermes + QVAC + offline verification By the end of this guide, you will have a personal Hermes Agent that sends inference requests only to a model running on your machine, preserves memory between sessions, invokes a restricted set of local tools, and completes a repeatable task after the internet connection is disabled. No cloud API key is required during operation, although you must download the software and model files beforehand. Hermes Agent is the agent layer around a language model. It manages conversations, state, memory, skills, tool descriptions, and the loop that connects a model request to an actual action. QVAC is responsible for running the model locally and exposing it through an OpenAI-compatible HTTP interface. User │ ▼ Hermes Agent ├── local sessions ├── persistent memory ├── skills and policies └── restricted tools │ │ HTTP over 127.0.0.1 ▼ QVAC OpenAI server │ ▼ local model │ ▼ CPU / GPU / RAM The roles must remain separate. The large language model produces a response or requests a function. Hermes controls the agent loop, memory, permissions, and function execution. QVAC loads the model weights, applies the model's chat template, and performs local inference. In this guide, “fully local” has a precise and limited meaning: Hermes and QVAC run on one computer, or inside an isolated local environment; model requests are sent only to 127.0.0.1; sessions, memory, skills, configuration, and logs remain on local storage; the tools used by the acceptance test require no external services; after dependencies and weights have been downloaded, the scenario passes without internet access. A local model does not automatically create a secure system. A tool can still execute a command, read an unrelated file, or contact an external address. Offline operation reduces one part of the attack surface; it does not replace operating-system permissions, path validation, or approval controls. A greeting is too weak to prove that an agent works. Our acceptance case therefore uses a small project directory and tests four independent layers: generation, tool calling, durable memory, and the absence of a required cloud route. Hermes must: read a local meeting note; extract the project, decision, and next action; write a Markdown summary inside an allowed output directory; remember a harmless formatting preference; recover that preference in a new session; repeat a new file task after external networking is disabled. Create a dedicated laboratory directory. Do not begin with your home directory, a repository containing credentials, or a cloud-synchronized folder. mkdir -p "$PWD/hermes-local-lab/inbox" mkdir -p "$PWD/hermes-local-lab/outbox" printf '%s\n' \ 'Project: local assistant.' \ 'Decision: store final notes as Markdown.' \ 'Next action: verify operation without a network.' \ "$PWD/hermes-local-lab/inbox/meeting.txt" export LAB DIR="$PWD/hermes-local-lab" find "$LAB DIR" -maxdepth 2 -type f -print printf 'Laboratory directory: %s\n' "$LAB DIR" Keep this terminal open or record the absolute value of LAB DIR. Agent prompts should use an absolute path rather than relying on an uncertain working directory. You need a Unix-like environment, Node.js and npm for QVAC, a Python environment supported by the Hermes release you install, Git, curl, and enough disk space for the selected model. On Windows, use WSL2 unless the current release explicitly documents another supported route. Before changing the machine, record the software and available resources: uname -a node --version npm --version python3 --version git --version curl --version | head -n 1 df -h . free -h 2 /dev/null || true CLI flags, model identifiers, and configuration formats can change between releases. Treat every command below as a configuration for the installed version, not as permission to ignore its built-in help. If a flag is rejected, stop and inspect --help instead of guessing a replacement. After installing QVAC, use its diagnostics to identify the available compute backend: qvac doctor A supported GPU backend can reduce latency, while CPU execution may still be adequate for a small model. The actual result depends on the machine, model build, context length, and runtime version; this article does not claim universal speed or memory measurements. Begin with a model variant using 4-bit quantization. Quantization reduces the storage and memory cost of weights, but it does not eliminate runtime buffers or the memory consumed by the context cache. The QVAC model registry available in your installed release is the source of truth. List or inspect that registry before copying a model identifier into the configuration. If the following identifiers are not present in your release, select the corresponding supported instruct model and use its exact identifier. Deployment profile Example model class Use QVAC identifier used below Installation check Qwen 4B, Q4 HTTP route and simple functions QWEN3 5 4B MULTIMODAL Q4 K M Balanced laboratory Qwen 9B, Q4 More reliable argument generation QWEN3 5 9B MULTIMODAL Q4 K M Larger agent workload Qwen 35B-A3B MoE, Q4 Longer multi-step work on capable hardware QWEN3 6 35B A3B MULTIMODAL Q4 K M A mixture-of-experts model activates only part of its expert network for each token, but its weights and working data still have to fit the runtime's memory strategy. Leave capacity for the operating system, Hermes, the context, and the KV cache. The smallest model is useful for validating the route, but it may emit invalid function arguments or skip steps. A larger model can improve agent behavior, yet no model size guarantees correct tool use. First complete the whole test with a model that fits comfortably; compare alternatives only after the infrastructure is stable. Internet access is required at this stage to retrieve the package and, later, the model weights. npm install -g @qvac/cli command -v qvac qvac --help qvac doctor If global npm installation asks for administrator privileges, do not solve it by piping an unreviewed process into sudo. Configure a user-owned npm prefix or use a managed Node.js installation. Confirm that the discovered qvac binary belongs to the installation you intended. Create a separate QVAC project directory: mkdir -p "$PWD/qvac-hermes" cd "$PWD/qvac-hermes" pwd The HTTP server loads models declared under serve.models. The object key becomes the model name visible to clients. Using a stable alias keeps Hermes independent of the underlying registry identifier. Create qvac.config.json in the current directory: { "serve": { "models": { "hermes-local": { "model": "QWEN3 6 35B A3B MULTIMODAL Q4 K M", "default": true, "preload": true, "config": { "ctx size": 65536, "tools": true } } } } } For a smaller model, change only the model value to an identifier verified in your installed QVAC registry, for example: "QWEN3 5 4B MULTIMODAL Q4 K M" or: "QWEN3 5 9B MULTIMODAL Q4 K M" ctx size is the maximum context the runtime attempts to hold. Hermes may send a large system instruction plus multiple tool schemas before the user's task begins. A small context can overflow early, while a large token limit increases KV-cache memory. If 65,536 tokens do not fit, lower the value, expose fewer tools, and configure Hermes with the same effective limit. Do not advertise a larger context to Hermes than QVAC can actually serve. The tools option enables function handling for the selected model. Without it, ordinary chat can succeed while the agent returns prose instead of a structured call. Bind QVAC to the loopback interface so that other devices cannot reach the port through the LAN. cd /absolute/path/to/qvac-hermes qvac serve openai \ --config "$PWD/qvac.config.json" \ --host 127.0.0.1 \ --port 11434 \ --verbose Keep this process in a dedicated terminal. The first start may download weights before loading them into memory. Wait until server initialization has completed. From another terminal, inspect the model endpoint: curl --fail --silent --show-error \ http://127.0.0.1:11434/v1/models The returned JSON should contain the client-facing identifier hermes-local. An empty list usually means the model is not ready. A 404 indicates a route, configuration, or CLI-version mismatch. Now send a minimal completion: curl --fail --silent --show-error \ http://127.0.0.1:11434/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "hermes-local", "messages": { "role": "user", "content": "Reply with exactly one word: ready" } , "temperature": 0, "max tokens": 16 }' Validate the response structure rather than punctuation or capitalization. It must contain a choices array with an assistant message. If HTTP succeeds but content is empty, inspect the verbose QVAC log and verify that the chosen registry entry is a chat-capable instruct model. This test separates QVAC and model behavior from the agent runtime. The described function performs no action; the request only asks the model to return a structured intention. curl --fail --silent --show-error \ http://127.0.0.1:11434/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "hermes-local", "messages": { "role": "user", "content": "Find the size of /lab/inbox/meeting.txt. Do not guess; use the function." } , "tools": { "type": "function", "function": { "name": "get file size", "description": "Return the size of an allowed local file", "parameters": { "type": "object", "properties": { "path": { "type": "string", "description": "Absolute path to the file" } }, "required": "path" , "additionalProperties": false } } } , "tool choice": "auto", "temperature": 0, "max tokens": 256 }' A successful result is not an alleged byte count. It is an item in message.tool calls whose function name is get file size and whose arguments value contains valid JSON with the requested path. Fail the test if: the model says that it will call a function but returns no tool calls; the function name is changed; arguments cannot be parsed as JSON; the model invents a file size; the requested path is replaced with another path. Do not install or debug Hermes until this boundary works. Confirm that "tools": true is active, restart QVAC after configuration changes, and test a more capable model if necessary. Successful chat completion is not evidence of successful function calling. Review remote installation scripts before executing them. One cautious sequence is: curl --fail --silent --show-error \ https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \ -o /tmp/hermes-install.sh less /tmp/hermes-install.sh bash /tmp/hermes-install.sh If your environment prohibits remote installation scripts, follow the manual installation procedure for the release you selected. Avoid mixing system Python packages with unrelated virtual environments. Open a new terminal if the installer changed PATH, then verify the command: command -v hermes hermes --help Hermes commonly stores user state under ~/.hermes/. That directory can contain sessions, memory, skills, logs, and provider configuration. Do not publish it or commit it to Git. find "$HOME/.hermes" -maxdepth 2 -type d -print 2 /dev/null Use Hermes' interactive model configuration so that the installed release writes its own supported configuration structure: hermes model Select a custom OpenAI-compatible endpoint and enter: API base URL: http://127.0.0.1:11434/v1 API key: local-qvac Model name: hermes-local Context length: 65536 If QVAC was started without an API-key requirement, local-qvac is merely a non-empty local placeholder for clients that require a value. It is not a cloud credential and must not be copied from a real account. The corresponding configuration is conceptually equivalent to: model: default: hermes-local provider: custom base url: http://127.0.0.1:11434/v1 The exact YAML structure can change. Do not replace the entire Hermes configuration with this short example: the existing file may also contain memory, channel, and skill settings. Confirm the saved route without printing credential fields: grep -nE 'provider:|base url:|default:' \ "$HOME/.hermes/config.yaml" The base URL should begin with http://127.0.0.1:11434/v1 http://127.0.0.1:11434/v1 , and no cloud provider domain should appear in the active model configuration. hermes Begin with a generation-only request: State the configured model name and base URL. Do not use tools and do not access the network. The model's answer is not authoritative. It can repeat prompt text or hallucinate configuration. The sources of truth are the Hermes configuration, QVAC request logs, and network observation. Next, establish an explicit working boundary: Working directory: /absolute/path/hermes-local-lab You may read only from inbox. You may write only to outbox. Do not modify source files. Do not make network requests. First present a plan, then wait for my next instruction. If the agent immediately changes files, stop it. A prompt influences model behavior but does not create an isolation boundary. File permissions and tool-level validation must enforce the same policy. The precise skill and permission controls depend on the installed Hermes version, but the security rule does not: expose the smallest directory and the smallest set of operations required by the task. The laboratory needs only these capabilities: read a file from the approved input directory; list files inside the laboratory; create a new file in outbox; read and write Hermes' local memory through its dedicated interface; request approval for anything outside those boundaries. Do not enable a browser, email, messaging, unrestricted shell access, or external MCP servers for this test. If Hermes runs in a container, expose individual directories rather than the entire home directory: php /host/hermes-local-lab/inbox - /lab/inbox:ro /host/hermes-local-lab/outbox - /lab/outbox:rw /host/hermes-state - /home/hermes/.hermes:rw For a native process, use a dedicated operating-system account or another isolation mechanism. Natural-language instructions remain useful, but the operating system must reject access that the agent should not have. An allowlist must compare normalized absolute paths. A raw string-prefix check can be bypassed with .., alternate separators, or symbolic links. requested path │ ▼ resolve with realpath │ ├── inside inbox → read ├── inside outbox → read and create └── elsewhere → deny Give Hermes this task after replacing with the absolute laboratory path: Read: <LAB DIR /inbox/meeting.txt Create: <LAB DIR /outbox/meeting-summary.md The new file must contain exactly these three sections: Project Decision Next action Do not modify the source file. After writing the result, read the new file again and report its path and section names. Do not accept the final chat message as evidence. Inspect the filesystem independently: test -f "$LAB DIR/outbox/meeting-summary.md" test -f "$LAB DIR/inbox/meeting.txt" sed -n '1,80p' "$LAB DIR/outbox/meeting-summary.md" find "$LAB DIR" -maxdepth 2 -type f \ -printf '%p\n' 2 /dev/null || \ find "$LAB DIR" -maxdepth 2 -type f -print The loop passes only if: the agent reads the source instead of inventing its contents; the write call targets a path inside outbox; exactly the expected output file is created; the input file remains unchanged; the agent rereads the output before completing; QVAC logs show local model requests only. Record the source hash before and after a repeated run: sha256sum "$LAB DIR/inbox/meeting.txt" A matching hash proves that this particular source file did not change. It does not prove that no file outside the laboratory was touched, which is why operating-system isolation remains important. Agent memory must be tested across independent sessions. A correct answer inside the original conversation may come from the active context rather than durable storage. In the first session, store a harmless synthetic preference: Remember this durable preference for the test project: final notes must contain no more than three sections. Do not store file paths, personal data, or secrets. Confirm exactly what you sent to persistent memory. Exit Hermes through its normal exit command. Start a new session and ask: What formatting limit did I set for final notes? If persistent memory does not contain the answer, say so plainly. The test passes when the new session recovers the three-section rule from persistent storage. Do not include the answer in the retrieval question. You can inspect modification times without printing all memory contents: find "$HOME/.hermes" -type f \ -printf '%TY-%Tm-%Td %TH:%TM %p\n' 2 /dev/null \ | sort \ | tail -n 30 On macOS, use its supported find options or file metadata tools. Do not infer the memory format from a directory name; internal storage can change between Hermes versions. Run the offline test only after the complete model has been downloaded. Otherwise, a startup failure merely proves that required files were never cached. Before disconnecting, confirm that: QVAC starts without another download; /v1/models returns hermes-local; Hermes sends completions through QVAC; the restricted tools can read and write laboratory files; memory survives a Hermes restart. Stop Hermes and QVAC. Disable Wi-Fi and wired networking, or use a temporary network namespace or firewall policy that denies external connections while preserving loopback. Do not block 127.0.0.1, because Hermes needs it to reach QVAC. Cold-start QVAC again: cd /absolute/path/to/qvac-hermes qvac serve openai \ --config "$PWD/qvac.config.json" \ --host 127.0.0.1 \ --port 11434 \ --verbose In a second terminal: curl --fail http://127.0.0.1:11434/v1/models hermes Give Hermes a new task that was not completed in the earlier session: Create offline-check.md in the allowed outbox. Write this exact line: OFFLINE ROUTE OK Add a short summary of inbox/meeting.txt. Do not use a browser, search, URL, or external source. After writing the file, read it again. Verify the artifact independently: test -f "$LAB DIR/outbox/offline-check.md" grep -F 'OFFLINE ROUTE OK' \ "$LAB DIR/outbox/offline-check.md" sed -n '1,80p' \ "$LAB DIR/outbox/offline-check.md" Confirm that QVAC listens only on loopback: ss -ltnp 2 /dev/null | grep ':11434' || \ lsof -nP -iTCP:11434 -sTCP:LISTEN The listening address should be 127.0.0.1:11434, not 0.0.0.0:11434. Disabling Wi-Fi alone may leave VPN, virtual-machine, container, or wired interfaces active. For a strict test, observe outbound connections or isolate the processes so that only loopback traffic is permitted. Acceptance criterion: after a cold restart with no external network, QVAC loads already downloaded weights, Hermes connects to 127.0.0.1, creates and rereads the requested file through a restricted tool, and recovers the synthetic preference from local memory. A small script can verify the HTTP boundary before every Hermes session. It checks model discovery and an ordinary completion; the earlier raw function test is still required. bash /usr/bin/env bash set -euo pipefail base url="${QVAC BASE URL:-http://127.0.0.1:11434/v1}" model="${QVAC MODEL:-hermes-local}" models json="$ curl --fail --silent --show-error \ "$base url/models" " printf '%s' "$models json" | grep -F "\"id\":\"$model\"" /dev/null || printf '%s' "$models json" | grep -F "\"id\": \"$model\"" /dev/null response json="$ curl --fail --silent --show-error \ "$base url/chat/completions" \ -H 'Content-Type: application/json' \ -d "{ \"model\": \"$model\", \"messages\": {\"role\": \"user\", \"content\": \"Reply: LOCAL OK\"} , \"temperature\": 0, \"max tokens\": 32 }" " printf '%s' "$response json" | grep -F 'choices' /dev/null printf '%s\n' 'QVAC HTTP OK' For a maintained test, parse JSON with jq and validate exact fields. The two grep patterns are only a minimal portable check and do not validate the complete response. Symptom Likely layer First check qvac: command not found CLI installation npm prefix -g and PATH Server exits while loading Model or memory qvac doctor, free RAM, model identifier /v1/models returns 404 HTTP configuration CLI help, config path, and server mode Model not found Client alias hermes-local must match in QVAC and Hermes Chat works but tools do not Function calling "tools": true and the raw curl function test Hermes contacts a cloud host Provider routing provider: custom and local base url A new session forgets the rule Persistent memory State path, write permissions, and memory-save operation Offline startup attempts a download Model cache Whether every model file completed downloading Responses are truncated Context or output limit ctx size, active tool count, Hermes limit The machine swaps heavily Insufficient RAM Smaller model, smaller context, fewer preloaded models The expected value in this setup includes /v1: http://127.0.0.1:11434/v1 Some clients append /v1 themselves. A duplicated path will appear in QVAC's HTTP log. Change one parameter at a time so that an address problem is not confused with a model or provider problem. Hermes sends the key declared under serve.models: hermes-local. The longer QVAC registry identifier tells the server which weights to resolve; it is not necessarily the name that the client should send. A configured value of 65,536 does not guarantee that the machine can allocate it efficiently. Reduce the context, expose only necessary tools, restart QVAC, and repeat the raw HTTP tests. Keep the effective limit consistent between server and client. Text that resembles JSON is not a valid function call. Inspect the raw response. If message.tool calls is absent, Hermes should not execute the text. Repairing this output with regular expressions can transform malformed model prose into unintended actions. Set a maximum number of agent steps and preserve the full message sequence. Verify that Hermes returns each tool result to the model with the expected role and call identifier. Give the task an explicit stopping condition: after one successful write and one verification read, finish. A long system prompt and multiple schemas require prompt processing before output begins. Watch QVAC logs and resource use. If generation never starts, reduce the model, context, or active tool set. Do not assign a universal timeout based on one machine. Store short, atomic, unambiguous preferences rather than full conversational transcripts. Test retrieval in a fresh session without embedding the answer in the question. Never place passwords, API tokens, or confidential document content in persistent memory. A disabled browser or Wi-Fi icon does not prove network isolation. Check listening sockets and outbound connections. VPNs, container bridges, virtual interfaces, and wired adapters can remain available. Use a firewall or network namespace when the proof matters. A local agent inherits the filesystem permissions of the account that runs it. The production boundary must therefore limit the consequences of an incorrect tool call, not merely its internet access. Bind QVAC to 127.0.0.1 unless remote access is explicitly required. Do not expose the complete home directory to Hermes. Mount or permission input data as read-only. Use a separate directory for generated artifacts. Prefer narrow file functions over a general-purpose shell. Require an approval gate for deletion, sending, publication, or access outside the working directory. Resolve symbolic links before applying path policy. Keep real credentials out of laboratory configuration and memory. Record the model identifier, package versions, and configuration hash. Back up ~/.hermes before upgrading or changing memory storage. sha256sum qvac.config.json npm list -g @qvac/cli --depth=0 hermes --help | head -n 5 Hermes releases may expose version information through different commands. Record only information returned by the installed build; do not invent a version string when no version flag is available. A valid benchmark gives every model the same prompts, function schemas, permissions, context budget, and acceptance checks. Do not compare a 4B model with one small function against a 35B model loaded with every Hermes skill. Use five local tasks: read one file and return an exact fact; create a file that follows a three-section schema; select the correct function from two similar functions; repair an argument after the path allowlist rejects it; stop after receiving the successful result instead of calling the function again. For each run, record: the model registry identifier and context size; the complete sanitized input; the ordered sequence of tool calls; whether names and JSON arguments are valid; the number of unnecessary calls; whether every path remained inside the boundary; whether the filesystem result passed deterministic checks; cold-start and warmed-run times measured on your own machine. A fast, persuasive answer that does not create the required file is a failed run. A model that reaches the result but repeats a dangerous call also fails. Determine the number of repetitions and pass criteria before reviewing results, and preserve raw logs. This guide intentionally reports no success percentages or latency figures because none can be honestly transferred to the reader without the original hardware, tasks, versions, and logs. Model quality. A compact local model may select tools, plan long chains, and recover from errors less reliably than a larger model. Hardware memory. Weight files are only part of consumption. The runtime needs working buffers, KV cache, Hermes memory, and operating-system capacity. Latency. Large system prompts and many function schemas make the first agent step expensive. Context. A model's advertised maximum does not prove that the selected runtime and device can use it efficiently. External tools. A local model does not make email, web search, SaaS APIs, or remote MCP servers local. Updates. CLI flags, registries, model templates, and configuration formats change. Pinning a working setup creates an upgrade responsibility. Security. A malicious local document can contain a prompt injection. Removing cloud access does not prevent the model from requesting a dangerous local action. Observability. Without logs, model, template, server, agent, and tool failures are difficult to distinguish. Backups. Local memory can disappear with disk failure unless it is backed up separately. Offline autonomy is not the same as universal usefulness. Without a network, the agent cannot retrieve current news, use remote business systems, or synchronize data. This architecture is best suited to local documents, private knowledge collections, drafting, classification, and workflows where data locality matters more than live external information. qvac doctor recognizes an available backend. The selected model exists in the installed QVAC registry. The model is declared in qvac.config.json. "tools": true is enabled for that model. QVAC listens only on 127.0.0.1:11434. /v1/models returns hermes-local. A direct curl request returns a structured chat completion. The direct function test returns structured tool calls. Hermes uses the custom provider and local /v1 URL. Hermes can access only the laboratory directories. The input directory is read-only at the system or container level. Filesystem results are checked independently of the agent's claims. Persistent memory is verified in a new session. Both processes cold-start after external networking is disabled. The offline task creates and rereads a new file. No real credentials are present in configuration, prompts, logs, or memory. If every item passes, you have more than a local chat interface. You have a minimal reproducible personal-agent environment: QVAC serves the model, Hermes manages memory and function execution, operating-system controls restrict access, and the offline test demonstrates that the working path does not require a cloud API. Add one specialized skill and one narrowly defined tool, then repeat the complete test suite. Avoid enabling a browser, unrestricted shell, email, and messaging at the same time: when a failure occurs, you need to know which layer violated its contract. Continue with the practical material in Agent Lab Journal guides, and use the glossary for definitions of agents, language models, context, tool calling, memory, MCP, and related engineering terms. ← All guides · Open the glossary → Agent Lab Journal Real experiments. Verifiable conclusions.