{"slug": "private-and-openai-comp-inference-engine-for-nvidia-linux-and-apple-silicon", "title": "Private and OpenAI-Comp. Inference Engine for Nvidia Linux and Apple Silicon", "summary": "Tara Vision released taraengine, an MIT-licensed, open-source local inference engine built on llama.cpp that runs Gemma, GLM and Qwen models on NVIDIA Linux machines and Apple Silicon Macs and exposes an OpenAI- and Anthropic-compatible /v1 API on localhost port 8272. Each taraengine release pins exactly one llama.cpp build, downloaded once and hashed against the lock file before unpacking and re-checked at every engine start, with catalog models pinned by size and sha256 and a KV cache tuned per model on an RTX 5090. The engine ships with no telemetry and no update checks, installs via a Homebrew cask on macOS or a curl install script on Linux as a LaunchAgent or systemd service, and is used natively by taracode since taracode 3.3.0.", "body_md": "**The Tara Vision Inference Engine**\n\n  Local, private and OpenAI-compatible: Gemma, GLM and Qwen on NVIDIA Linux machines and Apple Silicon Macs.\n\n  Built on llama.cpp. Made for taracode. No telemetry, ever.\n\n- **Everything above the kernels** - taraengine owns the catalog, the pull, the presets, the pinned runtime, the\nprocess supervision, the API and the measurement; the math is done by ggml through llama.cpp's server\n- **One pinned runtime** - Each release pins exactly one llama.cpp build, downloaded once, hashed against the\nlock before it is unpacked and checked again every time the engine starts\n- **A curated catalog** - Models pinned by size and sha256, with the projector and the draft model they come\nwith, and a KV cache chosen for each on an RTX 5090 board\n- **OpenAI and Anthropic APIs** -`/v1` is llama-server's surface, proxied as it streams, with tools, thinking\nand vision where the model has them\n- **Measured** - Every chat answer carries its timings, and`taraengine ps` shows what a loaded model holds in\nmemory, measured on NVIDIA and estimated on a Mac\n- **Privacy-first** - Localhost by default and a Bearer key when you set one; no telemetry, no update checks of\nits own, and the network only for the downloads that a pull, an install or an update needs\n- **No account required** - Open source, MIT-licensed; install it and run it\n\nThis release serves the pinned runtime, the catalog with its tuned presets, the control API and the OpenAI-compatible passthrough, on Apple Silicon Macs and NVIDIA Linux machines. It installs with one command and runs as a service, which fetches the llama.cpp build that its binary pins. taracode runs on it natively since taracode 3.3.0.\n\nOn an Apple Silicon Mac:\n\n```\nbrew install --cask tara-vision/taracode/taraengine\n```\n\nOn Linux:\n\n```\ncurl -fsSL https://engine.tara.vision/install.sh | bash\n```\n\nBoth install the binary and the service, a LaunchAgent on macOS and a systemd service on Linux, and the service\nfetches the pinned llama.cpp build for the machine on its own. [Install](#install) has the packages, the verified\ndownload by hand and the build from source.\n\n```\ntaraengine pull gemma4:12b\n```\n\nOn a Mac the first command that needs the engine starts the service. A command waits while the engine fetches its\nruntime. `gemma4:12b` suits 16 GB of RAM. On a smaller machine, pull the model that `taraengine doctor` names, and use\nthat name in step 3.\n\n```\ncurl http://127.0.0.1:8272/v1/chat/completions -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"gemma4:12b\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}'\n```\n\nThe chat request loads the model on first use. `taraengine doctor` shows the machine, the runtime and the daemon, and\nrecommends the catalog's model for the RAM it finds: for the machine of the daemon when one answers, and for this\nmachine when none does. taracode finds the engine at this address without configuration, and other\nOpenAI-compatible clients can use it too; a web page from another origin needs `allowed_origins`.\n\n| Command | What it does | \n|---|---|\n| `taraengine serve` | Runs the engine: supervises llama-server and serves the API on the listen address | \n| `taraengine pull <model>` | Pulls a catalog model with its siblings, or registers another GGUF ( `hf://...` or an absolute path) | \n| `taraengine list` | Lists the catalog and the pulled models | \n| `taraengine load <model>` | Loads a model, optionally at a context window ( `--context` ) or past a faulty mark (`--force` ), and waits until it is ready | \n| `taraengine ps` | Shows the loaded models and their memory | \n| `taraengine unload <model>` | Unloads a model | \n| `taraengine rm <model>` | Removes a pulled model and its siblings from disk | \n| `taraengine doctor` | Shows the machine, the runtime and the daemon, and names the model for your RAM | \n| `taraengine service install` | Installs the service for this binary and starts it: at login on macOS, at boot on Linux | \n| `taraengine service start` | Starts the service, and has it start at login or boot again | \n| `taraengine service stop` | Stops the service until the next `service start` | \n| `taraengine service restart` | Restarts the service, or starts it when it does not run | \n| `taraengine service status` | Shows the service and the engine it runs; ends with 3 when it does not run | \n| `taraengine service uninstall` | Stops the service and removes it; the models and the configuration stay | \n| `taraengine update` | Updates taraengine to the latest release the way it was installed ( `--check` only reports) | \n| `taraengine runtime install` | Downloads, verifies and unpacks the pinned llama.cpp build | \n| `taraengine runtime update` | Moves to the build this taraengine pins | \n| `taraengine runtime status` | Shows the installed build, and whether it is verified | \n| `taraengine version` | Prints the taraengine version | \n\nThe commands that talk to the daemon (`pull`, `list`, `ps`, `load`, `unload`, `rm` and `doctor`) take its address and\nkey from the configuration file; `--host`, `--api-key` and `TARAENGINE_API_KEY` win over it, as\n[Configuration](#configuration) explains. Every one of them but `doctor` waits for an engine that is starting, and\nprints what it is doing. `taraengine <command> --help` lists the flags.\n\n**macOS:** the Homebrew cask, from the taracode tap.\n\n```\nbrew install --cask tara-vision/taracode/taraengine\n```\n\nThe cask installs the binary, signed by Tara Vision, LLC and notarized by Apple, and the service, which starts with the\nfirst command that needs the engine and at every login. `brew upgrade` moves both to a new release and restarts the\nengine; `brew uninstall --cask taraengine` stops the service and removes it, and `--zap` removes the models, the\nconfiguration and the logs as well.\n\n**Linux, or a Mac without Homebrew:** the installer script.\n\n```\ncurl -fsSL https://engine.tara.vision/install.sh | bash\n```\n\nIt downloads the binary for the machine, checks it against the release's `checksums.txt`, installs it to\n`/usr/local/bin` (with sudo when the directory needs it), and installs the service: as you on macOS, as root on Linux.\nIts variables go after the pipe: `curl -fsSL https://engine.tara.vision/install.sh | TARAENGINE_VERSION=v0.3.0 bash`\ninstalls that release (v0.3.0 or later), `INSTALL_DIR=<dir>` another directory (on Linux one outside `/home`, `/root`,\n`/run/user`, `/tmp` and `/var/tmp`, which the service cannot reach), and `TARAENGINE_NO_SERVICE=1` the binary alone. On\nLinux it also installs the OpenMP library that llama.cpp needs (`libgomp1`, or `libgomp`) with the system's package\nmanager when the machine lacks it. Running it again updates taraengine.\n\n**Packages:** the release has a `.deb` and an `.rpm` for amd64 and arm64, which depend on the OpenMP library that\nllama.cpp needs. On Debian and Ubuntu:\n\n```\nsudo apt install ./taraengine_<version>_linux_amd64.deb\n```\n\nOn Fedora and RHEL:\n\n```\nsudo dnf install ./taraengine_<version>_linux_amd64.rpm\n```\n\nA package installs the binary as `/usr/bin/taraengine`, LICENSE and NOTICE under `/usr/share/doc/taraengine/`, and the\nsystem service. Removing the package removes the service; a Debian purge also removes `/etc/taraengine` and\n`/var/lib/taraengine`.\n\n**By hand, verified:** download the binary for your machine from the\n[releases page](https://github.com/tara-vision/taraengine/releases): `taraengine-darwin-arm64` for Apple Silicon,\n`taraengine-linux-amd64` or `taraengine-linux-arm64` for Linux. Every release also carries `checksums.txt`, a keyless\ncosign signature of it as a Sigstore bundle (`checksums.txt.sigstore.json`) and SLSA provenance.\n\n```\nBASE=https://github.com/tara-vision/taraengine/releases/latest/download\nBIN=taraengine-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')\ncurl -fsSL --fail-early -O $BASE/$BIN -O $BASE/checksums.txt -O $BASE/checksums.txt.sigstore.json && \\\n  cosign verify-blob --bundle checksums.txt.sigstore.json \\\n    --certificate-identity-regexp 'https://github.com/tara-vision/taraengine/.github/workflows/release.yml@refs/tags/v.*' \\\n    --certificate-oidc-issuer https://token.actions.githubusercontent.com checksums.txt && \\\n  grep \" $BIN\\$\" checksums.txt > $BIN.sha256 && sha256sum -c $BIN.sha256 && \\\n  sudo mkdir -p /usr/local/bin && sudo install -m 0755 $BIN /usr/local/bin/taraengine\n```\n\nOn Linux, install the OpenMP library that llama.cpp needs first, as the installer script and the packages do:\n`sudo apt-get install libgomp1` on Debian and Ubuntu, `sudo dnf install libgomp` on Fedora and RHEL. The steps form one\nchain, so a download, a signature or a checksum that fails stops it before anything is installed. On a Mac without\n`sha256sum`, use `shasum -a 256 -c $BIN.sha256` in its place. Then install the service: `taraengine service install` on\nmacOS, `sudo /usr/local/bin/taraengine service install` on Linux. `taraengine version` prints the version you installed.\nThe signature covers `checksums.txt`, and the checksums cover every binary and package of the release, so verifying the\none verifies the rest. The provenance can be checked with\n[slsa-verifier](https://github.com/slsa-framework/slsa-verifier). The macOS binary is signed and notarized, so macOS\nruns it also when a browser downloaded it.\n\n**Updates:** `taraengine update` brings taraengine to the latest release the way it came: through Homebrew for the cask,\nthe new package for a `.deb` or an `.rpm` (run it with sudo), or the binary in place (with sudo when its directory needs\nit: `sudo /usr/local/bin/taraengine update`), and then it restarts the service. `taraengine update --check` only says\nwhether a newer release is out. It looks the release up only when you run it.\n\nTo build from source, clone the repository and run `make build`; [Development](#development) lists the other\ntargets.\n\n`taraengine service install` installs the service for the binary that runs it and starts it; the cask, the installer\nscript and the packages run it for you. The service runs `taraengine serve --service`, which fetches the pinned\nruntime when it is missing and keeps its stderr to warnings.\n\n|  | macOS | Linux | \n|---|---|---|\n| What | a LaunchAgent of your user, `vision.tara.taraengine` | a systemd system service, `taraengine.service` , run as user`taraengine` | \n| Starts | at login | at boot | \n| Definition | `~/Library/LaunchAgents/vision.tara.taraengine.plist` | `/etc/systemd/system/taraengine.service` | \n| Configuration | `~/.config/taraengine/config.yaml` | `/etc/taraengine/config.yaml` , written once and then left alone | \n| Data | `~/Library/Application Support/taraengine` | `/var/lib/taraengine` | \n| Logs | `~/Library/Logs/taraengine/service.log` ,`logs/taraengine.log` of the data | `journalctl -u taraengine` ,`/var/lib/taraengine/logs/taraengine.log` | \n| Changed | as your user | with sudo | \n\n`service stop` stops the service until `service start`, across logins and reboots, and an install or an upgrade keeps a\nstopped service stopped. `service restart` restarts it, or starts a stopped one, which then runs at login or boot again;\n`service status` shows it and the engine it runs, and ends with 3 when it does not run; `service uninstall` stops it and\nremoves the definition, while the models and the configuration stay. On Linux, a normal user's commands reach the system\nservice at the default address; when the configuration sets an `api_key`, set `TARAENGINE_API_KEY`. The unit makes\n`/usr`, `/boot`, `/efi` and `/etc` read-only for the engine, hides `/home`, `/root` and `/run/user`, and gives it a\n`/tmp` and a `/var/tmp` of its own, so the binary and `data_dir` belong outside those five; an admin who needs `/home`\nfor data or models lifts `ProtectHome` with a drop-in (`sudo systemctl edit taraengine`), and the binary stays outside\nit all the same. The service imports a local GGUF file as the user `taraengine`, so the file has to be outside them too,\nsuch as under `/srv/models`, and readable by that user. Without systemd, as in a container, run `taraengine serve` under\nthe supervisor you have.\n\ntaraengine does not carry llama.cpp inside its binary. The service fetches it from the llama.cpp releases on GitHub\non its own, `taraengine runtime install` fetches it for a daemon you run by hand, and each taraengine release pins\nexactly one upstream build.\n\n- **Pinned.** The binary embeds`internal/runtime/runtime.lock.yaml` : the build number and the sha256 of every\nrelease archive it may install, per platform.\n- **Verified.** The download is hashed and compared with the lock before anything is unpacked. The unpacking refuses\nany entry that would land outside the target directory, and any link that does not point at a sibling file. The\nsha256 of the installed`llama-server` goes into a`manifest.json` next to it, and`taraengine serve` checks the\nbinary against that file when it starts. It refuses to start when the runtime is missing or does not match. Run by\nhand,`serve` never downloads a runtime itself; run by the service, as`serve --service` , it installs the pinned\nbuild first, trying again after 10, 30 and 60 seconds and then every five minutes, and removes the builds it no\nlonger pins once the new one verifies. A configured`runtime_dir` is left alone: nothing is fetched or removed for it.\n- **Kept whole.** The directory holds`llama-server` , its libraries and upstream's`LICENSE` . A failed install leaves\nthe previous one as it was.\n- **Moved on request.**`taraengine runtime update` moves to the build that the running taraengine pins, never to\nanother one, and says`already at <build>` when there is nothing to do.`taraengine runtime status` says whether\nthe runtime that the daemon would run on is verified, which is the`runtime_dir` of the configuration when it is set\nand the pinned build under the data directory otherwise, and`--dir-only` prints just that directory.\n\nThe build is chosen for the machine:\n\n| Machine | Build | \n|---|---|\n| Apple Silicon | Metal | \n| Linux with an NVIDIA driver of 580 or newer | CUDA 13.4 | \n| Linux amd64 with an NVIDIA driver of 570 or newer | CUDA 12.8 | \n| Any other Linux machine | CPU | \n\nThe driver version comes from `nvidia-smi`. The Linux arm64 builds are pinned too, but have not been tested on a\nmachine. Other systems, such as Intel Macs, are not supported in this release. A CUDA build needs the CUDA runtime\nlibraries of its major version. When `ldconfig` does not list them, `runtime install` also fetches upstream's CUDA\nruntime bundle, a few hundred megabytes; `--cudart` fetches it always and `--no-cudart` never. `taraengine doctor`\nshows the GPU, the RAM tier and the runtime it found, and names a model that needs a newer runtime than the pinned one.\n\nThe runtime lives in `<data dir>/runtime/<build>/`. The data directory is `~/Library/Application Support/taraengine`\non macOS, and `$XDG_DATA_HOME/taraengine` (default `~/.local/share/taraengine`) on Linux; the Linux service uses\n`/var/lib/taraengine`.\n\n`taraengine serve` runs in the foreground and starts one `llama-server` in router mode on a loopback port of its\nown. It writes the process id of that server to `child.pid` in the data directory each time the server starts, and\nremoves the file when it stops the server cleanly. If the server exits, taraengine ends the model processes that it\nleft behind and starts it again after a pause of 1, 2, 4 and then 8 seconds. The fifth exit within ten minutes ends\nthe restarts and marks the backend unhealthy until `serve` is started again.\n\nWhen a daemon dies without stopping its server (it was killed, or the machine ran out of memory), the server and its\nmodel processes keep running and keep their GPU memory. The next `serve` looks at `child.pid` before it starts its own\nserver. If that process is still there and runs the `llama-server` of the runtime in use, taraengine logs it and ends\nits process group, with SIGTERM and, ten seconds later, SIGKILL, so that a stale server cannot take the memory that a\nnew load needs. A server of another runtime directory, or any other process, is left alone.\n\nOne daemon runs per data directory. `serve` takes a lock on the file `lock` in the data directory and writes its own\nprocess id into it, and a second `serve` on the same data directory, whatever its listen address, stops at once with an\nerror that names the directory and that process id. The lock ends with the daemon, however it ends, and the file stays.\n\nA model whose load through `/api/load` fails twice in a row is marked faulty and refused, on `/api/load` and on `/v1`,\nuntil `taraengine load <model> --force`; `taraengine doctor` shows its exit code and the end of the server log. A load\nthat `/v1` starts by itself, and a server that dies in the middle of a chat, are not counted yet.\n\nSIGINT or SIGTERM stops the daemon: requests in flight get five seconds, and `llama-server` gets SIGTERM and, ten\nseconds later, SIGKILL.\n\nThe catalog is built into the binary. Each entry pins a Hugging Face file by size and sha256, together with the\nprojector that lets a model read images and the draft model that speeds it up (speculative decoding: a\nmulti-token-prediction head, `mtp`, or a DFlash drafter, `dflash`), when the model has them. A pull fetches all of\nthem, a sibling from its own repository when it names one.\n\n| Name | Tier | Size | Capabilities | KV cache | \n|---|---|---|---|---|\n| gemma4:e4b | small (default) | 4.6 GB | tools, thinking, vision | q8_0 | \n| gemma4:12b | 16 (default) | 7.2 GB | tools, thinking, vision | f16 | \n| gemma4:26b | 32 | 14.6 GB | tools, thinking, vision | f16 | \n| gemma4:31b | 48 (default) | 18.0 GB | tools, thinking, vision | f16 | \n| glm-4.7-flash | 32 (default) | 18.2 GB | tools, thinking | q8_0 | \n| qwen3.6:27b | 32 | 19.1 GB | tools, thinking, vision | q8_0 | \n| qwen3.8:27b | 32 | 19.0 GB | tools, thinking, vision | q8_0 | \n| qwen3.6:35b | 48 | 20.4 GB | tools, thinking, vision | q8_0 | \n\nSize is the main file, as `taraengine list` shows it; the projector and the draft file come on top. KV cache is the\ntype of the model's attention cache: q8_0 holds less memory than f16 at the same window, and an entry keeps f16 where\nq8_0 cost it more than a tenth of its speed on the RTX 5090 board. A `cache-type-k` or `cache-type-v` in\n`preset_global` sets another for every model, these included. llama.cpp quantizes the V cache only with flash\nattention, so with `flash-attn: \"off\"` in `preset_global` these models keep an f16 V cache. A tier is the RAM class a\nmodel is meant for. `small` suits any machine, and a machine with at least 14, 30 or 46 GiB of RAM counts as `16`,\n`32` or `48`. `(default)` marks the model that `taraengine doctor` recommends for a tier. Each model keeps the license\nof its authors; read it on the model's Hugging Face page before you use it.\n\n``` bash\n$ taraengine list\nNAME           STATUS     SIZE     TIER   CONTEXT  CAPABILITIES\ngemma4:e4b     available  4.6 GB   small  32768    tools,thinking,vision\ngemma4:12b     available  7.2 GB   16     32768    tools,thinking,vision\ngemma4:26b     available  14.6 GB  32     32768    tools,thinking,vision\n...\n```\n\n`STATUS` is `available` for a model in the catalog that is not on disk, `pulled` once it is, and `loading`, `loaded`\nor `sleeping` while the engine runs it; `failed` and `faulty` mark a load that did not work. `CONTEXT` is the context\nwindow in tokens that the model runs at, or would run at. Sizes below 1 GB, such as a small imported model, are shown in\nMB with one decimal, and from 1 GB in GB.\n\n```\ntaraengine pull gemma4:12b                                                # a catalog model\ntaraengine pull hf://ggml-org/models-moved/tinyllamas/stories260K.gguf    # any GGUF file on Hugging Face\ntaraengine pull /srv/models/my-model.gguf                                 # a GGUF file on this machine\n```\n\nA pull resolves each file with Hugging Face, refuses it when the sha256 that Hugging Face reports is not the one the\ncatalog pins, checks the free disk space, and downloads into a `.part` file, so that running the same pull again\nresumes an interrupted one. The complete file is hashed before it is moved into place; a file that fails the check is\ndeleted and never becomes a model. The command prints a line for every five percent of a download, in MB below 1 GB\nand in GB from there.\n\n``` bash\n$ taraengine pull gemma4:12b\nresolving gemma-4-12B-it-Q4_0.gguf\ndownloading gemma-4-12B-it-Q4_0.gguf 361.5 MB / 7.2 GB\n...\ndone gemma4:12b\n```\n\nA model from `hf://` or from a local path is named after its file without `.gguf`, has no tier and no promise of a\ntuned preset, and its capabilities are marked `(inferred)`: taraengine reads them from the chat template while the\nmodel runs and keeps them, so `taraengine list` still shows them after the model stops. Its native context window\ncomes from the file's header when it is pulled, and its default window is never larger; a load that asks for another\nwindow gets that one. A local file is linked, not copied, and `taraengine rm` removes the link and never the file. A\nname that the catalog owns is refused with a 409, and so is one that llama-server would serve under the same id as\nanother model's name (`gemma4:12b` and `gemma4-12b` are one id), and a different file under a name that an import\nalready holds.\n\nA gated or private repository needs `hf_token` in the configuration, or `TARAENGINE_HF_TOKEN` in the environment of\n`serve`. To use a catalog of your own, put a `registry.yaml` in the data directory; it replaces the built-in one. It\nis checked like the built-in one: a key that taraengine does not know, a `preset` key that does not tune how a model\nruns (the error lists the keys that do), two names that llama-server would serve under one id, a model with both an\n`mtp` and a `dflash` draft, and a second default for a tier are refused with an error that names them. A projector or\na draft is a `file` with its `sha256` and `size` in the entry's repository, or in one of its own with `hf_repo` (a\ndrafter published apart from its model); from a checkout of this repository, `go run ./tools/pinregistry <file>` fills\nthe pins. The engine itself sets\nwhere a model is, its context window, its projector, its draft and the draft's kind (`spec-type`); a preset tunes the\nrest, the drafter's token count with `spec-draft-n-max` among it.\n\n```\ntaraengine load gemma4:12b                  # loads it, and waits until it is ready\ntaraengine load gemma4:12b --context 16384  # loads it at another window, which the daemon remembers for the model\ntaraengine ps                               # what is loaded, and what it holds in memory\ntaraengine unload gemma4:12b\n```\n\n`taraengine rm <model>` deletes a pulled model and its projector and draft file from disk, each under its own\nrepository.\n\n``` bash\n$ taraengine load gemma4:12b\nloaded gemma4:12b at 32768 tokens of context, 10.0 GiB measured\n$ taraengine ps\nNAME        STATUS  CONTEXT  MEMORY\ngemma4:12b  loaded  32768    10.0 GiB measured\n```\n\nThe context window defaults to 32768 tokens, or to the model's native maximum when that is smaller. `context` in the\nconfiguration changes the default, and `context_by_tier` sets it per RAM tier. The memory of a loaded model is\nmeasured on NVIDIA, as the GPU memory that the driver reports for the model's process. On a Mac the GPU has no\nper-process figure, so the number is the process's resident memory and is labeled `estimated`. A dash means that\nthere is no figure.\nA model that a request through `/v1` brings up on its own gets its figure from a later read of the model list\n(`taraengine list`, `taraengine ps` or the API) that finds no load, unload or catalog change under way. A model left\nidle for `sleep_idle_seconds` sleeps: `ps` shows `sleeping` with the figure of its last run, the next request wakes\nit, and `taraengine load` of a sleeping model at its window answers at once.\n\nThe control API is under `/api` on the listen address, `http://127.0.0.1:8272` by default. Bodies are JSON, and an\nerror is `{\"error\":{\"message\":\"...\",\"hint\":\"...\"}}`.\n\n| Route | What it does | Errors | \n|---|---|---|\n| `GET /api/health` | `{\"status\":\"ok\"}` or`\"degraded\"` , with the backend's name; while the daemon starts,`\"starting\"` with a`detail` that names the step. Open without a key;`HEAD` works too. |  | \n| `GET /api/version` | The taraengine version, the runtime build and the platform. |  | \n| `GET /api/models` | The catalog and the pulled models: name, status, size, family, parameters, quantization, context, capabilities, tier, default and memory. | 503 | \n| `GET /api/models/{name}` | One model in detail: also its native context, a digest, the preset in effect and, after a failure, the end of the server log. | 400, 404, 503 | \n| `GET /api/loaded` | The loaded and sleeping models, with their status ( `loaded` or`sleeping` ), context window and memory (`bytes` and`source` ,`measured` or`estimated` ). | 503 | \n| `POST /api/load` | `{\"model\":\"...\",\"context\":N,\"force\":false}` , the last two optional.`context` is from 512 to 4194304 tokens; 0 or no value means the model's usual window. Loads the model and answers with it once it is up; a model that sleeps at that window is answered at once and wakes on its next request, and a model that is still loading is waited for. | 400, 404, 409 faulty, 502 load failed, 503, 504 over `load_timeout` | \n| `POST /api/unload` | `{\"model\":\"...\"}` . Answers 204. | 400, 404, 503 | \n| `POST /api/pull` | `{\"model\":\"...\"}` with a catalog name,`hf://owner/repo/file.gguf` or an absolute path. Streams the progress as server-sent events (`resolving` ,`downloading` ,`verifying` ,`done` ,`error` ); with`?wait=1` it answers once, with`{\"status\":\"done\"}` . | 400 bad reference, 404 unknown name, 409 that name is busy or taken by another model or file, 502 | \n| `DELETE /api/models/{name}` | Removes a pulled model and its sibling files. A model imported from a path or `hf://` whose file is already gone is dropped from the catalog all the same. Answers 204. | 400, 404, 409, 503 | \n| `GET /api/doctor` | The machine (GPU, memory, tier, driver), the runtime status and the recommended model per tier. |  | \n| `/v1/*` | llama-server's OpenAI and Anthropic API, proxied. | 400, 401, 403, 404, 405, 409, 503 | \n\nWhile the daemon starts, `GET` and `HEAD /api/health` and `GET /api/version` answer, and every other request that passes\nthe host, origin, key and JSON checks answers 503 with `Retry-After: 5` and the error\n`taraengine is starting: <detail>`.\n\n```\ncurl http://127.0.0.1:8272/api/loaded\ncurl http://127.0.0.1:8272/api/load -H 'Content-Type: application/json' -d '{\"model\":\"gemma4:12b\",\"context\":16384}'\ncurl http://127.0.0.1:8272/api/loaded -H \"Authorization: Bearer $TARAENGINE_API_KEY\"   # when api_key is set\n```\n\n`/v1` is llama-server's OpenAI and Anthropic surface, proxied without buffering the answers, so streams arrive as they\nare produced: `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models`, `/v1/messages` and\n`/v1/messages/count_tokens` among them.\n\n- **Models.** Name the model by its catalog name, such as`gemma4:12b` . A request for a pulled model that is not\nloaded loads it first, and with`models_max` at 1 it replaces the model that was loaded. A request has to name a\npulled model: a name the engine does not know, or has not pulled, is a 404 that lists up to three of the nearest\npulled names; a model marked faulty is a 409 until`taraengine load <model> --force` ; and a body without a`model` ,\nor one that is not a JSON object, is a 400.`/v1/models` lists the pulled models under their catalog names and\nnothing that the router found elsewhere. Only GET, HEAD and POST are served: OPTIONS is answered by the daemon\nitself, as described under Privacy and security, and every other method is a 405 with an`Allow` header. The`model` field of a chat answer carries the id that llama-server knows the model by, which writes a hyphen for the colon:`gemma4-12b` .\n- **Thinking.**`\"chat_template_kwargs\": {\"enable_thinking\": false}` switches thinking off and`true` on, and`\"reasoning_effort\": \"low\"` ,`\"medium\"` or`\"high\"` sets the effort where the model's template honors it. The\nthoughts come back in`reasoning_content` , on the message and on streamed deltas.\n- **Timings.** Every chat answer carries`usage` and`timings` (`prompt_n` ,`prompt_ms` ,`prompt_per_second` ,`predicted_n` ,`predicted_ms` ,`predicted_per_second` ,`cache_n` ); in a stream they arrive on the last chunk.\nAnswers that`/v1` passes on carry an`X-Taraengine-Version` header.\n- **Limits.** A request body over 32 MiB is refused with 413.\n\nThe JSON types of the API are in `pkg/api`, and a typed Go client is in `pkg/client`.\n\ntaraengine needs no configuration file: every key has a default. When there is one, it is\n`$XDG_CONFIG_HOME/taraengine/config.yaml` or `~/.config/taraengine/config.yaml`, on macOS too, and `--config FILE`\nnames another, which then has to exist. The Linux system service reads `/etc/taraengine/config.yaml`.\n\nA key that this version does not know is an error that names the key and its line and lists the accepted keys, so that a\nmisspelled `api_key` cannot leave an engine open without a word. `serve` also takes `--listen`, `--data-dir`,\n`--runtime-dir` and `--api-key`, which win over the file. The environment variables `TARAENGINE_API_KEY` and\n`TARAENGINE_HF_TOKEN` win over the file as well. The other commands read the same file for the data directory. They\nreach the daemon at `--host`, or, when `--host` is left out, at the `listen` address of the file,\n`http://127.0.0.1:8272` when there is none. The key is `--api-key`, else `TARAENGINE_API_KEY`, and either goes with any\naddress. The `api_key` of the file goes only with the address of the file: it belongs to the daemon that the file\nconfigures, so with `--host` it is not used, and a daemon that asks for a key then needs `--api-key` or\n`TARAENGINE_API_KEY`.\n\nKeep the key in the configuration file (with a mode that only you can read) or in the environment. Do not put it in\n`--api-key` on a machine that other people use: the arguments of a command show in the process list of every user.\n\n| Key | Default | What it does | \n|---|---|---|\n| `listen` | `127.0.0.1:8272` | The address the API listens on. See Privacy and security before you change it. | \n| `api_key` | none | The key that every route but `GET` and`HEAD /api/health` requires, as a Bearer token. | \n| `allowed_origins` | none | Web origins whose pages may use the API besides the pages served from `localhost` ,`127.0.0.1` and`[::1]` . Write each as the browser sends it,`scheme://host` or`scheme://host:port` , such as`https://app.example` ; an origin is matched whole, the case aside, and a value that is not an origin is a configuration error. See Privacy and security. | \n| `data_dir` | the data directory above | Where the runtime, the models, the presets, the state and the logs live. | \n| `models_max` | `1` | How many models stay loaded together. Loading another one replaces the first. | \n| `sleep_idle_seconds` | `900` | Seconds without a request after which a loaded model goes to sleep, at least 1. The next request wakes it. `-1` never sleeps;`0` is refused. | \n| `context` | `32768` | The default context window in tokens, at least 512, and never more than the model's own maximum. | \n| `context_by_tier` | none | A context window per RAM tier, such as `\"16\": 16384` ; the tiers are`small` ,`16` ,`32` ,`48` and`other` for models outside the catalog. | \n| `hf_token` | none | A Hugging Face token for gated or private repositories. It is sent to Hugging Face only and never logged. | \n| `load_timeout` | `600` | Seconds that a load may take before it is given up, at least 1. Raise it for a slow disk. | \n| `runtime_dir` | none | A runtime directory to run instead of the pinned build under the data directory. It has to hold the `manifest.json` that`runtime install` writes, and it is verified like the pinned build. | \n| `preset_global` | none | Extra llama-server options for every model, as preset keys (the long option name without the dashes), laid over the engine's own: `jinja: \"true\"` ,`flash-attn: auto` ,`n-gpu-layers: \"999\"` ,`reasoning-format: deepseek` and`load-on-startup: \"false\"` . A key set here also wins over the catalog's own preset for that key, as the KV cache of some models. Quote numbers. | \n\n```\n# ~/.config/taraengine/config.yaml\napi_key: change-me\nsleep_idle_seconds: -1\ncontext_by_tier:\n  \"16\": 16384\n  \"48\": 65536\npreset_global:\n  cache-type-k: q8_0\n  cache-type-v: q8_0\n```\n\nThe data directory holds:\n\n```\nruntime/<build>/    llama-server, its libraries, upstream's LICENSE and manifest.json\nmodels/             GGUF files by owner and repository, links to local files, index.json\npresets.ini         the llama-server presets, generated on every change: do not edit\nregistry.yaml       optional: a catalog that replaces the built-in one\nstate.json          faulty-model marks, remembered context windows, the last model loaded\nlock                held by the running daemon, with its process id: a second serve on this directory is refused\nchild.key           the generated key of the supervised llama-server (mode 0600)\nchild.pid           the process id of the supervised llama-server, removed by a clean stop, read at the next start\nllama-cache/        an empty download cache for llama-server, so that it never serves a model outside the catalog\nlogs/               taraengine.log and llama-server.log, rotated at 20 MB with three old files kept\n```\n\n**No telemetry.** taraengine sends no usage data and files no crash reports, and it looks for a new release only when\nyou run `taraengine update`. It connects only when something needs it: to `huggingface.co` on `taraengine pull`; to\n`github.com` on `taraengine runtime install`, `runtime update` and `taraengine update`, and when the service starts\nwithout a verified copy of the build it pins, as on the first start after an install or after an upgrade that pins a new\nbuild; and to `api.github.com` and then `engine.tara.vision` when `taraengine update` cannot read the latest version\nfrom the release page. Hugging Face and GitHub redirect downloads to their own content delivery hosts (under `hf.co` and\n`githubusercontent.com`). The installer script asks the same addresses; on Linux, when libgomp is missing, it has the\nsystem's package manager fetch it from the system's repositories, as `apt` and `dnf` do for a package's dependencies. A\ncask's update runs Homebrew, which connects as Homebrew does. Nothing else leaves the machine.\n\nA pull is carried out by the daemon, so it is the `serve` process that connects to Hugging Face, during the pull and for\na short while after it. Under the service, `serve --service` also connects to GitHub while it fetches its runtime. At\nany other time neither `serve` nor the `llama-server` it supervises holds a connection beyond the loopback. A Hugging\nFace token is sent to Hugging Face only, and it is never logged.\n\nWhat can reach the API:\n\n- **Localhost by default.**`listen` is`127.0.0.1:8272` . To serve other machines, set`listen` , for example to`0.0.0.0:8272` , and set an`api_key` . taraengine logs a warning when it listens beyond localhost without a key.\n- **A Bearer key.** With`api_key` set, every route but`GET` and`HEAD /api/health` needs`Authorization: Bearer <key>` , and a missing or wrong key is a 401. No other header counts: a client that sends its\nkey as`x-api-key` is refused.\n- **Web pages.** A page that you open in a browser can send requests to`127.0.0.1:8272` , and the browser attaches an`Origin` header to them. The daemon answers a request with an`Origin` header with 403 unless the page is served from`localhost` ,`127.0.0.1` or`[::1]` (http or https, any port) or its origin is listed under`allowed_origins` , with or\nwithout an`api_key` . The preflight that a browser sends before a request across origins\nis answered by the daemon, for the allowed origins only, and is never passed on to`llama-server` . The CORS headers\nthat`llama-server` puts on its own answers are removed, so that the headers of the daemon are the only ones a page\never sees. Programs that are not browsers, such as curl, the OpenAI and Anthropic SDKs and taracode, send no`Origin` and are not asked about it.\n- **JSON only.** Every`POST` under`/api` and`/v1` must be`application/json` , a charset parameter allowed, and\nanything else is a 400 with a hint. A web page can make the browser post a form or plain text to any address\nwithout asking first, and the body would otherwise be read as JSON all the same. The OpenAI and Anthropic SDKs send`application/json` ; with curl, send`-H 'Content-Type: application/json'` .\n- **The Host header.** With no`api_key` , the API refuses a request whose Host is not one of its own addresses\n(`localhost` ,`127.0.0.1` ,`[::1]` or the host in`listen` ) with a 403. That keeps out a web page that points its own\nDNS name at 127.0.0.1. With a key set, the key does that job. A daemon that listens on every interface skips the\ncheck.\n- **The supervised server.** The`llama-server` that taraengine starts listens on a loopback port with a generated\nkey. The key is in`child.key` in the data directory, readable by you only, and`llama-server` reads it from that\nfile, so it is on no command line. The`/v1` proxy swaps your key for it. The processes that`llama-server` starts\nfor each loaded model listen on random loopback ports and do not check that key, so any program on the same machine\ncan reach them, as it can with any inference server on localhost. Keep that in mind on a machine that you share with\npeople you do not trust. The server runs with an environment of its own:`PATH` ,`HOME` ,`TMPDIR` ,`LANG` , the`LC_*` and`CUDA_*` variables as you have them, and the runtime's library paths, and nothing else. It inherits neither\nyour API key nor your Hugging Face token, nor any`LLAMA_ARG_*` setting, and its`LLAMA_CACHE` is the empty`llama-cache` directory in the data directory, so it never serves a model that is not in the catalog.\n- **Verified downloads.** Runtime archives are checked against the sha256 values pinned in the binary, and model files\nagainst the values pinned in the catalog, or for`hf://` files against the checksum that Hugging Face reports. A\nmodel name is checked before it touches the file system, so it cannot carry a path.\n- **Logs.**`logs/taraengine.log` and`logs/llama-server.log` in the data directory are readable by you only. taraengine\ndoes not log prompts or answers, and the supervised`llama-server` runs at warning verbosity.\n\nTo report a vulnerability, see [SECURITY.md](https://github.com/tara-vision/taraengine/blob/main/SECURITY.md).\n\n```\nmake build            # Build the binary\nmake test             # Run the tests with the race detector\nmake coverage-gate    # Check the per-package coverage floors\nmake lint             # Run golangci-lint\nmake hygiene          # Check dashes, quotes, lab names and Go file sizes\nmake integration      # Run the real pinned runtime with a one-megabyte model; needs the network\nmake vuln             # Run govulncheck\n```\n\n[CONTRIBUTING.md](https://github.com/tara-vision/taraengine/blob/main/CONTRIBUTING.md) lists the checks that every change has to pass.\n\nContributions are welcome! Please read our [Contributing Guide](https://github.com/tara-vision/taraengine/blob/main/CONTRIBUTING.md)\nand [Code of Conduct](https://github.com/tara-vision/taraengine/blob/main/CODE_OF_CONDUCT.md).\n\nFor security issues, please see our [Security Policy](https://github.com/tara-vision/taraengine/blob/main/SECURITY.md).\n\ntaraengine is free, MIT-licensed and built without telemetry or a cloud service. If it saves you time, you can\n[sponsor Tara Vision on GitHub](https://github.com/sponsors/tara-vision) to keep the releases, the catalog and the\nruntime pins coming.\n\nBuilt on [llama.cpp](https://github.com/ggml-org/llama.cpp) by ggml-org (MIT). `taraengine runtime install` and the\nservice download an unmodified upstream build, and upstream's LICENSE stays next to it in the data directory. See\n[NOTICE](https://github.com/tara-vision/taraengine/blob/main/NOTICE).\n\nMIT License - see [LICENSE](https://github.com/tara-vision/taraengine/blob/main/LICENSE) for details.\n\n  Built with ❤️ by [Tara Vision](https://tara.vision) · Created by [Dejan Stefanoski](https://github.com/dayanstef)", "url": "https://wpnews.pro/news/private-and-openai-comp-inference-engine-for-nvidia-linux-and-apple-silicon", "canonical_source": "https://github.com/tara-vision/taraengine", "published_at": "2026-10-21 10:52:28+00:00", "updated_at": "2026-10-21 11:28:24.019747+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "artificial-intelligence", "ai-infrastructure", "developer-tools"], "entities": ["Tara Vision", "taraengine", "llama.cpp", "Gemma", "GLM", "Qwen", "NVIDIA", "Apple Silicon"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/private-and-openai-comp-inference-engine-for-nvidia-linux-and-apple-silicon", "markdown": "https://wpnews.pro/news/private-and-openai-comp-inference-engine-for-nvidia-linux-and-apple-silicon.md", "text": "https://wpnews.pro/news/private-and-openai-comp-inference-engine-for-nvidia-linux-and-apple-silicon.txt", "jsonld": "https://wpnews.pro/news/private-and-openai-comp-inference-engine-for-nvidia-linux-and-apple-silicon.jsonld"}}