{"slug": "tensorfold-zig-llm-server-for-metal-and-cuda-with-an-openai-compatible-api", "title": "TensorFold: Zig LLM Server for Metal and CUDA With an OpenAI-Compatible API", "summary": "TensorFold, a Zig-native LLM inference engine from developer ashhart, serves checkpoints on Apple Metal and NVIDIA CUDA from a single binary and exposes OpenAI-compatible HTTP APIs on port 8080 without requiring Python or MLX at serving time, according to its README. The GitHub repository trended in mid-November 2026, with the README pinning release v1.2.7, requiring NVIDIA driver R580 or newer for CUDA 13 kernels, and documenting a smallest start of mlx-community/Qwen3.5-2B-MLX-4bit at a pinned revision. Apple Silicon users install via the Homebrew tap ashhart/tensorfold/tensorfold, while Linux users pull tensorfold-1.2.7-linux-* release archives with the tensorfold-native wrapper as the entry binary.", "body_md": "**TensorFold is a Zig-native LLM inference engine that serves checkpoints on Apple Metal and NVIDIA CUDA from one binary, exposes OpenAI-style HTTP APIs on port 8080, and does not require Python or MLX at serving time according to its README.**\n\nThe [TensorFold GitHub repository](https://github.com/ashhart/TensorFold?ref=explainx) trended in mid-November 2026 as developers looked for alternatives to Python-heavy GPU servers. Stars climbed quickly because the README is unusually concrete: pinned Hugging Face pulls, release tarballs, qualified model rows and copy-paste curl tests. This post explains what shipped, how to try the documented quick start, and where TensorFold sits next to [llama.cpp](https://explainx.ai/blog/what-is-llama-cpp-run-models-locally-2026), [vLLM versus SGLang](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026), and Ollama-style wrappers.\n\n## TL;DR: TensorFold quick answers\n\n| Question | Answer | \n|---|---|\n| Do I need Python to serve? | README says serving needs no Python or MLX install | \n| Default API port | `8080` with`/v1/chat/completions` | \n| Mac install path | Homebrew tap `ashhart/tensorfold/tensorfold` | \n| Linux NVIDIA path | Release archive `tensorfold-1.2.7-linux-*` plus`tensorfold-native` wrapper | \n| Smallest documented start | `mlx-community/Qwen3.5-2B-MLX-4bit` with pinned revision | \n| Enterprise fleet control | Separate [TensorFold Starlight](https://github.com/ashhart/tf-starlight?ref=explainx) repo | \n| Agent contributor etiquette | [Agent Traffic Control](https://github.com/ashhart/ATC?ref=explainx) before editing upstream | \n\n## What TensorFold actually ships\n\nThe project's one-line pitch on GitHub is an **LLM inference engine for Metal, CUDA and Vulkan**. Under the hood the maintainers describe a **Zig binary** that owns checkpoint IO, tokenization and kernel dispatch. That design choice matters for operators who are tired of conda environments breaking CUDA wheels on upgrade weeks.\n\n**Serving surface.** After `tensorfold serve`, clients can hit standard OpenAI chat completions, plus Responses and Anthropic Messages endpoints listed in the README. `/v1/models` advertises the served alias you choose with `--name`. Parallel request caps exist: `--parallel N` or `--parallel auto` (up to eight streams when memory allows). Health metadata exposes active and queued connections when authentication is off.\n\n**Distribution.** Apple Silicon users get a Homebrew formula. Linux users pull gzipped release bundles with `tensorfold-native` as the entry binary. The README pins **v1.2.7** in its example and demands **NVIDIA driver R580 or newer** for CUDA 13 kernels. Those constraints are strict: TensorFold refuses older drivers at startup rather than failing mid-request.\n\n**Ecosystem extras beyond bare serving.**\n\n- **TensorFold Web** — a local launcher for picking models (`tensorfold web` ).\n- **Mac desktop app** — beta DMG in the`app/` tree bundling workers and metrics.\n- **Living Weights (`--slide`)** — experimental on-device weight edits for Nemotron checkpoints; README warns about recall limits and speed tax.\n- **Starlight** — enterprise multi-node governance in another repository.\n\nNone of those change the core news hook: a **native server** with first-class **Metal and CUDA** paths and an **OpenAI-compatible** wire format.\n\nTensorFold local LLM inference serves open-weight models on Apple Metal or NVIDIA CUDA from a Zig binary with an OpenAI-compatible HTTP API\n\n## Quick start: reproduce the README path\n\nThe maintainers standardize on **Qwen3.5-2B MLX 4-bit** because it is small and checked on Metal plus GB10 class hardware.\n\n### Apple Silicon\n\n```\nbrew install ashhart/tensorfold/tensorfold\ntensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3\ntensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking\n```\n\n### Linux with NVIDIA GPU\n\nDownload the platform archive, verify sha256, extract, then alias `tensorfold` to `tensorfold-native` exactly as the README shows for aarch64 GB10 or x86_64 hosts. Pull the same pinned Qwen revision, then serve with the same flags.\n\n### Smoke test\n\n```\ncurl -fsS http://127.0.0.1:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"local-model\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in one sentence.\"}],\"max_tokens\":128,\"temperature\":0}'\n```\n\nIf that returns JSON, your agent harness can point `baseURL` at `http://127.0.0.1:8080/v1` the same way we document for [llama-server in the llama.cpp guide](https://explainx.ai/blog/what-is-llama-cpp-run-models-locally-2026).\n\n## Model qualification table: read before you benchmark\n\nTensorFold is not a universal GGUF runner. The README's **Pick a model** section is a compatibility contract: each row lists **download command**, **machine scope**, and **native recipe** links.\n\nRepresentative rows developers asked about in early November threads:\n\n| Checkpoint family | Pull example | Hardware called out | Notes | \n|---|---|---|---|\n| Qwen3.5-2B MLX 4-bit | `mlx-community/Qwen3.5-2B-MLX-4bit@93760…` | Mac M5 Max or GB10 | Documented quick start | \n| Qwen3.8-27B MLX | `TensorFold/Qwen3.8-27B-MLX-4bit` | Mac with enough wired GPU RAM | May need `iogpu.wired_limit_mb` on 32 GB Macs | \n| Nemotron 3.5 Lightning 30B-A3B | `TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit` | Mac M1–M5 or GB10 | Includes MTP path | \n| GLM-5.3-Flash NVFP4 | `nvidia/GLM-5.3-Flash-NVFP4@423acf…` | Two GB10s tensor-parallel | Native CUDA recipe, experimental breadth | \n| Gemma 4 31B MLX | `mlx-community/gemma-4-31B-it-4bit@696d43…` | M5 Ultra or GB10 | Optional DFlash drafter | \n\nThe table also marks conversions that still need Hugging Face downloads plus Python conversion steps before native serve. Treat those rows as **two-phase** workflows, not one command.\n\nIf you are evaluating [Nemotron multimodal servers](https://explainx.ai/blog/nemotron-3-5-super-vl-120b-12b-multimodal-open-agents-nvidia-2026) or [multi-GB10 GLM-5.3 clusters](https://explainx.ai/blog/nvidia-dgx-spark-eight-node-cluster-1tb-glm-5-3-753b-roce-decode-4-43x-switchyard-2026), TensorFold's receipts are useful, but they are **per-model** receipts, not a blanket guarantee.\n\n## TensorFold vs llama.cpp vs Python servers\n\n**Versus llama.cpp.** llama.cpp remains the default for **GGUF** on CPU, Metal, CUDA and Vulkan with enormous community quant coverage. TensorFold focuses on **native MLX checkpoints** and its own CUDA stack, with explicit model recipes and pinned pulls. Choose llama.cpp when your artifact is GGUF and you want maximum forum support. Choose TensorFold when you want the maintainer's qualified MLX or CUDA paths and are willing to live inside their compatibility table.\n\n**Versus vLLM / SGLang.** Production GPU farms still lean on Python servers for MoE scale, radix caching and RL rollouts, as we covered in [vLLM vs SGLang](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026). TensorFold targets **single-node or small-cluster** operators who want a **native binary** without a conda stack. It is closer to \"run this release zip on a DGX Spark\" than \"schedule 200 GPUs in Kubernetes.\"\n\n**Versus Ollama.** Ollama optimizes pull-and-chat UX. TensorFold exposes lower-level flags (`tensorfold capabilities --json`, per-model `tensorfold info REPO`) and expects you to read recipes. That is a feature for SRE-minded builders, friction for casual chat users.\n\nTensorFold Zig LLM server routes Hugging Face checkpoints through Metal or CUDA kernels to an OpenAI-compatible local API for coding agents\n\n## Parallel requests, auth, and production gaps\n\nThe README documents queueing when `--parallel` slots fill: `/health` reports `live.connections` and `live.waiting`. That is enough for a home lab or a single developer laptop, but production teams still need answers the README does not fully centralize:\n\n- **TLS termination** — you front TensorFold with nginx or Caddy.\n- **API keys** — optional key files exist for features like Living Weights; default local serve may run open on localhost.\n- **Multi-node** — Starlight covers fleet policy; the base engine is not a full scheduler like Slurm plus[NVIDIA recovery automation](https://explainx.ai/blog/nvidia-recovery-open-source-automated-training-workload-recovery-slurm-8550-detections-2026) .\n\nTreat November 2026 TensorFold as **early-adopter infrastructure**, not a drop-in replacement for every vLLM deployment.\n\n## Living Weights and experimental flags\n\nThe README devotes significant space to **`--slide`**, which attempts to write new facts into weights during serve. The maintainers label it experimental: interference between facts, recall decay and decode slowdown on Mac when thinking modes are on. Interesting for research demos; risky for production agents without rollback discipline (`tensorfold slide rollback`).\n\nThis is the same caution we apply when covering [weight-editing research](https://explainx.ai/blog/microsoft-lazarus-dormant-neuron-agent-backdoor-weight-editing-2026): cool science, heavy operational caveats.\n\n## Release cadence and verification habits\n\nThe README's Linux quick start pins **tensorfold-1.2.7** tarballs with **sha256sum -c** before extract. That is the right habit for any binary you will put on a GPU host: verify checksum, keep the `bin` and library directories together, and read the startup line that states how many parallel streams the engine accepted. If the process refuses your driver, `nvidia-smi` is the first diagnostic; TensorFold's CUDA path targets **R580+** because its kernels are built against **CUDA 13**.\n\n`tensorfold capabilities --json` is the supported way to see what your build actually compiled in, which matters when a README row says \"Metal only\" or \"CUDA only\" for a given quantization path. Before you open a GitHub issue about a model, run `tensorfold info REPO` against the Hugging Face id; the README uses that command to print a memory floor so you do not serve a 70 GB checkpoint on a 24 GB card.\n\nWhen you outgrow one machine, resist bolting auth and scheduling into the open port yourself until you read [Starlight's install guide](https://github.com/ashhart/tf-starlight/blob/main/docs/install.md?ref=explainx). Starlight is where TensorFold talks about **OIDC**, **SCIM** and fleet budgets. The base engine remains a deliberate **single-node sharp tool**.\n\n## Contributing and agent coordination\n\nTensorFold's README asks contributors to run **Agent Traffic Control (ATC)** before coding so overlapping agent sessions do not collide. That pattern is new in open-source ML infra and worth watching: repositories that expect **coding agents** as first-class contributors need radar tooling, not just CONTRIBUTING.md prose.\n\nIf you automate benchmarking, pair ATC with the project's [evaluating guide](https://github.com/ashhart/TensorFold/blob/main/docs/evaluating.md?ref=explainx) rather than ad hoc prompts.\n\n## What people are asking on GitHub and HN\n\n**Is Vulkan production-ready?** The tagline mentions Vulkan; the quick start and qualification table emphasize Metal and CUDA. Treat Vulkan as advertised surface area with fewer published receipts today.\n\n**Windows?** Native Windows builds live in [docs/WINDOWS.md](https://github.com/ashhart/TensorFold/blob/main/docs/WINDOWS.md?ref=explainx) on the repository; Linux and Mac paths are the documented happy path.\n\n**Why Zig?** Maintainers argue for a single static binary and controlled kernel ownership. Practically, that means faster cold start and fewer pip conflicts, at the cost of a smaller third-party ecosystem than Python.\n\n**How does this relate to xAI dropping JAX/SGLang?** Musk's November Grok 4.8 post described rewriting RL and inference stacks in C++. TensorFold is unrelated product-wise, but it rides the same **compile-the-serving-path** zeitgeist we noted in [Grok 4.8's soft rollout](https://explainx.ai/blog/grok-4-8-soft-rollout-rl-cpp-stack-november-2026).\n\n## Security and supply-chain checklist\n\n1. **Pin pulls** with`@revision` hashes as the README demonstrates; do not serve floating`main` in production.\n2. **Verify release checksums** on Linux tarballs before extract.\n3. **Keep driver floors** — CUDA 13 plus R580 is mandatory on NVIDIA paths.\n4. **Sandbox keys** for`--slide` and API key files; README uses`umask 077` examples.\n5. **Agent access** — if you aim OpenCode or Claude Code at a local TensorFold endpoint, treat it like any local server: no inbound exposure without auth.\n\nFor broader agent hardening patterns, see explainx.ai coverage of [Claude Code sandboxing](https://explainx.ai/blog/claude-cage-vm-sandbox-claude-code-network-isolation-2026) and [MCP dependency drift](https://explainx.ai/blog/scoutcode-agent-harness-free-model-fallbacks-november-2026).\n\n## Related reading\n\n*Commands and version examples reflect the TensorFold README and release tags as of November 12, 2026. Verify `tensorfold --version`, driver levels and model rows before production cutover.*", "url": "https://wpnews.pro/news/tensorfold-zig-llm-server-for-metal-and-cuda-with-an-openai-compatible-api", "canonical_source": "https://explainx.ai/blog/tensorfold-zig-local-llm-inference-metal-cuda-api-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-12 00:00:00+00:00", "updated_at": "2026-11-12 12:53:36.490310+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools", "ai-products"], "entities": ["TensorFold", "ashhart", "Apple Metal", "NVIDIA CUDA", "Qwen3.5-2B-MLX-4bit", "TensorFold Starlight", "Homebrew", "Vulkan"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tensorfold-zig-llm-server-for-metal-and-cuda-with-an-openai-compatible-api", "markdown": "https://wpnews.pro/news/tensorfold-zig-llm-server-for-metal-and-cuda-with-an-openai-compatible-api.md", "text": "https://wpnews.pro/news/tensorfold-zig-llm-server-for-metal-and-cuda-with-an-openai-compatible-api.txt", "jsonld": "https://wpnews.pro/news/tensorfold-zig-llm-server-for-metal-and-cuda-with-an-openai-compatible-api.jsonld"}}