# TensorFold: Zig LLM Server for Metal and CUDA With an OpenAI-Compatible API

> Source: <https://explainx.ai/blog/tensorfold-zig-local-llm-inference-metal-cuda-api-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-12 00:00:00+00:00

**TensorFold is a Zig-native LLM inference engine that serves checkpoints on Apple Metal and NVIDIA CUDA from one binary, exposes OpenAI-style HTTP APIs on port 8080, and does not require Python or MLX at serving time according to its README.**

The [TensorFold GitHub repository](https://github.com/ashhart/TensorFold?ref=explainx) trended in mid-November 2026 as developers looked for alternatives to Python-heavy GPU servers. Stars climbed quickly because the README is unusually concrete: pinned Hugging Face pulls, release tarballs, qualified model rows and copy-paste curl tests. This post explains what shipped, how to try the documented quick start, and where TensorFold sits next to [llama.cpp](https://explainx.ai/blog/what-is-llama-cpp-run-models-locally-2026), [vLLM versus SGLang](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026), and Ollama-style wrappers.

## TL;DR: TensorFold quick answers

| Question | Answer | 
|---|---|
| Do I need Python to serve? | README says serving needs no Python or MLX install | 
| Default API port | `8080` with`/v1/chat/completions` | 
| Mac install path | Homebrew tap `ashhart/tensorfold/tensorfold` | 
| Linux NVIDIA path | Release archive `tensorfold-1.2.7-linux-*` plus`tensorfold-native` wrapper | 
| Smallest documented start | `mlx-community/Qwen3.5-2B-MLX-4bit` with pinned revision | 
| Enterprise fleet control | Separate [TensorFold Starlight](https://github.com/ashhart/tf-starlight?ref=explainx) repo | 
| Agent contributor etiquette | [Agent Traffic Control](https://github.com/ashhart/ATC?ref=explainx) before editing upstream | 

## What TensorFold actually ships

The project's one-line pitch on GitHub is an **LLM inference engine for Metal, CUDA and Vulkan**. Under the hood the maintainers describe a **Zig binary** that owns checkpoint IO, tokenization and kernel dispatch. That design choice matters for operators who are tired of conda environments breaking CUDA wheels on upgrade weeks.

**Serving surface.** After `tensorfold serve`, clients can hit standard OpenAI chat completions, plus Responses and Anthropic Messages endpoints listed in the README. `/v1/models` advertises the served alias you choose with `--name`. Parallel request caps exist: `--parallel N` or `--parallel auto` (up to eight streams when memory allows). Health metadata exposes active and queued connections when authentication is off.

**Distribution.** Apple Silicon users get a Homebrew formula. Linux users pull gzipped release bundles with `tensorfold-native` as the entry binary. The README pins **v1.2.7** in its example and demands **NVIDIA driver R580 or newer** for CUDA 13 kernels. Those constraints are strict: TensorFold refuses older drivers at startup rather than failing mid-request.

**Ecosystem extras beyond bare serving.**

- **TensorFold Web** — a local launcher for picking models (`tensorfold web` ).
- **Mac desktop app** — beta DMG in the`app/` tree bundling workers and metrics.
- **Living Weights (`--slide`)** — experimental on-device weight edits for Nemotron checkpoints; README warns about recall limits and speed tax.
- **Starlight** — enterprise multi-node governance in another repository.

None of those change the core news hook: a **native server** with first-class **Metal and CUDA** paths and an **OpenAI-compatible** wire format.

TensorFold local LLM inference serves open-weight models on Apple Metal or NVIDIA CUDA from a Zig binary with an OpenAI-compatible HTTP API

## Quick start: reproduce the README path

The maintainers standardize on **Qwen3.5-2B MLX 4-bit** because it is small and checked on Metal plus GB10 class hardware.

### Apple Silicon

```
brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking
```

### Linux with NVIDIA GPU

Download the platform archive, verify sha256, extract, then alias `tensorfold` to `tensorfold-native` exactly as the README shows for aarch64 GB10 or x86_64 hosts. Pull the same pinned Qwen revision, then serve with the same flags.

### Smoke test

```
curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'
```

If that returns JSON, your agent harness can point `baseURL` at `http://127.0.0.1:8080/v1` the same way we document for [llama-server in the llama.cpp guide](https://explainx.ai/blog/what-is-llama-cpp-run-models-locally-2026).

## Model qualification table: read before you benchmark

TensorFold is not a universal GGUF runner. The README's **Pick a model** section is a compatibility contract: each row lists **download command**, **machine scope**, and **native recipe** links.

Representative rows developers asked about in early November threads:

| Checkpoint family | Pull example | Hardware called out | Notes | 
|---|---|---|---|
| Qwen3.5-2B MLX 4-bit | `mlx-community/Qwen3.5-2B-MLX-4bit@93760…` | Mac M5 Max or GB10 | Documented quick start | 
| Qwen3.8-27B MLX | `TensorFold/Qwen3.8-27B-MLX-4bit` | Mac with enough wired GPU RAM | May need `iogpu.wired_limit_mb` on 32 GB Macs | 
| Nemotron 3.5 Lightning 30B-A3B | `TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit` | Mac M1–M5 or GB10 | Includes MTP path | 
| GLM-5.3-Flash NVFP4 | `nvidia/GLM-5.3-Flash-NVFP4@423acf…` | Two GB10s tensor-parallel | Native CUDA recipe, experimental breadth | 
| Gemma 4 31B MLX | `mlx-community/gemma-4-31B-it-4bit@696d43…` | M5 Ultra or GB10 | Optional DFlash drafter | 

The table also marks conversions that still need Hugging Face downloads plus Python conversion steps before native serve. Treat those rows as **two-phase** workflows, not one command.

If you are evaluating [Nemotron multimodal servers](https://explainx.ai/blog/nemotron-3-5-super-vl-120b-12b-multimodal-open-agents-nvidia-2026) or [multi-GB10 GLM-5.3 clusters](https://explainx.ai/blog/nvidia-dgx-spark-eight-node-cluster-1tb-glm-5-3-753b-roce-decode-4-43x-switchyard-2026), TensorFold's receipts are useful, but they are **per-model** receipts, not a blanket guarantee.

## TensorFold vs llama.cpp vs Python servers

**Versus llama.cpp.** llama.cpp remains the default for **GGUF** on CPU, Metal, CUDA and Vulkan with enormous community quant coverage. TensorFold focuses on **native MLX checkpoints** and its own CUDA stack, with explicit model recipes and pinned pulls. Choose llama.cpp when your artifact is GGUF and you want maximum forum support. Choose TensorFold when you want the maintainer's qualified MLX or CUDA paths and are willing to live inside their compatibility table.

**Versus vLLM / SGLang.** Production GPU farms still lean on Python servers for MoE scale, radix caching and RL rollouts, as we covered in [vLLM vs SGLang](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026). TensorFold targets **single-node or small-cluster** operators who want a **native binary** without a conda stack. It is closer to "run this release zip on a DGX Spark" than "schedule 200 GPUs in Kubernetes."

**Versus Ollama.** Ollama optimizes pull-and-chat UX. TensorFold exposes lower-level flags (`tensorfold capabilities --json`, per-model `tensorfold info REPO`) and expects you to read recipes. That is a feature for SRE-minded builders, friction for casual chat users.

TensorFold Zig LLM server routes Hugging Face checkpoints through Metal or CUDA kernels to an OpenAI-compatible local API for coding agents

## Parallel requests, auth, and production gaps

The README documents queueing when `--parallel` slots fill: `/health` reports `live.connections` and `live.waiting`. That is enough for a home lab or a single developer laptop, but production teams still need answers the README does not fully centralize:

- **TLS termination** — you front TensorFold with nginx or Caddy.
- **API keys** — optional key files exist for features like Living Weights; default local serve may run open on localhost.
- **Multi-node** — Starlight covers fleet policy; the base engine is not a full scheduler like Slurm plus[NVIDIA recovery automation](https://explainx.ai/blog/nvidia-recovery-open-source-automated-training-workload-recovery-slurm-8550-detections-2026) .

Treat November 2026 TensorFold as **early-adopter infrastructure**, not a drop-in replacement for every vLLM deployment.

## Living Weights and experimental flags

The README devotes significant space to **`--slide`**, which attempts to write new facts into weights during serve. The maintainers label it experimental: interference between facts, recall decay and decode slowdown on Mac when thinking modes are on. Interesting for research demos; risky for production agents without rollback discipline (`tensorfold slide rollback`).

This is the same caution we apply when covering [weight-editing research](https://explainx.ai/blog/microsoft-lazarus-dormant-neuron-agent-backdoor-weight-editing-2026): cool science, heavy operational caveats.

## Release cadence and verification habits

The README's Linux quick start pins **tensorfold-1.2.7** tarballs with **sha256sum -c** before extract. That is the right habit for any binary you will put on a GPU host: verify checksum, keep the `bin` and library directories together, and read the startup line that states how many parallel streams the engine accepted. If the process refuses your driver, `nvidia-smi` is the first diagnostic; TensorFold's CUDA path targets **R580+** because its kernels are built against **CUDA 13**.

`tensorfold capabilities --json` is the supported way to see what your build actually compiled in, which matters when a README row says "Metal only" or "CUDA only" for a given quantization path. Before you open a GitHub issue about a model, run `tensorfold info REPO` against the Hugging Face id; the README uses that command to print a memory floor so you do not serve a 70 GB checkpoint on a 24 GB card.

When you outgrow one machine, resist bolting auth and scheduling into the open port yourself until you read [Starlight's install guide](https://github.com/ashhart/tf-starlight/blob/main/docs/install.md?ref=explainx). Starlight is where TensorFold talks about **OIDC**, **SCIM** and fleet budgets. The base engine remains a deliberate **single-node sharp tool**.

## Contributing and agent coordination

TensorFold's README asks contributors to run **Agent Traffic Control (ATC)** before coding so overlapping agent sessions do not collide. That pattern is new in open-source ML infra and worth watching: repositories that expect **coding agents** as first-class contributors need radar tooling, not just CONTRIBUTING.md prose.

If you automate benchmarking, pair ATC with the project's [evaluating guide](https://github.com/ashhart/TensorFold/blob/main/docs/evaluating.md?ref=explainx) rather than ad hoc prompts.

## What people are asking on GitHub and HN

**Is Vulkan production-ready?** The tagline mentions Vulkan; the quick start and qualification table emphasize Metal and CUDA. Treat Vulkan as advertised surface area with fewer published receipts today.

**Windows?** Native Windows builds live in [docs/WINDOWS.md](https://github.com/ashhart/TensorFold/blob/main/docs/WINDOWS.md?ref=explainx) on the repository; Linux and Mac paths are the documented happy path.

**Why Zig?** Maintainers argue for a single static binary and controlled kernel ownership. Practically, that means faster cold start and fewer pip conflicts, at the cost of a smaller third-party ecosystem than Python.

**How does this relate to xAI dropping JAX/SGLang?** Musk's November Grok 4.8 post described rewriting RL and inference stacks in C++. TensorFold is unrelated product-wise, but it rides the same **compile-the-serving-path** zeitgeist we noted in [Grok 4.8's soft rollout](https://explainx.ai/blog/grok-4-8-soft-rollout-rl-cpp-stack-november-2026).

## Security and supply-chain checklist

1. **Pin pulls** with`@revision` hashes as the README demonstrates; do not serve floating`main` in production.
2. **Verify release checksums** on Linux tarballs before extract.
3. **Keep driver floors** — CUDA 13 plus R580 is mandatory on NVIDIA paths.
4. **Sandbox keys** for`--slide` and API key files; README uses`umask 077` examples.
5. **Agent access** — if you aim OpenCode or Claude Code at a local TensorFold endpoint, treat it like any local server: no inbound exposure without auth.

For broader agent hardening patterns, see explainx.ai coverage of [Claude Code sandboxing](https://explainx.ai/blog/claude-cage-vm-sandbox-claude-code-network-isolation-2026) and [MCP dependency drift](https://explainx.ai/blog/scoutcode-agent-harness-free-model-fallbacks-november-2026).

## Related reading

*Commands and version examples reflect the TensorFold README and release tags as of November 12, 2026. Verify `tensorfold --version`, driver levels and model rows before production cutover.*