Most LLM inference servers are NVIDIA-first, with AMD support buried in a README disclaimer. ZML just shipped LLMD, a free inference server that runs LLaMA, Gemma, Qwen, and Mistral models on NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI, and Apple Metal — from the same Docker image. It launched July 8 in alpha, and the people who built Docker and Hugging Face wrote the checks.
One Server, Five Chip Backends #
The promise is blunt: swap the Docker tag, swap the hardware. Running on NVIDIA looks like this:
docker run -p 8000:8000 --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda --model=hf://Qwen/Qwen3-8B
Switch to AMD by replacing :cuda
with :rocm
and adjusting the device flags. The API stays OpenAI-compatible on port 8000. Nothing else changes.
That sounds obvious until you try to actually do it with vLLM or SGLang. Those servers wrap Python around CUDA kernels — AMD works, but it’s second-class. ZML took a different approach: LLMD is built in Zig, compiled ahead-of-time via MLIR and OpenXLA into native binaries for each target. No Python runtime. No JIT compilation in the hot path. Flat, predictable latency across all five backends.
Full-Featured Across All Backends #
The chip portability would mean little if features degraded across backends. ZML claims the following work identically on NVIDIA, AMD, TPU, Intel, and Metal:
- Continuous batching
- Paged attention (DFlash)
- Tensor parallel sharding (automatic multi-device)
- Prefix caching
- Tool calling
- Prometheus metrics (
/metrics
)
Tensor parallel is the notable one — LLMD automatically shards models across multiple devices and handles device communication transparently, on any supported backend. Most multi-GPU solutions require backend-specific tuning. Model is zero-copy from HuggingFace Hub (hf://
), S3, or GCS — no pre-download required. The container weighs 2.4 GB.
Who Built This, and Why It Matters #
ZML is a 20-person startup in Paris founded by Steeve Morin, former VP of Engineering at Zenly before its Snap acquisition. They raised $20 million from 20VC, LocalGlobe, and Xavier Niel’s Kima Ventures. The angel round reads like a who’s-who of AI infrastructure: Solomon Hykes (Docker and Dagger creator), Clément Delangue and Julien Chaumond (Hugging Face co-founders), and Yann LeCun (Meta Chief AI Scientist, Turing Award winner).
When the person who invented the container and the people who built the dominant open-source model hub bet on an inference startup, that is a credibility signal worth tracking.
The Case for Multi-Chip Now #
The timing is deliberate. H100 SXM5 nodes are sitting at 36-52 week lead times from resellers. NVIDIA Blackwell is sold out until 2027. Meanwhile, AMD GPUs, Google TPUs, and Intel Arc accelerators sit underutilized — not because they can’t run models, but because the software to do it cleanly doesn’t exist yet. LLMD is a bet that it soon will.
Enterprise teams running on NVIDIA exclusively pay a vendor premium with no leverage. If you can run inference equally well on AMD and NVIDIA, you can negotiate with both. Community benchmarks show AMD RX 7900 XTX running at roughly 80–90% of RTX 4090 throughput for comparable model sizes — at significantly lower hardware cost. The gap is narrower than most NVIDIA-first teams assume.
What to Know Before You Try It #
LLMD is alpha software. ZML says explicitly it is not intended for production workloads yet. The container is available on Docker Hub, and the Hugging Face community test-drive is worth reading before you start.
“Free” does not mean “open source.” LLMD is free to use while ZML figures out its monetization. The underlying ZML framework is open source on GitHub — the server itself is not, yet. That distinction matters for teams that need to audit what they deploy.
For infrastructure teams watching the multi-chip inference space: TechCrunch’s coverage and ZML’s own announcement are the right starting points. The alpha is available now. Given the backers and the architecture, the clock from “alpha” to “production-ready” is probably moving faster than the version number suggests.