LLMD: Run LLM Inference on Any Chip, One Docker Tag

ZML, a Paris startup, released LLMD on July 8, a free, open-source inference server that runs LLaMA, Gemma, Qwen, and Mistral on NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI, and Apple Metal from a single Docker image, requiring only a tag change for different hardware. The server, built in Zig with MLIR and Bazel, offers an OpenAI-compatible API and features like continuous batching and paged attention, but is currently single-GPU with a max batch size of 16. ZML raised $20 million in July 2026 from 20VC, Kima Ventures, LocalGlobe, and Puzzle Ventures to develop the product.

If you’ve ever priced out running LLM inference on AMD hardware or a Google TPU and then looked at what it would actually take to port your stack there, you know the answer is usually “not worth it.” Every serious inference setup today is wired to NVIDIA CUDA — which means switching chips means rewriting code. ZML, a Paris startup, shipped LLMD https://zml.ai/posts/llmd/ on July 8 — a free, open-source inference server that replaces that rewrite with a Docker tag swap. LLMD runs LLaMA, Gemma, Qwen, and Mistral on NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI, and Apple Metal — from the same image, with the same OpenAI-compatible API. Your client code does not change. Only the image tag does. One Command, Five Chip Families Here is what the portability looks like in practice: NVIDIA CUDA docker run -p 8000:8000 --gpus=all -e HF TOKEN \ -it zmlai/llmd --model=hf://Qwen/Qwen3-8B AMD ROCm -- change the tag, keep everything else docker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF TOKEN \ -it zmlai/llmd:rocm --model=hf://Qwen/Qwen3-8B Google TPU docker run --net=host --privileged -e HF TOKEN \ -it zmlai/llmd:tpu --model=hf://Qwen/Qwen3-8B The API endpoint, the model path, the client library — none of that changes. If you are using the OpenAI Python SDK pointed at http://localhost:8000 , it works identically across all three. That is the point. Why Hardware Lock-In Is the Real Problem The reason this matters is that AI inference hardware is fragmenting fast. AWS Trainium 3 is real competition on cost — but it requires Amazon’s Neuron SDK, which means your stack only runs on AWS. Google’s TPU Ironwood requires JAX. AMD’s MI300X offers excellent VRAM capacity for large models at a fraction of H100 pricing, but ROCm support in most inference frameworks is second-class at best. The result: teams that could save 40–60% on inference costs by moving to alternative hardware cannot, because the switching cost is too high. You would be rewriting your serving layer, revalidating your model behavior, and retraining your ops team. ZML’s bet is that a chip-agnostic inference server breaks that lock-in at the infrastructure layer — before any of that becomes your problem. What’s Under the Hood LLMD is not a Python wrapper over CUDA kernels. ZML built their stack in Zig, using MLIR and Bazel, compiling model computation graphs into native binaries for each target platform via OpenXLA. This is a fundamentally different architecture than vLLM https://docs.vllm.ai/en/latest/ or TGI, which are Python-first and NVIDIA-first by design. The feature set is solid for an inference server: continuous batching, paged attention, tensor-parallel sharding, tool calling, and Prometheus metrics. The container clocks in at 2.4 GB — compact enough for fast autoscaling. The API is OpenAI-compatible, so dropping LLMD behind an existing codebase requires no client-side changes. The Caveats You Need to Know ZML is calling this a technical preview, and they mean it. LLMD is single-GPU only right now — no multi-GPU sharding. The maximum batch size is 16. There are no published throughput benchmarks against vLLM which hits 793 tokens per second in Red Hat’s testing . The supported model list is limited to LLaMA, Gemma, Qwen, and Mistral. If you need production inference today, you are still on vLLM https://github.com/vllm-project/vllm for NVIDIA or Ollama for local dev. LLMD does not replace either yet. What it does is give platform engineers a real option to test cross-chip portability without building the abstraction layer themselves. ZML raised $20 million in July 2026 — from 20VC, Kima Ventures, LocalGlobe, and Puzzle Ventures — which suggests they have runway to get LLMD to production-grade. The roadmap presumably includes multi-GPU support, broader model coverage, and performance parity with vLLM on NVIDIA. Should You Try It? If you are a platform engineer managing LLM inference costs, yes — spin it up today in a test environment on AMD or TPU hardware and see what the portability story looks like in practice. The 2.4 GB image pulls fast. The OpenAI API compatibility means your existing test suite runs against it without changes. The problem LLMD is solving is real enough that you want to understand the tool before you need it. If you are running production inference, stay on vLLM for now. But watch ZML closely. The cross-chip inference server gap is real, it is getting more expensive to ignore, and nobody else has shipped a credible answer to it yet. The repo is public https://github.com/zml/zml — worth a star and a bookmark.