{"slug": "zml-llmd-run-llms-on-any-chip-no-nvidia-required", "title": "ZML LLMD: Run LLMs on Any Chip — No NVIDIA Required", "summary": "ZML launched LLMD, a free inference server that runs LLaMA, Gemma, Qwen, and Mistral models on NVIDIA, AMD, Google TPU, Intel, and Apple hardware from a single Docker image. Built in Zig and compiled via MLIR and OpenXLA, it offers continuous batching, paged attention, and tensor parallel sharding across all backends. The Paris-based startup, backed by Docker and Hugging Face founders, aims to reduce vendor lock-in as NVIDIA hardware faces long lead times.", "body_md": "Most LLM inference servers are NVIDIA-first, with AMD support buried in a README disclaimer. ZML just shipped [LLMD](https://zml.ai/posts/llmd/), a free inference server that runs LLaMA, Gemma, Qwen, and Mistral models on NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI, and Apple Metal — from the same Docker image. It launched July 8 in alpha, and the people who built Docker and Hugging Face wrote the checks.\n\n## One Server, Five Chip Backends\n\nThe promise is blunt: swap the Docker tag, swap the hardware. Running on NVIDIA looks like this:\n\n```\ndocker run -p 8000:8000 --gpus=all -e HF_TOKEN   -it zmlai/llmd:cuda --model=hf://Qwen/Qwen3-8B\n```\n\nSwitch to AMD by replacing `:cuda`\n\nwith `:rocm`\n\nand adjusting the device flags. The API stays OpenAI-compatible on port 8000. Nothing else changes.\n\nThat sounds obvious until you try to actually do it with vLLM or SGLang. Those servers wrap Python around CUDA kernels — AMD works, but it’s second-class. ZML took a different approach: LLMD is built in Zig, compiled ahead-of-time via MLIR and OpenXLA into native binaries for each target. No Python runtime. No JIT compilation in the hot path. Flat, predictable latency across all five backends.\n\n## Full-Featured Across All Backends\n\nThe chip portability would mean little if features degraded across backends. ZML claims the following work identically on NVIDIA, AMD, TPU, Intel, and Metal:\n\n- Continuous batching\n- Paged attention (DFlash)\n- Tensor parallel sharding (automatic multi-device)\n- Prefix caching\n- Tool calling\n- Prometheus metrics (\n`/metrics`\n\n)\n\nTensor parallel is the notable one — LLMD automatically shards models across multiple devices and handles device communication transparently, on any supported backend. Most multi-GPU solutions require backend-specific tuning. Model loading is zero-copy from HuggingFace Hub (`hf://`\n\n), S3, or GCS — no pre-download required. The container weighs 2.4 GB.\n\n## Who Built This, and Why It Matters\n\nZML is a 20-person startup in Paris founded by Steeve Morin, former VP of Engineering at Zenly before its Snap acquisition. They raised $20 million from 20VC, LocalGlobe, and Xavier Niel’s Kima Ventures. The angel round reads like a who’s-who of AI infrastructure: Solomon Hykes (Docker and Dagger creator), Clément Delangue and Julien Chaumond (Hugging Face co-founders), and Yann LeCun (Meta Chief AI Scientist, Turing Award winner).\n\nWhen the person who invented the container and the people who built the dominant open-source model hub bet on an inference startup, that is a credibility signal worth tracking.\n\n## The Case for Multi-Chip Now\n\nThe timing is deliberate. H100 SXM5 nodes are sitting at 36-52 week lead times from resellers. NVIDIA Blackwell is sold out until 2027. Meanwhile, AMD GPUs, Google TPUs, and Intel Arc accelerators sit underutilized — not because they can’t run models, but because the software to do it cleanly doesn’t exist yet. LLMD is a bet that it soon will.\n\nEnterprise teams running on NVIDIA exclusively pay a vendor premium with no leverage. If you can run inference equally well on AMD and NVIDIA, you can negotiate with both. Community benchmarks show AMD RX 7900 XTX running at roughly 80–90% of RTX 4090 throughput for comparable model sizes — at significantly lower hardware cost. The gap is narrower than most NVIDIA-first teams assume.\n\n## What to Know Before You Try It\n\nLLMD is alpha software. ZML says explicitly it is not intended for production workloads yet. The container is available on [Docker Hub](https://hub.docker.com/r/zmlai/llmd), and the [Hugging Face community test-drive](https://huggingface.co/blog/erikkaum/test-driving-llmd-inference-engine) is worth reading before you start.\n\n“Free” does not mean “open source.” LLMD is free to use while ZML figures out its monetization. The underlying ZML framework is open source on GitHub — the server itself is not, yet. That distinction matters for teams that need to audit what they deploy.\n\nFor infrastructure teams watching the multi-chip inference space: [TechCrunch’s coverage](https://techcrunch.com/2026/07/08/hot-french-startup-zml-releases-free-product-to-speed-inference-across-lots-of-ai-chips/) and ZML’s own announcement are the right starting points. The alpha is available now. Given the backers and the architecture, the clock from “alpha” to “production-ready” is probably moving faster than the version number suggests.", "url": "https://wpnews.pro/news/zml-llmd-run-llms-on-any-chip-no-nvidia-required", "canonical_source": "https://byteiota.com/zml-llmd-run-llms-on-any-chip-no-nvidia-required/", "published_at": "2026-07-09 14:09:12+00:00", "updated_at": "2026-07-09 14:16:57.219890+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "ai-startups", "ai-products"], "entities": ["ZML", "LLMD", "NVIDIA", "AMD", "Google TPU", "Intel", "Apple Metal", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/zml-llmd-run-llms-on-any-chip-no-nvidia-required", "markdown": "https://wpnews.pro/news/zml-llmd-run-llms-on-any-chip-no-nvidia-required.md", "text": "https://wpnews.pro/news/zml-llmd-run-llms-on-any-chip-no-nvidia-required.txt", "jsonld": "https://wpnews.pro/news/zml-llmd-run-llms-on-any-chip-no-nvidia-required.jsonld"}}