The Universal LLM Server

Continuous batching

Serve concurrent requests efficiently without hand-building a scheduler around the model.

ZML/LLMD

A self-contained inference server for LLaMa, Gemma, Qwen and Mistral models, running across NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal.

docker run -p 8000:8000 \
  --shm-size=256GB --gpus=all \
  -e HF_TOKEN -it zmlai/llmd:cuda \
  --model=hf://Qwen/Qwen3-8B

Serving

Run long-context workloads with the memory behavior expected from a production server.

Shard model execution across multiple devices while LLMD handles communication.

Reuse shared prompt prefixes across requests to reduce repeated work.

Expose modern assistant workflows without leaving the LLMD serving path.

                    Monitor the server directly through the built-in
                    `/metrics` endpoint.

Native DFlash support ships for Gemma 4 series models, with Qwen support planned.

Try it

docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
  --model=hf://Qwen/Qwen3.6-27B
docker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \
  --model=hf://Qwen/Qwen3.6-27B
docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \
  --model=hf://Qwen/Qwen3.6-27B
docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \
  --model=hf://Qwen/Qwen3.6-27B
brew install zml/zml/llmd
llmd --model=hf://Qwen/Qwen3.6-27B

Storage

                LLMD uses ZML's VFS subsystem to load from Hugging Face, S3
                and GCS without a separate download step. Use
                `hf://`, `s3://` or
                `gs://` anywhere a model path is expected.

Platforms