vllm locally on 5060Ti 16GB x 2

A developer deployed vLLM locally on two NVIDIA RTX 5060 Ti 16GB GPUs using Docker Compose, configuring tensor parallelism, FP8 KV cache, and speculative decoding with MTP. The setup runs an OpenAI-compatible server with tool-call and reasoning parsers, achieving efficient LLM inference on consumer hardware.

| services: | | vllm-server: | | image: vllm/vllm-openai:latest | | container name: vllm-server | | restart: unless-stopped | | ports: | | - "${VLLM PORT}:8000" | | environment: | | - VLLM FP4 GEMM BACKEND=cutlass | | - NVIDIA VISIBLE DEVICES=all | | - NVIDIA DRIVER CAPABILITIES=compute,utility | | volumes: | | - ${HF CACHE DIR}:/root/.cache/huggingface | | ipc: host | | shm size: '16gb' | | deploy: | | resources: | | reservations: | | devices: | | - driver: nvidia | | count: all | | capabilities: gpu | | command: | | --model ${VLLM MODEL} | | --served-model-name ${VLLM SERVED NAME} | | --tensor-parallel-size ${VLLM TP SIZE} | | --gpu-memory-utilization ${VLLM MEMORY UTIL} | | --max-model-len ${VLLM MAX MODEL LEN} | | --max-num-seqs ${VLLM MAX NUM SEQS} | | --api-key ${API KEY} | | --language-model-only | | --enable-auto-tool-choice | | --generation-config vllm | | --trust-remote-code | | --kv-cache-dtype fp8 | | --reasoning-parser ${VLLM REASON PARSER} | | --tool-call-parser ${VLLM TOOL PARSER} | | --speculative-config '{"method": "mtp", "num speculative tokens": 2}' | | --default-chat-template-kwargs '{"preserve thinking": true}' | | --host 0.0.0.0 |