{"slug": "vllm-locally-on-5060ti-16gb-x-2", "title": "vllm locally on 5060Ti 16GB x 2", "summary": "A developer deployed vLLM locally on two NVIDIA RTX 5060 Ti 16GB GPUs using Docker Compose, configuring tensor parallelism, FP8 KV cache, and speculative decoding with MTP. The setup runs an OpenAI-compatible server with tool-call and reasoning parsers, achieving efficient LLM inference on consumer hardware.", "body_md": "|\nservices: |\n|\nvllm-server: |\n|\nimage: vllm/vllm-openai:latest |\n|\ncontainer_name: vllm-server |\n|\nrestart: unless-stopped |\n|\nports: |\n|\n- \"${VLLM_PORT}:8000\" |\n|\nenvironment: |\n|\n- VLLM_FP4_GEMM_BACKEND=cutlass |\n|\n- NVIDIA_VISIBLE_DEVICES=all |\n|\n- NVIDIA_DRIVER_CAPABILITIES=compute,utility |\n|\nvolumes: |\n|\n- ${HF_CACHE_DIR}:/root/.cache/huggingface |\n|\nipc: host |\n|\nshm_size: '16gb' |\n|\ndeploy: |\n|\nresources: |\n|\nreservations: |\n|\ndevices: |\n|\n- driver: nvidia |\n|\ncount: all |\n|\ncapabilities: [gpu] |\n|\ncommand: > |\n|\n--model ${VLLM_MODEL} |\n|\n--served-model-name ${VLLM_SERVED_NAME} |\n|\n--tensor-parallel-size ${VLLM_TP_SIZE} |\n|\n--gpu-memory-utilization ${VLLM_MEMORY_UTIL} |\n|\n--max-model-len ${VLLM_MAX_MODEL_LEN} |\n|\n--max-num-seqs ${VLLM_MAX_NUM_SEQS} |\n|\n--api-key ${API_KEY} |\n|\n--language-model-only |\n|\n--enable-auto-tool-choice |\n|\n--generation-config vllm |\n|\n--trust-remote-code |\n|\n--kv-cache-dtype fp8 |\n|\n--reasoning-parser ${VLLM_REASON_PARSER} |\n|\n--tool-call-parser ${VLLM_TOOL_PARSER} |\n|\n--speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 2}' |\n|\n--default-chat-template-kwargs '{\"preserve_thinking\": true}' |\n|\n--host 0.0.0.0 |", "url": "https://wpnews.pro/news/vllm-locally-on-5060ti-16gb-x-2", "canonical_source": "https://gist.github.com/bouroo/71d5dfe6131880956ff8e0cb5cc4105c", "published_at": "2026-07-12 05:51:10+00:00", "updated_at": "2026-07-12 06:09:07.285505+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["vLLM", "NVIDIA", "RTX 5060 Ti", "Docker", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/vllm-locally-on-5060ti-16gb-x-2", "markdown": "https://wpnews.pro/news/vllm-locally-on-5060ti-16gb-x-2.md", "text": "https://wpnews.pro/news/vllm-locally-on-5060ti-16gb-x-2.txt", "jsonld": "https://wpnews.pro/news/vllm-locally-on-5060ti-16gb-x-2.jsonld"}}