{"slug": "on-prem-llm-inference", "title": "On-Prem LLM Inference", "summary": "A technical guide details the production deployment of on-premises large language model inference using vLLM, covering an 8-GPU node running GLM-5.2/5.3 (NVFP4 MoE) and a single L40S running gemma-4-26B FP8, with specific version requirements and patches. The article includes installation commands, model downloads, and workarounds for FlashInfer and CUDA graph memory issues, emphasizing stable configurations for 256K context and MTP speculative decoding.", "body_md": "# vLLM serving, production shape\n\nContext: on-prem LLM inference — the kind where the data cannot leave. Two serving profiles: an 8-GPU node running GLM-5.2/5.3 (NVFP4 MoE, expert parallel, 256K context), and a single L40S running gemma-4-26B FP8. Ubuntu 24.04, vLLM in a uv-managed venv, systemd on top. The front door in front of these engines — keys, routing, TLS — is the [LiteLLM proxy entry](../litellm-proxy/). Hostnames genericized; the API key below is a placeholder.\n\n## Install\n\n```\ncurl -LsSf https://astral.sh/uv/install.sh | sh\nsource $HOME/.local/bin/env\nuv venv --python 3.12 --seed\nsource .venv/bin/activate\nuv pip install vllm --torch-backend=auto\n```\n\nThe unglamorous apt layer underneath: `ffmpeg`\n\n(torchcodec wants it), `numactl`\n\n(NUMA binding), `build-essential`\n\n+ `python3.12-dev`\n\n(Triton compiles), `ninja-build`\n\n.\n\n## Models\n\n```\npip install -U huggingface_hub\nhf download nvidia/GLM-5.2-NVFP4 --local-dir /models/GLM-5.2-NVFP4\nhf download google/gemma-4-26B-A4B-it --local-dir /models/gemma-4-26B-A4B-it\nhf download google/gemma-4-26B-A4B-it-assistant ...   # MTP draft model\nhf download Qwen/Qwen2.5-VL-72B-Instruct-AWQ ...      # vision\n```\n\n## What bit\n\n-\n— the pinned FlashInfer is older than the vLLM calling it. Upgrade FlashInfer and tell vLLM to stop checking versions:`trtllm_batch_decode_with_kv_cache_mla() got an unexpected keyword argument 'kv_scale_format'`\n\n```\npip install -U flashinfer-python flashinfer-cubin\nexport FLASHINFER_DISABLE_VERSION_CHECK=1\n```\n\n-\n— a literally half-finished backend class for the SM120 GPU. Patch it: back up the file, apply the patch script, restart vLLM.`Half finished FlashInferMLASparseSM120Impl`\n\n-\n**The CUDA graph memory hint nobody reads.** At`--gpu-memory-utilization 0.95`\n\n, vLLM logs that with CUDA graph memory profiling this is*effectively*0.9124 — and suggests 0.9876 for the same usable KV cache. Tempting; 0.98 OOMs at 256K context on this hardware. 0.95 with`--max-model-len 256K`\n\nis the stable point. -\n**Model upgrades pull vLLM upgrades.** GLM-5.3-NVFP4 needed vLLM 0.28.0; the node ran 0.25.0. Back up the whole venv first (`cp -a .venv .venv-vllm025-backup`\n\n), then:\n\n```\npip install -U \"vllm>=0.28.0\" --extra-index-url https://flashinfer.ai/whl\npip install -U flashinfer-cubin==0.6.16.post3 --extra-index-url https://flashinfer.ai/whl\n```\n\n-\n**MTP on gemma needed a patch** to`vllm/v1/spec_decode/llm_base_proposer.py`\n\n— the draft model's embedding path. Patch script writes its own`.bak`\n\n; restart vLLM after.\n\n## The serving commands\n\n8-GPU node, the final stable shape:\n\n```\nvllm serve /models/GLM-5.2-NVFP4 --served-model-name glm-5.2 \\\n  --tensor-parallel-size 8 --enable-expert-parallel \\\n  --disable-custom-all-reduce --numa-bind --performance-mode interactivity \\\n  --trust-remote-code --kv-cache-dtype fp8_e4m3 \\\n  --gpu-memory-utilization 0.95 --max-model-len 256K \\\n  --host <ip> --port 8000\n```\n\nSingle L40S, FP8, with MTP speculative decoding from the draft model:\n\n```\nvllm serve /models/gemma-4-26B-A4B-it --quantization fp8 \\\n  --served-model-name gemma-4-26b-a4b --tensor-parallel-size 1 \\\n  --gpu-memory-utilization 0.93 --kv-cache-dtype fp8 --max-model-len 32768 \\\n  --max-num-seqs 64 --enable-prefix-caching \\\n  --cudagraph-capture-sizes 1 2 4 8 16 --async-scheduling \\\n  --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 \\\n  --speculative-config '{\"method\":\"mtp\",\"model\":\"/models/gemma-4-26B-A4B-it-assistant\",\"num_speculative_tokens\":4}' \\\n  --trust-remote-code --host <ip> --port 8000\n```\n\n## Production shape\n\nCache directories *before* the first start — vLLM, Triton, torchinductor, CUDA and huggingface each want one:\n\n```\nmkdir /models /var/cache/huggingface /var/cache/vllm /var/cache/triton \\\n      /var/cache/torchinductor /var/cache/nv\n```\n\nEnvironment in `/etc/vllm/vllm.env`\n\n(`chmod 400`\n\n), the essentials:\n\n```\nVLLM_API_KEY=<key>\nHF_HOME=/var/cache/huggingface\nVLLM_CACHE_ROOT=/var/cache/vllm\nTRITON_CACHE_DIR=/var/cache/triton\nTORCHINDUCTOR_CACHE_DIR=/var/cache/torchinductor\nCUDA_MODULE_LOADING=LAZY\nCUDA_DEVICE_ORDER=PCI_BUS_ID\nNCCL_CUMEM_ENABLE=1\nPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\nFLASHINFER_DISABLE_VERSION_CHECK=1\nHF_HUB_OFFLINE=1        # model is on disk; do not phone home\n```\n\nThe systemd unit earns its keep on the restart and shutdown behavior — a 400K-context model takes minutes to load and seconds to corrupt if killed wrong:\n\n```\n[Service]\nType=simple\nEnvironmentFile=/etc/vllm/vllm.env\nExecStart=/root/.venv/bin/vllm serve ...\nRestart=on-failure\nRestartSec=10\nTimeoutStartSec=900\nTimeoutStopSec=120\nKillSignal=SIGINT\nKillMode=mixed\nLimitNOFILE=1048576\nOOMScoreAdjust=-900\n```\n\nFront it with nftables — default drop input, allow established, loopback, ICMP both families, SSH, and the API ports only:\n\n```\ntable inet filter {\n    chain input {\n        type filter hook input priority filter; policy drop;\n        ct state established,related accept\n        ct state invalid drop\n        iif lo accept\n        ip protocol icmp accept\n        ip6 nexthdr icmpv6 accept\n        tcp dport 22 accept\n        tcp dport 8000-8001 accept\n    }\n}\n```\n\nFor benchmarking, ShareGPT (Vicuna unfiltered split) through the OpenAI-compatible API with a small concurrency harness — numbers before and after every flag change, or the flag changes are just feelings.\n\n*This saved you a night? I do this for a living: info@wirt.ee.*", "url": "https://wpnews.pro/news/on-prem-llm-inference", "canonical_source": "https://wirt.ee/logbook/vllm-serving/", "published_at": "2026-09-03 12:30:43+00:00", "updated_at": "2026-09-03 12:53:27.798531+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["vLLM", "GLM-5.2", "GLM-5.3", "gemma-4-26B", "L40S", "FlashInfer", "LiteLLM", "NVIDIA"], "alternates": {"html": "https://wpnews.pro/news/on-prem-llm-inference", "markdown": "https://wpnews.pro/news/on-prem-llm-inference.md", "text": "https://wpnews.pro/news/on-prem-llm-inference.txt", "jsonld": "https://wpnews.pro/news/on-prem-llm-inference.jsonld"}}