vLLM for Baidu Kunlun

Baidu's vLLM Kunlun plugin, open-sourced on Dec 8, 2025, enables vLLM to run on Kunlun XPU hardware, supporting 20+ models including Qwen, Llama, DeepSeek, GLM, and Gemma4, with features like quantization, LoRA, and speculative decoding. The latest development version v0.25.1 adds support for Qwen3.5, Gemma4, and GLM MoE DSA, while v0.11.0 and v0.10.1.1 introduced multimodal models and performance optimizations.

📖 Documentation https://vllm-kunlun.readthedocs.io/en/latest/ | | https://vllm-kunlun.readthedocs.io/en/latest/quick start.html 🚀 Quick Start | https://vllm-kunlun.readthedocs.io/en/latest/installation.html 📦 Installation 💬 Slack - 2026/07 🚧 v0.25.1 under development — Added Qwen3.5 / Qwen3.5-MoE, Gemma4 text and multimodal , GLM MoE DSA, and DFlash speculative decoding - 2026/02 ⚡ Performance optimizations — Fused MoE with small batches, optimized attention metadata building, Multi-LoRA inference achieves 80%+ of non-LoRA performance - 2026/02 🔧 DeepSeek-V3.2 MTP support — Added MTP Multi-Token Prediction for DeepSeek-V3.2, with RoPE and decoding stage kernel optimizations - 2026/01 🔢 New quantization methods — Support for compressed-tensors W4A16, AWQ MoE W4A16, and DeepSeek-V3.2 W8A8 quantization - 2026/01 🛠️ CI/CD overhaul — Added E2E tests, unit test CI, ruff format checks, and modular CI workflow refactoring - 2025/12 🎉 v0.11.0 released — Added Qwen3-Omni, Qwen3-Next, Seed-OSS support Release Notes https://github.com/baidu/vLLM-Kunlun/releases/tag/v0.11.0 - 2025/12 📦 v0.10.1.1 released — 5+ multimodal models, AWQ/GPTQ quantization for dense models, Piecewise Kunlun Graph, vLLM V1 engine, Flash-Infer Top-K/Top-P sampling with 10-100× speedup Release Notes https://github.com/baidu/vLLM-Kunlun/releases/tag/v0.10.1.1 - 2025/12 🌟 Initial release of vLLM Kunlun — Open sourced on Dec 8, 2025 vLLM Kunlun vllm-kunlun is a community-maintained hardware plugin designed to seamlessly run vLLM https://github.com/vllm-project/vllm on the Kunlun XPU . It is the recommended approach for integrating the Kunlun backend within the vLLM community, adhering to the principles outlined in the RFC Hardware Pluggable https://github.com/vllm-project/vllm/issues/11162 . This plugin provides a hardware-pluggable interface that decouples the integration of the Kunlun XPU with vLLM. By utilizing vLLM Kunlun, popular open-source models — including Transformer-like, Mixture-of-Expert MoE , Embedding, and Multi-modal LLMs — can run effortlessly on the Kunlun XPU. Seamless Plugin Integration — Works as a standard vLLM platform plugin via Python entry points, no need to modify vLLM source code Broad Model Support — Supports 20+ mainstream LLMs including Qwen, Llama, DeepSeek, GLM, Gemma4, Kimi-K2, and multimodal models Quantization Support — W8A8 INT8 , AWQ, GPTQ, and compressed-tensors W4A16 for MoE and dense models LoRA Fine-Tuning — LoRA and Multi-LoRA adapter support for Qwen series models Piecewise Kunlun Graph — Hardware-accelerated graph optimization for high-performance inference FlashMLA Attention — Optimized multi-head latent attention for DeepSeek MLA architectures Speculative Decoding — MTP Multi-Token Prediction for DeepSeek-V3.2 and DFlash/EAGLE-style proposers Tensor Parallelism — Multi-device parallel inference with distributed execution support OpenAI-Compatible API — Serve models with the standard OpenAI API interface Hardware : Kunlun3 P800 OS : Ubuntu 20.04 Software :- Python = 3.10 - PyTorch = 2.5.1 KL3-customized xpytorch build, see Installation https://vllm-kunlun.readthedocs.io/en/latest/installation.html - vLLM matching version, see requirements.txt /baidu/vLLM-Kunlun/blob/v0.25.1-dev/requirements.txt - transformers == 5.2.0 Qwen3.5 requires transformers 5.x | Model | Support | Quantization | LoRA | Kunlun Graph | |---|---|---|---|---| | Qwen2 | ✅ | ✅ | ✅ | ✅ | | Qwen2.5 | ✅ | ✅ | ✅ | ✅ | | Qwen3 | ✅ | ✅ | ✅ | ✅ | | Qwen3-Moe | ✅ | ✅ | ✅ | ✅ | | Qwen3-Next | ✅ | ✅ | ✅ | ✅ | | Qwen3.5 | ✅ | ✅ | ✅ | | | Qwen3.5-Moe | ✅ | ✅ | ✅ | | | MiMo-V2-Flash | ✅ | ✅ | ✅ | | | Llama2 | ✅ | ✅ | ✅ | ✅ | | Llama3 | ✅ | ✅ | ✅ | ✅ | | Llama3.1 | ✅ | ✅ | ✅ | | | gpt-oss | ✅ | ✅ | || | Gemma4 | ✅ | ✅ | || | GLM4.5 | ✅ | ✅ | ✅ | | | GLM4.5Air | ✅ | ✅ | ✅ | | | GLM5 | ✅ | ✅ | ✅ | | | InternLM2 | ✅ | ✅ | || | Seed-OSS | ✅ | ✅ | || | DeepSeek-R1 | ✅ | ✅ | ✅ | | | DeepSeek-V3 | ✅ | ✅ | ✅ | | | DeepSeek-V3.2 | ✅ | ✅ | ✅ | | | Kimi-K2 | ✅ | ✅ | ✅ | | Model | Support | Quantization | LoRA | Kunlun Graph | |---|---|---|---|---| | Qwen2-VL | ✅ | ✅ | ✅ | | | Qwen2.5-VL | ✅ | ✅ | ✅ | | | Qwen3-VL | ✅ | ✅ | ✅ | | | Qwen3-VL-MoE | ✅ | ✅ | ✅ | | | Gemma4 | ✅ | ✅ | || | InternVL-2.5 | ✅ | ✅ | || | InternVL-3.5 | ✅ | ✅ | || | InternS1 | ✅ | ✅ | Current environment: 16-way concurrency, input/output size 2048. python -m vllm.entrypoints.openai.api server \ --host 0.0.0.0 \ --port 8356 \ --model <your-model-path \ --gpu-memory-utilization 0.9 \ --trust-remote-code \ --max-model-len 32768 \ --tensor-parallel-size 1 \ --dtype float16 \ --max num seqs 128 \ --max num batched tokens 32768 \ --block-size 128 \ --no-enable-prefix-caching \ --no-enable-chunked-prefill \ --distributed-executor-backend mp \ --served-model-name <your-model-name curl http://localhost:8356/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "<your-model-name ", "messages": {"role": "user", "content": "Hello "} , "max tokens": 512 }' | Version | Release Type | Documentation | |---|---|---| | v0.25.1 | Latest development version main | | Quick Start https://vllm-kunlun.readthedocs.io/en/latest/quick start.html · Installation https://vllm-kunlun.readthedocs.io/en/latest/installation.html vllm-kunlun/ ├── vllm kunlun/ Core plugin package │ ├── platforms/ Kunlun XPU platform implementation │ ├── models/ Model implementations DeepSeek, Qwen, Gemma4, InternVL, etc. │ ├── ops/ Custom operators │ │ ├── attention/ FlashMLA, paged attention, merge attention states │ │ ├── fla/ Flash linear attention operations │ │ ├── fused moe/ Fused MoE kernels │ │ ├── mamba/ Mamba / linear-attention state ops │ │ └── rotary embedding/ RoPE variants │ ├── quantization/ AWQ, GPTQ, compressed-tensors, moe wna16 │ ├── lora/ LoRA / Multi-LoRA support │ ├── v1/ vLLM V1 engine adaptations incl. spec decode: MTP, DFlash │ ├── distributed/ Communicators and distributed helpers │ ├── entrypoints/ OpenAI-compatible serving overrides and tool parsers │ ├── reasoning/ Reasoning parsers Qwen3, Gemma4 │ ├── compilation/ Torch compile wrapper for Kunlun Graph │ ├── transformers utils/ transformers config/tokenizer adaptations │ ├── csrc/ C++ extensions custom kernels │ └── config/ Model configuration overrides ├── tests/ Test suite ├── docs/ Documentation Sphinx-based, ReadTheDocs hosted ├── ci/ CI pipeline configurations ├── setup.py Legacy build script with C++ extensions └── pyproject.toml Modern Python build configuration hatchling We welcome contributions from the community Please read our Contributing Guide /baidu/vLLM-Kunlun/blob/v0.25.1-dev/CONTRIBUTING.md before submitting a PR. Use the following prefixes for PR titles: Attention — Attention mechanism features/optimizations Core — Core vllm-kunlun logic platform, attention, communicators, model runner Kernel — Compute kernels and ops Bugfix — Bug fixes Doc — Documentation improvements Test — Tests CI — CI/CD improvements Misc — Other changes We opened the project at Dec 8, 2025. We love open source and collaboration ❤️ https://www.star-history.com/?repos=baidu%2FvLLM-Kunlun&type=date&legend=top-left