{"slug": "a-new-speculative-decoding-drafter-for-vllm-and-sglang-called-eagle-4", "title": "A new Speculative decoding drafter for vLLM and SGLang called EAGLE-4", "summary": "EAGLE-4, a speculative decoding drafter released with the eagle-harness package, serves Qwen3.8-27B 3.40× faster than SGLang without speculation for a single request on four NVIDIA B300 GPUs, according to the EAGLE-4 paper and support matrix. The same drafter adapts its draft tree to batch size, hardware and workload while preserving the target model's output distribution, and also runs in vLLM (2.99× at one request) and experimental TokenSpeed (2.02× at 8 concurrent requests); on one AMD MI355X, SGLang with EAGLE-4 is 2.07× faster at 8 concurrent requests. Speedups fall as concurrency rises, to 1.35× in SGLang at 64 concurrent requests and 1.16× in vLLM.", "body_md": "**One Flexible Drafter for All Batch Sizes and Hardware**\n\n[**Paper (EAGLE-4)**](https://github.com/eagle-hq/EAGLE-4/blob/main/EAGLE4_Paper.pdf) ·\n  [**EAGLE-4 draft for Qwen3.8-27B**](https://huggingface.co/EAGLE-Corp/Qwen3.8-27B-EAGLE4) ·\n  **Documentation**\n\n    <sub>Figure 1(a) of the EAGLE-4 paper: Qwen3.8-27B on the SPEED-Bench qualitative split, 4× NVIDIA B300, one request, temperature 0.\n    The numbers this release measured with EAGLE-4 are in the [support matrix](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/support-matrix.md).</sub>\n  \n\nEAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) speeds up LLM inference with speculative decoding. One draft model adapts its draft tree to the batch size, the hardware and the workload, and keeps the target model's output distribution. On four B300 GPUs it serves Qwen3.8-27B 3.40× faster than SGLang without speculation for one request at a time.\n\nThis repository is `eagle-harness`, the package that runs EAGLE-4 drafts inside SGLang, vLLM and TokenSpeed.\n\nYou need Linux, four NVIDIA GPUs with a CUDA 13 driver (tested on B300) and about 70 GB of disk for\n[Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) and its\n[EAGLE-4 draft](https://huggingface.co/EAGLE-Corp/Qwen3.8-27B-EAGLE4), which download on the first start.\n\nStart an SGLang server with Docker:\n\n```\ndocker run --gpus all --ipc host -p 127.0.0.1:30000:30000 \\\n  -e HF_HOME=/hf -v ~/.cache/huggingface:/hf \\\n  ghcr.io/eagle-hq/eagle-4-sglang:0.1.0 \\\n  sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --host 0.0.0.0 \\\n  --disable-overlap-schedule --speculative-algorithm EAGLE_HARNESS \\\n  --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \\\n  --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \\\n  --speculative-algorithm-config '{}'\n```\n\nOr install with pip, in Python 3.12. Pip installs also need the [CUDA toolkit and a C++ compiler](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/install.md#requirements):\n\n```\npython3.12 -m venv .venv && source .venv/bin/activate\npip install \"eagle-harness[sglang]\" \"sglang==0.5.23.dev252+eagle.5\" \\\n  --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \\\n  --extra-index-url https://download.pytorch.org/whl/cu130 \\\n  --extra-index-url https://pypi.nvidia.com\nsglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 \\\n  --speculative-algorithm EAGLE_HARNESS \\\n  --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4\n```\n\nWhen the server logs `The server is fired up and ready to roll!`, send a request from another terminal:\n\n```\ncurl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' \\\n  -d '{\"model\": \"Qwen/Qwen3.8-27B\", \"messages\": [{\"role\": \"user\", \"content\": \"Explain how a hash table works.\"}], \"max_tokens\": 256}'\n```\n\nFor vLLM, TokenSpeed (experimental) and AMD MI355X, follow the [engine guides](#more).\n\nEach speedup divides EAGLE-4 throughput by the same engine's throughput without speculation, on Qwen3.8-27B at\ntemperature 0 on the SPEED-Bench client, with the measured recipe of each engine guide. The\n[support matrix](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/support-matrix.md) has the tokens/s, the correctness checks and how each number was measured.\n\n| Engine | GPUs | 1 request | 8 concurrent | 64 concurrent | \n|---|---|---|---|---|\n| SGLang | 4× B300 | 3.40× | 1.91× | 1.35× | \n| vLLM | 4× B300 | 2.99× | 1.70× | 1.16× | \n| TokenSpeed, experimental | 4× B300 | not measured | 2.02× | 1.42× | \n\nOn one AMD MI355X, SGLang with EAGLE-4 is 2.07× faster at 8 concurrent requests and 1.56× at 32.\n\n- [Install](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/install.md) : pins, wheel indexes, Docker images and the ROCm closure.\n- Engine guides with the measured recipes: [SGLang](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/quickstart-sglang.md) ,[vLLM](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/quickstart-vllm.md) ,[TokenSpeed, experimental](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/quickstart-tokenspeed.md) ,[AMD MI355X](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/quickstart-amd.md) .\n- [Support matrix](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/support-matrix.md) and[known limits](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/known-limits.md) .\n- [Calibration](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/calibration.md) ,[configuration](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/configuration.md) and[troubleshooting](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/troubleshooting.md) .\n- [How it works](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/architecture.md) ,[kernels](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/kernels.md) ,[compatibility](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/compatibility.md) ,[new models](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/new-model.md) and the[FAQ](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/faq.md) .\n\nIf you use EAGLE-4, cite the EAGLE-4 paper. EAGLE-4 builds on EAGLE, EAGLE-2 and EAGLE-3, which\n[`CITATION.cff`](https://github.com/eagle-hq/EAGLE-4/blob/main/CITATION.cff) also lists.\n\n```\n@misc{eagle4,\n    author = {EAGLE-4 Team},\n    title = {{EAGLE-4}: One Flexible Drafter for All Batch Sizes and Hardware},\n    year = {2026},\n    note = {Preprint}\n}\n```\n\n- Thanks to Aditya Nandakumar (@proaditya) and the contributors to the original EAGLE-4 PR 1 for the ROCm port of EAGLE-4 on AMD Instinct MI355X.\n- Thanks to Andy Luo (@andyluo7) and the contributors to the original EAGLE-4 PR 2 for the AMD AITER and Triton kernels on GFX950.\n- Thanks to Andy Luo (@andyluo7), Aditya Nandakumar (@proaditya) and the contributors to the merged AMD series in the original EAGLE-4 PR 4 for AITER/FA3 and Split-KV on GFX950.\n- Thanks to Nicolas Perrin (@nperrin-fr) and the TokenSpeed/LightSeek contributors for the TokenSpeed implementation in the original EAGLE-4 PR 3, including the MLP L2 warm-ahead and TGV SwiGLU work used in the TokenSpeed fork.\n- SPEED-Bench is by NVIDIA. Model weights are licensed by their owners.\n\nEagle 4 is **source-available software, not OSI-approved open-source software**.\nThe terms of the [Eagle 4 Source-Available License](https://github.com/eagle-hq/EAGLE-4/blob/main/LICENSE) govern its use,\nmodification, and distribution, and incorporate the [Eagle Trademark Policy](https://github.com/eagle-hq/EAGLE-4/blob/main/TRADEMARKS.md).\nSee also the [NOTICE](https://github.com/eagle-hq/EAGLE-4/blob/main/NOTICE) file.\n\nCommercial Use of a Competing Offering requires a separate written commercial license regardless of revenue. Other Commercial Use is permitted only within the conditions of the license, including the US $20 million Aggregate Revenue threshold measured over any consecutive 12-month period. If the threshold is exceeded, a separate commercial agreement is required before beginning or continuing any Commercial Use. The license specifies that the requirement continues after a subsequent revenue decrease.\n\nDownloading, cloning, forking, copying, accessing, modifying, using, hosting, distributing, or otherwise exercising any licensed right constitutes acceptance of the Source-Available License and incorporated Eagle Trademark Policy. Mere viewing or inspection does not itself constitute trademark use.\n\nFor separate commercial terms, see the\n[Commercial License Agreement](https://github.com/eagle-hq/EAGLE-4/blob/main/COMMERCIAL-LICENSE-AGREEMENT.md) and contact\n[licensing@eagle.ai](mailto:licensing@eagle.ai). Publishing that agreement does\n**not** grant commercial-license rights; it becomes binding only as stated in\nits Section 1. Files derived from\nsglang, vLLM, FlashInfer and other projects keep those projects' notices, listed in\n[`THIRD_PARTY_NOTICES.md`](https://github.com/eagle-hq/EAGLE-4/blob/main/THIRD_PARTY_NOTICES.md).\n\nBy submitting, uploading, pushing, committing, proposing, or otherwise providing\na pull request, patch, issue attachment, model, weight, dataset, documentation,\nor other contribution to this repository, you agree to the [Eagle Contribution\nLicense and Submission Terms](https://github.com/eagle-hq/EAGLE-4/blob/main/docs/legal/Eagle-Contribution-Terms-v1.0.pdf)\n(PDF). If you do not agree, do not submit a contribution. If you act for an\nemployer, client, partner or other entity, you represent that you have authority\nto submit the contribution and have obtained all required permissions.\nSubmitting a contribution does not change the license of existing Eagle code;\nEagle AI Corp controls outbound licensing for accepted contributions. See\n[`CONTRIBUTING.md`](https://github.com/eagle-hq/EAGLE-4/blob/main/CONTRIBUTING.md) for the licensing structure and for how to\nbuild, test and open a pull request.", "url": "https://wpnews.pro/news/a-new-speculative-decoding-drafter-for-vllm-and-sglang-called-eagle-4", "canonical_source": "https://github.com/eagle-hq/EAGLE-4", "published_at": "2026-10-21 16:57:34+00:00", "updated_at": "2026-10-21 17:12:49.231929+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "machine-learning"], "entities": ["EAGLE-4", "eagle-harness", "SGLang", "vLLM", "TokenSpeed", "Qwen3.8-27B", "NVIDIA B300", "AMD MI355X"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-new-speculative-decoding-drafter-for-vllm-and-sglang-called-eagle-4", "markdown": "https://wpnews.pro/news/a-new-speculative-decoding-drafter-for-vllm-and-sglang-called-eagle-4.md", "text": "https://wpnews.pro/news/a-new-speculative-decoding-drafter-for-vllm-and-sglang-called-eagle-4.txt", "jsonld": "https://wpnews.pro/news/a-new-speculative-decoding-drafter-for-vllm-and-sglang-called-eagle-4.jsonld"}}