A developer evaluated speculative decoding in vLLM on AMD Instinct MI300X and MI355X GPUs, testing draft methods including MTP, EAGLE-3, DFlash, and DSpark that propose candidate tokens verified in a …
Modal, Fireworks AI, and Baseten are offering hosted inference for Moonshot AI's Kimi K3 at roughly one-tenth the cost of direct access, using Nvidia GB300 NVL72 systems and AMD MI350X and MI355X acce…
DFlash, a block-diffusion speculative decoding drafter, delivered up to 5.02× single-request throughput on Qwen3.5-27B when run on AMD Instinct MI355X through vLLM on ROCm, according to the benchmark.…
A developer benchmarked DFlash, a diffusion-model-based speculative decoding drafter from Z-Lab, against Gemma-4-12B-it's native Assistant MTP model on an RTX 3060 with 12GB VRAM using llama.cpp. DFla…
Researchers led by Subham Sekhar Sahoo introduced Uno, a diffusion-based adapter that boosts large language model throughput by up to 3x, with the Uno-Qwen 8B variant achieving roughly 2.5x speedup at…
AMD's experiments with speculative decoding in vLLM on AMD Instinct MI300X and MI355X GPUs using the ROCm platform show that output-token throughput gains vary by drafting method, proposal length, mod…
Meta's Muse Glimmer 30B model can be run locally on an RTX 3090 GPU using llama.cpp, DFlash speculative decoding, and Pi, achieving speeds of 46 to 127 tokens per second for agentic coding tasks. The …
A developer's field guide benchmarks DFlash, a block-diffusion speculative decoding method, on 30B-scale models, arguing that tokens per second can mislead. The guide highlights DFlash's architectural…
A configuration using the DFlash speculative decoding drafter with Meta Muse Glimmer 30B on an NVIDIA RTX PRO 4000 Blackwell SFF GPU reached 84.64 tokens per second on a coding task, a 4.50 times spee…
NVIDIA's Nemotron 3.5 Lightning, a 30-billion-parameter hybrid Mixture-of-Experts model that activates only 3 billion parameters per forward pass, is now available on Amazon SageMaker JumpStart, deliv…
Meta has released Muse Glimmer-30B, a dense 30-billion-parameter model designed for on-device agentic AI, diverging from the prevailing Mixture-of-Experts trend. The model activates all parameters per…
Meta released Muse Glimmer, a 30-billion-parameter multimodal agentic model distilled from Muse Spark, under the Apache 2.0 license, designed to run on a single consumer GPU or Mac with no network cal…
Meta Superintelligence Labs released Muse Glimmer, a 30-billion-parameter multimodal open model under Apache 2.0, designed for local agent workloads on consumer hardware. The model scores 76 on SWE-Be…
Researchers introduced DominoTree, a training-free best-first draft tree method for speculative decoding that uses Domino's conditional correction to achieve up to 6.6x speedup over autoregressive dec…
Researchers propose DeLS-Spec, a speculative decoding method that decouples long- and short-context experts to accelerate LLM inference. By adding a lightweight local head to the existing DFlash model…
A developer reports that speculative decoding techniques MTP, EAGLE-3, and DFlash can significantly speed up local inference of large language models in llama-cpp. Testing on an NVidia 3060 RTX 12 GB …
Z-Lab's DFlash technique uses diffusion models to accelerate LLM token generation through speculative decoding, achieving up to 123 tokens per second for code generation in tests with Qwen 3 8B on lla…
DeepSeek-AI released DeepSpec, an open-source codebase for training and evaluating draft models for speculative decoding, supporting three draft model algorithms (DSpark, DFlash, Eagle3) and requiring…
DeepSeek open-sourced DSpark, a speculative decoding framework that accelerates per-user generation for DeepSeek-V4 by 57–85% over the MTP-1 baseline. The framework pairs a parallel draft backbone wit…
DeepSeek released DSpark, a speculative decoding framework now live in its DeepSeek-V4 Flash and Pro production API, delivering 51 to 400 percent throughput gains and up to 80 percent latency reductio…