NVIDIA released AIPerf, a ground-up rewrite and designated successor to GenAI-Perf, as a multiprocessed LLM inference benchmarking tool that coordinates worker processes and record-processor services …
A benchmark study by an independent researcher found that burstier request arrivals speed up LLM inference, contradicting standard intuition. The analysis of vLLM serving shows that higher burstiness …
VLLM's slack-aware preemption policies rescued urgent request attainment at high load in a benchmark on 6× A100 SXM4 80GB GPUs running Llama 70B, where the control policy collapsed to 0% urgent attain…
Researchers propose EntMTP, a training-free scheduler that dynamically adjusts multi-token prediction depth based on local entropy, achieving up to 1.36x speedup over Medusa baselines in LLM inference…
AgentTrove, one of the largest open-source collections of agentic interaction traces, is now available for streaming analysis without requiring full dataset downloads. Researchers can inspect conversa…