Apple releases RSI-Inference: evaluating agents for self-improving LLM inference

Apple ML Research released RSI-Inference Bench, a benchmark built from real-world vLLM and SGLang tasks that tests whether AI agents can improve the systems running AI models, spanning day-0 model support, inference optimization and bug fixes. Apple reported that after spending $100K in API tokens and 50K GPU hours, the best agent reached 26.7% pass@1, leaving the benchmark unsaturated. The release gives inference engineers a verifiable-reward evaluation covering the full inference stack, with binary and continuous scoring.

Introducing RSI-Inference Bench from 🍎 Apple ML Research 🌐 rsi-inference.com http://rsi-inference.com 💻 github.com/apple-aiml-res… https://github.com/apple-aiml-research/ml-rsi-inference-bench Can AI agents improve the systems that run AI? We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world vLLM https://twitter.com/hashtag/vLLM and SGLang https://twitter.com/hashtag/SGLang tasks, covering the full lifecycle of model inference engineering: 🚀 Day-0 Model Support ⚡ Inference Optimization 🔧 Bug Fix → Unsaturated: best agent at 26.7% pass@1 → Comprehensive coverage of the inference stack → Verifiable rewards: binary and continuous Watch the video to find out more 00:00