TileRT

TileRT's persistent engine on NVIDIA GPUs achieves up to 500 tokens/s/user on the InferenceX GLM5 FP8 744B benchmark on a single B200 decode server, approximately 3× faster than GB300 NVL72 running tr…

Xiaomi's MiMo-V2.5-Pro-UltraSpeed, a 1.02-trillion-parameter MoE model, achieved 1,000 tokens per second inference on standard cloud GPUs using FP4 quantization, DFlash speculative decoding, and TileR…