SkyPilot launched SkyPilot Endpoints, a production-ready LLM inference system that deploys and manages inference across multiple Kubernetes clusters from a single YAML specification. The system handle…
NVIDIA announced that DFlash speculative decoding boosts inference performance on Blackwell GPUs by up to 15x for gpt-oss-120b and nearly doubles interactivity for Llama 3.1 8B compared to EAGLE-3. Th…
Modelplane, an open-source control plane for AI inference built on Crossplane, is being released to manage GPU clusters as a single inference fleet, handling provisioning, model placement, autoscaling…
NVIDIA released a workflow for converting FP8-quantized CLIP model checkpoints into TensorRT engines, enabling faster inference and higher GPU throughput for production deployment. The process involve…
The open-source LLM inference landscape in 2025–2026 has split into three specialized tiers: throughput-oriented serving engines for production GPU clusters, portability-focused runtimes for consumer …