Tencent released Hy3, a 295B Mixture-of-Experts model with Apache 2.0 weights, achieving a 78.0 SWE-bench Verified score and leading open-weight models on tool and search benchmarks. A free API on Ope…
Microsoft Foundry now offers a curated catalog of Hugging Face open-weight models deployable on Foundry Managed Compute, with pre-staged weights in Azure and built-in enterprise security, governance, …
Tencent released Hy3, an open-source 295B-parameter Mixture-of-Experts model with 21B active parameters and 256K context length, under the Apache License 2.0. The model achieves state-of-the-art resul…
SGLang development is being augmented with agent-assisted workflows that encode procedural engineering knowledge into executable skills, covering LLM serving, GPU kernels, diffusion pipelines, and pro…
Tencent released Hy3, an Apache-2.0 open-weight Mixture-of-Experts model with 295B total parameters, 21B active parameters per token, and a 256K context window. The release includes official artifacts…
NVIDIA announced that its Confidential Computing technology for Blackwell GPUs achieves up to 98% of the inference performance of non-secure solutions, enabling hardware-rooted AI security without sig…
Speculative decoding accelerates large language model inference by 20-50% without quality loss, using a draft-verify mechanism that generates multiple tokens per forward pass. The technique amortizes …
MiniMax released M3, a 428-billion-parameter open-weight model, on June 7, achieving 59.0% on SWE-Bench Pro—slightly outperforming GPT-5.5's 58.6%—at $0.30 per million input tokens, making it 16 times…
Shanghai AI Laboratory's Agents-A1, a 35-billion-parameter Mixture-of-Experts model, matches or outperforms trillion-parameter rivals by training on longer task sequences instead of increasing paramet…
RadixArk released Miles, an open-source PyTorch-native framework for large-scale LLM reinforcement learning post-training. The framework integrates SGLang for rollout, NVIDIA Megatron-LM for training,…
PyTorch introduced the Cross-Repository CI Relay (CRCR), an automated pipeline that triggers and tracks CI in downstream repositories whenever changes are made to pytorch/pytorch. Results are displaye…
LLM inference optimization can reduce serving costs by 5-10x and latency by 3-5x, often determining whether an AI feature ships. The bottleneck is memory bandwidth during autoregressive decoding, and …
A new open-source tool called llm-d acts as an LLM-aware load balancer for distributed inference, intelligently routing requests across vLLM instances based on KV cache locality and GPU utilization. B…
Baidu released Unlimited OCR, an open-source 3-billion-parameter model that uses Reference Sliding Window Attention to parse multi-page documents in a single pass, eliminating the need for page-by-pag…
A developer built Modelplane, an open-source inference control plane using Crossplane, to manage GPU fleets across clouds, neoclouds, and on-premise environments. The platform allows platform teams to…
UC San Diego researchers developed DFlash, a speculative decoding method that uses a lightweight block diffusion model to draft entire token blocks in parallel, achieving up to 6.08x speedup on Qwen3-…
IBM Research, Red Hat, and NxtGen Cloud Technologies demonstrated that using llm-d to serve AI models on mixed GPU hardware can boost inference speeds by 3 to 5 times and double throughput, enabling e…
SkyPilot launched SkyPilot Endpoints, a production-ready LLM inference system that deploys and manages inference across multiple Kubernetes clusters from a single YAML specification. The system handle…
SGLang achieved a 5x throughput improvement for serving DeepSeek-V4 on NVIDIA GB300 hardware since its Day-0 launch, reaching ~11,200 tok/s/GPU at 50 tok/s/user through kernel and runtime optimization…
NVIDIA announced that DFlash speculative decoding boosts inference performance on Blackwell GPUs by up to 15x for gpt-oss-120b and nearly doubles interactivity for Llama 3.1 8B compared to EAGLE-3. Th…