Intel's Arc Pro B70 GPU crashes under sustained inference load in production, according to ModDog bot developer who pulled the card from his fleet after six weeks of testing. The 32GB card, priced at …
Microsoft's Azure Kubernetes Service engineering team published a reference implementation on June 29 that separates agent-request routing into three layers: semantic model selection via RouteLLM, gat…
Microsoft released a reference architecture for routing agent traffic on Azure Kubernetes Service that combines the Kubernetes Gateway API Inference Extension, agentgateway, and RouteLLM into one Open…
A developer built a small scheduler in Go to understand vLLM's scheduler for LLM inference, tracing each design decision back to its vLLM equivalent. The scheduler operates in a tick loop with three p…
VLLM's slack-aware preemption policies rescued urgent request attainment at high load in a benchmark on 6× A100 SXM4 80GB GPUs running Llama 70B, where the control policy collapsed to 0% urgent attain…
A new paper, FlowPrefill by Hsieh et al., proposes preempting long LLM inference prefills mid-forward-pass to rescue urgent requests that would otherwise miss their time-to-first-token (TTFT) service-…
Moonshot AI released Kimi K3, a 2.78-trillion-parameter mixture-of-experts model with 104.2 billion active parameters, native vision, and a 1,048,576-token context window, claiming it is the first ope…
LLM inference profitability depends on the trade-off between batch size and GPU count, which determines token latency and cost per million tokens. Applying this model to Kimi K3, which requires at lea…
Together AI introduces ThunderAgent, a system for high-throughput agentic inference that achieves up to 2.5× higher single-node throughput and 2.4× speedup on an 8-node cluster with near-linear scalin…
AgentHound, an open-source offensive security framework for AI agent infrastructure, has been released. The tool performs recon, fingerprinting, credential looting, model inversion, and persistence ac…
Moonshot AI's Kimi Linear attention architecture, introduced in October, now powers the company's 2.8-trillion-parameter Kimi K3 flagship model released in mid-July, marking the first production deplo…
A new hands-on guide teaches programmers how to run large language models on their own hardware, covering tools like Ollama, llama.cpp, vLLM, and Hugging Face's transformers. The guide explains local …
Flash Attention is an algorithm that speeds up training and inference of transformer models by using smart memory management on GPUs. The original version was released in 2022, followed by Flash Atten…
Moonshot AI released the full Kimi K3 open weights on July 27, 2026, a 2.8-trillion-parameter mixture-of-experts model with a one-million-token context window and a 1.4 TB download. The model uses nat…
Moonshot AI released Kimi K3, an open-source large language model with 2.8 trillion parameters and a 1-million-token context window, in July 2026. It is the first open-source model to surpass Claude a…
LOCKS, a new method from arXiv, enables efficient long-context decoding by giving each page of the key-value cache its own compact spectral summary, reconstructing within-page logits, and attending on…
Open-weight AI models like Kimi K3 are disrupting the market by enabling local deployment and customization without proprietary fees, according to the article. The shift lowers barriers for developers…
Moonshot AI released the full Kimi K3 model weights and technical report on July 27, 2026, making its 2.8 trillion parameter mixture-of-experts model available to developers and researchers. The model…
VLLM announces efficient day-0 support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model, achieving up to 370 tokens per second with speculative decoding on 16 NVIDIA GB300 …
Baseten has launched day-0 API support for Kimi K3, a new open frontier model from Moonshot AI with 2.8 trillion parameters, making it the largest open model to date. The API runs on NVIDIA GB300 NVL7…