LocalAI's 18 custom C and C++ inference engines, including vllm.cpp and depth-anything.cpp, outperform or match upstream Python-based engines while drastically reducing footprint, with vllm.cpp delive…
An engineer at vLLM analyzed concurrent KV block load coalescing in Mooncake and vLLM, finding that while the code path for single-flight optimization exists, real traces show almost no coalescable wo…
Baidu's vLLM Kunlun plugin, open-sourced on Dec 8, 2025, enables vLLM to run on Kunlun XPU hardware, supporting 20+ models including Qwen, Llama, DeepSeek, GLM, and Gemma4, with features like quantiza…
A technical advisory recommends that production systems using raw LLM responses implement a structured content lifecycle, treating model output as a draft rather than the final artifact. The approach,…
HexCore, a low-latency paged KV cache allocator for LLM inference written in C++20 and CUDA, has been released under the Apache License 2.0 by Rasuljanov Muhammadali. The CPU-side allocator and relate…
A research paper titled 'The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility' reveals that the choice of inference engine can significantly affect LLM output…
A new method called Functional Reconstruction improves draft-token acceptance in speculative decoding for multi-head latent attention (MLA) draft models, according to a paper on arXiv (2607.27269v1). …
Ollama's simplicity can mask when a local experiment becomes a shared inference service, and vLLM offers better scheduling and observability for production workloads. Migration is warranted when multi…
Moonshot AI released Kimi K3, a 2.8 trillion parameter Mixture of Experts (MoE) model, on July 27, 2026, making it the first open-weight system to reach the 3 trillion parameter class. The model requi…
On July 27, 2026, Moonshot AI released Kimi K3, a 2.8 trillion parameter Mixture of Experts (MoE) model, described as the first open-weight system to reach the 3 trillion parameter class. The model ac…
Moonshot AI released the full weights and technical report for Kimi K3 on July 27th, documenting a 2.8 trillion-parameter model that activates 104 billion parameters per token. The Beijing developer h…
DigitalOcean launched Kimi K3 on day 0, making it one of the most popular models on the platform and across the market, with the second most likes on Hugging Face and sixth most traffic on OpenCode. T…
Kubeflow 1.11, released at KubeCon Japan, introduces a pip-installable SDK that lets ML practitioners submit distributed training jobs, run hyperparameter searches, and fine-tune Llama 3.2 without wri…
Red Hat AI warns that enterprises relying on third-party hosted APIs for AI agent inference undermine their own data sovereignty, as every prompt and tool call routes through external datacenters. The…
LocalAI, a self-hosted runtime that runs AI workloads on user-controlled hardware, offers an OpenAI-compatible API supporting text generation, vision, speech, image and video generation, embeddings, a…
RunInfra enables running Kimi-Linear-48B, a distilled version of the full 2.78-trillion-parameter Kimi K3 model, on a single consumer GPU such as the RTX 5090 with 32 GB VRAM, achieving 113.83 tokens …
The accuracy gap between the best open-weight models and GPT-4o has shrunk to under 3% on structured data tasks, according to a developer's hands-on analysis. A fine-tuned Qwen2.5 72B model achieved 9…
A new Chrome extension called Explain This lets users select text on a page, right-click, and get a plain-language explanation from a local LLM running entirely in the browser via WebGPU, with no serv…
DeepSeek V4 Pro, a Mixture-of-Experts model with 1.6 trillion total parameters, scores 80.6% on SWE-bench Verified, tying Gemini 3.1 Pro and achieving the highest score for any open-weight model. Pric…
A developer built a pure-JAX inference engine for Google's Gemma 4 E2B model after discovering that quantization-aware-trained (QAT) checkpoints fail to load in vLLM on TPU due to a loader bug. The en…