Moonshot AI has released Kimi-K3, an image-text-to-text model under a permissive license that allows use, modification, and commercial distribution, with support for Transformers, vLLM, SGLang, and Do…
Moonshot AI released the open weights of its 2.8-trillion-parameter Kimi K3 sparse Mixture-of-Experts model on Hugging Face under an Apache 2.0 license, but the 594 GB MXFP4-quantized model requires a…
Netflix's AI Platform team published a detailed account of its LLM serving platform, revealing that version pinning between NVIDIA Triton Inference Server and vLLM, a Python GIL bottleneck, and KV-cac…
Netflix detailed its in-house LLM serving platform built on Triton and vLLM, revealing how it handles real-time and batch inference across CPUs and GPUs while managing version compatibility and constr…
Prefill-decode disaggregation separates the compute-bound prefill and memory-bound decode phases of LLM inference onto different hardware to solve the 'noisy neighbor' problem, according to a technica…
Prefill-decode disaggregation, now supported by vLLM, SGLang, LLM-d, NVIDIA Dynamo, and TensorRT-LLM, separates LLM inference into compute-bound prefill and memory-bound decode phases on different nod…
LightSeek Org released Shepherd Model Gateway (SMG), an engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments that centralizes worker lifecycle management and balance…
Karpenter v1.14.0 can autoscale GPU inference on Amazon EKS by provisioning spot GPU nodes on demand, bin-packing a vLLM v0.25.1 model server, and deleting nodes when traffic drops, eliminating static…
Deploying large language models locally requires matching hardware to model size, with quantization enabling massive models to run on consumer hardware. Ollama, LM Studio, and vLLM are recommended too…
OpenLake, an open-source storage engine for offloading LLM KV caches from GPU memory to RAM and NVMe, cuts GPU time by 48.2% for long-context inference, reducing a 1,169-second workload to 606 seconds…
ARIA, a voice-native 3D spatial AI security operations cockpit with governed autonomy, is now available under BSL 1.1 for evaluation and research. Developed by a solo developer, the platform runs enti…
A developer built a local-first voice-enabled AI assistant by combining Nous Research's open-source Hermes Agent framework with Kokoro TTS, achieving natural speech responses without cloud API costs o…
A senior engineer with 11 years of distributed systems experience explains the full LLM inference pipeline, from request arrival to text output, detailing the GGUF file structure and the distinction b…
OpenAI models are appearing on Hugging Face, offering researchers a rare opportunity to audit proprietary architectures through leaked weights, config files, and inference testing. The leaks expose hi…
A user reports that deploying the full 671B parameter DeepSeek-R1 model locally requires over 100GB of VRAM and is impractical on consumer hardware, with CUDA out-of-memory errors occurring even at sm…
Open-weight AI models offer freedom but require significant effort to deploy, according to a technical guide that argues the real value lies in deployment pipelines and developer ecosystems rather tha…
KV caching, which stores Key and Value tensors in GPU memory to avoid recalculating attention for every token, is the primary driver of high LLM inference costs because the cache grows linearly with s…
Ekorbia v0.6 ships a bundled inference engine that runs models locally without requiring Ollama or a terminal, while v0.7 delivers a visual refresh with simpler theme names and quieter defaults. The n…
Open-source AI is winning over closed-source lobbying due to deployment flexibility, cost, and hardware ecosystem scale, according to a developer analysis. The global infrastructure for running open w…
Ray AI libraries (Serve, Data, Train) now support Google TPU slices through a topology field that reserves a whole ICI-connected slice, preventing multi-host deployment hangs. Ray Serve serves LLMs vi…