Open models have reached impressive capability levels, but the surrounding platform infrastructure—including tool calling, streaming, file handling, web search, code execution, prompt caching, and tok…
Open-source coding agents can now run entirely on local consumer hardware such as a 2020-era NVIDIA RTX 3090, eliminating usage-based pricing from AI vendors. The setup uses a quantized Gemma4 model s…
A new study reveals that the multiplicative repetition penalty used across major LLM inference engines (HuggingFace, vLLM, llama.cpp) is fundamentally flawed because it branches on the sign of raw log…
MemStitch, a zero-copy context bridging gateway for vLLM, achieves a 25x speedup in Time-to-First-Token (TTFT) by allowing multi-agent GPU inference to share cached prefixes, reducing prefill latency …
A hybrid routing strategy that sends simple agent tasks to local AI models and complex ones to cloud APIs can cut costs by more than half for teams running high-volume workflows, according to a practi…
Two used RTX 3090s offer 48GB of VRAM for about half the cost of a single RTX 5090, enabling local 70B LLM inference at 14-21 tokens/sec, while the 5090's 32GB cannot run a quality 70B model. For mode…
Four months after Google Research and DeepMind published TurboQuant, a training-free algorithm suite claiming 6x memory reduction for KV cache, the promised official code has not shipped. Instead, the…
Google Cloud open-sourced k8s-aibom, an unprivileged Kubernetes controller that automatically detects running AI runtimes and generates CycloneDX Machine Learning Bill of Materials (ML-BOMs) from live…
A developer compared Llamafile and vLLM for serving local LLMs on an RTX 4090, finding Llamafile faster for single-user interactive use and simpler to set up, while vLLM excels under concurrent load a…
Together AI closed an $800 million Series C at an $8.3 billion valuation, reporting $1.15 billion in annual bookings and an inference engine that hits 500 tokens per second on DeepSeek-V3.1. The compa…
AugServe, a new framework for optimizing large language model inference, achieves a 4.7x increase in effective throughput compared to vLLM and a 3.3x boost over InferCept, while reducing time-to-first…
Embedcache, a new open-source proxy from developer Ajay6601, cuts embedding API costs by caching redundant requests, reporting that it can eliminate duplicate work for platform and ML-infra teams proc…
A hyper-optimized, zero-dependency C/CUDA inference engine for the Qwen 3.6 35B model on RTX 5090 Blackwell GPUs achieves 13.4k tokens/sec prefill throughput at 2,048 context depth and 270+ tokens/sec…
AMD's QuickReduce library now supports INT3 quantization for all-reduce communication in multi-GPU LLM inference, achieving a 22% reduction in on-wire data volume compared to INT4 on AMD Instinct MI35…
AMD has integrated an NVFP4 emulation pipeline into vLLM that enables AMD Instinct MI355 accelerators to serve standard NVFP4 quantized checkpoints directly, dequantizing weights to BF16 on-the-fly at…
Said Sef's Argocd-AI-Assistant is an Argo CD UI extension that adds an AI-powered Assistant tab to Kubernetes resource views, enabling natural language queries enriched with manifest, events, and opti…
Synapse, a security gate tool, introduces AI-powered false-positive triage that never deletes findings. The system uses a two-model consensus: a proposer model suggests a false-positive verdict, and a…
A developer deployed vLLM locally on two NVIDIA RTX 5060 Ti 16GB GPUs using Docker Compose, configuring tensor parallelism, FP8 KV cache, and speculative decoding with MTP. The setup runs an OpenAI-co…
Hugging Face announced that custom GPU kernels on its Hub are now signed artifacts governed by a trusted publisher model, requiring a dedicated repository type that replaces the old model-type format.…
A new scheduler for large language model serving allows decode nodes to assist prefill phases, cutting P95 time-to-first-token by up to 81% and improving service-level objective attainment by up to 79…