Four months after Google Research and DeepMind published TurboQuant, a training-free algorithm suite claiming 6x memory reduction for KV cache, the promised official code has not shipped. Instead, the…
Google Cloud open-sourced k8s-aibom, an unprivileged Kubernetes controller that automatically detects running AI runtimes and generates CycloneDX Machine Learning Bill of Materials (ML-BOMs) from live…
A developer compared Llamafile and vLLM for serving local LLMs on an RTX 4090, finding Llamafile faster for single-user interactive use and simpler to set up, while vLLM excels under concurrent load a…
Together AI closed an $800 million Series C at an $8.3 billion valuation, reporting $1.15 billion in annual bookings and an inference engine that hits 500 tokens per second on DeepSeek-V3.1. The compa…
AugServe, a new framework for optimizing large language model inference, achieves a 4.7x increase in effective throughput compared to vLLM and a 3.3x boost over InferCept, while reducing time-to-first…
Embedcache, a new open-source proxy from developer Ajay6601, cuts embedding API costs by caching redundant requests, reporting that it can eliminate duplicate work for platform and ML-infra teams proc…
A hyper-optimized, zero-dependency C/CUDA inference engine for the Qwen 3.6 35B model on RTX 5090 Blackwell GPUs achieves 13.4k tokens/sec prefill throughput at 2,048 context depth and 270+ tokens/sec…
AMD has integrated an NVFP4 emulation pipeline into vLLM that enables AMD Instinct MI355 accelerators to serve standard NVFP4 quantized checkpoints directly, dequantizing weights to BF16 on-the-fly at…
AMD's QuickReduce library now supports INT3 quantization for all-reduce communication in multi-GPU LLM inference, achieving a 22% reduction in on-wire data volume compared to INT4 on AMD Instinct MI35…
Said Sef's Argocd-AI-Assistant is an Argo CD UI extension that adds an AI-powered Assistant tab to Kubernetes resource views, enabling natural language queries enriched with manifest, events, and opti…
Synapse, a security gate tool, introduces AI-powered false-positive triage that never deletes findings. The system uses a two-model consensus: a proposer model suggests a false-positive verdict, and a…
A developer deployed vLLM locally on two NVIDIA RTX 5060 Ti 16GB GPUs using Docker Compose, configuring tensor parallelism, FP8 KV cache, and speculative decoding with MTP. The setup runs an OpenAI-co…
Hugging Face announced that custom GPU kernels on its Hub are now signed artifacts governed by a trusted publisher model, requiring a dedicated repository type that replaces the old model-type format.…
A new scheduler for large language model serving allows decode nodes to assist prefill phases, cutting P95 time-to-first-token by up to 81% and improving service-level objective attainment by up to 79…
A developer investigating FLUX.1 [dev] for commercial use found its non-commercial license and questioned whether AI model vendors can track self-hosted deployments. The article concludes that technic…
Ranvier Systems introduced a load-balancing technique for LLM serving that routes requests based on where the KV cache currently resides rather than historical prefix matches, reducing P99 time-to-fir…
Google DeepMind's Gemma team partnered with Hugging Face in the Fast Gemma Challenge, achieving a 5x inference speed boost for the Gemma 4 model on a single NVIDIA A10G GPU, with peak performance of 4…
VXControl released PentAGI, an open-source autonomous penetration testing tool that orchestrates multiple LLM agents and sandboxed Kali Linux environments to automate offensive security workflows. The…
A developer released a CUDA-native AI guardrail kernel written in C++20 that uses physical memory address binding and branchless multi-dimensional scanning to achieve zero-latency adversarial input de…
AWS introduced disaggregated prefill and decode (DPD) for LLM inference on SageMaker HyperPod, separating compute-bound prefill and memory-bound decode onto different GPU pools connected via EFA RDMA …