Abacus 0.6.0, the open-source coding agent from rethink, introduces persistent memory that improves over time, with six mechanisms including papercuts, memories, tethering, and a hive for delegation. …
Open-weight AI models are the only real hedge against a billionaire-led AI industry, according to a tech commentary, because they provide data sovereignty, latency control, and customization through f…
Amazon Web Services (AWS) introduced a tiered KV cache architecture on Amazon SageMaker HyperPod that extends the cache hierarchy beyond GPU and CPU memory into a shared, distributed NVMe pool using C…
A developer's local LLM crashed during a RAG experiment due to CUDA out-of-memory errors caused by an oversized 32,768-token context window, which inflated the KV cache. Fixing the issue by sliding LM…
Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter mixture-of-experts model with 95 billion active parameters per token, requires roughly 4.8 terabytes of memory at FP16/BF16 for weights alone, making local …
A developer documented their migration of a Cloud TPU workload from the deprecated Cloud TPU API to Compute Engine, moving a v6e-1 chip serving gemma-4-E2B-it under vLLM. The migration required mappin…
Proxima, an out-of-tree vLLM plugin implementing STAR-KV low-rank KV cache compression, serves 4x more concurrent requests at 8192 context and boots at 16384 context where plain vLLM refuses to start,…
NVIDIA introduced Nemotron 3.5 Lightning, a customizable open 30B mixture-of-experts model for always-on agents, delivering up to 4x faster token generation and 30% faster time to completion compared …
The maintainers of the open-source vLLM inference engine at Inferact are hiring, according to a post praising the team as among the most skilled engineers globally. The post highlights their role in b…
Most companies overpay for AI inference by 2x to 5x, sometimes by an order of magnitude, because their inference stack doesn't match their use case priorities, said Meryem Arik, co-founder of inferenc…
VLLM's batch-invariance documentation defines a feature ensuring a request produces the same inference result regardless of batch size, composition, request order, or scheduling under a fixed hardware…
A developer benchmarked Google's Gemma 4 E2B model on a Cloud TPU v6e-1 (Trillium) and found that while the chip offers 3.6x more KV cache memory than v5e-1, it costs 2.25x more and delivers only 1.62…
LiquidAI and collaborators released fuse-1 Lite, a 5.72 billion parameter mixture-of-experts model that transplants 960 coding experts from Qwen3.6-35B-A3B into LiquidAI's LFM2.5-2.6B host model, achi…
NVIDIA released Nemotron-3.5-Lightning, a 30B mixture-of-experts model with 3B active parameters, on the same day Modelplane, an open-source fleet-level control plane for inference, achieved zero-day …
Fuse-1 Lite, a 5.72B parameter mixture-of-experts coding model from LiquidAI, can run locally with VRAM needs ranging from 3.36 GB in 4-bit quantized form to about 12 GB in full bfloat16 precision, ac…
Leanpub has launched 'LLM Engineering, from Component to Production' by Ali Aouf, a practical, measurement-first guide to building a local-first agentic retrieval-augmented generation system. The book…
Meta CEO Mark Zuckerberg advocates for releasing 'imperfect' AI models early to leverage real-world telemetry and community feedback, a strategy that turns users into a massive QA team and accelerates…
Cast AI benchmark testing shows that continuous batching at batch size 8 reduces Llama 3.1 70B inference cost on a single H100 from approximately $0.60-$0.80 per million tokens to $0.15-$0.25 per mill…
Meta's Superintelligence Labs released Muse Glimmer, a 30-billion-parameter dense multimodal model with 131K context, under the Apache 2.0 license, marking a departure from the restrictive Llama Commu…
Meta announced it will release an open-weight version of Muse Spark 1.2 and released Muse Glimmer, a 30B agentic model with open weights under Apache 2.0, which can run on 24GB of VRAM. The model is q…