Cast AI benchmark testing shows that continuous batching at batch size 8 reduces Llama 3.1 70B inference cost on a single H100 from approximately $0.60-$0.80 per million tokens to $0.15-$0.25 per mill…
Meta's Superintelligence Labs released Muse Glimmer, a 30-billion-parameter dense multimodal model with 131K context, under the Apache 2.0 license, marking a departure from the restrictive Llama Commu…
Meta announced it will release an open-weight version of Muse Spark 1.2 and released Muse Glimmer, a 30B agentic model with open weights under Apache 2.0, which can run on 24GB of VRAM. The model is q…
A software engineer's analysis argues that MoE routing in transformer inference is fundamentally the same problem as CPU branch prediction, and that KV cache management techniques such as prefix cachi…
An engineer details the GPU memory requirements for self-hosting large language models, explaining that the KV cache, not model weights, is the primary bottleneck under real traffic. The post provides…
Mirafold, an MIT-licensed open-source tool, launched a browser-based generative UI layer for terminal coding agents Claude Code, Codex, and Gemini CLI, turning raw agent output into live cards, depend…
Meta released Muse Glimmer, a 30B-parameter multimodal AI model under the Apache 2.0 license, designed for local deployment in privacy-aware applications such as coding and document analysis. The mode…
Sail Research has open-sourced HTDYM (How To Deploy Your Model), an internal performance modeling infrastructure that helps determine the optimal hardware and sharding configuration for serving large …
A developer self-hosted a lite agent backend on a single Google Cloud TPU v5e chip, achieving 1,496 output tokens/sec aggregate throughput with 8.02 ms per-token latency for Gemma 4 E2B under vLLM. Th…
Intel has released LLM Scaler, a GenAI solution for text, image, and video generation optimized for Intel Arc Pro B60 and B70 GPUs, with the latest version intel/llm-scaler-vllm:0.21.0-b2 adding Multi…
An independent developer has released DeepSeek-V4-Flash-0731-Latent-Reasoning, a 284B-parameter MoE model with a 35.7M-parameter reasoning head that performs latent reasoning in a 1024-dimensional spa…
Nicholai Mitchko, InterSystems' director of AI enablement, released DeepSeek-V4-Flash-0731-Latent-Reasoning, a self-hosted latent-reasoning model built on DeepSeek-V4-Flash that requires Blackwell-cla…
Google has open-sourced TPU Raiden, an inference optimization library for KV-cache transfer in large language model serving, under the Apache-2.0 license on GitHub, positioning it as a direct counterp…
VLLM Launcher, a Windows desktop app for managing local LLM inference through WSL2, has been released as v0.1.0-alpha. The app supports multiple engines including vLLM, SGLang, and llama.cpp, and prov…
Red Hat published a post arguing that 'the CPU is back' for LLM inference, citing an Intel and Georgia Tech paper that found CPU-side tool processing accounts for 50–90% of total latency in agentic wo…
Liquid AI released LFM2.5-2.6B, an on-device agentic model with 2.69B total parameters, a 131,072-token context window, and a 128,000-token vocabulary, pre-trained on approximately 34 trillion tokens.…
ROSA, a robotics foundation model serving system for robot factories, improves factory productivity by up to 12.06x over conventional dedicated serving systems, according to a paper proposing the syst…
Pokee AI released Pokee-Isaac 28B, a 28B-parameter proprietary non-decoder-only model claiming a 10M-token context that fits on a single RTX 4090 (24GB) in quantized form. Vendor-reported benchmarks i…
A benchmark of 2,040 CRM agent rollouts found that over half of the small open-weights model gemma-4-E4B-it's 'failures' actually contained the correct answer but were graded zero because the model ne…
Prefill/decode disaggregation, which runs LLM prefill and decode phases on separate hardware pools, cut p99 inter-token latency by 66% (from 88ms to 30ms) in a Go simulation, at the cost of higher tim…