A software engineer's analysis argues that MoE routing in transformer inference is fundamentally the same problem as CPU branch prediction, and that KV cache management techniques such as prefix cachi…
An engineer details the GPU memory requirements for self-hosting large language models, explaining that the KV cache, not model weights, is the primary bottleneck under real traffic. The post provides…
Mirafold, an MIT-licensed open-source tool, launched a browser-based generative UI layer for terminal coding agents Claude Code, Codex, and Gemini CLI, turning raw agent output into live cards, depend…
Meta released Muse Glimmer, a 30B-parameter multimodal AI model under the Apache 2.0 license, designed for local deployment in privacy-aware applications such as coding and document analysis. The mode…
A developer self-hosted a lite agent backend on a single Google Cloud TPU v5e chip, achieving 1,496 output tokens/sec aggregate throughput with 8.02 ms per-token latency for Gemma 4 E2B under vLLM. Th…
Intel has released LLM Scaler, a GenAI solution for text, image, and video generation optimized for Intel Arc Pro B60 and B70 GPUs, with the latest version intel/llm-scaler-vllm:0.21.0-b2 adding Multi…
An independent developer has released DeepSeek-V4-Flash-0731-Latent-Reasoning, a 284B-parameter MoE model with a 35.7M-parameter reasoning head that performs latent reasoning in a 1024-dimensional spa…
Nicholai Mitchko, InterSystems' director of AI enablement, released DeepSeek-V4-Flash-0731-Latent-Reasoning, a self-hosted latent-reasoning model built on DeepSeek-V4-Flash that requires Blackwell-cla…
Google has open-sourced TPU Raiden, an inference optimization library for KV-cache transfer in large language model serving, under the Apache-2.0 license on GitHub, positioning it as a direct counterp…
VLLM Launcher, a Windows desktop app for managing local LLM inference through WSL2, has been released as v0.1.0-alpha. The app supports multiple engines including vLLM, SGLang, and llama.cpp, and prov…
Red Hat published a post arguing that 'the CPU is back' for LLM inference, citing an Intel and Georgia Tech paper that found CPU-side tool processing accounts for 50–90% of total latency in agentic wo…
Liquid AI released LFM2.5-2.6B, an on-device agentic model with 2.69B total parameters, a 131,072-token context window, and a 128,000-token vocabulary, pre-trained on approximately 34 trillion tokens.…
ROSA, a robotics foundation model serving system for robot factories, improves factory productivity by up to 12.06x over conventional dedicated serving systems, according to a paper proposing the syst…
Pokee AI released Pokee-Isaac 28B, a 28B-parameter proprietary non-decoder-only model claiming a 10M-token context that fits on a single RTX 4090 (24GB) in quantized form. Vendor-reported benchmarks i…
Prefill/decode disaggregation, which runs LLM prefill and decode phases on separate hardware pools, cut p99 inter-token latency by 66% (from 88ms to 30ms) in a Go simulation, at the cost of higher tim…
A developer detailed the deployment of Google's Gemma 4 2B model on a single Cloud TPU v5e chip using MCP and Antigravity CLI, achieving measured throughput and latency. The v5e chip, at roughly half …
Moonshot AI released the weights for Kimi K3 on July 27, a 2.8-trillion-parameter Mixture-of-Experts model with a 1-million-token context window, scoring third globally behind Claude Fable 5 Max and G…
Researchers introduced JudgeArena, an open-source framework unifying major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and …
An engineer's benchmarking guide introduces a cache hit rate metric for measuring prefix caching effectiveness in LLM serving, implemented in the open-source tool llmperf-rs. The metric calculates the…
Perplexity is hiring a Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure) in San Francisco with a salary range of $250k–485k/yr, to build a self-serve compute platform for its G…