Inferact, the company founded by the creators and core maintainers of vLLM, is hiring a remote Member of Technical Staff for Cloud Orchestration with a salary of $200k–400k/yr. The role involves build…
LLMPanel has launched an open-source platform that deploys large language models on any GPU cloud, including RunPod and Vast.ai, without requiring Kubernetes. The tool provisions containers, exposes O…
A LessWrong essay warns that malicious large language models (LLMs) could exploit vulnerabilities in inference engines like vLLM or SGLang to execute arbitrary code on the host machines where their we…
A developer's deep dive on tamiz.pro examines the shift from high-performance LLM serving engines like SGLang to local-first logic frameworks such as OpenLogi, which prioritize data privacy, determini…
A developer highlights the shift in generative AI tooling toward agentic workflows, local execution, and structured outputs. The post demonstrates how native structured outputs with JSON schemas and l…
A bug is a violation of a specification, and no specification exists that prefix caching's variable logits violate, according to an analysis of vLLM and SGLang issues. The vLLM PR #34046 adds an opt-i…
SemiAnalysis announced AgentX 1.0, the world's first fully open source multi-turn agentic coding inference benchmark at 1 million context, released under Apache 2.0, after spending more than $3M build…
On a single 8-GPU AMD Instinct MI355X node, AMD served Kimi Linear 48B-A3B with context lengths from 1024 tokens to 64Mi (67,108,864), using vLLM with FP8 KV cache and tensor parallelism 8, recording …
At Hot Chips 2026, Anurag Agarwal and Radhakrishna Giduthuri presented simulations and projections for High Bandwidth Flash (HBF), a new memory technology that combines SSD-like flash storage with HBM…
VLLM v0.27.1, the open-source inference engine, has unified its scheduler around a fixed token budget per step, making chunked prefill, prefix caching, and speculative decoding stackable by default, a…
Ducklab, an Apache-2.0 self-hosted software development harness built with a Go engine, CLI, and desktop app, has completed 416 runs at a total cost of $176 using local models first, and has developed…
A new 478 KB vector file enables 'abliteration without the weights' by removing refusal directions in activation space at inference time, allowing cybersecurity defenders to run capable models on thei…
Developers are buzzing over a new open-source AI model distributed through decentralized channels with no official landing page or marketing campaign, according to a developer blog post. The anonymous…
Anthropic's Claude Sonnet 5 charges $2.00 per million fresh input tokens, $2.50 to write into a five-minute cache, and $0.20 per read, making two identical calls cost $2.70 with caching versus $4.00 w…
A developer detailed the architecture of a private agentic operating system built on locally-hosted LLMs, drawing lessons from frameworks like Eliza and Hister. The system layers reasoning, memory, an…
Prompt caching in large language model inference engines such as vLLM works by reusing key-value (KV) caches through paged attention and automatic prefix caching, enabling faster responses and lower c…
Researchers introduced ELDR, an expert-locality-aware decode routing method for mixture-of-experts (MoE) models, which reduces median time-per-output-token by 5.9–13.9% across three MoE models and two…
LayoutLens, an AI-powered visual UI testing tool, catches layout and accessibility bugs using deterministic axe-core and geometry checks that run keyless and free in CI, with an optional vision-LLM ti…
A developer detailed the construction of a fully self-hosted RAG pipeline for a Gulf bank that required no data to leave its premises. The project, built on two used 3090 GPUs and open-source tools li…
Maneshwar, developer of the git-lrc AI code reviewer, explains that the perceived speed differences between LLM chatbots like ChatGPT and Gemini stem largely from the inference engine, not the model's…