A developer published a model-neutral Pi extension that displays live throughput metrics—time to first token (TTFT), decode tokens per second, and a client-side prompt-rate estimate—in the coding agen…
An engineering team reported that its Q2 LLM infrastructure bill reached $31,000 against a $12,000 budget after running multi-turn agentic loops in production. The team attributed the overrun to four …
A 2026 guide compares ROCm and Vulkan as backends for hosting local LLMs on AMD GPUs, concluding that the choice depends on the inference engine, GPU generation, and workload rather than being interch…
SkyDiscover-Synthesize (SkySynth), an autonomous pipeline that builds just-in-time specialized systems, synthesizes key-value stores up to 2.3× faster than Redis and FASTER, inference engines with up …
A developer published two Rust CLI demos, gemma-rust and gemma-rust-mcp, that query a self-hosted Gemma 4 E2B model through an OpenAI-compatible HTTP endpoint and through an MCP server, respectively. …
A user testing a multi-GPU LLM development machine reported that the system held up well under sustained load running the Qwen3.8 27B model, despite the machine shipping from 45 Drives with the wrong …
A developer detailed how load testing LLM API integrations can cost thousands of dollars in metered tokens, citing a $3,000 bill from a failed 100,000-request test. The engineer, who built an Autonomo…
A 2026 developer guide compares four local LLM inference engines for workstations and homelabs, highlighting Ollama for turnkey developer ergonomics and agent backends, vLLM for high-throughput batch …
A developer documented migrating an AWS EC2-hosted MCP server for managing Gemma 4 E2B on a vLLM deployment from the MCP Python SDK 1.x (FastMCP) to 2.x (MCPServer). The migration was prompted by an u…
Llm-d is a Kubernetes-native distributed inference serving stack, released open source under Apache 2.0, that adds inference-aware orchestration on top of vLLM and Kubernetes without replacing either.…
Swiss company SOKKAN Inference launched a sovereign "Swiss" inference tier running entirely on four Intel Arc Pro B60 GPUs in Meyrin, Geneva, with no NVIDIA hardware, serving gpt-oss-20b, Qwen3-Coder-…
A developer published a practical guide to fitting long-context LLM inference on 16 GB GPUs by budgeting VRAM for the KV cache, which grows with every active token and sequence. The guide provides a K…
A Kagi user publicly asked the privacy-focused search company to clarify whether outputs from its AI features — Assistant, Universal Summarizer, FastGPT, and Translate — are watermarked or otherwise m…
A Kubernetes Dynamic Resource Allocation (DRA) driver makes composable CXL memory a schedulable cluster resource for cross-node KV-cache reuse in LLM serving, according to an arXiv paper (2609.10790v1…
RunPod Serverless lets developers deploy any Hugging Face model as an OpenAI-compatible API endpoint with per-second billing, scale-to-zero, and flash boot cold starts, according to RunPod. The platfo…
A user detailed a home AI inference rig built on an AMD Epyc Rome 7532 CPU with 128GB DDR4, roughly 5TB of NVMe storage, and 4x AMD V620 GPUs on an open-air crypto-mining bench. The rig runs a LiteLLM…
A former B2B SaaS web developer who switched to inference engineering about a month ago published a skills map arguing the field requires only three service categories: routing and scheduling via Nvid…
Amazon SageMaker Inference launched prefix-aware routing, a routing strategy that directs requests sharing the same prompt prefix to the same instance so cached key-value pairs are reused. Benchmarks …
Amazon launched model caching for Amazon SageMaker Inference on HyperPod, a feature that pre-loads model weights and container images onto cluster nodes so inference pods can read from local NVMe stor…
Bedrock Robotics posted an internship opening for a 2027 Onboard Infrastructure Engineer, ML Inference role based in San Francisco, CA, on a hybrid schedule, with pay not published. The intern will in…