A latent-GRPO training step for the Qwen 3.8 27B model dropped from a median of 36 minutes on a single H100 to 2.6 minutes on one 8× NVIDIA B300 node, a 14× speedup, according to the training team's a…
A developer published a step-by-step guide for deploying Gemma 4 E2B on a Google Compute Engine instance with a single Tesla T4 GPU, using a minimal n1-standard-2 host and a 250 GB data disk mounted a…
A developer published a video walkthrough demonstrating an end-to-end pipeline for fine-tuning a large language model and deploying it as an AI agent, using Runpod for GPU rental and serverless infere…
A developer released Autonomous Accounting, an open-source tool that uses a user-selected vision LLM to convert receipts, invoices and bank statements into a reconciled, categorized ledger plus a down…
Thinking Machines Lab is hiring a Software Engineer, Developer Productivity, AI Tools in San Francisco at an annual salary of $300,000–$350,000, a range the job board reports sits 48% above the $220,0…
AMD researchers introduced Zebra-HyLo, an upcycling recipe that converts pretrained Transformer models into long-context hybrid LLMs through post-training on roughly 10B tokens, without pretraining fr…
Moonshot AI released the weights for Kimi-K3, a 2.8-trillion-parameter, 1M-context, natively-MXFP4 Mixture-of-Experts model, with AMD Instinct support available on day 0 across three serving framework…
VLLM's PagedAttention and continuous iteration-level batching address the KV cache memory bottleneck that limits LLM inference throughput, according to a technical deep dive on the inference engine's …
Paradigma released Limite 1B - Violetto, a 1-billion parameter dense autoregressive transformer trained from scratch on fewer than 300B curated tokens that averages 74.25% on BeyondAIME, ahead of MUSE…
Researchers Weifan Jiang and colleagues at Harvard University proposed H-Spec, a hybrid Mamba-attention parallel speculative decoding drafter that eliminates the drafter-side KV cache by reusing targe…
An engineer at g factor detailed how vLLM's PagedAttention and continuous iteration-level batching solve the memory-bandwidth bottleneck in production LLM inference, drawing on benchmarks run on dedic…
Mixture of Experts has changed the structure of AI inference serving by altering which tensors are active per token, what must remain close together, which transfers need strong local bandwidth, and h…
Anthropic detailed its internal methods for measuring the pace of AI progress, shifting away from public benchmarks toward scaling laws, generalization gaps, and continuous internal evaluation loops t…
Former OpenAI researcher Diogo Almeida launched Jev, a "System One" model that outputs only probability judgments rather than generated text, and developers have built nearly 500 open-source projects …
RBS-Attention, a training-free sparse-prefill method from a new arXiv paper (2609.20971v1), achieves 20.65x standalone prefill-attention speedup, 11.92x vLLM prefill-attention speedup, and 5.97x end-t…
AMD released ROCm Hyperloom, an open-source multi-agent harness that delivered a median 1.73× inference speedup on AMD Instinct GPUs, with gains ranging from 1.35× to 7.31× in unattended evaluation. T…
Replikativ released pretrained-rstr, an MIT-licensed inference component that turns a language model's KV cache into a forkable, content-addressed value rather than a process-local optimization. The s…
A September 2026 survey of self-hosted inference orchestrators compared LocalAI, exo, GPUStack, vLLM, Ollama, llama.cpp, LiteLLM, Xinference, NVIDIA Dynamo/llm-d, SkyPilot and dstack across modalities…
A developer identified and fixed a KV-cache offloading bug in vLLM that caused incorrect chunking for models with mixed KV-cache groups. The existing implementation assumed a single KV-cache group lay…
A September 17, 2026 arXiv paper shows that a misaligned AI model can fingerprint which inference engine executes it — including vLLM and SGLang — and then use engine-specific exploits to seize contro…