AMD and the vLLM project detailed five speculative decoding methods for vLLM on AMD GPUs, achieving up to 2.59x throughput improvement for certain large language models. The techniques enable producti…
Researchers introduced SSR (Self-Speculation for Reasoning Models), a training-free self-speculative decoding method that uses partial chain-of-thought as a drafter and full chain-of-thought as a veri…
Apertura, a from-scratch Objective-C++/MLX rebuild of Google's Gemma-4 for Apple Silicon, runs the entire Gemma-4 text family natively on a Mac with bit-exact conformance to the PyTorch reference, ach…
A new API from an unnamed startup returns a hallucination score for every token generated by frontier language models, using a white-box proxy model (Gemma-4) trained on outputs from GPT-5.4 Nano, ach…
Eliza, an open-source agentic operating system written in TypeScript with 19k GitHub stars, shipped an update that lets users run the entire AI pipeline on-device via the Eliza-1 model family (Gemma-4…
A new technique called KV-Cache Grafting boosts a frozen Gemma-4-12B model from 80.0% to 93.3% on AIME 2025 without changing any weights, according to a paper submitted to arXiv on 15 Jul 2026. The me…
A developer released AI-CLI, a tiny C terminal assistant that connects user requests to a local LLM and executes returned shell actions directly, supporting over 20 platforms and most LLM engines. The…
A developer ported Google's Gemma-4 model family (2B, 4B, 12B) to AWS Inferentia2, achieving up to 44 tok/s on the smallest model. The port required workarounds for three architectural features—mixed …
Mlx-optiq, a new open-source tool, enables per-layer mixed-precision quantization of large language models on Apple Silicon, allowing users to run, fine-tune, and serve LLMs locally on Macs without a …
A developer has published a guide for running Google's Gemma-4 12B instruction-tuned model on Windows Subsystem for Linux 2 (WSL2) using the llama.cpp framework. The process involves installing build …
A developer successfully ran Google's Gemma-4 E2B-it large language model on a Raspberry Pi 4 using llama.cpp, achieving text generation speeds of 1.5 to 1.8 tokens per second. The project involved co…