Two open-weight coding models, Kimi K2.7 Code from Moonshot AI and GLM-5.2 from Z.ai (Zhipu AI), launched days apart in June 2026, both designed for agentic coding workflows and supporting vLLM and SG…
Researchers present Fleet, a hierarchical task-based abstraction for multi-die GPUs that introduces Chiplet-tasks to exploit chiplet-level locality and synchronization. On AMD Instinct MI350 with Qwen…
Thinking Machines Lab, Inc. released Inkling, a general-purpose multimodal model with 975 billion total parameters and 41 billion active parameters, on July 15, 2026 under an Apache 2.0 license. The m…
Seven Python frameworks for orchestrating local AI agents are gaining adoption in 2026, according to a technical roundup. Ollama, a lightweight runtime for running open-source LLMs on local hardware, …
Version 1.21.0 of RFC BF introduces a YAML-based providers block that replaces hardcoded LLM providers with a config-declared registry supporting pluggable drivers for Anthropic, OpenAI, Gemini, DeepS…
A developer released AI-CLI, a tiny C terminal assistant that connects user requests to a local LLM and executes returned shell actions directly, supporting over 20 platforms and most LLM engines. The…
NVIDIA released Nemotron-Labs-TwoTower on July 1, achieving 2.42 times faster inference throughput at 98.7% of the baseline model's benchmark quality by adding a second neural network tower trained on…
A user inquired about deploying a multi-GPU environment on Hugging Face Spaces, requesting 8 GPUs with over 40GB VRAM each to run a generative model on 4 GPUs and a vLLM model on the remaining 4 GPUs,…
XGrammar, a grammar-constrained decoding library used by vLLM, SGLang, TensorRT-LLM, and MLC-LLM, masks invalid tokens before the model samples, guaranteeing valid JSON output and eliminating retry lo…
SGLang, an open-source LLM inference engine, introduces RadixAttention for prefix caching and grammar-constrained decoding, achieving 25x throughput improvement over vLLM for structured output tasks. …
Mistral released Leanstral 1.5, a formal verification agent built on Lean 4, on July 2, and in its first public test against 57 open-source repositories it found five bugs that human maintainers had n…
Atuin has open-sourced the Atuin AI server, allowing users to self-host the terminal-focused AI agent that provides agentic tools directly in the shell. The server supports any OpenAI-compatible endpo…
Intel's Arc Pro B70, priced at $949 for 32GB of GDDR6 but now selling above $1,100 due to market shortages, offers up to 85% higher token throughput and 6.2x faster time-to-first-token versus NVIDIA's…
A developer released trollbridge, a policy-enforced HTTP proxy designed to let coding agents operate freely while blocking unsafe requests. The tool uses deterministic allow/deny lists for most traffi…
Open models have reached impressive capability levels, but the surrounding platform infrastructure—including tool calling, streaming, file handling, web search, code execution, prompt caching, and tok…
Open-source coding agents can now run entirely on local consumer hardware such as a 2020-era NVIDIA RTX 3090, eliminating usage-based pricing from AI vendors. The setup uses a quantized Gemma4 model s…
A new study reveals that the multiplicative repetition penalty used across major LLM inference engines (HuggingFace, vLLM, llama.cpp) is fundamentally flawed because it branches on the sign of raw log…
MemStitch, a zero-copy context bridging gateway for vLLM, achieves a 25x speedup in Time-to-First-Token (TTFT) by allowing multi-agent GPU inference to share cached prefixes, reducing prefill latency …
A hybrid routing strategy that sends simple agent tasks to local AI models and complex ones to cloud APIs can cut costs by more than half for teams running high-volume workflows, according to a practi…
Two used RTX 3090s offer 48GB of VRAM for about half the cost of a single RTX 5090, enabling local 70B LLM inference at 14-21 tokens/sec, while the 5090's 32GB cannot run a quality 70B model. For mode…