A developer detailed the deployment of Google's Gemma 4 2B model on a single Cloud TPU v5e chip using MCP and Antigravity CLI, achieving measured throughput and latency. The v5e chip, at roughly half …
Moonshot AI released the weights for Kimi K3 on July 27, a 2.8-trillion-parameter Mixture-of-Experts model with a 1-million-token context window, scoring third globally behind Claude Fable 5 Max and G…
Researchers introduced JudgeArena, an open-source framework unifying major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and …
An engineer's benchmarking guide introduces a cache hit rate metric for measuring prefix caching effectiveness in LLM serving, implemented in the open-source tool llmperf-rs. The metric calculates the…
Perplexity is hiring a Member of Technical Staff (Software Engineer, GPU Cluster Infrastructure) in San Francisco with a salary range of $250k–485k/yr, to build a self-serve compute platform for its G…
On-premise AI, which involves running AI models on hardware owned by the organization, is gaining traction in 2026 due to data control, predictable costs, and reliable access, according to a guide fro…
Together AI is hiring a Forward Deployed Engineer (Inference & Post-Training) in Singapore, requiring Mandarin proficiency and permanent residency or citizenship. The role involves optimizing inferenc…
Mistral released Shieldstral, a 3B-parameter Apache-2.0 guard model that moderates text and images by answering a plain-language yes/no question about a policy in a single forward pass, emitting a sin…
AI applications run in data centers, and their performance depends on a five-layer stack of software, hardware, and infrastructure, not just the model. The article explains that training and inference…
Mistral AI released Shieldstral, a 3B-parameter open-weights multimodal moderation model that outputs toxicity and safety scores across multiple axes, trained on ~600K human-judged examples covering h…
Google DeepMind released DiffusionGemma, an open-weight language model that generates text via discrete diffusion, achieving roughly 1,500 output tokens per second on a single H100 compared to about 3…
DeepSeek's V4 Flash 0731 build corrupts integer fields in strict JSON schema outputs when thinking mode is enabled, failing 8 of 13 runs across two request paths, while disabling thinking fixes all ru…
A single AMD MI300X GPU with 192 GB of HBM3 now serves DeepSeek's 284B-parameter DeepSeek-V4-Flash-0731 checkpoint in mixed FP4+FP8 format, requiring nine patch overlays against a vLLM ROCm nightly pl…
A production AI infrastructure engineer reports that six months of operating an on-prem RAG pipeline for internal document search over 2M legal documents with a Llama-3.1-70B model revealed critical p…
A production configuration for running DeepSeek-V4-Flash-0731 on a single AMD MI300X GPU achieves 168.6 tok/s median single-stream decode and 542 tok/s aggregate across 8 concurrent streams, with the …
Homebench, a new open-source command-line tool, benchmarks local large language models for speed, memory, and quality, presenting results in a live terminal leaderboard. The tool, installable via pip,…
Carsen Klock launched Gainz.fast, a new site for benchmarking local AI inference speed, reporting that the Laguna XS 2.1 model on an AMD R9700 with llama.cpp HIP achieved 143.3 tokens per second, a 31…
A guide from Autonomous AI outlines how to build a local AI server by budget tier, emphasizing that VRAM is the most critical specification for running AI models locally. The guide provides VRAM requi…
NVIDIA Dynamo, an Apache-2.0 datacenter-scale inference orchestration layer from NVIDIA, sits above vLLM, SGLang, and TensorRT-LLM to coordinate multi-GPU clusters, with published benchmarks showing u…
A debate on r/SelfHostedAI asks whether self-hosted local AI will go mainstream or remain a niche for enthusiasts, with arguments citing hardware and setup friction versus closing model quality gaps a…