Together AI is hiring a Forward Deployed Engineer (Inference & Post-Training) in Singapore, requiring Mandarin proficiency and permanent residency or citizenship. The role involves optimizing inferenc…
Mistral released Shieldstral, a 3B-parameter Apache-2.0 guard model that moderates text and images by answering a plain-language yes/no question about a policy in a single forward pass, emitting a sin…
AI applications run in data centers, and their performance depends on a five-layer stack of software, hardware, and infrastructure, not just the model. The article explains that training and inference…
Mistral AI released Shieldstral, a 3B-parameter open-weights multimodal moderation model that outputs toxicity and safety scores across multiple axes, trained on ~600K human-judged examples covering h…
Google DeepMind released DiffusionGemma, an open-weight language model that generates text via discrete diffusion, achieving roughly 1,500 output tokens per second on a single H100 compared to about 3…
DeepSeek's V4 Flash 0731 build corrupts integer fields in strict JSON schema outputs when thinking mode is enabled, failing 8 of 13 runs across two request paths, while disabling thinking fixes all ru…
A single AMD MI300X GPU with 192 GB of HBM3 now serves DeepSeek's 284B-parameter DeepSeek-V4-Flash-0731 checkpoint in mixed FP4+FP8 format, requiring nine patch overlays against a vLLM ROCm nightly pl…
A production AI infrastructure engineer reports that six months of operating an on-prem RAG pipeline for internal document search over 2M legal documents with a Llama-3.1-70B model revealed critical p…
A production configuration for running DeepSeek-V4-Flash-0731 on a single AMD MI300X GPU achieves 168.6 tok/s median single-stream decode and 542 tok/s aggregate across 8 concurrent streams, with the …
Homebench, a new open-source command-line tool, benchmarks local large language models for speed, memory, and quality, presenting results in a live terminal leaderboard. The tool, installable via pip,…
Carsen Klock launched Gainz.fast, a new site for benchmarking local AI inference speed, reporting that the Laguna XS 2.1 model on an AMD R9700 with llama.cpp HIP achieved 143.3 tokens per second, a 31…
NVIDIA Dynamo, an Apache-2.0 datacenter-scale inference orchestration layer from NVIDIA, sits above vLLM, SGLang, and TensorRT-LLM to coordinate multi-GPU clusters, with published benchmarks showing u…
A debate on r/SelfHostedAI asks whether self-hosted local AI will go mainstream or remain a niche for enthusiasts, with arguments citing hardware and setup friction versus closing model quality gaps a…
Moonshot AI released the open weights for its K3 model, a 2.8-trillion-parameter sparse MoE with MXFP4 quantization, ranking #1 on Frontend Code Arena and #3 on the Artificial Analysis Intelligence In…
NVIDIA's NeMo team released Molt, a PyTorch-native agentic reinforcement learning framework designed to be compact enough for researchers and AI coding assistants to grasp, with roughly 8.6K lines of …
AMD's Instinct MI355X GPUs now run Moonshot AI's 2.8-trillion-parameter Kimi K3 model at 952 tokens per second per node, undercutting Nvidia's B300 on cost per token despite Nvidia's 1.65x raw through…
A 180-line roofline model built by an independent developer shows speculative decoding on Llama-3-70B collapses at batch 224, turning a 1.92x speedup into a 0.74x slowdown by batch 512, while the same…
DeepSeek released DeepSeek-V4-Flash-0731 on Hugging Face and moved its V4-Flash API to public beta on July 31, 2026, claiming major gains in agentic and coding benchmarks from re-post-training rather …
A team of four developers won third place at the Dell x NVIDIA AI Hackathon by building Squidward, a 100% air-gapped, self-improving IT firewall that runs locally on a Dell GB10 with a Qwen3.6-27B mod…
A benchmark study by an independent researcher found that burstier request arrivals speed up LLM inference, contradicting standard intuition. The analysis of vLLM serving shows that higher burstiness …