Pokee AI released Pokee-Isaac 28B, a 28B-parameter proprietary non-decoder-only model claiming a 10M-token context that fits on a single RTX 4090 (24GB) in quantized form. Vendor-reported benchmarks i…
An engineer's benchmarking guide introduces a cache hit rate metric for measuring prefix caching effectiveness in LLM serving, implemented in the open-source tool llmperf-rs. The metric calculates the…
Together AI is hiring a Forward Deployed Engineer (Inference & Post-Training) in Singapore, requiring Mandarin proficiency and permanent residency or citizenship. The role involves optimizing inferenc…
MiniMax released the core weights for its H3 video model, making it the first publicly downloadable model to lead a category in Artificial Analysis' video arena, ranking first for video editing, secon…
Cloudflare published an engineering post detailing how it quantizes models on Workers AI, including Moonshot's Kimi K2.6 with FP8 KV cache and Z.ai's GLM 5.2 with INT4 weights, achieving up to 41% hig…
Cloudflare's production benchmarks for Moonshot's Kimi K2.6 and Z.ai's GLM 5.2 show that quantizing the decode phase, not the prefill, yields the biggest cost and throughput gains for trillion-paramet…
Cloudflare's Workers AI has implemented three optimizations—quantizing the KV cache, compressing model weights, and protecting the shared cache—to run Moonshot's Kimi K2.6 and Z.ai's GLM 5.2 more effi…
MiniMax has released two H3 base checkpoints on Hugging Face under a standard community license, but the full 2K workflow still requires hosted services. The 288 GB repository includes H3-Base-FL2VA a…
NVIDIA Dynamo, an Apache-2.0 datacenter-scale inference orchestration layer from NVIDIA, sits above vLLM, SGLang, and TensorRT-LLM to coordinate multi-GPU clusters, with published benchmarks showing u…
AMD's Instinct MI355X GPUs now run Moonshot AI's 2.8-trillion-parameter Kimi K3 model at 952 tokens per second per node, undercutting Nvidia's B300 on cost per token despite Nvidia's 1.65x raw through…
AMD released Instella-MoE-16B-A3B, a fully open Mixture-of-Experts language model with 16B total parameters and 2.8B active per token, trained from scratch on Instinct MI300X and MI325X GPUs. The mode…
A research paper titled 'The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility' reveals that the choice of inference engine can significantly affect LLM output…
LocalAI, a self-hosted runtime that runs AI workloads on user-controlled hardware, offers an OpenAI-compatible API supporting text generation, vision, speech, image and video generation, embeddings, a…
Moonshot AI released the 2.8-trillion-parameter Kimi-K3 open-weight model on July 27, 2026, and Fixstars successfully ran inference on a single-node NVIDIA B300 x8 system using SGLang's DCP support, a…
A developer detailed how to build a secure local AI agent using SGLang and Olares, warning that local LLMs are not automatically secure and require defenses against prompt injection, tool misuse, and …
Together AI introduces ThunderAgent, a system for high-throughput agentic inference that achieves up to 2.5× higher single-node throughput and 2.4× speedup on an 8-node cluster with near-linear scalin…
Moonshot AI released the full Kimi K3 open weights on July 27, 2026, a 2.8-trillion-parameter mixture-of-experts model with a one-million-token context window and a 1.4 TB download. The model uses nat…
Moonshot AI released the full Kimi K3 model weights and technical report on July 27, 2026, making its 2.8 trillion parameter mixture-of-experts model available to developers and researchers. The model…
Baseten has launched day-0 API support for Kimi K3, a new open frontier model from Moonshot AI with 2.8 trillion parameters, making it the largest open model to date. The API runs on NVIDIA GB300 NVL7…
Moonshot AI has released Kimi-K3, an image-text-to-text model under a permissive license that allows use, modification, and commercial distribution, with support for Transformers, vLLM, SGLang, and Do…