LoRA (Low-Rank Adaptation) and QLoRA have become widely adopted methods for efficiently fine-tuning large language models with a fraction of the parameters, solving the problem of massive GPU requirem…
GPU utilization in production Kubernetes clusters averages just 5%, according to Cast AI's 2026 State of Kubernetes Optimization Report, while hardware costs continue to rise. The GPU Cost Optimizatio…
A developer tested Qwen-AgentWorld-35B-A3B, a 35-billion-parameter model designed for agentic reasoning, and found it excels in state tracking and tool-use reliability. The model demonstrated discipli…
An engineer provides a formula to estimate KV cache memory consumption for large language models, showing that the KV cache often becomes the bottleneck before model weights. For Llama 3.1 70B at 128K…
AI inference is profitable, with frontier providers reporting 70%-80% gross margins, contradicting claims that it is unprofitable. Cost estimates show inference costs around $1 per million tokens, whi…
Slack detailed its four-phase evolution to a multi-cloud AI serving platform, moving from self-managed SageMaker to AWS Bedrock and Google Cloud Vertex AI, which improved complex reasoning quality by …
A developer probed 15 Ampere GA100 cards, including CMP 170HX and A100 variants, and published a cross-variant register comparison. The analysis confirms that restriction fuses are product-line identi…
Photoroom reduced inference latency of its convolutional segmentation backbone by 22% on A100 GPUs by switching to PyTorch's channels-last memory format. The change required only four lines of code an…
A developer confirmed that GLM-5.2, a 753B-parameter DeepSeek-Sparse-Attention MoE model, runs on 8x A100 80GB GPUs using vLLM PR #38476, which adds a Triton sparse-MLA backend for Ampere architecture…
A developer explains that the KV cache is the biggest operational bottleneck in production LLM serving on GPUs, consuming more memory than model weights for workloads with high concurrency or long con…
Researchers at arXiv analyzed the performance of Med-DDPM, a 3D generative diffusion model for MRI synthesis, across three generations of NVIDIA GPUs. They identified inefficiencies in memory access a…
NVIDIA released Qwen3.6-35B-A3B-NVFP4, a post-training FP4-quantized variant of Alibaba's 35B MoE model that fits on a single H100 by reducing VRAM from ~71 GB to ~23 GB. The quantization targets weig…
Nvidia H100 GPUs cost $30,000 to $40,000 to buy in 2026, with cloud rental rates ranging from $1.03 to $12.29 per GPU-hour depending on the provider. Neo-clouds offer the cheapest spot instances, whil…
A developer open-sourced AutoMegakernel, a tool that compiles any HuggingFace model into a single persistent megakernel, reducing overhead by launching one kernel per forward pass. It includes a stati…
A widely cited claim that AI inference GPUs last only three years under high utilization is based on an anonymous source from a paid expert network, raising questions about its reliability. Google has…
A developer found that self-hosting open-source AI models can cost between $900 and $4,900 per month when factoring in hidden expenses like GPU rental and maintenance time. The analysis shows that usi…
Researchers have developed POLARIS-9B, a 9-billion-parameter language model that generates long-form stories with improved quality and length adherence, using a novel training recipe that combines a f…
Researchers have developed a Marchenko-Pastur (MP) random-matrix approach for pruning deep neural networks that requires minimal post-pruning fine-tuning. The method provides deterministic data-path c…
The subscription cost for GLM 5.1, a leading open-source AI model from Chinese provider Z.ai, has doubled to $160 per month for its maximum tier, challenging the narrative that open-source models are …
Photoroom reduced its vision-language model inference costs by approximately 62% within three weeks by deploying Bifrost as a semantic caching layer in front of the VLM step of its product-photo diffu…