ParseHawk, a new local-first document AI tool, launched on Hacker News, enabling users to extract structured JSON from PDFs, scans, and images entirely on their own hardware without sending data to th…
A single H200 GPU with 141GB HBM3e cannot comfortably run DeepSeek V4 Flash (284B total, 13B active parameters) due to VRAM constraints, even with 2TB system RAM for offloading. The model requires an …
Ora Computing launched an automated LLM compression engine that reduces model size by up to 70% with minimal accuracy loss, enabling deployment on edge devices, on-prem servers, or cloud infrastructur…
A developer built Modelplane, an open-source inference control plane using Crossplane, to manage GPU fleets across clouds, neoclouds, and on-premise environments. The platform allows platform teams to…
A developer achieved 3x better batch throughput with Qwen3.5 by fixing vLLM on AMD's Strix Halo using the Lemonade Server, enabling more efficient AI inference on AMD hardware.…
UC San Diego researchers developed DFlash, a speculative decoding method that uses a lightweight block diffusion model to draft entire token blocks in parallel, achieving up to 6.08x speedup on Qwen3-…
A developer built an 11-chapter interactive guide explaining how LLM inference works, centered around nano-vLLM, a 1,200-line Python reimplementation of the vLLM serving engine. The guide covers algor…
A developer built a production-ready LangGraph ReAct agent that exposes an OpenAI-compatible API, supports multi-model switching via a gateway, and includes one-line tracing with Langfuse. The deploym…
IBM Research, Red Hat, and NxtGen Cloud Technologies demonstrated that using llm-d to serve AI models on mixed GPU hardware can boost inference speeds by 3 to 5 times and double throughput, enabling e…
Developers can deploy Llama 3.1 8B behind vLLM's OpenAI-compatible API on a rented GPU instance, achieving thousands of output tokens per second through continuous batching. The tutorial covers instal…
SkyPilot launched SkyPilot Endpoints, a production-ready LLM inference system that deploys and manages inference across multiple Kubernetes clusters from a single YAML specification. The system handle…
NVIDIA's confidential computing (CC) on Blackwell GPUs introduces performance overheads ranging from 1% to over 100% for inference and 18–32% for training, according to benchmarks from Tinfoil. The pr…
Modelplane, an open-source control plane for AI inference built on Crossplane, is being released to manage GPU clusters as a single inference fleet, handling provisioning, model placement, autoscaling…
VLLM, a large-model inference serving framework, uses Python for control flow but pushes arithmetic into compiled C++ and CUDA kernels to avoid interpreter overhead. The Python/C++ boundary crossing i…
Neuralwatt launched the first AI inference API with energy-based pricing, charging per kilowatt-hour instead of per token to provide transparency into power consumption and cost. The platform offers r…
Local AI serving engines like Ollama and vLLM offer different trade-offs between ease of use and throughput, with Ollama ideal for single users and vLLM for high-concurrency production workloads. The …
A new Local LLM Hardware Calculator helps users estimate memory requirements for running large language models on their own hardware, factoring in weights, KV cache, and overhead. The tool also compar…
A developer confirmed that GLM-5.2, a 753B-parameter DeepSeek-Sparse-Attention MoE model, runs on 8x A100 80GB GPUs using vLLM PR #38476, which adds a Triton sparse-MLA backend for Ampere architecture…
An AI infrastructure engineer at a major industrial company seeks advice on transitioning from SRE-focused work to a proper software engineering role in AI infrastructure, asking for skills, resources…
A developer explains that the KV cache is the biggest operational bottleneck in production LLM serving on GPUs, consuming more memory than model weights for workloads with high concurrency or long con…