Richael (dh8116), a Year 11 student in Auckland, built Soulor AI, an AI companion app with persistent memory and a five-perspective Simulation Mode, on a LoRA fine-tune of Qwen3-14B served with vLLM o…
A developer published a walkthrough for replacing cloud LLM APIs with local small language models for routine tasks like JSON parsing, log analysis, and ticket routing. The guide provides a production…
A user on the Level1Techs forum reported running four AMD Radeon R9700 GPUs on an AM4 platform to requantize a DeepSeek model to MXFP4 and run it in the vLLM engine, with the first layer of a 48-layer…
Prime Intellect launched Prime Inference, a serverless and reserved-capacity serving platform for frontier open-source models that processed nearly a trillion tokens per day internally before public r…
IQuest released IQuest-Q1, an open-weight 320B-parameter Mixture-of-Experts coding model that activates only 15B parameters per token and supports a 524,288-token context window, with official deploym…
A developer repacked Google's quantization-aware-trained Gemma 4 weights for vLLM and served them on a single Google Cloud TPU v5e chip, publishing the scripts and per-record results on GitHub. The in…
Janus, an MIT-licensed single Go binary from the Vibra-Ingenn project, launched as a Show HN on October 1 with 42+ points, wrapping llama.cpp's Vulkan backend to run GGUF models on AMD, Intel, or NVID…
Clustering two NVIDIA DGX Sparks over a 200 GB RoCE RDMA cable and running tensor-parallel inference with vLLM lets the pair serve models in the 150 to 160 GB range, such as DeepSeek V4 Flash, that do…
A technical blog post argues that non-reproducible LLM metadata extraction should be treated as a data-cleaning and statistical problem rather than a determinism problem, citing He (2025), who obtaine…
A developer documented running two unlocked NVIDIA CMP 170HX mining cards (GA100, A100-class) for LLM inference on a single AMD EPYC 7443 / ASRock Rack ROMED8-2T host, unlocking them to 64 GB and 74 S…
A developer evaluated speculative decoding in vLLM on AMD Instinct MI300X and MI355X GPUs, testing draft methods including MTP, EAGLE-3, DFlash, and DSpark that propose candidate tokens verified in a …
Researchers introduced Decode-Latency Feedback Prefill (DLFP), a model-free controller implemented in vLLM that resizes prefill chunks based on observed decode intervals, cutting P99 inter-token laten…
Developer Rizwan Reza released Smartly, a CLI tool that sends one plain English sentence to an LLM, validates the reply is a single shell command line, and runs it, with install via `brew install rizw…
A developer's Kubernetes operator, pd-ratio-coordinator, failed to compile on its committed KRAG-1 branch, with two real bugs — a DeepCopy type mismatch in register.go and four unqualified Bottleneck …
A developer benchmarked the same Gemma 4 E2B build served from an Amazon SageMaker endpoint and from a plain EC2 instance with an identical NVIDIA T4 or L4 GPU, finding decode speed and answers match …
A developer deployed Gemma 4 (E2B, E4B, 12B and 26B A4B) to Amazon SageMaker on a single NVIDIA T4 (ml.g4dn.xlarge), measuring decode throughput at 0.77x to 0.82x of an L4 while producing identical an…
Kong released AI Gateway 2.2, adding native support for TypeSafe JEV as a provider and format, support for Skills APIs, and a new passthrough mode for custom or non-standard AI interfaces such as vLLM…
IQuest released IQuest-Q1, an open-weight 320B-parameter Mixture-of-Experts language model with 15B active parameters per token and a 512K-token context window, built for agentic coding and multi-step…
IQuest released IQuest-Q1, a 320-billion-parameter Mixture-of-Experts model with roughly 15 billion active parameters per token, an 88-layer transformer, 256 experts (8 active), and a 524,288-token co…
AMD detailed its AMD Inference Microservices (AIMs), standardized Docker-based inference microservices that automatically select a runtime profile from inputs including model, precision, engine, laten…