A developer outlines inference optimization strategies for small-to-mid-scale deployments, focusing on KV cache management, quantization, and latency tradeoffs. The post provides practical guidance fo…
An independent researcher tested four widely used quantization saliency metrics (AWQ, GPTQ, OWQ, SpQR) by directly perturbing weight channels in small LLMs and measuring real damage. The study found t…
A developer explains quantization techniques for large language models, including GPTQ, AWQ, GGUF, and bitsandbytes, which can reduce model size by up to 75% while preserving most accuracy. The post d…
A comprehensive analysis of 15+ large language model quantization methods categorizes them into four paradigms: CPU-optimized GGUF-based approaches, GPU-native weight-only kernels, NVIDIA's floating-p…
A developer implemented GPTQ quantization from scratch on a nanoGPT model, achieving only 1.1% perplexity degradation across 61 quantized layers. The implementation uses second-order optimization to r…
NVIDIA has released a GenAI LLM Certification Lab that guides developers through building a production-ready fine-tuning and optimization pipeline. The lab covers data preparation, LoRA fine-tuning wi…
Rudrite Research compares three methods for shrinking large language models: AWQ scales salient weights, GPTQ compensates rounding with second-order math, and BitNet trains ternary weights to turn mat…
Local AI users are accumulating hundreds of gigabytes of model files, including multiple quantizations of the same model, image and video generation weights, and RAG datasets, often with duplicates th…
GGUF, GPTQ, and AWQ are the three dominant formats for running quantized large language models locally, each optimized for different hardware and use cases. GGUF, the format used by llama.cpp and its …