NF4

A hands-on benchmark of Qwen3.8 27B on an RTX 4090 found that 4-bit quantization (NF4 and AWQ INT4) preserves near-baseline quality, with MMLU scores of 59.8% and 59.5% versus 61.2% for FP16, while 1-…

GGUF, GPTQ, and AWQ are the three dominant formats for running quantized large language models locally, each optimized for different hardware and use cases. GGUF, the format used by llama.cpp and its …

A new paired-minimum detectable effect (MDE) budget for 4-bit quantization benchmarks, derived from classical sample-size calculations, provides benchmark designers a one-line formula to pre-register …