Portail, a sovereign Rust AI gateway, has released a BitNet B1.58 SIMD tensor engine achieving 16.9k tokens per second with 8.0x FP16 compression, enabling real-time sliding window response latencies …
Indian smartphone manufacturers are pushing 12GB or 16GB of RAM as the new standard for mid-range devices to support on-device AI features, which consume significant memory through model weights and K…
DigitalOcean published a tutorial on June 19 demonstrating how to compress large language models using SparseGPT and Wanda pruning methods for GPU cloud deployment, targeting reduced inference costs a…
A developer calculated the actual VRAM requirements for running Llama 3 8B and Gemma 2 9B locally, revealing that the KV cache can consume far more memory than the model weights, especially at longer …
A developer argues that balanced ternary (-1, 0, +1) could replace binary for AI hardware, citing 20× model compression, 3× inference speedup, and 8× power reduction. Microsoft's BitNet b1.58 demonstr…
GGUF, GPTQ, and AWQ are the three dominant formats for running quantized large language models locally, each optimized for different hardware and use cases. GGUF, the format used by llama.cpp and its …
A new paired-minimum detectable effect (MDE) budget for 4-bit quantization benchmarks, derived from classical sample-size calculations, provides benchmark designers a one-line formula to pre-register …
The Kog AI team implemented a single-kernel LLM inference engine on AMD MI300X GPUs, achieving over 3,000 output tokens per second per request for a 2B-parameter model in FP16 precision. The monokerne…
Researchers have developed ThriftAttention, a mixed-precision attention algorithm that selectively computes only 5% of query-key blocks in FP16 precision while processing the remaining 95% in FP4, rec…