AirLLM, an open-source inference library, claims to run a 70B-parameter large language model on a 4GB GPU without quantization, distillation, or pruning. The claim is technically true, achieved by loa…
AirLLM, an open-source inference library, now supports running the 2.8T-parameter Kimi K3 model on a single 4GB GPU, using only 3.72GB of VRAM on an RTX 6000 Ada, by streaming one expert at a time for…
DeepSeek released V4 Flash, a 284B-parameter MoE model with 13B active parameters per token, featuring FP4+FP8 hybrid attention and 1M native context. The open-weight model, available under MIT licens…
The author quit their FAANG job in 2024 to become an independent researcher and built a $48K GPU server called "grumbl" with six RTX 6000 Ada GPUs. They chose these GPUs over A100s and H100s based on …