A developer stretched Qwen 35B's context window from 4,096 to 32,768 tokens on a 12GB RTX 4070 by quantizing the KV cache to q8_0 using llama.cpp flags -ctk q8_0 -ctv q8_0, achieving an 8× context inc…
Owner-submitted benchmarks show that memory bandwidth, not TFLOPS, determines local LLM generation speed. An AMD RX 7900 XTX with 122 TFLOPS generates text at 39 tokens per second, while an older RTX …
Llama.cpp developer released ggrun, an auto-tuning tool that measures GPU, RAM, and PCIe topology to compute optimal multi-GPU and MoE expert placement for GGUF models, serving an OpenAI-compatible AP…
A developer released NanoEuler, a GPT-2-scale language model built entirely from scratch in C/CUDA without any machine learning libraries. The project includes a hand-written byte-level BPE tokenizer,…
A researcher reports that a byte transformer with a zero-parameter input layer (HSL-embedding-zero) performs comparably to learned embeddings across text, image, audio, radar, and lidar modalities, wi…