A developer ran a 56M-parameter language model across three ESP32-S3 N16R8 microcontrollers using ESP-NOW wireless communication, achieving distributed inference of a 50.3M-parameter PLE-based transfo…
A new research project called BCMT (Blockwise Causal Memory Transformer) proposes an alternative architecture for long-context language modeling that achieves O(TL) computational complexity compared t…
A new study reveals that the multiplicative repetition penalty used across major LLM inference engines (HuggingFace, vLLM, llama.cpp) is fundamentally flawed because it branches on the sign of raw log…
Researchers introduced Gaussian Mixture Attention (GMA), a probabilistic attention mechanism that replaces pairwise query-key comparisons with routing through K learned Gaussian mixture components, ac…
A new 306M-parameter language model architecture, the Cognitive Categorical Transformer (CCT), achieved 21.27 validation perplexity on WikiText-103, a 12% relative improvement over a fine-tuned GPT-2 …