A from-scratch Rust language model called PSSA, which replaces transformer attention with a recurrent state-space core and a 512-slot episodic memory bank, reached 3.98 training cross-entropy versus 4…
Researchers proposed FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of large language models over wireless networks that transmits intermediate LoRA activations ins…
A developer ran a 56M-parameter language model across three ESP32-S3 N16R8 microcontrollers using ESP-NOW wireless communication, achieving distributed inference of a 50.3M-parameter PLE-based transfo…
A new research project called BCMT (Blockwise Causal Memory Transformer) proposes an alternative architecture for long-context language modeling that achieves O(TL) computational complexity compared t…
A new study reveals that the multiplicative repetition penalty used across major LLM inference engines (HuggingFace, vLLM, llama.cpp) is fundamentally flawed because it branches on the sign of raw log…
Researchers introduced Gaussian Mixture Attention (GMA), a probabilistic attention mechanism that replaces pairwise query-key comparisons with routing through K learned Gaussian mixture components, ac…
A new 306M-parameter language model architecture, the Cognitive Categorical Transformer (CCT), achieved 21.27 validation perplexity on WikiText-103, a 12% relative improvement over a fine-tuned GPT-2 …