A developer found a scale-selection rule for NVFP4 gradient quantization that reduces mean squared error by 14% on real gradient tensors compared to the state-of-the-art MS-EDEN estimator, but trainin…
Researchers at the Max Planck Institute for Intelligent Systems, ELLIS Tübingen, and ETH Zürich built LittleLearner, a 5B-parameter language model pretrained exclusively on 88 billion tokens of U.S. e…
A new controlled study from the LittleLearner project, led by researchers including Ryan Cotterell of ETH Zürich and Wieland Brendel of the Max Planck Institute for Intelligent Systems, found that pos…
Researchers at the University of Zurich released LittleLearner, a family of language models (0.6B, 1.3B, and 5B parameters) trained from scratch on LittleCurriculum, an 88B-token corpus filtered to U.…
Hyperbolic Sparse Autoencoders (HyperSAE) outperform standard Euclidean Sparse Autoencoders (FlatSAEs) in reconstructing Google Gemma-2-2B activations, reducing reconstruction mean squared error (MSE)…
Researchers released OpenLanguageModel (OLM), an open-source PyTorch library for building and pretraining small language models with readable, composable code. The library includes 27 presets across n…
A 135M-class depth-recurrent transformer trained on FineWeb-Edu converges to a per-token fixed point, with mean successive-output KL divergence falling from 3.9e-1 at the second loop to 8.5e-6 by the …
A developer trained a 113M-parameter language model for earthquake science from scratch using open-access papers, Wikipedia, and FineWeb-Edu on two NVIDIA A30 GPUs. The model achieved a 35% reduction …
A new AI model called continuous-query limited memory language model (CO-LMLM) combines minimal cost with high factual precision by storing knowledge in an external knowledge base and fetching it as n…
Researchers at arXiv published a retrospective on SWAVE, a complex-valued recurrent language model, revealing that its Resonance Head suffered from a failure mode called cos-domination collapse, which…
Researchers introduced Gated DeltaNet-2, a linear attention model that decouples the erase and write operations in recurrent state updates using separate channel-wise gates. The model outperforms Mamb…
Researchers at HASH built a 25M-parameter transformer decoder, vec2slug, that generates URL slugs directly from vector embeddings, achieving 14-19x faster and 85x cheaper performance than a Haiku-clas…