WordPiece, Unigram, and SentencePiece are three subword tokenization algorithms that differ in how they split text, with WordPiece using a likelihood-maximizing merge criterion and a ## continuation m…
A new study improving the Huth et al. fMRI language decoding pipeline achieved a mean METEOR score of 0.149 and BLEU-1 score of 0.200, an 11% relative METEOR gain over baseline, but the fMRIFlamingo m…
A new arXiv paper argues that the alignment techniques used to improve large language models like GPT-2 represent an expression of 'optimization culture' that conflates measurable improvement with val…
Researchers at arXiv improved the Huth et al. fMRI encoding baseline by expanding voxel selection to 15K and substituting GPT-2 medium for GPT-1, achieving mean METEOR = 0.149 and BLEU-1 = 0.200 for s…
A new study reveals that low-precision training can cause AI model parameters to freeze when gradient updates fall below half the unit in the last place (ULP) of a weight, effectively halting learning…
Researchers have introduced identity bridge supervision to overcome the curse of two-hop reasoning in large language models, enabling out-of-distribution generalization even in simple one-layer transf…
A study using GPT-2 models fine-tuned on impossible languages and then reconstructed to natural English found that recovered structures exhibit shorter dependency lengths than the original text, revea…
A technical walkthrough demonstrates the arithmetic behind attention in GPT-2 by computing one token's contextual embedding from raw scores, showing that attention weights and values are produced by s…
Researchers analyzing GPT-2 checkpoints found recurring spectral patterns in weight matrices, but reusing these spectra as initialization for new models failed to improve performance. The study sugges…
Researchers at arXiv analyzed eleven pretrained GPT-2-style checkpoints and found shared depth trends in weight spectra, but their initialization schemes imitating component-wise magnitudes and spectr…
Andrej Karpathy's open-source 'autoresearch' repository, released March 7, 2026, enables AI agents to autonomously run machine learning experiments, achieving an 11% training time reduction for GPT-2 …
A developer building a GPT-2 implementation in JAX discovered that token embeddings and the output head account for nearly half of the model's 163 million parameters, while attention layers use fewer …
Efficient Long-hOrizon (ELO) learning, a new learned optimizer, surpasses traditional models like AdamW and Muon across diverse tasks including language modeling and image classification, requiring le…
Researchers have developed a method to compress key-value caches in AI models using contractive iterated-map codes, achieving 36-54x memory reduction on GPT-2 with a perplexity cost of 11-15%. The tec…
Researchers introduced LEXIC, a lightweight eye-tracking extension that injects word-level difficulty signals into gaze-only models, achieving AUROC gains of up to +2.9 percentage points on reading co…
An interactive guide demonstrates how Transformer models use attention, embeddings, and positional encoding to predict moves in a fading Tic-Tac-Toe game, making the architecture's inner workings acce…
Researchers have published the BABEL codec, the first complete and certified bidirectional decode of GPT-2 Small's internal state, mapping every dimension of the model's hidden layers to plain English…
A technical walkthrough explains how GPT-2's decoder processes a 768-wide vector through 12 layers to predict the next word, detailing every matrix multiplication and parameter count without hand-wavi…
Experiments building on a mathematical theory of transformers reveal that the architecture drives tokens to cluster and collapse through layers, but trained weights learn to resist this clustering, en…
Giles Thomas completed building and training a GPT-2 small model from scratch using JAX, achieving a test loss of 3.418784, outperforming both his PyTorch model (3.538161) and the original GPT-2 small…