An experiment using Claude Code and 10 parallel agents to evade AI detection on a corpus of 84 blog posts failed, with Pangram flagging nearly every post as 100% AI-generated. The author, who wrote th…
Vocabulary size in tokenization creates a three-way tradeoff between sequence length, embedding table size, and training signal sparsity, with a metric called fertility measuring how many tokens each …
WordPiece, Unigram, and SentencePiece are three subword tokenization algorithms that differ in how they split text, with WordPiece using a likelihood-maximizing merge criterion and a ## continuation m…
Byte-Pair Encoding (BPE), originally a data compression algorithm published by Philip Gage in 1994, was adapted in 2016 by Sennrich, Haddow, and Birch for subword tokenization in neural machine transl…
A language model does not see letters or pixels; it sees a sequence of integers from a vocabulary table, and the raw input is just bytes. Unicode assigns every character a unique integer code point, b…
The encoder-decoder (seq2seq) architecture, proposed by Cho et al. and Sutskever et al. in 2014, compresses the entire source sentence into a single fixed-length vector, causing performance to degrade…
Recurrent neural networks (RNNs) process sequences by maintaining a hidden state that compresses the entire history into a fixed-size vector, as formalized in Elman's 1990 paper "Finding Structure in …
GloVe, FastText, and word2vec are all word embedding methods that produce dense vector representations of words, but they differ in their underlying approaches: GloVe uses a weighted least-squares reg…
Google researchers led by Tomas Mikolov published word2vec in 2013, a method that made high-quality dense word embeddings cheap to train at billion-token scale on a single machine, transforming them f…
N-gram language models suffer from the curse of sparsity: as context length increases, the number of possible n-grams grows exponentially (Vⁿ), far exceeding any training corpus, leading to most unsee…
A language model assigns a probability to the next token given all previous tokens, a task that the chain rule shows is sufficient to capture any pattern in language. Claude Shannon demonstrated in 19…
A technical blog post explains that AdamW with warmup and cosine decay is the optimizer almost everyone uses today to train large language models, tracing the evolution from vanilla SGD through moment…
Gradient descent is the core algorithm behind deep learning, enabling neural networks with millions of parameters to minimize loss by iteratively adjusting weights in the direction opposite to the gra…
Cross-entropy loss is the standard loss function for training language models like GPT, LLaMA, and BERT because it is differentiable and captures probabilistic confidence, unlike accuracy which is a s…
Language models generate text by constructing a categorical probability distribution over their vocabulary at each decoding step, converting raw neural network outputs called logits into valid probabi…
A blog post explains that the L2 norm measures a vector's magnitude as its Euclidean distance from the origin, while the dot product quantifies similarity between vectors and is the core operation beh…
The first post in the 'Holding the LLM Stack in Your Head' series rebuilds intuition for linear algebra, explaining that vectors and matrices are the foundation of neural network operations. The autho…
A Google employee wrote a dependency-ordered series of over 80 posts walking through the full LLM stack from linear algebra to agent protocols, motivated by a fear of not understanding the layers bene…
A blog post by an anonymous author draws a parallel between the song 'Naked (But Still Stripping)' by D-A-D and the existential experience of large language models, exploring how LLMs generate text wi…