Transformer Basics

A technical reference on transformer architecture details that most tokenizers prepend a BOS token, which acts as a fixed landmark or "attention sink," while a space prefix yields the wrong token. The reference notes that every layer contains two sublayers — an MLP that moves information and attention that transforms it — and that all attention heads write to the same residual stream, with the MLP performing no cross-token communication. It also lists cross entropy loss and a notation table covering token embeddings, weight matrices, the residual stream, query/key/value/output matrices, residual stream dimension, and number of attention heads.