Transformer Basics

A technical reference on transformer architecture details that most tokenizers prepend a BOS token, which acts as a fixed landmark or "attention sink," while a space prefix yields the wrong token. The reference notes that every layer contains two sublayers — an MLP that moves information and attention that transforms it — and that all attention heads write to the same residual stream, with the MLP performing no cross-token communication. It also lists cross entropy loss and a notation table covering token embeddings, weight matrices, the residual stream, query/key/value/output matrices, residual stream dimension, and number of attention heads.

Transformer Basics Most tokenizers prepend a BOS token. - ↳ fixed landmark or “attention sink” - space prefix gives you the wrong token Two sublayers at every layer 1. MLP → move info 2. Attention → transform info All attention heads write to same residual stream. MLP ⇒ ↳ NO cross token communication Cross entropy loss Notation Reference | symbol | meaning | |---|---| | | token embedding / a -dim vector | | | weight matrix | | | residual stream | | | query, key, value, output matrices | | | residual stream dimension | | | number of attention heads |