{"slug": "transformer-basics", "title": "Transformer Basics", "summary": "A technical reference on transformer architecture details that most tokenizers prepend a BOS token, which acts as a fixed landmark or \"attention sink,\" while a space prefix yields the wrong token. The reference notes that every layer contains two sublayers — an MLP that moves information and attention that transforms it — and that all attention heads write to the same residual stream, with the MLP performing no cross-token communication. It also lists cross entropy loss and a notation table covering token embeddings, weight matrices, the residual stream, query/key/value/output matrices, residual stream dimension, and number of attention heads.", "body_md": "# Transformer Basics\n\nMost tokenizers prepend a **BOS** token.\n\n- ↳ fixed landmark or “attention sink”\n- space prefix gives you the wrong token\n\n## Two sublayers at every layer\n\n1. MLP → move info\n2. Attention → transform info\n\nAll attention heads write to same residual stream.\n\nMLP ⇒\n\n↳ NO cross token communication\n\n## Cross entropy loss\n\n## Notation Reference\n\n| symbol | meaning | \n|---|---|\n|  | token embedding / a -dim vector | \n|  | weight matrix | \n|  | residual stream | \n|  | query, key, value, output matrices | \n|  | residual stream dimension | \n|  | number of attention heads |", "url": "https://wpnews.pro/news/transformer-basics", "canonical_source": "https://garden.azl.au/ai/attention/transformer-basics", "published_at": "2026-09-14 22:44:33+00:00", "updated_at": "2026-09-14 23:02:12.759160+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "neural-networks", "natural-language-processing"], "entities": ["BOS token", "MLP", "attention heads", "residual stream", "cross entropy loss"], "alternates": {"html": "https://wpnews.pro/news/transformer-basics", "markdown": "https://wpnews.pro/news/transformer-basics.md", "text": "https://wpnews.pro/news/transformer-basics.txt", "jsonld": "https://wpnews.pro/news/transformer-basics.jsonld"}}