{"slug": "in-which-i-lose-my-mind-over-embeddings-hplm-chapter-2", "title": "In Which I Lose My Mind over Embeddings (HPLM Chapter 2)", "summary": "Andrey Kurenkov's blog post on The Hundred-Page Language Models Book details Chapter 2, which explains tokenization, embeddings, and Word2vec, highlighting the surprising property that embedding vectors support arithmetic such as 'king' - 'man' + 'woman' ≈ 'queen'.", "body_md": "## In Which I Lose My Mind Over Embeddings (HPLM Chapter 2)\n\nJuly 31, 2026\n\nChapter 2 of\n[The Hundred-Page Language Models Book](https://thelmbook.com/)\nis when things started to get really exciting for me.\n\n### Chapter 2 – Language Modeling Basics\n\nChapter 2 introduces the fundamental building blocks of machine learning with natural language input and outputs: tokenization, embeddings, and model evaluation frameworks. Tokenization is the process of converting input words into “tokens” that are recognizable by language models, and embeddings convert those tokens into dense numerical representations. Together, they tackle the problem of efficiently representing arbitrarily large vocabularies in machine-readable formats.\n\nI had heard the word “embeddings”, of course, and I understood, at a high level, that embedding models are something important for LLMs, and are used to capture semantic similarity between words. More than that, I could not have told you. HPLM explains the concepts beautifully:\n\nImagine that you have a language with a vocabulary of 10000 words. (Note that this is a very small\nlanguage. The [Oxford English\nDictionary](https://www.oed.com/) contains over 500,000 words, and modern LLMs model multiple languages\nsimultaneously.) The most straightforward way to represent a word in this vocabulary is with a\n** one-hot vector**, an array of 10000 bits, with each index in the array\ncorresponding to a word in the vocabulary.\n\nOne-hot vectors are easy to understand and perfectly machine-readable, but they have two major drawbacks: they are incredibly memory inefficient, wasting most of their space storing useless 0s, and they fail to encode any useful relationships between words. Indices are assigned to words arbitrarily, and words that are very similar (e.g. “happy” and “glad”) may end up far away from each other, whereas words with adjacent indices may have no meaningful semantic relationship.\n\n### Word2vec\n\nThe primary insight of embeddings is that it is possible to generate an alternative representation of tokens which is both more dense than one-hot encoding and represents semantic relationships between words. HPLM goes into detail on Word2vec, a family of algorithms used to generate word embeddings. Word2vec comes in several flavors, but they all share the same core conceptual idea. Imagine a snippet of text containing a target word (in this case “cat”) and several surrounding context words:\n\nIf you were to train a neural network such that, when given the target word as input, it produces\nthe context words with high likelihood, then if you were to pass in a similar word, one that is\nlikely to appear in similar contexts (e.g. “kitten”), you should expect the network to\nproduce similar output. The structure of the network looks like this: two layers, an input layer\nthat scales from the vocabulary size down to the embedding dimension size and an output layer that\ndoes the reverse. The output of the first layer is a vector of floats, also called the\n** embedding vector**, which can be used as a dense representation of the\ninput word when training or using a language model.\n\n### The part that blew my mind\n\nSo far so good. It makes intuitive sense that the vectors for “cat” and “kitten” should be similar i.e. close to each other in vector-space. But then, the author casually drops this bomb – these embedding vectors support simple mathematical operations. For example, if you take the embedding vector for “king”, subtract the vector for “man”, and add the vector for “woman”, you get a vector very similar to the one for “queen”.\n\nWhat?\n\nHow?\n\nThat sounds completely made up.\n\nI had a little chat with Claude, trying to understand why this might be true. Claude made the argument to me that, within the vectors for “king” and “queen”, there is some subcomponent representing the concept of “royalty”. Honestly, that seems like nonsense to me.\n\nIt gets a little clearer when I think of the operations as if they were happening in the space of\ntraining examples, or at least I can squint at it and pretend that it makes sense. Imagine all of\nthe context examples for the word “king”. Some of them will include context about\ngender, many will not. When you subtract the vector for “man”, you’re not\nremoving only the examples that include both “king” and “man”. You are\nremoving *all* of the context examples for the word “man”, as if the concept of\n“man” is being removed from your language entirely. That doesn’t land you on some\ngenderless concept of “monarch”; it moves you somewhere really weird.\n\nNow, add in all of the example texts for “woman”. Not only are we adding back in some examples for “queen”, we’re also moving our whole language back from the strange universe in which “man” doesn’t exist.\n\nOk, no, maybe this doesn’t actually help.\n\n### Visualizing embedding vectors\n\nObsessed with this idea, I decided to vibe-code an app that would let me play around with more of\nthis word vector math. You can try it out here\n([Word2Vec\nArithmetic](https://www.maayanroth.com/word2vec_math/),\n[GitHub\nrepo](https://github.com/maayanroth/word2vec_math)). Over a smallish vocabulary of about 1600 words, you can add and subtract them together,\nand get a list of the closest resulting word vectors from a larger vocabulary of 30K words, along\nwith a visualization of the operation in vector space. Distance is measured by cosine similarity\nbetween vectors.\n\nThe quality of an embedding model should grow as the model is trained on a larger data set, and I\nwanted to see if I could visualize this improvement. I trained two embedding models, one on\n[text8](https://www.kaggle.com/datasets/gupta24789/text8-word-embedding),\nwhich is the first 100MB of English Wikipedia, and one on\n[enwik9](https://mattmahoney.net/dc/textdata.html), the\nfirst 1GB of English Wikipedia. I also downloaded one pretrained embedding model,\n[Stanford\nNLP’s 6-billion token GloVe vectors](https://nlp.stanford.edu/projects/glove/).\n\nSo does it work? Well, sort of.\n\nThe “king - man + woman” equation returns queen for all three vector models. Interestingly, the result vector computed by the enwik9 model is just a little bit closer to its actual vector for “queen” than the result that comes out of the GloVe model.\n\nThe impact of model size starts to show up in “walking - walk + swim”. The two larger models return “swimming”, which is pretty cute. The small text8 model returns “crawling”. Initially, I was worried that this might be a vocabulary mismatch issue, that maybe “swimming” was simply missing from the top 30K most confidently-trained tokens in the text8 model. Once I restricted all three models to a common shared vocabulary, I was able to confirm that this is a genuine example of model quality improving as model training data size increases.\n\nAnother interesting result, where you really see the impact of model quality most clearly:\n\nBut a lot of the equations are simply nonsense, and Claude’s explanation that “king - man” should expose some kind of “royalty” vector didn’t show up at all.\n\nAnd with that, I’ve sufficiently scratched this itch, and I’m ready to move on to Chapter 3 – Recurrent Neural Networks.", "url": "https://wpnews.pro/news/in-which-i-lose-my-mind-over-embeddings-hplm-chapter-2", "canonical_source": "https://www.maayanroth.com/blog/posts/hundred-page-lm-book-chapter-2.html", "published_at": "2026-08-17 18:26:53+00:00", "updated_at": "2026-08-17 18:41:22.293584+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "machine-learning"], "entities": ["Andrey Kurenkov", "The Hundred-Page Language Models Book", "Word2vec", "Claude", "Oxford English Dictionary"], "alternates": {"html": "https://wpnews.pro/news/in-which-i-lose-my-mind-over-embeddings-hplm-chapter-2", "markdown": "https://wpnews.pro/news/in-which-i-lose-my-mind-over-embeddings-hplm-chapter-2.md", "text": "https://wpnews.pro/news/in-which-i-lose-my-mind-over-embeddings-hplm-chapter-2.txt", "jsonld": "https://wpnews.pro/news/in-which-i-lose-my-mind-over-embeddings-hplm-chapter-2.jsonld"}}