{"slug": "strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number", "title": "Strawberry Has 2 R’s? How ChatGPT Actually Works Under the Hood (Every Number Measured)", "summary": "An experiment loading Qwen3-4B, released in 2025 under Apache 2.0, onto a local GPU found the open-weight model answered \"2\" when asked how many times the letter r appears in \"strawberry\" with its thinking mode off, then answered \"3\" with the switch flipped. The test traced the question \"Which city is the Eiffel Tower in?\" through every layer and reported that OpenAI's o200k_base tokenizer (vocabulary size 200,019) encodes \" strawberry\" as a single token, #101830, so the network never sees the individual characters, and that English words pack into single tokens while Indic scripts shatter into byte fragments requiring nearly 2× the compute. The authors argue an LLM is a stateless next-token predictor whose only memory is the reconstructed conversation history fed in each turn.", "body_md": "Ask a modern AI to write a five-paragraph essay on the French Revolution, and it finishes in three seconds. Ask that same AI a question a seven-year-old can answer on their fingers:\n\n***“How many times does the letter*** *** r appear in*** *** strawberry?\"***\n\nWith its “Thinking” switch turned off, the open model we tested (Qwen3-4B) answers instantly and confidently: **\"2.\"**\n\nWrong. Then we flipped one switch, asked the exact same model again, and it answered: **“3.”**\n\nWhy does a machine that can write Python compilers and pass medical licensing exams fail to count three letters in a ten-letter word? And what actually happens inside those billions of parameters between the moment your finger hits **Enter** and the moment the first word appears on your screen?\n\nChatGPT’s proprietary weights are locked behind OpenAI’s API. But modern open-weight models like **Qwen3–4B** (released in 2025 under Apache 2.0) are built on the exact same transformer recipe. So we loaded Qwen3-4B and its pre-assistant base model (Qwen3-4B-Base) onto our own GPU, hooked probes into every single layer, and followed one simple question all the way through:\n\n***“Which city is the Eiffel Tower in?”***\n\nEvery token ID, vector dimension, attention percentage, and probability below was measured directly from the live model.\n\n*🎥* ***Prefer watching the visual breakdown?*** *Watch our full 13-minute animated explainer here:*\n\n**The Misconception:** *“ChatGPT only reads my latest message, and it remembers who I am as we chat.”*\n\nWhen you press **Enter**, the neural network does not receive a neat little string containing only your question. In fact, the model has **zero memory** of you between messages.\n\nInstead, the application wrapper constructs a single flat text document from scratch every single turn. It stitches together:\n\nIn Qwen3-4B, the raw text fed into the model looks like this:\n\n```\n<|im_start|>userWhich city is the Eiffel Tower in?<|im_end|><|im_start|>assistant\n```\n\nThe model’s only job in the universe is to **predict the next single piece of text** that continues this document.\n\nWait — if a language model only predicts *one* piece of text, how does a full paragraph appear on your screen?\n\nIt runs in an **autoregressive loop**:\n\n*(Under the hood, production inference engines cache earlier layer calculations in GPU memory — the KV-Cache — so they don’t recompute past words from scratch, which we measure in* *Episode 2**).*\n\n***Takeaway #1:*** *An LLM is a stateless next-piece predictor. The conversation history* is *the memory.*\n\n**The Misconception:** *“AI reads words and letters the way humans do.”*\n\nBefore your text ever touches a neural network layer, a deterministic preprocessor called a **tokenizer** chops your string into common chunks called **tokens** and swaps each chunk for an integer ID.\n\nWhen we run \"Which city is the Eiffel Tower in?\" through OpenAI's GPT-4o-era tokenizer (o200k_base, vocabulary size 200,019), it splits into 8 tokens. Common short words stay whole, while longer words like \"unbelievable\" get cleaved into sub-word fragments: un | bel | ievable.\n\nNow look at what happens to \" strawberry\":\n\n``` python\nimport tiktoken\nenc = tiktoken.get_encoding(\"o200k_base\")print(enc.encode(\" strawberry\"))# Output: [101830]\n```\n\nIn o200k_base, \" strawberry\" (with a leading space) is **a single token:** **#101830** (and in Qwen3's tokenizer, a single token #72600).\n\nThe neural network **never receives the characters** **s-t-r-a-w-b-e-r-r-y**. It only sees the integer 101830. Asking an LLM to count the r's inside token #101830 without writing anything down is like asking you to count the letters in a word you have only ever heard spoken aloud as a barcode number.\n\nTokenization also explains why non-English languages cost more compute:\n\nBecause English dominated the tokenizer’s training corpus, English words get packed into single tokens while Indic scripts get shattered into smaller byte/character fragments — requiring nearly **2× the forward passes** for the same sentence.\n\n***Takeaway #2:*** *LLMs read numbered chunks (**tokens), never individual letters.*\n\nAn integer ID like 101830 has no mathematical meaning on its own—token #101830 isn't \"closer\" to token #101831 just because their IDs are adjacent.\n\nSo in Step 1 of the neural network, an **Embedding Lookup Table** swaps each token ID for a long list of floating-point numbers — a vector of length **2,560** in Qwen3-4B.\n\nThink of those 2,560 numbers as GPS coordinates in a high-dimensional **Map of Meaning**. Nobody hand-coded this map; the coordinates were learned automatically during training so that words used in similar contexts land in the same neighborhood.\n\nWhen we queried the real embedding table of Qwen3-4B for the nearest neighbors of specific tokens using cosine similarity, here is what lives right next door:\n\nEven directions in this space carry approximate geometric meaning. If you take the vector step from v(\"France\") → v(\"Paris\") and apply that exact same offset starting from v(\"Japan\"), the closest city vector you land on is **\"Tokyo\"**.\n\n*(Where the popular analogy breaks down: this space has 2,560 dimensions, not 2, and vector arithmetic is noisy — when we tested* *v(\"India\") + (v(\"Paris\") - v(\"France\")), the closest vector was still just* *\"Delhi\" because* *\"Delhi\" was already right next to* *\"India\").*\n\n***Takeaway #3:*** *Every token becomes a 2,560-number coordinate vector in a learned geometric map of meaning.*\n\nStatic dictionary coordinates have a fatal flaw: context.\n\nConsider this classic sentence from the 2017 Transformer paper:\n\n“The animal didn’t cross the street because it was too tired.”\n\nWhat does **“it”** refer to? You know immediately that *it* is the **animal**, not the street. How does the transformer figure that out?\n\nThrough **Self-Attention**. At each layer, every token broadcasts three learned vectors:\n\nWhen we inspected Qwen3-4B processing that sentence, one attention head looking backward from the token \"it\" placed **92% of its attention weight directly on** **\"animal\"** and **0.0% on** **\"street\"**.\n\nEach of the 36 layers in Qwen3-4B runs **32 attention heads in parallel**, and different heads specialize in different grammatical patterns without ever being explicitly programmed:\n\n*(One crucial rule: in autoregressive models like GPT and Qwen, attention is causal — words can only look backward at earlier tokens, never forward into the unwritten future).*\n\n***Takeaway #4:*** *Self-attention allows each token to dynamically route and blend information from relevant earlier tokens.*\n\nAfter the 32 attention heads mix context together, each token vector passes through a **Feed-Forward Network (MLP / SwiGLU)** — which researchers (Geva et al., 2021; Meng et al., 2022) have shown acts like a massive key-value memory bank storing factual associations.\n\nOne **Attention block + one Feed-Forward block = 1 Transformer Layer**. Qwen3-4B stacks **36 of these layers** on top of each other.\n\nUsing an interpretability technique called the **Logit Lens** (projecting intermediate layer hidden states directly onto the output vocabulary matrix before the model reaches the top floor), we peeked at Qwen3-4B-Base's top guess at every floor as it processed \"The Eiffel Tower is in the city of\":\n\nAt the top of Layer 36, the neural network does **not** output a word. It outputs a list of ~151,936 raw scores (**logits**) — one for every token in its vocabulary — which are converted via **Softmax** into a probability distribution summing to 100%.\n\nOn \"The Eiffel Tower is in the city of\", Qwen3-4B-Base outputs:\n\nTo actually pick the next token, the sampler rolls a weighted die biased by a parameter called **Temperature (****T)**:\n\nThat weighted dice roll is why clicking **Regenerate** gives you a different phrasing every time. The model’s weights never change while you chat; it is not “learning from you” in real time.\n\nWhere did those 4.02 billion numbers come from? Every modern chat LLM goes through three distinct training stages:\n\nThe model begins as random noise. It is fed trillions of tokens of internet text with one self-supervised objective: **predict the next token**. Every wrong guess triggers backpropagation, nudging billions of weights slightly toward the true token.\n\nTo put the scale in perspective: Meta’s open **Llama 3** model was pre-trained on **15.6 trillion tokens** across up to **16,000 H100 GPUs** (3.8 × 10²⁵ FLOPs). If you read 8 hours a day at 250 words per minute, reading 15.6 trillion tokens would take you **~250,000 years**.\n\nA raw pre-trained model (Qwen3-4B-Base) is a document completer, not a helpful assistant—ask it a question and it might reply with three more worksheet questions. In SFT, human annotators write thousands of ideal <|im_start|>user ... <|im_start|>assistant conversations, teaching the model the turn-taking assistant format.\n\nThe model generates multiple candidate answers to a prompt, human raters rank them from best to worst, a **Reward Model** learns to predict human preferences, and the LLM is optimized to maximize that reward score. (Whenever ChatGPT shows you two side-by-side responses and asks *“Which response do you prefer?”*, you are generating RLHF preference data!).\n\n**The Misconception:** *“When AI hallucinates, it looked up the wrong row in its database.”*\n\nRemember: there is **no database inside an LLM** — only compressed statistical weights trained to produce plausible-sounding continuations.\n\nTo see hallucination happen live, we asked Qwen3-4B our exact same running question, but swapped \"Eiffel\" for a completely fictional name we invented:\n\n***“Which city is the Kesharpur Tower in?”***\n\nWithout a millisecond of hesitation, Qwen3-4B replied:\n\n“The Kesharpur Tower is located in Kesharpur, India… in the Bhagalpur district of Bihar… a symbol of the city.”\n\nThere is no Kesharpur Tower. Why did the model invent a detailed history in Bihar instead of saying *“I don’t know”*?\n\nThree interlocking mechanisms cause this:\n\nFinally, let’s return to our cold-open mystery. Why did flipping the **Thinking** switch allow the exact same Qwen3-4B model to get \"strawberry\" right (3 R's) when it previously failed (2 R's)?\n\nBecause with Thinking ON, before giving its final answer, the model wrote out a scratchpad:\n\n```\n<think>Let's spell out the word \"strawberry\" letter by letter:1: S2: T3: R  (1)4: A5: W6: B7: E8: R  (2)9: R  (3)10: YThere are 3 R's.</think>\n```\n\nBy spelling \"s-t-r-a-w-b-e-r-r-y\" separated by spaces and newlines, **each individual letter became its own separate token** on the page! Once the letters were broken out of token #101830 (#72600 in Qwen3) into individual tokens, the model's attention heads could actually see and count them.\n\n**Models need tokens to think.**\n\nWait — what *is* that <think> switch under the hood? How did the model learn to write a scratchpad and suddenly say **\"Wait—\"** when it makes a mistake? And why does turning Thinking Mode ON sometimes make accuracy **crash from 100% down to 37.5%**?\n\nThat is the subject of **Episode 2**: 👉 **Read Part 2: Why AI Says “Wait” — How Thinking Models Actually Work**\n\n[Strawberry Has 2 R’s? How ChatGPT Actually Works Under the Hood (Every Number Measured)](https://pub.towardsai.net/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number-measured-0c6499992573) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number", "canonical_source": "https://pub.towardsai.net/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number-measured-0c6499992573?source=rss----98111c9905da---4", "published_at": "2026-10-22 05:01:27+00:00", "updated_at": "2026-10-22 05:10:05.862326+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "artificial-intelligence", "generative-ai"], "entities": ["Qwen3-4B", "Qwen3-4B-Base", "OpenAI", "ChatGPT", "o200k_base", "tiktoken", "KV-Cache"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number", "markdown": "https://wpnews.pro/news/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number.md", "text": "https://wpnews.pro/news/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number.txt", "jsonld": "https://wpnews.pro/news/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number.jsonld"}}