# Strawberry Has 2 R’s? How ChatGPT Actually Works Under the Hood (Every Number Measured) > Source: > Published: 2026-10-22 05:01:27+00:00 Ask a modern AI to write a five-paragraph essay on the French Revolution, and it finishes in three seconds. Ask that same AI a question a seven-year-old can answer on their fingers: ***“How many times does the letter*** *** r appear in*** *** strawberry?"*** With its “Thinking” switch turned off, the open model we tested (Qwen3-4B) answers instantly and confidently: **"2."** Wrong. Then we flipped one switch, asked the exact same model again, and it answered: **“3.”** Why does a machine that can write Python compilers and pass medical licensing exams fail to count three letters in a ten-letter word? And what actually happens inside those billions of parameters between the moment your finger hits **Enter** and the moment the first word appears on your screen? ChatGPT’s proprietary weights are locked behind OpenAI’s API. But modern open-weight models like **Qwen3–4B** (released in 2025 under Apache 2.0) are built on the exact same transformer recipe. So we loaded Qwen3-4B and its pre-assistant base model (Qwen3-4B-Base) onto our own GPU, hooked probes into every single layer, and followed one simple question all the way through: ***“Which city is the Eiffel Tower in?”*** Every token ID, vector dimension, attention percentage, and probability below was measured directly from the live model. *🎥* ***Prefer watching the visual breakdown?*** *Watch our full 13-minute animated explainer here:* **The Misconception:** *“ChatGPT only reads my latest message, and it remembers who I am as we chat.”* When you press **Enter**, the neural network does not receive a neat little string containing only your question. In fact, the model has **zero memory** of you between messages. Instead, the application wrapper constructs a single flat text document from scratch every single turn. It stitches together: In Qwen3-4B, the raw text fed into the model looks like this: ``` <|im_start|>userWhich city is the Eiffel Tower in?<|im_end|><|im_start|>assistant ``` The model’s only job in the universe is to **predict the next single piece of text** that continues this document. Wait — if a language model only predicts *one* piece of text, how does a full paragraph appear on your screen? It runs in an **autoregressive loop**: *(Under the hood, production inference engines cache earlier layer calculations in GPU memory — the KV-Cache — so they don’t recompute past words from scratch, which we measure in* *Episode 2**).* ***Takeaway #1:*** *An LLM is a stateless next-piece predictor. The conversation history* is *the memory.* **The Misconception:** *“AI reads words and letters the way humans do.”* Before your text ever touches a neural network layer, a deterministic preprocessor called a **tokenizer** chops your string into common chunks called **tokens** and swaps each chunk for an integer ID. When we run "Which city is the Eiffel Tower in?" through OpenAI's GPT-4o-era tokenizer (o200k_base, vocabulary size 200,019), it splits into 8 tokens. Common short words stay whole, while longer words like "unbelievable" get cleaved into sub-word fragments: un | bel | ievable. Now look at what happens to " strawberry": ``` python import tiktoken enc = tiktoken.get_encoding("o200k_base")print(enc.encode(" strawberry"))# Output: [101830] ``` In o200k_base, " strawberry" (with a leading space) is **a single token:** **#101830** (and in Qwen3's tokenizer, a single token #72600). The neural network **never receives the characters** **s-t-r-a-w-b-e-r-r-y**. It only sees the integer 101830. Asking an LLM to count the r's inside token #101830 without writing anything down is like asking you to count the letters in a word you have only ever heard spoken aloud as a barcode number. Tokenization also explains why non-English languages cost more compute: Because English dominated the tokenizer’s training corpus, English words get packed into single tokens while Indic scripts get shattered into smaller byte/character fragments — requiring nearly **2× the forward passes** for the same sentence. ***Takeaway #2:*** *LLMs read numbered chunks (**tokens), never individual letters.* An integer ID like 101830 has no mathematical meaning on its own—token #101830 isn't "closer" to token #101831 just because their IDs are adjacent. So in Step 1 of the neural network, an **Embedding Lookup Table** swaps each token ID for a long list of floating-point numbers — a vector of length **2,560** in Qwen3-4B. Think of those 2,560 numbers as GPS coordinates in a high-dimensional **Map of Meaning**. Nobody hand-coded this map; the coordinates were learned automatically during training so that words used in similar contexts land in the same neighborhood. When we queried the real embedding table of Qwen3-4B for the nearest neighbors of specific tokens using cosine similarity, here is what lives right next door: Even directions in this space carry approximate geometric meaning. If you take the vector step from v("France") → v("Paris") and apply that exact same offset starting from v("Japan"), the closest city vector you land on is **"Tokyo"**. *(Where the popular analogy breaks down: this space has 2,560 dimensions, not 2, and vector arithmetic is noisy — when we tested* *v("India") + (v("Paris") - v("France")), the closest vector was still just* *"Delhi" because* *"Delhi" was already right next to* *"India").* ***Takeaway #3:*** *Every token becomes a 2,560-number coordinate vector in a learned geometric map of meaning.* Static dictionary coordinates have a fatal flaw: context. Consider this classic sentence from the 2017 Transformer paper: “The animal didn’t cross the street because it was too tired.” What does **“it”** refer to? You know immediately that *it* is the **animal**, not the street. How does the transformer figure that out? Through **Self-Attention**. At each layer, every token broadcasts three learned vectors: When we inspected Qwen3-4B processing that sentence, one attention head looking backward from the token "it" placed **92% of its attention weight directly on** **"animal"** and **0.0% on** **"street"**. Each of the 36 layers in Qwen3-4B runs **32 attention heads in parallel**, and different heads specialize in different grammatical patterns without ever being explicitly programmed: *(One crucial rule: in autoregressive models like GPT and Qwen, attention is causal — words can only look backward at earlier tokens, never forward into the unwritten future).* ***Takeaway #4:*** *Self-attention allows each token to dynamically route and blend information from relevant earlier tokens.* After the 32 attention heads mix context together, each token vector passes through a **Feed-Forward Network (MLP / SwiGLU)** — which researchers (Geva et al., 2021; Meng et al., 2022) have shown acts like a massive key-value memory bank storing factual associations. One **Attention block + one Feed-Forward block = 1 Transformer Layer**. Qwen3-4B stacks **36 of these layers** on top of each other. Using an interpretability technique called the **Logit Lens** (projecting intermediate layer hidden states directly onto the output vocabulary matrix before the model reaches the top floor), we peeked at Qwen3-4B-Base's top guess at every floor as it processed "The Eiffel Tower is in the city of": At the top of Layer 36, the neural network does **not** output a word. It outputs a list of ~151,936 raw scores (**logits**) — one for every token in its vocabulary — which are converted via **Softmax** into a probability distribution summing to 100%. On "The Eiffel Tower is in the city of", Qwen3-4B-Base outputs: To actually pick the next token, the sampler rolls a weighted die biased by a parameter called **Temperature (****T)**: That weighted dice roll is why clicking **Regenerate** gives you a different phrasing every time. The model’s weights never change while you chat; it is not “learning from you” in real time. Where did those 4.02 billion numbers come from? Every modern chat LLM goes through three distinct training stages: The model begins as random noise. It is fed trillions of tokens of internet text with one self-supervised objective: **predict the next token**. Every wrong guess triggers backpropagation, nudging billions of weights slightly toward the true token. To put the scale in perspective: Meta’s open **Llama 3** model was pre-trained on **15.6 trillion tokens** across up to **16,000 H100 GPUs** (3.8 × 10²⁵ FLOPs). If you read 8 hours a day at 250 words per minute, reading 15.6 trillion tokens would take you **~250,000 years**. A raw pre-trained model (Qwen3-4B-Base) is a document completer, not a helpful assistant—ask it a question and it might reply with three more worksheet questions. In SFT, human annotators write thousands of ideal <|im_start|>user ... <|im_start|>assistant conversations, teaching the model the turn-taking assistant format. The model generates multiple candidate answers to a prompt, human raters rank them from best to worst, a **Reward Model** learns to predict human preferences, and the LLM is optimized to maximize that reward score. (Whenever ChatGPT shows you two side-by-side responses and asks *“Which response do you prefer?”*, you are generating RLHF preference data!). **The Misconception:** *“When AI hallucinates, it looked up the wrong row in its database.”* Remember: there is **no database inside an LLM** — only compressed statistical weights trained to produce plausible-sounding continuations. To see hallucination happen live, we asked Qwen3-4B our exact same running question, but swapped "Eiffel" for a completely fictional name we invented: ***“Which city is the Kesharpur Tower in?”*** Without a millisecond of hesitation, Qwen3-4B replied: “The Kesharpur Tower is located in Kesharpur, India… in the Bhagalpur district of Bihar… a symbol of the city.” There is no Kesharpur Tower. Why did the model invent a detailed history in Bihar instead of saying *“I don’t know”*? Three interlocking mechanisms cause this: Finally, let’s return to our cold-open mystery. Why did flipping the **Thinking** switch allow the exact same Qwen3-4B model to get "strawberry" right (3 R's) when it previously failed (2 R's)? Because with Thinking ON, before giving its final answer, the model wrote out a scratchpad: ``` Let's spell out the word "strawberry" letter by letter:1: S2: T3: R (1)4: A5: W6: B7: E8: R (2)9: R (3)10: YThere are 3 R's. ``` By spelling "s-t-r-a-w-b-e-r-r-y" separated by spaces and newlines, **each individual letter became its own separate token** on the page! Once the letters were broken out of token #101830 (#72600 in Qwen3) into individual tokens, the model's attention heads could actually see and count them. **Models need tokens to think.** Wait — what *is* that switch under the hood? How did the model learn to write a scratchpad and suddenly say **"Wait—"** when it makes a mistake? And why does turning Thinking Mode ON sometimes make accuracy **crash from 100% down to 37.5%**? That is the subject of **Episode 2**: 👉 **Read Part 2: Why AI Says “Wait” — How Thinking Models Actually Work** [Strawberry Has 2 R’s? How ChatGPT Actually Works Under the Hood (Every Number Measured)](https://pub.towardsai.net/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number-measured-0c6499992573) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.