{"slug": "chain-of-thought-why-think-step-by-step-actually-works", "title": "Chain of Thought — why 'think step by step' actually works", "summary": "A developer explains that chain-of-thought prompting works because it effectively increases the model's compute budget by turning a single forward pass into multiple passes, using written tokens as external memory. The technique helps for multi-step problems like math or planning but does nothing for single-fact recall or one-pass classification.", "body_md": "📺 Prefer to watch?\n\n[90-second YouTube Short]· 💬[Telegram]\n\n*Originally published on software-engineer-blog.com.*\n\nYou already know the trick: add \"think step by step\" to your prompt and the model's answer gets better. Almost nobody explains *why* — and the real reason has nothing to do with motivation or effort.\n\nHere's the floor. When a transformer generates a token, it runs through the same neural network layers every time. The stack depth is fixed at model-creation time. Whether you ask it \"2+2\" or \"Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left?\", the model gets the same amount of *layered* computation to produce each output token.\n\nThat compute budget never grows with problem difficulty.\n\nNow imagine you ask for just the answer: \"Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left? Answer only the number.\"\n\nThe model has to solve a three-step problem (multiply 3 × 8 = 24, multiply 4 × 5 = 20, subtract 24 − 20 = 4) in a single forward pass. It needs to hold \"24\" and \"20\" somewhere while computing the final step. But it's only got one forward pass, one set of layer outputs, and nowhere internal to stash intermediate values. So it guesses. It might say 19.\n\nIt didn't get the math wrong because it's bad at math. It got it wrong because you handed it the wrong compute budget for the job.\n\nNow ask the same question and let it write the steps: \"Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left? Think step by step.\"\n\nThree mechanical things happen:\n\n**1. The model becomes a loop.**\n\nEvery token the model emits is appended to the input context and fed back in on the next forward pass. So if it writes \"First, 3 × 8 = 24\", that token sequence gets read again when computing the next token. Write 40 reasoning tokens, run 40 forward passes instead of 1.\n\n**2. The written steps are external memory.**\n\nInstead of holding \"24\" in some internal embedding space and hoping it survives the next layer stack, the model writes \"24\" to the output — and then reads it back. The scratchpad is not hidden. It's right there in the text.\n\n**3. Each step conditions on prior work.**\n\nWhen the model generates \"then 24 − 20\", it's reading the output from step 1 (\"3 × 8 = 24\") and step 2 (\"5 × 4 = 20\") off the screen. No internal state decay. No lossy compression. Just tokens it can see.\n\nThis prediction is tight: chain of thought helps when the problem *has* steps, and does nothing when it doesn't.\n\n| Task Type | Why Steps Help | Why Steps Don't Help | |\n|---|---|---|---|\nMulti-step math |\nThree multiplies and a subtract need three separate forward passes to avoid state collapse. Steps buy those passes. | — | |\nLogic / planning |\nDecomposing into sub-goals (\"first find X, then use X to find Y\") is steps. Writing them out reads them back. | — | |\nSingle-fact recall |\n— | \"What's the capital of France?\" is one lookup, one forward pass. No steps exist. Writing fake steps adds latency and buys nothing. | |\nCategorization |\n— | If the model can classify an email as spam in one pass, additional \"reasoning\" tokens are just decoration. |\n\nThe pattern: if the ground truth solution has serial dependencies (step N depends on the output of step N−1), steps help. If it's a single lookup or one-pass classification, they don't.\n\n**The written reasoning is not an audit log.**\n\nThe model can land on an answer and then construct a plausible chain of reasoning to justify it. The text reads like \"here's how I thought,\" but it might be \"here's a story that fits the answer I already generated.\" You're getting a rationalization, not a replay of the computation. This matters when you're trying to trust the intermediate steps or debug where a wrong answer came from.\n\n**It costs latency and tokens.**\n\nEach reasoning token is a forward pass. Forty tokens = 40× the inference latency compared to asking for just the answer. For simple tasks, you've made the model slower to get the right answer. For hard tasks, you've made it *possible* to get the right answer, and latency is the price.\n\nThis entire story was the standard explanation during the prompt-engineering era (2022–2024). But modern reasoning models (OpenAI o-series, Anthropic Claude extended thinking, Google Gemini) do chain of thought *internally*, in a hidden token stream that doesn't appear in your output. The model spends thinking tokens you don't see, then emits the final answer.\n\nFrom the user's perspective: same model, same weights, same job. But now the reasoning is hidden and you pay for it upfront, not in your output tokens.\n\n\"Think step by step\" is still mechanically sound — it still works — but it's largely a prompt-era artifact. Modern models abstract away the need to write it explicitly. If you're using one of those models, you're already paying for internal reasoning; adding \"think step by step\" to the prompt often does nothing.\n\nIf you operate an LLM serving layer, chain of thought reframes the latency budget:\n\nThe first pass is the same. But you're now serializing 40 forward passes instead of 1. If you're batching requests, the model could process other users' prompts during those intermediate passes — but not for *this* user. You've increased the time-in-flight for that request.\n\nReasoning models hide this cost by processing the hidden thinking tokens within the system's \"thinking budget\" before returning the final answer, but the wall-clock time is still spent. The difference: the user doesn't see the intermediate work, and you can potentially amortize batches better because the output is shorter.\n\nReach for explicit chain of thought when you're solving multi-step math, logic, or planning problems on a base model and you can tolerate the latency cost. Reach for it *not* when you're doing single-fact recall or using a modern reasoning model — either the steps don't exist or the model already handles it internally.\n\nWatch the [90-second reel](https://youtube.com) for the same idea, compressed.", "url": "https://wpnews.pro/news/chain-of-thought-why-think-step-by-step-actually-works", "canonical_source": "https://dev.to/vahid_aghajani_60ce9dbec9/chain-of-thought-why-think-step-by-step-actually-works-5dlp", "published_at": "2026-07-27 06:46:38+00:00", "updated_at": "2026-07-27 07:01:45.910976+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "natural-language-processing"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/chain-of-thought-why-think-step-by-step-actually-works", "markdown": "https://wpnews.pro/news/chain-of-thought-why-think-step-by-step-actually-works.md", "text": "https://wpnews.pro/news/chain-of-thought-why-think-step-by-step-actually-works.txt", "jsonld": "https://wpnews.pro/news/chain-of-thought-why-think-step-by-step-actually-works.jsonld"}}