{"slug": "a-calculator-compiled-into-a-transformer", "title": "A calculator, compiled into a transformer", "summary": "A new compiler called Torchwright, created by researcher Rob, compiles fixed computation graphs into transformer weights, and its first dedicated digit-level multiplier achieves 100% accuracy on all 3,000,000 valid three-digit expressions. In a direct test, five of six frontier models scored below 30% on 500 random five-digit multiplication problems without scratchpad reasoning, while the compiled transformer, with no training, exactly computes multiplication. The work demonstrates that transformers can express exact arithmetic algorithms, separate from what models learn.", "body_md": "LLMs’ weakness at math was once a running complaint. Tool calling, scale, and hidden reasoning scratchpads have made the problem much less conspicuous, but the weakness remains. To test that directly, I turned off reasoning and gave six frontier models the same 500 random five-digit multiplication problems. Each model had to answer directly, with no scratchpad work along the way. Most struggled: five of the six scored below 30%, and none reached 90% accuracy.\n\nWhat models learn and what transformers can express are different questions.\nGiven a fixed maximum operand length, there ought to be a transformer with the\nright weights that can multiply every pair of operands exactly. Most research on\nthis gap has approached it as a problem of learning.[1](#fn-1) I\nwondered about a different route: could I\nbuild a transformer directly from the same basic algorithms taught in grade\nschool?\n\nThat question led to more than one calculator. I built four in all, each implementing the same calculator in a different way, and compiled each one directly into transformer weights. Nothing is trained. All four predict exactly the same answers; what differs is how they get there. The grade-school implementation is just one of those routes.\n\n## Building the grade-school calculator\n\nI built all four calculators with Torchwright, a compiler I wrote that turns\nfixed computation graphs into transformer weights. For each calculator, I use\nPython to assemble one such graph from a restricted set of primitives that\nTorchwright knows how to realize in a transformer. [Introducing\nTorchwright](/posts/torchwright-intro/) explains how the compiler works. The\ncompiler machinery is shared across all four calculators, but this section\nfocuses on the grade-school version.\n\nAll four calculators use the same input format and digit representation. They\naccept expressions of the form `A op B\\n`\n\n, where `op`\n\nis `+`\n\n, `-`\n\n, or `*`\n\n. Each\ncalculator is built to handle operands up to a fixed number of digits (set at\ncompilation time). It treats shorter operands as left-padded with zeros, giving\nthe arithmetic a fixed number of digit positions to work with. At each digit\nposition, the calculator represents the value there with a one-hot vector: the\ncoordinate assigned to that value is set to 1 and the rest are 0. Concatenating\nthe vectors from two positions produces one of 100 distinct lookup-table keys,\none for each possible pair of values. The Python loops below are part of the\nconstruction process: they assemble tables and operations that the compiler\nturns into weights. The loops themselves never run at inference time.\n\nThe grade-school calculator is the first of the four implementations\n([source](https://github.com/physicsrob/torchwright/blob/main/examples/calculator_simple.py)).\nIts [three-digit\ncheckpoint](https://huggingface.co/physicsrob/torchwright-calculator-simple-max-digits-3)\naccepts operands from 0 through 999. I have exhaustively verified that it\ngenerates the correct answer for all 3,000,000 valid expressions that it\nsupports.\n\nThe intermediate values inside the model are not necessarily exact, but they do not need to be. Approximation error can shift the final logits and narrow the gap between the correct next token and the alternatives without changing which logit is largest. As long as the correct token remains on top at every generation step, greedy decoding produces the exact answer.\n\nAs far as I can tell, this is the first dedicated digit-level multiplier\ncompiled into runnable transformer weights. Compiled transformers themselves\naren’t new—[Tracr](https://arxiv.org/abs/2301.05062) compiles RASP programs into\nweights—and neither is compiled arithmetic: [addition and subtraction have been\ncompiled into a language model](https://arxiv.org/abs/2304.01665), and [addition\nand parity have been constructed with explicit layer\ncounts](https://arxiv.org/abs/2410.18077).\nMultiplication, though, has appeared only as [approximate gadgets inside an\nexistence proof](https://arxiv.org/abs/2305.15408), never as an instantiated\nmodel.\n\n### Arithmetic\n\nIn the grade-school construction, addition and subtraction work much the same\nway. Both move from right to left, one column at a time. For addition, a lookup\nmaps `(a, b, carry)`\n\nto an output digit and the carry for the next column;\nsubtraction follows the same pattern with a borrow. Multiplication eventually\nworks column by column too, but it needs some setup first: the graph must produce\nand arrange all the one-digit products.\n\nEach one-digit product comes from the familiar `0`\n\nthrough `9`\n\ntimes table. For\nevery pair of operand positions, the graph looks up the two digits and receives\ntheir product split into tens and ones: `7 * 8`\n\nbecomes `(5, 6)`\n\n. The following\nPython dictionary defines the 100-row mapping used to construct each lookup:\n\n```\nfor a in range(10):\n    for b in range(10):\n        key = torch.cat(\n            [embedding.get_embedding(str(a)), embedding.get_embedding(str(b))]\n        )\n        product_table[key] = torch.tensor(\n            [float(a * b // 10), float(a * b % 10)]\n        )\n```\n\nThe graph then puts the two parts of each product into their place-value columns:\n\n```\nfor i in range(n):\n    for j in range(n):\n        product = onehot_lookup(\n            concat([seq1[i], seq2[j]]), product_table, default_product\n        )\n        tens = slice_columns(product, 0, 1)\n        ones = slice_columns(product, 1, 1)\n        columns[i + j].append(tens)\n        columns[i + j + 1].append(ones)\n```\n\nThe operands, product columns, and final result all run from most significant\ndigit to least. With that ordering, the product of digits at positions `i`\n\nand\n`j`\n\ncontributes its tens digit to column `i + j`\n\nand its ones digit to the\ncolumn immediately to the right.\n\nTake `12 * 34`\n\n. The lookup for `1 * 3`\n\nreturns `03`\n\n: the 0 lands in the\nthousands column and the 3 in the hundreds column. In the tens column, 6 from\n`2 * 3`\n\nmeets 4 from `1 * 4`\n\n, making 10 before carrying. The units column gets\n8 from `2 * 4`\n\n.\n\nThis is the grade-school arrangement with the partial-product rows removed. On paper, we write and add those rows; the graph instead collects their contributions directly into columns, then propagates carries from right to left. Because several one-digit products can pile into one column, the column totals and resulting carries can be much larger than in addition.\n\nFor `12 * 34`\n\n, the three-digit checkpoint’s six result slots contain `000408`\n\nafter the carry sweep. The final formatting stage trims the leading zeros,\nproducing `408`\n\n.\n\nBehind the scenes, the calculator computes all three operations for every prompt. Addition uses the final column sweep without the preceding product stage, while subtraction follows an analogous sweep with borrows and uses a comparison to determine the sign. Only at the end does a switch select the result matching the operator in the prompt. The graph is fixed, so it has no control flow to skip the other two calculations.\n\n### How this construction scales\n\nThe compiler turns parallelism primarily into width and serial dependencies into depth. Independent graph operations can share a transformer layer when there is enough capacity, while a chain of dependent operations must extend across layers.\n\nEach call to `onehot_lookup`\n\ncreates its own copy of the times table: an\nindependent feed-forward network (FFN) node with 100 hidden units, one per\npossible digit pair. Each unit recognizes one concatenated one-hot key and\ncontributes that table entry’s tens-and-ones output, storing the lookup directly\nin the FFN’s weights. The compiled transformer cannot reuse a single table\nacross digit pairs, so with two -digit operands, the graph needs one lookup\nfor each of the pairs of digit positions, for a total of lookup\ncircuits and FFN hidden units.\n\nBecause those lookups are independent, the compiler can pack them into the\nsame transformer layer as long as its FFN is wide enough. A three-digit\ncalculator therefore contains nine lookup circuits, using 900 hidden units. That\nallocation does not shrink with the prompt: the calculator contains the same\nnine lookups for `999*999`\n\nand `2*3`\n\n.\n\nThe lookups can all happen at once, but the carries have to wait their turn. Each column needs the carry from the column to its right before it can finish. Supporting one more operand digit adds two product columns and extends this chain. As a result, the FFN capacity for digit products grows quadratically with operand width, while the serial depth for carrying grows linearly.\n\n### Producing the answer\n\nAt this point, the arithmetic graph can compute the complete answer as a fixed sequence of token vectors. Now it has to get that answer out. Because the model is causal, each position predicts only one next token; the whole sequence cannot emerge at once.\n\nThe newline anchors generation. At that position, attention gathers the prompt\ndigits into two fixed-width operand vectors and latches them there. It also\ninitializes `steps_since`\n\n, a counter that tracks how many answer tokens have\nbeen emitted. From the newline onward, each position retrieves those operands,\nrecomputes the complete answer, and uses the counter to expose the corresponding\nanswer slot.[2](#fn-2)\n\nFor `12*34\\n`\n\n, the formatted answer is `408`\n\n. At the newline position,\nthe counter is zero, so the gate exposes slot zero and selects `4`\n\n. Subsequent\npositions repeat the computation as the counter advances, exposing `0`\n\n, `8`\n\n,\nand then the end-of-sequence token; only the operands, not the answer, remain\nlatched.\n\nThe published Hugging Face checkpoint makes this concrete. With `d_model=2048`\n\nand `d_hidden=4096`\n\n, the three-digit grade-school graph compiles into a 27-layer\ntransformer. That is the depth the compiler realized for this particular\nconfiguration, not a lower bound on every possible architecture. The checkpoint\nitself uses Hugging Face’s unmodified Phi-3 implementation; its weights are\ncompiler-generated rather than trained. It loads through the usual `pipeline()`\n\ninterface:\n\n``` python\nfrom transformers import pipeline\n\ngenerate = pipeline(\"text-generation\", model=\"physicsrob/torchwright-calculator-simple-max-digits-3\")\nprint(generate(\"12*34\\n\", return_full_text=False)[0][\"generated_text\"])\n# 408\n```\n\n## Better algorithms\n\nHardware designers have met this carry problem before. In digital circuits,\ndepth means latency, so they learned to replace one-column-at-a-time carry\nchains with parallel trees. The [hardware-style\ncalculator](https://github.com/physicsrob/torchwright/blob/main/examples/calculator_advanced.py)\nborrows two of those techniques: carry-lookahead for addition and\nsubtraction,[3](#fn-3) and carry-save reduction for\nmultiplication.[4](#fn-4) Instead of waiting for carries to travel\nfrom column to column, these algorithms combine information in a tree, reducing\nthe longest dependency chain from linear to logarithmic growth. Each tree level\nis built from independent small lookups that can share a transformer layer, so\nthat logarithmic depth carries through compilation.\n\nThe external interface stays the same: its [three-digit\ncheckpoint](https://huggingface.co/physicsrob/torchwright-calculator-advanced-max-digits-3)\naccepts the same inputs as the grade-school calculator. Internally, the graph is\nwider and more intricate, and the tree introduces some overhead. At three\ndigits, the hardware-style calculator actually compiles two layers deeper — 29\nlayers rather than 27. It pulls ahead only as operands widen; at ten digits, it\nneeds 36 layers versus the grade-school calculator’s 43.\n\nSo far, the token stream has been a delivery mechanism, not a workspace. The\ngrade-school and hardware-style calculators complete the arithmetic within each\nforward pass; generation only exposes the finished answer one digit at a time.\nThe third implementation changes that: the [scratchpad\ncalculator](https://github.com/physicsrob/torchwright/blob/main/examples/calculator_scratchpad.py)\nuses [generated tokens as working memory](https://arxiv.org/abs/2112.00114).\nInstead of explaining its work in prose, it emits compact records of carries,\npartial digits, and formatting state for later positions to read back. Its\n[three-digit\ncheckpoint](https://huggingface.co/physicsrob/torchwright-calculator-scratchpad-max-digits-3)\naccepts the same input range, but emits these records before the final answer.\n\nThe scratchpad does not eliminate the serial chain; it relocates it. In the\ngrade-school calculator, the column-by-column work stretches across transformer\nlayers. In the scratchpad calculator, it stretches across generated positions,\nwith each position performing the next local step. Wider operands still require\nproportionally more sequential work, but that work adds tokens rather than\nlayers: [the serial depth has moved into the token\nstream](https://arxiv.org/abs/2402.12875). At the fixed model width used for the\ncomparison below, the layer count stays essentially flat at 18.[5](#fn-5)\n\nThere is one more way to avoid a long dependency chain: avoid computing the\nanswer at all. Every checkpoint supports a fixed maximum number of operand\ndigits, so its input set is finite. The fourth implementation, the [memorizing\ncalculator](https://github.com/physicsrob/torchwright/blob/main/examples/calculator_memorize.py),\nexploits that fact. It stores the complete formatted answer for every expression\nand retrieves it with a lookup. Its [two-digit\ncheckpoint](https://huggingface.co/physicsrob/torchwright-calculator-memorize-max-digits-2)\naccepts operands from 0 through 99.\n\nMemorization is shallow at two digits because it trades computation for storage.\nAdd one operand digit, however, and the lookup table grows a hundredfold (tenfold\nfor each operand digit). At a fixed model width, that growth causes both parameter and\nlayer counts to explode.[6](#fn-6)\n\nThe plots are measurements, not lower bounds.[7](#fn-7) They show\nwhat the compiler produced at the chosen hyperparameters. Within their supported\nranges, however, all four implement the same calculator function. I verified\nevery supported expression for all four published checkpoints.[8](#fn-8)\n\n## Versus the frontier\n\nWith all four calculators in hand, we can return to the frontier models from the opening. I tested the same six LLMs on direct-answer multiplication from three through seven digits, with reasoning disabled throughout.\n\nAt each length, I sampled 500 expressions whose operands both had exactly that\nmany digits. Every system received the same set. I asked each frontier model for\na plain integer at temperature zero and counted a response as correct only if\nits parsed value exactly matched the product. I retried non-conforming responses\nrather than scoring them; if a model never produced a direct answer, the\nproblem counted as wrong.[9](#fn-9) Provider accounting confirmed\nzero reasoning tokens on every attempt.\n\n| Model | 3×3 | 4×4 | 5×5 | 6×6 | 7×7 |\n|---|---|---|---|---|---|\n| GPT-5.6 Sol | 99.2% | 69.4% | 21.4% | 5.2% | 0.0% |\n| Claude Opus 5† | 95.4% | 78.0% | 13.0% | 1.0% | 0.0% |\n| Grok 4.3 | 78.4% | 25.2% | 2.4% | 0.2% | 0.0% |\n| DeepSeek V4 Pro | 99.6% | 55.2% | 26.4% | 2.2% | 0.0% |\n| Kimi K3 | 97.4% | 64.8% | 6.4% | 0.2% | 0.0% |\n| Qwen 3.7 Max | 100.0% | 99.0% | 87.4% | 55.2% | 7.4% |\n| Ours\n|\n\n† Claude’s scores partly reflect format noncompliance. At six digits, 42% of problems never received a direct answer after four attempts and were counted as wrong; format noncompliance was negligible for the other five models.\n\nAccuracy among the frontier models fell steeply as operands lengthened. Most\nwere near-perfect at three digits; at seven, every model except Qwen 3.7 Max\nscored exactly zero.[11](#fn-11) Qwen is the outlier throughout,\nholding on well past the others; I don’t know why. The compiled calculator, by\ncontrast, answered all 2,500 expressions correctly.\n\nThis is not a like-for-like test of general capability. The compiled calculator was built specifically for this grammar and a fixed maximum operand length. The point is narrower: exact multiplication can be implemented in transformer weights, even though trained general-purpose models do not reliably perform it when required to answer directly.\n\n## Where the computation lives\n\nWhen I started this, I mostly wanted to know whether I could take the multiplication algorithm taught in grade school and put it directly into transformer weights. I could. The three-digit checkpoint gets all 3,000,000 expressions in its domain right. Zero training.\n\nBut building four versions made a different point clear: a transformer does not necessarily come with one natural way to compute. The grade-school calculator puts its serial work in layers. The hardware-style version spends more width to shorten that chain. The scratchpad version moves the chain into generated tokens. The memorizing version avoids the chain altogether and pays in an exploding parameter count.\n\nAt a fixed input length, exact multiplication is not an impressive existence result. The domain is finite; a big enough lookup table will do. What I find satisfying is that ordinary algorithms survive the trip into transformer weights with their structure still visible. Parallel work becomes width. Dependencies become depth. Autoregressive steps become another place to put serial computation.\n\nThis answers one side of the question from the opening. Exact multiplication fits inside transformer weights. Why don’t trained models reliably perform it without intermediate work? I don’t know. In these models, I chose where the computation lived. How training might find an equivalent computation is a different problem.\n\nI started with calculators because they were simple enough that I could understand every step. What I did not expect was that the four versions would make the transformer’s resource tradeoffs this literal. The computation has to live somewhere.\n\n## Notes\n\nThe learning literature approaches LLM arithmetic from three angles:\n[measuring where frontier models break down](https://arxiv.org/abs/2305.18654),\n[training recipes that let small transformers learn arithmetic](https://arxiv.org/abs/2311.14737),\nand [asking why gradient descent so rarely finds the algorithm](https://arxiv.org/abs/2510.00184).\n\nExact equality is not available as a primitive, so the answer-slot gate tests\nit indirectly. For slot `i`\n\n, the gate checks whether the integer-valued\nposition counter lies between `i - 0.5`\n\nand `i + 0.5`\n\n. The only integer in\nthat interval is `i`\n\n.\n\nCarry-lookahead replaces the column-by-column sweep with a tree. In addition, each column gets one of three statuses: start a carry when its digits sum past nine, pass an incoming carry when they sum to exactly nine, or stop it otherwise. The tree combines neighboring columns into pairs, then groups of four, eight, and so on. This doubling resolves every carry in logarithmically many levels.\n\nThe same tree handles borrows in subtraction.\n\nCarry-save reduction first shrinks each column’s pile of partial-product digits without propagating carries. It combines up to eleven digits at the same place value into two: a sum digit that stays in place and a carry digit sent to the next-higher column.\n\nWhy eleven? Eleven decimal digits sum to at most 99, so the result still fits in two decimal digits. In binary, the same constraint permits only three input bits, producing the textbook 3:2 compressor. Base 10 permits an 11:2 compressor.\n\nRepeating this reduction for logarithmically many rounds leaves two digits per column. One final carry-lookahead addition produces the answer.\n\nThe scratchpad calculator uses 18 layers through seven-digit operands and 19 at eight. At the comparison width, the compiler cannot schedule larger versions — the scratchpad hits a width wall of its own.\n\nAn extrapolation at the comparison width puts the memorizing calculator’s three-digit table at roughly 200 layers and 700 million parameters, so the published checkpoint stops at two digits.\n\nOne configuration detail: the three-digit demo checkpoint published on\nHugging Face uses `d_model=2048`\n\nand `d_hidden=4096`\n\n, smaller than the shared\nconfiguration used for the comparison. Despite that difference, the\ngrade-school graph compiles to 27 layers in both configurations. All other\nlayer counts in the article use the shared comparison configuration.\n\nI exhaustively checked the full domains of all four published checkpoints: 3,000,000 expressions each for the grade-school, hardware-style, and scratchpad calculators, and 30,000 for the memorizing calculator. All four produced the expected output for every expression.\n\nIn the frontier-model evaluation, responses were capped at 24 tokens: enough room for any answer, but none for working. If a response did not conform, I discarded it and tried again, allowing up to four attempts per problem. Including retries, the evaluation made 17,290 attempts in all. Each model stayed pinned to the same provider.\n\nFormat compliance was a negligible issue for five models. Claude Opus 5, however, emitted working despite the instruction on over half its attempts. As a result, 42% of its six-digit problems never received a direct answer and were counted as wrong.\n\n“Ours” is the grade-school construction from above, compiled for seven-digit\noperands; the same checkpoint was used for every column. Frontier models\nreceived `What is {a} * {b}? Respond with only the answer as a plain integer. No commas, no explanation, no working -- just the digits.`\n\nThe\ncompiled calculator received `{a}*{b}\\n`\n\n, the fixed input grammar it was\nbuilt to accept. The underlying expressions and exact-match scoring were\nidentical.\n\nThe shape of the frontier models’ decline matches [the published record for\nGPT-4](https://arxiv.org/abs/2305.18654): 59% at three-digit multiplication,\n4% at four, zero at five.\n\n## References\n\n- Nye et al. (2021).\n[Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://arxiv.org/abs/2112.00114). - Dziri et al. (2023).\n[Faith and Fate: Limits of Transformers on Compositionality](https://arxiv.org/abs/2305.18654). NeurIPS 2023. - Shen et al. (2023).\n[Positional Description Matters for Transformers Arithmetic](https://arxiv.org/abs/2311.14737). - Liu & Low (2023).\n[Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://arxiv.org/abs/2305.14201). - Lindner et al. (2023).\n[Tracr: Compiled Transformers as a Laboratory for Interpretability](https://arxiv.org/abs/2301.05062). - Weng et al. (2023).\n[Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks](https://arxiv.org/abs/2304.01665). - Feng et al. (2023).\n[Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://arxiv.org/abs/2305.15408). NeurIPS 2023. - Li et al. (2024).\n[Chain of Thought Empowers Transformers to Solve Inherently Serial Problems](https://arxiv.org/abs/2402.12875). ICLR 2024. - Shaw et al. (2024).\n[ALTA: Compiler-Based Analysis of Transformers](https://arxiv.org/abs/2410.18077). TMLR. - Bai et al. (2025).\n[Why Can’t Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls](https://arxiv.org/abs/2510.00184).\n\n## Citation\n\nRobert Porter. \"A calculator, compiled into a transformer.\" Out of Distribution, August 2026. https://ood.dev/posts/calculator/\n\n```\n@misc{porter2026calculator,\n  author       = {Porter, Robert},\n  title        = {A calculator, compiled into a transformer},\n  year         = {2026},\n  month        = {aug},\n  howpublished = {\\url{https://ood.dev/posts/calculator/}},\n  note         = {Out of Distribution (blog)}\n}\n```\n\n", "url": "https://wpnews.pro/news/a-calculator-compiled-into-a-transformer", "canonical_source": "https://ood.dev/posts/calculator/", "published_at": "2026-08-09 00:00:00+00:00", "updated_at": "2026-08-10 01:50:40.172946+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-infrastructure"], "entities": ["Torchwright", "Rob", "Tracr", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/a-calculator-compiled-into-a-transformer", "markdown": "https://wpnews.pro/news/a-calculator-compiled-into-a-transformer.md", "text": "https://wpnews.pro/news/a-calculator-compiled-into-a-transformer.txt", "jsonld": "https://wpnews.pro/news/a-calculator-compiled-into-a-transformer.jsonld"}}