Shared circuits predict whether LLMs generalize across formats in arithmetic reasoning

A new arXiv study (2609.04463v1) finds that the overlap between the internal circuits LLMs use for numeric arithmetic (2+5) and verbal arithmetic ('two plus five') predicts how well the models generalize across formats in English, Spanish, and Italian. Using attribution patching, researchers localized circuits for each format and showed that circuit overlap accounts for relative difficulty across verbal formats, which formats generalize best, and which items are solved correctly, rivaling supervised probes without labeled data.

arXiv:2609.04463v1 Announce Type: new Abstract: In many forms of reasoning, including arithmetic reasoning, generalizing across superficial changes in input format is effortless for humans: anyone who can solve 2+5 can also solve 'two plus five'. In contrast, LLMs are more brittle to surface variations of the prompts: for example, they solve numeric arithmetic problems almost perfectly but are substantially less accurate on verbal renditions of the same problems. Here, we ask whether generalization across formats can be predicted from the models' internals. Using attribution patching, we first independently localize the circuit that each model recruits to solve numeric arithmetic problems 2+5 vs. verbal ones, in three languages: English 'two plus five' , Spanish 'dos m\'as cinco' , and Italian 'due pi\ u cinque' ; then, we test whether overlap with the model's own numeric circuit predicts its generalization to the verbal formats. Indeed, we find support for this idea at three levels: circuit overlap accounts for the relative difficulty of the three verbal formats, for which models generalize best, and for which items are solved correctly, rivaling supervised probes while requiring no labeled data.