GSM-Symbolic

A March 2026 replication of the GSM-Symbolic study found that GPT-4o, Claude Opus 4.6, and Claude Haiku 4.5 no longer show catastrophic performance drops on confounded math problems when ambiguous sam…

Large language models (LLMs) show reduced accuracy on math problems when simple details like names or numbers are changed, and a new study finds that using code execution methods does not improve this…

A new study of small language models reveals that chain-of-thought prompting for arithmetic relies on a positional shortcut: the model copies whichever number appears last before the answer delimiter,…