Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models A study of eight large language models and fine-tuned encoders on GLUE and the three rounds of ANLI found that every system answering directly retains 36-72% of its above-chance performance without nouns on MNLI, versus only 14-29% on ANLI, according to the arXiv paper 2611.16843v2. The authors report no link between pre-training exposure and robustness after checking contamination indicators, including exact n-gram counts in the public pre-training data of two models, and cite 368 new human-validated items unseen by any model on which all eight LLMs kept more performance without nouns on MNLI-style items than on items lacking its cues. The authors conclude that robustness to perturbation on familiar NLI benchmarks reflects dataset shortcuts rather than memorisation, and says more about the benchmark than the model. arXiv:2611.16843v2 Announce Type: replace Abstract: Pre-trained encoders fine-tuned on natural language inference NLI and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g. by removing all nouns or shuffling the words. Does the same hold for large language models LLMs , and if so, is it because they have seen the benchmarks during pre-training? In this paper we revisit these perturbation experiments with eight LLMs alongside fine-tuned encoders on GLUE and the three rounds of ANLI. Regardless of model size, every system that answers directly retains 36-72% of its above-chance performance without nouns on MNLI compared to only 14-29% on ANLI. Only a model that reasons before answering departs from this pattern. We also check contamination indicators including exact $n$-gram counts in the public pre-training data of two of the models and find no link between exposure and robustness. Our strongest evidence comes from 368 new, human-validated items that none of the models can have seen. All eight LLMs keep more of their performance without nouns on items written in MNLI's style than on items written without its cues. When the perturbation instead removes the function words carrying such cues, this advantage reverses. In light of these results, we argue that robustness to perturbation on familiar NLI benchmarks reflects dataset shortcuts rather than memorisation, and tells us more about the benchmark than about the model.