Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants

A controlled French surface-variant benchmark of 4,200 original-low-high triplets (12,600 input instances) across seven task types found that accuracy falls and Brier score and negative log-likelihood (NLL) rise as perturbations intensify, according to the arXiv paper 2610.33493v1. Evaluating six models from two LLM families, the authors report that larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently shrink with scale, and keyboard typos cause the largest losses in both accuracy and probability quality at high intensity. Calibration temperatures fitted on original inputs transferred effectively to surface variants, and combining calibration with input restoration improved Brier score and NLL over either method alone while preserving restored accuracy.

arXiv:2610.33493v1 Announce Type: new Abstract: French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain insufficiently understood. To address this gap, we construct a controlled French surface-variant benchmark covering seven task types, with 4,200 original-low-high triplets 12,600 input instances . Each high-intensity variant retains all low-intensity edits and adds further changes. We evaluate six models from two LLM families. The results show that accuracy decreases and Brier score and negative log-likelihood NLL increase with perturbation intensity. Larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently decrease with scale. Keyboard typos cause the largest losses in accuracy and the quality of predicted probabilities at the predefined high-intensity settings. Finally, calibration temperatures fitted on original inputs transfer effectively to surface variants. Combining calibration with input restoration further improves Brier score and NLL over either method alone while preserving restored accuracy. Our benchmark provides a unified framework for jointly evaluating task accuracy and prediction reliability under common French surface variations.