Benchmarking LLM Competence on Logical Inference over Probability Operators

Researchers introduced a benchmark for reasoning over probability operators, containing 14,320 procedurally-generated English prompts across fifteen inference templates, and found that most of the 29 evaluated large language models show answer biases independent of logical form, with only 9 exceeding random chance. The study, posted on arXiv (2607.27405v1), highlights systematic preferences for Yes or No answers and biases across question form, verb phrases, and name gender and origin.

arXiv:2607.27405v1 Announce Type: new Abstract: Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural-language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law. While large language models are increasingly evaluated on logical reasoning tasks, disentangling principled, symbolic reasoning from clever surface-level pattern matching is fraught with difficulty. We introduce a benchmark for reasoning over probability operators--inference over sentences with gradable epistemic modals e.g., probably, might, must containing 14,320 procedurally-generated English prompts across fifteen inference templates, systematically varying question form, negation strategy, and surface content. Evaluating 29 models, we find that most show answer biases independent of the logical form, a systematic preference for Yes or No. We summarize this with a competence floor: the worse of a model's accuracy on Yes-correct and No-correct items. Only 9 of 29 models exceed random chance. We also test variations in question form, verb phrases/activity, and both the gender and origin of names used in the prompts, finding biases across every axis.