Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

A new arXiv paper (2611.06715v1) introduces Rel-Bloom, a controlled multilingual diagnostic of 450 multiple-choice questions built from 30 Wikidata relations, and reports that accuracy drops sharply from Apply to Analyze in every evaluated open-weight model series, reaching 47.2 percentage points for the Large Granite bucket, while neither Gemini model showed that drop. Across 43 models, six languages and two multilingual settings (226,800 predictions), Gemini 3.1 Pro averaged 99.5% on the other four Bloom-inspired levels but only 60.7% on Evaluate items, which the authors say remain a bottleneck even for models near ceiling elsewhere. The authors conclude that greater model capability does not by itself ensure reliable reasoning over supplied relational evidence, and released resources at a Hugging Face Space hosted by QCRI.

arXiv:2611.06715v1 Announce Type: new Abstract: Retrieval-augmented generation RAG assumes that language models can reason over relations stated in their context rather than fall back on associations memorized during pretraining. We study this evidence-conditioned relational reasoning in isolation, with the facts supplied directly in the prompt rather than retrieved. \textsc{Rel-Bloom} is a controlled multilingual diagnostic of 450 multiple-choice questions built from 30 Wikidata relations; the same source relations are instantiated at five Bloom-inspired levels Remember, Understand, Apply, Analyze, Evaluate and under three veracity conditions Factual, Counterfactual, Neutralized . Across 43 models, six languages, and two multilingual settings 226,800 predictions , three findings stand out. First, accuracy drops sharply from Apply to Analyze in every evaluated open-weight model series, reaching 47.2 percentage points for the Large Granite bucket, but not for either Gemini model. Second, Evaluate items remain a bottleneck even for models near ceiling on the other task types: Gemini~3.1 Pro averages 99.5\% on the other four levels but 60.7\% on Evaluate. Third, item- and relation-level difficulty rankings are consistently preserved across languages, indicating a difficulty structure partly shared across languages within this benchmark. Across knowledge conditions, Counterfactual contexts are generally harder than Factual ones, while removing entity identities can improve higher-level performance in some model regimes. Greater model capability therefore does not by itself ensure reliable reasoning over supplied relational evidence. Relevant resources are available at: https://huggingface.co/spaces/QCRI/non-parametric-memory-relational-reasoning.