Aligning MLLMs with the Latent Structure of Human Cognition via Behavior-Derived Semantic Dimensions

A new arXiv paper (2610.41951v1) proposes aligning Multimodal Large Language Models with human similarity judgments using a behavior-derived semantic embedding space of 66 interpretable dimensions built from 4.70 million human Odd-One-Out judgments. The framework infers the most salient decision dimension for each triplet and converts it into a dimension-grounded linguistic rationale, improving model consistency with human similarity judgments while retaining competitive performance on standard multimodal benchmarks. Searchlight Representational Similarity Analysis on an independent fMRI Natural Scenes Dataset showed increased representational alignment between model and human, and the code is available on GitHub.

arXiv:2610.41951v1 Announce Type: new Abstract: Multimodal Large Language Models MLLMs have achieved impressive performance across a wide range of reasoning tasks, yet it remains unclear whether their internal representations and decision criteria align with the latent structure of human cognition. In this work, we propose a novel framework to align MLLMs with the multidimensional mental representations underlying human similarity judgements. We build on a previously established behavior-derived semantic embedding space, in which objects are represented along 66 interpretable dimensions derived from 4.70 million human Odd-One-Out O1O judgements. For each triplet, we infer the most salient decision dimension and convert it into a dimension-grounded linguistic rationale. This enables MLLMs to learn both the final choice and a justification consistent with the inferred behavioral criterion. Experimental results demonstrate that our approach improves model consistency with human similarity judgements while retaining competitive performance on standard multimodal benchmarks. Furthermore, using searchlight Representational Similarity Analysis RSA on an independent fMRI Natural Scenes Dataset NSD , we observe increased representational alignment between model and human. Ultimately, our findings point to a data-driven route for incorporating human cognitive structure into MLLMs alignment. The code is available \href{https://github.com/ening-open/cogalign}{here}.