Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Model

A study of Mistral-7B found that demographic identity is readable and faithfully arranged in attention heads, but not causally used, with a single head (L11 H16) faithful across all six attribute types and selection-corrected fidelity up to rho=0.63, while causal use was clearest in one of the least faithful types and replacing the entire identity moved predictions by under 2% of their error. The findings, based on representational similarity analysis against Pew ground truth over 169 demographic cells and 1,089 read-out locations, suggest that readability, faithfulness, and causal use are dissociable properties, complicating the debate on whether LLMs can simulate populations.

arXiv:2608.18768v1 Announce Type: new Abstract: Large language models are widely used to simulate survey respondents, yet their answers are homogeneous and unfaithful to real inter-group differences. We ask where demographic group identity lives inside an LLM, how faithfully its geometry mirrors real inter-group opinion structure, and whether it uses what it encodes. Using representational similarity analysis against Pew ground truth over 169 demographic cells, we score 1,089 read-out locations in Mistral-7B and intervene causally across six attribute types. Four results. 1 The standard last-token residual read-out understates the model: attention-head read-outs dominate it in five of six types, with selection-corrected fidelity up to rho=0.63 -- roughly 70% of the measurement-reliability ceiling -- surviving a lexical-similarity control. 2 A single head L11 H16 is significantly faithful in all six types as a fixed location, while race-based types stay weak and prompt-fragile. Both phenomena replicate -- the analogous head significant in five of six types, weakest on the same race type -- across three checkpoints of a second model family, where ten billion training tokens barely move the map. 3 Causal use does not follow fidelity: the clearest causal pathway sits in one of the least faithful types p=0.002, cluster-robust, fixed depth , the most faithful type shows no correction-surviving single-layer effect, and replacing the entire identity moves predictions by under 2% of their error. 4 A 128-dimensional probe of the single head lands 21-31% closer to survey truth than the model's own answers -- yet recovers almost none of the per-question group ordering, no better than the answers themselves. Readable, faithfully arranged, and causally used are three dissociable properties of the same model; treating them as one claim is what keeps the "can LLMs simulate populations" debate unresolved.