Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

Researchers propose grain calibration to validate whether large language models measure theoretical constructs correctly, not just reliably. The method decomposes constructs into clause-level components and tests each with extractive evidence, revealing whether the LLM uses the intended theory or a correlate.

arXiv:2606.28574v1 Announce Type: new Abstract: When a large language model LLM codes a construct in text as a human annotator would, that agreement makes the LLM a reliable coder. Yet reliability leaves construct validity untouched. The instrument may be theory-naive, reaching the code through a correlate that meets none of the demands the construct's theory makes, and no current method tells that apart from genuine measurement. We propose grain calibration as a method that closes the gap. It decomposes a construct into clause-level components, tests each against the text with extractive evidence, and combines the results through an explicit, theory-derived rule. Because the rule is stated rather than lodged in one opaque pass, its structure is evidence about the process rather than the output. It shows which components settled a code, and, when the code is wrong, whether a component was missed or an adjacent construct mistaken for it. Validation shifts from scoring an instrument's outputs against an annotator to showing that the instrument runs on the construct its theory specifies.