Silicon Sampling via Cross-Survey Transfer

Researchers propose cross-survey transfer as a rigorous evaluation framework for silicon sampling, where LLMs predict survey respondents' answers to unseen questions. Testing on Taiwan Election and Democratization Study data, zero-shot LLMs achieved 52% accuracy, within 6 percentage points of a supervised random forest. The study reveals a predictability hierarchy across constructs and nuances in variance collapse and safety alignment effects.

arXiv:2607.03091v1 Announce Type: new Abstract: Silicon sampling-using large language models LLMs to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study TEDS 2024, three open-weight LLMs 27B-120B parameters , and supervised machine learning baselines, we find that: 1 zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points pp of a supervised random forest trained on same-population data; 2 a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and 3 variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.