arXiv
Survey simulation gets patterns right, not people
Open-weight language models are beginning to reproduce relationships across a survey population, even though they remain weak at predicting any one person. The researchers tested three model families against 2,058 human respondents across 15 survey instruments and found strong agreement in the shape of many cross-survey correlations. In a university survey lab, this could make locally run models a first filter for questionnaires, provided each model release is independently calibrated.
Today, a survey still needs people. Researchers must recruit and pay them, then wait for answers to learn how attitudes and traits move together across a population. A language model can fill out a questionnaire cheaply, but an unconditioned model often gives a narrow stream of socially acceptable answers. It can sound like a person without representing the differences between people.
What a useful synthetic population must preserve
The important question is not whether a model can produce a believable answer. It is whether many simulated answers keep the same relationships found among real people. For example, if two human survey measures tend to rise together, a useful simulation should preserve that pattern. It should also preserve when two measures move in opposite directions.
This is different from predicting one person's answer. A model may be poor at guessing what a particular person will say while still reproducing a broad relationship across thousands of people. The reverse is also possible: it may match average answers while losing the variation that makes survey data useful.
The researchers tested this across different questionnaires, rather than asking a model to repeat the survey that created its persona. They used a national panel of 2,058 people and 15 psychometric instruments, which are structured tests of traits or attitudes. For each test, a model received one person's exact answers to one instrument, called the seed, and then completed another instrument. The comparison was between the pattern made by these simulated answers and the pattern made by the human answers.
From plausible answers to population patterns
The three open-weight model families reproduced a meaningful part of those human patterns. Across 139 ordered pairs of instruments, the correlation between the simulated and human correlations reached 0.703 for gemma4:31b, 0.731 for llama3.1:70b, and 0.701 for qwen3.8:27b. In most pairs, the models also got the direction right: whether the relationship was positive or negative.
That is the useful change. The models were not shown a whole population and asked to copy its summary. They used one person's answers to one survey to generate answers to another, then did this across many people. The result suggests that an open-weight model can act as a partly calibrated population simulator, not just a machine for writing plausible individual responses.
But the word “partly” matters. Agreement between a simulated person's held-out score and that same person's real score was only 0.210 for the strongest of the three models. The models were much better at preserving the shape of relationships across people than at identifying any one person.
Calibration is therefore the scarce ability. An unconditioned llama3.1:70b persona gave answers with less variation than people. Giving it information from the Big Five personality test brought its response variation closer to the human level. Turning up the model's sampling temperature, a setting intended to make answers more varied, instead increased calibration error in every tested case.
The behavior was not fixed. A newer Llama release performed worst in a smaller comparison. In another test, reversing which questionnaire came first changed the simulated correlation. These results weaken the idea that a fluent synthetic respondent is automatically evidence of a valid synthetic population.
A cheaper first pass for survey research
If these findings survive comparisons with simpler statistical methods and larger independent panels, a university survey lab could run several open-weight models on its own computers using de-identified past responses. It could test which links between proposed questions look stable before asking people to spend time on every discarded pilot. Organizations could keep sensitive research prompts on their own infrastructure instead of sending them to a vendor's API.
That future depends on continuous checking. Researchers would need to calibrate the exact model release and use more than one seed measure. Human data would still be necessary when the size of an effect matters, not merely whether it points up or down. The authors also report no non-LLM baseline, and say that shared respondents and reused instruments make their results descriptive rather than suitable for significance tests that assume independent data. A small behavioral-task extension was preliminary as well.
So the likely replacement is not a room full of synthetic people. It is a cheaper first filter for population-level ideas—provided each new model is tested before anyone trusts its patterns.