arXiv
A stable synthetic population changes with the model
Language models can produce a stable synthetic population without producing a reliable picture of real people. The authors tested nine open-weight models by asking them to assign preferences across travel destinations, restaurants, and products, repeating each request up to 20 times. If such simulations are validated against human data, a city planning office could show the exact model checkpoint beside its ridership forecast.
A language model can be asked to stand in for survey respondents, shoppers, or travelers. Ask it the same question again and again, and it may return a neat list of what a population supposedly prefers. That apparent steadiness is useful—but it can also hide a basic problem: the list may describe the model, not the people.
A repeatable answer is not a measured opinion
A preference distribution is simply a set of choices with numbers attached. For example, a system might assign probabilities to 300 possible air-travel destinations, restaurants, or Home & Kitchen products. The numbers look like survey results, but the model has not measured anyone’s choices. It has produced stated probabilities from patterns in its training and later tuning.
This creates a difference between reproducibility and accuracy. Reproducibility means getting similar answers each time. Accuracy, or calibration, means that those answers match the world. A model can have the first property without the second. Synthetic populations can still be useful when surveys, behavior records, or experiments are unavailable, but they may carry patterns from training data instead of representing the group being simulated.
Changing the model is therefore not like installing an ordinary software update. Models can differ in their training data, fine-tuning, and other post-training methods. A larger model is not automatically a later version of the same simulated population.
The population moves when the model moves
The authors evaluated nine open-weight models from the Llama, Qwen, and Mistral families. They asked each model, under identical settings, to give explicit probability distributions for the three sets of choices, repeating the request up to 20 times.
Within a model, the answers often settled into a recognizable shape. Across repeated samples, the average overlap among the five most likely choices was 0.72. In 36 of 74 pairs of samples, those five choices were exactly the same. That is enough to make a synthetic survey look dependable.
But the dependable-looking population changed when the model changed. The researchers compared models at different sizes and used a distance measure called Jensen-Shannon divergence. In plain language, it asks how far apart two probability lists are. The lists from different sizes were substantially different, and the same pattern appeared in tests of Qwen and Mistral models.
The result was not a smooth progression in which a bigger model simply sharpened the same preferences. One model spread its probability across more choices; another concentrated it on a smaller group. Even when the prompt and the choices stayed fixed, the model’s size and family changed which destinations rose to the top. The paper calls the practical object a model-specific synthetic population, rather than a single, abstract “LLM prior.”
The method itself also matters. The experiment asked for probabilities in free-form JSON, then adjusted the results because the raw numbers often did not add up to one. Directly asking for numbers is a measurement method, and those numbers do not have to match the probabilities hidden inside the model’s word-generation process.
What a careful simulation would look like
In a city planning office, a report estimating who would use a new transit route could print the exact model checkpoint beside its ridership chart. The chart would sit next to results from a human survey, not be presented as neutral evidence from imaginary residents.
At a retailer’s planning meeting, analysts could run the same demand simulation across several models before ordering stock. A model swap would be treated as a change in assumptions, not as a routine software update. That would require benchmark data showing which models track the customers in question.
Those safeguards matter because this paper did not compare its generated distributions with real human preference data. Its conclusions are also limited to enumerable discrete-choice tasks, open-weight models, and English prompts. If the result holds against human data, simulations and decision-support systems will need to record, version, and test the exact model behind every generated preference distribution. Until then, repeated agreement can show that a model is reproducible—not that it describes real travelers, diners, or shoppers.