arXiv
Cultural Divergence Preservation catches synthetic surveys flattening countries
A synthetic survey should be judged not only by whether it matches each country, but by whether it preserves the differences between countries. The researchers tested four open-weight models and found that a prompting method with strong ordinary accuracy also produced CDP values as low as 14%, meaning much of the human divergence had vanished. In a public-health office, that warning could trigger targeted human interviews before outreach money is allocated, if comparable human baselines and direction-sensitive checks are maintained.
A synthetic survey can give each country answers that look close to human results and still get the main story wrong. The countries may appear too similar, as if national differences have been smoothed away. Or they may look more stereotyped than real populations do. Researchers who trust only a country-by-country accuracy score can then mistake a model’s tidy average for public opinion and use it to make cross-country comparisons.
What a synthetic survey misses
Synthetic respondents are attractive because they can produce large survey-like samples without recruiting and paying human participants. A model can be asked to answer as an invented person, or simply as someone from a named country. This is called persona prompting.
One common check compares two response distributions using Jensen-Shannon divergence, or JSD. It measures how different two sets of answers are. A low JSD between a model’s answers and each country’s human answers sounds reassuring. But it does not ask whether the countries still differ from one another in the right amount.
Those differences can reflect culture, but also translation, when the survey was conducted, how people tend to use answer scales, and who was sampled. That makes the human data an imperfect reference, not a simple picture of national character. Figure 1 describes the motivation for checking the relationships among countries rather than only their separate scores.
A score for the missing distance
The researchers introduce Cultural Divergence Preservation, or CDP. It compares the distance between a model’s country-level response distributions with the distance between the matching human distributions for the same survey area.
The result is a ratio. A CDP of 100% means the model preserves the amount of cross-country difference found in the human data. Below 100% means the model has flattened that difference. Above 100% means it has amplified it. This lets researchers measure flattening or exaggeration separately from the error in each country’s distribution.
The audit tested four open-weight models, three ways of prompting personas, and two survey areas: items from the World Values Survey and the Big Five Personality Test. It used 300 simulated respondents for each country, persona method, and survey model. One prompt style, called DeepPersona-Inspired, often looked best by ordinary distance measures. But it also produced the strongest flattening in every block, with CDP values from 14% to 59%.
The opposite problem appeared with Country-only Cultural Prompting. It was over-divergent in all eight blocks, with CDP values from 184% to 501%. A country label alone could therefore make the simulated populations look more different than the human populations they were meant to represent.
What the check could change
In a public-health office comparing vaccine attitudes across countries, an analyst could see that a synthetic pilot had flattened the differences before using it to allocate outreach money. The office could then commission targeted human interviews instead of treating the model’s smooth average as a map of public opinion. This would give under-surveyed countries a way to challenge datasets that portray them as one undifferentiated global population.
That future depends on keeping comparable human survey baselines. CDP must be recalibrated when the countries, questions, or response scale change. It also does not check whether a country’s simulated deviation points in the same direction as its human counterpart: a model might preserve the overall amount of difference while assigning it to the wrong countries. The authors also say the flattening seen with DeepPersona-Inspired prompting may partly come from its fixed, country-unconditioned persona taxonomy, rather than from an inherent property of the models themselves.