Synthetic data check catches flat school weeks fidelity scores miss

Synthetic educational data can pass ordinary accuracy checks while deleting the changes over time that make a useful research question visible. The paper tested four cohorts of Japanese lower-secondary learners with a graph-based check and found real week-to-week variation in learner groups was 2.6 to 4.9 times larger than in the synthetic cohorts. In a school research portal, that could warn an analyst before a flat synthetic timeline leads to weeks of work, if the check holds across other datasets and generators.

A school week can look ordinary in a synthetic dataset even when a real class is changing underneath it. A class may split into different study habits, gather around an exam, or settle back after a break. But researchers working with privacy-protected education records usually see only the synthetic copy first. They may spend weeks building a question on it, only to discover after controlled testing on the real records that the pattern was never there. Worse, a pattern that would have suggested a good question may have been erased before anyone could look for it.

What ordinary accuracy leaves out

Synthetic data is made to resemble real records without exposing what can be learned about a particular person. Differential privacy helps limit that risk, but privacy protection can reduce how useful the data is for analysis.

A common check compares each variable separately. It asks whether study time has similar averages, variances, or other summary statistics in the synthetic and real data. That can make the copy look convincing. Yet it may miss connections between variables, learners, and weeks. A dataset can preserve what each column looks like while losing the way several columns change together over time.

That matters in a system where researchers first explore synthetic records and then ask a data custodian to test selected conclusions on sensitive real records. If the synthetic term has been flattened, the missing feature is not only a failed conclusion. It is a research question nobody thought to submit.

A test for a flat timeline

The paper proposes looking at a weekly map of learner behaviour rather than checking one variable at a time. It examined four annual cohorts, from 2022 through 2025, containing 117 to 120 Japanese lower-secondary learners. Each cohort covered 18 school weeks, separated by a six-week summer break. The records described weekly study time in six time-of-day bands, estimated from digital-textbook operation logs.

For each week, the authors connected learners whose standardised weekly profiles were close enough to one another. In this proximity graph, each connected group is a rough cluster of learners with similar behaviour. The number of groups is called β0. It is a coarse measure, not a full description of the class, and it depends strongly on the distance threshold chosen.

The useful signal is how that number changes from week to week. The authors compared the real cohorts with synthetic ones, and also with 200 shuffled versions of each synthetic dataset. The shuffling kept each week's separate feature counts but disrupted the original ordering and joint changes. If the synthetic timeline was genuinely carrying structure, it should look different from its own shuffled versions.

It did not look different enough. At a common threshold, the week-to-week variation in β0 was 2.6 to 4.9 times larger in the real cohorts than in the synthetic cohorts. At four thresholds chosen in advance, none of the 16 synthetic variation values reached any of the 16 real values. Three of the four synthetic cohorts could not be distinguished from their shuffled versions; all four real cohorts could.

The synthetic data also put unusually high study activity in the documented mid-term examination week in every cohort. The real cohorts did not. The released private summaries had not required the synthetic data to preserve which time bands and weeks were zero together, or the identity and order of weekly changes. Those omissions help explain how separate summaries could look right while the timeline lost its shape.

What a warning could change

In a school district's research portal, an analyst could open a synthetic term of study logs and see a warning that its calm week-to-week pattern is indistinguishable from shuffled data. That warning would arrive before the analyst spent weeks building a study around a pattern that cannot support exploration. A data custodian could run the check using only the synthetic dataset and its 200 permutations, before using scarce access to the real records.

If the check works across more datasets and generators, it could screen out releases that flatten a cohort's temporal structure even when standard fidelity scores look good. Platforms that preserve joint trajectories and links among learners would have a clearer target than matching separate averages and variances.

This is not a general certificate that synthetic data is suitable for an educational claim. The evidence comes from one subject, one institution, one platform, and four cohorts, and the authors do not claim that the size of the gap will generalise. The check tests one structural property. Measuring the size of a real-versus-synthetic difference still requires access to the real data. Its value is earlier and narrower: it can reveal that a release has become too flat before the most valuable questions disappear with it.

Sources

arXiv, “What fidelity metrics miss: a structural check on synthetic educational data”