arXiv

AI-text detector scores need a version number

October 13, 2026 2:30 PM

AI-text detectors cannot remain fair and reliable as language models change underneath them. Researchers tested 4,000 scientific abstracts against rewrites from 23 model versions and found that detection of GPT-5 rewrites fell to 26.6% when the detector had only seen earlier versions. If publishers keep using these screens, editors may need to treat a score as provisional evidence and check each model generation over time.

An AI-writing score can affect whether a scientific paper is delayed, challenged, or rejected. But the same score can also let undisclosed AI assistance pass if editors raise the threshold to avoid accusing human writers. A detector sold as a general screen is not necessarily checking the models that authors are using now: a model update can change its word choices and style without warning.

Why a detector can go wrong

AI-text detectors do not read a hidden label in a document. They look for statistical patterns in the language and use those patterns to guess whether a person or a language model produced the text. That makes the detector’s threshold important. Set it low, and it catches more AI-written passages but also falsely accuses more people. Set it high, and false accusations become rarer, but more AI-written text is missed.

In scientific publishing, the score is meant to be a first signal for an editor, not proof of who wrote a manuscript or whether its use of AI broke a rule. That distinction becomes harder to maintain when the model itself keeps changing. A detector trained on yesterday’s writing may not recognise tomorrow’s writing, even when both came from the same company.

The model changed, and the screen stopped seeing it

The researchers tested this problem with 4,000 scientific abstracts published before ChatGPT and rewrites made by 23 language-model versions from OpenAI, Meta, and Alibaba. The versions covered releases from June 2023 to August 2026. The rewrites were a simulation of AI assistance, not observations of real authors working on papers.

They set their detectors so that only 1% of human-written calibration abstracts would be falsely flagged. When a detector was trained on the exact model version it was screening, it detected almost all of the rewrites. But when it had been trained only on earlier versions from the same vendor, its success could collapse. For GPT-5 rewrites, detection fell to 26.6%; for Muse-Glimmer rewrites, it fell to 3.8%.

The reason was not simply that the new models were better at writing. Their vocabulary and patterns had shifted. The researchers found a strong link between the size of that shift and the detector’s failure to carry over from one version to another. Figure 2 maps both changes: where the language of two versions was farther apart, detection tended to transfer less well.

Trying to cover every version creates the opposite problem. A collection of version-specific detectors caught 98.6% of Muse-Glimmer rewrites, but falsely flagged 11.9% of human-written abstracts. A single detector trained on mixed versions avoided most of those false accusations, yet missed 33.8% of the Muse-Glimmer rewrites. The tested commercial detector labelled 79.8% of those rewrites as fully human-written.

These are not just competing technical scores. They describe a choice an editor cannot quietly avoid. A system tuned to make false accusations rare will miss some AI writing. A system tuned to catch nearly everything will accuse too many people. Model turnover means there is no fixed setting that solves both problems across the whole changing roster.

Refer to caption — Figure 2: Cross-version detection and vocabulary closeness for all pairs of the 23 LLM versions. (a) Each cell shows the TPR at 1% FPR of the detector trained on the rewrites of the row version and evaluated on the rewrites of the column version, averaged over the 10 random splits. (b) Each cell shows the closeness between the rewrites of the row version and the rewrites of the column version, measured by the JSD. In both panels, each cell is colored by the percentile rank of its value among all 23×2323\\times 23 cells of that panel. In panel (a), a rank of 100 means the highest TPR. In panel (b), a rank of 100 means the smallest JSD. Taken together, detectors trained only on earlier versions worked for most new versions, and their largest failures coincided with the largest vocabulary shifts between successive versions. This raises the question of whether an institution can keep up with the LLM turnover by collecting rewrites of every new LLM version and retraining its own detectors. Source: Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing（arXiv）

What an editorial office could do instead

At a journal office, a manuscript would no longer be rejected because a dashboard turns red. The editor would ask for an AI-use disclosure and examine the work, the data, and the author’s response. A detector result could still be a provisional clue, but not a stand-alone reason for an editorial sanction.

That approach depends on publishers treating detection as a continuing maintenance task rather than a one-time purchase. They would need to test scores against new and older model versions and keep checking how often human writing is falsely flagged. The authors studied English abstracts from one journal and used a two-stage rewrite, while real AI assistance can involve grammar correction, drafting, repeated editing, or research support. Before these scores can carry more weight, publishers would have to find out how well the findings apply beyond that narrow simulation.

Sources

arXiv, “Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing”