arXiv
Early-warning system fairness controls reshuffle advising mistakes
A college can make a vendor AI’s fairness dashboard look better without making the distribution of student help fairer, because downstream controls mainly redistribute mistakes. Researchers tested six such controls on 168,550 Ontario college records without opening or retraining the model. If this pattern holds in real deployments, an advising office could become a place where students can question a risk flag and advisers can see its trade-offs, provided contracts expose the needed information and the college has capacity to respond.
Colleges already use early-warning systems to decide which students receive scarce advising, tutoring, financial-aid counselling, or other help. But the people making those decisions may see only a score or a flag. They cannot inspect the training data, the model’s settings, or the reasons behind its ranking.
That creates a problem close to everyday life. A student who needs help can be left off the list. Another can be labelled “at risk” because of patterns connected to age, gender, or residency, and carry the stigma of a warning that may not fit. The college has a dashboard, but not necessarily control over the machine behind it.
What fairness means after the model is bought
An early-warning system estimates whether a student will complete a programme. Advisers then use the estimate to decide whom to contact. A vendor may provide the output while keeping the model itself closed.
The college can still change the point at which a student is flagged. This is called a post-hoc intervention: an adjustment made after the prediction, without opening or retraining the model. It does not make the prediction more accurate. It changes who is put above or below the action line.
That line forces a choice between two kinds of mistake. A student who would have benefited from help can be missed. A student who would not have needed it can receive an unnecessary flag, taking time from a limited advising service. Measures of fairness can also disagree when groups have different completion rates. Making one measure look better may make another worse.
The dashboard can improve while help does not
The researchers tested six post-hoc methods on a research replica of the kind of tree-based model chosen by an Ontario college’s automated machine-learning system. They used 168,550 first-semester student records, covering 2008 to 2023. During testing, the replica was treated as a black box: none of the methods could inspect, alter, or retrain it.
The result was not one general fairness fix. The methods could move headline measures, but often by moving errors between groups. Calibration reduced some gaps, for example, while slightly widening the difference in false alarms between domestic and international students.
One method brought the share of students flagged to roughly 0.47–0.50 across groups. That may look balanced on a chart. It would also flag about half the students, which the paper says goes beyond plausible advising capacity. A fair-looking distribution is not much help if advisers cannot act on the list.
Another method nearly equalised the rate at which men and women who were truly at risk were identified. But it also sharply increased false alarms: the female rate rose from 0.561 to 0.837, while the male rate rose from 0.406 to 0.734. The errors had not disappeared. They had been rearranged.
The same issue appeared in the residency results. Two methods labelled the smaller international group as disadvantaged even though international students already had a higher positive rate than domestic students. One adjustment changed 706 residency predictions, all for international students. Their positive rate rose from 0.916 to 0.956, while domestic measures stayed unchanged.
The paper also warns that its “Unknown Gender” category combined non-binary identities, other responses, non-reporting, and missing data. The methods assessed age, gender, and residency separately, not their intersections. Small groups provide little information for calculations that depend on group-level data.
What a fairer system would require
At a college advising office, a student could ask why they were placed on—or omitted from—a risk list. An adviser could see which error trade-off produced the flag and appeal the classification, rather than treating a vendor score as final. That would make the advising slot a decision people can question, not a consequence of an unseen threshold.
For that scene to exist, contracts would need to provide meaningful audit information and institutions would need enough advising capacity to respond. Colleges would also need a say in how errors are traded off, not just a choice among downstream controls.
The result is not a verdict on every early-warning system. The researchers say their replica could not be checked against the operational vendor model, and the work used one institution. If the pattern holds in real deployments, changing outputs after purchase will not be a sufficient safeguard. The model’s design, the meaning of its target, and the human capacity to act on its flags would all matter.