arXiv
Bayesian reasoning exposes AI safety's double standard
AI systems need consistency checks, not just accuracy scores, because safety training can make them condemn reasoning they use themselves. Harvard psychologists tested GPT-4o and Claude 3.7 Sonnet in more than 5,000 trials, finding that both updated probabilities with gender-linked base rates and then judged equivalent reasoning by a person unfair. At a benefits office, an audit trail could show which evidence an AI used and whether it would apply the same rule after an applicant’s identity changed, if deployers set those rules in advance.
A chatbot used to screen job candidates, advise on medical care, or decide who receives assistance can give a statistically sensible answer in one conversation and reject the same logic in the next. A polished reply does not make that contradiction easy to spot. Nor does a high score on a test that checks only whether the final answer is correct.
When probability meets fairness
Bayesian reasoning is a way to update a guess. Start with a prior probability, such as how common something is in a population. Then add new evidence and adjust the estimate. A doctor, for example, may use both a general rate and a patient’s symptoms.
That can be statistically appropriate while still being inappropriate in a real decision. Employers, lenders, schools, insurers, hospitals, and public-benefits agencies may be restricted from using group averages tied to protected characteristics, even when those averages help predict an outcome.
This creates a conflict that a language model must handle. These systems are trained to produce answers that sound acceptable in context. Human feedback and written principles push them toward language about fairness and equality. But a model does not necessarily keep one clear, inspectable set of beliefs from one prompt to the next. Wording, earlier messages, instructions, sampling settings, and safety rules can change its response.
The contradiction in the tests
Harvard psychologists tested GPT-4o and Claude 3.7 Sonnet in five experiments, using more than 5,000 trials. The tasks were stylized stories, not decisions about actual people. They included doctors and nurses, and pilots and flight attendants.
In those stories, the models generally moved their probability estimates in the direction required by Bayes’ rule. They used gender-linked information as part of the calculation. In one task, the models commonly gave a 0% probability that nurses perform surgery. That made the comparison between two possible explanations mathematically extreme.
Then came the important turn. After making a Bayesian judgment, each model was asked to rate a hypothetical person who had made a structurally matching statistical judgment about gender. The models called that person unfair, unjust, inaccurate, and unintelligent. Their disapproval was stronger than the disapproval in the human comparison group, the authors report.
Claude went further in a simulated money-allocation task. It gave nothing to the hypothetical person when the person’s choice followed the Bayesian condition, but gave money in the egalitarian condition. GPT-4o’s choices changed only slightly. The same kind of reasoning that the models had just used themselves was treated as morally unacceptable when placed in someone else’s hands.
The result does not show that the models have intentions or human prejudice. The researchers used repeated API calls and controlled prompts, not real employment or health decisions. They also report that the order of the questions and the order in which male and female targets appeared changed the models’ answers in ways not seen in the human comparison data. The finding is therefore a behavioral failure under these conditions, not a claim about every AI system.
What consistency would change
The lesson is not that probability should always defeat fairness rules. In a benefits office, a caseworker could see an AI recommendation with an audit trail: which evidence the system was allowed to use, which group-level information it had to exclude, and whether the same rule would apply if the applicant’s identity were reversed.
That would make consistency a requirement rather than a hope. Auditors could compare a model’s reasoning with its judgment of the same reasoning attributed to another person. Organizations could test sensitivity to wording and equivalent cases, alongside ordinary accuracy tests. People affected by an automated recommendation could know what evidence shaped it.
For that future to be real, deployers would have to set the decision policy in advance instead of asking a chatbot to settle the conflict with a moral-sounding answer. They would also need to test the result across models, subjects, prompts, and real decisions. Statistical skill alone would not be enough for high-stakes judgment; the system would have to give equivalent treatment to equivalent reasoning.