arXiv
Interchangeable hallucination neurons send auditors to groups
The useful discovery is not a fixed set of hallucination neurons, but evidence that a detectable warning can be spread across interchangeable parts of a model. An arXiv preprint tested two Gemma-family models on three question-answering datasets and found that 19 of 22 selected neurons were strongly correlated with an unselected feature. In a hospital office, an AI assistant could flag an unreliable internal state instead of switching off one supposedly bad neuron, if the warning remains reliable across medical tasks and preserves useful reasoning.
A chatbot can produce a confident false answer, and researchers can often find a pattern in the model’s hidden activity that warns of the mistake. What they cannot safely say is that the warning lives in one small set of “hallucination neurons” that can simply be switched off. The same information may be carried by many internal features, so changing the neurons picked in one analysis could alter more than the unwanted behavior.
A signal is not a cause
A model’s hidden activity is made up of many numerical signals, called features here. A feature can be part of the model’s internal calculation without being a single-purpose control for one behavior. Transformer models distribute information across many such activations, and a change to one neuron can affect other behavior too.
The paper separates two questions that are easy to confuse. A detector can predict when a model is likely to hallucinate. That does not prove that the detector’s chosen features are the mechanism producing the hallucination, or that changing them will prevent it safely.
One common method uses L1-regularized regression. It is designed to choose a short list of features. But when many features contain similar information, the method may select one representative from a larger, correlated group. The selected neuron can therefore look important while being replaceable by another feature carrying much of the same signal.
Testing whether the neurons are special
The arXiv preprint, dated 24 September 2026, tested instruction-tuned Gemma 3 4B and MedGemma 4B models on three question-answering datasets: TriviaQA, BioASQ, and NQ-Open. Each model has 34 transformer layers and 10,240 feedforward features per layer, giving 348,160 features to choose from.
For each question, the identification procedure generated 10 answers. It kept only questions for which every answer was correct or none was correct. Sparse probes could then detect a pattern linked to hallucination. On Gemma 3 4B, their AUROC scores—a measure of how well a detector separates two outcomes—were 0.811 on TriviaQA, 0.974 on BioASQ, and 0.628 on NQ-Open. MedGemma showed the same basic pattern, with scores of 0.735, 0.955, and 0.612.
But the selected neurons did not behave like a unique control panel. Across three Gemma settings, 19 of 22 selected neurons had a correlation above 0.7 with a feature that had not been selected. Repeating the selection also produced only moderate overlap: bootstrap Jaccard similarity ranged from 0.45 to 0.69. When the researchers changed the ranking method, just 0% to 25% of the neurons selected by L1 appeared in an equally sized top-ranked set. No selected neuron was shared across all three Gemma dataset settings.
The causal test was weaker than the detector’s scores might suggest. Suppressing selected neurons changed judge accuracy from 0.496 to 0.472 on Gemma TriviaQA, and from 0.486 to 0.477 on MedGemma BioASQ. The paper reports these differences as statistically significant, but they are small. The causal tests used 500 held-out samples per condition and five random seeds. The authors did not test whether the interventions preserved general capabilities such as MMLU performance.
From neurons to groups
If this diagnostic approach holds up, an auditor should treat a detected pattern as evidence about a correlated group or subspace of model activity, not as the location of a behavior in a few special neurons. That gives safety teams a checklist: test whether features are correlated, whether the result survives resampling, whether ranking methods agree, whether an intervention has proper controls, and whether the same features appear across datasets or models.
In a hospital office, a clinician using an AI assistant might see a warning that an answer is drawing on a model state associated with unreliable responses, rather than being told that one supposedly bad neuron has been switched off. That scene depends on the warning remaining reliable across medical tasks and on any intervention preserving the model’s useful reasoning. Before neuron editing can become a credible route to safer deployed models, it must show both that it changes the target behavior and that it does not damage unrelated abilities.