October 19, 2026
Technology Watch

From the laboratory to industry: following new technology on its way

← Front page

arXiv

Visual-fMRI decoders start before your scan

Image drawn by AI

A shared visual-fMRI decoder turns separate brain-imaging archives into training material for one another, though its common geometry is partly borrowed from CLIP. The authors trained BRAID-fMRI across eight datasets and reported 35.0% average Top-10 retrieval in eight matched participants. In a hospital imaging suite, a short standard visual task could let researchers search a large image library without first collecting hours of person-specific scans, if the transfer works with clinical data.

Today, a visual fMRI decoder usually belongs to one person and one experiment. fMRI scanners measure slow changes in blood oxygen, not the firing of nerve cells directly. Those measurements also change with the person, scanner, session and task. So a new participant often needs calibration scans, and a new study may need its own decoder. Expensive brain-imaging data remain trapped in separate collections rather than adding up.

Why the old approach does not travel well

A decoder is a system that tries to work backward from brain measurements to the image someone saw. A common method first predicts features from a vision model, then searches through a gallery of possible images for the closest match.

This work uses CLIP, a vision model that places images with related meanings near one another. A picture of a dog and another picture of a dog may land close together even if they are not identical. That makes the decoder useful for finding an image or a similar one, but it also creates a warning: a decoder trained against CLIP may inherit CLIP’s categories. It does not prove that those categories are the brain’s own filing system.

The main score is called Top-10 retrieval. The system ranks candidate images and gets credit if the correct one appears among the first ten. The score depends on how many candidates there are and what those candidates are. Anatomical registration can line up roughly similar regions across brains, but it does not remove the differences in how those regions function.

One shared starting point for many scans

The authors trained one model, called BRAID-fMRI, on eight visual-fMRI datasets. Together, the collection contained 430,007 single-trial pairs of brain measurements and visual stimuli. The model divided the brain into 379 anatomical regions, turned each region’s response into a token, and used a shared Transformer to combine them.

It then mapped the combined brain pattern into the fixed 512-dimensional image space of CLIP. In plain terms, the model learns to place a scan near the image that produced it, while also learning from scans collected in other datasets. It can therefore use the differences among studies as training material rather than treating each study as a sealed box.

The authors report retrieval across datasets and participants. In eight matched participants from two datasets, the average Top-10 score was 35.0%, compared with 27.1% for a retrieval-only baseline and 21.1% for ridge regression. In a stricter test, the target dataset, its participants and its stimuli were all kept out of training. Participant-specific embeddings were also disabled. Adding more source datasets improved the held-out target in all seven reported cases, with the largest gain reported for NSD at 92.1%.

That is the important change: transfer becomes a shared-pretraining problem instead of a demand to build or adapt a separate model for every person and experiment.

Refer to caption
Figure 1: \\sharpfontCross-dataset visual-fMRI harmonization and the BRAID-fMRI model.a, Visual stimuli are presented during fMRI acquisition, and single-trial responses are estimated to obtain stimulus-specific brain activity patterns. b, Eight visual-fMRI datasets spanning static-image and dynamic-video experiments are brought into a common processing framework. fMRI responses are represented using 360 HCP-MMP1.0 cortical parcels and 19 additional regions, while stimuli are represented in a common 512-dimensional CLIP ViT-B/32 image space. c, ROI-wise responses from participants in eight visual-fMRI datasets are mapped to ROI tokens and augmented with learned ROI embeddings and, in participant-conditioned analyses, learned participant embeddings. A Transformer shared across datasets and participants contextualizes the tokens, which are attention-pooled and projected into the 512-dimensional image space of a frozen CLIP ViT-B/32 encoder. Participant embeddings are omitted in the zero-shot transfer experiments. A multi-positive contrastive objective aligns neural responses with their corresponding stimuli while treating repeated observations of the same stimulus as positives. ### One model retrieves visual stimuli across datasets and participants Source: Many Brains, One Geometry: A Shared Visual-Semantic Space for Cross-Dataset fMRI Decoding（arXiv）
Refer to caption
Figure 3: \\sharpfontSemantic organization, network contributions and zero-shot cross-dataset transfer.a, t-SNE visualization of 40 kk-means clusters obtained from normalized CLIP embeddings of unique validation stimuli. Ten clusters used for subsequent analyses are highlighted, with representative stimuli shown. b, Dataset composition of the selected clusters. Stacked bars show the number of unique stimuli contributed by each dataset. Cluster labels (C0–C39) denote arbitrary kk-means cluster identifiers and are unrelated to retrieval rank; ranks indicate ordering by Top-10 retrieval accuracy. c, Top: unique-stimulus Top-10 accuracy for all clusters, ordered by performance; bottom: number of unique stimuli in the same ordering. Every fourth cluster in the ranked sequence was retained, yielding ten clusters spanning the full performance range. d, Global unique-stimulus Top-10 accuracy after masking each functional or anatomical ROI group. The dashed line marks the unablated baseline; annotations give percentage-point changes. e,Dataset-specific ablation effects. f, Participant-level effects for the four most disruptive groups. Points denote participants, violins show distributions, boxes show medians and interquartile ranges, and black circles show means. Asterisks denote one-sided Wilcoxon signed-rank tests of retrieval impairment, with Benjamini–Hochberg correction across nine groups. g, Zero-shot participant and dataset transfer with participant embeddings disabled throughout training and evaluation. For each target, the entire dataset and all of its participants remain held out while source datasets and participants are cumulatively added to pretraining. Values are percentage changes relative to step 0; annotations report the net change from the first to the final step.### A shared visual backbone supports retrieval across datasets Source: Many Brains, One Geometry: A Shared Visual-Semantic Space for Cross-Dataset fMRI Decoding（arXiv）

What a usable version would change

In a hospital imaging suite, a participant might complete a short, standard visual task. Researchers could then search a large image library for what the person was shown without first collecting many hours of person-specific training scans. Small laboratories could begin with a decoder trained on pooled data, and brain-computer-interface developers could avoid building a large bespoke dataset for every user.

That future depends on more than a high score in this collection. The result must survive independent replication and work with clinically practical amounts of data, scanners and populations unlike those in the research datasets. The model’s shared geometry is also partly supplied by CLIP, so it does not establish direct access to someone’s visual experience or reveal the brain’s native semantic geometry. Video datasets were reduced to their central frame, meaning the work does not separately establish decoding of actions or changes over time. The authors also used held-out evaluation data during training and checkpoint selection, monitoring a minibatch retrieval proxy. Until these limits are resolved, the advance is best understood as a reusable starting point—not a finished window into another person’s mind.

Sources

Many Brains, One Geometry: A Shared Visual-Semantic Space for Cross-Dataset fMRI Decoding