Figure 1: Overview of the human image similarity judgment-brain encoding model alignment framework. (a) Each of the eight NSD participants arranged an identical set of 100 images according to their perceived similarity, producing a subject-specific human similarity judgment matrix. (b) A generic model of human perception (here called Population DreamSim) was adapted to each participant’s observed similarity judgments to create perceptual similarity models that reflect that person’s judgments (here called personalized DreamSim). Population and personalized DreamSim models were used to generate similarity values between each pair of images in a given participant’s approximately 9,000 training set images. (c) Each participant’s brain encoder was fine-tuned using both population and personalized DreamSim estimates of image similarity to increase alignment of the brain encoder’s predictions of brain responses with human perception. (d) The resulting representations were evaluated by their alignment with human similarity judgments and their preservation of across-subject similarity structure.How a person perceives the similarity of two images, or human perceptual similarity, is a popular area of study in the broader field of psychophysics. Behavioral similarity experiments in these areas ask individuals to organize stimuli according to their perceived relationships. Representational similarity analysis (RSA) is a statistical approach that provides a framework for comparing similarity structures across various , multi-modal spaces ([Kriegeskorte et al., 2008](https://arxiv.org/html/#bib.bib25); [Sucholutsky et al., 2024](https://arxiv.org/html/#bib.bib24); [Nili et al., 2014](https://arxiv.org/html/#bib.bib44); [Kriegeskorte and Kievit, 2013](https://arxiv.org/html/#bib.bib43); [Simkova et al., 2026](https://arxiv.org/html/#bib.bib37)). RSA can be applied in this context to perceptual similarity, measured brain activity via fMRI, and encoding model predictions of brain activity to enable comparison of how image similarity spaces are conserved across the modalities. Analysis of these representational spaces can also allow quantification of either the similarity of representations across modalities within an individual or how the across-subject correlations of similarity spaces are preserved. A brain encoding model trained on individual participant data may be subject-specific in the sense that it predicts that participant’s neural activity in an identifiable way (compared to other subjects’ data), while still failing to reflect that person’s perceptual relationships between stimuli. 出典: Alignment of visual brain encoding models with individual perceptual similarity judgments（arXiv）Figure 3: (a) Within-subject alignment of an individuals’ image-pair similarity judgments for the 100 images in the perceptual task compared to the representational similarity of: population DreamSim, personalized DreamSim, baseline brain encoder, brain encoder + population DreamSim, and brainencoder + personalized DreamSim, for the same image-pairs. Bars above pairs of boxplots indicate those with significant differences (p<0.05p<0.05). (b) Human similarity judgment alignment (Spearman ρ\\rho) between each subject-specific model (rows) and each subject’s human similarity judgments (columns); red boxes mark the maximum value for each column. Boxes on the diagonal indicate that person’s model best matched to their observed similarity judgments compared to others’ similarity judgments. (c) Intersubject correlations of representational spaces for human similarity judgments vs intersubject correlations of representational spaces the same modalities listed in panel (b). Each of the 28 points represents a unique pair of individuals, where the x-axis is the correlation of those two individuals’ human similarity judgment RSMs and the y-axis is the correlation of those two individuals’ RSMs for that specific modality.We compared three brain encoders: the baseline brain encoder, the baseline brain encoder finetuned with Population DreamSim (Brain Encoder + Population DreamSim), and the baseline brain encoder finetuned with Personalized DreamSim (Brain Encoder + Personalized DreamSim). Within subject alignment of human judgment RSMs and brain encoder RSMs increased significantly across the three models: Baseline Brain Encoder (mean ρ=0.192\\rho=0.192, all p≤0.032p\\leq 0.032), Brain Encoder + Population DreamSim (mean ρ=0.287\\rho=0.287, all p<0.001p<0.001), and Brain Encoder + Personalized DreamSim (mean ρ=0.377\\rho=0.377, all p<0.001p<0.001), see Fig. [3](https://arxiv.org/html/#S4.F3 "Figure 3 ‣ 4.2 Personalized DreamSim models and Brain Encoder Finetuning ‣ 4 Results ‣ Alignment of visual brain encoding models with individual perceptual similarity judgments")a. The Baseline Brain Encoder RSMs had significantly lower correlation with human similarity judgment RSMs compared to the Brain Encoder + Population DreamSim (W=2W=2, p=0.023p=0.023) and the Brain Encoder + Personalized DreamSim (W=0W=0, p=0.008p=0.008), the latter two also being significantly different (W=0W=0, p=0.008p=0.008). The Brain encoder + personalized DreamSim also had higher correlations to human similarity judgments within individuals compared to just Personalized DreamSim (mean ρ=0.377\\rho=0.377 vs 0.3140.314 W=0W=0, p=0.008p=0.008). 出典: Alignment of visual brain encoding models with individual perceptual similarity judgments（arXiv）
