arXiv

LLM scores give one call seven measurements

October 1, 2026 8:30 AM

An LLM score is not automatically a property of the document it describes: changing the model can change the data before analysis begins. The authors had seven model families score 1,946 earnings-call transcripts with the same instructions, yet their rankings often diverged enough to alter later financial tests. If cross-provider checks become routine, a researcher could see whether a score survives the choice of instrument before relying on it.

A transcript can look like a fixed object: words on a page that anyone can count and study. But when a researcher asks an LLM to turn that transcript into a number for sentiment, risk, or management clarity, the number may depend on which company made the model. The document has not changed. The data has.

When text becomes a measurement

LLMs are increasingly used as measuring instruments for research, finance, compliance, and company analysis. Instead of searching for a fixed list of words, a researcher gives a model a written definition—such as “political risk”—and asks it to interpret a document and return a score.

That score is a measurement, much like a temperature reading. Reliability asks whether the reading stays similar when the instrument changes. Validity asks a different question: whether the reading captures the thing it claims to measure. A measurement can be consistent without being meaningful, or meaningful in principle without being consistent.

The distinction matters because a researcher can quietly choose one model, create a column of scores, and then treat that column as a property of the company. The model and its version become part of the result, even if the research paper does not make that choice prominent.

Seven models, one transcript

The authors tested this problem on 1,946 earnings-call transcripts from S&P 500 firms in 2024. Seven model families scored the same text using identical prompts, numerical scales, and temperature 0. They had no external tools, retrieval, or extra scaffolding. The models produced 13 measures covering topics such as sentiment, uncertainty, climate risk, political risk, specificity, and corporate culture.

If the models were measuring the same underlying qualities, companies should generally keep the same order. Instead, the average pairwise rank correlation across the measures was 0.52. Rank correlation is a way to ask whether two lists put the same items near the top and bottom. The result means that agreement was real but far from complete.

The differences were not equally large everywhere. Sentiment had much stronger agreement. The rule-based specificity ratio had much weaker agreement. This ratio tries to measure how specific a conversation is, but the models’ word counts were themselves part of the problem. When the researchers replaced those counts with the true word count, agreement rose from 0.23 to 0.47.

The disagreement could change more than a table of rankings. For several measures, changing only the LLM changed the size, direction, or statistical significance of regressions connecting the scores to announcement returns. A result that passes a standard test with one model might not pass it with another.

The paper’s Figure 1 compares the providers’ scores for constructs with high and low agreement, including sentiment, integrity, and the specificity ratio. It also compares the LLM scores with traditional, non-LLM benchmarks.

Refer to caption — Figure 1: Inter-provider score correlations: highest- and lowest-agreement constructs. Pairwise correlations between the seven LLM providers’ scores (upper triangle: Pearson; lower triangle: Spearman) for the two constructs with the highest mean pairwise correlation—sentiment and innovation—and the two with the lowest—integrity and the specificity ratio. The last row and column report the correlation of each LLM measure with the corresponding traditional, non-LLM benchmark for that construct. The colour scale is identical to the correlation heatmaps for all constructs reported in the Online Appendix (Appendix [A](https://arxiv.org/html/#A1 "Appendix A Correlation Heatmaps ‣ Same Text, Different Numbers: The Divergence of LLM-Based Measures")). Source: Same Text, Different Numbers: The Divergence of LLM-Based Measures（arXiv）

A more careful dataset

A practical response is to use several models and average their scores. In the study, a seven-provider average reached the researchers’ relative-reliability benchmark of 0.80 for 10 of the 13 measures. But it met their stricter absolute-dependability benchmark for only five. Averaging can reduce the quirks of individual models; it cannot decide what “political risk” or “clarity” should mean.

If checks across providers become affordable and routine, a university finance lab could open a dataset and see model-specific scores, an ensemble score, and a sensitivity report before running its main analysis. A company or regulator could ask which models agreed before using an AI-generated assessment as a threshold.

That future depends on researchers naming the model and version, testing results across providers, and comparing scores with human-coded or directly checkable benchmarks. Until then, an LLM-derived variable is not a fact pulled from a document. It is a measurement made by a particular instrument.

Sources

arXiv, “Same Text, Different Numbers: The Divergence of LLM-Based Measures”