arXiv
LLM grading scores without diagnosing
An LLM can imitate the shape of a detailed rubric while failing to tell a student what they have misunderstood. Researchers tested three models on 3,041 undergraduate computer-science answers and found that their scores tracked instructor grades, but their feedback identified misconceptions far less often. In a university programming lab, the system could handle clearly procedural work while sending conceptual or design-heavy answers to an instructor, if institutions validate those limits.
A student can receive an AI-generated rubric with separate scores, praise, and advice for revision. It can look like a careful explanation of what the student understands. But a weak answer may still leave the student unsure which idea is wrong, or how badly it has been misunderstood. The neat rows of a rubric can create the appearance of a diagnosis without doing the work of one.
What a useful rubric must reveal
An analytic rubric is meant to separate different skills. It might show that a student has remembered the facts but has weak reasoning, or that the ideas are sound but poorly communicated. That separation matters because each problem calls for different work.
A score alone is not the same thing. A machine can give a mark that is close to an instructor's mark and still offer poor help for learning. Useful formative feedback must identify the error or misconception, not just list something that is missing.
This difference is especially important in computer science. A procedural question may have clear expected steps, making it easier to judge. A design question asks someone to weigh trade-offs and explain their reasoning. Those answers are harder to assess from a fixed ideal answer.
When detail becomes decoration
The study analyzed 3,041 authentic answers to 50 open-ended computer-science questions from undergraduate courses at Universidad Francisco de Vitoria in Spain. The answers came from the 2025–2026 academic year and already had instructor grades on a 0–10 scale, along with written comments.
DeepSeek-chat-V3.2, Qwen-flash-2025-07-28, and Gemini-2.5-flash-lite-001 graded every answer using the same instructions. Each model saw the question, the student's answer, and the instructor-defined ideal answer. It did not see the human grade or other students' answers.
The models' scores matched instructor scores reasonably well overall. Their score correlations ranged from 0.785 to 0.815. Gemini had the smallest average error, at 1.22 points. But performance changed with the subject. Testing and test-driven development questions were graded most reliably. Architecture and integration, and Spring Boot and REST questions, produced the largest errors. On Spring Boot, Qwen's average error was 2.36 points, and it had a strong tendency to give scores that were too high.
The larger problem appeared inside the detailed rubrics. The separate LLM scores moved together very closely, with correlations from 0.82 to 0.99. In effect, several columns often repeated one broad judgment instead of measuring clearly different skills. The study's variance inflation factor reached 44.6, another sign that the dimensions were not acting independently.
The feedback also rarely found the thing a struggling student most needs. Teachers identified misconceptions in 15.0% of their comments. The LLMs did so in only 4.8% to 7.4% of comments. For answers graded 0–2 by humans, teachers found misconceptions in 31.5% of comments, compared with 5% to 7% for the models.
The machines were also more reassuring when they should have been more careful. They praised 47% to 57% of the lowest-scoring answers; teachers praised 22.6%. Human comments averaged 59 words, while LLM comments averaged 23 to 28 words. Human feedback became more negative for poor work and more positive for strong work. LLM feedback stayed positive across grade levels.
What a safer assessment looks like
In a university programming lab, a student whose design answer is fundamentally wrong would not receive three reassuring AI scores and a generic suggestion to add detail. The system would send the answer to an instructor, or clearly label its response as score-only.
That would not mean abandoning automated marking. If these patterns hold beyond this dataset, universities could use LLMs to screen or score work with well-defined procedural answers. They should not treat the systems as independent providers of formative diagnosis.
Before detailed AI feedback deserves the authority of an instructor's rubric, the systems would need to show that they can make independent judgments about rubric categories, detect misconceptions, and change their feedback to match the quality of the work. Institutions would also need to test the tools by subject and use them only for abilities they have actually validated.