AlGhamdi, Rayed · Nov 9, 2026
Reliability of Claude Code as an Agentic AI Grading System for Handwritten University Exams
Markdown · JSON · Original source
Rayed AlGhamdi examines whether Code, an deployment of Claude Opus 4.6, can grade handwritten university exams reliably. In one Saudi computer graphics course, the system graded papers from 60 consenting students using three rubric files. AlGhamdi reports strong agreement with the author's human grading on total scores (r = 0.97; MAE = 0.85 marks), alongside a small but significant AI-positive bias of 0.68 marks. The study also reports excellent test-retest consistency (ICC = 0.96) and accurate feedback under its coding scheme. But drawing items were an outlier, and recurring failure modes included keyword matching without conceptual depth, ambiguity over corrected multiple-choice answers, and attention drifting on hand-drawn diagrams. AlGhamdi argues that AI grading at this capability level can assist instructors on objective and short-form items under structural safeguards, but is not yet ready to replace instructor judgement at boundary cases or on drawing items. The human was the author's own grading, so the study assesses agreement with one domain expert rather than inter-rater reliability.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
cs.AI · Article
Source details
Original source date: Nov 9, 2026 (2026-11-09).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T04:03:44.041Z. First observed: 2026-11-11T05:06:26.810Z. Retrieved: 2026-11-12T04:02:45.367Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The study concerns 60 consenting students in a single Saudi computer graphics course; the evidence does not establish whether results generalize to other courses or exam settings. The author was the sole subject-matter expert and human , so the study does not assess inter-rater reliability. Reliability was estimated on disjoint samples: 20 papers for RQ1, RQ2 and RQ4, and 15 papers for RQ3.
Open question
Would Code show similar agreement and failure patterns in other courses or against multiple independent human graders?
Source-stated
The study examined Claude Code as a grader of handwritten university exams, using papers from 60 consenting students in one Saudi computer graphics course.
Claude Code graded the 20-mark mid-term papers of 60 consenting students in a Saudi computer graphics course
Source-stated
AlGhamdi reports strong agreement with human grading on total scores, alongside a small, significant AI-positive bias.
AI grading agreed strongly with human grading on totals (r = 0.97; MAE = 0.85 marks) with a small but significant AI-positive bias of 0.68 marks
Source-stated
The study reports excellent test-retest consistency and accurate AI-generated feedback.
showed excellent test-retest consistency (ICC = 0.96), and produced accurate feedback (92.8% diagnostic; 97.8% confirmation)
Source-stated
Drawing items were an outlier, and the study identifies recurring grading failure modes involving conceptual depth, corrected multiple-choice answers, and attention to diagrams.
Drawing items were the outlier on three of four measures. Three failure modes recurred: keyword matching without conceptual depth, selection-state ambiguity on corrected multiple-choice answers, and region-of-attention drift on hand-drawn diagrams.
Source-stated
AlGhamdi concludes that the system may assist instructors on some item types under safeguards, but should not replace instructor judgement at boundary cases or on drawing items.
AI grading at this capability level can assist instructors on objective and short-form items under structural safeguards but is not yet ready to replace instructor judgement at boundary cases or on drawing items.
Source-stated
Because the author was the sole subject-matter expert providing the human benchmark, the findings measure agreement with that expert rather than inter-rater reliability.
The author, as the sole subject-matter expert for the course, provided the human benchmark, so the design assesses agreement with a domain-expert reference rather than inter-rater reliability.
Source and contributor context
AlGhamdi, Rayed · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 613477bd-4a55-400e-88a4-d67b5367b84f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T04:03:44.041Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 41e72d12d7aa6b13da76a23e60b86f9ae81eaad86657b69de507dd55b275f15b. Source text hash: d805cb4a314b12d538e16e0ed258f4b7f004cfd3f71503ff9e45163095c04f42.
Revision and source-check history
Version 1 · 2026-11-12T04:03:44.041Z
Reliability of Claude Code as an Agentic AI Grading System for Handwritten University Exams
Rayed AlGhamdi examines whether Code, an deployment of Claude Opus 4.6, can grade handwritten university exams reliably. In one Saudi computer graphics course, the system graded papers from 60 consenting students using three rubric files. AlGhamdi reports strong agreement with the author's human grading on total scores (r = 0.97; MAE = 0.85 marks), alongside a small but significant AI-positive bias of 0.68 marks. The study also reports excellent test-retest consistency (ICC = 0.96) and accurate feedback under its coding scheme. But drawing items were an outlier, and recurring failure modes included keyword matching without conceptual depth, ambiguity over corrected multiple-choice answers, and attention drifting on hand-drawn diagrams. AlGhamdi argues that AI grading at this capability level can assist instructors on objective and short-form items under structural safeguards, but is not yet ready to replace instructor judgement at boundary cases or on drawing items. The human was the author's own grading, so the study assesses agreement with one domain expert rather than inter-rater reliability.
Original source
cs.AI · Article
Source details
Original source date: Nov 9, 2026 (2026-11-09).
Date basis: Date basis not provided.
Added to Escape Meridian: 2026-11-12T04:03:44.041Z. First observed: 2026-11-11T05:06:26.810Z. Retrieved: 2026-11-12T04:02:45.367Z.
Limits
The study concerns 60 consenting students in a single Saudi computer graphics course; the evidence does not establish whether results generalize to other courses or exam settings. The author was the sole subject-matter expert and human , so the study does not assess inter-rater reliability. Reliability was estimated on disjoint samples: 20 papers for RQ1, RQ2 and RQ4, and 15 papers for RQ3.
Open question
Would Code show similar agreement and failure patterns in other courses or against multiple independent human graders?
Source-stated
The study examined Claude Code as a grader of handwritten university exams, using papers from 60 consenting students in one Saudi computer graphics course.
Claude Code graded the 20-mark mid-term papers of 60 consenting students in a Saudi computer graphics course
Source-stated
AlGhamdi reports strong agreement with human grading on total scores, alongside a small, significant AI-positive bias.
AI grading agreed strongly with human grading on totals (r = 0.97; MAE = 0.85 marks) with a small but significant AI-positive bias of 0.68 marks
Source-stated
The study reports excellent test-retest consistency and accurate AI-generated feedback.
showed excellent test-retest consistency (ICC = 0.96), and produced accurate feedback (92.8% diagnostic; 97.8% confirmation)
Source-stated
Drawing items were an outlier, and the study identifies recurring grading failure modes involving conceptual depth, corrected multiple-choice answers, and attention to diagrams.
Drawing items were the outlier on three of four measures. Three failure modes recurred: keyword matching without conceptual depth, selection-state ambiguity on corrected multiple-choice answers, and region-of-attention drift on hand-drawn diagrams.
Source-stated
AlGhamdi concludes that the system may assist instructors on some item types under safeguards, but should not replace instructor judgement at boundary cases or on drawing items.
AI grading at this capability level can assist instructors on objective and short-form items under structural safeguards but is not yet ready to replace instructor judgement at boundary cases or on drawing items.
Source-stated
Because the author was the sole subject-matter expert providing the human benchmark, the findings measure agreement with that expert rather than inter-rater reliability.
The author, as the sole subject-matter expert for the course, provided the human benchmark, so the design assesses agreement with a domain-expert reference rather than inter-rater reliability.
Source and contributor context
AlGhamdi, Rayed · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
No suitable media was found by the supported page extraction.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Agentic AI
AI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Claude
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
arXiv
arXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Editorial provenance
Version 1. Publication ID: 613477bd-4a55-400e-88a4-d67b5367b84f.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: passed by anthropic/claude-opus-5.5 on 2026-11-12T04:03:44.041Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 41e72d12d7aa6b13da76a23e60b86f9ae81eaad86657b69de507dd55b275f15b. Source text hash: d805cb4a314b12d538e16e0ed258f4b7f004cfd3f71503ff9e45163095c04f42.
passed · 2026-11-12T04:03:44.041Z · anthropic/claude-opus-5.5
Publication: 613477bd-4a55-400e-88a4-d67b5367b84f
attribution — pass: The reading attributes the study to Rayed AlGhamdi, the sole listed author on arXiv. Findings are framed as reported by the study, and the conclusion is presented as AlGhamdi's argument rather than as established fact.
Authors:Rayed AlGhamdi
summary — pass: The summary leads with the actor and the development, then gives the results, failure modes, conclusion and benchmark limitation. A non-specialist can follow it because the plain words 'strong agreement' and 'excellent consistency' carry the meaning. Terms such as r, MAE, ICC and test-retest suit term-library tooltips. Two small fidelity repairs: keep the exact terms 'selection-state ambiguity' and 'region-of-attention drift' instead of paraphrasing them, and add that agreement was measured on samples of 20 and 15 papers, not all 60 (this appears only under limitations). The question is a fair unresolved issue.
Reliability was estimated on disjoint samples: 20 papers for RQ1, RQ2 and RQ4, and 15 papers for RQ3.
date — pass: originalPublished is 2026-11-09, which matches the arXiv v1 submission date. The later feed date of 2026-11-11 is correctly not used.
[Submitted on 9 Nov 2026]
support — pass: Every claim is backed by a verbatim abstract excerpt, and the limitations are source-stated. Minor note: the text of claim 5 says the system 'should not replace' instructor judgement, while the source says it is 'not yet ready to replace' it. The summary preserves 'not yet', so this is not a material error. Restoring 'not yet ready to' in the claim text would keep the source's temporal condition.
is not yet ready to replace instructor judgement at boundary cases or on drawing items
media — pass: The selected media is the verified arXiv abstract page, the direct original source, and it matches the source URL.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Claude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore ClaudeAI systems that plan and carry out a series of steps toward a goal, often by using other software, rather than giving one answer.
Definition referenceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referencearXiv is a research manuscript repository where authors share papers that may not yet have been peer reviewed.
Explore arXivA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceClaude is Anthropic’s family of AI models and assistants for tasks such as writing, analysis and coding.
Explore Claude