LLM Performance on a Real, Double-Marked GCSE Benchmark

Researchers introduced a dataset of 32,534 double-marked GCSE student responses and tested large language models (LLMs) against examiner consensus. Top-performing LLMs agreed more closely with examiners than examiners agreed with each other, achieving high scores in subjective tasks like English essay marking and handling handwritten Maths papers. The findings suggest cost-effective automated marking solutions are feasible.

arXiv:2606.24973v1 Announce Type: new Abstract: We introduce a dataset of 32,534 double-marked real student responses to GCSE mock exams GCSEs are the UK's national exams, taken at age ~16 , spanning 328 questions across five subjects and including handwritten work. We test whether off-the-shelf large language models agree with examiners as closely as the two examiners agree with each other. We find that models overwhelmingly agree well with the examiner consensus across subjects, with the top performing models agreeing more closely with examiners than examiners agree with each other. Models achieve high scores for subjective tasks like English essay marking, as well as handling complex and messy handwritten Maths paper scripts. Agreement is uniform near the examiner line, and not massively discriminated by model size, providing cost-effective automated marking solutions.