Table 1_Large language models as grading assistants in public health education: a method-comparison study of essay-style exam assessment.docx

Repeated grading across multiple sessions was performed for the best-aligned model.</p>Results<p>Agreement with human examiners was limited in fast mode and improved in reasoning or thinking modes; however, reproducibility across sessions and implementation conditions remained limited. In the single-file upload setting, ChatGPT showed the strongest agreement with human examiners (weighted kappa 0.718, 95% CI 0.578–0.859), followed by Kimi (weighted kappa 0.571, 95% CI 0.373–0.768).

ChatGPT achieved 50.0% exact agreement and 90.6% agreement within ±1 grade, with corresponding values of 28.1% and 78.1% for Kimi. Gemini and Gemini Pro showed the highest internal consistency across the file upload conditions. Repeated grading by ChatGPT across 5 days showed moderate variation.

LLMs used fewer extreme grades than did human examiners.</p>Conclusions<p>Some LLMs, particularly ChatGPT and Kimi, showed moderate alignment with human examiners, but the alignemnet was not consistently stable across repeated sessions or operational settings. LLMs may currently be better suited as supervised grading assistants than as autonomous graders.</p>