标题： 评估大型语言模型在高等教育中对学生作文自动评估的可靠性和有效性 显示英文标题

标题： Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education

摘要： 本研究调查了五种先进的大型语言模型（LLMs），Claude 3.5、DeepSeek v2、Gemini 2.5、GPT-4和Mistral 24B，在真实世界高等教育情境中自动作文评分的可靠性和有效性。 总共评估了67篇意大利语学生作文，这些作文是作为大学心理学课程的一部分撰写的，使用了四个标准的评分量表（相关性、连贯性、原创性和可行性）。 每个模型在三个提示复制中对所有作文进行评分，以评估模型内稳定性。 人机评分一致性始终较低且不显著（二次加权Kappa），模型内跨复制的可靠性同样较弱（中位数肯德尔W < 0.30）。 系统性评分差异显现出来，包括倾向于提高连贯性以及对依赖上下文的维度处理不一致。 模型间一致性分析显示，连贯性和原创性有中等程度的收敛，但相关性和可行性几乎没有一致性。 尽管研究范围有限，但这些发现表明，当前的LLMs可能在需要学科洞察力和上下文敏感性的任务中难以复制人类判断。 在评估开放性学术作品时，特别是解释性领域，人工监督仍然至关重要。 显示英文摘要

摘要： This study investigates the reliability and validity of five advanced Large Language Models (LLMs), Claude 3.5, DeepSeek v2, Gemini 2.5, GPT-4, and Mistral 24B, for automated essay scoring in a real world higher education context. A total of 67 Italian-language student essays, written as part of a university psychology course, were evaluated using a four-criterion rubric (Pertinence, Coherence, Originality, Feasibility). Each model scored all essays across three prompt replications to assess intra-model stability. Human-LLM agreement was consistently low and non-significant (Quadratic Weighted Kappa), and within-model reliability across replications was similarly weak (median Kendall's W < 0.30). Systematic scoring divergences emerged, including a tendency to inflate Coherence and inconsistent handling of context-dependent dimensions. Inter-model agreement analysis revealed moderate convergence for Coherence and Originality, but negligible concordance for Pertinence and Feasibility. Although limited in scope, these findings suggest that current LLMs may struggle to replicate human judgment in tasks requiring disciplinary insight and contextual sensitivity. Human oversight remains critical when evaluating open-ended academic work, particularly in interpretive domains.