标题： 超越共识：重新思考教育人工智能标注中的真实值 显示英文标题

标题： Beyond Agreement: Rethinking Ground Truth in Educational AI Annotation

摘要： 人类作为评估者往往存在严重的不完美之处。他们常常带有偏见、不可靠，不适合定义“真实标签”。然而，鉴于在教育应用中使用人工智能生成大量训练数据的迫切需求，传统的评分者间信度（IRR）指标如Cohen's kappa仍然是验证标注数据的核心方法。IRR仍然是许多教育数据机器学习流程的基石。例如，对话中导师行为的分类或机器评分评估中开放回答的标注。本文认为，过度依赖人类IRR作为标注质量的把关人，会阻碍以有效且预测性方式对数据进行分类的进展，从而促进学习改进。为解决这一问题，我们提出了五个互补的评估方法示例，如多标签标注方案、基于专家的方法和闭环有效性。我们认为，这些方法能够比仅依靠IRR方法产生更优质的训练数据和后续模型，从而提升学生学习效果和更具操作性的见解。我们还强调了外部有效性的重性，例如通过建立验证导师行为的程序并证明其在多种导师行为类别（如提供提示）中的适用性。我们呼吁该领域重新思考标注质量和真实标签——将有效性与教育影响置于共识之上。 显示英文摘要

摘要： Humans can be notoriously imperfect evaluators. They are often biased, unreliable, and unfit to define "ground truth." Yet, given the surging need to produce large amounts of training data in educational applications using AI, traditional inter-rater reliability (IRR) metrics like Cohen's kappa remain central to validating labeled data. IRR remains a cornerstone of many machine learning pipelines for educational data. Take, for example, the classification of tutors' moves in dialogues or labeling open responses in machine-graded assessments. This position paper argues that overreliance on human IRR as a gatekeeper for annotation quality hampers progress in classifying data in ways that are valid and predictive in relation to improving learning. To address this issue, we highlight five examples of complementary evaluation methods, such as multi-label annotation schemes, expert-based approaches, and close-the-loop validity. We argue that these approaches are in a better position to produce training data and subsequent models that produce improved student learning and more actionable insights than IRR approaches alone. We also emphasize the importance of external validity, for example, by establishing a procedure of validating tutor moves and demonstrating that it works across many categories of tutor actions (e.g., providing hints). We call on the field to rethink annotation quality and ground truth--prioritizing validity and educational impact over consensus alone.