标题： 在良好的 GRACEs 中：知识蒸馏的有原则的教师选择 显示英文标题

标题： In Good GRACEs: Principled Teacher Selection for Knowledge Distillation

摘要： 知识蒸馏是一种高效策略，利用大型“教师”语言模型生成的数据来训练更小但功能强大的“学生”模型，但选择适合特定学生-任务组合的最佳教师需要昂贵的试错过程。 我们提出了一种轻量级评分GRACE，用于量化教师在对学生模型进行微调时的有效性。 GRACE在不访问验证器、教师logits、教师内部结构或测试数据的情况下，测量学生的梯度分布特性。 从信息论的角度来看，GRACE与基于梯度算法的留一稳定性相关，这控制了蒸馏学生的泛化性能。 在GSM8K和MATH数据集上，GRACE与蒸馏后的LLaMA和OLMo学生的性能有很强的相关性（最高达86%的Spearman相关性）。 特别是，使用GRACE选择的教师训练学生，可以在性能上比盲目使用表现最好的教师提高多达7.4%。 此外，GRACE可以为蒸馏中的关键设计选择提供指导，包括（1）生成时应使用的最佳温度，（2）在尺寸限制下应使用的最佳教师，以及（3）在特定模型家族中应使用的最佳教师。 总的来说，我们的研究结果表明，GRACE可以高效且有效地识别出与给定学生高度兼容的教师，并提供如何进行蒸馏的细粒度指导。 显示英文摘要

摘要： Knowledge distillation is an efficient strategy to use data generated by large "teacher" language models to train smaller capable "student" models, but selecting the optimal teacher for a specific student-task combination requires expensive trial-and-error. We propose a lightweight score called GRACE to quantify how effective a teacher will be for post-training a student model. GRACE measures distributional properties of the student's gradients without access to a verifier, teacher logits, teacher internals, or test data. From an information-theoretic perspective, GRACE connects to leave-one-out stability of gradient-based algorithms, which controls the generalization performance of the distilled students. On GSM8K and MATH, GRACE correlates strongly (up to 86% Spearman correlation) with the performance of the distilled LLaMA and OLMo students. In particular, training a student using the GRACE-selected teacher can improve the performance by up to 7.4% over naively using the best-performing teacher. Further, GRACE can provide guidance on crucial design choices in distillation, including (1) the best temperature to use when generating from the teacher, (2) the best teacher to use given a size constraint, and (3) the best teacher to use within a specific model family. Altogether, our findings demonstrate that GRACE can efficiently and effectively identify a strongly compatible teacher for a given student and provide fine-grained guidance on how to perform distillation.