标题： 深度神经网络训练的不一致、不稳定性和泛化差距 显示英文标题

标题： Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training

摘要： 由于深度神经网络具有高度的表达能力，找到具有小泛化差距（训练数据和未见数据性能之间的差异）的解决方案非常重要。 专注于训练的随机性，我们首先进行了一项理论分析，其中泛化差距的界限取决于我们所谓的模型输出的不一致性和不稳定性，这些可以在未标记的数据上进行估计。 基于此分析的实证研究显示，在各种情况下，不稳定性与不一致性对泛化差距具有很强的预测性。 特别是，我们的发现表明，不一致性比损失景观的尖锐性更能可靠地指示泛化差距。 此外，我们展示了算法减少不一致性可以带来更好的性能。 这些结果也为现有的方法如协同蒸馏和集成提供了理论基础。 显示英文摘要

摘要： As deep neural networks are highly expressive, it is important to find solutions with small generalization gap (the difference between the performance on the training data and unseen data). Focusing on the stochastic nature of training, we first present a theoretical analysis in which the bound of generalization gap depends on what we call inconsistency and instability of model outputs, which can be estimated on unlabeled data. Our empirical study based on this analysis shows that instability and inconsistency are strongly predictive of generalization gap in various settings. In particular, our finding indicates that inconsistency is a more reliable indicator of generalization gap than the sharpness of the loss landscape. Furthermore, we show that algorithmic reduction of inconsistency leads to superior performance. The results also provide a theoretical basis for existing methods such as co-distillation and ensemble.