标题： 迈向理解为何Adam对于Transformer收敛更快 显示英文标题

标题： Toward Understanding Why Adam Converges Faster Than SGD for Transformers

摘要： 虽然随机梯度下降（SGD）仍然是深度学习中最受欢迎的优化算法，但在一些深度学习应用中，如训练变压器，自适应算法如Adam已经显示出经验上的优势。然而，为什么Adam在这些情况下比SGD收敛快得多仍然是一个疑问。在本文中，我们提出了一种使用新概念方向锐度来解释Adam为何比SGD收敛更快的解释。我们认为优化算法的性能与更新步骤的方向锐度密切相关，并表明与自适应算法相比，SGD的方向锐度要差得多。我们进一步观察到，只有少数坐标导致了SGD的不良锐度和缓慢收敛，并建议使用坐标剪切作为解决SGD和其他优化算法的方法。我们在各种设置下展示了坐标剪切对锐度减少和加速优化算法收敛的效果。我们表明，当只有少数坐标具有不良锐度时，坐标剪切可以改善局部损失减少。我们得出结论，自适应坐标缩放的锐度减少效果是Adam在实践中成功的原因，并建议使用坐标剪切作为一种通用技术来加速深度学习优化。 显示英文摘要

摘要： While stochastic gradient descent (SGD) is still the most popular optimization algorithm in deep learning, adaptive algorithms such as Adam have established empirical advantages over SGD in some deep learning applications such as training transformers. However, it remains a question that why Adam converges significantly faster than SGD in these scenarios. In this paper, we propose one explanation of why Adam converges faster than SGD using a new concept directional sharpness. We argue that the performance of optimization algorithms is closely related to the directional sharpness of the update steps, and show SGD has much worse directional sharpness compared to adaptive algorithms. We further observe that only a small fraction of the coordinates causes the bad sharpness and slow convergence of SGD, and propose to use coordinate-wise clipping as a solution to SGD and other optimization algorithms. We demonstrate the effect of coordinate-wise clipping on sharpness reduction and speeding up the convergence of optimization algorithms under various settings. We show that coordinate-wise clipping improves the local loss reduction when only a small fraction of the coordinates has bad sharpness. We conclude that the sharpness reduction effect of adaptive coordinate-wise scaling is the reason for Adam's success in practice and suggest the use of coordinate-wise clipping as a universal technique to speed up deep learning optimization.