标题： Adam减少一种特殊的锐度：最小化流形附近的理论见解 显示英文标题

标题： Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold

摘要： 尽管Adam优化器在实践中很受欢迎，但大多数理论分析将随机梯度下降（SGD）作为Adam的代理，关于Adam找到的解有何不同了解甚少。 在本文中，我们表明Adam隐式地减少了一种由其自适应更新塑造的独特尖锐度度量，导致与SGD定性不同的解。 更具体地说，当训练损失较小时，Adam在最小值流形周围游走，并以自适应方式采取半梯度来最小化这种尖锐度度量，我们通过使用随机微分方程的连续时间近似严格表征了这种行为。 我们进一步展示了这种行为在一种广泛研究的设置中与SGD的不同：当用标签噪声训练过参数化模型时，SGD已被证明可以最小化Hessian矩阵的迹，$\tr(\mH)$，而我们证明Adam则最小化$\tr(\Diag(\mH)^{1/2})$。 在使用对角线线性网络解决稀疏线性回归时，这种区别使Adam能够比SGD实现更好的稀疏性和泛化能力。 最后，我们的分析框架不仅适用于Adam，还适用于一大类自适应梯度方法，包括RMSProp、Adam-mini、Adalayer和Shampoo，并提供了一个统一的观点来理解这些自适应优化器如何减少尖锐度，我们希望这能为未来优化器的设计提供见解。 显示英文摘要

摘要： Despite the popularity of the Adam optimizer in practice, most theoretical analyses study Stochastic Gradient Descent (SGD) as a proxy for Adam, and little is known about how the solutions found by Adam differ. In this paper, we show that Adam implicitly reduces a unique form of sharpness measure shaped by its adaptive updates, leading to qualitatively different solutions from SGD. More specifically, when the training loss is small, Adam wanders around the manifold of minimizers and takes semi-gradients to minimize this sharpness measure in an adaptive manner, a behavior we rigorously characterize through a continuous-time approximation using stochastic differential equations. We further demonstrate how this behavior differs from that of SGD in a well-studied setting: when training overparameterized models with label noise, SGD has been shown to minimize the trace of the Hessian matrix, $\tr(\mH)$, whereas we prove that Adam minimizes $\tr(\Diag(\mH)^{1/2})$ instead. In solving sparse linear regression with diagonal linear networks, this distinction enables Adam to achieve better sparsity and generalization than SGD. Finally, our analysis framework extends beyond Adam to a broad class of adaptive gradient methods, including RMSProp, Adam-mini, Adalayer and Shampoo, and provides a unified perspective on how these adaptive optimizers reduce sharpness, which we hope will offer insights for future optimizer design.