标题： 具有直通估计器的量化模型的高维学习动态 显示英文标题

标题： High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator

摘要： 量化神经网络训练优化一个离散的、不可微的目标。 直通估计器（STE）通过替代梯度实现反向传播，并被广泛使用。 尽管之前的研究主要集中在替代梯度的性质及其收敛性上，但量化超参数（如位宽和量化范围）对学习动态的影响仍大多未被探索。 我们理论证明，在高维极限下，STE动态收敛到一个确定性的常微分方程。 这表明STE训练在泛化误差上表现出一个平台期，随后出现急剧下降，平台期长度取决于量化范围。 定点分析量化了与未量化线性模型的渐近偏差。 我们还将用于随机梯度下降的分析技术扩展到权重和输入的非线性变换。 显示英文摘要

摘要： Quantized neural network training optimizes a discrete, non-differentiable objective. The straight-through estimator (STE) enables backpropagation through surrogate gradients and is widely used. While previous studies have primarily focused on the properties of surrogate gradients and their convergence, the influence of quantization hyperparameters, such as bit width and quantization range, on learning dynamics remains largely unexplored. We theoretically show that in the high-dimensional limit, STE dynamics converge to a deterministic ordinary differential equation. This reveals that STE training exhibits a plateau followed by a sharp drop in generalization error, with plateau length depending on the quantization range. A fixed-point analysis quantifies the asymptotic deviation from the unquantized linear model. We also extend analytical techniques for stochastic gradient descent to nonlinear transformations of weights and inputs.