标题： 宽神经网络使用权重衰减训练确实表现出神经崩溃 显示英文标题

标题： Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

摘要： 深度神经网络（DNNs）在收敛时通过一种称为神经坍缩的高度对称几何结构，一致地在最后一层表示训练数据。 这一实证证据推动了一系列理论研究，旨在证明神经坍缩的出现，主要集中在无约束特征模型上。 在此模型中，倒数第二层的特征是自由变量，这使得模型与数据无关，从而质疑了其捕捉DNN训练的能力。 我们的工作解决了这一问题，摆脱了无约束特征，研究以至少两个线性层结尾的DNN。 我们首先证明了神经坍缩的通用保证，假设（i）低训练误差和线性层的平衡性（用于类内可变性坍缩），以及（ii）线性部分之前的特征的有界条件（用于类均值的正交性，以及它们与权重矩阵的对齐）。 然后我们表明，这些假设适用于带有权重衰减的梯度下降训练：（i）对于具有宽第一层的网络，我们证明了低训练误差和平衡性，（ii）对于几乎最优或在大学习率下稳定的解，我们进一步证明了有界条件。 综合来看，我们的结果是首次在DNN端到端训练中展示了神经坍缩。 显示英文摘要

摘要： Deep neural networks (DNNs) at convergence consistently represent the training data in the last layer via a highly symmetric geometric structure referred to as neural collapse. This empirical evidence has spurred a line of theoretical research aimed at proving the emergence of neural collapse, mostly focusing on the unconstrained features model. Here, the features of the penultimate layer are free variables, which makes the model data-agnostic and, hence, puts into question its ability to capture DNN training. Our work addresses the issue, moving away from unconstrained features and studying DNNs that end with at least two linear layers. We first prove generic guarantees on neural collapse that assume (i) low training error and balancedness of the linear layers (for within-class variability collapse), and (ii) bounded conditioning of the features before the linear part (for orthogonality of class-means, as well as their alignment with weight matrices). We then show that such assumptions hold for gradient descent training with weight decay: (i) for networks with a wide first layer, we prove low training error and balancedness, and (ii) for solutions that are either nearly optimal or stable under large learning rates, we additionally prove the bounded conditioning. Taken together, our results are the first to show neural collapse in the end-to-end training of DNNs.