标题： 结合显式和隐式正则化在深度网络中的高效学习 显示英文标题

标题： Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks

摘要： 对隐式正则化的工作研究了优化过程中的梯度轨迹，以解释为什么深度网络更倾向于某些类型的解决方案。 在深度线性网络中，已经证明梯度下降在矩阵完成/分解任务中会隐式地正则化为低秩解决方案。 增加深度不仅提高了这些任务的性能，还作为一种加速的预条件，进一步增强了这种对低秩性的偏倚。 受此启发，我们提出了一种显式惩罚来模拟这种隐式偏倚，该惩罚仅在某些自适应梯度优化器（例如 Adam）上生效。 这种组合可以使退化的单层网络实现与深度线性网络相当的泛化误差的低秩近似，使深度不再是学习的必要条件。 尽管结构简单，该单层网络在各种参数和数据条件下，在矩阵完成任务中仍表现出竞争力或优于各种方法。 结合优化器的归纳偏倚，我们的研究结果表明，显式正则化可以在设计不同且理想的正则化形式中发挥作用，而对这种相互作用更细致的理解可能是必要的。 显示英文摘要

摘要： Works on implicit regularization have studied gradient trajectories during the optimization process to explain why deep networks favor certain kinds of solutions over others. In deep linear networks, it has been shown that gradient descent implicitly regularizes toward low-rank solutions on matrix completion/factorization tasks. Adding depth not only improves performance on these tasks but also acts as an accelerative pre-conditioning that further enhances this bias towards low-rankedness. Inspired by this, we propose an explicit penalty to mirror this implicit bias which only takes effect with certain adaptive gradient optimizers (e.g. Adam). This combination can enable a degenerate single-layer network to achieve low-rank approximations with generalization error comparable to deep linear networks, making depth no longer necessary for learning. The single-layer network also performs competitively or out-performs various approaches for matrix completion over a range of parameter and data regimes despite its simplicity. Together with an optimizer's inductive bias, our findings suggest that explicit regularization can play a role in designing different, desirable forms of regularization and that a more nuanced understanding of this interplay may be necessary.