标题： 理解贝叶斯神经网络中单元级别的先验知识 显示英文标题

标题： Understanding Priors in Bayesian Neural Networks at the Unit Level

摘要： 我们研究具有高斯权重先验和一类类似ReLU的非线性的深度贝叶斯神经网络。具有高斯先验的贝叶斯神经网络众所周知可以产生L2，“权重衰减”，正则化。我们的结果描述了单元激活层面更复杂的正则化效应。我们的主要结果表明，单元在激活前后的先验分布随着层的深度增加而变得越来越重尾。我们证明第一层单元是高斯的，第二层单元是次指数的，更深层的单元由次威布尔分布表征。我们的结果为深度贝叶斯神经网络提供了新的理论见解，我们通过模拟实验验证了这些结果。 显示英文摘要

摘要： We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, "weight decay", regularization. Our results characterize a more intricate regularization effect at the level of the unit activations. Our main result establishes that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer. We show that first layer units are Gaussian, second layer units are sub-exponential, and units in deeper layers are characterized by sub-Weibull distributions. Our results provide new theoretical insight on deep Bayesian neural networks, which we corroborate with simulation experiments.