PEP: Parameter Ensembling by Perturbation

Ensembling is now recognized as an effective approach for increasing the predictive performance and calibration of deep networks. We introduce a new approach, Parameter Ensembling by Perturbation (PEP), that constructs an ensemble of parameter values as random perturbations of the optimal parameter set from training by a Gaussian with a single variance parameter.The variance is chosen to maximize the log-likelihood of the ensemble average ($\mathbb{L}$) on the validation data set. Empirically, and perhaps surprisingly, $\mathbb{L}$ has a well-defined maximum as the variance grows from zero (which corresponds to the baseline model). Conveniently, calibration level of predictions also tends to grow favorably until the peak of $\mathbb{L}$ is reached. In most experiments, PEP provides a small improvement in performance, and, in some cases, a substantial improvement in empirical calibration. We show that this "PEP effect" (the gain in log-likelihood) is related to the mean curvature of the likelihood function and the empirical Fisher information. Experiments on ImageNet pre-trained networks including ResNet, DenseNet, and Inception showed improved calibration and likelihood. We further observed a mild improvement in classification accuracy on these networks. Experiments on classification benchmarks such as MNIST and CIFAR-10 showed improved calibration and likelihood, as well as the relationship between the PEP effect and overfitting; this demonstrates that PEP can be used to probe the level of overfitting that occurred during training. In general, no special training procedure or network architecture is needed, and in the case of pre-trained networks, no additional training is needed.

PEP：通过扰动进行参数合并

如今，组装被认为是提高深度网络的预测性能和校准的有效方法。我们引入了一种新方法，即“通过扰动进行参数合并（PEP）”，该方法将参数值的集合构造为最佳参数集的随机扰动，该扰动是通过高斯对单个方差参数的训练而得到的。.. 选择方差以使整体平均值的对数似然（ 大号 ）验证数据集。根据经验，也许令人惊讶， 大号 当方差从零开始增长时，它具有明确定义的最大值（对应于基线模型）。方便地，预测的校准水平也趋于有利地增长，直到达到峰值为止。 大号 到达了。在大多数实验中，PEP的性能都有很小的提高，在某些情况下，经验校准也有很大的提高。我们表明，这种“ PEP效应”（对数似然的增益）与似然函数的平均曲率和经验费舍尔信息有关。在包括ResNet，DenseNet和Inception在内的ImageNet预训练网络上进行的实验表明，标定和可能性有所提高。我们进一步观察到这些网络上分类准确度的温和提高。在诸如MNIST和CIFAR-10之类的分类基准上进行的实验显示出改进的校准和可能性，以及PEP效果和过度拟合之间的关系。这表明PEP可用于探测训练过程中发生的过度拟合水平。一般来说， （阅读更多）