Neural Networks Preserve Invertibility Across Iterations: A Possible Source of Implicit Data Augmentation

Determining what kind of representations neural networks learn, and how this may relate to generalization, remains a challenging problem. Previous work has utilized a rich set of methods to invert layer representations of neural networks, i.e. given some reference activation $\Phi_0$ and a layer function $r_{\ell}$, find $x$ which minimizes $||\Phi_0 - r_{\ell}(x)||^2$ .We show that neural networks can preserve invertibility across several iterations. That is, it is possible to interpret activations produced in some later iteration in the context of the layer function of the current iteration. For convolutional and fully connected networks, the lower layers maintain such a consistent representation for several iterations, while in the higher layers invertibility holds for fewer iterations. Adding skip connections such as those found in Resnet allows even higher layers to preserve invertibility across several iterations. We believe the fact that higher layers may interpret weight changes made by lower layers as changes to the data may produce implicit data augmentation. This implicit data augmentation may eventually yield some insight into why neural networks can generalize even with so many parameters.

神经网络保留迭代的可逆性：隐式数据增强的可能来源

确定神经网络学习什么样的表示形式以及如何将其与泛化联系仍然是一个具有挑战性的问题。先前的工作利用了丰富的方法来反转神经网络的层表示，即给定一些参考激活 Φ0 和一个图层功能 [Rℓ ， 找 X 最小化 ||Φ0-[Rℓ（X）||2 。.. 我们证明了神经网络可以在多次迭代中保持可逆性。即，可以在当前迭代的层函数的上下文中解释在某些后续迭代中产生的激活。对于卷积网络和完全连接的网络，较低的层在多次迭代中保持这种一致的表示，而在较高的层中，可逆性保持较少的迭代。添加跳过连接（例如在Resnet中找到的连接）可以使更高的层在多次迭代中保持可逆性。我们认为，较高的层可以解释较低层的权重变化，因为对数据的更改可能会产生隐式的数据增强。这种隐式的数据增强最终可以使人深入了解为什么神经网络即使具有如此多的参数也能泛化。 （阅读更多）