Robustness-aware 2-bit quantization with real-time performance for neural network

Quantized neural network (NN) with a reduced bit precision is an effective solution to reduces the computational and memory resource requirements and plays a vital role in machine learning. However, it is still challenging to avoid the significant accuracy degradation due to its numerical approximation and lower redundancy.In this paper, a novel robustness-aware 2-bit quantization scheme is proposed for NN base on binary NN and generative adversarial network(GAN), witch improves the performance by enriching the information of binary NN, efficiently extract the structural information and considering the robustness of the quantized NN. Specifically, using shift addition operation to replace the multiply-accumulate in the quantization process witch can effectively speed the NN. Meanwhile, a structural loss between the original NN and quantized NN is proposed to such that the structural information of data is preserved after quantization. The structural information learned from NN not only plays an important role in improving the performance but also allows for further fine tuning of the quantization network by applying the Lipschitz constraint to the structural loss. In addition, we also for the first time take the robustness of the quantized NN into consideration and propose a non-sensitive perturbation loss function by introducing an extraneous term of spectral norm. The experiments are conducted on CIFAR-10 and ImageNet datasets with popular NN( such as MoblieNetV2, SqueezeNet, ResNet20, etc). The experimental results show that the proposed algorithm is more competitive under 2-bit-precision than the state-of-the-art quantization methods. Meanwhile, the experimental results also demonstrate that the proposed method is robust under the FGSM adversarial samples attack.

具有实时性能的神经网络鲁棒性2位量化

降低比特精度的量化神经网络（NN）是减少计算和内存资源需求的有效解决方案，并且在机器学习中起着至关重要的作用。但是，避免由于其数值逼近和较低的冗余度而导致的显着精度下降仍然具有挑战性。.. 本文提出了一种新的基于二进制NN和生成对抗网络（GAN）的NN感知鲁棒性2比特量化方案，通过丰富二进制NN的信息，有效提取结构信息并考虑到算法的改进来提高性能。量化NN的鲁棒性。具体地，在量化过程中使用移位加法运算来代替乘法累加可以有效地加速NN。同时，提出了原始神经网络与量化神经网络之间的结构损失，以使得数据在量化后得以保留。从NN学习到的结构信息不仅在提高性能方面起着重要作用，而且还可以通过将Lipschitz约束应用于结构损失来进一步量化网络。此外，我们还首次考虑了量化NN的鲁棒性，并通过引入频谱范数的无关项提出了一种非敏感的扰动损耗函数。实验是在具有流行NN（例如MoblieNetV2，SqueezeNet，ResNet20等）的CIFAR-10和ImageNet数据集上进行的。实验结果表明，该算法在2位精度下比最新的量化方法更具竞争力。同时，实验结果还表明，该方法在FGSM对抗样本攻击下具有较强的鲁棒性。实验是在具有流行NN（例如MoblieNetV2，SqueezeNet，ResNet20等）的CIFAR-10和ImageNet数据集上进行的。实验结果表明，该算法在2位精度下比最新的量化方法更具竞争力。同时，实验结果还表明，该方法在FGSM对抗样本攻击下具有较强的鲁棒性。实验是在具有流行NN（例如MoblieNetV2，SqueezeNet，ResNet20等）的CIFAR-10和ImageNet数据集上进行的。实验结果表明，该算法在2位精度下比最新的量化方法更具竞争力。同时，实验结果还表明，该方法在FGSM对抗样本攻击下具有较强的鲁棒性。 （阅读更多）