TBNN: totally-binary neural network for image classification

Abstract

Most binary networks apply full precision convolution at the first layer. Changing the first layer to the binary convolution will result in a significant loss of accuracy. In this paper, we propose a new approach to solve this problem by widening the data channel to reduce the information loss of the first convolutional input through the sign function. In addition, widening the channel increases the computation of the first convolution layer, and the problem is solved by using group convolution. The experimental results show that the accuracy of applying this paper’s method to state-of-the-art (SOTA) binarization method is significantly improved, proving that this paper’s method is effective and feasible.

References

[1] HeR, SunS, YangJ, et al.. Knowledge distillation as efficient pre-training: faster convergence, higher data-efficiency, and better transferability[C], 2022, New York, IEEE: 9161-9171

[2] ZhangL, ChenX, TuX, et al.. Wavelet knowledge distillation: towards efficient image-to-image translation[C], 2022, New York, IEEE: 12464-12474

[3] ZhongY, LinM, NanG, et al.. IntraQ: learning synthetic images with intra-class heterogeneity for zero-shot network quantization[C], 2022, New York, IEEE: 12339-12348

[4] LiuC, DingW, XiaX, et al.. Circulant binary convolutional networks: enhancing the performance of 1-bit DCNNs with circulant back propagation[C], 2019, New York, IEEE: 2691-2699

[5] LiuZ, ShenZ, SavvidesM, et al.. ReActNet: towards precise binary neural network with generalized activation functions[C], 2020, Cham, Springer: 143-159Virtual

[6] ZHOU S, WU Y, NI Z, et al. Dorefa-net: training low bit width convolutional neural networks with low bit width gradients[EB/OL]. (2016-06-20) [2022-06-22]. https://arxiv.org/pdf/1606.06160.pdf.

[7] DingR, ChinT W, LiuZ, et al.. Regularizing activation distribution for training binarized deep networks[C], 2019, New York, IEEE: 11408-11417

[8] HOWARD A G, ZHU M, CHEN B, et al. Mobilenets: efficient convolutional neural networks for mobile vision applications[EB/OL]. (2017-04-17) [2022-06-22]. https://arxiv.org/pdf/1704.04861.pdf.

[9] ZhangX, ZhouX, LinM, et al.. Shufflenet: an extremely efficient convolutional neural network for mobile devices[C], 2018, New York, IEEE: 6848-6856

[10] MEHTA S, RASTEGARI M. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer[EB/OL]. (2021-10-17) [2022-06-22]. https://arxiv.org/pdf/2110.02178v2.pdf.

[11] QinH, GongR, LiuX, et al.. Forward and backward information retention for accurate binary neural networks[C], 2020, New York, IEEE: 2250-2259

[12] LiuZ, WuB, LuoW, et al.. Bi-real net: enhancing the performance of 1-bit CNNs with improved representational capability and advanced training algorithm[C], 2018, Berlin, Heidelberg, Springer-Verlag: 722-737

[13] LinX, ZhaoC, PanW. Towards accurate binary convolutional neural network[J]. Advances in neural information processing systems, 2017, 30: 345-353

[14] SU Z, FANG L, GUO D, et al. FTBNN: rethinking non-linearity for 1-bit CNNs and going beyond[EB/OL]. (2010-09-29) [2022-06-22]. https://www.xueshufan.com/reader/3096361616.

[15] KIM H, KIM K, KIM J, et al. Binaryduo: reducing gradient mismatch in binary activation network by coupling binary activations[EB/OL]. (2002-06-05) [2022-06-22]. https://arxiv.org/pdf/2002.06517v1.pdf.

[16] XieS, GirshickR, DollárP, et al.. Aggregated residual transformations for deep neural networks[C], 2017, New York, IEEE: 1492-1500

[17] CholletF. Xception: deep learning with depth wise separable convolutions[C], 2017, New York, IEEE: 1251-1258

[18] HubaraI, CourbariauxM, SoudryD, et al.. Binarized neural networks[J]. Advances in neural information processing systems, 2016, 29: 4107-4115

[19] GongR, LiuX, JiangS, et al.. Differentiable soft quantization: bridging full-precision and low-bit neural networks[C], 2019, New York, IEEE: 4852-4861