Efficient deep neural network training via decreasing precision with layer capacity

Abstract

Low-precision training has emerged as a practical approach, saving the cost of time, memory, and energy during deep neural networks (DNNs) training. Typically, the use of lower precision introduces quantization errors that need to be minimized to maintain model performance, often neglecting to consider the potential benefits of reducing training precision. This paper rethinks low-precision training, highlighting the potential benefits of lowering precision: (1) low precision can serve as a form of regularization in DNN training by constraining excessive variance in the model; (2) layer-wise low precision can be seen as an alternative dimension of sparsity, orthogonal to pruning, contributing to improved generalization in DNNs. Based on these analyses, we propose a simple yet powerful technique – DPC (Decreasing Precision with layer Capacity), which directly assigns different bit-widths to model layers, without the need for an exhaustive analysis of the training process or any delicate low-precision criteria. Thorough extensive experiments on five datasets and fourteen models across various applications consistently demonstrate the effectiveness of the proposed DPC technique in saving computational cost (−16.21%–−44.37%) while achieving comparable or even superior accuracy (up to +0.68%, +0.21% on average). Furthermore, we offer feature embedding visualizations and conduct further analysis with experiments to investigate the underlying mechanisms behind DPC’s effectiveness, enhancing our understanding of low-precision training. Our source code will be released upon paper acceptance.

Graphical abstract

Keywords

low precision / efficient training / generalization / regularization / bit-width assignment

References

[1] Du J, Shen M, Du Y. A distributed in-situ CNN inference system for IOT applications. In: Proceedings of the 38th IEEE International Conference on Computer Design (ICCD). 2020, 279–287

[2] Yang C, Wu Z, Chee J, De Sa C, Udell M. How low can we go: trading memory for error in low-precision training. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[3] Zhu F, Gong R, Yu F, Liu X, Wang Y, Li Z, Yang X, Yan J. Towards unified int8 training for convolutional neural network. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 1966–1976

[4] Micikevicius P, Narang S, Alben J, Diamos G, Elsen E, Garcia D, Ginsburg B, Houston M, Kuchaiev O, Venkatesh G, Wu H. Mixed precision training. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[5] Zhou S, Wu Y, Ni Z, Zhou X, Wen H, Zou Y. DoReFa-Net: training low bitwidth convolutional neural networks with low bitwidth gradients. 2016, arXiv preprint arXiv: 1606.06160

[6] Yang L, Jin Q. FracBits: mixed precision quantization via fractional bit-widths. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 10612–10620

[7] E lthakeb A T, P illigundla P, M ireshghallah F, Y azdanbakhsh A, E smaeilzadeh H . ReLeQ: a reinforcement learning approach for automatic deep quantization of neural networks. IEEE Micro, 2020, 40( 5): 37–45

[8] Yang H, Duan L, Chen Y, Li H. BSQ: exploring bit-level sparsity for mixed-precision neural network quantization. In: Proceedings of the 9th International Conference on Learning Representations, 2021

[9] Ma Z, He J, Qiu J, Cao H, Wang Y, Sun Z, Zheng L, Wang H, Tang S, Zheng T, Lin J, Feng G, Huang Z, Gao J, Zeng A, Zhang J, Zhong R, Shi T, Liu S, Zheng W, Tang J, Yang H, Liu X, Zhai J, Chen W. BaGuaLu: targeting brain scale pretrained models with over 37 million cores. In: Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2022, 192–204

[10] Lee S, Park J, Jeon D. Toward efficient low-precision training: data format optimization and hysteresis quantization. In: Proceedings of the 10th International Conference on Learning Representations. 2021

[11] Sun X, Wang N, Chen C Y, Ni J M, Agrawal A, Cui X, Venkataramani S, El Maghraoui K, Srinivasan V V, Gopalakrishnan K. Ultra-low precision 4-bit training of deep neural networks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 33: 1796–1807

[12] Fu Y, You H, Zhao Y, Wang Y, Li C, Gopalakrishnan K, Wang Z, Lin Y. FracTrain: fractionally squeezing bit savings both temporally and spatially for efficient DNN training. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1017

[13] Koster U, Webb T J, Wang X, Nassar M, Bansal A K, Constable W H, Elibol O H, Gray S, Hall S, Hornof L, Khosrowshahi A, Kloss C, Pai R J, Rao N. Flexpoint: an adaptive numerical format for efficient training of deep neural networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 1740–1750

[14] Das D, Mellempudi N, Mudigere D, Kalamkar D, Avancha S, Banerjee K, Sridharan S, Vaidyanathan K, Kaul B, Georganas E, Heinecke A, Dubey P, Corbal J, Shustrov N, Dubtsov R, Fomenko E, Pirogov V. Mixed precision training of convolutional neural networks using integer operations. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[15] Fox S, Rasoulinezhad S, Faraone J, Leong P, Leong P. A block minifloat representation for training deep neural networks. In: Proceedings of the 9th International Conference on Learning Representations. 2020

[16] Sun X, Choi J, Chen C Y, Wang N, Venkataramani S, Srinivasan V V, Cui X, Zhang W, Gopalakrishnan K. Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 441

[17] Zhang D, Yang J, Ye D, Hua G. LQ-Nets: learned quantization for highly accurate and compact deep neural networks. In: Proceedings of the 15th European Conference on Computer Vision-ECCV 2018. 2018, 373–390

[18] Choi J, Wang Z, Venkataramani S, Chuang P I J, Srinivasan V, Gopalakrishnan K. PACT: parameterized clipping activation for quantized neural networks. In: Proceedings of the 6th International Conference on Learning Representations, 2018

[19] Chen J, Gai Y, Yao Z, Mahoney M W, Gonzalez J E. A statistical framework for low-bitwidth training of deep neural networks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 75

[20] Fu F, Hu Y, He Y, Jiang J, Shao Y, Zhang C, Cui B. Don’t waste your bits! Squeeze activations and gradients for deep neural networks via TINYSCRIPT. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 309

[21] Wu S, Li G, Chen F, Shi L. Training and inference with integers in deep neural networks. In: Proceedings of the 6th International Conference on Learning Representations, 2018

[22] Y ang Y, D eng L, W u S, Y an T, X ie Y, L i G . Training high-performance and large-scale deep neural networks with full 8-bit integers. Neural Networks, 2020, 125: 70–82

[23] Wang N, Choi J, Brand D, Chen C Y, Gopalakrishnan K. Training deep neural networks with 8-bit floating point numbers. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 7686–7695

[24] G randvalet Y, C anu S, B oucheron S . Noise injection: theoretical prospects. Neural Computation, 1997, 9( 5): 1093–1108

[25] Neelakantan A, Vilnis L, Le Q V, Sutskever I, Kaiser L, Kurach K, Martens J. Adding gradient noise improves learning for very deep networks. In: Proceedings of the 5th International Conference on Learning Representations. 2017

[26] Fu Y, Guo H, Li M, Yang X, Ding Y, Chandra V, Lin Y. CPT: efficient deep neural network training via cyclic precision. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[27] M ocanu D C, M ocanu E, S tone P, N guyen P H, G ibescu M, L iotta A . Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 2018, 9( 1): 2383

[28] Evci U, Gale T, Menick J, Castro P S, Elsen E. Rigging the lottery: making all tickets winners. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 2943–2952

[29] Liu S, Chen T, Chen X, Shen L, Mocanu D C, Wang Z, Pechenizkiy M. The unreasonable effectiveness of random pruning: return of the most naive baseline for sparse training. In: Proceedings of the 10th International Conference on Learning Representations, ICLR 2022. 2022, 21

[30] Zhou A, Yao A, Guo Y, Xu L, Chen Y. Incremental network quantization: towards lossless CNNs with low-precision weights. In: Proceedings of the 5th International Conference on Learning Representations, 2017

[31] Krishnamoorthi R. Quantizing deep convolutional networks for efficient inference: a whitepaper. 2018, arXiv preprint arXiv: 1806.08342

[32] Banner R, Nahshan Y, Soudry D. Post training 4-bit quantization of convolutional networks for rapid-deployment. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 714

[33] Wang K, Liu Z, Lin Y, Lin J, Han S. HAQ: hardware-aware automated quantization with mixed precision. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 8604–8612

[34] Guo C, Tang J, Hu W, Leng J, Zhang C, Yang F, Liu Y, Guo M, Zhu Y. OliVe: accelerating large language models via hardware-friendly outlier-victim pair quantization. In: Proceedings of the 50th Annual International Symposium on Computer Architecture. 2023, 3

[35] Guo C, Zhang C, Leng J, Liu Z, Yang F, Liu Y, Guo M, Zhu Y. ANT: exploiting adaptive numerical data type for low-bit deep neural network quantization. In: Proceedings of the 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). 2022, 1414–1433

[36] O nan A, K orukoğlu S, B ulut H . A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification. Information Processing & Management, 2017, 53( 4): 814–833

[37] Achille A, Rovere M, Soatto S. Critical learning periods in deep networks. In: Proceedings of the 7th International Conference on Learning Representations. 2018

[38] A chille A, S oatto S . Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 2018, 19( 1): 1947–1980

[39] Raghu M, Poole B, Kleinberg J, Ganguli S, Sohl Dickstein J. On the expressive power of deep neural networks. In: Proceedings of the 34th International Conference on Machine Learning. 2017, 2847–2854

[40] Martinez B, Yang J, Bulat A, Tzimiropoulos G. Training binary neural networks with real-to-binary convolutions. In: Proceedings of the 8th International Conference on Learning Representations. 2020

[41] Banner R, Hubara I, Hoffer E, Soudry D. Scalable methods for 8-bit training of neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing systems. 2018, 5151–5159

[42] Park E, Yoo S. PROFIT: a novel training method for sub-4-bit MobileNet models. In: Proceedings of the 16th European Conference on Computer Vision. 2020, 430–446

[43] Yang G, Zhang T, Kirichenko P, Bai J, Wilson A G, De Sa C. SWALP: stochastic weight averaging in low precision training. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 7015–7024

[44] Han R, Si M, Demmel J, You Y. Dynamic scaling for low-precision learning. In: Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2021, 480–482

[45] Feng B, Wang Y, Geng T, Li A, Ding Y. APNN-TC: accelerating arbitrary precision neural networks on ampere GPU tensor cores. In: Proceedings of the SC21: International Conference for High Performance Computing, Networking, Storage and Analysis. 2021, 1–14

[46] Knorr F, Thoman P, Fahringer T. ndzip-gpu: efficient lossless compression of scientific floating-point data on GPUs. In: SC21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 2021, 1–13

[47] Savarese P, Yuan X, Li Y, Maire M. Not all bits have equal value: heterogeneous precisions via trainable noise. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 35769–35782

[48] Xi H, Li C, Chen J, Zhu J. Training transformers with 4-bit integers. In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023). 2023

[49] Huang X, Shen Z, Li S, Liu Z, Hu X, Wicaksana J, Xing E, Cheng K T. SDQ: stochastic differentiable quantization with mixed precision. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 9295–9309

[50] Chakrabarti A, Moseley B. Backprop with approximate activations for memory-efficient network training. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 218

[51] Jin S, Li G, Song S L, Tao D. A novel memory-efficient deep learning training framework via error-bounded lossy compression. In: Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2021, 485–487

[52] Chen J, Zheng L, Yao Z, Wang D, Stoica I, Mahoney M, Gonzalez J. ActNN: reducing training memory footprint via 2-bit activation compressed training. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 1803–1813

[53] Zhuang B, Shen C, Tan M, Liu L, Reid I. Towards effective low-bitwidth convolutional neural networks. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 7920–7928

[54] Hardt M, Recht B, Singer Y. Train faster, generalize better: Stability of stochastic gradient descent. In: Proceedings of the 33rd International Conference on Machine Learning. 2016, 1225–1234

[55] Z hang C, B engio S, H ardt M, R echt B, V inyals O . Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 2021, 64( 3): 107–115

[56] Neyshabur B, Li Z, Bhojanapalli S, LeCun Y, Srebro N. Towards understanding the role of over-parametrization in generalization of neural networks. 2018, arXiv preprint arXiv: 180512076

[57] Poggio T, Torre V, Koch C. Computational vision and regularization theory. In: Fischler M A, Firschein Q, eds. Readings in Computer Vision: Issues, Problem, Principles, and Paradigms. Los Altos: Morgan Kaufmann, 1987, 638–643

[58] Dettmers T, Zettlemoyer L. The case for 4-bit precision: k-bit inference scaling laws. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 370

[59] Su J, Chen Y, Cai T, Wu T, Gao R, Wang L, Lee J D. Sanity-checking pruning methods: random tickets can win the jackpot. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 33: 1712

[60] Frankle J, Carbin M. The lottery ticket hypothesis: finding sparse, trainable neural networks. In: Proceedings of the 7th International Conference on Learning Representations. 2019

[61] Mostafa H, Wang X. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 4646–4655

[62] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770–778

[63] Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L C. MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 4510–4520

[64] Wang Y, Jiang Z, Chen X, Xu P, Zhao Y, Lin Y, Wang Z. E²-train: training state-of-the-art CNNs with over 80% energy savings. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 462

[65] Krizhevsky A. Learning multiple layers of features from tiny images. University of Toronto, 2012

[66] Deng J, Dong W, Socher R, Li L J, Li K, Fei-Fei L. ImageNet: a large-scale hierarchical image database. In: Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009, 248–255

[67] Baevski A, Auli M. Adaptive input representations for neural language modeling. In: Proceedings of the 7th International Conference on Learning Representations. 2019

[68] Merity S, Keskar N S, Socher R. Regularizing and optimizing LSTM language models. In: Proceedings of the 6th International Conference on Learning Representations. 2018

[69] Wang X, Yu F, Dou Z Y, Darrell T, Gonzalez J E. SkipNet: learning dynamic routing in convolutional networks. In: Proceedings of the 15th European Conference on Computer Vision-ECCV 2018. 2018, 420–436

[70] Howard A G, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. MobileNets: efficient convolutional neural networks for mobile vision applications. 2017, arXiv preprint arXiv: 1704.04861

[71] Hou L, Zhu J, Kwok J, Gao F, Qin T, Liu T Y. Normalization helps training of quantized LSTM. In: Proceedings of the 33rd Neural Information Processing Systems. 2019, 660

[72] K rizhevsky A, S utskever I, H inton G E . ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60( 6): 84–90

[73] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014, arXiv preprint arXiv: 1409.1556

[74] He C, Li S, Soltanolkotabi M, Avestimehr S. PipeTransformer: automated elastic pipelining for distributed training of transformers. 2021, arXiv preprint arXiv: 2102.03161

[75] Raghu M, Gilmer J, Yosinski J, Sohl-Dickstein J. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 6078–6087

[76] Nielsen M A. Neural Networks and Deep Learning. San Francisco, CA, USA: Determination Press, 2015

[77] Frantar E, Ashkboos S, Hoefler T, Alistarh D. GPTQ: accurate post-training quantization for generative pre-trained transformers. In: Proceedings of the 11th International Conference on Learning Representations. 2023

[78] Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLORA: efficient finetuning of quantized LLMs. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2024, 441

[79] Huang G, Liu Z, Van Der Maaten L, Weinberger K Q. Densely connected convolutional networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017, 2261–2269

RIGHTS & PERMISSIONS

The Author(s) 2024. This article is published with open access at link.springer.com and journal.hep.com.cn