Optimizing knowledge distillation for efficient breast ultrasound image segmentation: Insights and performance enhancement

Abstract

Most modern models designed for ultrasound (US) image segmentation are characterized by high computational and memory requirements, limiting their practical utility in point-of-care US settings. Consequently, researchers have devised innovative approaches to compress these large models, enabling the training of smaller networks capable of achieving comparable generalization performance. Among these strategies, knowledge distillation (KD) has emerged as particularly suitable for scenarios involving small datasets or where significant efficiency improvements are desired. While previous KD-based methods have focused on extracting comprehensive information from diverse levels of teacher representation, they often overlook the identification of the most effective representation level. Additionally, many existing techniques propose intricate strategies that present implementation challenges. To address this gap, our study concentrates on selecting optimal teacher representations from various levels. Through an exhaustive analysis of KD pathways, loss functions, and the impact of augmentation, we offer valuable insights into the mechanisms underlying knowledge transfer from the teacher to the student networks. Our proposed methodology significantly enhances student performance, elevating the Dice similarity score from 73% to 80%, while the teacher model achieves 81%. Notably, our student model achieves this improvement with only 0.82 million parameters, compared to the teacher model’s 96 million parameters.

Keywords

Ultrasound / Image segmentation / Model compression / Knowledge distillation

Funding

This work was supported by The Natural Sciences and Engineering Research Council of Canada (NSERC) (NSERC RGPIN-2020-04612).

Conflict of interest

The authors declare that they have no competing interests.

References

[1] Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning in medical image analysis. Med Image Anal. 2017; 42:60-88. doi: 10.1016/j.media.2017.07.005

[2] Toennies KD. Guide to Medical Image Analysis. Berlin: Springer; 2017.

[3] Zhou K. Medical Image Recognition, Segmentation and Parsing: Machine Learning and Multiple Object Approaches. United States: Academic Press; 2015.

[4] Xian M, Zhang Y, Cheng HD, Xu F, Zhang B, Ding J. Automatic breast ultrasound image segmentation: A survey. Pattern Recogn. 2018; 79:340-355. doi: 10.1016/j.patcog.2018.02.012

[5] Shen D, Wu G, Suk HI. Deep learning in medical image analysis. Annu Rev Biomed Eng. 2017; 19:221-248. doi: 10.1146/annurev-bioeng-071516-044442

[6] Hesamian MH, Jia W, He X, Kennedy P. Deep learning techniques for medical image segmentation: Achievements and challenges. J Digit Imaging. 2019; 32:582-596. doi: 10.1007/s10278-019-00227-x

[7] Ghosh D, Kumar A, Ghosal P, Chowdhury T, Sadhu A, Nandi D. Breast Lesion Segmentation in Ultrasound Images Using Deep Convolutional Neural Networks. In: 2020 IEEE Calcutta Conference (CALCON). IEEE; 2020. p. 318-322. doi: 10.1109/CALCON49167.2020.9106568

[8] Isensee F, Jaeger PF, Kohl SA, Petersen J, Maier-Hein KH. nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nat Methods. 2021; 18:203-211. doi: 10.1038/s41592-020-01008-z

[9] Moore CL, Copel JA. Point-of-care ultrasonography. N Engl J Med. 2011;364:749-757. doi: 10.1056/NEJMra0909487

[10] Zieleskiewicz L, Muller L, Lakhal K, et al. Point-of-care ultrasound in intensive care units: Assessment of 1073 procedures in a multicentric, prospective, observational study. Intensive Care Med. 2015; 41:1638-1647. doi: 10.1007/s00134-015-3952-5

[11] Fujioka T, Kubota K, Hsu JF, et al. Examining the effectiveness of a deep learning-based computer-aided breast cancer detection system for breast ultrasound. J Med Ultrasonics. 2023; 50:511-520. doi: 10.1007/s10396-023-01332-9

[12] Ding W, Zhang H, Zhuang S, Zhuang Z, Gao Z. Multi-view stereoscopic attention network for 3D tumor classification in automated breast ultrasound. Expert Syst Appl. 2023; 234:120969. doi: 10.1016/j.eswa.2023.120969

[13] Balasubramaniam S, Velmurugan Y, Jaganathan D, Dhanasekaran S. A modified LeNet CNN for breast cancer diagnosis in ultrasound images. Diagnostics (Basel). 2023; 13:2746. doi: 10.3390/diagnostics13172746

[14] Neill JO. An Overview of Neural Network Compression, arXiv preprint arXiv:2006. 03669; 2020. doi: 10.48550/arXiv.2006.03669

[15] Cheng Y, Wang D, Zhou P, Zhang T. A survey of model compression and acceleration for deep neural networks, arXiv preprint arXiv:1710.09282; 2017. doi: 10.48550/arXiv.1710.09282

[16] Wang L, Yoon KJ. Knowledge Distillation and Student-teacher Learning for Visual Intelligence: A Review and New Outlooks. In: IEEE Transactions on Pattern Analysis and Machine Intelligence; 2021. doi: 10.1109/TPAMI.2021.3055564

[17] Hanson S, Pratt L. Comparing Biases for Minimal Network Construction with Back-Propagation. Advances in Neural Information Processing Systems. In: NeurIPS Proceedings; 1988.

[18] LeCun Y, Denker J, Solla S. Optimal Brain Damage. Advances in Neural Information Processing Systems. In: NeurIPS Proceedings; 1989.

[19] Hassibi B, Stork D. Second Order Derivatives for Network Pruning: Optimal Brain Surgeon, Advances in Neural Information Processing Systems. In: NeurIPS Proceedings; 1992.

[20] Gong Y, Liu L, Yang M, Bourdev L. Compressing Deep Convolutional Networks Using Vector Quantization, arXiv preprint arXiv:1412.6115; 2014. doi: 10.48550/arXiv.1412.6115

[21] Wu J, Leng C, Wang Y, Hu Q, Cheng J. Quantized Convolutional Neural Networks for Mobile Devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016. p. 4820-4828. doi: 10.1109/CVPR.2016.521

[22] Vanhoucke V, Senior A, Mao MZ, et al. Improving the Speed of Neural Networks on CPUs. In: Proceeding Deep Learning and Unsupervised Feature Learning NIPS Workshop.Vol. 1; 2011. p. 4.

[23] Ba J, Caruana R. Do Deep Nets Really Need to be Deep? Advances in Neural Information Processing Systems. In: NeurIPS Proceedings; 2014. doi: 10.48550/arXiv.1312.6184

[24] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531 2; 2015. doi: 10.48550/arXiv.1503.02531

[25] Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y. Fitnets: Hints for thin deep nets, arXiv preprint arXiv:1412.6550; 2014. doi: 10.48550/arXiv.1412.6550

[26] Gou J, Yu B, Maybank SJ, Tao D. Knowledge distillation: A survey. Int J Comput Vision. 2021; 129:1789-1819. doi: 10.1007/s11263-021-01453-z

[27] Yap MH, Pons G, Mart’i J, et al. Automated breast ultrasound lesions detection using convolutional neural networks. IEEE J Biomed Health Inform. 2017; 22:1218-1226. doi: 10.1109/JBHI.2017.2731873

[28] Xu K, Rui L, Li Y, Gu L. Feature Normalized Knowledge Distillation for Image classification. In: European Conference on Computer Vision. Springer; 2020. p. 664-680. doi: 10.1007/978-3-030-58595-2_40

[29] Zagoruyko S, Komodakis N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928; 2016. doi: 10.48550/arXiv.1612.03928

[30] Wang H, Zhang D, Song Y, et al. Segmenting Neuronal Structure in 3D Optical Microscope Images via Knowledge Distillation with Teacher-Student Network. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).IEEE; 2019. p. 228-231. doi: 10.1109/ISBI.2019.8759326

[31] He T, Shen C, Tian Z, Gong D, Sun C, Yan Y. Knowledge Adaptation for Efficient Semantic Segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019. p. 578-587. doi: 10.48550/arXiv.1903.04688

[32] Liu Y, Chen K, Liu C, Qin Z, Luo Z, Wang J. Structured Knowledge Distillation for Semantic Segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019. p. 2604-2613. doi: 10.1109/CVPR.2019.00271

[33] Tung F, Mori G. Similarity-Preserving Knowledge Distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019. p. 1365-1374. doi: 10.1109/ICCV.2019.00145

[34] Gao Z, Chung J, Abdelrazek M, et al. Privileged modality distillation for vessel border detection in intracoronary imaging. IEEE Trans Med Imaging. 2019; 39:1524-1534. doi: 10.1109/TMI.2019.2952939

[35] Dou Q, Liu Q, Heng PA, Glocker B. Unpaired multi-modal segmentation via knowledge distillation. IEEE Trans Med Imaging. 2020; 39:2415-2425. doi: 10.1109/TMI.2019.2963882

[36] Chen Z, Guo X, Woo PY, Yuan Y. Super-resolution enhanced medical image diagnosis with sample affinity interaction. IEEE Trans Med Imaging. 2021; 40:1377-1389. doi: 10.1109/TMI.2021.3055290

[37] Ho TKK, Gwak J. Utilizing knowledge distillation in deep learning for classification of chest X-ray abnormalities. IEEE Access. 2020; 8:160749-160761. doi: 10.1109/ACCESS.2020.3020802

[38] Li K, Yu L, Wang S, Heng PA. Towards cross-modality medical image segmentation with online mutual knowledge distillation. Proc AAAI Conf Art Intell. 2020; 34:775-783. doi: 10.1609/aaai.v34i01.5421

[39] Qin D, Bu JJ, Liu Z, et al. Efficient medical image segmentation based on knowledge distillation. IEEE Trans Med Imaging. 2021; 40:3820-3831. doi: 10.1109/TMI.2021.3098703

[40] Mangalam K, Salzamann M. On compressing u-net using knowledge distillation. arXiv preprint arXiv:1812.00249; 2018. doi: 10.48550/arXiv.1812.00249

[41] Owen JP, Blazes M, Manivannan N, et al. Student becomes teacher: Training faster deep learning lightweight networks for automated identification of optical coherence tomography b-scans of interest using a student-teacher framework. Biomed Opt Express. 2021; 12:5387-5399. doi: 10.1364/BOE.433432

[42] Vaze S, Xie W, Namburete AI. Low-memory CNNs enabling real-time ultrasound segmentation towards mobile deployment. IEEE J Biomed Health Inform. 2020; 24:1059-1069. doi: 10.1109/JBHI.2019.2961264

[43] Cao Z, Yang G, Chen Q, Chen X, Lv F. Breast tumor classification through learning from noisy labeled ultrasound images. Med Phys. 2020; 47:1048-1057. doi: 10.1002/mp.13966

[44] Lee K, Lee H, El Fakhri G, Woo J, Hwang JY. Self-Supervised Domain Adaptive Segmentation of Breast Cancer Via Test- Time Fine-Tuning. In: International Conference on Medical Image Computing and Computer Assisted Intervention. Springer; 2023. p. 539-550. doi: 10.1007/978-3-031-43907-0_52

[45] Ronneberger O, Fischer P, Brox T. U-net: Convolutional Networks for Biomedical Image Segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention. Springer; 2015. p. 234-241. doi: 10.1007/978-3-319-24574-4_28

[46] Fan J, Liu D, Chang H, Huang H, Chen M, Cai W. Taxonomy Adaptive Cross-domain adaptation in Medical Imaging Via Optimization Trajectory Distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2023. p. 21174-21184. doi: 10.48550/arXiv.2307.14709

[47] Yap MH, Goyal M, Osman F, et al. End-to-end breast ultrasound lesions recognition with a deep learning approach. In: Medical Imaging 2018: Biomedical Applications in Molecular, Structural, and Functional Imaging. In: Proceeding International Society for Optics and Photonics. Vol. 10578; 2018. p. 1057819. doi: 10.1117/12.2293498

[48] Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2015. p. 3431-3440. doi: 10.1109/CVPR.2015.7298965

[49] Abraham N, Khan NM. A Novel Focal Tversky Loss Function with Improved Attention u-net for Lesion Segmentation. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE; 2019. p. 683-687. doi: 10.1109/ISBI.2019.8759329

[50] Zhuang Z, Li N, Joseph Raj AN, Mahesh VG, Qiu S. An RDAU-net model for lesion segmentation in breast ultrasound images. PLoS One. 2019; 14:e0221535. doi: 10.1371/journal.pone.0221535

[51] Costa MGF, Campos JBM, de Aquino e Aquino G, de Albuquerque Pereira WC, Costa Filho CFF. Evaluating the performance of convolutional neural networks with direct acyclic graph architectures in automatic segmentation of breast lesion in us images. BMC Med Imaging. 2019; 19:1-13. doi: 10.1186/s12880-019-0389-2

[52] Liang Y, He R, Li Y, Wang Z. Simultaneous Segmentation and Classification of Breast Lesions from Ultrasound Images Using Mask R-CNN. In: 2019 IEEE International Ultrasonics Symposium (IUS). IEEE; 2019. p. 1470-1472. doi: 10.1109/ULTSYM.2019.8926185

[53] He K, Gkioxari G, Doll’ar P, Girshick R. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision; 2017. p. 2961-2969. doi: 10.48550/arXiv.1703.06870

[54] Amiri M, Brooks R, Behboodi B, Rivaz H. Two-stage ultrasound image segmentation using u-net and test time augmentation. Int J Comput Assist Radiol Surg. 2020; 15:981-988. doi: 10.1007/s11548-020-02158-3

[55] Lee H, Park J, Hwang JY. Channel attention module with multiscale grid average pooling for breast cancer segmentation in an ultrasound image. IEEE Trans Ultrason Ferroelectr Freq Control. 2020; 67:1344-1353. doi: 10.1109/TUFFC.2020.2972573

[56] Shareef B, Xian M, Vakanski A. Stan: Small Tumor-aware Network for Breast Ultrasound Image Segmentation. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE; 2020. p. 1-5. doi: 10.1109/ISBI45749.2020.9098691

[57] Shareef B, Vakanski A, Xian M, Freer PE. ESTAN: Enhanced small tumor-aware network for breast ultrasound image segmentation, arXiv preprint arXiv:2009.12894; 2020.

[58] Singh VK, Abdel-Nasser M, Akram F, et al. Breast tumor segmentation in ultrasound images using contextual-information-aware deep adversarial learning framework. Expert Syst Appl. 2020; 162:113870. doi: 10.1016/j.eswa.2020.113870

[59] Mirza M, Osindero S. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784; 2014. doi: 10.48550/arXiv.1411.1784

[60] Chen LC, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587; 2017. doi: 10.48550/arXiv.1706.05587

[61] Fu J, Liu J, Tian H, et al. ; Dual Attention Network for Scene Segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019. p. 3146-3154. doi: 10.1109/CVPR.2019.00326

[62] Hu J, Shen L, Sun G. Squeeze-and-Excitation Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018. p. 7132-7141. doi: 10.48550/arXiv.1709.01507

[63] Hussain S, Xi X, Ullah I, et al. Contextual level-set method for breast tumor segmentation. IEEE Access. 2020; 8:189343-189353. doi: 10.1109/ACCESS.2020.3029684

[64] Qu X, Shi Y, Hou Y, Jiang J. An attention-supervised full-resolution residual network for the segmentation of breast ultrasound images. Med Phys. 2020; 47:5702-5714. doi: 10.1002/mp.14470

[65] Pohlen T, Hermans A, Mathias M, Leibe B. Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017. p. 4151-4160. doi: 10.48550/arXiv.1611.08323

[66] Ning Z, Wang K, Zhong S, Feng Q, Zhang Y. Cf2-net: Coarse-to-fine fusion convolutional network for breast ultrasound image segmentation, arXiv preprint arXiv:2003. 10144; 2020. doi: 10.48550/arXiv.2003.10144

[67] Behboodi B, Amiri M, Brooks R, Rivaz H. Breast Lesion Segmentation in Ultrasound Images with Limited Annotated Data. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE; 2020. p. 1834-1837. doi: 10.1109/ISBI45749.2020.9098685

[68] Gao C, Ye H, Cao F, Wen C, Zhang Q, Zhang F. Multiscale fused network with additive channel-spatial attention for image segmentation. Knowl Based Syst. 2021; 214:106754. doi: 10.1016/j.knosys.2021.106754

[69] Su R, Zhang D, Liu J, Cheng C. MSU-Net: Multi-scale u-net for 2D medical image segmentation. Front Genet. 2021; 12:140. doi: 10.3389/fgene.2021.639930

[70] Xu M, Huang K, Chen Q, Qi X. MSSA-Net: Multi-scale Self-attention Network for Breast Ultrasound Image Segmentation. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE; 2021. p. 827-831. doi: 10.1109/ISBI48211.2021.9433899

[71] Huang H, Chen H, Xu H, et al. Cross-tissue/organ transfer learning for the segmentation of ultrasound images using deep residual u-net. J Med Biol Eng. 2021; 41:137-145. doi: 10.1007/s40846-020-00585-w

[72] Yeung M, Sala E, Schonlieb CB, Rundo L. Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Comput Med Imaging Graph. 2022; 95:102026. doi: 10.1016/j.compmedimag.2021.102026

[73] Xu C, Qi Y, Wang Y, Lou M, Pi J, Ma Y. ARF-Net: An adaptive receptive field network for breast mass segmentation in whole mammograms and ultrasound images. Biomed Signal Process Control. 2022; 17:103178. doi: 10.1016/j.bspc.2021.103178

[74] Lou M, Meng J, Qi Y, Li X, Ma Y. MCRNet: Multi-level context refinement network for semantic segmentation in breast ultrasound imaging. Neurocomputing. 2022; 470:154-169.

[75] Yang K, Suzuki A, Ye J, Nosato H, Izumori A, Sakanashi H. CTG-Net: Cross-task guided network for breast ultrasound diagnosis. PLoS One. 2022; 17:e0271106. doi: 10.1016/j.neucom.2021.10.102

[76] Xie S, Girshick R, Dollar P, Tu Z, He K. Aggregated Residual Transformations for Deep Neural Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017. p. 1492-1500. doi: 10.48550/arXiv.1611.05431

[77] Howard A, Sandler M, Chu G, et al.; Searching for Mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019. p. 1314-1324. doi: 10.48550/arXiv.1905.02244

[78] Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. Imagenet: A Large-Scale Hierarchical Image Database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE; 2009. p. 248-255. doi: 10.1109/CVPR.2009.5206848

[79] Iakubovskii P. Segmentation Models Pytorch; 2019. Available from: https://github.com/qubvel/segmentation_models.pytorch [Last accessed on 2024 Dec 19].

[80] Buslaev A, Iglovikov VI, Khvedchenya E, Parinov A, Druzhinin M, Kalinin AA. Albumentations: Fast and flexible image augmentations. Information. 2020; 11:125. doi: 10.3390/info11020125