DeepPolyp: an artificial intelligence framework for polyp detection and segmentation

Abstract

Aim: Colorectal cancer is a leading cause of cancer-related mortality, emphasising the need for accurate polyp segmentation during colonoscopy for early detection. Existing methods often struggle to generalize effectively across diverse clinical scenarios. This study introduces DeepPolyp, an artificial intelligence framework designed for comprehensive benchmarking and real-time clinical deployment of polyp segmentation models.

Methods: Transformer-based segmentation models, SegFormer and SSFormer, were trained from scratch using an extensive dataset comprising public collections (CVC-ClinicDB, ETIS-LaribPolypDB, Kvasir) and recently augmented datasets (PolypDataset-TCNoEndo, PolypGen). Training involved standardized data augmentation, learning rate schedules, and early stopping. Models were evaluated using Dice and Intersection over Union (IoU) metrics. Real-time inference performance was assessed on an NVIDIA Jetson Orin device with ONNX and TensorRT optimizations.

Results: SegFormer-B4 achieved the highest accuracy (Dice: 0.9843, IoU: 0.9694), but was not selected for clinical deployment due to computational constraints. SegFormer-B2 provided comparable accuracy (Dice: 0.9787, IoU: 0.9588) with significantly faster inference (94 ms per frame), offering an optimal balance suitable for real-time clinical use. SSFormer showed lower accuracy and slower inference, limiting its practical deployment.

Conclusions: DeepPolyp enables systematic evaluation of polyp segmentation models, assisting in selecting models based on both performance and computational efficiency. Despite superior accuracy from SegFormer-B4, SegFormer-B2 was selected for clinical deployment due to its advantageous balance between accuracy and real-time execution efficiency.

Keywords

Polyp segmentation / transformer models / deep learning / real-time inference / colonoscopy / edge computing

References

[1] Shalata W, Gluzman A, Man S, Cohen AY, Jama AA, Gothelf I, et al. Colorectal Cancer in Elderly Patients: Insights into Presentations, Prognosis, and Patient Outcomes. Medicina (Kaunas). 2024; 60:1951.

[2] Lopes SR, Martins C, Santos IC, Teixeira M, Gamito É, Alves AL. Colorectal cancer screening: A review of current knowledge and progress in research. World J Gastrointest Oncol. 2024; 16:1119-33.

[3] Luo Z, Dong X, Wang L, Zheng Y, Wang C, Xie J, et al. Potential reduction of global colorectal cancer, 1990-2021. J Natl Cancer Cent. 2025; 5:313-21.

[4] Jha D, Ali S, Tomar NK, Johansen HD, Johansen D, Rittscher J, et al. Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning. IEEE Access. 2021; 9:40496-510.

[5] Wu H, Zhao Z, Zhong J, Wang W, Wen Z, Qin J. PolypSeg+: A Lightweight Context-Aware Network for Real-Time Polyp Segmentation. IEEE Trans Cybern. 2023; 53:2610-21.

[6] Wickstrøm K, Kampffmeyer M, Jenssen R. Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps. Med Image Anal. 2020; 60:101619.

[7] Mei J, Zhou T, Huang K, Zhang Y, Zhou Y, Wu Y, et al. A survey on deep learning for polyp segmentation: techniques, challenges and future trends. Vis Intell. 2025; 3:1.

[8] Banik D, Roy K, Bhattacharjee D, Nasipuri M, Krejcar O. Polyp-Net: A Multimodel Fusion Network for Polyp Segmentation. IEEE Trans Instrum Meas. 2021; 70:1-12.

[9] Du Y, Jiang Y, Tan S, Liu S, Li Z, Li G, et al. Highlighted Diffusion Model as Plug-In Priors for Polyp Segmentation. IEEE J Biomed Health Inform. 2025; 29:1209-20.

[10] Nie M, An X, Xing Y, Wang Z, Wang Y, Lü J. Artificial intelligence algorithms for real-time detection of colorectal polyps during colonoscopy: a review. Am J Cancer Res. 2024; 14:5456-70.

[11] Park SY, Sargent D, Spofford I, Vosburgh KG, A-Rahim Y. A colon video analysis framework for polyp detection. IEEE Trans Biomed Eng. 2012; 59:1408-18.

[12] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N, Hornegger J, Wells W, Frangi A, editors. Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science. Cham: Springer; 2015. pp. 234-41.

[13] Jin Y, Hu Y, Jiang Z, Zheng Q. Polyp segmentation with convolutional MLP. Vis Comput. 2023; 39:4819-37.

[14] Cao X, Fan K, Xu C, Ma H, Jiao K. CMNet: deep learning model for colon polyp segmentation based on dual-branch structure. J Med Imaging (Bellingham). 2024; 11:024004.

[15] Hussain MS, Asgher U, Nisar S, Socha V, Shaukat A, Wang J, et al. Enhanced accuracy with Segmentation of Colorectal Polyp using NanoNetB, and Conditional Random Field Test-Time Augmentation. Front Robot AI. 2024; 11:1387491.

[16] Li K, Fathan MI, Patel K, Zhang T, Zhong C, Bansal A, et al. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations. PLoS One. 2021; 16:e0255809.

[17] Ali S, Jha D, Ghatwary N, Realdon S, Cannizzaro R, Salem OE, et al. A multi-centre polyp detection and segmentation dataset for generalisability assessment. Sci Data. 2023; 10:75.

[18] Wang J, Huang Q, Tang F, Meng J, Su J, Song S. Stepwise Feature Fusion: Local Guides Global. arXiv 03635 [Preprint].2022 [cited 2024 Feb 28]. Available from:https://arxiv.org/abs/2203.03635

[19] Xie E, Wang W, Yu Z, Anandkumar A, Alvarez JM, Luo P. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. Adv Neural Inf Process Syst. 2021; 34:12077-90.

[20] Bernal J, Sánchez FJ, Fernández-Esparrach G, Gil D, Rodríguez C, Vilariño F. WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Comput Med Imaging Graph. 2015; 43:99-111.

[21] Akbari M, Mohrekesh M, Nasr-Esfahani E, Soroushmehr SMR, Karimi N, Samavi S, et al. Polyp Segmentation in Colonoscopy Images Using Fully Convolutional Network. Annu Int Conf IEEE Eng Med Biol Soc. 2018; 2018:69-72.

[22] Silva J, Histace A, Romain O, Dray X, Granado B. Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. Int J Comput Assist Radiol Surg. 2014; 9:283-93.

[23] Jha D, Smedsrud PH, Riegler MA, Halvorsen P, de Lange T, Johansen D, et al. Kvasir-SEG: A Segmented Polyp Dataset. In: Ro YM, Kim J, Choi J, Cheng W, Chu W, Cui P, et al., editors. MultiMedia Modeling - MMM 2020 - Lecture Notes in Computer Science; 2020 Jan 5-8; Daejeon, South Korea. Cham: Springer; 2020. pp. 451-62.

[24] Ali S, Dmitrieva M, Ghatwary N, Bano S, Polat G, Temizel A, et al. Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy. Med Image Anal. 2021; 70:102002.

[25] Ali S, Ghatwary N, Jha D, Isik-Polat E, Polat G, Yang C, et al. Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge. Sci Rep. 2024; 14:2032.

[26] Mahmood T, Rehman A, Saba T, Nadeem L, Bahaj SAO. Recent Advancements and Future Prospects in Active Deep Learning for Medical Image Segmentation and Classification. IEEE Access. 2023; 11:113623-52.

[27] Fan D, Ji G, Zhou T, Chen G, Fu H, Shen J, et al. PraNet: Parallel Reverse Attention Network for Polyp Segmentation. In: Martel AL, Abolmaesumi P, Stoyanov D, Mateus D, Zuluaga MA, Zhou SK, editors. Medical Image Computing and Computer-Assisted Intervention - MICCAI 2020 - MICCAI 2020 - Lecture Notes in Computer Science; 2020 Oct 4-8; Lima, Peru. Cham: Springer; 2020. pp. 263-73.

[28] Jha D, Smedsrud PH, Riegler MA, Johansen D, De Lange T, Halvorsen P. ResUNet++: An Advanced Architecture for Medical Image Segmentation. 2019 IEEE International Symposium on Multimedia (ISM); 2019 Dec 9-11; San Diego, USA. IEEE; 2019. pp. 225-2255.

[29] Tomar NK, Jha D, Bagci U. DilatedSegNet: A Deep Dilated Segmentation Network for Polyp Segmentation. In: Dang-Nguyen D, Gurrin C, Smeaton AF, Larson M, Rudinac S, Dao M, et al., editors. MultiMedia Modeling - MMM 2023 - Lecture Notes in Computer Science; 2023 Jan 9-12; Bergen, Norway. Cham: Springer; 2023. pp. 334-44.

[30] Srivastava A, Jha D, Chanda S, Pal U, Johansen H, Johansen D, et al. MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation. IEEE J Biomed Health Inform. 2022; 26:2252-63.

[31] Huang C, Wu H, Lin Y. HarDNet-MSEG: A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS. arXiv 07172 [Preprint].2021 [cited 2024 Feb 27]. Available from:https://arxiv.org/abs/2101.07172

[32] Dong B, Wang W, Fan D, Li J, Fu H, Shao L. Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers. arXiv 06932 [Preprint].2021 [cited 2024 Feb 26]. Available from:https://arxiv.org/abs/2108.06932

[33] Sanderson E, Matuszewski BJ. FCN-Transformer Feature Fusion for Polyp Segmentation. In: Yang G, Aviles-Rivero A, Roberts M, Schönlieb CB, editors. Medical Image Understanding and Analysis - MIUA 2022 - Lecture Notes in Computer Science; 2022 Jul 27-29; Cambridge, UK. Cham: Springer; 2022. pp. 892-907.

[34] Zhang Y, Liu H, Hu Q. TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation. arXiv 08005 [Preprint].2021 [cited 2024 Jan 26]. Available from:https://arxiv.org/abs/2102.08005

[35] Dumitru R, Peteleaza D, Craciun C. Using DUCK-Net for polyp image segmentation. Sci Rep. 2023; 13:9803.

[36] Su Y, Cheng J, Zhong C, Zhang Y, Ye J, He J, et al. FeDNet: Feature Decoupled Network for polyp segmentation from endoscopy images. Biomed Signal Process Control. 2023; 83:104699.

[37] Zhang R, Lai P, Wan X, Fan D, Gao F, Wu X, et al. Lesion-aware Dynamic Kernel for Polyp Segmentation. arXiv 04904 [Preprint]. 2023 [cited 2024 Feb 2]. Available from:https://arxiv.org/abs/2301.04904

[38] Zhou T, Zhou Y, He K, Gong C, Yang J, Fu H, et al. Cross-level Feature Aggregation Network for Polyp Segmentation. Pattern Recognit. 2023; 140:109555.

[39] Qiu X, Wang Z, Zhang M, Xu Z, Fan J, Xu L. BDG-Net: Boundary Distribution Guided Network for Accurate Polyp Segmentation. arXiv 00767 [Preprint].2022 [cited 2024 Jan 10]. Available from:https://arxiv.org/abs/2201.00767

[40] Yin Z, Liang K, Ma Z, Guo J. Duplex Contextual Relation Network For Polyp Segmentation. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI); 2022 Mar 28-31; Kolkata, India. IEEE; 2022. pp. 1-5.

[41] Duc NT, Oanh NT, Thuy NT, Triet TM, Dinh VS. ColonFormer: An Efficient Transformer Based Method for Colon Polyp Segmentation. IEEE Access. 2022; 10:80575-86.

[42] Liao T, Yang C, Lo Y, Lai K, Shen P, Lin Y. HarDNet-DFUS: An Enhanced Harmonically-Connected Network for Diabetic Foot Ulcer Image Segmentation and Colonoscopy Polyp Segmentation. arXiv 07313 [Preprint].2022 [cited 2024 Jan 11]. Available from:https://arxiv.org/abs/2209.07313

[43] Tang F, Huang Q, Wang J, Hou X, Su J, Liu J. DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation. arXiv 11677 [Preprint].2022 [cited 2024 Jan 11]. Available from:https://arxiv.org/abs/2212.11677

[44] Zhang W, Fu C, Zheng Y, Zhang F, Zhao Y, Sham C. HSNet: A hybrid semantic network for polyp segmentation. Comput Biol Med. 2022; 150:106173.

[45] Kim T, Lee H, Kim D. UACANet: Uncertainty Augmented Context Attention for Polyp Segmentation. In: Proceedings of the 29th ACM International Conference on Multimedia; 2021 Oct 20-24; China. ACM; 2021. pp. 2167-75.

[46] Zhao X, Jia H, Pang Y, Lv L, Tian F, Zhang L, et al. M²SNet: Multi-scale in Multi-scale Subtraction Network for Medical Image Segmentation. arXiv 10894 [Preprint].2023 [cited 2024 Jan 24]. Available from:https://arxiv.org/abs/2303.10894

[47] Zhao X, Zhang L, Lu H. Automatic Polyp Segmentation via Multi-scale Subtraction Network. arXiv 05082 [Preprint].2021 [cited 2024 Jan 24]. Available from:https://arxiv.org/abs/2108.05082

[48] Wei J, Hu Y, Zhang R, Li Z, Zhou S, Gui S. Shallow Attention Network for Polyp Segmentation. arXiv 00882 [Preprint].2021 [cited 2024 Jan 24]. Available from:https://arxiv.org/abs/2108.00882

[49] Lou A, Guan S, Loew M. CaraNet: context axial reverse attention network for segmentation of small medical objects. J Med Imaging (Bellingham). 2023; 10:014005.

[50] Tomar NK, Jha D, Riegler MA, Johansen HD, Johansen D, Rittscher J, et al. FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation. IEEE Trans Neural Netw Learn Syst. 2023; 34:9375-88.

[51] Patel K, Bur AM, Wang G. Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentation. Proc Int Robot Vis Conf. 2021; 2021:181-8.

[52] Guo X, Yang C, Liu Y, Yuan Y. Learn to Threshold: ThresholdNet With Confidence-Guided Manifold Mixup for Polyp Segmentation. IEEE Trans Med Imaging. 2021; 40:1134-46.

[53] Tajbakhsh N, Gurudu SR, Liang J. Automated Polyp Detection in Colonoscopy Videos Using Shape and Context Information. IEEE Trans Med Imaging. 2016; 35:630-44.