ChromTR: chromosome detection in raw metaphase cell images via deformable transformers

Abstract

Chromosome karyotyping is a critical way to diagnose various hematological malignancies and genetic diseases, of which chromosome detection in raw metaphase cell images is the most critical and challenging step. In this work, focusing on the joint optimization of chromosome localization and classification, we propose ChromTR to accurately detect and classify 24 classes of chromosomes in raw metaphase cell images. ChromTR incorporates semantic feature learning and class distribution learning into a unified DETR-based detection framework. Specifically, we first propose a Semantic Feature Learning Network (SFLN) for semantic feature extraction and chromosome foreground region segmentation with object-wise supervision. Next, we construct a Semantic-Aware Transformer (SAT) with two parallel encoders and a Semantic-Aware decoder to integrate global visual and semantic features. To provide a prediction with a precise chromosome number and category distribution, a Category Distribution Reasoning Module (CDRM) is built for foreground–background objects and chromosome class distribution reasoning. We evaluate ChromTR on 1404 newly collected R-band metaphase images and the public G-band dataset AutoKary2022. Our proposed ChromTR outperforms all previous chromosome detection methods with an average precision of 92.56% in R-band chromosome detection, surpassing the baseline method by 3.02%. In a clinical test, ChromTR is also confident in tackling normal and numerically abnormal karyotypes. When extended to the chromosome enumeration task, ChromTR also demonstrates state-of-the-art performances on R-band and G-band two metaphase image datasets. Given these superior performances to other methods, our proposed method has been applied to assist clinical karyotype diagnosis.

Keywords

chromosome detection / deformable transformer / metaphase cell image / distribution reasoning

References

[1] GersenSL. The Principles of Clinical Cytogenetics. Springer, 2013

[2] Patterson D. Molecular genetic analysis of Down syndrome. Hum Genet 2009; 126(1): 195–214

[3] Craig JM, Bickmore WA. Genes and genomes: chromosome bands–flavours to savour. BioEssays 1993; 15(5): 349–354

[4] Rack K, Van den Berg E, Haferlach C, Beverloo HB, Costa D, Espinet B, Foot N, Jeffries S, Martin K, O’Connor S, Schoumans J, Talley P, Telford N, Stioui S, Zemanova Z, Hastings RJ. European recommendations and quality assurance for cytogenomic analysis of haematological neoplasms: reponse to the comments from the francophone group of hematological cytogenetics (gfch). Leukemia 2020; 34(8): 2262–2264

[5] McGowan-JordanJSimonsASchmidM. ISCN 2016: an international system for human cytogenomic nomenclature (2016). 2016

[6] Ganguly BB, Shouvik M, Kadam NN, Banerjee D, Agarwal M. Experience of conventional cytogenetics in elderly cytopenic indian patients suspected with myelodysplastic syndromes. Blood 2016; 128(22): 5488–5488

[7] MetaSystems. Ikaros. 2024. Available at the website of metasystems-international.com/en/products/ikaros/

[8] Marková J, Michková P, Burčková K, Březinová J, Michalová K, Dohnalová A, Maaloufová JS, Soukup P, Vítek A, Cetkovský P, Schwarz J. Prognostic impact of DNMT3A mutations in patients with intermediate cytogenetic risk profile acute myeloid leukemia. Eur J Haematol 2012; 88(2): 128–135

[9] Havelka M, Bytyutskyy D, Symonová R, Ráb P, Flajšhans M. The second highest chromosome count among vertebrates is observed in cultured sturgeon and is associated with genome plasticity. Genet Sel Evol 2016; 48(1): 12

[10] Priya PK, Mishra VV, Roy P, Patel H. A study on balanced chromosomal translocations in couples with recurrent pregnancy loss. J Hum Reprod Sci 2018; 11(4): 337

[11] Krishna Chandran R, Geetha N, Sakthivel KM, Suresh Kumar R, Jagathnath Krishna KMN, Sreedharan H. Impact of additional chromosomal aberrations on the disease progression of chronic myelogenous leukemia. Front Oncol 2019; 9: 88

[12] Paulis M, Susani L, Castelli A, Suzuki T, Hara T, Straniero L, Duga S, Strina D, Mantero S, Caldana E, Sergi LS, Villa A, Vezzoni P. Chromosome transplantation: a possible approach to treat human X-linked disorders. Mol Ther Methods Clin Dev 2020; 17: 369–377

[13] Kurtovic-Kozaric A, Mehinovic L, Malesevic R, Mesanovic S, Jaros T, Stomornjak-Vukadin M, Mackic-Djurovic M, Ibrulj S, Kurtovic-Basic I, Kozaric M. Ten-year trends in prevalence of down syndrome in a developing country: impact of the maternal age and prenatal screening. Eur J Obstet Gynecol Reprod Biol 2016; 206: 79–83

[14] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998; 86(11): 2278–2324

[15] GravesA. Long short-term memory. In: Supervised Sequence Labelling with Recurrent Neural Networks. Springer. 2012. 37–45

[16] VaswaniAShazeer NParmarNUszkoreitJJonesL GomezANKaiser LPolosukhinI. Attention is all you need. In: Advances in Neural Information Processing Systems. Vol 30 (NIPS 2017). 2017

[17] Gomes B, Ashley EA. Artificial intelligence in molecular medicine. N Engl J Med 2023; 388(26): 2456–2465

[18] MaoWZhuM SunZShenS WuLYChenH ShenC. De novo protein design using geometric vector field networks. 2023. arXiv: 2310.11802

[19] Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, Mahendiran T, Moraes G, Shamdas M, Kern C, Ledsam JR, Schmid MK, Balaskas K, Topol EJ, Bachmann LM, Keane PA, Denniston AK. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health 2019; 1(6): e271–e297

[20] Lin Z, Zhang D, Shi D, Xu R, Tao Q, Wu L, He M, Ge Z. Contrastive pre-training and linear interaction attention-based transformer for universal medical reports generation. J Biomed Inform 2023; 138: 104281

[21] Rajpurkar P, Lungren MP. The current and future state of AI interpretation of medical images. N Engl J Med 2023; 388(21): 1981–1990

[22] Al-Kharraz MS, Elrefaei LA, Fadel MA. Automated system for chromosome karyotyping to recognize the most common numerical abnormalities using deep learning. IEEE Access 2020; 8: 157727–157747

[23] Xiao L, Luo C, Yu T, Luo Y, Wang M, Yu F, Li Y, Tian C, Qiao J. Deepacev2: automated chromosome enumeration in metaphase cell images using deep convolutional neural networks. IEEE Trans Med Imaging 2020; 39(12): 3920–3932

[24] Wang J, Zhou C, Chen S, Hu J, Wu M, Jiang X, Xu C, Qian D. Chromosome detection in metaphase cell images using morphological priors. IEEE J Biomed Health Inform 2023; 27(9): 4579–4590

[25] XiaoLLuo CLuoYYuTTianC QiaoJZhao Y. Deepace: automated chromosome enumeration in metaphase cell images using deep convolutional neural networks. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Cham: Springer International Publishing, 2019. 595–603

[26] Bai H, Zhang T, Lu C, Chen W, Xu F, Han ZB. Chromosome extraction based on u-net and yolov3. IEEE Access 2020; 8: 178563–178569

[27] LuoCYuT LuoYWangM YuFLiY TianCQiao JXiaoL. Deepacc: automate chromosome classification based on metaphase images using deep learning framework fused with prior knowledge. 2020. arXiv: 2006.15528

[28] Tseng JJ, Lu CH, Li JZ, Lai HY, Chen MH, Cheng FY, Kuo CE. An open dataset of annotated metaphase cell images for chromosome identification. Sci Data 2023; 10(1): 104

[29] DingWChang LGuCWuK. Classification of chromosome karyotype based on faster-rcnn with the segmatation and enhancement preprocessing model. In: 2019 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISPBMEI). IEEE. 2019. 1–5

[30] YouDXiaP ChenQWu MXiangSWangJ. AutoKary2022: A Large-Scale Densely Annotated Dataset for Chromosome Instance Segmentation. 2023 IEEE International Conference on Multimedia and Expo (ICME), Brisbane, Australia, 2023. 1577–1582

[31] Zhou R, Yu L, Chen D, Zhang H, Szczerbicki E. Kemr-net: a knowledge-enhanced mask refinement network for chromosome instance segmentation. Cybern Syst 2023; 55(3): 708–718

[32] WangPHu WZhangJWenYXuC QianD. Enhanced rotated mask r-cnn for chromosome segmentation. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE. 2021. 2769–2772

[33] Xie N, Li X, Li K, Yang Y, Shen HT. Statistical karyotype analysis using cnn and geometric optimization. IEEE Access 2019; 7: 179445–179453

[34] PijackovaKGotthans TGotthansJ. Deep learning pipeline for chromosome segmentation. In: 2022 32nd International Conference Radioelektronika (RADIOELEKTRONIKA). IEEE. 2022. 1–5

[35] Huang R, Lin C, Yin A, Chen H, Guo L, Zhao G, Fan X, Li S, Yang J. A clinical dataset and various baselines for chromosome instance segmentation. IEEE/ACM Trans Comput Biol Bioinformatics 2022; 19(1): 31–39

[36] Liu H, Wang G, Song S, Huang D, Zhang L. Rc-net: regression correction for end-to-end chromosome instance segmentation. Front Genet 2022; 13: 895099

[37] MarcCNCzibula G. Karysom: an unsupervised learning based approach for human karyotyping using self-organizing maps. In: 2018 IEEE 14th International Conference on Intelligent Computer Communication and Processing (ICCP). IEEE. 2018. 167–174

[38] LinCZhaoG YinADingB GuoLChenH. A multistages chromosome segmentation and mixed classification method for chromosome automatic karyotyping. In: International Conference on Web Information Systems and Applications. Springer. 2020. 365–376

[39] WuYYueY TanXWangW LuT. End-to-end chromosome karyotyping with data augmentation using gan. In: 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE. 2018. 2456–2460

[40] Wu Y, Tan X, Lu T. A new multiple-distribution gan model to solve complexity in end-to-end chromosome karyotyping. Complexity 2020; 2020: 8923838

[41] Andrade MF, Dias LV, Macario V, Lima FF, Hwang SF, Silva JC, Cordeiro FR. A study of deep learning approaches for classification and detection chromosomes in metaphase images. Mach Vis Appl 2020; 31: 65

[42] DoughertyAWYou J. A kernel-based adaptive fuzzy c-means algorithm for m-fish image segmentation. In: 2017 International Joint Conference on Neural Networks (IJCNN). IEEE. 2017. 198–205

[43] Song S, Bai T, Zhao Y, Zhang W, Yang C, Meng J, Ma F, Su J. A new convolutional neural network architecture for automatic segmentation of overlapping human chromosomes. Neural Process Lett 2022; 54: 285–301

[44] HuangKLin CHuangRZhaoGYinA ChenHGuo LShanCNieRLiS. A novel chromosome instance segmentation method based on geometry and deep learning. In: 2021 International Joint Conference on Neural Networks (IJCNN). IEEE. 2021. 1–8

[45] Mei L, Yu Y, Shen H, Weng Y, Liu Y, Wang D, Liu S, Zhou F, Lei C. Adversarial multiscale feature learning framework for overlapping chromosome segmentation. Entropy (Basel) 2022; 24(4): 522

[46] Liu X, Wang S, Lin JCW, Liu S. An algorithm for overlapping chromosome segmentation based on region selection. Neural Comput Appl 2024; 36: 133–142

[47] Wang G, Liu H, Yi X, Zhou J, Zhang L. Arms net: overlapping chromosome segmentation based on adaptive receptive field multiscale network. Biomed Signal Process Control 2021; 68: 102811

[48] ChenPCai JYangL. Chromosome segmentation via data simulation and shape learning. In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE. 2020. 1637–1640

[49] Chen X, Cai Q, Ma N, Li H. Chrosegnet: an attentionbased model for chromosome segmentation with enhanced processing. Appl Sci (Basel) 2023; 13(4): 2308

[50] CaoXLanF LiuCMLam TWLuoR. Chromseg: twostage framework for overlapping chromosome segmentation and reconstruction. In: 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE. 2020. 2335–2342

[51] Altinsoy E, Yang J, Yilmaz C. Fully-automatic raw G-band chromosome image segmentation. IET Image Process 2020; 14(9): 1920–1928

[52] HuRLKarnowski JFadelyRPommierJP. Image segmentation to distinguish between overlapping human chromosomes. 2017; arXiv: 1712.07639

[53] YilmazICYang JAltinsoyEZhouL. An improved segmentation for raw g-band chromosome images. In: 2018 5th International Conference on Systems and Informatics (ICSAI). IEEE. 2018. 944–950

[54] RedmonJFarhadi A. Yolo9000: better, faster, stronger. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 7263–7271

[55] RenSHeK GirshickRSun J. Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems. Vol 28 (NIPS 2015). 2015

[56] HeKGkioxariG DollarPGirshick R. Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision. 2017. 2961–2969

[57] LinTYDollar PGirshickRHeKHariharan BBelongieS. Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 2117–2125

[58] ZhuXSuW LuLLiB WangXDai J. deformable detr: deformable transformers for end-to-end object detection. 2020; arXiv: 2010.04159

[59] Qin Y, Wen J, Zheng H, Huang X, Yang J, Song N, Zhu YM, Wu L, Yang GZ. Varifocal-net: a chromosome classification approach using deep convolutional networks. IEEE Trans Med Imaging 2019; 38(11): 2569–2581

[60] XiaCWangJ QinYGuY ChenBYang J. An end-to-end combinatorial optimization method for r-band chromosome recognition with grouping guided attention. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022. 3–13

[61] Xia C, Wang J, Qin Y, Wen J, Liu Z, Song N, Wu L, Chen B, Gu Y, Yang J. Karyonet: chromosome recognition with end-toend combinatorial optimization network. IEEE Trans Med Imaging 2023; 42(10): 2899–2911

[62] Kuhn HW. The hungarian method for the assignment problem. Nav Res Logist Q 1955; 2(1–2): 83–97

[63] LinTYGoyal PGirshickRHeKDollarP. Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision. 2017. 2980–2988

[64] RezatofighiHTsoiNGwakJ SadeghianAReid ISavareseS. Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. 658–666

[65] Russell BC, Torralba A, Murphy KP, Freeman WT. Labelme: a database and web-based tool for image annotation. Int J Comput Vis 2008; 77(1–3): 157–173

[66] PaszkeAGross SMassaFLererABradburyJ ChananGKilleen TLinZGimelsheinNAntigaL . Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems. 2019. Vol 32

[67] CarionNMassa FSynnaeveGUsunierNKirillovA ZagoruykoS. End-to-end object detection with transformers. In: European Conference on Computer Vision. Springer. 2020. 213–229

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (No. 81670137), SJTU Trans-med Awards Research (No. 20220102), and State Key Laboratory of Medical Genomics Support (No. 201802)

Compliance with ethics guidelines

Conflict of interests Chao Xia, Jiyue Wang, Xin You, Yaling Fan, Bing Chen, and Jie Yang declare that they have no conflict of interest. Saijuan Chen is one of Editors-in-Chief of Frontiers of Medicine, and she was excluded from the peer-review process and all editorial decisions related to the acceptance and publication of this article. Peer-review was handled independently by the other editors to minimise bias.

This study does not involve a research protocol that requires the approval of relevant institutional review board or ethics committee.

RIGHTS & PERMISSIONS

2024 Higher Education Press