Deep learning in digital pathology image analysis: a survey

Abstract

deep learning (DL) has achieved state-of-the-art performance in many digital pathology analysis tasks. Traditional methods usually require hand-crafted domain-specific features, and DL methods can learn representations without manually designed features. In terms of feature extraction, DL approaches are less labor intensive compared with conventional machine learning methods. In this paper, we comprehensively summarize recent DL-based image analysis studies in histopathology, including different tasks (e.g., classification, semantic segmentation, detection, and instance segmentation) and various applications (e.g., stain normalization, cell/gland/region structure analysis). DL methods can provide consistent and accurate outcomes. DL is a promising tool to assist pathologists in clinical diagnosis.

Keywords

pathology / deep learning / segmentation / detection / classification

References

[1] Cardesa A, Zidar N, Alos L, Nadal A, Gale N, Klöppel G. The Kaiser’s cancer revisited: was Virchow totally wrong? Virchows Arch 2011; 458(6): 649–657

[2] Gurcan MN, Boucheron LE, Can A, Madabhushi A, Rajpoot NM, Yener B. Histopathological image analysis: a review. IEEE Rev Biomed Eng 2009; 2: 147–171

[3] Andrion A, Magnani C, Betta PG, Donna A, Mollo F, Scelsi M, Bernardi P, Botta M, Terracini B. Malignant mesothelioma of the pleura: interobserver variability. J Clin Pathol 1995; 48(9): 856–860

[4] Wolberg WH, Street WN, Heisey DM, Mangasarian OL. Computer-derived nuclear features distinguish malignant from benign breast cytology. Hum Pathol 1995; 26(7): 792–796

[5] Thiran JP, Macq B. Morphological feature extraction for the classification of digital images of cancerous tissues. IEEE Trans Biomed Eng 1996; 43(10): 1011–1020

[6] Choi HK, Jarkrans T, Bengtsson E, Vasko J, Wester K, Malmström PU, Busch C. Image analysis based grading of bladder carcinoma. Comparison of object, texture and graph based methods and their reproducibility. Anal Cell Pathol 1997; 15(1): 1–18

[7] Hamilton PW, Bartels PH, Thompson D, Anderson NH, Montironi R, Sloan JM. Automated location of dysplastic fields in colorectal histology using image texture analysis. J Pathol 1997; 182(1): 68–75

[8] Esgiar AN, Naguib RN, Sharif BS, Bennett MK, Murray A. Fractal analysis in the detection of colonic cancer images. IEEE Trans Inf Technol Biomed 2002; 6(1): 54–58

[9] Spyridonos P, Ravazoula P, Cavouras D, Berberidis K, Nikiforidis G. Computer-based grading of haematoxylin-eosin stained tissue sections of urinary bladder carcinomas. Med Inform Internet Med 2001; 26(3): 179–190

[10] Wiltgen M, Gerger A, Smolle J. Tissue counter analysis of benign common nevi and malignant melanoma. Int J Med Inform 2003; 69(1): 17–28

[11] Nielsen B, Albregtsen F, Danielsen HE. The use of fractal features from the periphery of cell nuclei as a classification tool. Anal Cell Pathol 1999; 19(1): 21–37

[12] Esgiar AN, Naguib RN, Sharif BS, Bennett MK, Murray A. Microscopic image analysis for quantitative measurement and feature identification of normal and cancerous colonic mucosa. IEEE Trans Inf Technol Biomed 1998; 2(3): 197–203

[13] Weyn B, van de Wouwer G, Kumar-Singh S, van Daele A, Scheunders P, van Marck E, Jacob W. Computer-assisted differential diagnosis of malignant mesothelioma based on syntactic structure analysis. Cytometry 1999; 35(1): 23–29

[14] Keenan SJ, Diamond J, McCluggage WG, Bharucha H, Thompson D, Bartels PH, Hamilton PW. An automated machine vision system for the histological grading of cervical intraepithelial neoplasia (CIN). J Pathol 2000; 192(3): 351–362

[15] Demir C, Gultekin SH, Yener B. Learning the topological properties of brain tumors. IEEE/ACM Trans Comput Biol Bioinformatics 2005; 2(3): 262–270

[16] Gunduz-Demir C. Mathematical modeling of the malignancy of cancer using graph evolution. Math Biosci 2007; 209(2): 514–527

[17] Weinstein RS, Graham AR, Richter LC, Barker GP, Krupinski EA, Lopez AM, Erps KA, Bhattacharyya AK, Yagi Y, Gilbertson JR. Overview of telepathology, virtual microscopy, and whole slide imaging: prospects for the future. Hum Pathol 2009; 40(8): 1057–1069

[18] Krupinski EA, Bhattacharyya AK, Weinstein RS. Telepathology and Digital Pathology Research. Springer, Cham. 2016: 41–54

[19] Farahani N, Parwani AV, Pantanowitz L. Whole slide imaging in pathology: advantages, limitations, and emerging perspectives. Pathol Lab Med Int 2015; 7: 23–33

[20] Ying X, Monticello TM. Modern imaging technologies in toxicologic pathology: an overview. Toxicol Pathol 2006; 34(7): 815–826

[21] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 521(7553): 436–444

[22] LeCun Y, Boser BE, Denker JS, Henderson D, Howard RE, Hubbard WE, Jackel LD. Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems. 1990: 396–404

[23] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE 1998; 86(11): 2278–2324

[24] Wang P, Ge R, Xiao X, Cai Y, Wang G, Zhou F. Rectified-linear-unit-based deep learning for biomedical multi-label data. Interdiscip Sci 2017; 9(3): 419–422

[25] Madabhushi A, Lee G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med Image Anal 2016; 33: 170–175

[26] Epstein JI. An update of the Gleason grading system. J Urol 2010; 183(2): 433–440

[27] Frierson HF Jr, Wolber RA, Berean KW, Franquemont DW, Gaffey MJ, Boyd JC, Wilbur DC. Interobserver reproducibility of the Nottingham modification of the Bloom and Richardson histologic grading scheme for infiltrating ductal carcinoma. Am J Clin Pathol 1995; 103(2): 195–198

[28] Khan AM, Rajpoot N, Treanor D, Magee D. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans Biomed Eng 2014; 61(6): 1729–1738

[29] Vahadane A, Peng T, Sethi A, Albarqouni S, Wang L, Baust M, Steiger K, Schlitter AM, Esposito I, Navab N. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans Med Imaging 2016; 35(8): 1962–1971

[30] Janowczyk A, Basavanhally A, Madabhushi A. Stain normalization using Sparse AutoEncoders (StaNoSA): application to digital pathology. Comput Med Imaging Graph 2017; 57: 50–61

[31] Bentaieb A, Hamarneh G. Adversarial stain transfer for histopathology image analysis. IEEE Trans Med Imaging 2018; 37(3): 792–802

[32] Roy S, Kumar Jain A, Lal S, Kini J. A study about color normalization methods for histopathology images. Micron 2018; 114: 42–61

[33] Elston CW, Ellis IO. Author Commentary: “Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up. C. W. Elston & I. O. Ellis. Histopathology 1991; 19; 403–410.” Histopathology 2002; 41(3A): 151

[34] Chow KH, Factor RE, Ullman KS. The nuclear envelope environment and its cancer connections. Nat Rev Cancer 2012; 12(3): 196–209

[35] Dey P. Cancer nucleus: morphology and beyond. Diagn Cytopathol 2010; 38(5): 382–390

[36] Chen H, Wang X, Heng PA. Automated mitosis detection with deep regression networks. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE. 2016: 1204–1207

[37] Cireşan D, Giusti A, Gambardella LM, Schmidhuber J. Deep neural networks segment neuronal membranes in electron microscopy images. Adv Neural Inf Process Syst 2012: 2843–2851

[38] Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2015: 234–241

[39] Zhou Y, Chang H, Barner KE, Parvin B. Nuclei segmentation via sparsity constrained convolutional regression. Proc IEEE Int Symp Biomed Imaging 2015; 2015: 1284–1287

[40] Song Y, Zhang L, Chen S, Ni D, Lei B, Wang T. Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning. IEEE Trans Biomed Eng 2015; 62(10): 2421–2433

[41] Xing F, Xie Y, Yang L. An automatic learning-based framework for robust nucleus segmentation. IEEE Trans Med Imaging 2016; 35(2): 550–566

[42] Pan X, Li L, Yang H, Liu Z, Yang J, Zhao L, Fan Y. Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing 2016; 229: S0925231216313765

[43] Zhang L, Sonka M, Lu L, Summers RM, Yao J. Combining fully convolutional networks and graph-based approach for automated segmentation of cervical cell nuclei. 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). IEEE. 2017: 406–409

[44] Alom MZ, Yakopcic C, Taha TM, Asari VK. Nuclei segmentation with recurrent residual convolutional neural networks based U-Net (R2U-Net). NAECON 2018-IEEE National Aerospace and Electronics Conference. IEEE. 2018: 228–233

[45] Xie Y, Xing F, Kong X, Su H, Yang L. Beyond classification: structured regression for robust cell detection using convolutional neural network. Med Image Comput Comput Assist Interv 2015; 9351: 358–365

[46] Sirinukunwattana K, Ahmed Raza SE, Tsang YW, Snead DRJ, Cree IA, Rajpoot NM. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 2016; 35(5): 1196–1206

[47] Xu J, Xiang L, Liu Q, Gilmore H, Wu J, Tang J, Madabhushi A. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Trans Med Imaging 2016; 35(1): 119–130

[48] Xie Y, Xing F, Shi X, Kong X, Su H, Yang L. Efficient and robust cell detection: a structured regression approach. Med Image Anal 2018; 44: 245–254

[49] Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J. Mitosis detection in breast cancer histology images with deep neural networks. Med Image Comput Comput Assist Interv 2013; 16(Pt 2): 411–418

[50] Wang H, Cruz-Roa A, Basavanhally A, Gilmore H, Shih N, Feldman M, Tomaszewski J, Gonzalez F, Madabhushi A. Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. J Med Imaging (Bellingham) 2014; 1(3): 034003

[51] Chen H, Dou Q, Wang X, Qin J, Heng PA. Mitosis detection in breast cancer histology images via deep cascaded networks. Thirtieth AAAI Conference on Artificial Intelligence. 2016

[52] Albarqouni S, Baur C, Achilles F, Belagiannis V, Demirci S, Navab N. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans Med Imaging 2016; 35(5): 1313–1321

[53] Li C, Wang X, Liu W, Latecki LJ. DeepMitosis: mitosis detection via deep detection, verification and segmentation networks. Med Image Anal 2018; 45: 121–133

[54] Ma M, Shi Y, Li W, Gao Y, Xu J. A novel two-stage deep method for mitosis detection in breast cancer histology images. 2018 24th International Conference on Pattern Recognition (ICPR). IEEE. 2018: 3892–3897

[55] Li C, Wang X, Liu W, Latecki LJ, Wang B, Huang J. Weakly supervised mitosis detection in breast histopathology images using concentric loss. Med Image Anal 2019; 53: 165–178

[56] Yang L, Zhang Y, Guldner IH, Zhang S, Chen DZ. 3D segmentation of glial cells using fully convolutional networks and k-terminal cut. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 658–666

[57] Akram SU, Kannala J, Eklund L, Heikkilä J. Cell proposal network for microscopy image analysis. 2016 IEEE International Conference on Image Processing (ICIP). IEEE. 2016: 3199–3203

[58] Akram SU, Kannala J, Eklund L, Heikkilä J. Cell segmentation proposal network for microscopy image analysis. Deep Learning and Data Labeling for Medical Applications. Springer, Cham. 2016: 21–29

[59] Chen H, Qi X, Yu L, Dou Q, Qin J, Heng PA. DCAN: deep contour-aware networks for object instance segmentation from histology images. Med Image Anal 2017; 36: 135–146

[60] Song Y, Tan EL, Jiang X, Cheng JZ, Ni D, Chen S, Lei B, Wang T. Accurate cervical cell segmentation from overlapping clumps in pap smear images. IEEE Trans Med Imaging 2017; 36(1): 288–300

[61] Kumar N, Verma R, Sharma S, Bhargava S, Vahadane A, Sethi A. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans Med Imaging 2017; 36(7): 1550–1560

[62] Ho DJ, Fu C, Salama P, Dunn KW, Delp EJ. Nuclei detection and segmentation of fluorescence microscopy images using three dimensional convolutional neural networks. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE. 2018: 418–422

[63] Naylor P, Laé M, Reyal F, Walter T. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Trans Med Imaging 2019; 38(2): 448–459

[64] Zhou Y, Onder OF, Dou Q, Tsougenis E, Chen H, Heng PA. Cia-net: Robust nuclei instance segmentation with contour-aware information aggregation. International Conference on Information Processing in Medical Imaging. Springer, Cham. 2019: 682–693

[65] Arganda-Carreras I, Seung HS, Cardona A, Schindelin J. Segmentation of neuronal structures in EM stacks challenge–ISBI 2012. 2012

[66] Oren A, Fernandes J. The Bethesda system for the reporting of cervical/vaginal cytology. J Am Osteopath Assoc 1991; 91(5): 476–479

[67] Naik S, Doyle S, Feldman M, Tomaszewski J, Madabhushi A. Gland segmentation and computerized gleason grading of prostate histology by integrating low-, high-level and domain specific information. MIAAB workshop. 2007: 1–8

[68] Karvelis PS, Fotiadis DI, Georgiou I, Syrrou M. A watershed based segmentation method for multispectral chromosome images classification. Conf Proc IEEE Eng Med Biol Soc 2006; 2006: 3009–3012

[69] Petushi S, Garcia FU, Haber MM, Katsinis C, Tozeren A. Large-scale computations on histology images reveal grade-differentiating parameters for breast cancer. BMC Med Imaging 2006; 6(1): 14

[70] Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 2017; 39(4): 640–651

[71] Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B, Rueckert D. Attention U-net: learning where to look for the pancreas. arXiv 2018; 1804.03999

[72] Zeng Z, Xie W, Zhang Y, Lu Y. RIC-Unet: an improved neural network based on Unet for nuclei segmentation in histology images. IEEE Access 2019; 7: 21420–21428

[73] Chen JM, Li Y, Xu J, Gong L, Wang LW, Liu WL, Liu J. Computer-aided prognosis on breast cancer with hematoxylin and eosin histopathology images: a review. Tumour Biol 2017; 39(3): 1010428317694550

[74] Veta M, Pluim JP, van Diest PJ, Viergever MA. Breast cancer histopathology image analysis: a review. IEEE Trans Biomed Eng 2014; 61(5): 1400–1411

[75] Sommer C, Fiaschi L, Hamprecht FA, Gerlich DW. Learning-based mitotic cell detection in histopathological images. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE. 2012: 2306–2309

[76] Veta M, van Diest PJ, Pluim JPW. Detecting mitotic figures in breast cancer histopathology images. Medical Imaging 2013: Digital Pathology. International Society for Optics and Photonics. 2013; 8676: 867607

[77] Khan AM, Eldaly H, Rajpoot NM. A gamma-Gaussian mixture model for detection of mitotic cells in breast cancer histopathology images. J Pathol Inform 2013; 4(4): 149–152

[78] Paul A, Dey A, Mukherjee DP, Sivaswamy J, Tourani V. Regenerative random forest with automatic feature selection to detect mitosis in histopathological breast cancer images. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2015: 94–102

[79] Cruz-Roa AA, Arevalo Ovalle JE, Madabhushi A, González Osorio FA. A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. Med Image Comput Comput Assist Interv 2013; 16(Pt 2): 403–410

[80] Veta M, van Diest PJ, Willems SM, Wang H, Madabhushi A, Cruz-Roa A, Gonzalez F, Larsen AB, Vestergaard JS, Dahl AB, Cireşan DC, Schmidhuber J, Giusti A, Gambardella LM, Tek FB, Walter T, Wang CW, Kondo S, Matuszewski BJ, Precioso F, Snell V, Kittler J, de Campos TE, Khan AM, Rajpoot NM, Arkoumani E, Lacle MM, Viergever MA, Pluim JP. Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 2015; 20(1): 237–248

[81] Roux L, Racoceanu D, Loménie N, Kulikova M, Irshad H, Klossa J, Capron F, Genestie C, Naour GL, Gurcan MN. Mitosis detection in breast cancer histological images. An ICPR 2012 contest. J Pathol Inform 2013; 4: 8

[82] Yang F, Mackey MA, Ianzini F, Gallardo G, Sonka M. Cell segmentation, tracking, and mitosis detection using temporal context. Med Image Comput Comput Assist Interv 2005; 8(Pt 1): 302–309

[83] Payer C, Štern D, Feiner M, Bischof H, Urschler M. Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks. Med Image Anal 2019; 57: 106–119

[84] Hariharan B, Arbeláez P, Girshick R, Malik J. Simultaneous detection and segmentation. European Conference on Computer Vision. Springer, Cham. 2014: 297–312

[85] Veta M, Van Diest PJ, Pluim JPW. Cutting out the middleman: measuring nuclear area in histopathology slides without segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 632–639

[86] Kronqvist P, Kuopio T, Collan Y. Morphometric grading of invasive ductal breast cancer. I. Thresholds for nuclear grade. Br J Cancer 1998; 78(6): 800–805

[87] Mommers EC, Page DL, Dupont WD, Schuyler P, Leonhart AM, Baak JP, Meijer CJ, van Diest PJ. Prognostic value of morphometry in patients with normal breast tissue or usual ductal hyperplasia of the breast. Int J Cancer 2001; 95(5): 282–285

[88] Veta M, Kornegoor R, Huisman A, Verschuur-Maes AH, Viergever MA, Pluim JP, van Diest PJ. Prognostic value of automatically extracted nuclear morphometric features in whole slide images of male breast cancer. Mod Pathol 2012; 25(12): 1559–1565

[89] Maška M, Ulman V, Svoboda D, Matula P, Matula P, Ederra C, Urbiola A, España T, Venkatesan S, Balak DMW, Karas P, Bolcková T, Štreitová M, Carthel C, Coraluppi S, Harder N, Rohr K, Magnusson KEG, Jaldén J, Blau HM, Dzyubachyk O, Křížek P, Hagen GM, Escuredo DP, Carretero DJ, Carbayo MJL, Barrutia AM, Meijering E, Kozubek M, Solorzano CO. A benchmark for comparison of cell tracking algorithms. Bioinformatics 2014; 30(11): 1609–1617

[90] Vu QD, Graham S, To NN. Minh, Muhammad S, Talha Q, Navid AK, Ali KS, Tahsin K, Keyvan F, Tianhao Z, Rajarsi G, Tae KJ, Nasir R, Joel S. Methods for segmentation and classification of digital microscopy tissue images. Front Bioeng Biotechnol 2019; 7: 53

[91] Naylor P, Laé M, Reyal F, . Nuclei segmentation in histopathology images using deep neural networks[C]//2017 IEEE 14th International Symposium On Biomedical Imaging (ISBI 2017). IEEE. 2017: 933–936

[92] Kumar N, Verma R, Anand D, Zhou Y, Onder OF, Tsougenis E, Chen H, Heng P, Li J, Hu Z, Wang Y, Koohbanani NA, Jahanifar M, Tajeddin NZ, Gooya A, Rajpoot N, Ren X, Zhou S, Wang Q, Shen D, Yang C, Weng C, Yu W, Yeh C, Yang S, Xu S, Yeung PH, Sun P, Mahbod A, Schaefer G, Ellinger I, Ecker R, Smedby O, Wang C, Chidester B, Ton T, Tran M, Ma J, Do MN, Graham S, Vu QD, Kwak JT, Gunda A, Chunduri R, Hu C, Zhou X, Lotfi D, Safdari R, Kascenas A, O’Neil A, Eschweiler D, Stegmaier J, Cui Y, Yin B, Chen K, Tian X, Gruening P, Barth E, Arbel E, Remer L, Ben-Dor A, Sirazitdinova E, Kohl M, Braunewell S, Li Y, Xie X, Shen L, Ma J, Baksi KD, Khan MA, Choo J, Colomer A, Naranjo V, Pei L, Iftekharuddin KM, Roy K, Bhattacharjee D, Pedraza A, Bueno MG, Devanathan S, Radhakrishnan S, Koduganty P, Wu Z, Cai G, Liu X, Wang Y, Sethi A. A multi-organ nucleus segmentation challenge. IEEE Trans Med Imaging 2020; 39(5): 1380–1391

[93] Neubeck A, Van Gool L. Efficient non-maximum suppression. 18th International Conference on Pattern Recognition (ICPR'06). IEEE. 2006, 3: 850–855

[94] Maurer CR, Qi R, Raghavan V. A linear time algorithm for computing exact euclidean distance transforms of binary images in arbitrary dimensions. IEEE Trans Pattern Anal Mach Intell 2003; 25(2): 265–270

[95] Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 2017; 39(6): 1137–1149

[96] Zhang R, Cheng C, Zhao X, Li X. Multiscale mask R-CNN-based lung tumor detection using PET imaging. Mol Imaging 2019; 18: 1536012119863531

[97] Wang S, Rong R, Yang DM, Fujimoto J, Yan S, Cai L, Yang L, Luo D, Behrens C, Parra ER, Yao B, Xu L, Wang T, Zhan X, Wistuba II, Minna J, Xie Y, Xiao G. Computational staining of pathology images to study the tumor microenvironment in lung cancer. Cancer Res 2020; 80(10): 2056–2066

[98] Johnson JW. Adapting mask-RCNN for automatic nucleus segmentation. arXiv 2018; 1805.00500

[99] Hoda SA, Hoda RS. Rubin’s pathology: clinicopathologic foundations of medicine. JAMA 2007; 298(17): 2070–2075

[100] Sirinukunwattana K, Pluim JPW, Chen H, Qi X, Heng PA, Guo YB, Wang LY, Matuszewski BJ, Bruni E, Sanchez U, Böhm A, Ronneberger O, Cheikh BB, Racoceanu D, Kainz P, Pfeiffer M, Urschler M, Snead DRJ, Rajpoot NM. Gland segmentation in colon histology images: the GlaS challenge contest. Med Image Anal 2017; 35: 489–502

[101] Gunduz-Demir C, Kandemir M, Tosun AB, Sokmensuer C. Automatic segmentation of colon glands using object-graphs. Med Image Anal 2010; 14(1): 1–12

[102] Hess KR, Varadhachary GR, Taylor SH, Wei W, Raber MN, Lenzi R, Abbruzzese JL. Metastatic patterns in adenocarcinoma. Cancer 2006; 106(7): 1624–1633

[103] Ryan DP, Hong TS, Bardeesy N. Pancreatic adenocarcinoma. N Engl J Med 2014; 371(11): 1039–1049

[104] Fleming M, Ravula S, Tatishchev SF, Wang HL. Colorectal carcinoma: pathologic aspects. J Gastrointest Oncol 2012; 3(3): 153–173

[105] Travis WD, Brambilla E, Geisinger KR. Histological grading in lung cancer: one system for all or separate systems for each histological type? Eur Respir J 2016; 47(3): 720–723

[106] Xu Y, Li Y, Liu M, Wang Y, Lai M, Eric C. Gland instance segmentation by deep multichannel side supervision. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 496–504

[107] Kainz P, Pfeiffer M, Urschler M. Segmentation and classification of colon glands with deep convolutional neural networks and total variation regularization. PeerJ 2017; 5: e3874

[108] Li W, Manivannan S, Akbar S, Zhang J, Trucco E, McKenna SJ. Gland segmentation in colon histology images using hand-crafted features and convolutional neural networks. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE. 2016: 1405–1408

[109] Xu Y, Li Y, Wang Y, Liu M, Fan Y, Lai M, Chang EIC. Gland instance segmentation using deep multichannel neural networks. IEEE Trans Biomed Eng 2017; 64(12): 2901–2912

[110] BenTaieb A, Hamarneh G. Topology aware fully convolutional networks for histology gland segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham. 2016: 460–468

[111] BenTaieb A, Kawahara J, Hamarneh G. Multi-loss convolutional networks for gland analysis in microscopy. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE. 2016: 642–645

[112] Wu HS, Xu R, Harpaz N, Burstein D, Gil J. Segmentation of microscopic images of small intestinal glands with directional 2-D filters. Anal Quant Cytol Histol 2005; 27(5): 291–300

[113] Farjam R, Soltanian-Zadeh H, Jafari-Khouzani K, Zoroofi RA. An image analysis approach for automatic malignancy determination of prostate pathological images. Cytometry B Clin Cytom 2007; 72(4): 227–240

[114] Wu HS, Xu R, Harpaz N, Burstein D, Gil J. Segmentation of intestinal gland images with iterative region growing. J Microsc 2005; 220(Pt 3): 190–204

[115] Fu H, Qiu G, Shu J, Ilyas M. A novel polar space random field model for the detection of glandular structures. IEEE Trans Med Imaging 2014; 33(3): 764–776

[116] Sirinukunwattana K, Snead DR, Rajpoot NM. A stochastic polygons model for glandular structures in colon histology images. IEEE Trans Med Imaging 2015; 34(11): 2366–2378

[117] Monaco JP, Tomaszewski JE, Feldman MD, Hagemann I, Moradi M, Mousavi P, Boag A, Davidson C, Abolmaesumi P, Madabhushi A. High-throughput detection of prostate cancer in histological sections using probabilistic pairwise Markov models. Med Image Anal 2010; 14(4): 617–629

[118] Diamond J, Anderson NH, Bartels PH, Montironi R, Hamilton PW. The use of morphological characteristics and texture analysis in the identification of tissue composition in prostatic neoplasia. Hum Pathol 2004; 35(9): 1121–1131

[119] Doyle S, Madabhushi A, Feldman M, Tomaszeweski J. A boosting cascade for automated detection of prostate cancer from digitized histology. Med Image Comput Comput Assist Interv 2006; 9(Pt 2): 504–511

[120] Tabesh A, Teverovskiy M, Pang HY, Kumar VP, Verbel D, Kotsianti A, Saidi O. Multifeature prostate cancer diagnosis and Gleason grading of histological images. IEEE Trans Med Imaging 2007; 26(10): 1366–1378

[121] Nguyen K, Sarkar A, Jain AK. Structure and context in prostatic gland segmentation and classification. Med Image Comput Comput Assist Interv 2012; 15(Pt 1): 115–123

[122] Jacobs JG, Panagiotaki E, Alexander DC. Gleason grading of prostate tumours with max-margin conditional random fields. International Workshop on Machine Learning in Medical Imaging. Springer, Cham. 2014: 85–92

[123] Sabata B, Babenko B, Monroe R, Srinivas C. Automated analysis of pin-4 stained prostate needle biopsies. International Workshop on Prostate Cancer Imaging. Springer, Berlin, Heidelberg. 2010: 89–100

[124] Altunbay D, Cigir C, Sokmensuer C, Gunduz-Demir C. Color graphs for automated cancer diagnosis and grading. IEEE Trans Biomed Eng 2010; 57(3): 665–674

[125] Fakhrzadeh A, Sporndly-Nees E, Holm L, Hendriks CLL. Analyzing tubular tissue in histopathological thin sections. 2012 International Conference on Digital Image Computing Techniques and Applications (DICTA). IEEE. 2012: 1–6

[126] Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 2012: 1097–1105

[127] Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. Proceedings of the IEEE Conference on computer Vision and Pattern Recognition. 2015: 1–9

[128] Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. Imagenet: a large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 248–255

[129] Manivannan S, Li W, Akbar S, Wang R, Zhang J, McKenna SJ. An automated pattern recognition system for classifying indirect immunofluorescence images of hep-2 cells and specimens. Pattern Recognit 2016; 51: 12–26

[130] Xu Y, Jia Z, Wang LB, Ai Y, Zhang F, Lai M, Chang EIC. Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinformatics 2017; 18(1): 281

[131] Xu Y, Mo T, Feng Q, Zhong P, Lai M, Chang EI. Deep learning of feature representation with multiple instance learning for medical image analysis. IEEE International Conference on Acoustics, Speech and Signal Processing 2014; 1626–1630

[132] Xu Y, Jia Z, Ai Y, Zhang F, Lai M, Change EI. Deep convolutional activation features for large scale brain tumor histopathology image classification and segmentation. IEEE International Conference on Acoustics, Speech and Signal Processing. 2015: 947–951

[133] Hou L, Samaras D, Kurc TM, Gao Y, Davis JE, Saltz JH. Patch-based convolutional neural network for whole slide tissue image classification. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2016; 2016: 2424–2433

[134] Källén H, Molin J, Heyden A, Lundstrom C, Astrom K. Towards grading gleason score using generically trained deep convolutional neural networks. IEEE International Symposium on Biomedical Imaging. 2016: 1163–1167

[135] Xu J, Luo X, Wang G, Gilmore H, Madabhushi A. A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 2016; 191: 214–223

[136] Qaiser T, Tsang YW, Epstein D, Rajpoot N. Tumor segmentation in whole slide images using persistent homology and deep convolutional features. Annual Conference on Medical Image Understanding and Analysis. 2017; 723: 320–329

[137] Jia Z, Huang X, Chang EI, Xu Y. Constrained deep weak supervision for histopathology image segmentation. IEEE Trans Med Imaging 2017; 36(11): 2376–2388

[138] Courtiol P, Tramel EW, Sanselme M, Wainrib G. Classification and disease localization in histopathology using only global labels: a weakly-supervised approach. 2018

[139] Wang X, Chen H, Gan C, Lin H, Dou Q, Tsougenis E, Huang Q, Cai M, Heng PA. Weakly supervised deep learning for whole slide lung cancer image analysis. IEEE Trans Cybern 2019; [Epub ahead of print] doi: 10.1109/TCYB.2019.2935141

[140] Mercan C, Aksoy S, Mercan E, Shapiro LG, Weaver DL, Elmore JG. From patch-level to roi-level deep feature representations for breast histopathology classification. Medical Imaging 2019: Digital Pathology. 2019. 109560H

[141] Xu Y, Jiao L, Wang S, Wei J, Fan Y, Lai M, Chang EI. Multi-label classification for colon cancer using histopathological images. Microsc Res Tech 2013; 76(12): 1266–1277

[142] Jiao L, Qi C, Li S, Yan X. Colon cancer detection using whole slide histopathological images. IFMBE Proc 2013; 39: 1283–1286

[143] Adankon MM, Cheriet M. Support vector machine. Comput Sci 2002; 1(4): 1–28

[144] Golland P, Hata N, Barillot C, Hornegger J, Howe R. Preface. Medical image computing and computer-assisted intervention—MICCAI 2014. Med Image Comput Comput Assist Interv 2014; 17(Pt 1): V–VI

[145] Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, Lecun Y. Overfeat: integrated recognition, localization and detection using convolutional networks. International Conference on Learning Representations. 2014

[146] Liaw A, Wiener M. Classification and regression by random forest. R News 2007; 2: 18–22

[147] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv 2014; 1409.1556v6

[148] Yan X, Zhu JY, Chang EI, Tu Z. Multiple clustered instance learning for histopathology cancer image classification, segmentation and clustering. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2012; 964–971

[149] Xu Y, Zhang J, Chang EI, Lai M, Tu Z. Context-constrained multiple instance learning for histopathology image segmentation. Med Image Comput Comput Assist Interv 2012; 15(Pt 3): 623–630

[150] Xu Y, Zhu JY, Chang EI, Lai M, Tu Z. Weakly supervised histopathology cancer image segmentation and classification. Med Image Anal 2014; 18(3): 591–604

[151] Xu Y, Li Y, Shen Z, Wu Z, Gao T, Fan Y, Lai M, Chang EI. Parallel multiple instance learning for extremely large histopathology image analysis. BMC Bioinformatics 2017; 18(1): 360

[152] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2016: 770–778

[153] Hagerty RG, Butow PN, Ellis PM, Dimitry S, Tattersall MH. Communicating prognosis in cancer care: a systematic review of the literature. Ann Oncol 2005; 16(7): 1005–1053

[154] Norgeot B, Glicksberg BS, Butte AJ. A call for deep-learning healthcare. Nat Med 2019; 25(1): 14–15

[155] Uramoto H, Tanaka F. Recurrence after surgery in patients with NSCLC. Transl Lung Cancer Res 2014; 3(4): 242–249

[156] Wang X, Janowczyk A, Zhou Y, Thawani R, Fu P, Schalper K, Velcheti V, Madabhushi A. Prediction of recurrence in early stage non-small cell lung cancer using computer extracted nuclear features from digital H&E images. Sci Rep 2017; 7(1): 13543

[157] Vaidya P, Wang X, Bera K, Khunger A, Choi H, Patil P, Velcheti V, Madabhushi A. Raptomics: integrating radiomic and pathomic features for predicting recurrence in early stage lung cancer. Medical Imaging 2018: Digital Pathology International Society for Optics and Photonics. 2018: 105810M

[158] Sanchez-Cespedes M, Parrella P, Esteller M, Nomoto S, Trink B, Engles JM, Westra WH, Herman JG, Sidransky D. Inactivation of LKB1/STK11 is a common event in adenocarcinomas of the lung. Cancer Res 2002; 62(13): 3659–3662

[159] Shackelford DB, Abt E, Gerken L, Vasquez DS, Seki A, Leblanc M, Wei L, Fishbein MC, Czernin J, Mischel PS, Shaw RJ. LKB1 inactivation dictates therapeutic response of non-small cell lung cancer to the metabolism drug phenformin. Cancer Cell 2013; 23(2): 143–158

[160] Parums DV. Current status of targeted therapy in non-small cell lung cancer. Drugs Today (Barc) 2014; 50(7): 503–525

[161] Wang S, Yang DM, Rong R, Zhan X, Fujimoto J, Liu H, Minna J, Wistuba II, Xie Y, Xiao G. Artificial intelligence in lung cancer pathology image analysis. Cancers (Basel) 2019; 11(11): 1673

[162] Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D, Moreira AL, Razavian N, Tsirigos A. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med 2018; 24(10): 1559–1567

[163] Kather JN, Pearson AT, Halama N, Jäger D, Krause J, Loosen SH, Marx A, Boor P, Tacke F, Neumann UP, Grabsch HI, Yoshikawa T, Brenner H, Chang-Claude J, Hoffmeister M, Trautwein C, Luedde T. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat Med 2019; 25(7): 1054–1056

[164] Le DT, Uram JN, Wang H, Bartlett BR, Kemberling H, Eyring AD, Skora AD, Luber BS, Azad NS, Laheru D, Biedrzycki B, Donehower RC, Zaheer A, Fisher GA, Crocenzi TS, Lee JJ, Duffy SM, Goldberg RM, de la Chapelle A, Koshiji M, Bhaijee F, Huebner T, Hruban RH, Wood LD, Cuka N, Pardoll DM, Papadopoulos N, Kinzler KW, Zhou S, Cornish TC, Taube JM, Anders RA, Eshleman JR, Vogelstein B, Diaz LA Jr. Pd-1 blockade in tumors with mismatch-repair deficiency. N Engl J Med 2015; 372(26): 2509–2520

[165] Nagpal K, Foote D, Liu Y, Chen PHC, Wulczyn E, Tan F, Olson N, Smith JL, Mohtashamian A, Wren JH, Corrado GS, MacDonald R, Peng LH, Amin MB, Evans AJ, Sangoi AR, Mermel CH, Hipp JD, Stumpe MC. Development and validation of a deep learning algorithm for improving gleason scoring of prostate cancer. NPJ Digit Med 2019; 2(1): 1–10

[166] Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. 2016: 2818–2826

[167] Ing N, Ma Z, Li J, Salemi H, Arnold C, Knudsen BS, Gertych A. Semantic segmentation for prostate cancer grading by convolutional neural networks. Medical Imaging 2018: Digital Pathology International Society for Optics and Photonics. 2018: 105811B

[168] Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 2017; 39(12): 2481–2495

[169] Li W, Li J, Sarma KV, Ho KC, Shen S, Knudsen BS, Gertych A, Arnold CW. Path R-CNN for prostate cancer diagnosis and gleason grading of histological images. IEEE Trans Med Imaging 2019; 38(4): 945–954

Acknowledgements

This work was supported by the National Science and Technology Major Project of the Ministry of Science and Technology of China (No. 2017YFC0110903), Microsoft Research under the eHealth program, the National Natural Science Foundation of China (No. 81771910), the Beijing Natural Science Foundation in China (No. 4152033), the Technology and Innovation Commission of Shenzhen in China (No. shenfagai2016-627), the Beijing Young Talent Project in China, the Fundamental Research Funds for the Central Universities of China (No. SKLSDE-2017ZX-08) from the State Key Laboratory of Software Development Environment in Beihang University in China, and the 111 Project in China (No. B13003).

Compliance with ethics guidelines

Shujian Deng, Xin Zhang, Wen Yan, Eric I-Chao Chang, Yubo Fan, Maode Lai, and Yan Xu declare that they have no conflicts of interest. This manuscript is a review article and does not involve a research protocol requiring approval by the relevant institutional review board or ethics committee.

RIGHTS & PERMISSIONS

2020 Higher Education Press