MFCIS: an automatic leaf-based identification pipeline for plant cultivars using deep learning and persistent homology

Abstract

Recognizing plant cultivars reliably and efficiently can benefit plant breeders in terms of property rights protection and innovation of germplasm resources. Although leaf image-based methods have been widely adopted in plant species identification, they seldom have been applied in cultivar identification due to the high similarity of leaves among cultivars. Here, we propose an automatic leaf image-based cultivar identification pipeline called MFCIS (Multi-feature Combined Cultivar Identification System), which combines multiple leaf morphological features collected by persistent homology and a convolutional neural network (CNN). Persistent homology, a multiscale and robust method, was employed to extract the topological signatures of leaf shape, texture, and venation details. A CNN-based algorithm, the Xception network, was fine-tuned for extracting high-level leaf image features. For fruit species, we benchmarked the MFCIS pipeline on a sweet cherry ( Prunus avium L.) leaf dataset with >5000 leaf images from 88 varieties or unreleased selections and achieved a mean accuracy of 83.52%. For annual crop species, we applied the MFCIS pipeline to a soybean ( Glycine max L. Merr.) leaf dataset with 5000 leaf images of 100 cultivars or elite breeding lines collected at five growth periods. The identification models for each growth period were trained independently, and their results were combined using a score-level fusion strategy. The classification accuracy after score-level fusion was 91.4%, which is much higher than the accuracy when utilizing each growth period independently or mixing all growth periods. To facilitate the adoption of the proposed pipelines, we constructed a user-friendly web service, which is freely available at http://www.mfcis.online.

References

[1] Sohn, H. B. et al. Barcode system for genetic identification of soybean [ Glycine max (L.) Merrill] cultivars using InDel markers specific to dense variation blocks . Front. Plant Sci. 8, 520 (2017).

[2] Korir, N. K. et al. Plant variety and cultivar identification: advances and prospects. Crit. Rev. Biotechnol. 33, 111-125 (2013).

[3] Jamali, S. et al. Identification and distinction of soybean commercial cultivars using morphological and microsatellite markers. Iranian J. Crop Sci. 13, 131-145 (2011).

[4] Wu, K. et al. Genetic analysis and molecular characterization of Chinese sesame (Sesamum indicum L.) cultivars using Insertion-Deletion (InDel) and Simple Sequence Repeat (SSR) markers . BMC Genet. 15, 35 (2014).

[5] Lee, S. H. et al. How deep learning extracts and learns leaf features for plant classification. Pattern Recognit. 71, 1-13 (2017).

[6] Zhao, C., Chan, S. S. F., Cham, W.-K. & Chu, L. M. Plant identification using leaf shapes: a pattern counting approach. Pattern Recognit. 48, 3203-3215 (2015).

[7] Price, C. A. et al. Leaf extraction and analysis framework graphical user interface: segmenting and analyzing the structure of leaf veins and areoles. Plant Physiol. 155, 236-245 (2011).

[8] De Vylder, J., Vandenbussche, F. & Hu, Y. et al. Rosette tracker: an open source image analysis tool for automatic quantification of genotype effects. Plant Physiology 160, 1149-1159 (2012).

[9] Zhou, J. et al. Leaf-GP: an open and automated software application for measuring growth phenotypes for arabidopsis and wheat. Plant Methods 13, 117 (2017).

[10] Biot, E. et al. Multi-scale quantification of morphodynamics: MorphoLeaf software for 2D shape analysis. Development 143, 3417-3428 (2016).

[11] Minervini, M. et al. Phenotiki: an open software and hardware platform for affordable and easy image-based phenotyping of rosette-shaped plants. Plant J. 90, 204-216 (2017).

[12] Neto, J. C. et al. Plant species identification using Elliptic Fourier leaf shape analysis. Computers Electron. Agriculture 50, 121-134 (2006).

[13] Cope, J. S. et al. in International Symposium on Visual Computing (eds Bebis, G. et al. ) 669-677 (Springer, 2010).

[14] Chaki, J. & Parekh, R. Plant leaf recognition using shape based features and neural network classifiers. Int. J. Adv. Comp. Sci. Appl. 2, 41-47 (2011).

[15] Naresh, Y. & Nagendraswamy, H. Classification of medicinal plants: an approach using modified LBP with symbolic representation. Neurocomputing 173, 1789-1797 (2016).

[16] Pradeep Kumar, T., Veera Prasad Reddy, M. & Bora, P. K. Leaf identification using shape and texture features. Proceedings of International Conference on Computer Vision and Image Processing (eds Raman B., Kumar S., Roy P. P., Sen D.) 531-541 (Springer Singapore, 2017).

[17] Tharwat, A., Gaber, T., Awad, Y. M., Dey, N. & Hassanien, A. E. Plants identification using feature fusion technique and bagging classifier. (eds Gaber T., Hassanien A. E., El-Bendary N., Dey N. ). The 1st International Conference on Advanced Intelligent System and Informatics (AISI2015), November 28-30, 2015, Beni Suef, Egypt. 461-471 (Springer International Publishing, 2016).

[18] Codizar, A. L. & Solano, G. Plant leaf recognition by venation and shape using artificial neural networks. In: 2016 7th International Conference on Information, Intelligence, Systems & Applications (IISA). 1-4 (IEEE, 2016).

[19] Yang, C. Plant leaf recognition by integrating shape and texture features. Pattern Recognit. 112, 107809 (2021).

[20] Liu, C. et al. A novel identification method for apple (Malus domestica Borkh.) cultivars based on a deep convolutional neural network with leaf image input . Symmetry 12, 217 (2020).

[21] Baldi, A. et al. A leaf-based back propagation neural network for oleander (Nerium oleander L.) cultivar identification . Computers Electron. Agriculture 142, 515-520 (2017).

[22] Yu, X. et al. Patchy image structure classification using multi-orientation region transform. in Proceedings of the AAAI Conference on Artificial Intelligence. 12741-12748 (AAAI, 2020).

[23] Edelsbrunner, H & Harer, J . in Persistent Homology-a Survey (eds Goodman, J. E., Pach, J., Pollack, R.). 257-282 (Contemporary Mathematics American Mathematical Society, 2008).

[24] Li, M. et al. Topological data analysis as a morphometric method: using persistent homology to demarcate a leaf morphospace. Front. Plant Sci. 9, 553 (2018).

[25] Reininghaus, J. et al. A stable multi-scale kernel for topological machine learning, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 4741-4748 (IEEE, Boston, MA, USA, 2015).

[26] Li, C., Ovsjanikov, M. & Chazal, F. Persistence-based structural recognition. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1995-2002 (IEEE Computer Society, 2014).

[27] Dey, T., Mandal, S. & Varcho, W. Improved image classification using topological persistence. in Proceedings of the Conference on Vision, Modeling and Visualization. 161-168 (Eurographics Association, 2017).

[28] MacLane, S. Homology. Bull. Am. Math. Soc. 70, 329-331 (1964).

[29] Qaiser, T. et al. Tumor segmentation in whole slide images using persistent homology and deep convolutional features. in Annual Conference on Medical Image Understanding and Analysis. 320-329 (Springer, 2017).

[30] Qaiser, T. et al. Fast and accurate tumor segmentation of histology images using persistent homology and deep convolutional features. Med. Image Anal. 55, 1-14 (2019).

[31] Zeppelzauer, M. et al. A study on topological descriptors for the analysis of 3d surface texture. Computer Vis. Image Underst. 167, 74-88 (2018).

[32] Chollet, F. Xception: Deep learning with depthwise separable convolutions. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1251-1258 (IEEE Computer Society, 2017).

[33] Hofer, C. et al. Deep learning with topological signatures. In: Advances in Neural Information Processing Systems. 1634-1644 (Curran Associates Inc., 2017).

[34] Turner, K., Mukherjee, S. & Boyer, D. M. Persistent homology transform for modeling shapes and surfaces. Inf. Inference.: A J. IMA 3, 310-344 (2014).

[35] Deng, J. et al. Imagenet: a large-scale hierarchical image database. in 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248-255 (IEEE, 2009).

[36] Adams, H. et al. Persistence images: A stable vector representation of persistent homology. J. Mach. Learn. Res. 18, 218-252 (2017).

[37] Bubenik, P. Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res. 16, 77-102 (2015).

[38] Wang, B. et al. From species to cultivar: Soybean cultivar recognition using joint leaf image patterns by multi-scale sliding chord matching. Biosyst. Eng. 194, 99-111 (2020).

[39] Heiberger, R. M., & Neuwirth E. One-way ANOVA. In: R through Excel. 165-191 (Springer, 2009).

[40] Ling, H. & Jacobs, D. W. Shape classification using the inner-distance. IEEE Trans. Pattern Anal. Mach. Intell. 29, 286-299 (2007).

[41] Wang, B. & Gao, Y. Hierarchical string cuts: a translation, rotation, scale, and mirror invariant descriptor for fast shape retrieval. IEEE Trans. Image Process 23, 4101-4111 (2014).

[42] Kaya, A. et al. Analysis of transfer learning for deep neural network-based plant classification models. Computers Electron. Agriculture 158, 20-29 (2019).

[43] Yanping, Z. & Liu, W. WeizhenLiuBioinform/mfcis: source code of mfcis. (Version 1.0.2). Zenodo https://doi.org/10.5281/zenodo.4739746 (2021).

[44] Barré, P. et al. LeafNet: a computer vision system for automatic plant species identification. Ecol. Inform. 40, 50-56 (2017).

[45] Beghin, T. et al. Shape and texture-based plant leaf classification. in International Conference on Advanced Concepts for Intelligent Vision Systems, 345-353 (Springer, 2010).

[46] Blonder, B. et al. X-ray imaging of leaf venation networks. N. Phytologist 196, 1274-1282 (2012).

[47] Gan, Y. et al. Automatic hierarchy classification in venation networks using directional morphological filtering for hierarchical structure traits extraction. Computational Biol. Chem. 80, 187-194 (2019).

[48] Cui, F. & Yang, G. Score level fusion of fingerprint and finger vein recognition. J. Computational Inf. Syst. 7, 5723-5731 (2011).

[49] Park, H.-A. & Park, K. R. Iris recognition based on score level fusion by using SVM. Pattern Recognit. Lett. 28, 2019-2028 (2007).

[50] Ghosh, S. et al. Software for systems biology: from tools to integrated platforms. Nat. Rev. Genet. 12, 821-832 (2011).

[51] Smulders, M., Booy, I. & Vosman, B. Use of molecular and biochemical methods for identification of plant varieties throughout the agri-chain. (eds Trienekens, J. H. & Zuurbier, P. J. P.) In Proceedings of the 2nd International Conference on Chain Management in Agri-and Food Business. 591-600 (Department of Management studies Wageningen Agricultural University, May 1996).

[52] Park, H. et al. Molecular identification of sweet potato accessions using ARMS-PCR based on SNPs. J. Plant Biotechnol. 47, 124-130 (2020).

[53] Fufa, H. et al. Comparison of phenotypic and molecular marker-based classifications of hard red winter wheat cultivars. Euphytica 145, 133-146 (2005).

[54] Kim, M. et al. Genome-wide SNP discovery and core marker sets for DNA barcoding and variety identification in commercial tomato cultivars. Sci. Horticulturae 276, 109734 (2021).

[55] Patzak, J., Henychová, A., Paprštein, F. & Sedlák, J . Evaluation of S-incompatibility locus, genetic diversity and structure of sweet cherry (Prunus avium L.) genetic resources by molecular methods and phenotypic characteristics . J. Horticultural Sci. Biotechnol. 95, 84-92 (2020).

[56] Pourkhaloee, A. et al. Molecular analysis of genetic diversity, population structure, and phylogeny of wild and cultivated tulips (Tulipa L.) by genic microsatellites . Horticulture Environ. Biotechnol. 59, 875-888 (2018).

[57] Cho, K. H. et al. Sequence-characterized amplified region markers and multiplex-polymerase chain reaction assays for kiwifruit cultivar identification. Horticulture Environ., Biotechnol. 61, 395-406 (2020).

[58] Agarwal, M., Shrivastava, N. & Padh, H. Advances in molecular marker techniques and their applications in plant sciences. Plant Cell Rep. 27, 617-631 (2008).

[59] Nadeem, M. A. et al. DNA molecular markers in plant breeding: current status and recent advancements in genomic selection and genome editing. Biotechnol. Biotechnological Equip. 32, 261-285 (2018).

[60] Yamaç, S. S. & Todorovic, M. Estimation of daily potato crop evapotranspiration using three different machine learning algorithms and four scenarios of available meteorological data. Agric. Water Manag. 228, 105875 (2020).

[61] Reisi Gahrouei, O., McNairn, H., Hosseini, M. & Homayouni, S. Estimation of crop biomass and leaf area index from multitemporal and multispectral imagery using machine learning approaches. Can. J. Remote Sens. 46, 84-99 (2020).

[62] Colmer, J. et al. SeedGerm: a cost-effective phenotyping platform for automated seed imaging and machine-learning based phenotypic analysis of crop seed germination. N. Phytologist 228, 778-793 (2020).

[63] Danner, M., Berger, K., Wocher, M., Mauser, W. & Hank, T. Efficient RTM-based training of machine learning regression algorithms to quantify biophysical & biochemical traits of agricultural crops. ISPRS J. Photogramm. Remote Sens. 173, 278-296 (2021).

[64] Zeiler, M. D. & Fergus R. in Visualizing and Understanding Convolutional Networks (eds Fleet D., Pajdla T., Schiele B., Tuytelaars T.). Computer Vision-ECCV 2014. 818-833 (Springer International Publishing, 2014).

[65] Erhan, D., Bengio, Y., Courville, A. & Vincent, P. Visualizing higher-layer features of a deep network. Univ. Montr. 1341, 1 (2009).

[66] Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations.(2014).

[67] Islam, M. R. Feature and score fusion based multiple classifier selection for iris recognition. Computational Intell. Neurosci. 2014, e380585 (2014).

[68] Yang, J. et al. Feature fusion: parallel strategy vs. serial strategy. Pattern Recognit. 36, 1369-1381 (2003).

[69] Bryson, A. E. et al. Composite modeling of leaf shape across shoots discriminates Vitis species better than individual leaves. Preprint at bioRxiv https://doi.org/10.1101/2020.06.22.163899 (2020).