High throughput saliency-based quantification of grape powdery mildew at the microscopic level for disease resistance breeding

Abstract

Imaging-based high throughput phenotyping (HTP) systems have demonstrated promising solutions to enhance genetic understanding of grapevine powdery mildew (PM) resistance and have accelerated PM-resistant cultivar breeding. The accuracy and throughput of extracting phenotypic traits from images are still the bottleneck of modern HTP systems, especially at the microscopic level. The goal of this study was to develop a saliency-based processing pipeline for the quantification of PM infection in microscopic images and comprehensively evaluate its performance for genetic analyses. An input image was segregated into subimages that were classified as infected or healthy by a pretrained CNN classifier. Saliency maps from the classification were generated post-hoc and used for the quantification of PM infection in the input image at the pixel level without the use of mask annotations. A total of seven phenotypic traits were extracted from images collected for a biparental population. Experimental results showed that optimal combinations of convolutional neural network and saliency methods achieved strong measurement correlations (r = 0.74 to 0.75) with human assessments at the image patch level, and the traits calculated by the saliency-based processing pipeline were highly correlated (r = 0.87 to 0.88) with reference PM infection ratings at the leaf image level. The high quantification accuracy of the saliency-based pipeline led to the increased explanation of phenotypic variance and reliable identification of quantitative trait loci. Therefore, the saliency-based processing pipeline can be used as an effective and efficient analysis tool for PM disease research and breeding programs in the future, especially agricultural and life science studies requiring microscopic image analysis.

References

[1] Tuck B, Gartner W, Appiah G . Economic contribution of vineyards and wineries of the north . 2017; 2015: 1-18.

[2] Fuller KB, Alston JM, Sambucci OS . The value of powdery mildew resistance in grapes: evidence from California. Wine Econ Policy. 2014; 3: 90-107.

[3] Araus JL, Kefauver SC, Zaman-Allah M et al. Translating high-throughput phenotyping into genetic gain. Trends Plant Sci. 2018; 23: 451-66.

[4] Furbank RT, Tester M . Phenomics-technologies to relieve the phenotyping bottleneck. Trends Plant Sci. 2011; 16: 635-44.

[5] Boursianis AD, Papadopoulou MS, Diamantoulakis P et al. Internet of things (IoT) and agricultural unmanned aerial vehicles (UAVs) in smart farming: a comprehensive review. Internet of Things. 2022; 18: 1-17.

[6] Cadle-Davidson L, Gadoury D, Fresnedo-Ramírez J et al. Lessons from a phenotyping center revealed by the genome-guided mapping of powdery mildew resistance loci. Phytopathology. 2016; 106: 1159-69.

[7] Bierman A, LaPlumm T, Cadle-Davidson L et al. A high-throughput phenotyping system using machine vision to quantify severity of grapevine powdery mildew. Plant Phenomics. 2019; 2019: 1-13.

[8] Lück S, Strickert M, Lorbeer M et al. “Macrobot”: an automated segmentation-based system for powdery mildew disease quantification. Plant Phenomics. 2020; 2020: 1-13.

[9] Mahmud MS, Zaman QU, Esau TJ et al. Real-time detection of strawberry powdery mildew disease using a mobile machine vision system. Agronomy. 2020; 10: 1027.

[10] Shin J, Chang YK, Heung B et al. A deep learning approach for RGB image-based powdery mildew disease detection on strawberry leaves. Comput Electron Agric. 2021; 183: 1-8.

[11] Adeel A, Khan MA, Sharif M et al. Diagnosis and recognition of grape leaf diseases: an automated system based on a novel saliency approach and canonical correlation analysis based multiple features fusion. Sustainable Computing: Informatics and Systems. 2019; 24: 1-11.

[12] Chandel AK, Khot LR, Sallato B . Apple powdery mildew infestation detection and mapping using high-resolution visible and multispectral aerial imaging technique. Sci Hortic. 2021; 287: 1-7.

[13] Pineda M, Barón M, Pérez-Bueno M-L . Thermal imaging for plant stress detection and phenotyping. Remote Sens. 2021; 13: 68.

[14] Yao J, Sun D, Cen H et al. Phenotyping of Arabidopsis drought stress response using kinetic chlorophyll fluorescence and multicolor fluorescence imaging. Front Plant Sci. 2018; 9: 603.

[15] Tardaguila J, Stoll M, Gutiérrez S et al. Smart applications and digital technologies in viticulture: a review. Smart Agricultural Technology. 2021; 1: 1-13.

[16] Bélanger MC, Roger JM, Cartolaro P et al. Detection of powdery mildew in grapevine using remotely sensed UV-induced fluorescence. Int J Remote Sens. 2008; 29: 1707-24.

[17] Oberti R, Marchi M, Tirelli P et al. Automatic detection of powdery mildew on grapevine leaves by image analysis: optimal view-angle range to increase the sensitivity. Comput Electron Agric. 2014; 104: 1-8.

[18] Kamilaris A, Prenafeta-Boldú FX . Deep learning in agriculture: a survey. Comput Electron Agric. 2018; 147: 70-90.

[19] Jiang Y, Li C . Convolutional neural networks for image-based high-throughput plant phenotyping: a review. Plant Phenomics. 2020; 2020: 1-22.

[20] Singh AK, Ganapathysubramanian B, Sarkar S et al. Deep learning for plant stress phenotyping: trends and future perspectives. Trends Plant Sci. 2018; 23: 883-98.

[21] Ghosal S, Blystone D, Singh AK et al. An explainable deep machine vision framework for plant stress phenotyping. Proc Natl Acad Sci. 2018; 115: 4613-8.

[22] Lin K, Gong L, Huang Y et al. Deep learning-based segmentation and quantification of cucumber powdery mildew using convolutional neural network. Front Plant Sci. 2019; 10: 155.

[23] Fuentes A, Yoon S, Kim SC et al. A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition. Sensors. 2017; 17: 2022.

[24] Arrieta AB, Diaz-Rodriguez N, Del Ser J et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion. 2020; 58: 82-115.

[25] Vilone G, Longo L . Explainable artificial intelligence: a systematic review arXiv preprint arXiv:2006.00093. 2020.

[26] Adadi A, Berrada M . Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE access. 2018; 6: 52138-60.

[27] Toda Y, Okura F . How convolutional neural networks diagnose plant disease. Plant Phenomics. 2019; 2019: 1-14.

[28] Zeiler MD, Fergus R . Visualizing and understanding convolutional networks . In: In European Conference on Computer Vision.Springer: Cham, 2014, 818-33.

[29] Simonyan K, Vedaldi A, Zisserman A . Deep inside convolutional networks: visualising image classification models and saliency maps arXiv preprint arXiv:1312.6034. 2013.

[30] Selvaraju RR, Cogswell M, Das A et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision.IEEE, 2017, 618-26.

[31] Smilkov D, Thorat N, Kim B et al. Smoothgrad: removing noise by adding noise arXiv preprint arXiv:1706.03825. 2017.

[32] Shrikumar A, Greenside P, Kundaje A . Learning important features through propagating activation differences. In: International Conference on Machine Learning.ACM, 2017, 3145-53.

[33] Brahimi M, Arsenovic M, Laraba S et al. Deep Learning for Plant Diseases: Detection and Saliency Map Visualisation. In: Zhou J, Fang C (eds.), Human and Machine Learning.Springer: Cham, 2018, 93-117.

[34] Brahimi M, Boukhalfa K, Moussaoui A . Deep learning for tomato diseases: classification and symptoms visualization. Appl Artif Intell. 2017; 31: 299-315.

[35] Brahimi M, Mahmoudi S, Boukhalfa K et al. Deep interpretable architecture for plant diseases classification. 2019 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA). 2019; 111-6.

[36] Qiu T, Underhill A, Sapkota SD et al. Deep learning-based saliency maps for the quantification of grape powdery mildew at the microscopic level. 2021 ASABE Annual International Virtual Meeting. 2021; 1: 1-13.

[37] Moreno-Camacho CA, Montoya-Torres JR, Jaegler A et al. Sustainability metrics for real case applications of the supply chain network design problem: a systematic literature review. J Clean Prod. 2019; 231: 600-18.

[38] Adebayo J, Gilmer J, Muelly M et al. Sanity checks for saliency maps arXiv preprint arXiv:1810.03292. 2018.

[39] Springenberg JT, Dosovitskiy A, Brox T et al. Striving for simplicity: the all convolutional net arXiv preprint arXiv:1412.6806. 2014.

[40] Simonyan K, Zisserman A . Very deep convolutional networks for large-scale image recognition arXiv preprint arXiv:1409.1556. 2014.

[41] He K, Zhang X, Ren S et al. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE, 2016, 770-8.

[42] Szegedy C, Vanhoucke V, Ioffe S et al. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE, 2016, 2818-26.

[43] Chen L-C, Papandreou G, Schroff F et al. Rethinking atrous convolution for semantic image segmentation arXiv preprint arXiv:1706.05587. 2017.

[44] Rastas P . Lep-MAP3: robust linkage mapping even for low-coverage whole genome sequencing data. Bioinformatics. 2017; 33: 3726-32.

[45] Zou C, Karn A, Reisch B et al. Haplotyping the Vitis collinear core genome with rhAmpSeq improves marker transferability in a diverse genus. Nat Commun. 2020; 11: 1-11.

[46] Karn A, Zou C, Brooks S et al. Discovery of the REN11 locus from Vitis aestivalis for stable resistance to grapevine powdery mildew in a family segregating for several unstable and tissue-specific quantitative resistance loci. Front Plant Sci. 2021; 12: 1-14.

[47] Nie W, Zhang Y, Patel A . A theoretical explanation for perplexing behaviors of backpropagation-based visualizations . In: International Conference on Machine Learning.Nature Pub. Group: London, 2018, 3809-18.

[48] Chen C, Li O, Tao C et al. This looks like that: deep learning for interpretable image recognition arXiv preprint arXiv:1806.10574. 2018.

[49] Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence. 2019; 1: 206-15.