Combining computer vision and deep learning to enable ultra-scale aerial phenotyping and precision agriculture: A case study of lettuce production

Abstract

Aerial imagery is regularly used by crop researchers, growers and farmers to monitor crops during the growing season. To extract meaningful information from large-scale aerial images collected from the field, high-throughput phenotypic analysis solutions are required, which not only produce high-quality measures of key crop traits, but also support professionals to make prompt and reliable crop management decisions. Here, we report AirSurf, an automated and open-source analytic platform that combines modern computer vision, up-to-date machine learning, and modular software engineering in order to measure yield-related phenotypes from ultra-large aerial imagery. To quantify millions of in-field lettuces acquired by fixed-wing light aircrafts equipped with normalised difference vegetation index (NDVI) sensors, we customised AirSurf by combining computer vision algorithms and a deep-learning classifier trained with over 100,000 labelled lettuce signals. The tailored platform, AirSurf-Lettuce, is capable of scoring and categorising iceberg lettuces with high accuracy (>98%). Furthermore, novel analysis functions have been developed to map lettuce size distribution across the field, based on which associated global positioning system (GPS) tagged harvest regions have been identified to enable growers and farmers to conduct precision agricultural practises in order to improve the actual yield as well as crop marketability before the harvest.

References

[1] Gubbi, J., Buyya, R., Marusic, S. & Palaniswami, M . Internet of Things (IoT): a vision, architectural elements, and future directions. Futur Gener. Comput Syst. 29, 1645-1660 (2013).

[2] Mou, B. Mutations in lettuce improvement. Int J. Plant Genom. 2011, 1-7 (2011).

[3] Koudela, M. & Petříková, K. Nutrients content and yield in selected cultivars of leaf lettuce (Lactuca sauva L. var crispa) . Hortic. Sci. 35, 99-106 (2008).

[4] Tei, F., Scaife, A. & Aikman, D. Growth of lettuce, onion, and red beet. 1. Growth Anal. light Intercept., Radiat. use Effic.. Ann. Bot. 78, 633-643 (1996).

[5] BBC. Iceberg lettuces and broccoli rationed as vegetable crisis hits supermarkets. BBC News. 2017; 1-3. Published online 3 February 2017. https://www.bbc.com/news/uk-38851097.

[6] Jie, H. Lee Sing Kong . Growth and photosynthetic characteristics of lettuce (Lactuca sativa L.) under fluctuating hot ambient temperatures with the manipulation of cool root-zone temperature . J. Plant Physiol. 152, 387-391 (1998).

[7] Johkan, M., Shoji, K. & Goto, F. Hashida S nosuke, Yoshihara T. Blue light-emitting diode light irradiation of seedlings improves seedling quality and growth after transplanting in red leaf lettuce. HortScience 45, 1809-1814 (2010).

[8] Ogbodo, E. N., Okorie, P. O. & Utobo, E. B. Growth and Yield of Lettuce (Lactuca sativa L.) At Abakaliki Agro-Ecological Zone of Southeastern Nigeria . World J. Agric Sci. 6, 141-148 (2010).

[9] Tudela, J. A., Hernández, N., Pérez-Vicente, A. & Gil, M. I. Growing season climates affect quality of fresh-cut lettuce. Postharvest Biol. Technol. 123, 60-68 (2017).

[10] Sankaran, S. et al. Low-altitude, high-resolution aerial imaging systems for row and field crop phenotyping: a review. Eur. J. Agron. 70, 112-123 (2015).

[11] Duan T., et al. Comparison of ground cover estimates from experiment plots in cotton, sorghum and sugarcane based on images and ortho-mosaics captured by UAV. Funct. Plant Biol. https://doi.org/10.1071/FP16123 (2016).

[12] Meyer, G. E. & Neto, J. C. Verification of color vegetation indices for automated crop imaging applications. Comput. Electron. Agric. 63, 282-293 (2008).

[13] Yang, G. et al. Unmanned aerial vehicle remote sensing for field-based crop phenotyping: current status and perspectives. Front Plant Sci. 8, 1-27 (2017).

[14] Rousseau D., Dee H., Pridmore T. Imaging Methods for Phenotyping of Plant Traits. In Phenomics in Crop Plants: Trends, Options and Limitations. (ed. Kumar S.) 61-74 (Springer India, 2015).

[15] Tsaftaris, S. A., Minervini, M. & Scharr, H. Machine learning for plant phenotyping needs image processing. Trends Plant Sci. 21, 989-991 (2016).

[16] Burridge, J., Jochua, C. N., Bucksch, A. & Lynch, J. P. Legume shovelomics: High-Throughput phenotyping of common bean (Phaseolus vulgaris L.) and cowpea (Vigna unguiculata subsp, unguiculata) root architecture in the field . F. Crop. Res. 192, 21-32 (2016).

[17] Bradley, D. et al. Evolution of flower color pattern through selection on regulatory small RNAs. Science 358, 925-928 (2017).

[18] Angermueller, C., Pärnamaa, T., Parts, L. & Stegle, O. Deep learning for computational biology. Mol. Syst. Biol. 12, 878 (2016).

[19] Großkinsky, D. K. et al. Phenotyping in the fields: Dissecting the genetics of quantitative traits and digital farming. New Phytol. 207, 950-952 (2015).

[20] Zaman-Allah, M. et al. Unmanned aerial platform-based multi-spectral imaging for field phenotyping of maize. Plant Methods 11, 35 (2015).

[21] Simms, D. M., Waine, T. W., Taylor, J. C. & Juniper, G. R. The application of time-series MODIS NDVI profiles for the acquisition of crop information across Afghanistan. Int J. Remote Sens. 35, 6234-6254 (2014).

[22] Pound, M. P. et al. Deep machine learning provides state-of-the-art performance in image-based plant phenotyping. Gigascience 6, 1-10 (2017).

[23] Rosebrock A. Deep Learning for Computer Vision with Python . https://doi.org/10.1017/CBO9781107415324.004 (2017).

[24] Reza, A. M. Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement. J. VLSI Signal Process Syst. Signal Image Video Technol. 38, 35-44 (2004).

[25] Shipman J. W. Tkinter 8.5 reference: a GUI for Python. New Mexico. http://www.nmt.edu/tcc/help/pubs/tkinter/ (2013).

[26] Zhou, J. et al. Leaf-GP: An open and automated software application for measuring growth phenotypes for arabidopsis and wheat. Plant Methods 13, 1-31 (2017).

[27] Alex K., Ilya S., Geoffrey E. H. ImageNet Classification with Deep Convolutional Neural Networks. In: Neural information processing systems. Neural Information Processing Systems (NIPS): Lake Tahoe, Nevada, 2012, pp 1097-1105.

[28] Choromanska A., Henaff M., Mathieu M., Arous G Ben & LeCun Y. The Loss Surface of Multilayer Networks. arXiv14120233 [cs] (2014).

[29] Matsugu, M., Mori, K., Mitari, Y. & Kaneda, Y. Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw. 16, 555-559 (2003).

[30] Neubeck, A. & Van Gool, L. Efficient non-maximum suppression. Proc. Int. Conf. Pattern Recognit. 3, 850-855 (2006).

[31] Hazan, E. Introduction to online convex optimization. Found. Trends Optim. 2, 151-166 (2015).

[32] Hunter, J. D. Matplotlib: a 2D graphics environment. Comput Sci. Eng. 9, 90-95 (2007).

[33] Furbank, R. T. et al. Phenomics - technologies to relieve the phenotyping bottleneck. Trends Plant Sci. 16, 635-644 (2011).

[34] Tardieu, F., Cabrera-Bosquet, L., Pridmore, T. & Bennett, M. Plant phenomics, from sensors to knowledge. Curr. Biol. 27, R770-R783 (2017).

[35] ZHOU, J. et al. Plant phenomics: history, present status and challenges. J. Nanjing Agric Univ. 41, 580-588 (2018).

[36] Cendrero-Mateo M. P., et al. Field phenotyping: challenges and opportunities. Terr. Ecosyst. Res. Infrastruct. 53-80 (2017).

[37] Galinato, S. P. & Miles, C. A. Economic profitability of growing lettuce and tomato in western washington under high tunnel and open-field production systems. HortTechnology 23, 453-461 (2013).

[38] Mulderij R. OVERVIEW GLOBAL LETTUCE MARKET. Tholen. http://www.freshplaza.com/article/163938/OVERVIEW-GLOBAL-LETTUCE-MARKET (2016).

[39] Shukla M., Jharkharia S. Agri-fresh produce supply chain management: A state-of-the-art literature review. https://doi.org/10.1108/01443571311295608 (2013).

[40] Lamsal, K., Jones, P. C. & Thomas, B. W. Harvest logistics in agricultural systems with multiple, independent producers and no on-farm storage. Comput. Ind. Eng. 91, 129-138 (2016).

[41] Schiefer, G. New technologies and their impact on the agri-food sector: an economists view. Comput. Electron. Agric. 43, 163-172 (2004).

[42] Singh, A., Ganapathysubramanian, B., Singh, A. K. & Sarkar, S. Machine learning for high-throughput stress phenotyping in plants. Trends Plant Sci. 21, 110-124 (2016).

[43] Pedregosa, F. et al. Scikit-learn: machine learning in python. J. Mach. Learn Res 12, 2825-2830 (2011).

[44] van der Walt, S. et al. Scikit-image: image processing in Python. PeerJ 2, 1-18 (2014).

[45] Reynolds, D. et al. CropSight: a scalable and open-source information management system for distributed plant phenotyping and IoT-based crop management. Gigascience 8, 1-35 (2019).

[46] Reynolds D., et al. What is cost-efficient phenotyping? Optimizing costs for different scenarios. Plant Sci. https://doi.org/10.1016/j.plantsci.2018.06.015 (2018).

[47] Neveu, P. et al. Methods Dealing with multi-source and multi-scale information in plant phenomics: the ontology-driven Phenotyping Hybrid Information System. New Phytol. 221, 588-601 (2018).

[48] Liu, S., Baret, F., Andrieu, B., Burger, P. & Hemmerlé, M. Estimation of wheat plant density at early stages using high resolution imagery. Front Plant Sci. 8, 1-10 (2017).

[49] Frasconi, C. et al. An automatic machine able to perform variable rate application of flame weeding: design and assembly. Chem. Eng. Trans. 58, 301-306 (2017).

[50] Brann D., Extension Grain Specialist, Soil Environmental Sciences, Virginia Technology. A Comprehensive Approach Precision Farming: Management is the KEY. Virginia Cooperative Extension (2009).