Deep learning image segmentation and extraction of blueberry fruit traits associated with harvestability and yield

Abstract

Fruit traits such as cluster compactness, fruit maturity, and berry number per clusters are important to blueberry breeders and producers for making informed decisions about genotype selection related to yield traits and harvestability as well as for plant management. The goal of this study was to develop a data processing pipeline to count berries, to measure maturity, and to evaluate compactness (cluster tightness) automatically using a deep learning image segmentation method for four southern highbush blueberry cultivars (‘Emerald’, ‘Farthing’, ‘Meadowlark’, and ‘Star’). An iterative annotation strategy was developed to label images that reduced the annotation time. A Mask R-CNN model was trained and tested to detect and segment individual blueberries with respect to maturity. The mean average precision for the validation and test dataset was 78.3% and 71.6% under 0.5 intersection over union (IOU) threshold, and the corresponding mask accuracy was 90.6% and 90.4%, respectively. Linear regression of the detected berry number and the ground truth showed an R² value of 0.886 with a root mean square error (RMSE) of 1.484. Analysis of the traits collected from the four cultivars indicated that ‘Star’ had the fewest berries per clusters, ‘Farthing’ had the least mature fruit in mid-April, ‘Farthing’ had the most compact clusters, and ‘Meadowlark’ had the loosest clusters. The deep learning image segmentation technique developed in this study is efficient for detecting and segmenting blueberry fruit, for extracting traits of interests related to machine harvestability, and for monitoring blueberry fruit development.

References

[1] Sargent, S. A. et al. Harvest of southern highbush blueberry with a modified, over-the-row mechanical harvester: use of handheld shakers and soft catch surfaces. Agriculture 10, 4 (2020).

[2] Gallardo, R. K. & Zilberman, D. The economic feasibility of adopting mechanical harvesters by the highbush blueberry industry. HortTechnology 26, 299-308 (2016).

[3] Brown, G. K. et al. Estimates of mechanization effects on fresh blueberry quality. Appl. Eng. Agric. 12, 21-26 (1996).

[4] Takeda, F. et al. Assessment of the V45 blueberry harvester on rabbiteye blueberry and southern highbush blueberry pruned to V-shaped canopy. HortTechnology 18, 130-138 (2008).

[5] Ballington, J . Germplasm resources available to meet future needs for blueberry cultivar improvement. Fruit. Var. J. 44, 54-62 (1990).

[6] International Organisation of Vine and Wine, Compendium of international methods of wine and must analysis. International Organisation of Vine and Wine. (Paris, France, 2015).

[7] Tello, J. & Forneck, A. A double-sigmoid model for grapevine bunch compactness development. OENO One 52, 307-316 (2018). https://oeno-one.eu/article/view/2132#.

[8] Cubero, S. et al. A new method for assessment of bunch compactness using automated image analysis. Aust. J. grape wine Res. 21, 101-109 (2015).

[9] Rist, F. et al. High-precision phenotyping of grape bunch architecture using fast 3D sensor and automation. Sensors 18, 763 (2018).

[10] Hed, B., Ngugi, H. K. & Travis, J. W. Relationship between cluster compactness and bunch rot in Vignoles grapes. Plant Dis. 93, 1195-1201 (2009).

[11] Schwartze, C. D. & Myhre, A. S. Growing Blueberries in the Puget Sound Region of Washington (State College of Washington, Institute of Agricultural Sciences, Washington, 1954).

[12] Lyrene, P. Breeding southern highbush blueberries. Plant Breed. Rev. 30, 353-414 (2007).

[13] Brightwell, W. T. A Comparison of the Ethel and Walker Varieties as Parents in Blueberry Breeding. (The Ohio State University, 1956).

[14] Gilbert, J. L. et al. Potential for increasing southern highbush blueberry flavor acceptance by breeding for major volatile components. HortScience 48, 835-843 (2013).

[15] El Agamy, S., Aly, M. & Biggs, R. Fruit maturity as related to ethylene in ‘Delite’ blueberry. Proc Fla State Hortic Soc 95, 245-246 (1982).

[16] Kalt, W., McRae, K. & Hamilton, L. Relationship between surface color and other maturity indices in wild lowbush blueberries. Can. J. Plant Sci. 75, 485-490 (1995).

[17] Yang, C., Lee, W. S. & Gader, P. Hyperspectral band selection for detecting different blueberry fruit maturity stages. Comput. Electron. Agric. 109, 23-31 (2014).

[18] Yang, C., Lee, W. S. & Williamson, J. G. Classification of blueberry fruit and leaves based on spectral signatures. Biosyst. Eng. 113, 351-362 (2012).

[19] LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436-444 (2015).

[20] Papandreou, G. et al. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. 2015 IEEE International Conference on Computer Vision (ICCV). pp. 1742-1750 (Santiago, Chile, 2015).

[21] Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3431-3440 (Boston, MA, USA, 2015).

[22] Ronneberger, O., P. Fischer, and T. Brox . U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. (Springer, 2015).

[23] He, K., et al. Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV). pp. 2980-2988 (Venice, Italy, 2017).

[24] Ren, S. et al. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 1137-1149 (2017).

[25] Chen, L.-C. et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell 40, 834-848 (2017).

[26] Chen, L.-C. et al. Rethinking atrous convolution for semantic image segmentation. arXiv: 1706.05587 (2017).

[27] Chen, L.-C. et al. Encoder-decoder with atrous separable convolution for semantic image segmentation. arXiv: 1802.02611 (2018).

[28] Wu, H. et al. FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation. arXiv: 1903.11816 (2019).

[29] Zhu, Y. et al. Improving semantic segmentation via video propagation and label relaxation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8848-8857 (Long Beach, CA, USA, 2019).

[30] Milella, A. et al. In-field high throughput grapevine phenotyping with a consumer-grade depth camera. Computers Electron. agriculture 156, 293-306 (2019).

[31] Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556. (2014)

[32] Santos, T. T. et al. Grape detection, segmentation and tracking using deep neural networks and three-dimensional association. Computers and Electronics in Agriculture 170, 2020 (2020).

[33] Redmon, J. et al., You only look once: unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 779-788 (Las Vegas, NV, USA, 2016).

[34] Redmon, J. & Farhadi, A. YOLOv3: An Incremental Improvement. https://arxiv.org/abs/1804.02767 (2018).

[35] Zabawa, L. et al. Detection of single grapevine berries in images using fully convolutional neural networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 2571-2579 (Long Beach, CA, USA, 2019).

[36] Grimm, J. et al. An adaptable approach to automated visual detection of plant organs with applications in grapevine breeding. Biosyst. Eng. 183, 170-183 (2019).

[37] Tan, Y., Q. Hu, & Z. Du . Batch mode active learning with nonlocal self-similarity prior for semantic segmentation. In 2019 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2019).

[38] Yao, X. et al. Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Trans. Geosci. Remote Sens. 54, 3660-3671 (2016).

[39] Olmstead, J. W. & Finn, C. E. Breeding highbush blueberry cultivars adapted to machine harvest for the fresh market. HortTechnology 24, 290-294 (2014).

[40] Sargent, S. A. et al. Fruit detachment force of southern highbush blueberry: an aid to selection of cultivars suitable for mechanical harvest. In HortScience (2010).

[41] Tan, K. et al. Recognising blueberry fruit of different maturity using histogram oriented gradients and colour features in outdoor scenes. Biosyst. Eng. 176, 59-72 (2018).

[42] Chen, X. et al. New approach of simultaneous, multi‐perspective imaging for quantitative assessment of the compactness of grape bunches. Aust. J. Grape Wine Res. 24, 413-420 (2018).

[43] Lin, T.-Y. et al. Microsoft COCO: Common objects in context (Springer, 2014).

[44] Everingham, M. et al. The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88, 303-338 (2010).