Easy domain adaptation method for filling the species gap in deep learning-based fruit detection

Abstract

Fruit detection and counting are essential tasks for horticulture research. With computer vision technology development, fruit detection techniques based on deep learning have been widely used in modern orchards. However, most deep learning-based fruit detection models are generated based on fully supervised approaches, which means a model trained with one domain species may not be transferred to another. There is always a need to recreate and label the relevant training dataset, but such a procedure is time-consuming and labor-intensive. This paper proposed a domain adaptation method that can transfer an existing model trained from one domain to a new domain without extra manual labeling. The method includes three main steps: transform the source fruit image (with labeled information) into the target fruit image (without labeled information) through the CycleGAN network; Automatically label the target fruit image by a pseudo-label process; Improve the labeling accuracy by a pseudo-label self-learning approach. Use a labeled orange image dataset as the source domain, unlabeled apple and tomato image dataset as the target domain, the performance of the proposed method from the perspective of fruit detection has been evaluated. Without manual labeling for target domain image, the mean average precision reached 87.5% for apple detection and 76.9% for tomato detection, which shows that the proposed method can potentially fill the species gap in deep learning-based fruit detection.

References

[1] L. Jian, Z. Mingrui, and G. Xifeng, A fruit detection algorithm based on r-fcn in natural scene. In Proc. Chinese Control And Decision Conference (CCDC), 487-492, (IEEE, 2020).

[2] Ge, Y., Xiong, Y. & From, P. J. Symmetry-based 3d shape completion for fruit localisation for harvesting robots. Biosyst. Eng. 197, 188-202 (2020).

[3] Gené-Mola, J. et al. Fruit detection, yield prediction and canopy geometric characterization using lidar with forced air flow. Comput. Electron. Agric. 168, 105121 (2020).

[4] Wang, Z., Walsh, K. & Koirala, A. Mango fruit load estimation using a video-based mangoyolo-Kalman filter-Hungarian algorithm method. Sensors 19, 2742 (2019).

[5] Yu, Y., Zhang, K., Yang, L. & Zhang, D. Fruit detection for strawberry harvesting robot in non-structural environment based on mask-rcnn. Comput. Electron. Agric. 163, 104846 (2019).

[6] Dai, N., Xie, H., Yang, X., Zhan, K. and Liu, J. Recognition of cutting region for pomelo picking robot based on machine vision. In Proc. ASABE Annual International Meeting, p. 1, American Society of Agricultural and Biological Engineers, (2019).

[7] Liu, W. et al, Ssd: single shot multibox detector. In Proc. European Conference on Computer Vision, 21-37, Springer, 2016.

[8] Redmon, J. Divvala, S., Girshick, R. and Farhadi, A. You only look once: Unified, real-time object detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 779-788, (2016).

[9] Redmon, J. and Farhadi, A. Yolo9000: better, faster, stronger. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 7263-7271, (2017).

[10] Farhadi, A., Redmon, J. Yolov3: An incremental improvement[J]. Computer Vision and Pattern Recognition, (2018).

[11] Girshick, R. Donahue, J., Darrell, T. and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. 580-587, (2014).

[12] Wang, X., Shrivastava, A. and Gupta, A. A-fast-rcnn: hard positive generation via adversary for object detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2606-2615, (2017).

[13] Ren, S. He, K. Girshick, R. and Sun, J. Faster r-cnn: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 91-99, (2015).

[14] Balasubramanian, V.N., Guo, W., Chandra, A.L. and Desai, S.V. Computer vision with deep learning for plant phenotyping in agriculture: a survey. Adv. Comput. Commun., 1-26, (2020).

[15] Russakovsky, O. et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211-252 (2015).

[16] Mu, Y., Chen, T.- S., Ninomiya, S. & Guo, W. Intact detection of highly occluded immature tomatoes on plants using deep learning techniques. Sensors 20, 2984 (2020).

[17] Bilen, H., Pedersoli, M. & Tuytelaars, T. Weakly supervised object detection with posterior regularization. Proc. BMVC 2014, 1-12 (2014).

[18] Bilen, H., Pedersoli, M. and Tuytelaars, T. Weakly supervised object detection with convex clustering. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 1081-1089, (2015).

[19] Bilen, H. and Vedaldi, A. Weakly supervised deep detection networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2846-2854, (2016).

[20] Cinbis, R. G., Verbeek, J. & Schmid, C. Weakly supervised object localization with multi-fold multiple instance learning. IEEE Trans. pattern Anal. Mach. Intell. 39, 189-203 (2016).

[21] Papadopoulos, D.P. Uijlings, J.R., Keller, F. and Ferrari, V. Training object class detectors with click supervision. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 6374-6383, (2017).

[22] Bellocchio, E., Ciarfuglia, T. A., Costante, G. & Valigi, P. Weakly supervised fruit counting for yield estimation using spatial consistency. IEEE Robot. Autom. Lett. 4, 2348-2355 (2019).

[23] Bellocchio, E., Costante, G., Cascianelli, S., Fravolini, M. L. & Valigi, P. Combining domain adaptation and spatial consistency for unseen fruits counting: a quasi-unsupervised approach. IEEE Robot. Autom. Lett. 5, 1079-1086 (2020).

[24] Lu, H., Cao, Z., Xiao, Y., Zhuang, B. & Shen, C. Tasselnet: counting maize tassels in the wild via local counts regression network. Plant Methods 13, 79 (2017).

[25] Ghosal, S. et al. A weakly-supervised deep learning framework for Sorghum head detection and counting. Plant Phenomics 2019, 1-14 (2019).

[26] Chandra, A. L., Desai, S. V., Balasubramanian, V. N., Ninomiya, S. & Guo, W. Active learning with point supervision for cost-effective panicle detection in cereal crops. Plant Methods 16, 1-16 (2020).

[27] Settles, B. “ Active learning literature survey,” tech. rep., University of Wisconsin-Madison Department of Computer Sciences, 2009.

[28] Huang, S.-J., Jin, R. and Zhou, Z.-H. Active learning by querying informative and representative examples. In Advances in Neural Information Processing Systems, 892-900, (2010).

[29] Wachs, J. P., Stern, H. I., Burks, T. & Alchanatis, V. Low and high-level visual feature-based apple detection from multi-modal images. Precis. Agric. 11, 717-735 (2010).

[30] Dubey, S.R., Dixit, P., Singh, N., and Gupta, J.P. Infected fruit part detection using k-means clustering segmentation technique. (2013).

[31] Zhang, P. & Xu, L. Unsupervised segmentation of greenhouse plant images based on statistical method. Sci. Rep. 8, 1-13 (2018).

[32] Sa, I. et al. Deepfruits: a fruit detection system using deep neural networks. Sensors 16, 1222 (2016).

[33] Bargoti, S. and Underwood, J. Deep fruit detection in orchards. In Proc. IEEE International Conference on Robotics and Automation (ICRA), 3626-3633, (IEEE, 2017).

[34] Muresan, H. & Oltean, M. Fruit recognition from images using deep learning. Acta Univ. Sapientiae, Inform. 10, 26-42 (2018).

[35] Koirala, A., Walsh, K., Wang, Z. & McCarthy, C. Deep learning for real-time fruit detection and orchard fruit load estimation: bBenchmarking of ‘mangoyolo’. Precis. Agric. 20, 1107-1135 (2019).

[36] Kestur, R., Meduri, A. & Narasipura, O. Mangonet: a deep semantic segmentation architecture for a method to detect and count mangoes in an open orchard. Eng. Appl. Artif. Intell. 77, 59-69 (2019).

[37] Häni, N., Roy, P. & Isler, V. Minneapple: a benchmark dataset for apple detection and segmentation. IEEE Robot. Autom. Lett. 5, 852-858 (2020).

[38] Goodfellow, I. et al, Generative adversarial nets. In Advances in Neural Information Processing Systems, 2672-2680, (2014).

[39] Stein, G. J., & Roy, N. Genesis-rt: generating synthetic images for training secondary real-world tasks. In Proc. IEEE International Conference on Robotics and Automation (ICRA). 7151-7158, (2018).

[40] Zhang, J. et al. Vr-goggles for robots: Real-to-sim domain adaptation for visual control. IEEE Robot. Autom. Lett. 4, 1148-1155 (2019).

[41] Roy, P., Häni, N., & Isler, V. Semantics-aware image to image translation and domain transfer. arXiv preprint arXiv:1904.02203.(2019).

[42] Valerio Giuffrida, M., Dobrescu, A., Doerner, P., & Tsaftaris, S. A. (2019). Leaf counting without annotations using adversarial unsupervised domain adaptation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019).

[43] Zhu, J.Y., Park, T., Isola, P. and Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. IEEE International Conference on Computer Vision, 2223-2232, (2017).

[44] Wang, J. et al, A deep learning-based in-field fruit counting method using video sequences. https://www.plant-phenotyping.org/CVPPP2020-Programme.

[45] Liang, Q. et al, A real-time detection framework for ontree mango based on ssd network. In Proc. International Conference on Intelligent Robotics and Applications, 423-436, (Springer, 2018).

[46] Tsironis, V., Bourou, S. & Stentoumis, C. Tomatod: evaluation of object detection algorithms on a new real-world tomato dataset. Int. Arch. Photogramm., Remote Sens. Spat. Inform. Sci. 43, 1077-1084 (2020).