Deep-learning-based in-field citrus fruit detection and tracking

Abstract

Fruit yield estimation is crucial for establishing fruit harvest and marketing strategies. Recently, computer vision and deep learning techniques have been used to estimate citrus fruit yield and have exhibited notable fruit detection ability. However, computer-vision-based citrus fruit counting has two key limitations: inconsistent fruit detection accuracy and double-counting of the same fruit. Using oranges as the experimental material, this paper proposes a deep-learning-based orange counting algorithm using video sequences to help overcome these problems. The algorithm consists of two sub-algorithms, OrangeYolo for fruit detection and OrangeSort for fruit tracking. The OrangeYolo backbone network is partially based on the YOLOv3 algorithm, which has been improved upon to detect small objects (fruits) at multiple scales. The network structure was adjusted to detect small-scale targets while enabling multiscale target detection. A channel attention and spatial attention multiscale fusion module was introduced to fuse the semantic features of the deep network with the shallow textural detail features. OrangeYolo can achieve mean Average Precision (mAP) values of 0.957 in the citrus dataset, higher than the 0.905, 0.911, and 0.917 achieved with the YOLOv3, YOLOv4, and YOLOv5 algorithms. OrangeSort was designed to alleviate the double-counting problem associated with occluded fruits. A specific tracking region counting strategy and tracking algorithm based on motion displacement estimation were established. Six video sequences taken from two fields containing 22 trees were used as the validation dataset. The proposed method showed better performance (Mean Absolute Error (MAE) = 0.081, Standard Deviation (SD) = 0.08) than video-based manual counting and produced more accurate results than the existing standards Sort and DeepSort (MAE = 0.45 and 1.212; SD = 0.4741 and 1.3975).

References

[1] Anderson NT, Walsh KB, Wulfsohn D . Technologies for Forecasting Tree Fruit Load and Harvest Timing-from Ground. Agronomy (Basel). 2021; 11: 1409.

[2] Koirala A, Walsh KB, Wang Z . Attempting to estimate the unseen-correction for occluded fruit in tree fruit load estimation by machine vision with deep learning. Agronomy. 2021; 11: 347.

[3] Koirala A, Walsh KB, Wang Z et al. Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’. Precis Agric. 2019; 20: 1107-35.

[4] Apolo-Apolo OE, Martínez-Guanter J, Egea G et al. Deep learning techniques for estimation of the yield and size of citrus fruits using a UAV. Eur J Agron. 2020; 115: 126030.

[5] Mu Y, Chen TS, Ninomiya S et al. Intact detection of highly occluded immature tomatoes on plants using deep learning techniques. Sensors. 2020; 20: 2984.

[6] Bargoti S, Underwood J . Deep fruit detection in orchards[C]. In: 2017 IEEE International Conference on Robotics and Automation (ICRA).IEEE, 2017, 3626-33.

[7] Sa I, Ge Z, Dayoub F et al. Deepfruits: a fruit detection system using deep neural networks. Sensors. 2016; 16: 1222.

[8] Gan H, Lee WS, Alchanatis V et al. Active thermal imaging for immature citrus fruit detection. Biosyst Eng. 2020; 198: 291-303.

[9] Wan S, Goudos S . Faster R-CNN for multi-class fruit detection using a robotic vision system. Comput Netw. 2020; 168: 107036.

[10] Gao F, Fu L, Zhang X et al. Multi-class fruit-on-plant detection for apple in SNAP system using faster R-CNN. Comput Electron Agric. 2020; 176: 105634.

[11] Kestur R, Meduri A, Narasipura O . MangoNet: a deep semantic segmentation architecture for a method to detect and count mangoes in an open orchard. Eng Appl Artif Intell. 2019; 77: 59-69.

[12] Akiva P, Dana K, Oudemans P et al. Finding berries: segmentation and counting of cranberries using point supervision and shape priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020; 2020: 50-1.

[13] Gené-Mola J, Sanz-Cortiella R, Rosell-Polo JR et al. Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry. Comput Electron Agric. 2020; 169: 105165.

[14] Kang H, Chen C . Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Comput Electron Agric. 2020; 171: 105302.

[15] Chen SW, Shivakumar SS, Dcunha S et al. Counting apples and oranges with deep learning: a data-driven approach. IEEE Robotics and Automation Letters. 2017; 2: 781-8.

[16] Sun G, Wang X, Yang H et al. A canopy information measurement method for modern standardized apple orchards based on UAV multimodal information. Sensors. 2020; 20: 2985.

[17] Wang Z, Walsh K, Koirala A. Mango fruit load estimation using a video based MangoYOLO-Kalman filter-hungarian algorithm method. Sensors. 2019;19:2742.

[18] Roy P, Kislay A, Plonski PA et al. Vision-based preharvest yield mapping for apple orchards. Comput Electron Agric. 2019; 164: 104897.

[19] Liu X, Chen SW, Aditya S et al. Robust fruit counting: Combining deep learning, tracking, and structure from motion. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).IEEE, 2018, 1045-52.

[20] Liu X, Chen SW, Liu C et al. Monocular camera based fruit counting and mapping with semantic data association. IEEE Robotics and Automation Letters. 2019; 4: 2296-303.

[21] Gené-Mola J, Gregorio E, Cheein FA et al. Fruit detection, yield prediction and canopy geometric characterization using LiDAR with forced air flow. Comput Electron Agric. 2020; 168: 105121.

[22] Stein M, Bargoti S, Underwood J . Image based mango fruit detection, localisation and yield estimation using multiple view geometry. Sensors. 2016; 16: 1915.

[23] Gan H, Lee WS, Alchanatis V . A prototype of an immature citrus fruit yield mapping system. 2017 ASABE Annual International Meeting. American Society of Agricultural and Biological Engineers. 2017; 2017: 1.

[24] Redmon J, Farhadi A . Yolov3: an incremental improvement arXiv preprint arXiv:1804.02767. 2018.

[25] Luo W, Li Y, Urtasun R et al. Understanding the effective receptive field in deep convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016; 2016: 4905-13.

[26] Zhang S, Zhu X, Lei Z et al. S3fd: single shot scale-invariant face detector. Proceedings of the IEEE international conference on computer vision. 2017; 2017: 192-201.

[27] Redmon J, Farhadi A . YOLO9000: better, faster, stronger. IEEE Conference on Computer Vision & Pattern Recognition. 2017; 2017: 6517-25.

[28] Bewley A, Ge Z, Ott L et al. Simple online and realtime tracking. In: 2016 IEEE International Conference on Image Processing (ICIP).IEEE. 2016, 3464-8.

[29] Wojke N, Bewley A, Paulus D . Simple online and realtime tracking with a deep association metric. 2017 IEEE international conference on image processing (ICIP) IEEE. 2017; 2017: 3645-9.

[30] Liu S, Huang D, Wang Y . Learning spatial fusion for single-shot object detection arXiv preprint arXiv:1911.09516. 2019.

[31] Zhang Z, He T, Zhang H et al. Bag of freebies for training object detection neural networks arXiv preprint arXiv:1902.04103. 2019.

[32] Lin TY, Maire M, Belongie S et al. Microsoft coco: Common objects in context. In: European Conference on Computer Vision.Springer: Cham, 2014, 740-55.

[33] Lin TY, Dollár P, Girshick R et al. Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017; 2017: 2117-25.

[34] Liu S, Qi L, Qin H et al. Path aggregation network for instance segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018; 2018: 8759-68.

[35] Tan M, Pang R, Le QV . Efficientdet: scalable and efficient object detection[C]. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020; 2020: 10781-90.

[36] Hu J, Shen L, Sun G . Squeeze-and-excitation networks[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018; 2018: 7132-41.

[37] Woo S, Park J, Lee JY et al. Cbam: convolutional block attention module[C]. Computer Vision - ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII. 2018; 11211: 3-19.

[38] Milan A, Leal-Taixé L, Reid I et al. MOT16: a benchmark for multi-object tracking arXiv preprint arXiv:1603.00831. 2016.

[39] Wang J, Zhang W, Chen K et al. A deep learning-based in-field fruit counting method using video sequences https://www.plant-phenotyping.org/CVPPP2020-Programme. 2020.

[40] Bochkovskiy A, Wang CY, Liao HYM . Yolov4: optimal speed and accuracy of object detection arXiv preprint arXiv:2004.10934. 2020.

[41] Yun S, Han D, Oh SJ et al. Cutmix: regularization strategy to train strong classifiers with localizable features. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019; 2019: 6023-32.