EasyDAM_V4: Guided-GAN-based cross-species data labeling for fruit detection with significant shape difference

Abstract

Traditional agriculture is gradually being combined with artificial intelligence technology. High-performance fruit detection technology is an important basic technology in the practical application of modern smart orchards and has great application value. At this stage, fruit detection models need to rely on a large number of labeled datasets to support the training and learning of detection models, resulting in higher manual labeling costs. Our previous work uses a generative adversarial network to translate the source domain to the target fruit images. Thus, automatic labeling is performed on the actual dataset in the target domain. However, the method still does not achieve satisfactory results for translating fruits with significant shape variance. Therefore, this study proposes an improved fruit automatic labeling method, EasyDAM_V4, which introduces the Across-CycleGAN fruit translation model to achieve spanning translation between phenotypic features such as fruit shape, texture, and color to reduce domain differences effectively. We validated the proposed method using pear fruit as the source domain and three fruits with large phenotypic differences, namely pitaya, eggplant, and cucumber, as the target domain. The results show that the EasyDAM_V4 method achieves substantial cross-fruit shape translation, and the average accuracy of labeling reached 87.8, 87.0, and 80.7% for the three types of target domain datasets, respectively. Therefore, this research method can improve the applicability of the automatic labeling process even if significant shape variance exists between the source and target domain.

Acknowledgements

This study was partially supported by the National Natural Science Foundation of China (NSFC) Program 62276009 and the Japan Science and Technology Agency (JST) AIP Acceleration Research JPMJCR21U3.

Author contributions

W.Z., Y.L., and W.G. conceived the ideas and designed the methodology; W.Z., Y.L., and W.G. implemented the technical pipeline, conducted the experiments, and analyzed the results; Y.L. and C.W. analyzed the data with input from W.Z. and W.G.; C.W., C.Z., and G.C. conducted the supplementary experiment. All authors discussed and wrote the manuscript and provided final approval for publication.

Data availability

The dataset and pretrained model used in this paper are available at https://github.com/I3-Laboratory/EasyDAM_dataset.

Conflict of interest

The authors declare that they have no conflicts of interest.

Supplementary data

Supplementary data is available at Horticulture Research online.

References

[1] Boquete MT, Muyle A, Alonso C. Plant epigenetics: phenotypic and functional diversity beyond the DNA sequence. Am J Bot. 2021; 108 :553-8

[2] Muthukumar P, Jat GS, Kalia P. et al. Morphological characterization and screening of Solanum habrochaites accessions for late blight ( Phytophthora infestans ) disease resistance. Genet Resour Crop Evol. 2023;1-9

[3] Munaweera TIK, Jayawardana NU, Rajaratnam R. et al. Modern plant biotechnology as a strategy in addressing climate change and attaining food security. Agric Food Sec. 2022; 11 :1-28

[4] Sun G, Lu H, Zhao Y. et al. AirMeasurer: open-source software to quantify static and dynamic traits derived from multiseason aerial phenotyping to empower genetic mapping studies in rice. New Phytol. 2022; 236 :1584-604

[5] Zhu Y, Sun G, Ding G. et al. Large-scale field phenotyping using backpack LiDAR and CropQuant-3D to measure structural variation in wheat. Plant Physiol. 2021; 187 :716-38

[6] Zahir SADM, Omar AF, Jamlos MF. et al. A review of visible and near-infrared (Vis-NIR) spectroscopy application in plant stress detection. Sensors Actuators A Phys. 2022; 338 :113468

[7] Zhang W, Chen K, Wang J. et al. Easy domain adaptation method for filling the species gap in deep learning-based fruit detection. Hortic Res. 2021; 8 :119

[8] Zhang W, Chen K, Zheng C. et al. EasyDAM_V2: efficient data labeling method for multishape, cross-species fruit detection. Plant Phenomics. 2022; 2022 :9761674

[9] Zhang W, Liu Y, Zheng C. et al. EasyDAM_V3: automatic fruit labeling based on optimal source domain selection and data synthesis via a knowledge graph. Plant Phenomics. 2023; 5 :0067

[10] Shamsolmoali P, Zareapoor M, Granger E. et al. Image synthesis with adversarial networks: a comprehensive survey and case studies. Inf Fusion. 2021; 72 :126-46

[11] Mo S, Cho M, Shin J. InstaGAN: Instance-aware image-to-image translation. Proc Int Conf Learn Represent (ICLR). 2018:1-70

[12] Chen Y, Xia S, Zhao J. et al. Appearance and shape based image synthesis by conditional variational generative adversarial network. Knowl Based Syst. 2020; 193 :105450

[13] Roy P, Häni N, Isler V. Semantics-aware image to image translation and domain transfer. arXiv preprint arXiv:1904.02203. 2019

[14] Chen Z, Kim VG, Fisher M. et al. DECOR-GAN: 3D shape detailization by conditional refinement. In: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 15735-44

[15] Wu R. Geometry-aware image-to-image translation. Dissertation,. The Chinese University of Hong Kong; 2021:

[16] Li R, Li X, Hui K-H. et al. SP-GAN: sphere-guided 3D shape generation and manipulation. ACM Trans Graphics. 2021; 40 :151

[17] Zhang J, Hou J. Unpaired image-to-image translation network for semantic-based face adversarial examples generation. In: Proceedings of the 2021 Symposium on Great Lakes Symposium on VLSI. 2021, 449-54

[18] Gokaslan A, Ramanujan V, Ritchie D. et al. Improving shape deformation in unsupervised image-to-image translation. In: Proceedings of the European Conference on Computer Vision (ECCV). 2018:649-65

[19] Huang S, He C, Cheng R. SoloGAN: multi-domain multimodal unpaired image-to-image translation via a single generative adversarial network. IEEE Trans Artif Intell. 2022; 3 :722-37

[20] Hedjazi MA, Genc Y. Efficient texture-aware multi-GAN for image inpainting. Knowl Based Syst. 2021; 217 :106789

[21] Hu X. Multi-texture GAN: exploring the multi-scale texture translation for brain MR images. arXiv preprint arXiv:2102.07225. 2021

[22] Karras T, Laine S, Aila T. A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019:4401-10

[23] Karras T, Laine S, Aittala M. et al. Analyzing and improving the image quality of styleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:8110-9

[24] Karras T, Aittala M, Laine S. et al. Alias-free generative adversarial networks. Adv Neural Inf Proces Syst. 2021; 34 :852-63

[25] Wu X, Shao J, Gao L. et al. Unpaired image-to-image translation from shared deep space. In: 25th IEEE International Conference on Image Processing (ICIP). 2018:2127-31

[26] Johnson J, Alahi A, Fei-Fei L. Perceptual losses for real-time style transfer and super-resolution. In: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer: Cham, 2016,694-711

[27] Liu M-Y, Breuel T, Kautz J. Unsupervised image-to-image translation networks. Proc 31st Int Conf Neural Inf Process Sys. 2017:700-8

[28] Bergmann U, Jetchev N, Vollgraf R. Learning texture manifolds with the periodic spatial Gan. Proc Int Conf Mach Learn. 2017:469-77

[29] Shen Y, Yang C, Tang X. et al. InterFaceGAN: interpreting the disentangled face representation learned by GANs. IEEE Trans Pattern Anal Mach Intell. 2020; 44 :2004-18

[30] Sainburg T, Thielk M, Theilman B. et al.. Generative adversarial interpolative autoencoding: Adversarial training on latent space interpolations encourage convex latent distributions. arXiv:1807.06650. 2018

[31] Chen X, Duan Y, Houthooft R. et al. InfoGAN: interpretable representation learning by information maximizing generative adversarial nets. Adv Neural Inf Proces Syst. 2016; 29, https://arxiv.org/abs/1606.03657

[32] Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell. 2013; 35 :1798-828

[33] Selvaraju RR, Cogswell M, Das A. et al. Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision. 2017:618-26.

[34] Sosa J, Buitrago L. A review of latent space models for social networks. Rev Colomb Estad. 2021; 44 :171-200

[35] Kim B, Lee KH, Xue L. et al. A review of dynamic network models with latent variables. Stat Surv. 2018; 12 :105

[36] Bojanowski P, Joulin A, Lopez-Paz D. et al.. Optimizing the latent space of generative networks. Proc Int Conf Mach Learn. 2018:600-9

[37] Ma S, Fu J, Chen CW. et al. DA-GAN: instance-level image translation by deep attention generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018:5657-66

[38] Shen Z, Huang M, Shi J. et al. Towards instance-level image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019:3683-92

[39] Wang Z, Simoncelli EP, Bovik AC. Multiscale structural similarity for image quality assessment. In: The Thirty-Seventh IEEE Asilomar Conference on Signals, Systems and Computers. 2003; 2 :1398-402

[40] Thompson A. Fruits-360 dataset. https://www.kaggle.com/moltean/fruits, 2017.(18 May 2020, date last accessed)

[41] Minut M-D. Fruits-262 dataset: a dataset containing a vast majority of the popular and known fruits. 2021.(25 May 2021, date last accessed)

[42] Zhou X, Wang D, Krähenbühl P. Objects as points. arXiv 2019

[43] Tian Z, Shen C, Chen H. et al. FCOS: fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019:9627-36

[44] Ge Z, Liu S, Wang F. et al.. YOLOX: exceeding YOLO series in 2021. arXiv preprint arXiv:2107.08430 2021

[45] Singh B, Dhinakaran DP, Vijai C. et al. Artificial intelligence in agriculture. J Surv Fish Sci. 2023; 10 :6601-11

[46] Hu F, Lin C, Peng J. et al. Rapeseed leaf estimation methods at field scale by using terrestrial LiDAR point cloud. Agronomy. 2022; 12 :2409

[47] Meshram V, Patil K, Meshram V. et al. Machine learning in agriculture domain: a state-of-art survey. Artif Intell Life Sci. 2021; 1 :100010

[48] Montoya-Cavero L-E, de León D, Torres R. et al. Vision systems for harvesting robots: produce detection and localization. Comput Electron Agric. 2022; 192 :106562