
Design and control algorithm of a motion sensing-based fruit harvesting robot
Ziwen CHEN, Yuhang CHEN, Hui LI, Pei WANG
Design and control algorithm of a motion sensing-based fruit harvesting robot
● An optimized four-step inverse kinematic solution method ensures smooth and precise motion with minimal mechanical interference. | |
● The robot achieves a fast response time of 74.4 ms, with an average target-picking duration reduced to 6.5 seconds after operator training. | |
● The system simplifies the picking process using gesture recognition. |
In response to the demand of automatic fruit identification and harvesting, this paper presents a human-robot collaborative picking robot based on somatosensory interactive servo control. The robot system mainly consisted of four parts: picking execution mechanism, hand information acquisition system, human-machine interaction interface, and human-robot collaborative picking strategy. A six-degree-of-freedom robotic arm was designed as the picking execution mechanism. The D-H method is employed for both forward and inverse kinematic modeling of the robotic arm. A four-step inverse kinematic optimal solution selection method, including mechanical interference, correctness, rationality, and smoothness of motion, is proposed. The working principle and use of the Leap Motion controller for hand information acquisition were introduced. Data from three types of hand movements were collected and analyzed. Spatial mapping method between the Leap Motion interaction space and operating range of the robotic arm was proposed to achieve a direct correspondence between the cubic interaction box and the cylindrical space of the fan ring of the robotic arm. The test results demonstrated that the average response time of the double-click picking command was 332 ms. The average time consumption for somatosensory control targeting was 6.5 s. The accuracy rate of the picking gesture judgment was 96.7%.
Harvesting robots / human-machine interaction / human-robot collaboration / somatosensory control / Leap Motion controller
Tab.1 The fruit set and numbers of seeds and polyploids recovered from the 2x × 4x crosses |
Cross | No. pollinated flowers | No. fruits set | No. seeds obtained | No. seeds germinated | No. plantlets obtained | No. diploids | No. triploids | No. tetraploids | |||
---|---|---|---|---|---|---|---|---|---|---|---|
Dev. | Undev. | Dev. | Undev. | ||||||||
Orah × PCS | 210 | 79 | 323 | 930 | 115 | 271 | 145 | 37 | 90 | 18 | |
Orah × PO | 238 | 128 | 859 | 906 | 99 | 108 | 132 | 35 | 81 | 16 | |
Orah × SP | 263 | 80 | 490 | 511 | 101 | 87 | 88 | 75 | 11 | 2 | |
Total | 711 | 287 | 1672 | 2347 | 315 | 466 | 365 | 147 | 182 | 36 |
Fig.1 Embryo rescue, plant regeneration and transplantation for citrus triploid production. (a) Young fruits 85 d after pollination. (b) Germination of developed seeds after approximately two weeks of culturing in vitro on germination medium. (c) Germination of undeveloped seeds after about four weeks of culturing in vitro on germination medium. (d) Regeneration of shoots from embryoids after their transfer to the shoot-induction medium. (e) A shoot grafted in vitro to the rootstock (Poncirus trifoliata). (f) Transplanted seedlings in a greenhouse. |
Fig.2 Ploidy determination for regenerated citrus plantlets using flow cytometry and chromosome counting. (a–c) Histograms of diploid progeny (peak= 50), triploid progeny (peak= 75) and tetraploid progeny (peak= 100). (d–f) Chromosome counting for diploid (2n= 2x= 18), triploid (2n= 3x= 27) and tetraploid (2n= 4x= 36) plantlets. Scale bars= 5 mm. |
Fig.3 Determining the genetic origin of triploids and tetraploids using KASP genotyping and aa × bbbb type SNP markers. Genotyping plots of (a) 43 randomly selected triploid progeny and (b) 36 tetraploid progeny with SNP marker Chr2-25841537 demonstrating their hybrid origins. Green, blue, red and gray represent the genotypes of maternal parents, paternal parents, triploid or tetraploid progeny and negative controls, respectively. |
Tab.2 Genotypic analysis of nine SNP markers (aa × bbbb type) in the triploid and tetraploid hybrid populations |
SNP marker | Orah (aa) | Male parents (bbbb) | NI | abb | aabb |
---|---|---|---|---|---|
Chr2-24850985 | CC | AAAA | 79 | 43 | 36 |
Chr2-25841537 | GG | CCCC | 79 | 43 | 36 |
Chr3-18395328 | AA | GGGG | 78 | 43 | 35 |
Chr3-24832283 | CC | TTTT | 79 | 43 | 36 |
Chr4-8664085 | GG | CCCC | 77 | 42 | 35 |
Chr4-8689111 | GG | TTTT | 79 | 43 | 36 |
Chr5-12876197 | CC | TTTT | 79 | 43 | 36 |
Chr6-1932038 | TT | CCCC | 79 | 43 | 36 |
Chr9-815315 | AA | GGGG | 76 | 43 | 33 |
Note: NI, number of individuals genotyped; abb and aabb, number of individuals of each genotype. |
Fig.4 Determining the mechanism of 2n megagametophyte formation in the 36 tetraploids using KASP genotyping and ab × aaaa/bbbb type SNP markers. (a) Under pericenteomeric locus Chr5-17395118, the maternal genotype (green) is GA, the paternal genotype (blue) is GGGG, and the tetraploid plantlets (red) clustered with their parents; the genotypes of the tetraploids are GGAA and GGGG with a GG contribution from the paternal parent and therefore homozygous AA and GG for the 2n megagametophyte. (b) Under the centromere distal locus Chr5-24798525, the maternal genotype (green) is TC, the paternal genotype (blue) is TTTT, and the tetraploid plantlets (red) clustered into three groups; the genotypes of the tetraploids are TTCC, TTTT and TTTC with a TT contribution from the paternal parent and therefore homozygous CC, TT and TC for the 2n megagametophyte. |
Tab.3 Genotypes of 18 tetraploids from ‘Orah × PCS’ hybridization generated using eight pericentromeric SNP markers and nine centromere distal SNP markers |
SNP markers | Orah | Male parents | OPCS 1 | OPCS 2 | OPCS 3 | OPCS 4 | OPCS 5 | OPCS 6 | OPCS 7 | OPCS 8 | OPCS 9 | OPCS 10 | OPCS 11 | OPCS 12 | OPCS 13 | OPCS 14 | OPCS 15 | OPCS 16 | OPCS 17 | OPCS 18 | Het | PHR |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Chr1-12169985 | GA | GGGG | GG | AA | GG | GG | AA | AA | GG | GG | GG | AA | AA | GG | GG | AA | GG | AA | AA | GG | 0 | 0 |
Chr3-7913461 | GA | GGGG | GG | AA | GG | GG | AA | AA | GG | AA | AA | AA | AA | GG | AA | GG | AA | GG | GG | GG | 0 | 0 |
Chr5-17395118 | GA | GGGG | GG | AA | AA | GG | AA | GG | AA | AA | GG | AA | AA | GG | AA | GG | AA | GG | GG | AA | 0 | 0 |
Chr5-18735709 | TA | TTTT | TT | TT | AA | TT | TT | AA | TT | TT | TT | TT | TT | AA | TT | AA | TT | TT | TT | TT | 0 | 0 |
Chr6-5337355 | TG | TTTT | TT | TT | GG | GG | GG | TT | GG | TT | TT | TT | GG | GG | TT | TT | GG | GG | GG | TT | 0 | 0 |
Chr7-20190172 | CA | CCCC | CC | AA | CC | AA | AA | CC | AA | AA | CC | CC | CC | CC | CC | AA | CC | AA | AA | AA | 0 | 0 |
Chr8-7202641 | AG | AAAA | AA | AA | GG | AA | GG | AA | AA | AA | GG | GG | AA | GG | AA | AA | GG | GG | GG | GG | 0 | 0 |
Chr9-8606082 | GT | GGGG | TT | GG | GG | GG | TT | TT | GG | GG | GG | TT | GG | TT | TT | GG | TT | GG | TT | GG | 0 | 0 |
Chr5-1014992 | CA | CCCC | AA | CC | AA | AA | CC | AA | AA | AA | AA | AA | AA | AA | AA | CC | AA | CC | AA | AA | 0 | 0 |
Chr5-1051787 | AT | AAAA | TT | AA | AA | AA | AA | TT | TT | TT | AA | TT | TT | AA | AA | AA | AA | AA | AA | AA | 0 | 0 |
Chr5-1103777 | TC | TTTT | TC | TC | TC | TC | TT | TC | TC | TC | TC | TC | TC | CC | TC | TT | TC | TC | TC | TC | 15 | 83.33 |
Chr5-1323430 | CT | CCCC | TT | CC | CC | CC | CC | TT | TT | TT | TT | TT | TT | CC | CC | CC | CC | CC | CC | TT | 0 | 0 |
Chr5-1580076 | TC | TTTT | CC | TT | CC | CC | TT | CC | CC | CC | CC | CC | CC | CC | CC | TT | CC | TT | CC | TT | 0 | 0 |
Chr5-22348846 | GC | GGGG | GG | GG | GG | CC | CC | GG | CC | CC | CC | CC | GG | GG | GG | CC | GG | CC | GG | CC | 0 | 0 |
Chr5-24661722 | AG | AAAA | GG | AA | GG | GG | GG | GG | GG | GG | AA | GG | GG | GG | GG | AA | GG | AA | GG | AA | 0 | 0 |
Chr5-24798525 | TC | TTTT | TT | TT | TT | TC | TC | TC | TC | CC | TT | TT | TC | TC | TT | TC | TC | CC | TT | TT | 8 | 44.44 |
Chr5-26120337 | CT | CCCC | CC | CC | TT | TT | CC | TT | TT | TT | TT | CC | TT | TT | TT | CC | TT | CC | TT | CC | 0 | 0 |
Het | 1 | 1 | 1 | 2 | 1 | 2 | 2 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | ||||
PHR | 5.88 | 5.88 | 5.88 | 11.76 | 5.88 | 11.76 | 11.76 | 5.88 | 5.88 | 5.88 | 11.76 | 5.88 | 5.88 | 5.88 | 11.76 | 5.88 | 5.88 | 5.88 |
Tab.4 Genotypes of 18 tetraploids from ‘Orah × PO’ and ‘Orah × SP’ hybridizations generated using eight pericentromeric SNP markers and nine centromere distal SNP markers |
SNP markers | Orah | Male parents | OPO1 | OPO2 | OPO3 | OPO4 | OPO5 | OPO6 | OPO7 | OPO8 | OPO9 | OPO10 | OPO11 | OPO12 | OPO13 | OPO14 | OPO15 | OPO16 | OSP1 | OSP2 | Het | PHR |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Chr1-12169985 | GA | GGGG | GG | GG | AA | GG | AA | AA | AA | AA | GG | GG | AA | AA | GG | GG | AA | GG | AA | AA | 0 | 0 |
Chr3-7913461 | GA | GGGG | AA | AA | GG | AA | GG | GG | GG | AA | GG | GG | GG | AA | AA | GG | GG | AA | GG | GG | 0 | 0 |
Chr5-17395118 | GA | GGGG | AA | AA | GG | AA | GG | AA | GG | AA | AA | GG | GG | AA | GG | AA | AA | GG | AA | AA | 0 | 0 |
Chr5-18735709 | TA | TTTT | AA | AA | TT | AA | TT | AA | AA | AA | TT | TT | TT | AA | TT | AA | TT | TT | TT | TT | 0 | 0 |
Chr6-5337355 | TG | TTTT | TT | TT | TT | GG | TT | GG | GG | TT | TT | GG | TT | GG | GG | GG | TT | GG | GG | GG | 0 | 0 |
Chr7-20190172 | CA | CCCC | CC | CC | CC | AA | CC | CC | AA | CC | CC | AA | CC | CC | CC | AA | AA | AA | CC | CC | 0 | 0 |
Chr8-7202641 | AG | AAAA | AA | AA | AA | AA | GG | AA | GG | AA | AA | AA | AA | GG | GG | AA | AA | GG | GG | GG | 0 | 0 |
Chr9-8606082 | GT | GGGG | GG | GG | GG | TT | TT | GG | GG | GG | TT | TT | TT | GG | GG | GG | TT | GG | GG | GG | 0 | 0 |
Chr5-1014992 | CA | CCCC | CC | AA | AA | CC | CC | CC | CC | CC | CC | AA | AA | CC | AA | CC | AA | CC | CC | AA | 0 | 0 |
Chr5-1051787 | AT | AAAA | AA | AA | AA | AA | AA | AA | AA | AA | TT | AA | AA | AA | AA | AA | AA | AA | AA | AA | 0 | 0 |
Chr5-1103777 | TC | TTTT | TC | TC | TC | TC | TC | TC | TC | TC | TC | TC | TC | TC | TC | TC | TC | TT | TT | TC | 16 | 88.89 |
Chr5-1323430 | CT | CCCC | CC | CC | CC | CC | CC | CC | CC | CC | CC | TT | CC | CC | CC | CC | CC | CC | CC | CC | 0 | 0 |
Chr5-1580076 | TC | TTTT | TT | CC | CC | TT | TT | TT | TT | TT | TT | CC | CC | TT | CC | TT | CC | TT | TT | CC | 0 | 0 |
Chr5-22348846 | GC | GGGG | GG | CC | CC | GG | GG | GG | GG | GG | GG | CC | GG | GG | GG | GG | CC | CC | CC | GG | 0 | 0 |
Chr5-24661722 | AG | AAAA | AA | GG | AA | AA | AA | AA | AA | AA | AA | AA | GG | AA | GG | AA | GG | GG | GG | GG | 0 | 0 |
Chr5-24798525 | TC | TTTT | TT | TC | CC | TT | TT | TT | TT | TT | TT | CC | TT | TT | TT | TT | TC | TC | TC | TC | 5 | 27.78 |
Chr5-26120337 | CT | CCCC | CC | TT | CC | CC | CC | CC | CC | CC | CC | TT | TT | CC | TT | CC | CC | TT | CC | CC | 0 | 0 |
Het | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 2 | ||||
PHR | 5.88 | 11.76 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 5.88 | 11.76 | 5.88 | 5.88 | 11.76 |
[1] |
Liu J. Research progress analysis of robotic harvesting technologies in green house. Transactions of the Chinese Society for Agricultural Machinery, 2017, 48(12): 1−18 (in Chinese)
|
[2] |
Song J, Zhang T, Xu L, Tang X. Research actuality and prospect of picking robot for fruits and vegetables. Transactions of the Chinese Society for Agricultural Machinery, 2006, 37(5): 158−162 (in Chinese)
|
[3] |
Hayashi S, Shigematsu K, Yamamoto S, Kobayashi K, Kohno Y, Kamata J, Kurita M . Evaluation of a strawberry-harvesting robot in a field test. Biosystems Engineering, 2010, 105(2): 160–171
CrossRef
Google scholar
|
[4] |
Xu L, Zhang T. Present situation of fruit and vegetable harvesting robot and its key problems and measures in application. Transactions of the Chinese Society of Agricultural Engineering, 2004, 20(5): 38−42 (in Chinese)
|
[5] |
Fang J. Present situation and development of mobile harvesting robot. Transactions of the Chinese Society of Agricultural Engineering, 2004, 20(2): 273−278 (in Chinese)
|
[6] |
Si Y, Qiao J, Liu G, Liu Z, Gao D. Recognition and shape features extraction of apples based on machine vision. Transactions of the Chinese Society for Agricultural Machinery, 2009, 40(8): 161−165 (in Chinese)
|
[7] |
Feng J, Wang S, Liu G, Zeng L. A separating method of adjacent apples based on machine vision and chain code information. In: Li D, Chen Y, eds. Computer and Computing Technologies in Agriculture V. CCTA 2011. IFIP Advances in Information and Communication Technology, vol 368. Berlin, Heidelberg: Springer, 2012, 258–267
|
[8] |
Miao Z, Shen Y, Wang X, Zhou X, Liu C. Imaged recognition algorithm and experiment of overlapped fruits in natural environment. Transactions of the Chinese Society for Agricultural Machinery, 2016, 47(6): 21−26 (in Chinese)
|
[9] |
Van Henten E J, Schenk E J, van Willigenburg L G, Meuleman J, Barreiro P . Collision-free inverse kinematics of the redundant seven-link manipulator used in a cucumber picking robot. Biosystems Engineering, 2010, 106(2): 112–124
CrossRef
Google scholar
|
[10] |
Bac C W, Roorga T, Reshef R, Berman S, Hemming J, van Henten E J . Analysis of a motion planning problem for sweet pepper harvesting in a dense obstacle environment. Biosystems Engineering, 2016, 146: 85–97
CrossRef
Google scholar
|
[11] |
Barth R, Hemming J, van Henten E J . Design of eye-in-hand sensing and servo control framework for harvesting robotics in dense vegetation. Biosystems Engineering, 2016, 146: 71–84
CrossRef
Google scholar
|
[12] |
Tu J, Liu C, Li Y, Zhou J, Yuan J. Apple recognition method based on illumination invariant graph. Transactions of the Chinese Society of Agricultural Engineering, 2010, 26(S2): 26−31 (in Chinese)
|
[13] |
Song H, Qu W, Wang D, Yu X, He D. Shadow removal method of apples based on illumination invariant image. Transactions of the Chinese Society of Agricultural Engineering, 2014, 30(24): 168−176 (in Chinese)
|
[14] |
Huang L W, He D J . Apple recognition in natural tree canopy based on fuzzy 2-partition entropy. International Journal of Digital Content Technology and Its Applications, 2013, 7(1): 107–115
CrossRef
Google scholar
|
[15] |
Wang D, Xu Y, Song H, He D, Zhang H. Fusion of K-means and Ncut algorithm to realize segmentation and reconstruction of two overlapped apples without blocking by branches and leaves. Transactions of the Chinese Society of Agricultural Engineering, 2015, 31(10): 227−234 (in Chinese)
|
[16] |
Wang D, Song H, He D. Research advance on vision system of apple picking robot. Transactions of the Chinese Society of Agricultural Engineering, 2017, 33(10): 59−69 (in Chinese)
|
[17] |
Li X. The Design and Implementation of Virtual Human Body Dissection Teaching System Based on Kinect Gesture Recognition. Beijing: Beijing University of Technology, 2014 (in Chinese)
|
[18] |
Lu H, Zhang Y, Niu M, Shui W, Zhou M. Three-dimensional virtual reassembling for broken artifact fragments based on Leap Motion. Journal of System Simulation, 2015, 27(12): 3006−3011 (in Chinese)
|
[19] |
Liu C, Li Q. Leap Motion somatosensory controller and its application in aircraft structure display system. Computer Applications and Software, 2016, 33(4): 227−229 (in Chinese)
|
[20] |
Liu J, Jia S, Wang P, Huo D. Study of 3D product display technology based on Leap Motion. Journal of Dalian Jiaotong University, 2016, 37(4): 110−113 (in Chinese)
|
[21] |
Wu F, Ding Y, Ding W, Xie T. Design of human computer interaction system of virtual crops based on Leap Motion. Transactions of the Chinese Society of Agricultural Engineering, 2016, 32(23): 144−151 (in Chinese)
|
[22] |
Xu C, Wang Q, Chen H, Mei S, Du L. Design and simulation of artificial limb picking robot based on somatosensory interaction. Transactions of the Chinese Society of Agricultural Engineering, 2017, 33(S1): 49−55 (in Chinese)
|
[23] |
Quan L, Li C, Feng Z, Liu J. Algorithm of works’ decision for three arms robot in greenhouse based on control with motion sensing technology. Transactions of the Chinese Society for Agricultural Machinery, 2017, 48(3): 14−23 (in Chinese)
|
[24] |
Fu D, Xu L, Li D, Xin L. Automatic Detection and Segmentation of Stems of Potted Tomato Plant Using Kinect. In: Proceedings Volume 9159, Proceedings of the Sixth International Conference on Digital Image Processing (ICDIP), Athens, Greece, 5–6 April 2014. Society of Photo-Optical Instrumentation Engineers (SPIE), 2014, 915905
|
[25] |
Pérez-Ruíz M, Slaughter D C, Fathallah F A, Gliever C J, Miller B J . Co-robotic intra-row weed control system. Biosystems Engineering, 2014, 126: 45–55
CrossRef
Google scholar
|
[26] |
Denavit J, Hartenberg R S . A kinematic notation for lower pair mechanisms based on matrices. Journal of Applied Mechanics, 1955, 22(2): 215–221
CrossRef
Google scholar
|
[27] |
Wang W. Leap Motion Human–Computer Interaction Application Development. Xi’an: Xidian University Press, 2015 (in Chinese)
|
[28] |
Weichert F, Bachmann D, Rudak B, Fisseler D . Analysis of the accuracy and robustness of the Leap Motion Controller. Sensors, 2013, 13(5): 6380–6393
CrossRef
Google scholar
|
Supplementary files
FASE-21385-OF-XQM_suppl_1 (48 KB)
/
〈 |
|
〉 |