Robotic Cell Micromanipulation Skill Learning via Imitation-Enhanced Reinforcement Learning

Abstract

Humans can learn complex and dexterous manipulation tasks by observing videos, imitating and exploring. Multiple end- effectors manipulation of free micron-sized deformable cells is one of the challenging tasks in robotic micromanipulation. We propose an imitation-enhanced reinforcement learning method inspired by the human learning process that enables robots to learn cell micromanipulation skills from videos. Firstly, for the microscopic robot micromanipulation videos, a multi-task observation (MTO) network is designed to identify the two end-effectors and the manipulated objects to obtain the spatio-temporal trajectories. The spatiotemporal constraints of the robot's actions are obtained by the task-parameterised hidden Markov model (THMM). To simultaneously address the safety and dexterity of robot micromanipulation, an imitation learning optimisation-based soft actor-critic (ILOSAC) algorithm is proposed in which the robot can perform skill learning by demonstration and exploration. The proposed method is capable of performing complex cell manipulation tasks in a realistic physical environment. Experiments indicated that compared with current methods and manual remote manipulation, the proposed framework achieved a shorter operation time and less deformation of cells, which is expected to facilitate the development of robot skill learning.

Keywords

deep learning / intelligent robots / intelligent systems / robotics

Funding

This work was supported in part with the General Programme of the National Natural Science Foundation of China (Grant 62576312), the Key Research and Development Program of Zhejiang Province (Grant 2025C01132) and the Shandong Province Key R&D Plan Project (Grant 2022LZGC020).

Conflicts of Interest

The authors declare no confiicts of interest.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

[1] Y. Zhang, B. K. Chen, X. Liu and Y. Sun, “Autonomous Robotic Pick-and-Place of Microobjects,” IEEE Transactions on Robotics 26, no. 1 (2009): 200-207, https://doi.org/10.1109/TRO.2009.2034831.

[2] L. Zheng, Y. Jia, D. Dong, et al., “3d Navigation Control of Unteth-ered Magnetic Microrobot in Centimeter-Scale Workspace Based on Field-of-View Tracking Scheme,” IEEE Transactions on Robotics 38, no. 3 (2021): 1583-1598, https://doi.org/10.1109/tro.2021.3118205.

[3] S. Zhuang, C. Dai, G. Shan, C. Ru, Z. Zhang, and Y. Sun, “Robotic Rotational Positioning of End-Effectors for Micromanipulation,” IEEE Transactions on Robotics 38, no. 4 (2022): 2251-2261, https://doi.org/10.1109/tro.2022.3142671.

[4] C. Dai, G. Shan, H. Liu, C. Ru, and Yu Sun, “Robotic Manipulation of Sperm as a Deformable Linear Object,” IEEE Transactions on Robotics 38, no. 5 (2022): 2799-2811, https://doi.org/10.1109/tro.2022.3158200.

[5] Y. Zhang, X. Guo, Q. Wang, et al., “Automated Dissection of Intact Single Cell From Tissue Using Robotic Micromanipulation System,” IEEE Robotics and Automation Letters 8, no. 8 (2023): 4705-4712, https://doi.org/10.1109/lra.2023.3287364.

[6] C. Dai, Z. Zhang, Y. Lu, et al., “Robotic Manipulation of Deformable Cells for Orientation Control,” IEEE Transactions on Robotics 36, no. 1 (2019): 271-283, https://doi.org/10.1109/tro.2019.2946746.

[7] S. Hu and D. Sun, “Automatic Transportation of Biological Cells With a Robot-Tweezer Manipulation System,” International Journal of Robotics Research 30, no. 14 (2011): 1681-1694, https://doi.org/10.1177/0278364911413479.

[8] X. Liu, K. Kim, Y. Zhang, and Yu Sun, “Nanonewton Force Sensing and Control in Microrobotic Cell Manipulation,” International Journal of Robotics Research 28, no. 8 (2009): 1065-1076, https://doi.org/10.1177/0278364909340212.

[9] S. Permana, E. Grant, G. M. Walker, and J. A. Yoder, “A Review of Automated Microinjection Systems for Single Cells in the Embryogen-esis Stage,” IEEE 21, no. 5 (2016): 2391-2404, https://doi.org/10.1109/tmech.2016.2574871.

[10] M. Xie, J. K. Mills, Y. Wang, M. Mahmoodi, and D. Sun, “Auto-mated Translational and Rotational Control of Biological Cells With a Robot-Aided Optical Tweezers Manipulation System,” IEEE Trans-actions on Automation Science and Engineering 13, no. 2 (2015): 543-551, https://doi.org/10.1109/tase.2015.2411271.

[11] S. Yang and K. W. C. Lai, “A Novel Micropipette Robot for Cell Manipulation Based on Dielectrophoresis and Electroosmotic Vortex,” in 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO) IEEE, 2021), 24-29.

[12] S. Schaal, A. Ijspeert, and A. Billard, “Computational Approaches to Motor Learning by Imitation,” Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 358, no. 1431 (2003): 537-547, https://doi.org/10.1098/rstb.2002.1258.

[13] Y. Ma, De Xu and F. Qin, “Efficient Insertion Control for Precision Assembly Based on Demonstration Learning and Reinforcement Learning,” IEEE Transactions on Industrial Informatics 17, no. 7 (2020): 4492-4502, https://doi.org/10.1109/tii.2020.3020065.

[14] H. Su, W. Qi, Y. Hu, H. R. Karimi, G. Ferrigno, and E. De Momi, “An Incremental Learning Framework for Human-Like Redundancy Optimization of Anthropomorphic Manipulators,” IEEE Transactions on Industrial Informatics 18, no. 3 (2020): 1864-1872, https://doi.org/10.1109/tii.2020.3036693.

[15] L. Chen, Y. Wang, Z. Miao, et al., “Transformer-Based Imitative Reinforcement Learning for Multi-Robot Path Planning,” IEEE Trans-actions on Industrial Informatics 19, no. 10 (2023): 10233-10243, https://doi.org/10.1109/tii.2023.3240585.

[16] T. Hester, M. Vecerik, O. Pietquin, et al. “Deep q-Learning From Demonstrations ” , in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, no. 1 (2018), https://doi.org/10.1609/aaai.v32i1.11757.

[17] A. Nurbayeva, A. Shintemirov, and M. Rubagotti, “Deep Imitation Learning of Nonlinear Model Predictive Control Laws for Safe Physical Human-Robot Interaction,” IEEE Transactions on Industrial Informatics 19, no. 7 (2022): 8384-8395, https://doi.org/10.1109/TII.2022.3217833.

[18] M. Vecerik, Todd H. J. Scholz, et al., “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems With Sparse Rewards,” arXiv Preprint arXiv:1707.08817 (2017).

[19] A. Rajeswaran, V. Kumar, A. Gupta, et al., “Learning Complex Dexterous Manipulation With Deep Reinforcement Learning and Demonstrations,” in Proceedings of Robotics: Science and Systems (2018).

[20] I. Radosavovic, X. Wang, L. Pinto, and J. Malik, “State-Only Imitation Learning for Dexterous Manipulation,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) IEEE, 2021), 7865-7871.

[21] M. B. Hafez and S. Wermter, “Continual Robot Learning Using Self- Supervised Task Inference,” IEEE Transactions on Cognitive and Developmental Systems 16, no. 3 (2024): 947-960, https://doi.org/10.1109/TCDS.2023.3315513.

[22] W. Wan, Y. Zhu, R. Shah, and Y. Zhu, “Lotus: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discov-ery,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) 2024, https://doi.org/10.1109/ICRA57147.2024.10611129.

[23] R. Mori, T. Aoyama, T. Kobayashi, K. Sakamoto, M. Takeuchi, and Y. Hasegawa, “Real-Time Spatiotemporal Assistance for Micromanipu-lation Using Imitation Learning,” IEEE Robotics and Automation Letters 9, no. 4 (2024): 3506-3513, https://doi.org/10.1109/lra.2024.3366011.

[24] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation Learning: A Survey of Learning Methods,” ACM Computing Surveys 50, no. 2 (2017): 1-35, https://doi.org/10.1145/3054912.

[25] T. P. Lillicrap, J. J. Hunt, A. Pritzel, et al., “Continuous Control With Deep Reinforcement Learning,” arXiv Preprint arXiv:1509.02971 (2015).

[26] T. Haarnoja, A. Zhou, K. Hartikainen, et al., “Soft actor-critic Al-gorithms and Applications,” arXiv Preprint arXiv:1812.05905 (2018).

[27] C. Dai, G. Shan, X. Liu, C. Ru, L. Xin, and Yu Sun, “Automated Orientation Control of Motile Deformable Cells,” IEEE Transactions on Automation Science and Engineering 20, no. 3 (2022): 2126-2134, https://doi.org/10.1109/tase.2022.3194845.