Human-in-the-loop transfer learning in collision avoidance of autonomous robots

Abstract

Neural networks have demonstrated exceptional performance across a range of applications. Yet, their training often demands substantial time and data resources, presenting a challenge for autonomous robots operating in real-world environments where real-time learning is difficult. To mitigate this constraint, we propose a novel human-in-the-loop framework that harnesses human expertise to mitigate the learning challenges of autonomous robots. Our approach centers on directly incorporating human knowledge and insights into the robot’s learning pipeline. The proposed framework incorporates a mechanism for autonomous learning from the environment via reinforcement learning, utilizing a pre-trained model that encapsulates human knowledge as its foundation. By integrating human-provided knowledge and evaluation, we aim to bridge the division between human intuition and machine learning capabilities. Through a series of collision avoidance experiments, we validated that incorporating human knowledge significantly improves both learning efficiency and generalization capabilities. This collaborative learning paradigm enables robots to utilize human common sense and domain-specific expertise, resulting in faster convergence and better performance in complex environments. This research contributes to the development of more efficient and adaptable autonomous robots and seeks to analyze how humans can effectively participate in robot learning and the effects of such participation, illuminating the intricate interplay between human cognition and artificial intelligence.

Keywords

Human-in-the-loop / Transfer learning / Autonomous robots / Neural networks / Reinforcement learning

1 CRediT authorship contribution statement

Minako Oriyama: Writing - original draft, Visualization, Validation, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Pitoyo Hartono: Writing - review & editing, Validation, Supervision, Resources, Project administration, Methodology, Conceptualization. Hideyuki Sawada: Writing - review & editing, Validation, Supervision, Resources, Project administration, Methodology, Funding acquisition, Conceptualization.

2 Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

3 Acknowledgment

This research is supported by the research funding of Waseda University, Japan.

References

[1] Eduardo Mosqueira-Rey, Elena Hernández-Pereira, David Alonso-Ríos, José Bobes-Bascarán, Ángel Fernández-Leal, Human-in-the-loop machine learn-ing: a state of the art, Artif. Intell. Rev. 56 (4) (2022) 3005-3054, http://dx.doi.org/10.1007/s10462-022-10246-w.

[2] Eitan Netzer, Amir B. Geva, Human-in-the-loop active learning via brain computer interface, Ann. Math. Artif. Intell. 88 (11-12) (2020) 1191-1205, http://dx.doi.org/10.1007/s10472-020-09689-0

[3] Issei Suzuki, Pitoyo Hartono, Human-in-the-loop: infusing knowledge into neural networks, in: 2024 IEEE International Conference on Mecha-tronics and Automation, ICMA, IEEE, 2024, pp. 1665-1670, http://dx. doi.org/10.1109/icma61710.2024.10633061

[4] Deniz Yilmaz, Barkan Ugurlu, Erhan Oztop, Human-in-the-loop training leads to faster skill acquisition and adaptation in reinforcement learning-based robot control, in: 2024 IEEE 18th International Conference on Advanced Motion Control, AMC, IEEE, 2024, pp. 1-6, http://dx.doi.org/10. 1109/amc58169.2024.10505631

[5] Yonghao Long, Wang Wei, Tao Huang, Yuehao Wang, Qi Dou, Human-in-the-loop embodied intelligence with interactive simulation environment for surgical robot learning, IEEE Robot. Autom. Lett. 8 (8) (2023) 4441-4448, http://dx.doi.org/10.1109/lra.2023.3284380

[6] Hee Rak Beom, Hyung Suck Cho, A sensor-based navigation for a mobile robot using fuzzy logic and reinforcement learning, IEEE Trans. Syst. Man Cybern. 25 (3) (1995) 464-477, http://dx.doi.org/10.1109/21.364859

[7] Guoqian Pan, Yong Xiang, Xiaorui Wang, Zhongquan Yu, Xinzhi Zhou, Research on path planning algorithm of mobile robot based on reinforce-ment learning, Soft Comput. 26 (18) (2022) 8961-8970, http://dx.doi.org/10.1007/s00500-022-07293-4

[8] Napat Karnchanachari, Miguel I. Valls, David Hoeller, Marco Hutter,Prac-tical reinforcement learning for MPC: Learning from sparse objectives in under an hour on a real robot, 2020, URL https://arxiv.org/abs/2003.03200.

[9] Binyu Wang, Zhe Liu, Qingbiao Li, Amanda Prorok, Mobile robot path planning in dynamic environments through globally guided reinforcement learning, IEEE Robot. Autom. Lett. 5 (4) (2020) 6932-6939, http://dx.doi.org/10.1109/lra.2020.3026638

[10] Siti Sendari, Muladi, Firman Ardiyansyah, Samsul Setumin, Norrima Binti Mokhtar, Hsien-I Lin, Pitoyo Hartono, Common-sensical incentive reward in deep actor-critic reinforcement learning for mobile robot navigation, Int. J. Innovative Comput. Inf. Control 20 (2) (2024) 373-389, http://dx.doi.org/10.24507/ijicic.20.02.373, URL https://www.scopus.com/inward/record.uri?eid=2-s2.0-85186911357&doi=10.24507%2fijicic.20.02. 373&partnerID=40&md5=1b41795e03a07d1af74236f5854d811a.

[11] Kana Ogawa, Pitoyo Hartono, Infusing common-sensical prior knowledge into topological representations of learning robots, Artif. Life Robot. 27(3)(2022) 576-585, http://dx.doi.org/10.1007/s10015-022-00776-5

[12] Sinno Jialin Pan, Qiang Yang,A survey on transfer learning, IEEE Trans. Knowl. Data Eng. 22 (10) (2010) 1345-1359, http://dx.doi.org/10.1109/tkde.2009.191, URL http://dx.doi.org/10.1109/TKDE.2009.191.

[13] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, Chun-fang Liu, A survey on deep transfer learning, in:Věra Kůrková Yannis Manolopoulos, Barbara Hammer, Lazaros Iliadis, Eds.), Artificial Neural Networks and Machine Learning - ICANN 2018, Springer International Publishing, Cham, 2018, pp. 270-279.

[14] Heramb Nemlekar, Neel Dhanaraj, Angelos Guan, Satyandra K. Gupta, Stefanos Nikolaidis, Transfer learning of human preferences for proactive robot assistance in assembly tasks, in: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’23, ACM, 2023, pp. 575-583, http://dx.doi.org/10.1145/3568162.3576965, URL http://dx.doi.org/10.1145/3568162.3576965.

[15] Michael J. Sorocky, Siqi Zhou, Angela P. Schoellig, Experience selection us-ing dynamics similarity for efficient multi-source transfer learning between robots, in: 2020 IEEE International Conference on Robotics and Automa-tion, ICRA, IEEE, 2020, pp. 2739-2745, http://dx.doi.org/10.1109/icra40945.2020.9196744, URL http://dx.doi.org/10.1109/ICRA40945.2020.9196744.

[16] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel, Sim-to-real transfer of robotic control with dynamics randomization, in: 2018 IEEE International Conference on Robotics and Automation, ICRA, IEEE, 2018, pp. 3803-3810, http://dx.doi.org/10.1109/icra.2018.8460528, URL http://dx.doi.org/10.1109/ICRA.2018.8460528.

[17] Tianhao Zhang, Runyu Tian, Hongqi Yang, Chen Wang, Jinan Sun, Shikun Zhang, Guangming Xie, From simulation to reality: A learning framework for fish-like robots to perform control tasks, IEEE Trans. Robot. 38 (6)(2022) 3861-3878, http://dx.doi.org/10.1109/tro.2022.3181014, URL http://dx.doi.org/10.1109/TRO.2022.3181014.

[18] Andrei A. Rusu, Mel Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pas-canu, Raia Hadsell,Sim-to-real robot learning from pixels with progressive nets, 2018, URL https://arxiv.org/abs/1610.04286.

[19] Naijun Liu, Yinghao Cai, Tao Lu, Rui Wang, Shuo Wang, Real-sim-real transfer for real-world robot control policy learning with deep reinforce-ment learning, Appl. Sci. 10 (5) (2020) 1555, http://dx.doi.org/10.3390/app10051555, URL http://dx.doi.org/10.3390/app10051555.

[20] Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupta, Aravind Rajeswaran, Vikash Kumar, Real world offline reinforcement learning with realistic data source, in: 2023 IEEE International Conference on Robotics and Automation, ICRA, IEEE, 2023, pp. 7176-7183, http://dx.doi.org/10.1109/icra48891.2023.10161474, URL http://dx.doi.org/10.1109/ICRA48891.2023.10161474.

[21] Charles Sun, Je¸ drzej Orbik, Coline Manon Devin, Brian H. Yang, Abhishek Gupta, Glen Berseth, Sergey Levine, Fully autonomous real-world reinforce-ment learning with applications to mobile manipulation, in:Aleksandra Faust, David Hsu, Eds.),Proceedings of the 5th Confer-ence on Robot Learning, in: Proceedings of Machine Learning Research, vol. 164, PMLR, 2022, pp. 308-319, URL https://proceedings.mlr.press/v164/sun22a.html.

[22] Laura Smith, Ilya Kostrikov, Sergey Levine,A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning, 2022, URL https://arxiv.org/abs/2208.07860.

[23] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, Pieter Abbeel,DayDreamer: World models for physical robot learning, 2022, URL https://arxiv.org/abs/2206.14176.

[24] R. Andrews, S. Geva,On the effects of initialising a neural network with prior knowledge, in:ICONIP’99. ANZIIS’99 & ANNES’99 & ACNN’99. 6th International Conference on Neural Information Processing. Proceedings (Cat. No.99EX378), vol. 1, 1999, pp. 251-256, http://dx.doi.org/10.1109/ICONIP.1999.843995.

[25] Andrew Silva, Matthew Gombolay, Encoding human domain knowledge to warm start reinforcement learning, Proc. AAAI Conf. Artif. Intell. 35 (6)(2021) 5042-5050, http://dx.doi.org/10.1609/aaai.v35i6.16638, URL http://dx.doi.org/10.1609/aaai.v35i6.16638.

[26] Robert M. French, Catastrophic forgetting in connectionist networks, Trends in Cognitive Sciences 3 (4) (1999) 128-135, http://dx.doi.org/10.1016/S1364-6613(99)01294-2, URL https://www.sciencedirect.com/science/article/pii/S1364661399012942.

[27] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell, Overcoming catastrophic forgetting in neural networks, Proc. Natl. Acad. Sci. 114 (13) (2017) 3521-3526, http://dx.doi.org/10.1073/pnas.1611835114, URL http://dx.doi.org/10.1073/pnas.1611835114.

[28] Pinxin Long, Tingxiang Fanl, Xinyi Liao, Wenxi Liu, Hao Zhang, Jia Pan, Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning, in: 2018 IEEE International Conference on Robotics and Automation, ICRA, IEEE, 2018, pp. 6252-6259, http://dx.doi.org/10.1109/icra.2018.8461113, URL http://dx.doi.org/10.1109/ICRA.2018.8461113.

[29] Bashra Kadhim Oleiwi, Asif Mahfuz, Hubert Roth, Application of fuzzy logic for collision avoidance of mobile robots in dynamic-indoor environ-ments, in: 2021 2nd International Conference on Robotics, Electrical and Signal Processing Techniques, ICREST, IEEE, 2021, pp. 131-136, http://dx.doi.org/10.1109/icrest51555.2021.9331072, URL http://dx.doi.org/10.1109/ICREST51555.2021.9331072.

[30] Pitoyo Hartono, Sachiko Kakita, Fast reinforcement learning for simple physical robots, Memetic Comput. 1 (4) (2009) 305-313, http://dx.doi.org/10.1007/s12293-009-0015-x, URL http://dx.doi.org/10.1007/s12293-009-0015-x.

[31] Xiaogang Ruan, Dingqi Ren, Xiaoqing Zhu, Jing Huang, Mobile robot navigation based on deep reinforcement learning, in: 2019 Chinese Control and Decision Conference, CCDC, IEEE, 2019, pp. 6174-6178, http://dx.doi.org/10.1109/ccdc.2019.8832393, URL http://dx.doi.org/10.1109/CCDC.2019.8832393.

[32] Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, Jiayu Zhou, Transfer learning in deep reinforcement learning: A survey, IEEE Trans. Pattern Anal. Mach. Intell. 45 (11) (2023) 13344-13362, http://dx.doi.org/10.1109/tpami.2023.3292075, URL http://dx.doi.org/10.1109/TPAMI.2023.3292075.

[33] Karthik Karur, Nitin Sharma, Chinmay Dharmatti, Joshua E. Siegel, A survey of path planning algorithms for mobile robots, Vehicles 3 (3) (2021) 448-468, http://dx.doi.org/10.3390/vehicles3030027, URL http://dx.doi.org/10.3390/vehicles3030027.

[34] Bianca Sangiovanni, Gian Paolo Incremona, Marco Piastra, Antonella Fer-rara, Self-configuring robot path planning with obstacle avoidance via deep reinforcement learning, IEEE Control Syst. Lett. 5 (2) (2021) 397-402, http://dx.doi.org/10.1109/lcsys.2020.3002852, URL http://dx.doi.org/10.1109/LCSYS.2020.3002852.

[35] Minako Oriyama, Pitoyo Hartono, Hideyuki Sawada, Human-guided trans-fer learning for autonomous robot, in: Neural Information Processing, Springer Nature Singapore, 2023, pp. 186-198, http://dx.doi.org/10.1007/978-981-99-8126-7_15, URL http://dx.doi.org/10.1007/978-981-99-8126-7_15.