Deep reinforcement learning-based dynamic path planning of flexible needle in robotic puncturing

Abstract

Flexible needle puncture path planning in surgical robots faces significant challenges, such as limited adaptability to multi-objective environments and poor real-time performance. These issues affect both the accuracy and efficiency of needle puncture, restricting its application in complex medical scenarios. This paper proposes a deep reinforcement learning-based method to improve flexible needle path planning. To address the limitations of traditional models in accurately capturing needle dynamics, a hierarchical tissue model based on the unicycle framework is designed, which integrates kinematic and mechanical models. This approach considers the varying forces from different tissues on the needle at various positions. A dynamic multi-objective environment and obstacle model are also constructed, along with a target prioritization scheme for multi-objective optimization. Additionally, a prioritized experience replay (PER) mechanism is introduced to improve data efficiency in the learning process. This method enhances the needle’s adaptability and robustness in dynamic environments. Simulation results demonstrate that the model improved real-time performance and precision in dynamic multi-objective environments, making intelligent decisions based on target priorities and accelerating the exploration of optimal strategies.

Keywords

Reinforcement learning / Flexible needle puncture / Path planning / Dynamic environment / Multi-objective optimization

CRediT authorship contribution statement

Jiewu Leng: Supervision, Funding acquisition, Conceptualization. Tengliang Zhu: Writing – review & editing, Writing – original draft, Data curation. Zhiqiang Huang: Writing – original draft, Software, Methodology. Rongjie Li: Writing – review & editing, Data curation. Xueliang Zhou: Validation, Investigation. Jun Lin: Visualization, Software, Data curation. Ding Zhang: Supervision, Project administration, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by the Guangdong Provincial Basic and Applied Basic Research Fund, China (2022B1515020006) and the Science and Technology Program of Guangzhou, China (2024A04J6301).

References

[1] C. Wang, L. Guo, J. Zhu, L. Zhu, C. Li, H. Zhu, A. Song, L. Lu, G. Teng, N. Navab, Review of robotic systems for thoracoabdominal puncture interventional surgery, APL Bioeng. 8 (2024).

[2] K. Wu, B. Li, Y. Zhang, X. Dai, Review of research on path planning and control methods of flexible steerable needle puncture robot, Comput. Assist. Surg. 27 (2022) 91-112.

[3] H.M. Muzzammil, Y. Zhang, H. Ejaz, Q. Yuan, M. Muddassir, A review on tissue-needle interaction and path planning models for bevel tip type flexible needle minimal intervention, Math. Biosci. Eng. 21 (2024) 523-561.

[4] J. Leng, M. Zhou, Y. Xiao, H. Zhang, Q. Liu, W. Shen, Q. Su, L. Li, Digital twins-based remote semi-physical commissioning of flow-type smart manufacturing systems, J. Clean Prod. 306 (2021) 127278.

[5] Y. Matsuo, Y. LeCun, M. Sahani, D. Precup, D. Silver, M. Sugiyama, E. Uchibe, J. Morimoto, Deep learning, reinforcement learning, and world models, Neural Netw. 152 (2022) 267-275.

[6] T.T. Nguyen, N.D. Nguyen, S. Nahavandi, Deep reinforcement learning for multiagent systems : A review of challenges, solutions, and applications, IEEE T Cybern. 50 (2020) 3826-3839.

[7] J. Lin, Z. Huang, T. Zhu, J. Leng, K. Huang, Deep reinforcement learning-based robotic puncturing path planning of flexible needle, Processes 12 (2024) 2852.

[8] J. Xiong, X. Li, Y. Gan, Z. Xia, Path Planning for Flexible Needle Insertion System Based on Improved Rapidly-Exploring Random Tree Algorithm, IEEE, 2015, pp. 1545-1550.

[9] A. Hong, Q. Boehler, R. Moser, A. Zemmar, L. Stieglitz, B.J. Nelson, 3D path planning for flexible needle steering in neurosurgery, Int. J. Med. Robot. Comp. 15 (2019).

[10] Z.Y. , J.Z. , Z.H. , Q.Z. , 3-D path planning using improved RRT* algorithm for robot-assisted flexible needle insertion in multilayer tissues, IEEE Can. J. Elect. Com. 45 (2022) 50-62.

[11] Z. Tan, D. Zhang, H. Liang, Q. Wang, W. Cai, A new path planning method for bevel-tip flexible needle insertion in 3D space with multiple targets and obstacles, Control Theor. Tech. 20 (2022) 525-535.

[12] Y. Lee, X. Tan, C. Chng, C. Chui, Simulation of Robot-Assisted Flexible Needle Insertion using Deep Q-Network, IEEE, 2019, pp. 342-346.

[13] X. Tan, Y. Lee, C. Chng, K. Lim, C. Chui, Robot-assisted flexible needle insertion using universal distributional deep reinforcement learning, Int. J. Comput. Ass. Rad. 15 (2020) 341-349.

[14] J. Yan, Z. Liu, J. Leng, J.L. Zhao, C. Chen, D. Zhang, Y. Tao, Y. Wang, T. Liu, C. Zhang, Y. Tong, D. Mourtzis, L. Wang, Human-centric artificial intelligence towards industry 5.0: retrospect and prospect, J. Ind. Inf. Integr. 47 (2025) 100903.

[15] J. Leng, K. Zuo, C. Xu, X. Zhou, S. Zheng, J. Kang, Q. Liu, X. Chen, W. Shen, L. Wang, R.X. Gao, Physics-informed machine learning in intelligent manufacturing: a review, J. Intell. Manuf. (2025).

[16] B. Zhang, C.H. Liu, J. Tang, Z. Xu, J. Ma, W. Wang, Learning-based energy-efficient data collection by unmanned vehicles in smart cities, IEEE T Ind. Inform. 14 (2018) 1666-1676.

[17] X. Wang, Y. Wang, X. Su, L. Wang, C. Lu, H. Peng, J. Liu, Deep reinforcement learning-based air combat maneuver decision-making : literature review, implementation tutorial and future direction, Artif. Intell. Rev. 57 (2024).

[18] K. Zhu, T. Zhang, Deep reinforcement learning based mobile robot navigation : A review, Tsinghua Sci. Technol. 26 (2021) 674-691.

[19] J. Guo, J. Leng, J.L. Zhao, X. Zhou, Y. Yuan, Y. Lu, D. Mourtzis, Q. Qi, S. Huang, X. Song, Q. Liu, L. Wang, Industrial metaverse towards industry 5.0: Connotation, architecture, enablers, and challenges, J. Manuf. Syst. 76 (2024) 25-42.

[20] J. Leng, K. Zheng, R. Li, C. Chen, B. Wang, Q. Liu, X. Chen, W. Shen, AIGC-empowered smart manufacturing: Prospects and challenges, Robot. Cim-Int. Manuf. 97 (2026) 103076.

[21] J. Leng, R. Li, J. Xie, X. Zhou, X. Li, Q. Liu, X. Chen, W. Shen, L. Wang, Federated learning-empowered smart manufacturing and product lifecycle management: A review, Adv. Eng. Inform. 65 (2025) 103179.

[22] J. Leng, X. Su, Z. Liu, L. Zhou, C. Chen, X. Guo, Y. Wang, R. Wang, C. Zhang, Q. Liu, X. Chen, W. Shen, L. Wang, Diffusion model-driven smart design and manufacturing: Prospects and challenges, J. Manuf. Syst. 82 (2025) 561-577.

[23] C. You, J. Lu, D. Filev, P. Tsiotras, Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning, Robot. Auton. Syst. 114 (2019) 1-18.

[24] Autonomous Highway Driving using Deep Reinforcement Learning.

[25] L. Chen, X. Hu, B. Tang, Y. Cheng, Conditional DQN-based motion planning with fuzzy logic for autonomous driving, IEEE T Intell. Transp. 23 (2022) 2966-2977.

[26] J. Xin, H. Zhao, D. Liu, M. Li, Application of Deep Reinforcement Learning in Mobile Robot Path Planning, IEEE, 2017, pp. 7112-7116.

[27] L. Tai, G. Paolo, M. Liu, Virtual-to-real Deep Reinforcement Learning : Continuous Control of Mobile Robots for Mapless Navigation, in: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2017, pp. 31-36.

[28] X. Lei, P. Dong, Z. Zhang, R.P. Gordon, G.R. Pennock, Dynamic path planning of unknown environment based on deep reinforcement learning, J Robot. 2018 (2018) 1-10.

[29] L. Lv, S. Zhang, D. Ding, Y. Wang, Path planning via an improved DQN-based learning policy, IEEE Access 7 (2019) 67319-67330.

[30] H. Bae, G. Kim, J. Kim, D. Qian, S. Lee, Multi-robot path planning method using reinforcement learning, Appl. Sci.-Basel 9 (2019) 3057.

[31] L. Jiang, H. Huang, Z. Ding, Path planning for intelligent robots based on deep Q-learning with experience replay and heuristic knowledge, IEEE-CAA J. Autom. 7 (2020) 1179-1189.

[32] J. Yu, Y. Su, Y. Liao, The path planning of mobile robot by neural networks and hierarchical reinforcement learning, Front Neurorobotics, 14 (2020).

[33] J. Gao, W. Ye, J. Guo, Z. Li, Deep reinforcement learning for indoor mobile robot path planning, Sensors-Basel, 20 (2020).

[34] S. Guo, X. Zhang, Y. Du, Y. Zheng, Z. Cao, Path planning of coastal ships based on optimized DQN reward function, J. Mar. Sci. Eng. 9 (2021).

[35] R. Pan, M. Bagherzadeh, T.A. Ghaleb, L. Briand, Test case selection and prioritization using machine learning : a systematic literature review, Empir. Softw. Eng. 27 (2022).

[36] L. Zeng, Q. Liu, S. Shen, X. Liu, Improved double deep q network-based task scheduling algorithm in edge computing for makespan optimization, Tsinghua Sci. Technol. 29 (2024) 806-817.

[37] Y. Yu, J. Li, S.A. Solomon, J. Min, J. Tu, W. Guo, C. Xu, Y. Song, W. Gao, All-printed soft human-machine interface for robotic physicochemical sensing, Sci. Robot. 7 (2022).