A Five-Dimensional Three-Layer Digital Twin to Train a Reinforcement Learning Agent for Interaction Control of a Robotic Exoskeleton in Adolescent Idiopathic Scoliosis Rehabilitation

Abstract

Adolescent idiopathic scoliosis (AIS) is a sideway curvature of the spinal column combined with a vertebral rotation that usually occurs in adolescents without any known causes. Bracing, the most common conservative treatment of AIS, has not fully exploited the benefits of the active control approaches powered by artificial intelligence (AI), although AI has entered a wide range of applications. The correction forces exerted by the brace are controlled passively by regulating the tightness of the brace's strap. Besides, training the learning-based control methods using a virtual model is of high importance in the AIS brace treatment, since training using trial and error on human subjects may result in unexpected pressure and injuries on the patient's torso. However, digital twin (DT) modeling, an emerging technology, has not been implemented into the AIS brace treatment yet. In this paper, reinforcement learning-based position-based impedance control (RLPIC) is proposed to enable a robotic brace to learn the desired physical interaction between the robotic brace and the human torso. A five-dimensional (5D) three-layer DT is also developed to be used for training the RLPIC in a simulated environment. The 5D three-layer DT consists of a physical system, a three-layer digital model of the physical system, including the robotic brace, human torso, and the physical human–robot interaction (HRI), a bidirectional connection between them, and an optimization dimension. A neural network-based regression model is also proposed to estimate the unknown parameters of the digital model. Numerical simulations and real-time experiments are performed to validate the 5D three-layer DT model. The proposed RLPIC trained using the 5D three-layer DT is verified using numerical simulations in terms of position tracking, velocity tracking, and HRI control. It is concluded that the proposed learning-based interaction control approach can improve the HRI control by learning the desired interaction in a simulated environment.

Keywords

adolescent idiopathic scoliosis / deep reinforcement learning / digital twin / reinforcement learning-based impedance control / robotic rehabilitation

References

[1] T. Maruyama, K. Takesita, T. Kitagawa, and Y. Nakao, “Milwaukee Brace,” Physiotherapy Theory and Practice 27, no. 1 (2011): 43–46, https://doi.org/10.3109/09593985.2010.503992.

[2] M. J. Khan, V. M. Srinivasan, and A. H. Jea, “The History of Bracing for Scoliosis,” Clinical Pediatrics 55, no. 4 (2016): 320–325, https://doi.org/10.1177/0009922815615829.

[3] H. R. Weiss and M. Werkmann, “ ‘Brace Technology’ Thematic Series—The Scoliologic® Chêneau Light™ Brace in the Treatment of Scoliosis,” Scoliosis 5, no. 1 (2010): 19, https://doi.org/10.1186/1748-7161-5-19.

[4] C. T. Price, D. S. Scott, F. E. Reed, , and M. Riddick, “Nighttime Bracing for Adolescent Idiopathic Scoliosis With the Charleston Bending Brace. Preliminary Report,” Spine 15, no. 12 (1990): 1294–1299, https://doi.org/10.1097/00007632-199012000-00011.

[5] R. S. Fayssoux, R. H. Cho, and M. J. Herman, “A History of Bracing for Idiopathic Scoliosis in North America,” Clinical Orthopaedics & Related Research 468, no. 3 (2010): 654–664, https://doi.org/10.1007/s11999-009-0888-5.

[6] A. G. Veldhuizen, J. Cheung, G. J. Bulthuis, and G. Nijenbanning, “A New Orthotic Device in the Non-Operative Treatment of Idiopathic Scoliosis,” Medical Engineering & Physics 24, no. 3 (2002): 209–218, https://doi.org/10.1016/S1350-4533(02)00008-5.

[7] M. S. Wong, J. C. Y. Cheng, T. P. Lam, et al., “The Effect of Rigid Versus Flexible Spinal Orthosis on the Clinical Efficacy and Acceptance of the Patients With Adolescent Idiopathic Scoliosis,” Spine 33, no. 12 (2008): 1360–1365, https://doi.org/10.1097/BRS.0b013e31817329d9.

[8] A. Ali, V. Fontanari, M. Fontana, and W. Schmölz, “Spinal Deformities and Advancement in Corrective Orthoses,” Bioengineering 8, no. 1 (2021): 2, https://doi.org/10.3390/bioengineering8010002.

[9] P. Joon-Hyuk, P. Stegall, and S. K. Agrawal. “Dynamic Brace for Correction of Abnormal Postures of the Human Spine.” in 2015 IEEE International Conference on Robotics and Automation (ICRA), (2015), 5922–5927, https://doi.org/10.1109/ICRA.2015.7140029.

[10] A. Ali, V. Fontanari, W. Schmölz, and S. K. Agrawal, “Active Soft Brace for Scoliotic Spine: A Finite Element Study to Evaluate In-Brace Correction,” Robotics 11, no. 2 (2022): 37, https://doi.org/10.3390/robotics11020037.

[11] R. Ray, L. Nouaille, B. Colobert, L. Calistri, and G. Poisson, “Design and Position Control of a Robotic Brace Dedicated to the Treatment of Scoliosis,” Robotica 41, no. 5 (2023): 1466–1482, https://doi.org/10.1017/S0263574722001825.

[12] A. L. Jutinico, J. C. Jaimes, F. M. Escalante, J. C. Perez-Ibarra, M. H. Terra, and A. A. G. Siqueira, “Impedance Control for Robotic Rehabilitation: A Robust Markovian Approach,” Frontiers in Neurorobotics 11 (2017): 11–43, https://doi.org/10.3389/fnbot.2017.00043.

[13] Y. Zhou, J. She, Z.-T. Liu, C. Xu, and Z. Yang. “Implementation of Impedance Control for Lower-Limb Rehabilitation Robots.” in 2021 4th IEEE International Conference on Industrial Cyber-Physical Systems (ICPS), Victoria, bc, Canada, (2021), 700–704, https://doi.org/10.1109/ICPS49255.2021.9468210.

[14] J. Bai, A. Song, T. Wang, and H. Li, “A Novel Backstepping Adaptive Impedance Control for an Upper Limb Rehabilitation Robot,” Computers & Electrical Engineering 80 (2019): 106465, https://doi.org/10.1016/j.compeleceng.2019.106465.

[15] Z. Li, Z. Huang, W. He, and C. Y. Su, “Adaptive Impedance Control for an Upper Limb Robotic Exoskeleton Using Biological Signals,” IEEE Transactions on Industrial Electronics 64, no. 2 (2017): 1664–1674, https://doi.org/10.1109/TIE.2016.2538741.

[16] L. Luo, L. Peng, Z. Hou, and W. Wang. “An Adaptive Impedance Controller for Upper Limb Rehabilitation Based on Estimation of Patients’ Stiffness.” in 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), Macau, Macao (2017), 532–537, https://doi.org/10.1109/ROBIO.2017.8324471.

[17] F. Farhadiyadkuri, A. M. Popal, S. S. Paiwand, and X. Zhang, “Interaction Dynamics Modeling and Adaptive Impedance Control of Robotic Exoskeleton for Adolescent Idiopathic Scoliosis,” Computers in Biology and Medicine 145 (2022): 105495, https://doi.org/10.1016/j.compbiomed.2022.105495.

[18] F. Farhadiyadkuri and X. Zhang, “Novel Interaction Control in Adolescent Idiopathic Scoliosis Treatment Using a Robotic Brace,” Journal of Intelligent & Robotic Systems 109 (2023): 73, https://doi.org/10.1007/s10846-023-02010-1.

[19] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (The MIT Press, 2018).

[20] M. P. Deisenroth, “A Survey on Policy Search for Robotics,” Foundations and Trends in Robotics 2, no. 1–2 (2013): 1–142, https://doi.org/10.1561/2300000021.

[21] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement Learning in Robotics: A Survey,” International Journal of Robotics Research 32, no. 11 (2013): 1238–1274, https://doi.org/10.1177/0278364913495721.

[22] P. Kormushev, S. Calinon, and D. Caldwell, “Reinforcement Learning in Robotics: Applications and Real-World Challenges,” Robotics 2, no. 3 (2013): 122–148, https://doi.org/10.3390/robotics2030122.

[23] K. Chatzilygeroudis, V. Vassiliades, F. Stulp, S. Calinon, and J. B. Mouret, “A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials,” IEEE Transactions on Robotics 36, no. 2 (2020): 328–347, https://doi.org/10.1109/TRO.2019.2958211.

[24] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep Reinforcement Learning: A Brief Survey,” IEEE Signal Processing Magazine 34, no. 6 (2017): 26–38, https://doi.org/10.1109/MSP.2017.2743240.

[25] T. Erol, A. F. Mendi, and D. Doğan. “The Digital Twin Revolution in Healthcare,” 2020 4th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) 22–24, (2020): 1–7, https://doi.org/10.1109/ISMSIT50672.2020.9255249.

[26] B. R. Barricelli, E. Casiraghi, and D. Fogli, “A Survey on Digital Twin: Definitions, Characteristics, Applications, and Design Implications,” IEEE Access 7 (2019): 167653–167671, https://doi.org/10.1109/ACCESS.2019.2953499.

[27] H. X. Nguyen, R. Trestian, D. To, and M. Tatipamula, “Digital Twin for 5G and Beyond,” IEEE Communications Magazine 59, no. 2 (2021): 10–15, https://doi.org/10.1109/MCOM.001.2000343.

[28] M. Grieves, “ Digital Twin: Manufacturing Excellence Through Virtual Factory Replication,” White Paper (2014): 1, https://www.3ds.com/fileadmin/PRODUCTS-SERVICES/DELMIA/PDF/Whitepaper/DELMIA-APRISO-Digital-Twin-Whitepaper.pdf.

[29] A. Thelen, X. Zhang, O. Fink, et al., “A Comprehensive Review of Digital Twin—Part 1: Modeling and Twinning Enabling Technologies,” Structural and Multidisciplinary Optimization 65, no. 12 (2022): 354, https://doi.org/10.1007/s00158-022-03425-4.

[30] A. Thelen, X. Zhang, O. Fink, et al., “A Comprehensive Review of Digital Twin—Part 2: Roles of Uncertainty Quantification and Optimization, a Battery Digital Twin, and Perspectives,” Structural and Multidisciplinary Optimization 66, no. 1 (2022): 1, https://doi.org/10.1007/s00158-022-03410-x.

[31] W. Wang, Y. He, F. Li, J. Li, J. Liu, and X. Wu, “Digital Twin Rehabilitation System Based on Self-Balancing Lower Limb Exoskeleton,” Technology and Health Care 31, no. 1 (2023): 103–115, https://doi.org/10.3233/THC-220087.

[32] M. M. R. Khan, M. S. H. Sunny, T. Ahmed, et al., “Development of a Robot-Assisted Telerehabilitation System With Integrated IIoT and Digital Twin,” IEEE Access 11 (2023): 70174–70189, https://doi.org/10.1109/ACCESS.2023.3291803.

[33] F. Tao, H. Zhang, A. Liu, and A. Y. C. Nee, “Digital Twin in Industry: State-of-the-Art,” IEEE Transactions on Industrial Informatics 15, no. 4 (2019): 2405–2415.

[34] H. D. Taghirad, Parallel Robots: Mechanics and Control (CRC Press, 2013), https://doi.org/10.1201/b16096.

[35] A. Ahmad, N. Abu Osman, H. Mokhtar, W. Mehmood, and N. A. Kadri, “Analysis of the Interface Pressure Exerted by the Chêneau Brace in Patients With Double-Curve Adolescent Idiopathic Scoliosis,” Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine 233, no. 9 (2019): 901–908, https://doi.org/10.1177/0954411919856144.

[36] V. M. Pham, A. Houilliez, A. Schill, A. Carpentier, B. Herbaux, and A. Thevenon, “Study of the Pressures Applied by a Chêneau Brace for Correction of Adolescent Idiopathic Scoliosis,” Prosthetics & Orthotics International 32, no. 3 (2008): 345–355, https://doi.org/10.1080/03093640802016092.

[37] J. C. Gesbert, B. Colobert, L. Rakotomanana, and P. Violas, “Idiopathic Scoliosis and Brace Treatment: An Innovative Device to Assess Corrective Pressure,” Computer Methods in Biomechanics and Biomedical Engineering 24, no. 2 (2021): 131–136, 1080/10255842.2020.1813729.

[38] F. K. Fuss, A. Ahmad, A. M. Tan, R. Razman, and Y. Weizman, “Pressure Sensor System for Customized Scoliosis Braces,” Sensors 21, no. 4 (2021): 1153, https://doi.org/10.3390/s21041153.

[39] A. S. Anand, G. Zhao, H. Roth, and A. Seyfarth. “A Deep Reinforcement Learning Based Approach Towards Generating Human Walking Behavior With a Neuromuscular Model.” in 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), Toronto, ON, Canada (2019), 537–543, https://doi.org/10.1109/Humanoids43949.2019.9035034.

[40] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. “Proximal Policy Optimization Algorithms.” (2017), https://doi.org/10.48550/arXiv.1707.06347.

[41] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable Baselines3 (2023), https://github.com/DLR-RM/stable-baselines3.

[42] A. Chawla, S. Mukherjee, and B. Karthikeyan, “Mechanical Properties of Soft Tissues in the Human Chest, Abdomen and Upper Extremities,” Institution of Engineers, Journal of Mechanical Engineering, Technical Report (2013): 1.

[43] P. T. Boggs and J. W. Tolle, “Sequential Quadratic Programming,” Acta Numerica 4 (1995): 1–51, https://doi.org/10.1017/S0962492900002518.

[44] H. F. N. Al-Shuka, S. Leonhardt, W. H. Zhu, R. Song, C. Ding, and Y. Li, “Active Impedance Control of Bioinspired Motion Robotic Manipulators: An Overview,” Applied Bionics and Biomechanics 2018 (2018): 8203054, https://doi.org/10.1155/2018/8203054.

[45] A. Y. Mersha, S. Stramigioli, and R. Carloni. “Variable Impedance Control for Aerial Interaction,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (2014): 3435–3440, https://doi.org/10.1109/IROS.2014.6943041.

[46] X. Zhang, L. Sun, Z. Kuang, and M. Tomizuka, “Learning Variable Impedance Control via Inverse Reinforcement Learning for Force-Related Tasks,” IEEE Robotics and Automation Letters 6, no. 2 (2021): 2225–2232, https://doi.org/10.1109/LRA.2021.3061374.

[47] L. Roveda, J. Maskani, P. Franceschi, et al., “Model-Based Reinforcement Learning Variable Impedance Control for Human-Robot Collaboration,” Journal of Intelligent & Robotic Systems 100 (2020): 417–433, https://doi.org/10.1007/s10846-020-01183-3.

RIGHTS & PERMISSIONS

2025 The Author(s). International Journal of Mechanical System Dynamics published by John Wiley & Sons Australia, Ltd on behalf of Nanjing University of Science and Technology.