Multimodal wearable sensors-driven KsFormer model for continuous multi-step ahead prediction of lower limb joint moments and ground reaction forces

Abstract

Accurate real-time prediction of lower limb biomechanics is critical for enhancing assistive robotic control systems. While machine learning approaches have emerged as computationally efficient alternatives to traditional musculoskeletal multibody dynamics simulations, existing methods faced persistent challenges including limited prediction accuracy, inefficient multi-modal feature integration, and latency constraints inherent to current-time prediction frameworks. Therefore, this study aimed to propose a wearable sensors-driven deep learning model (KsFormer) for continuous multi-step ahead prediction of sagittal-plane joint moments (hip, knee, and ankle) and three-dimensional ground reaction forces (GRFs) across complete gait cycles. The encoder–decoder model KsFormer was specifically designed for cross-modal feature extraction and integration, with its encoder adopting a three-stage hierarchical processing pipeline. The preprocessed inertial measurement unit (IMU) kinematics and surface electromyography (sEMG) data recorded by wearable sensors served as the inputs of KsFormer. The prediction results were then validated by comparing them to those from gold-standard musculoskeletal simulations and force plate measurements. The results demonstrated exceptional predictive performance with mean Pearson correlation coefficients exceeding 0.9 across six walking speeds and three running speeds patterns, achieving low error rates ($\overline{\text{RMSE}}$ = 0.092 N m/kg for joint moments; $\overline{\text{RMSE}}$ = 0.032 body weight for GRFs). Additionally, the proposed model enabled accurate and continuous biomechanical prediction 240–960 ms prior to motion initiation, significantly outperforming conventional current-time prediction approaches. This study provided a more practical method for real-time lower limb biomechanics feedback to the assistive robotic system in the real-world environment, enabling dynamic torque adjustment and pilot gait pattern recognition.

Keywords

Deep learning / Ground reaction force / Joint moment / Lower limb biomechanics / Wearable sensor

CRediT authorship contribution statement

Hao Zhou: Writing – original draft, Visualization, Validation, Software, Methodology, Formal analysis, Data curation. Yinghu Peng: Writing – review & editing, Visualization, Software, Methodology, Data curation, Conceptualization. Xiaohui Li: Visualization, Software, Data curation. Xueyan Lyu: Visualization, Validation, Software, Data curation. Dahua Shou: Writing – review & editing, Supervision, Funding acquisition. Guanglin Li: Writing – review & editing, Supervision, Funding acquisition. Lin Wang: Writing – review & editing, Supervision, Funding acquisition.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors greatly appreciate the comprehensive kinetic and EMG dataset provided by Wang et al. (University of Twente, DOI: https://doi.org/10.5281/zenodo.6457662)

This work was supported in part by National Key Research and Development Program of China (2024YFE0216500), in part by the National Natural science Foundation of China (62573402), in part by Shenzhen Strategic Emerging Industry Support Plans (XMHT20230115002), in part by Shenzhen Sustainable Development Sci-Tech project (KCXFZ20230731093501003), in part by Shenzhen Science and Technology Program (KQTD20210811090217009), in part by Shenzhen Science and Technology Program (JCYJ20240813154923031), and in part by the Guangdong Basic and Applied Basic Research Foundation (2025A1515011989).

References

[1] D.G. Schmitz, R.W. Nuckols, S. Lee, T. Akbas, K. Swaminathan, C.J. Walsh, D.G. Thelen, Modulation of Achilles tendon force with load carriage and exosuit assistance, Sci. Robot. 7 (71) (2022) eabq1514.

[2] F. Barberi, E. Anselmino, A. Mazzoni, M. Goldfarb, S. Micera, Toward the development of user-centered neurointegrated lower limb prostheses, IEEE Rev. Biomed. Eng. 17 (2023) 212-228.

[3] P. Slade, M.J. Kochenderfer, S.L. Delp, S.H. Collins, Personalizing exoskeleton assistance while walking in the real world, Nature 610 (7931) (2022) 277-282.

[4] D.E. Miller, G.R. Tan, E.M. Farina, A.L. Sheets-Singer, S.H. Collins, Characterizing the relationship between peak assistance torque and metabolic cost reduction during running with ankle exoskeletons, J. Neuroeng. Rehabil. 19 (1) (2022) 46.

[5] A. Lakmazaheri, S. Song, B.B. Vuong, B. Biskner, D.M. Kado, S.H. Collins, Optimizing exoskeleton assistance to improve walking speed and energy economy for older adults, J. NeuroEng. Rehabil. 21 (1) (2024) 1.

[6] P.W. Franks, G.M. Bryan, R. Reyes, M.P. O’Donovan, K.N. Gregorczyk, S.H. Collins, The effects of incline level on optimized lower-limb exoskeleton assistance: A case series, IEEE Trans. Neural Syst. Rehabil. Eng. 30 (2022) 2494-2505.

[7] K.L. Poggensee, S.H. Collins, Lower limb biomechanics of fully trained exoskeleton users reveal complex mechanisms behind the reductions in energy cost with human-in-the-loop optimization, Front. Robot. AI 11 (2024) 1283080.

[8] R.W. Nuckols, C.-K. Chang, D. Kim, A. Eckert-Erdheim, D. Orzel, L. Baker, T. Baker, N.C. Wendel, B. Quinlivan, P. Murphy, et al., Design and evaluation of an independent 4-week, exosuit-assisted, post-stroke community walking program, Ann. New York Acad. Sci. 1525 (1) (2023) 147-159.

[9] L.H. Sloot, L.M. Baker, J. Bae, F. Porciuncula, B.F. Clément, C. Siviy, R.W. Nuckols, T. Baker, R. Sloutsky, D.K. Choe, et al., Effects of a soft robotic exosuit on the quality and speed of overground walking depends on walking ability after stroke, J. Neuroeng. Rehabil. 20 (1) (2023) 113.

[10] C.L. Dembia, N.A. Bianco, A. Falisse, J.L. Hicks, S.L. Delp, OpenSim Moco: musculoskeletal optimal control, PLoS Comput. Biol. 16 (12) (2020) e1008493.

[11] M.E. Lund, M.S. Andersen, M. de Zee, J. Rasmussen, Scaling of musculoskeletal models from static and dynamic trials, Int. Biomech. 2 (1) (2015) 1-11.

[12] Y. Peng, D.W.-C. Wong, Y. Wang, T.L.-W. Chen, Q. Tan, Z. Chen, Z. Jin, M. Zhang, Immediate effects of medially posted insoles on lower limb joint contact forces in adult acquired flatfoot: a pilot study, Int. J. Environ. Res. Public Health 17 (7) (2020) 2226.

[13] S. Skals, M.K. Jung, M. Damsgaard, M.S. Andersen, Prediction of ground reaction forces and moments during sports-related movements, Multibody Syst. Dyn. 39 (3) (2017) 175-195.

[14] X. Wang, J. Guo, Q. Tian, A forward-inverse dynamics modeling framework for human musculoskeletal multibody system, Acta Mech. Sin. 38 (11) (2022) 522140.

[15] H. Hatze, The fundamental problem of myoskeletal inverse dynamics and its implications, J. Biomech. 35 (1) (2002) 109-115.

[16] M. Abdullah, A.A. Hulleck, R. Katmah, K. Khalaf, M. El-Rich, Multibody dynamics-based musculoskeletal modeling for gait analysis: a systematic review, J. NeuroEng. Rehabil. 21 (1) (2024) 178.

[17] D. Xu, H. Zhou, W. Quan, X. Ma, T.-E. Chon, J. Fernandez, F. Gusztav, A. Kovács, J.S. Baker, Y. Gu, New insights optimize landing strategies to reduce lower limb injury risk, Cyborg Bionic Syst. 5 (2024) 0126.

[18] B.X. Liew, D. Rügamer, Q. Mei, Z. Altai, X. Zhu, X. Zhai, N. Cortes, Smooth and accurate predictions of joint contact force time-series in gait using over parameterised deep neural networks, Front. Bioeng. Biotechnol. 11 (2023) 1208711.

[19] D. Liu, Y. Ma, J. Wang, M. Hou, C. Zhang, Convolutional neural network based ground reaction forces and center of pressure estimation during stair walking using multi-level kinematics, Expert Syst. Appl. 243 (2024) 122868.

[20] L. Xiang, Y. Gu, Z. Gao, P. Yu, V. Shim, A. Wang, J. Fernandez, Integrating an LSTM framework for predicting ankle joint biomechanics during gait using inertial sensors, Comput. Biol. Med. 170 (2024) 108016.

[21] T. Sun, D. Li, B. Fan, T. Tan, P.B. Shull, Real-time ground reaction force and knee extension moment estimation during drop landings via modular lstm modeling and wearable imus, IEEE J. Biomed. Health Inform. 27 (7) (2023) 3222-3233.

[22] M. Mansour, K. Serbest, M. Kutlu, M. Cilli, Estimation of lower limb joint moments based on the inverse dynamics approach: a comparison of machine learning algorithms for rapid estimation, Med. Biol. Eng. Comput. 61 (12) (2023) 3253-3276.

[23] C. Wang, P.P. Chan, B.M. Lam, S. Wang, J.H. Zhang, Z.Y. Chan, R.H. Chan, K.K. Ho, R.T. Cheung, Real-time estimation of knee adduction moment for gait retraining in patients with knee osteoarthritis, IEEE Trans. Neural Syst. Rehabil. Eng. 28 (4) (2020) 888-894.

[24] S.M. Moghadam, T. Yeung, J. Choisne, A comparison of machine learning models’ accuracy in predicting lower-limb joints’ kinematics, kinetics, and muscle forces from wearable sensors, Sci. Rep. 13 (1) (2023) 5046.

[25] J. Camargo, D. Molinaro, A. Young, Predicting biological joint moment during multiple ambulation tasks, J. Biomech. 134 (2022) 111020.

[26] Y. Zhang, J. Xiong, H. Xian, C. Chen, X. Chen, H. Liang, C. Fu, Y. Leng, Joint moment estimation for hip exoskeleton control: A generalized moment feature generation method, Biomim. Intell. Robot. (2025), 100246.

[27] L. Zhang, X. Zhu, E.M. Gutierrez-Farewik, R. Wang, Ankle joint torque prediction using an NMS solver informed-ANN model and transfer learning, IEEE J. Biomed. Health Inform. 26 (12) (2022) 5895-5906.

[28] G. Giarmatzis, E.I. Zacharaki, K. Moustakas, Real-time prediction of joint forces by motion capture and machine learning, Sensors 20 (23) (2020) 6933.

[29] M. Du, B. Lv, B. Fan, X. Li, J. Yu, F. Yi, T. Liu, S. Cai, Ankle moment estimation based on a novel distributed plantar pressure sensing system, IEEE J. Biomed. Health Inform. (2024).

[30] Z. Wang, C. Chen, H. Chen, Y. Zhou, X. Wang, X. Wu, Dual transformer network for predicting joint angles and torques from Multi-Channel EMG signals in the lower limbs, IEEE J. Biomed. Health Inform. (2025).

[31] Y. Wang, X. Cheng, L. Jabban, X. Sui, D. Zhang, Motion intention prediction and joint trajectories generation toward lower limb prostheses using EMG and IMU signals, IEEE Sens. J. 22 (11) (2022) 10719-10729.

[32] C. Zhao, K. Liu, H. Zheng, W. Song, Z. Pei, W. Chen, Cross-Modality Self-Attention and Fusion-Based neural network for lower limb locomotion mode recognition, IEEE Trans. Autom. Sci. Eng. 22 (2025) 5411-5424, https://doi.org/10.1109/TASE.2024.3421276.

[33] H. Wang, A. Basu, G. Durandau, M. Sartori, A wearable real-time kinetic measurement sensor setup for human locomotion, Wearable Technol. 4 (2023) e11.

[34] P. Konrad, The ABC of EMG: A Practical Introduction to Kinesiological Electromyography, Noraxon INC., USA, (2005).

[35] S.L. Delp, F.C. Anderson, A.S. Arnold, P. Loan, A. Habib, C.T. John, E. Guendelman, D.G. Thelen, OpenSim: open-source software to create and analyze dynamic simulations of movement, IEEE Trans. Biomed. Eng. 54 (11) (2007) 1940-1950.

[36] A. Seth, J.L. Hicks, T.K. Uchida, A. Habib, C.L. Dembia, J.J. Dunne, C.F. Ong, M.S. DeMers, A. Rajagopal, M. Millard, et al., OpenSim: Simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement, PLoS Comput. Biol. 14 (7) (2018) e1006223.

[37] M. Żuk, M. Syczewska, C. Pezowicz, Use of the surface electromyography for a quantitative trend validation of estimated muscle forces, Biocybern. Biomed. Eng. 38 (2) (2018) 243-250.

[38] M. Zheng, P. Lu, W. Wu, R. Song, Isometric plantarflexion moment prediction based on a Compartment-Specific HD-sEMG-Driven musculoskeletal model, IEEE Trans. Biomed. Eng. 71 (8) (2024) 2311-2320.

[39] T. Lindeberg, Discrete approximations of Gaussian smoothing and Gaussian derivatives, J. Math. Imaging Vision 66 (5) (2024) 759-800.

[40] H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, M. Long, Timesnet: Temporal 2d-variation modeling for general time series analysis, Int. Conf. Learn. Represent. (2023).

[41] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818-2826.

[42] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, W. Zhang, Informer: Beyond efficient transformer for long sequence time-series forecasting, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 11106-11115.

[43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Adv. Neural Inf. Process. Syst. 30 (2017).

[44] Y. Nie, N.H. Nguyen, P. Sinthong, J. Kalagnanam, A time series is worth 64 words: Long-term forecasting with transformers, Int. Conf. Learn. Represent. (2023), p. 657.

[45] A. Zeng, M. Chen, L. Zhang, Q. Xu, Are transformers effective for time series forecasting?, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 11121-11128.

[46] Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, L. Sun, Transformers in time series: A survey, Int. Jt. Conf. Artif. Intell.(IJCAI), (2023), p. 661.

[47] R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 618-626.

[48] W. Zhong, X. Fu, M. Zhang, A muscle synergy-driven ANFIS approach to predict continuous knee joint movement, IEEE Trans. Fuzzy Syst. 30 (6) (2022) 1553-1563.

[49] Y. Lu, H. Wang, B. Zhou, C. Wei, S. Xu, Continuous and simultaneous estimation of lower limb multi-joint angles from sEMG signals based on stacked convolutional and LSTM models, Expert Syst. Appl. 203 (2022) 117340.

[50] Y. Peng, W. Wang, L. Wang, H. Zhou, Z. Chen, Q. Zhang, G. Li, Smartphone videos-driven musculoskeletal multibody dynamics modelling workflow to estimate the lower limb joint contact forces and ground reaction forces, Med. Biol. Eng. Comput. 62 (12) (2024) 3841-3853.