Motion Trajectory Estimation for Hand Grasping States Using a Deep Learning Approach

Abstract

Predicting the final grasp tendency at the start of movement in prosthetic hands is crucial for improved control. Biological data, such as 3D movement and muscle activity, have been using by researchers to predict the final grasp. Early prediction of the intended grasp allows the prosthetic device to initiate control actions before the motion is complete, resulting in faster and more intuitive responses. Most machine learning algorithms are trained to predict the gesture of the final grasp. The aim of this study is to accurately estimate the final grasp state using inertial measurement unit (IMU) data. This estimation, based on movement trajectories, will allow prosthetic devices to respond more quickly to user actions. Deep Learning model was trained using movement data collected from a prosthetic hand controlled certain gesture trajectories without any human involvement. Data such as acceleration, angular velocity, and orientation were gathered through IMU sensors to create 3D orientation matrices representing the movement process. A deep convolutional neural network was used for training, with data labeled by the final grasp states. The deep learning algorithm successfully predicted the final hand motion with 93% accuracy. This trained model enables the generation of smooth supervisory trajectories, facilitating faster and more accurate control of the prosthesis. The proposed model demonstrates significant potential in improving prosthetic hand control by predicting the final hand movement at an early stage of motion, contributing to more responsive and effective prosthetic devices.

Keywords

trajectory estimation / deep learning / prosthesis

References

[1] Yadav, D., & Veer, K. (2023). Recent trends and challenges of surface electromyography in prosthetic applications. Biomedical Engineering Letters, 13(3), 353-373. https://doi.org/10.1007/s13534-023-00281-z

[2] Lee, K. H., Min, J. Y., & Byun, S. (2021). Electromyogram—based classification of hand and finger gestures using artificial neural networks. Sensors, 22(1), 225. https://doi.org/10.3390/s22010225

[3] Shi, H., Jiang, X., Dai, C., & Chen, W. (2024). EMG—based multi—user hand gesture classification via unsupervised transfer learning using unknown calibration gestures. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 32, 1119-1131. https://doi.org/10.1109/TNSRE.2024.3372002

[4] Mahmood, N. T., Al—Muifraje, M. H., Saeed, T. R., & Kaittan, A. H. (2020). Upper prosthetic design based on EMG: A systematic review. IOP Conference Series: Materials Science and Engineering, 978(1), 012025. https://doi.org/10.1088/1757-899X/978/1/012025

[5] Boyer, M., Bouyer, L., Roy, J. S., & Campeau—Lecours, A. (2023). Reducing noise, artifacts and interference in single—channel EMG signals: A review. Sensors, 23(6), 2927. https://doi.org/10.3390/s23062927

[6] Kumar, S. A., Chand, R., Chand, R. P., & Sharma, B. (2023). Linear manipulator: Motion control of an n—link robotic arm mounted on a mobile slider. Heliyon, 9(1), e12867. https://doi.org/10.1016/j.heliyon.2023.e12867

[7] Weingartshofer, T., Bischof, B., Meiringer, M., Hartl—Nesic, C., & Kugi, A. (2023). Optimization—based path planning framework for industrial manufacturing processes with complex continuous paths. Robotics and Computer—Integrated Manufacturing, 82, 102516. https://doi.org/10.1016/j.rcim.2022.102516

[8] Nasr, A., Bell, S., & McPhee, J. (2023). Optimal design of active—passive shoulder exoskeletons: A computational modeling of human—robot interaction. Multibody System Dynamics, 57(1), 73-106. https://doi.org/10.1007/s11044-022-09855-8

[9] Peternel, L., Petrič, T., Oztop, E., & Babič, J. (2014). Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi—modal human—in—the—loop approach. Autonomous Robots, 36(1), 123-136. https://doi.org/10.1007/s10514-013-9361-0

[10] Gozzi, N., Malandri, L., Mercorio, F., & Pedrocchi, A. (2022). XAI for myo—controlled prosthesis: Explaining EMG data for hand gesture classification. Knowledge—Based Systems, 240, 108053. https://doi.org/10.1016/j.knosys.2021.108053

[11] Uyanik, C., Hussaini, S. F., Erdemir, E., Kaplanoglu, E., & Sekmen, A. (2019). A deep learning approach for final grasping state determination from motion trajectory of a prosthetic hand. Procedia Computer Science, 158, 19-26. https://doi.org/10.1016/j.procs.2019.09.023

[12] Dudley, J., Yin, L., Garaj, V., & Kristensson, P. O. (2023). Inclusive immersion: A review of efforts to improve accessibility in virtual reality, augmented reality and the metaverse. Virtual Reality, 27(4), 2989-3020. https://doi.org/10.1007/s10055-023-00850-8

[13] Liu, Y., Jiang, L., Liu, H., & Ming, D. (2021). A systematic analysis of hand movement functionality: Qualitative classification and quantitative investigation of hand grasp behavior. Frontiers in Neurorobotics, 15, 658075. https://doi.org/10.3389/fnbot.2021.658075

[14] Schreuders, T. A. R., Brandsma, J. W., & Stam, H. J. (2014). Functional anatomy and biomechanics of the hand. In M. T. Duruöz (Ed.), Hand function: A practical guide to assessment (pp. 3-22). Springer. https://doi.org/10.1007/978-1-4614-9449-2_1

[15] Liu, M. J., Xiong, C. H., Xiong, L., & Huang, X. L. (2016). Biomechanical characteristics of hand coordination in grasping activities of daily living. PLOS ONE, 11(1), e0146193. https://doi.org/10.1371/journal.pone.0146193

[16] Lee, K. S., & Jung, M. C. (2015). Ergonomic evaluation of biomechanical hand function. Safety and Health at Work, 6(1), 9-17. https://doi.org/10.1016/j.shaw.2014.09.002

[17] Fowler, N. K., & Nicol, A. C. (2001). Functional and biomechanical assessment of the normal and rheumatoid hand. Clinical Biomechanics, 16(8), 660-666. https://doi.org/10.1016/S0268-0033(01)00057-2

[18] Gu, C., Lin, W., He, X., Zhang, L., & Zhang, M. (2023). IMU—based motion capture system for rehabilitation applications: A systematic review. Biomimetic Intelligence and Robotics, 3(2), 100097. https://doi.org/10.1016/j.birob.2023.100097

[19] Caramia, C., Torricelli, D., Schmid, M., Munoz—Gonzalez, A., Gonzalez—Vargas, J., Grandas, F., & Pons, J. L. (2018). IMU—based classification of Parkinson’s disease from gait: A sensitivity analysis on sensor location and feature selection. IEEE Journal of Biomedical and Health Informatics, 22(6), 1765-1774. https://doi.org/10.1109/JBHI.2018.2865218

[20] Arlotti, J. S., Carroll, W. O., Afifi, Y., Talegaonkar, P., Albuquerque, L., Burch, R. F. V., :::, & Petway, A. (2022). Benefits of IMU—based wearables in sports medicine: Narrative review. International Journal of Kinesiology and Sports Science, 10(1), 36-43. https://doi.org/10.7575/aiac.ijkss.v.10n.1p.36

[21] Deng, L., & Yu, D. (2014). Deep learning: Methods and applications. Foundations and Trends in Signal Processing, 7(3—4), 197-387. https://doi.org/10.1561/2000000039

[22] Sekmen, A., Koku, A. B., Parlaktuna, M., Abdul—Malek, A., & Vanamala, N. (2017). Unsupervised deep learning for subspace clustering. In 2017 IEEE International Conference on Big Data, 2089-2094. https://doi.org/10.1109/BigData.2017.8258156

[23] Xiong, D., Zhang, D., Zhao, X., & Zhao, Y. (2021). Deep learning for EMG—based human—machine interaction: A review. IEEE/CAA Journal of Automatica Sinica, 8(3), 512-533. https://doi.org/10.1109/JAS.2021.1003865

[24] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. https://doi.org/10.1038/nature14539

[25] Ghafari, M., Mailman, D., Hatami, P., Peyton, T., Yang, L., Dang, W., & Qin, H. (2022). A comparison of YOLO and Mask—RCNN for detecting cells from microfluidic images. In International Conference on Artificial Intelligence in Information and Communication, 204-209. https://doi.org/10.1109/ICAIIC54071.2022.9722616

[26] Tran, T. V., Khaleghian, S., Zhao, J., & Sartipi, M. (2022). SIMCal: A high—performance toolkit for calibrating traffic simulation. In 2022 IEEE International Conference on Big Data, 2895-2902. https://doi.org/10.1109/BigData55660.2022.10021057

[27] Rahmati, M., Cho, J., Fell, N., & Sartipi, M. (2022). Developing prediction models for 30—day readmission after stroke among medicare beneficiaries. In SoutheastCon 2022, 551-557. https://doi.org/10.1109/SoutheastCon48659.2022.9763936

[28] Lee, W., Seong, J. J., Ozlu, B., Shim, B. S., Marakhimov, A., & Lee, S. (2021). Biosignal sensors and deep learning—based speech recognition: A review. Sensors, 21(4), 1399. https://doi.org/10.3390/s21041399

[29] Ong, Z. C., Seet, Y. C., Khoo, S. Y., & Noroozi, S. (2018). Development of an economic wireless human motion analysis device for quantitative assessment of human body joint. Measurement, 115, 306-315. https://doi.org/10.1016/j.measurement.2017.10.056

[30] Barraza Madrigal, J. A., Contreras Rodríguez, L. A., Cardiel Pérez, E., Hernández Rodríguez, P. R., & Sossa, H. (2023). Hip and lower limbs 3D motion tracking using a double—stage data fusion algorithm for IMU/MARG—based wearables sensors. Biomedical Signal Processing and Control, 86, 104938. https://doi.org/10.1016/j.bspc.2023.104938

[31] Tian, J., Xiang, A., Feng, Y., Yang, Q., & Liu, H. (2024). Enhancing disease prediction with a hybrid CNN—LSTM framework in EHRs. Journal of Theory and Practice of Engineering Science, 4(02), 8-14. https://doi.org/10.53469/jtpes.2024.04(02).02

[32] Dua, N., Singh, S. N., Challa, S. K., Semwal, V. B., & Morla, L. S. S. K. (2022). A survey on human activity recognition using deep learning techniques and wearable sensor data. In Machine Learning, Image Processing, Network Security and Data Sciences: 4th International Conference, 52-71. https://doi.org/10.1007/978-3-031-24352-3_5

[33] Chen, B., Zhou, Y., Chen, C., Sayeed, Z., Hu, J., Qi, J., :::, & Palacio, C. (2023). Volitional control of upper—limb exoskeleton empowered by EMG sensors and machine learning computing. Array, 17, 100277. https://doi.org/10.1016/j.array.2023.100277

[34] Dai, N., Lei, I. M., Li, Z., Li, Y., Fang, P., & Zhong, J. (2023). Recent advances in wearable electromechanical sensors—Moving towards machine learning—assisted wearable sensing systems. Nano Energy, 105, 108041. https://doi.org/10.1016/j.nanoen.2022.108041

[35] Chryssolouris, G., Alexopoulos, K., & Arkouli, Z. (2023). Artificial intelligence in manufacturing equipment, automation, and robots. In G. Chryssolouris, K. Alexopoulos, & Z. Arkouli (Eds.), A perspective on artificial intelligence in manufacturing (pp. 41-78). Springer. https://doi.org/10.1007/978-3-031-21828-6_3

[36] Li, S., Zheng, P., Liu, S., Wang, Z., Wang, X. V., Zheng, L., & Wang, L. (2023). Proactive human—robot collaboration: Mutual—cognitive, predictable, and self—organising perspectives. Robotics and Computer—Integrated Manufacturing, 81, 102510. https://doi.org/10.1016/j.rcim.2022.102510

[37] Atasoy, A., Kaya, E., Toptas, E., Kuchimov, S., Kaplanoglu, E., & Ozkan, M. (2016). 24 DOF EMG controlled hybrid actuated prosthetic hand. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 5059-5062. https://doi.org/10.1109/EMBC.2016.7591864

[38] Tan, H., & Kawamura, K. (2015). Generation of acceptable actions using imitation learning, intention recognition, and cognitive control. In 2015 24th IEEE International Symposium on Robot and Human Interactive Communication, 389-393. https://doi.org/10.1109/ROMAN.2015.7333662

[39] Gadaleta, M., Merelli, L., & Rossi, M. (2016). Human authentication from ankle motion data using convolutional neural networks. In 2016 IEEE Statistical Signal Processing Workshop, 1-5. https://doi.org/10.1109/SSP.2016.7551815

[40] Dehzangi, O., Taherisadr, M., & ChangalVala, R. (2017). IMU—based gait recognition using convolutional neural networks and multi—sensor fusion. Sensors, 17(12), 2735. https://doi.org/10.3390/s17122735

[41] Peng, Y., Sakai, Y., Funabora, Y., Yokoe, K., Aoyama, T., & Doki, S. (2025). Funabot—Sleeve: A wearable device employing McKibben artificial muscles for haptic sensation in the forearm. IEEE Robotics and Automation Letters, 10(2), 1944-1951. https://doi.org/10.1109/LRA.2025.3528229

[42] Peng, Y., Yang, X., Li, D., Ma, Z., Liu, Z., Bai, X., & Mao, Z. (2025). Predicting flow status of a flexible rectifier using cognitive computing. Expert Systems with Applications, 264, 125878. https://doi.org/10.1016/j.eswa.2024.125878

[43] Moreno, J. C., de Lima, E. R., Ruíz, A. F., Brunetti, F. J., & Pons, J. L. (2006). Design and implementation of an inertial measurement unit for control of artificial limbs: Application on leg orthoses. Sensors and Actuators B: Chemical, 118(1—2), 333-337. https://doi.org/10.1016/j.snb.2006.04.039

[44] Stephenson, R. M., Naik, G. R., & Chai, R. (2017). A system for accelerometer—based gesture classification using artificial neural networks. In 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 4187-4190. https://doi.org/10.1109/EMBC.2017.8037779

[45] Anand, A., Sharma, M., Srivastava, R., Kaligounder, L., & Prakash, D. (2017). Wearable motion sensor based analysis of swing sports. In 2017 16th IEEE International Conference on Machine Learning and Applications, 261-267. https://doi.org/10.1109/ICMLA.2017.0-149

[46] Zhao, H., Zhang, X., Chen, X., & Zhou, P. (2024). A robust myoelectric pattern recognition framework based on individual motor unit activities against electrode array shifts. Computer Methods and Programs in Biomedicine, 257, 108434. https://doi.org/10.1016/j.cmpb.2024.108434

[47] Sanchez—Londono, D., Posada—Ceron, S., Barbieri, G., & Rodriguez, C. F. (2022). A design of machinery learning activity to develop critical thinking. International Journal of Mechanical Engineering Education, 50(3), 704-725. https://doi.org/10.1177/03064190211057824

[48] Sollerman, C., & Ejeskär, A. (1995). Sollerman hand function test: A standardised method and its use in tetraplegic patients. Scandinavian Journal of Plastic and Reconstructive Surgery and Hand Surgery, 29(2), 167-176. https://doi.org/10.3109/02844319509034334

[49] Taylor, C. L., & Schwarz, R. J. (1955). The anatomy and mechanics of the human hand. Artificial Limbs, 2(2), 22-35.

[50] Ulkir, O., Akgun, G., Toptas, E., & Kaplanoglu, E. (2020). Design and myoelectric control of an active orthosis device using finite state machine algorithm. Technium: Romanian Journal of Applied Sciences and Technology, 2(7), 286-296. https://doi.org/10.47577/technium.v2i7.2122