Reducing user training burden for myoelectric prosthetic hand control with CNN-LDA temporal-spectral transfer learning

Abstract

Modern myoelectric prosthetic hands continue to face reliability challenges due to the non-stationarities of surface electromyography (sEMG) signals, which are highly sensitive to limb positions, electrode shifts, and grasp forces. While data abundance is a common strategy to mitigate these issues, it significantly increases users’ training burden. Hand gesture recognition, which maps the spatiotemporal patterns of muscle activation to key hand gestures for daily activities, remains the standard control strategy for advanced prosthetic hands. However, while temporal information can be reliably extracted from the sEMG signals, spatial information is highly dependent on electrode placements, which vary significantly between subjects. Previous research in myoelectric hand gesture transfer learning has primarily focused on transferring either spatial information or combined spatiotemporal information, leaving the transfer of temporal information alone largely unexplored. We propose a temporal-spectral cross-subject transfer learning framework using multi-stream convolutional neural networks (CNNs), where each stream processes only a single sEMG channel. Evaluated on the Transradial Amputee sEMG Multi-Contraction Forces Dataset, our framework has achieved training accuracy of 92.73% for medium contraction force and generalization accuracy of 74.53%, outperforming several models for sEMG hand gesture recognition. It also significantly improves recognition accuracy compared to the self-training baseline with the same architecture (repeated measures t-test p $\approx$ 0.032). By excluding spatial knowledge transfer, our approach maintains high robustness even under extreme cases of channel mismatch between source and target subjects. Moreover, this study highlights the importance of CNN architecture design, and spatially agnostic feature extraction for advancing myoelectric control systems.

Keywords

Deep learning / Transfer learning / Muscle–computer interface / Surface electromyography / Prosthetic hand

CRediT authorship contribution statement

Hongquan Le: Writing – original draft, Visualization, Validation, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Marc in het Panhuis: Writing – review & editing, Supervision, Resources, Project administration, Investigation, Funding acquisition. Gursel Alici: Writing – review & editing, Supervision, Resources, Project administration, Investigation, Funding acquisition, Formal analysis, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The research presented in this article was made possible through financial support from the ARC Centre of Excellence for Electromaterials Science, Australia (CE140100012), the ARC-Discovery Project, Australia (DP210102911), and the University of Wollongong, Australia .

References

[1] N.H.H. Mohamad Hanif, P. Chappell, N. White, A. Cranny, A psychophysical investigation on vibrotactile sensing for transradial prosthesis users, Cogent Eng. 5 (1) (2018) 1539943, https://doi.org/10.1080/23311916.2018.1539943.

[2] R. Clement, K.E. Bugler, C.W. Oliver, Bionic prosthetic hands: A review of present technology and future aspirations, Surg. 9 (6) (2011) 336-340.

[3] F. Cordella, et al., Literature review on needs of upper limb prosthesis users, Front. Neurosci. 10 (2016) 209.

[4] P. Weiner, J. Starke, S. Rader, F. Hundhausen, T. Asfour, Designing prosthetic hands with embodied intelligence: The KIT prosthetic hands, Front. Neurorobotics 16 (2022) 815716, https://doi.org/10.3389/fnbot.2022.815716.

[5] I. Kyranou, S. Vijayakumar, M.S. Erden, Causes of performance degradation in electromyographic pattern recognition in upper limb prostheses, Front. Neurorobotics 12 (2018) 58, https://doi.org/10.3389/fnbot.2018.00058.

[6] D. Yang, Y. Gu, N.V. Thakor, H. Liu, Improving the functionality, robustness, and adaptability of myoelectric control for dexterous motion restoration, Exp. Brain Res. 237 (2) (2019) 291-311, https://doi.org/10.311.

[7] R.N. Khushaba, M. Takruri, J.V. Miro, S. Kodagoda, Towards limb position invariant myoelectric pattern recognition using time-dependent spectral features, Neural Netw. 55 (2014) 42-58, https://doi.org/10.1016/j.neunet.2014.03.010.

[8] J.L. Betthauser, L.E. Osborn, R.R. Kaliki, N.V. Thakor, Electrode-shift tolerant myoelectric movement-pattern classification using extreme learning for adaptive sparse representations, 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS, IEEE, 2017, pp. 1-4, https://doi.org/10.1109/BIOCAS.2017.8325201.

[9] A.H. Al-Timemy, R.N. Khushaba, G. Bugmann, J. Escudero, Improving the performance against force variation of EMG controlled multifunctional upper-limb prostheses for transradial amputees, IEEE Trans. Neural Syst. Rehabil. Eng. 24 (6) (2015) 650-661, https://doi.org/10.661.

[10] M.M.-C. Vidovic, H.-J. Hwang, S. Amsüss, J.M. Hahne, D. Farina, K.-R. Müller, Improving the robustness of myoelectric pattern recognition for upper limb prostheses by covariate shift adaptation, IEEE Trans. Neural Syst. Rehabil. Eng. 24 (9) (2015) 961-970, https://doi.org/10.970.

[11] A. Fougner, E. Scheme, A.D. Chan, K. Englehart, Ø. Stavdahl, Resolving the limb position effect in myoelectric pattern recognition, IEEE Trans. Neural Syst. Rehabil. Eng. 19 (6) (2011) 644-651, https://doi.org/10.651.

[12] X. Zhu, J. Liu, D. Zhang, X. Sheng, N. Jiang, Cascaded adaptation framework for fast calibration of myoelectric control, IEEE Trans. Neural Syst. Rehabil. Eng. 25 (3) (2016) 254-264.

[13] F. Zhuang, et al., A comprehensive survey on transfer learning, Proc. IEEE 109 (1) (2020) 43-76, https://doi.org/10.76.

[14] R. Soroushmojdehi, S. Javadzadeh, A. Pedrocchi, M. Gandolla, Transfer learning in hand movement intention detection based on surface electromyography signals, Front. Neurosci. 16 (2022) 977328, . https://doi.org/10.3389/fnins.2022.977328

[15] U. Cote-Allard, et al., Deep learning for electromyographic hand gesture signal classification using transfer learning, IEEE Trans. Neural Syst. Rehabil. Eng. 27 (4) (2019) 760-771, https://doi.org/10.771.

[16] K.-T. Kim, C. Guan, S.-W. Lee, A subject-transfer framework based on single-trial EMG analysis using convolutional neural networks, IEEE Trans. Neural Syst. Rehabil. Eng. 28 (1) (2019) 94-103, https://doi.org/10.103.

[17] I. Ketykó, F. Kovács, K.Z. Varga, Domain adaptation for sEMG-based gesture recognition with recurrent neural networks, 2019 International Joint Conference on Neural Networks, IJCNN, IEEE, (2019), https://doi.org/10.1109/IJCNN.2019.8852018.

[18] H. Le, M. in Het Panhuis, G.M. Spinks, G. Alici, Quantifying covariate shift and improving electromyography driven gesture recognition with calibration and sample selection, 2024 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, IEEE, 2024, pp. 1434-1440, https://doi.org/10.1109/AIM55361.2024.10636995.

[19] J. Liu, X. Sheng, D. Zhang, J. He, X. Zhu, Reduced daily recalibration of myoelectric prosthesis classifiers based on domain adaptation, IEEE J. Biomed. Health Inform. 20 (1) (2014) 166-176, https://doi.org/10.176.

[20] G. Yao, T. Lei, J. Zhong, A review of convolutional-neural-network-based action recognition, Pattern Recognit. Lett. 118 (2019) 14-22, https://doi.org/10.1016/j.patrec.2018.05.018.

[21] R. Poppe, A survey on vision-based human action recognition, Image Vis. Comput. 28 (6) (2010) 976-990.

[22] Z. Teng, G. Xu, R. Liang, M. Li, S. Zhang, Evaluation of synergy-based hand gesture recognition method against force variation for robust myoelectric control, IEEE Trans. Neural Syst. Rehabil. Eng. 29 (2021) 2345-2354, https://doi.org/10.1109/TNSRE.2021.3124744.

[23] F. Onay, A. Mert, Phasor represented EMG feature extraction against varying contraction level of prosthetic control, Biomed. Signal Process. Control. 59 (2020) 101881, . https://doi.org/10.1016/j.bspc.2020.101881

[24] A. Phinyomark, F. Quaine, S. Charbonnier, C. Serviere, F. Tarpin-Bernard, Y. Laurillau, EMG feature evaluation for improving myoelectric pattern recognition robustness, Expert Syst. Appl. 40 (12) (2013) 4832-4840, https://doi.org/10.4840.

[25] A.J. Fuglevand, D.A. Winter, A.E. Patla, Models of recruitment and rate coding organization in motor-unit pools, J. Neurophysiol. 70 (6) (1993) 2470-2488, https://doi.org/10.2488.

[26] C. Prahm, et al., Counteracting electrode shifts in upper-limb prosthesis control via transfer learning, IEEE Trans. Neural Syst. Rehabil. Eng. 27 (5) (2019) 956-962.

[27] F. Han, B. Reily, W. Hoff, H. Zhang, Space–time representation of people based on 3D skeletal data: a review, Comput. Vis. Image Underst. 158 (2017) 85-105, https://doi.org/10.1016/j.cviu.2017.01.011,05/01/2017.

[28] W. Wei, Q. Dai, Y. Wong, Y. Hu, M. Kankanhalli, W. Geng, Surface-electromyography-based gesture recognition by multi-view deep learning, IEEE Trans. Biomed. Eng. 66 (10) (2019) 2964-2973, https://doi.org/10.2973.

[29] A.T. Belyea, K.B. Englehart, E.J. Scheme, A proportional control scheme for high density force myography, J. Neural Eng. 15 (2018) 046029.

[30] N. Jiang, I. Vujaklija, H. Rehbaum, B. Graimann, D. Farina, Is accurate mapping of EMG signals on kinematics needed for precise online myoelectric control?, IEEE Trans. Neural Syst. Rehabil. Eng. 22 (3) (2013) 549-558, https://doi.org/10.558.

[31] E. Scheme, K. Englehart, Training strategies for mitigating the effect of proportional control on classification in pattern recognition–based myoelectric control, JPO: J. Prosthetics Orthot. 25 (2) (2013) 76-83.

[32] M. Atzori, M. Cognolato, H. Müller, Deep learning with convolutional neural networks applied to electromyography data: A resource for the classification of movements for prosthetic hands, Front. Neurorobotics 10 (2016) 9, https://doi.org/10.3389/fnbot.2016.00009.

[33] W. Wei, Y. Wong, Y. Du, Y. Hu, M. Kankanhalli, W. Geng, A multi-stream convolutional neural network for sEMG-based gesture recognition in muscle-computer interface, Pattern Recognit. Lett. 119 (2019) 131-138, https://doi.org/10.1016/j.patrec.2017.12.005.

[34] A. Ameri, M.A. Akhaee, E. Scheme, K. Englehart, A deep transfer learning approach to reducing the effect of electrode shift in EMG pattern recognition-based control, IEEE Trans. Neural Syst. Rehabil. Eng. (2019), https://doi.org/10.1109/TNSRE.2019.2962189.

[35] A. Ameri, M.A. Akhaee, E. Scheme, K. Englehart, Regression convolutional neural network for improved simultaneous EMG control, J. Neural Eng. 16 (2019) 036015.

[36] H. Chen, Y. Zhang, G. Li, Y. Fang, H. Liu, Surface electromyography feature extraction via convolutional neural network, Int. J. Mach. Learn. Cybern. 2019, pp. 1-12.

[37] Z. Wang, Representation Learning on Time Series with Symbolic Approximation and Deep Learning, University of Maryland, Baltimore County, (2016).

[38] Y. Zou, L. Cheng, Z. Li, A multimodal fusion model for estimating human hand force: Comparing surface electromyography and ultrasound signals, IEEE Robot. Autom. Mag. 29 (4) (2022) 10-24, https://doi.org/10.24.

[39] H. Le, M. in Het Panhuis, G.M. Spinks, G. Alici, Hybrid LDA-CNN framework for robust end-to-end myoelectric hand gesture recognition under dynamic conditions, Robotics, (2025).

[40] H. Le, G.M. Spinks, M. in Het Panhuis, G. Alici, Cross-day myoelectric gesture recognition with hybrid multistream CNN-bidirectional LSTM, in: 2025 IEEE International Conference on Mechatronics, ICM’25, Wollongong, Australia, 2025, http://dx.doi.org/10.1109/ICM62621.2025.10934890.

[41] T. Hastie, R. Tibshirani, J.H. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer, (2009).

[42] Y. Wu, B. Zheng, Y. Zhao, Dynamic gesture recognition based on LSTM-CNN, 2018 Chinese Automation Congress, CAC, IEEE, 2018, pp. 2446-2450, https://doi.org/10.1109/CAC.2018.8623035.

[43] E. Rahimian, S. Zabihi, A. Asif, D. Farina, S.F. Atashzar, A. Mohammadi, Hand gesture recognition using temporal convolutions and attention mechanism, ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, IEEE, 2022, pp. 1196-1200, https://doi.org/10.1109/ICASSP43922.2022.9746174.

[44] S. Zabihi, E. Rahimian, A. Asif, A. Mohammadi, Trahgr: transformer for hand gesture recognition via electromyography, IEEE Trans. Neural Syst. Rehabil. Eng. (2023), https://doi.org/10.1109/TNSRE.2023.3324252.

[45] T.R. Farrell, Determining delay created by multifunctional prosthesis controllers, J. Rehabil. Res. Dev. 48 (6) (2011) xxi.

[46] T. Farrell, R.F. Weir, Analysis window induced controller delay for multifunctional prostheses, in: Myoelectric Symposium, 2008.

[47] T.R. Farrell, R.F. Weir, The optimal controller delay for myoelectric prostheses, IEEE Trans. Neural Syst. Rehabil. Eng. 15 (1) (2007) 111-118.

[48] H. Le, M. in Het Panhuis, G. Alici, Literature survey on machine learning techniques for enhancing accuracy of myoelectric hand gesture recognition in real-world prosthetic hand control, Biomim. Intell. Robot. (2025), Article 100250, https://doi.org/10.1016/j.birob.2025.100250,2025/07/11/.

[49] J.L. Betthauser, et al., Stable responsive EMG sequence prediction and adaptive reinforcement with temporal convolutional networks, IEEE Trans. Biomed. Eng. 67 (6) (2019) 1707-1717, https://doi.org/10.1717.

[50] R. Menon, G. Di Caterina, H. Lakany, L. Petropoulakis, B.A. Conway, J.J. Soraghan, Study on interaction between temporal and spatial information in classification of EMG signals for myoelectric prostheses, IEEE Trans. Neural Syst. Rehabil. Eng. 25 (10) (2017) 1832-1842, https://doi.org/10.1842.

[51] H. Le, M. in Het Panhuis, G.M. Spinks, G. Alici, The effect of dataset size on EMG gesture recognition under diverse limb positions, 2024 10th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics, BioRob, IEEE, 2024, pp. 303-308, https://doi.org/10.1109/BioRob60516.2024.10719858.

[52] T. Garikayi, D. Van den Heever, S. Matope, Analysis of surface electromyography signal features on osteomyoplastic transtibial amputees for pattern recognition control architectures, Biomed. Signal Process. Control. 40 (2018) 10-22.

[53] G. Rognini, et al., Multisensory bionic limb to achieve prosthesis embodiment and reduce distorted phantom limb perceptions, J. Neurol. Neurosurg. Psychiatry 90 (7) (2019) 833-836.

[54] J. Ye, Least squares linear discriminant analysis, in: Proceedings of the 24th International Conference on Machine Learning, 2007, pp. 1087-1093.

[55] A. Radmand, E. Scheme, K. Englehart, A characterization of the effect of limb position on EMG features to guide the development of effective prosthetic control schemes, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE, 2014, pp. 662-667, https://doi.org/10.1109/EMBC.2014.6943678.

[56] J. Liu, D. Zhang, X. Sheng, X. Zhu, Quantification and solutions of arm movements effect on sEMG pattern recognition, Biomed. Signal Process. Control. 13 (2014) 189-197, https://doi.org/10.1016/j.bspc.2014.05.001.

[57] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, C.C. Loy, Domain generalization: a survey, IEEE Trans. Pattern Anal. Mach. Intell. (2022), https://doi.org/10.1109/TPAMI.2022.3195549.

[58] H. Le, G.M. Spinks, M. in Het Panhuis, G. Alici, Feature significance and generalizability of myoelectric hand gesture recognition under varying limb positions, in: 2025 IEEE International Conference on Mechatronics, ICM’25, Wollongong, Australia, 2025.

[59] S. Zhang, H. Zhou, R. Tchantchane, G. Alici, Hand gesture recognition across various limb positions using a multi-modal sensing system based on self-adaptive data-fusion and convolutional neural networks (CNNs), IEEE Sensors J. (2024), https://doi.org/10.1109/JSEN.2024.3389963.

[60] M.R. Islam, D. Massicotte, P. Massicotte, W.P. Zhu, Surface EMG-based intersession/intersubject gesture recognition by leveraging lightweight all-ConvNet and transfer learning, IEEE Trans. Instrum. Meas. 73 (2024) 1-16, https://doi.org/10.1109/TIM.2024.3381288.