High-accuracy dynamic gesture recognition: A universal and self-adaptive deep-learning-assisted system leveraging high-performance ionogels-based strain sensors

Abstract

Gesture recognition utilizing flexible strain sensors is a highly valuable technology widely applied in human–machine interfaces. However, achieving rapid detection of subtle motions and timely processing of dynamic signals remain a challenge for sensors. Here, highly resilient and durable ionogels are developed by introducing micro-scale incompatible phases in macroscopic homogeneous polymeric network. The compatible network disperses in conductive ionic liquid to form highly resilient and stretchable skeleton, while incompatible phase forms hydrogen bonds to dissipate energy thus strengthening the ionogels. The ionogels-derived strain sensors show highly sensitivity, fast response time (<10 ms), low detection limit (∼50 µm), and remarkable durability (>5000 cycles), allowing for precise monitoring of human motions. More importantly, a self-adaptive recognition program empowered by deep-learning algorithms is designed to compensate for sensors, creating a comprehensive system capable of dynamic gesture recognition. This system can comprehensively analyze both the temporal and spatial features of sensor data, enabling deeper understanding of the dynamic process underlying gestures. The system accurately classifies 10 hand gestures across five participants with impressive accuracy of 93.66%. Moreover, it maintains robust recognition performance without the need for further training even when different sensors or subjects are involved. This technological breakthrough paves the way for intuitive and seamless interaction between humans and machines, presenting significant opportunities in diverse applications, such as human–robot interaction, virtual reality control, and assistive devices for the disabled individuals.

Keywords

deep-learning algorithms / dynamic gesture recognition / human–machine interaction / ionogels / self-adaptive recognition program / strain sensors

References

[1] Tan P, Han X, Zou Y, et al. Self-powered gesture recognition wristband enabled by machine learning for full keyboard and multicommand input. Adv Mater. 2022; 34(21): 2200793.

[2] Huang J, Wang H, Li J, et al. High-performance flexible capacitive proximity and pressure sensors with spiral electrodes for continuous human-machine interaction. ACS Mater Lett. 2022; 4(11): 2261-2272.

[3] Ke QH, Liu J, Bennamoun M, An SJ, Sohel F, Boussaid F. Chapter 5-computer vision for human-machine interaction. In: Leo M, Farinella GM, eds. Computer Vision for Assistive Healthcare: Computer Vision and Pattern Recognition. Academic Press; 2018: 127-145.

[4] Wong WK, Juwono FH, Khoo BTT. Multi-features capacitive hand gesture recognition sensor: a machine learning approach. IEEE Sens J. 2021; 21(6): 8441-8450.

[5] Li L, Jiang S, Shull PB, Gu G. SkinGest: artificial skin for gesture recognition via filmy stretchable strain sensors. Adv Robot. 2018; 32(21): 1112-1121.

[6] Tchantchane R, Zhou H, Zhang S, Alici G. A review of hand gesture recognition systems based on noninvasive wearable sensors. Adv Intell Syst. 2023; 5(10): 2300207.

[7] Oudah M, Al-Naji A. Chahl J. Hand gesture recognition based on computer vision: a review of techniques. J Imaging. 2020; 6(8): 73.

[8] Yu J, Qin M, Zhou S. Dynamic gesture recognition based on 2D convolutional neural network and feature fusion. Sci Rep. 2022; 12: 4345.

[9] Mantecón T, del-Blanco CR. Jaureguizar F, García N. A real-time gesture recognition system using near-infrared imagery. PLoS ONE. 2019; 14(10): e0223320.

[10] Yu L, Abuella H, Islam MZ, O’Hara JF, Crick C, Ekin S. Gesture recognition using reflected visible and infrared lightwave signals. IEEE Trans Hum Mach Syst. 2021; 51(1): 44-55.

[11] Smith KA, Csech C, Murdoch D, Shaker G. Gesture recognition using mm-wave sensor for human-car interface. IEEE Sensors Lett. 2018; 2(2): 1-4.

[12] Ahmed S, Kallu KD, Ahmed S, Cho SH. Hand gestures recognition using radar sensors for human-computer-interaction: a review. Remote Sens. 2021; 13(3): 527.

[13] Liu Y, Fan Y, Wu Z, Yao J, Long Z. Ultrasound-based 3-D gesture recognition: signal optimization, trajectory, and feature classification. IEEE Trans Instrum Meas. 2023; 72: 1-12.

[14] Mo F, Huang Y, Li Q, et al. A highly stable and durable capacitive strain sensor based on dynamically super-tough hydro/organo-gels. Adv Funct Mater. 2021; 31(28): 2010830.

[15] Wu X, Luo X, Song Z, Bai Y, Zhang B, Zhang G. Ultra-robust and sensitive flexible strain sensor for real-time and wearable sign language translation. Adv Funct Mater. 2023; 33(36): 2303504.

[16] Wang M, Yan Z, Wang T, et al. Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nature Electronics. 2020; 3: 563-570.

[17] Si Y, Chen S, Li M, Li S, Pei Y, Guo X. Flexible strain sensors for wearable hand gesture recognition: from devices to systems. Adv Intell Syst. 2022; 4(2): 2100046.

[18] Niu B, Yang S, Yang Y, Hua T. Highly conductive fiber with design of dual conductive Ag/CB layers for ultrasensitive and wide-range strain sensing. SmartMat. 2023; 4(6): e1178.

[19] Chen Y, Chen L, Geng B, et al. Triple-network-based conductive polymer hydrogel for soft and elastic bioelectronic interfaces. SmartMat. 2024; 5(3): e1229.

[20] Wang L, Wang H, Wan, QC, Gao JF. Recent development of conductive polymer composite-based strain sensors. J Polym Sci. 2023; 61(24): 3167-3185.

[21] Zhang S, Sun X, Guo X, et al. A wide-range-response piezoresistive-capacitive dual-sensing breathable sensor with spherical-shell network of MWCNTs for motion detection and language assistance. Nanomaterials. 2023; 13(5): 843.

[22] Yu X, Wu Z, Weng L, et al. Flexible strain sensor enabled by carbon nanotubes-decorated electrospun TPU membrane for human motion monitoring. Adv Mater Interfaces. 2023; 10(11): 2202292.

[23] Zhou K, Dai K, Liu C, Shen C. Flexible conductive polymer composites for smart wearable strain sensors. SmartMat. 2020; 1(1): e1010.

[24] Liu J, Chen X, Sun B, et al. Stretchable strain sensor of composite hydrogels with high fatigue resistance and low hysteresis. J Mater Chem A. 2022; 10(48): 25564-25574.

[25] Yuk H, Wu J, Zhao X. Hydrogel interfaces for merging humans and machines. Nat Rev Mater. 2022; 7: 935-952.

[26] Ge G, Lu Y, Qu X, et al. Muscle-inspired self-healing hydrogels for strain and temperature sensor. ACS Nano. 2020; 14(1): 218-228.

[27] Ma L, Wang J, He J, et al. Ultra-sensitive, durable and stretchable ionic skins with biomimetic micronanostructures for multisignal detection, high-precision motio. monitoring, and underwater sensing. J Mater Chem A. 2021; 9(47): 26949-26962.

[28] Wang Y, Sun S, Wu P. Adaptive ionogel paint from room-temperature autonomous polymerization of α-thioctic acid for stretchable and healable electronics. Adv Funct Mater. 2021; 31(24): 2101494.

[29] Li T, Wang Y, Li S, Liu X, Sun J. Mechanically robust, elastic, and healable ionogels for highly sensitive ultra-durable ionic skins. Adv Mater. 2020; 32(32): 2002706.

[30] Zhang Z, Yu Y, Yu H, Feng Y, Feng W. Water-resistant conductive organogels with sensation and actuation functions for artificial neuro-sensory muscular systems. SmartMat. 2022; 3(4): 632-643.

[31] Xu L, Huang Z, Deng Z, et al. A transparent, highly stretchable, solvent-resistant, recyclable multifunctional ionogel with underwater self-healing and adhesion for reliable strain sensors. Adv Mater. 2021; 33(51): 2105306.

[32] Kim YM, Moon HC. Ionoskins: nonvolatile, highly transparent, ultrastretchable ionic sensory platforms for wearable electronics. Adv Funct Mater. 2020; 30(4): 1907290.

[33] Qiu W, Chen G, Zhu H, Zhang Q, Zhu S. Enhanced stretchability and robustness towards flexible ionotronics via double-network structure and ion-dipole interactions. Chem Eng J. 2022; 434: 134752.

[34] Lu Y, Qu X, Wang S, et al. Ultradurable, freeze-resistant, and healable MXene-based ionic gels for multi-functional electronic skin. Nano Res. 2022; 15: 4421-4430.

[35] Hao S, Li T, Yang X, Song H. Ultrastretchable, adhesive, fast self-healable, and three-dimensiona. printable photoluminescent ionic skin based on hybrid network ionogels. ACS Appl Mater Interfaces. 2022; 14(1): 2029-2037.

[36] Kim JH, Cho KG, Cho DH, Hong K, Lee KH. Ultra-sensitive and stretchable ionic skins for high-precision motion monitoring. Adv Funct Mater. 2021; 31(16): 2010199.

[37] Zhang Y, Wang Y, Guan Y, Zhang Y. Peptide-enhanced tough, resilient and adhesive eutectogels for highly reliable strain/pressure sensing under extreme conditions. Nat Commun. 2022; 13: 6671.

[38] Yu Q, Qin Z, Ji F, et al. Low-temperature tolerant strain sensors based on triple crosslinked organohydrogels with ultrastretchability. Chem Eng J. 2021; 404: 126559.

[39] Zhao G, Lv B, Wang H, et al. Ionogel-based flexible stress and strain sensors. Int J Smart Nano Mater. 2021; 12(3): 307-336.

[40] Fan X, Liu S, Jia Z, et al. Ionogels: recent advances in design, material properties and emerging biomedical applications. Chem Soc Rev. 2023; 52(7): 2497-2527.

[41] Sun J, Yuan Y, Lu G, Li L, Zhu X, Nie J. A transparent, stretchable, stable, self-adhesive ionogel-based strain sensor for human motion monitoring. J Mater Chem C. 2019; 7(36): 11244-11250.

[42] Zhang J, Liu E, Hao S, et al. 3D printable, ultra-stretchable, self-healable, and self-adhesiv. dual cross-linked nanocomposite ionogels as ultra-durable strain sensors for motion detection and wearable human-machine interface. Chem Eng J. 2022; 431: 133949.

[43] Chen J, Wang F, Zhu G, et al. Breathable strain/temperature sensor based on fibrous networks of ionogels capable of monitoring human motion, respiration, and proximity. ACS Appl Mater Interfaces. 2021; 13(43): 51567-51577.

[44] Yuan Y, Zhou J, Lu G, Sun J, Tang L. Highly stretchable, transparent, and self-adhesiv. ionic conductor for high-performance flexible sensors. ACS Appl Polym Mater. 2021; 3(3): 1610-1617.

[45] Ma H, Qin H, Xiao X, et al. Robust hydrogel sensors for unsupervised learning enabled sign-to-verbal translation. InfoMat. 2023; 5(7): e12419.

[46] Wang M, Zhang P, Shamsi M, et al. Tough and stretchable ionogels by in situ phase separation. Nat Mater. 2022; 21: 359-365.

[47] Zhang M, Yu R, Tao X, et al. Mechanically robust and highly conductive ionogels for soft ionotronics. Adv Funct Mater. 2023; 33(10): 2208083.

[48] Sun L, Huang H, Ding Q, et al. Highly transparent, stretchable, and self healable ionogel for multifunctional sensors, triboelectric nanogenerator, and wearable fibrous electronics. Adv Fiber Mater. 2022; 4: 98-107.

[49] Saleha TA, Elsharif AM, Asiri S, Mohammed A, Dafalla H. Synthesis of carbon nanotubes grafted with copolymer of acrylic acid acrylamide for phenol removal. Environ Nanotechnol Monit Manag. 2020; 14: 100302.

[50] Meng X, Qiao Y, Do C, et al. Hysteresis-free nanoparticle-reinforced hydrogels. Adv Mater. 2022; 34(7): 2108243.

RIGHTS & PERMISSIONS

2024 The Authors. SmartMat published by Tianjin University and John Wiley & Sons Australia, Ltd.