Highly Sensitive and Mechanically Stable MXene Textile Sensors for Adaptive Smart Data Glove Embedded with Near-Sensor Edge Intelligence

Abstract

Smart data gloves capable of monitoring finger activities and inferring hand gestures are of significance to human–machine interfaces, robotics, healthcare, and Metaverse. Yet, most current smart data gloves present unstable mechanical contacts, limited sensitivity, as well as offline training and updating of machine learning models, leading to uncomfortable wear and suboptimal performance during practical applications. Herein, highly sensitive and mechanically stable textile sensors are developed through the construction of loose MXene-modified textile interface structures and a thermal transfer printing method with the melting-infiltration-solidification adhesion procedure. Then, a smart data glove with adaptive gesture recognition is reported, based on the integration of 10-channel MXene textile bending sensors and a near-sensor adaptive machine learning model. The near-sensor adaptive machine learning model achieves a 99.5% accuracy using the proposed post-processing algorithm for 14 gestures. Also, the model features the ability to locally update model parameters when gesture types change, without additional computation on any external device. A high accuracy of 98.1% is still preserved when further expanding the dataset to 20 gestures, where the accuracy is recovered by 27.6% after implementing the model updates locally. Lastly, an auto-recognition and control system for wireless robotic sorting operations with locally trained hand gestures is demonstrated, showing the great potential of the smart data glove in robotics and human–machine interactions.

Keywords

Smart data glove / Gesture recognition / Machine learning / Wearable sensors / Robotics / Textile sensors

References

[1] Araromi OA, Graule MA, Dorsey KL, Castellanos S, Foster JR, Hsu WH, Passy AE, Vlassak JJ, Weaver JC, Walsh CJ, Wood RJ. Ultra-sensitive and resilient compliant strain gauges for soft machines. Nature, 2020, 587: 219,

[2] Zeissler K. Gesture recognition gets an update. Nat Electron, 2023, 6: 272,

[3] Gu G, Zhang N, Xu H, Lin S, Yu Y, Chai G, Ge L, Yang H, Shao Q, Sheng X, Zhu X, Zhao X. A soft neuroprosthetic hand providing simultaneous myoelectric control and tactile feedback. Nat Biomed Eng, 2023, 7: 589,

[4] Duan S, Shi Q, Hong J, Zhu D, Lin Y, Li Y, Lei W, Lee C, Wu J. Water-modulated biomimetic hyper-attribute-gel electronic skin for robotics and skin-attachable wearables. ACS Nano, 2023, 17: 1355,

[5] Sun Z, Zhu M, Shan X, Lee C. Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions. Nat Commun, 2022, 13: 5224,

[6] Giese MA, Poggio T. Neural mechanisms for the recognition of biological movements. Nat Rev Neurosci, 2003, 4: 179,

[7] Zhou Z, Chen K, Li X, Zhang S, Wu Y, Zhou Y, Meng K, Sun C, He Q, Fan W, Fan E, Lin Z, Tan X, Deng W, Yang J, Chen J. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays. Nat Electron, 2020, 3: 571,

[8] Shi Q, Zhang Z, He T, Sun Z, Wang B, Feng Y, Shan X, Salam B, Lee C. Deep learning enabled smart mats as a scalable floor monitoring system. Nat Commun, 2020, 11: 4609,

[9] Zhang W, Wang J. Dynamic hand gesture recognition based on 3D convolutional neural network models. 2019 IEEE 16th International Conference on Networking, Sensing and Control (ICNSC); 2019;9.

[10] Wang M, Yan Z, Wang T, Cai P, Gao S, Zeng Y, Wan C, Wang H, Pan L, Yu J, Pan S, He K, Lu J, Chen X. Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nat Electron, 2020, 3: 563,

[11] Duan S, Zhao F, Yang H, Hong J, Shi Q, Lei W, Wu J. A pathway into metaverse: gesture recognition enabled by wearable resistive sensors. Adv Sensor Res, 2023, 2: 2200054,

[12] Xiong Y, Han J, Wang Y, Wang ZL, Sun Q. Emerging iontronic sensing: materials, mechanisms, and applications. Research, 2022, 2022: 9867378,

[13] Guo ZH, Zhang Z, An K, He T, Sun Z, Pu X, Lee C. A Wearable Multidimensional Motion Sensor for AI-Enhanced VR Sports. Research, 2023, 6: 0154,

[14] Meng K, Xiao X, Liu Z, Shen S, Tat T, Wang Z, Lu C, Ding W, He X, Yang J, Chen J. Kirigami-inspired pressure sensors for wearable dynamic cardiovascular monitoring. Adv Mater, 2022, 34: 2202478,

[15] Tchantchane R, Zhou H, Zhang S, Alici G. A review of hand gesture recognition systems based on noninvasive wearable sensors. Adv Intell Syst, 2023, 5: 2300207,

[16] Si Y, Chen S, Li M, Li S, Pei Y, Guo X. Flexible strain sensors for wearable hand gesture recognition: from devices to systems. Adv Intell Syst, 2022, 4: 2100046,

[17] Wen F, Zhang Z, He T, Lee C. AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nat Commun, 2021, 12: 5378,

[18] Zhu C, Wu J, Yan J, Liu X. Advanced fiber materials for wearable electronics. Adv Fiber Mater, 2023, 5: 12,

[19] Wang W, Yu A, Zhai J, Wang ZL. Recent progress of functional fiber and textile triboelectric nanogenerators: towards electricity power generation and intelligent sensing. Adv Fiber Mater, 2021, 3: 394,

[20] Shi Q, Sun J, Hou C, Li Y, Zhang Q, Wang H. Advanced functional fiber and smart textile. Adv Fiber Mater, 2019, 1: 3,

[21] Liu X, Miao J, Fan Q, Zhang W, Zuo X, Tian M, Zhu S, Zhang X, Qu L. Recent progress on smart fiber and textile based wearable strain sensors: materials, fabrications and applications. Adv Fiber Mater, 2022, 4: 361,

[22] Zhu M, Sun Z, Zhang Z, Shi Q, He T, Liu H, Chen T, Lee C. Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications. Sci Adv, 2020, 6: eaaz8693,

[23] Wen F, Sun Z, He T, Shi Q, Zhu M, Zhang Z, Li L, Zhang T, Lee C. Machine learning glove using self-powered conductive superhydrophobic triboelectric textile for gesture recognition in VR/AR applications. Adv Sci, 2020, 7: 2000261,

[24] Kim YG, Song JH, Hong S, Ahn SH. Piezoelectric strain sensor with high sensitivity and high stretchability based on kirigami design cutting. npj Flex Electron., 2022, 6: 52,

[25] Bai H, Li S, Barreiros J, Tu Y, Pollock CR, Shepherd RF. Stretchable distributed fiber-optic sensors. Science, 2020, 370: 848,

[26] Wong WK, Juwono FH, Khoo BTT. Multi-features capacitive hand gesture recognition sensor: a machine learning approach. IEEE Sens J, 2021, 21: 8441,

[27] Wang T, Zhao Y, Wang Q. A flexible iontronic capacitive sensing array for hand gesture recognition using deep convolutional neural networks. Soft Robot, 2022, 10: 443,

[28] Zhu D, Duan S, Liu J, Diao S, Hong J, Xiang S, Wei X, Xiao P, Xia J, Lei W, Wang B, Shi Q, Wu J. A double-crack structure for bionic wearable strain sensors with ultra-high sensitivity and a wide sensing range. Nanoscale, 2024, 16: 5409,

[29] Ota S, Ando A, Chiba D. A flexible giant magnetoresistive device for sensing strain direction. Nat Electron, 2018, 1: 124,

[30] Guo X, Hong W, Zhao Y, Zhu T, Liu L, Li H, Wang Z, Wang D, Mai Z, Zhang T, Yang J, Zhang F, Xia Y, Hong Q, Xu Y, Yan F, Wang M, Xing G. Bioinspired dual-mode stretchable strain sensor based on magnetic nanocomposites for strain/magnetic discrimination. Small, 2023, 19: 2205316,

[31] Duan S, Wei X, Zhao F, Yang H, Wang Y, Chen P, Hong J, Xiang S, Luo M, Shi Q, Shen G, Wu J. Bioinspired Young's modulus-hierarchical E-skin with decoupling multimodality and neuromorphic encoding outputs to biosystems. Adv Sci, 2023, 10: 2304121,

[32] Wei X, Li H, Yue W, Gao S, Chen Z, Li Y, Shen G. A high-accuracy, real-time, intelligent material perception system with a machine-learning-motivated pressure-sensitive electronic skin. Matter, 2022, 5: 1481,

[33] Bai N, Xue Y, Chen S, Shi L, Shi J, Zhang Y, Hou X, Cheng Y, Huang K, Wang W, Zhang J, Liu Y, Guo CF. A robotic sensory system with high spatiotemporal resolution for texture recognition. Nat Commun, 2023, 14: 7121,

[34] Gu J, Li F, Zhu Y, Li D, Liu X, Wu B, Wu HA, Fan X, Ji X, Chen Y, Liang J. Extremely robust and multifunctional nanocomposite fibers for strain-unperturbed textile electronics. Adv Mater, 2023, 35: 2209527,

[35] Zhou X, Wang Z, Xiong T, He B, Wang Z, Zhang H, Hu D, Liu Y, Yang C, Li Q, Chen M, Zhang Q, Wei L. Fiber crossbars: an emerging architecture of smart electronic textiles. Adv Mater, 2023, 35: 2370364,

[36] Duan S, Wang J, Lin Y, Hong J, Lin Y, Xia Y, Li Y, Zhu D, Lei W, Su W, Wang B, Cui Z, Yuan W, Wu J. Highly durable machine-learned waterproof electronic glove based on low-cost thermal transfer printing for amphibious wearable applications. Nano Res, 2023, 16: 5480,

[37] Kim KK, Kim M, Pyun K, Kim J, Min J, Koh S, Root SE, Kim J, Nguyen B-NT, Nishio Y, Han S, Choi J, Kim CY, Tok JBH, Jo S, Ko SH, Bao Z. A substrate-less nanomesh receptor with meta-learning for rapid hand task recognition. Nat Electron., 2023, 6: 64

[38] Liu X, Sacks J, Zhang M, Richardson AG, Lucas TH, Spiegel JVd. The virtual trackpad: an electromyography-based, wireless, real-time, low-power, embedded hand-gesture-recognition system using an event-driven artificial neural network. IEEE Trans Circuits Syst II Express Briefs, 2017, 64: 1257

[39] Moin A, Zhou A, Rahimi A, Menon A, Benatti S, Alexandrov G, Tamakloe S, Ting J, Yamamoto N, Khan Y, Burghardt F, Benini L, Arias AC, Rabaey JM. A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat Electron, 2021, 4: 54,

[40] Tkach D, Huang H, Kuiken TA. Study of stability of time-domain features for electromyographic pattern recognition. J Neuroeng Rehabil, 2010, 7: 21,

[41] Young AJ, Hargrove LJ, Kuiken TA. The effects of electrode size and orientation on the sensitivity of myoelectric pattern recognition systems to electrode shift. IEEE Trans Biomed Eng, 2011, 58: 2537,

[42] Xiang S, Nie F, Zhang C. Learning a Mahalanobis distance metric for data clustering and classification. Pattern Recognit, 2008, 41: 3600,

[43] Kuo RJ, Rakhmat Setiawan M, Nguyen TPQ. Sequential clustering and classification using deep learning technique and multi-objective sine-cosine algorithm. Comput Ind Eng, 2022, 173,

[44] Duan S, Lin Y, Wang Z, Tang J, Li Y, Zhu D, Wu J, Tao L, Choi C-H, Sun L, Xia J, Wei L, Wang B. Conductive porous MXene for bionic, wearable, and precise gesture motion sensors. Research, 2021, 2021: 9861467,

[45] Deng W, Yang T, Jin L, Yan C, Huang H, Chu X, Wang Z, Xiong D, Tian G, Gao Y, Zhang H, Yang W. Cowpea-structured PVDF/ZnO nanofibers based flexible self-powered piezoelectric bending motion sensor towards remote control of gestures. Nano Energy, 2019, 55: 516,

[46] Nakamoto H, Ootaka H, Tada M, Hirata I, Kobayashi F, Kojima F. Stretchable strain sensor with anisotropy and application for joint angle measurement. IEEE Sens J, 2016, 16: 3572,

[47] Wang H, Tong Y, Zhao X, Tang Q, Liu Y. Flexible, high-sensitive, and wearable strain sensor based on organic crystal for human motion detection. Org Electron, 2018, 61: 304,

[48] Dong H, Sun J, Liu X, Jiang X, Lu S. Highly sensitive and stretchable MXene/CNTs/TPU composite strain sensor with bilayer conductive structure for human motion detection. ACS Appl Mater Interfaces, 2022, 14: 15504,

[49] Fan J, Yuan M, Wang L, Xia Q, Zheng H, Zhou A. MXene supported by cotton fabric as electrode layer of triboelectric nanogenerators for flexible sensors. Nano Energy, 2023, 105,

[50] Zhou H, Huang W, Xiao Z, Zhang S, Li W, Hu J, Feng T, Wu J, Zhu P, Mao Y. Deep-learning-assisted noncontact gesture-recognition system for touchless human-machine interfaces. Adv Funct Mater, 2022, 32: 2208271,

Funding

National Key R&D Program of China(2022YFB3603403); the National Natural Science Foundation of China(62075040); the Start-up Research Fund of Southeast University(RF1028623164); the National Natural Science Foundation of China(623B2021)