The Touch-Code Glove: a multimodal mapping interface with triboelectric-digital encoding for intuitive robot training

Abstract

Current human-robot interaction (HRI) systems for training embodied intelligent robots often suffer from limited motion dimensionality and unintuitive control. This work presents the Touch-Code Glove, a multimodal HRI interface integrating functional materials, structural intelligence, and deep-learning decoding. A triboelectric digital interface is embedded into the Wrist-pad via a mosaic-patterned array of polyamide/polytetrafluoroethylene-doped silicone rubber films, generating polarity-dependent digital signal pairs upon contact. A co-electrode layout enables 16 touch points with minimal wiring, allowing multiplexed, programmable tactile input via sliding or multi-point gestures. Coupled triboelectric signals are accurately decoded using a convolutional neural network and long short-term memory model, achieving over 98% recognition accuracy. Complementarily, a double-network conductive hydrogel composed of sodium alginate, polyacrylamide, and sodium chloride is integrated into the Finger-fibers and the Wrist-pad to provide strain-sensing capabilities with excellent stretchability, high linearity, low hysteresis, and long-term stability. The system incorporates three concurrent sub-mapping strategies: gesture-driven control, wrist posture-based movement, and touch path-guided input, which together enable real-time control of robotic hands and arms without requiring professional training. This triboelectric-hydrogel hybrid interface offers a materials-centric solution for intelligent, wearable, and accessible HRI, paving the way for next-generation multimodal robotic control systems in assistive and industrial applications.

Keywords

Human-robot interface / multimodal sensing / triboelectric encoding / motion mapping / flexible electronics / embodied intelligent robots

References

[1] Bartolozzi C,Donati E.Author Correction: Embodied neuromorphic intelligence.Nat Commun2022;13:1415 PMCID:PMC8917186

[2] Zhao Z,Wang J,Zhong C.Exploring embodied intelligence in soft robotics: a review.Biomimetics2024;9:248 PMCID:PMC11047907

[3] Mon-Williams R,Long R,Lucas CG.Embodied large language models enable robots to complete complex tasks in unpredictable environments.Nat Mach Intell2025;7:592-601 PMCID:PMC12088599

[4] Khan AT,Cao X.Human guided cooperative robotic agents in smart home using beetle antennae search.Sci China Inf Sci2022;65:122204

[5] Eirale A,Tagliavini L,Chiaberge M.Marvin: an innovative omni-directional robotic assistant for domestic environments.Sensors2022;22:5261 PMCID:PMC9322347

[6] Gentile C,Buonocore LR.Manipulation tasks in hazardous environments using a teleoperated robot: a case study at CERN.Sensors2023;23:1979 PMCID:PMC9963582

[7] Szczurek KA,Matheson E,Di Castro M.Multimodal multi-user mixed reality human–robot interface for remote operations in hazardous environments.IEEE Access2023;11:17305-33

[8] Soori M,Arezoo B.Intelligent robotic systems in Industry 4.0: a review.J Adv Manuf Sci Technol2024;4:2024007

[9] Arents J.Smart industrial robot control trends, challenges and opportunities within manufacturing.Appl Sci2022;12:937

[10] Hou C,Yang X.A piezoresistive-based 3-axial MEMS tactile sensor and integrated surgical forceps for gastrointestinal endoscopic minimally invasive surgery.Microsyst Nanoeng2024;10:141 PMCID:PMC11427553

[11] Hou C,Wang F.A highly integrated 3D MEMS force sensing module with variable sensitivity for robotic-assisted minimally invasive surgery.Adv Funct Mater2023;33:2302812

[12] Mengaldo G,Brunton SL.A concise guide to modelling the physics of embodied intelligence in soft robotics.Nat Rev Phys2022;4:595-610

[13] Liu W,Liu J.Touchless interactive teaching of soft robots through flexible bimodal sensory interfaces.Nat Commun2022;13:5030 PMCID:PMC9412806

[14] Fan H,Fuh JYH,Li B.Embodied intelligence in manufacturing: leveraging large language models for autonomous industrial robotics.J Intell Manuf2025;36:1141-57

[15] Sun Z,Lee C.Progress in the triboelectric human–machine interfaces (HMIs)-moving from smart gloves to AI/haptic enabled HMI in the 5G/IoT era.Nanoenergy Adv2021;1:81-120

[16] Wang T,Li S.Multimodal human–robot interaction for human-centric smart manufacturing: a survey.Adv Intell Syst2024;6:2300359

[17] Chen S,Cao Y,Cao C.Soft robotic manipulation system capable of stiffness variation and dexterous operation for safe human–machine interactions.Adv Mater Technol2021;6:2100084

[18] Sun T,Liu Z.Machine learning-coupled vertical graphene triboelectric pressure sensors array as artificial tactile receptor for finger action recognition.Nano Energy2024;123:109395

[19] Zhu M,Chen T.Low cost exoskeleton manipulator using bidirectional triboelectric sensors enhanced multiple degree of freedom sensory system.Nat Commun2021;12:2692 PMCID:PMC8113469

[20] Fang P,Zeng Z.A multi-module sensing and Bi-directional HMI integrating interaction, recognition, and feedback for intelligent robots.Adv Funct Mater2024;34:2310254

[21] Hou C,Yang Z.A delta-parallel-inspired human machine interface by using self-powered triboelectric nanogenerator toward 3D and VR/AR manipulations.Adv Mater Technol2021;6:2000912

[22] He T,Shi Q.Self-powered glove-based intuitive interface for diversified control applications in real/cyber space.Nano Energy2019;58:641-51

[23] Hang CZ,Xi SY.Highly stretchable and self-healing strain sensors for motion detection in wireless human-machine interface.Nano Energy2020;76:105064

[24] Zhu M,Zhang Z.Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications.Sci Adv2020;6:eaaz8693 PMCID:PMC7209995

[25] Yang B,Qu X.Triboelectric-inertial sensing glove enhanced by charge-retained strategy for human-machine interaction.Adv Sci2025;12:e2408689 PMCID:PMC11744583

[26] Zhao Z,Li Y.Embedding high-resolution touch across robotic hands enables adaptive human-like grasping.Nat Mach Intell2025;7:889-900

[27] Lee GH,Kim H.Rapid meniscus-guided printing of stable semi-solid-state liquid metal microgranular-particle for soft electronics.Nat Commun2022;13:2643 PMCID:PMC9098628

[28] Yin L,Lv J.A self-sustainable wearable multi-modular E-textile bioenergy microgrid system.Nat Commun2021;12:1542 PMCID:PMC7943583

[29] Jan AA,Kim S.A skin-wearable and self-powered laminated pressure sensor based on triboelectric nanogenerator for monitoring human motion.Soft Sci2024;4:10

[30] Zhang W,Li G.Self-powered triboelectric-responsive microneedles with controllable release of optogenetically engineered extracellular vesicles for intervertebral disc degeneration repair.Nat Commun2024;15:5736 PMCID:PMC11233569

[31] Jin G,Geng J.Bioinspired soft caterpillar robot with ultra-stretchable bionic sensors based on functional liquid metal.Nano Energy2021;84:105896

[32] Sun Z,Shan X.Augmented tactile-perception and haptic-feedback rings as human-machine interfaces aiming for immersive interactions.Nat Commun2022;13:5224 PMCID:PMC9445040

[33] Wu P,Huang X.Liquid metal-based strain-sensing glove for human-machine interaction.Soft Sci2023;3:35

[34] Li Z,Tang W.Crossmodal sensory neurons based on high-performance flexible memristors for human-machine in-sensor computing system.Nat Commun2024;15:7275 PMCID:PMC11344147

[35] Liao X,Zhang X.A bioinspired analogous nerve towards artificial intelligence.Nat Commun2020;11:268 PMCID:PMC6959309

[36] Wang W,Zhong D.Neuromorphic sensorimotor loop embodied by monolithically integrated, low-voltage, soft e-skin.Science2023;380:735-42

[37] Liu F,Christou A,Kaboli M.Neuro-inspired electronic skin for robots.Sci Robot2022;7:eabl7344

[38] Niu H,Gao S.Perception-to-cognition tactile sensing based on artificial-intelligence-motivated human full-skin bionic electronic skin.Adv Mater2022;34:e2202622

[39] Xu J,Sun B.Stretchable, adhesive, and bioinspired visual electronic skin with strain/temperature/pressure multimodal non-interference sensing.ACS Appl Mater Interfaces2023;15:33774-83

[40] Guo X,Zhu Y.Zero-biased bionic fingertip E-skin with multimodal tactile perception and artificial intelligence for augmented touch awareness.Adv Mater2024;36:e2406778

[41] Li S,Li X.Bioinspired robot skin with mechanically gated electron channels for sliding tactile perception.Sci Adv2022;8:ade0720

[42] Li S,Wang L.Skin-inspired quadruple tactile sensors integrated on a robot hand enable object recognition.Sci Robot2020;5:abc8134

[43] Chen T,Zhu M.Triboelectric self-powered wearable flexible patch as 3D motion control interface for robotic manipulator.ACS Nano2018;12:11561-71

[44] Shao B,Wu TC.Large-area, untethered, metamorphic, and omnidirectionally stretchable multiplexing self-powered triboelectric skins.Nat Commun2024;15:1238 PMCID:PMC10858173

[45] Xie X,Zhao C.Neuromorphic computing-assisted triboelectric capacitive-coupled tactile sensor array for wireless mixed reality interaction.ACS Nano2024;18:17041-52 PMCID:PMC11223466

[46] Sun Y,Li D.Stretchable, multiplexed, and bimodal sensing electronic armor for colonoscopic continuum robot enhanced by triboelectric artificial synapse.Adv Mater2025;37:e2502203 PMCID:PMC12369696