Bimodal Coupling Haptic Perceptron for Accurate Contactless Gesture Perception and Material Identification

Abstract

A bimodal coupled multifunctional tactile perceptron for contactless gesture recognition and material identification is proposed to address the challenges posed by limited functionality, signal interference from multimodal collaborative work, and the high power consumption of traditional tactile sensors. This perceptron integrates a capacitive sensor and a triboelectric sensor symmetrically, employing an energy complementarity strategy to reduce power consumption and implementing symmetrical distribution of two sensors for physical isolation to prevent signal interference. The capacitive sensor detects external pressure, providing information on material properties such as hardness, softness, and deformation, with a wide linear response range of 0–745.3 kPa. The triboelectric sensor captures the electron affinity of measured object. Further, by utilising machine learning algorithms, a system for contactless gesture recognition and material identification is engineered. This system demonstrates a remarkable accuracy rate of 98.5% when recognising 5 gestures, and achieves a perfect identification (100%) of 10 different materials aided by incorporating capacitive and triboelectric response. These results greatly advance the progress of tactile perceptrons with high integration, low power consumption, and multifunctionality, enhancing their effectiveness and reliability in smart device applications.

Graphical Abstract

References

[1] ZareiM, LeeG, LeeSG, ChoK. Advances in biodegradable electronic skin: material progress and recent applications in sensing, robotics, and human–machine interfaces. Adv Mater, 2023, 35: 2203193

[2] QiaoH, SunS, WuP. Non-equilibrium-growing aesthetic ionic skin for fingertip-like strain-undisturbed tactile sensation and texture recognition. Adv Mater, 2023, 35: 2300593

[3] QuX, LiuZ, TanP, WangC, LiuY. Artificial tactile perception smart finger for material identification based on triboelectric sensing. Sci Adv, 2022, 8: eabq2521

[4] LuD, LiuT, MengX, LuoB, YuanJ. Wearable triboelectric visual sensors for tactile perception. Adv Mater, 2023, 35: 2209117

[5] LiL, LiuY, SongC, ShengS, YangL. Wearable alignment-free microfiber-based sensor chip for precise vital signs monitoring and cardiovascular assessment. Adv Fiber Mater, 2022, 4: 475

[6] NiuL, WangJ, WangK, PanH, JiangG. High-speed sirospun conductive yarn for stretchable embedded knitted circuit and self-powered wearable device. Adv Fiber Mater, 2023, 5: 154

[7] WenD-L, PangY-X, HuangP, WangY-L, ZhangX-R. Silk fibroin-based wearable all-fiber multifunctional sensor for smart clothing. Adv Fiber Mater, 2022, 4: 873

[8] LiD, ZhouJ, YaoK, LiuS, HeJ. Touch IoT enabled by wireless self-sensing and haptic-reproducing electronic skin. Sci Adv, 2022, 8: eade2450

[9] ShenZ, ZhangZ, ZhangN, LiJ, ZhouP. High-stretchability, ultralow-hysteresis conducting polymer hydrogel strain sensors for soft machines. Adv Mater, 2022, 34: 2203650

[10] LvC, TianC, JiangJ, DangY, LiuY. Ultrasensitive linear capacitive pressure sensor with wrinkled microstructures for tactile perception. Adv Sci, 2023, 10: 2206807

[11] WangS, WangX, WangQ, MaS, XiaoJ. Flexible optoelectronic multimodal proximity/pressure/temperature sensors with low signal interference. Adv Mater, 2023, 35: 2304701

[12] ZhuH, LuoH, CaiM, SongJ. A multifunctional flexible tactile sensor based on resistive effect for simultaneous sensing of pressure and temperature. Adv Sci, 2024, 11: 2307693

[13] QuJ, MaoB, LiZ, XuY, ZhouK. Recent progress in advanced tactile sensing technologies for soft grippers. Adv Funct Mater, 2023, 33: 2306249

[14] ZhaoC, WangY, TangG, RuJ, ZhuZ. Ionic flexible sensors: Mechanisms, materials, structures, and applications. Adv Funct Mater, 2022, 32: 2110417

[15] WeiX, WangB, WuZ, WangZL. An open-environment tactile sensing system: Toward simple and efficient material identification. Adv Mater, 2022, 34: 2203073

[16] JiJ, ZhaoW, WangY, LiQ, WangG. Templated laser-induced-graphene-based tactile sensors enable wearable health monitoring and texture recognition via deep neural network. ACS Nano, 2023, 17: 20153

[17] YangL, WangZ, WangH, JinB, MengC. Self-healing, reconfigurable, thermal-switching, transformative electronics for health monitoring. Adv Mater, 2023, 35: 2207742

[18] CuiZ, WangW, XiaH, WangC, TuJ. Freestanding and scalable force-softness bimodal sensor arrays for haptic body-feature identification. Adv Mater, 2022, 34: 2207016

[19] PyoS, LeeJ, BaeK, SimS, KimJ. Recent progress in flexible tactile sensors for human-interactive systems: from sensors to advanced applications. Adv Mater, 2021, 33: 2005902

[20] LiuY, YiuC, SongZ, HuangY, YaoK. Electronic skin as wireless human-machine interfaces for robotic vr. Sci Adv, 2022, 8: eabl6700

[21] MishraS, KimY-S, IntarasirisawatJ, KwonY-T, LeeY. Soft, wireless periocular wearable electronics for real-time detection of eye vergence in a virtual reality toward mobile eye therapies. Sci Adv, 2020, 6: eaay1729

[22] SharA, GlassP, ParkSH, JoungD. 3D printable one-part carbon nanotube-elastomer ink for health monitoring applications. Adv Funct Mater, 2023, 33: 2211079

[23] SunwooS-H, HanSI, JungD, KimM, NamS. Stretchable low-impedance conductor with Ag–Au–Pt core–shell–shell nanowires and in situ formed Pt nanoparticles for wearable and implantable device. ACS Nano, 2023, 17: 7550

[24] SongK, ZhaoR, WangZL, YangY. Conjuncted pyro-piezoelectric effect for self-powered simultaneous temperature and pressure sensing. Adv Mater, 2019, 31: 1902831

[25] ShenZ, ZhuX, MajidiC, GuG. Cutaneous ionogel mechanoreceptors for soft machines, physiological sensing, and amputee prostheses. Adv Mater, 2021, 33: 2102069

[26] LeeJH, KimSH, HeoJS, KwakJY, ParkCW. Heterogeneous structure omnidirectional strain sensor arrays with cognitively learned neural networks. Adv Mater, 2023, 35: 2208184

[27] LuY, KongD, YangG, WangR, PangG. Machine learning-enabled tactile sensor design for dynamic touch decoding. Adv Sci, 2023, 10: 2303949

[28] LiS, XuJ, MuY, WangP, ZhuH. Fatigue-resistant and hysteresis-free composite fibers with a heterogeneous hierarchical structure. Adv Fiber Mater, 2023, 5: 1643

[29] WangZ, ZhouH, LiuD, ChenX, WangD. A structural gel composite enabled robust underwater mechanosensing strategy with high sensitivity. Adv Funct Mater, 2022, 32: 2201396

[30] ZhuS, LiY, YelemulatiH, DengX, LiY. An artificial remote tactile device with 3D depth-of-field sensation. Sci Adv, 2022, 8: eabo5314

[31] SeoB, ChaY, ChoiY, KimS, ChoiW. Rationally designed micropixelation-free tactile sensors via contour profile of triboelectric field propagation. Nano Energy, 2023, 109 108255

[32] BoutryCM, NegreM, JordaM, VardoulisO, ChortosA. A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics. Sci Robot, 2018, 3: eaau6914

[33] LiJ, LiJ, TangY, LiuZ, ZhangZ. Touchable gustation via a Hoffmeister gel iontronic sensor. ACS Nano, 2023, 17: 5129

[34] WenF, ZhangZ, HeT, LeeC. Ai enabled sign language recognition and Vr space bidirectional communication using triboelectric smart glove. Nat Commun, 2021, 12: 5378

[35] LiZ, MaY, ZhangK, WanJ, ZhaoD. Air permeable vibrotactile actuators for wearable wireless haptics. Adv Funct Mater, 2023, 33: 2211146

[36] ParkJ, KangD-H, ChaeH, GhoshSK, JeongC. Frequency-selective acoustic and haptic smart skin for dual-mode dynamic/static human-machine interface. Sci Adv, 2022, 8: eabj9220

[37] WeiC, LinW, LiangS, ChenM, ZhengY. An all-in-one multifunctional touch sensor with carbon-based gradient resistance elements. Nano-Micro Lett, 2022, 14: 131

[38] ChenZ, ZhangS, LuoS, SunF, FangB. Tacchi: a pluggable and low computational cost elastomer deformation simulator for optical tactile sensors. IEEE Robot Autom Lett, 2023, 8: 1239

[39] LiS, HuangJ, WangM, DengK, GuoC. Structural electronic skin for conformal tactile sensing. Adv Sci, 2023, 10: 2304106

[40] ShiQ, SunZ, LeX, XieJ, LeeC. Soft robotic perception system with ultrasonic auto-positioning and multimodal sensory intelligence. ACS Nano, 2023, 17: 4985

[41] YeG, WanY, WuJ, ZhuangW, ZhouZ. Multifunctional device integrating dual-temperature regulator for outdoor personal thermal comfort and triboelectric nanogenerator for self-powered human-machine interaction. Nano Energy, 2022, 97 107148

[42] ZouH, ZhangY, GuoL, WangP, HeX. Quantifying the triboelectric series. Nat Commun, 2019, 10: 1427

Funding

National Natural Science Foundation of China(No. 52071282)

Applied Basic Research Key Project of Yunnan(No. 202001BB050046)

Scientific Research Fund of Education Department of Yunnan Province(No. 2024Y041)

National Training Programs of Innovation and Entrepreneurship for Undergraduates(No. 202210673002)

RIGHTS & PERMISSIONS

Donghua University, Shanghai, China