Electronic skins with multimodal sensing and perception

Abstract

Multiple types of sensory information are detected and integrated to improve perceptual accuracy and sensitivity in biological cognition. However, current studies on electronic skin (e-skin) systems have mainly focused on the optimization of the modality-specific data acquisition and processing. Endowing e-skins with the abilities of multimodal sensing and even perception that can achieve high-level perception behaviors has been insufficiently explored. Moreover, the perception progress of multisensory e-skin systems is faced with challenges at both device and software levels. Here, we provide a perspective on the multisensory fusion of e-skins. The recent progress in e-skins realizing multimodal sensing is reviewed, followed by bottom-up and top-down multimodal perception. With the deepening understanding of neuroscience and the rapid advance of novel algorithms and devices, multimodal perception function becomes possible and will promote the development of highly intelligent e-skin systems.

Keywords

Electronic skins / multimodal sensing / perception fusion

References

[1] Lumpkin EA.Mechanisms of sensory transduction in the skin.Nature2007;445:858-65

[2] Ohyama T,Fetter RD.A multilevel multimodal circuit enhances action selection in drosophila.Nature2015;520:633-9

[3] Tan H,Tao Q,van Dijken S.Bioinspired multisensory neural network with crossmodal integration and recognition.Nat Commun2021;12:1120 PMCID:PMC7893014

[4] Ernst MO.Humans integrate visual and haptic information in a statistically optimal fashion.Nature2002;415:429-33

[5] Macaluso E.Multisensory spatial interactions: a window onto functional integration in the human brain.Trends Neurosci2005;28:264-71

[6] Green AM.Multisensory integration: resolving sensory ambiguities to build novel representations.Curr Opin Neurobiol2010;20:353-60 PMCID:PMC2901182

[7] Ohshiro T,DeAngelis GC.A normalization model of multisensory integration.Nat Neurosci2011;14:775-82 PMCID:PMC3102778

[8] Hagmann CE.Multisensory integration of redundant trisensory stimulation.Atten Percept Psychophys2016;78:2558-68 PMCID:PMC5564512

[9] Zhu B,Liu Y.Skin-inspired haptic memory arrays with an electrically reconfigurable architecture.Adv Mater2016;28:1559-66

[10] Chen S,Lou Z,Shen G.Recent developments in graphene-based tactile sensors and e-skins.Adv Mater Technol2018;3:1700248

[11] Jeon S,Trung TQ,Lee N.Flexible multimodal sensors for electronic skin: principle, materials, device, array architecture, and data acquisition method.Proc IEEE2019;107:2065-83

[12] Xu K,Takei K.Multifunctional skin-inspired flexible sensor systems for wearable electronics.Adv Mater Technol2019;4:1800628

[13] Li H,Liang Z.Wearable skin-like optoelectronic systems with suppression of motion artifacts for cuff-less continuous blood pressure monitor.Natl Sci Rev2020;7:849-62 PMCID:PMC8288864

[14] Wu X,Khan Y.A potentiometric mechanotransduction mechanism for novel electronic skins.Sci Adv2020;6:eaba1062 PMCID:PMC7439546

[15] Choi I,Lee SH.Bottom-up and top-down modulation of multisensory integration.Curr Opin Neurobiol2018;52:115-22

[16] Li H,Li X.Epidermal inorganic optoelectronics for blood oxygen measurement.Adv Healthc Mater2017;6:1601013

[17] Boutry CM,Jorda M.A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics.Sci Robot2018;3:eaau6914

[18] Choi S,Jung D.Highly conductive, stretchable and biocompatible Ag-Au core-sheath nanowire composite for wearable and implantable bioelectronics.Nat Nanotechnol2018;13:1048-56

[19] Wang M,Leow WR.Enhancing the matrix addressing of flexible sensory arrays by a highly nonlinear threshold switch.Adv Mater2018;30:e1802516

[20] Yang JC,Kwon SY,Bao Z.Electronic skin: recent progress and future prospects for skin-attachable devices for health monitoring, robotics, and prosthetics.Adv Mater2019;31:e1904765

[21] Lee S,Hassani FA.Nanomesh pressure sensor for monitoring finger manipulation without sensory interference.Science2020;370:966-70

[22] Wang Y,Yokota T.A durable nanomesh on-skin strain gauge for natural skin motion monitoring with minimum mechanical constraints.Sci Adv2020;6:eabb7043 PMCID:PMC7423357

[23] Yang X,Wang S.Ultrathin, stretchable, and breathable epidermal electronics based on a facile bubble blowing method.Adv Electron Mater2020;6:2000306

[24] Wang M,Wang T.Artificial skin perception.Adv Mater2021;33:e2003014

[25] Tien NT,Kim DI.A flexible bimodal sensor array for simultaneous sensing of pressure and temperature.Adv Mater2014;26:796-804

[26] Won SM,Kim BH.Multimodal sensing with a three-dimensional piezoresistive structure.ACS Nano2019;13:10972-9

[27] Li G,Wang L.Skin-inspired quadruple tactile sensors integrated on a robot hand enable object recognition.Sci Robot2020;5:eabc8134

[28] Senkowski D,Foxe JJ.Crossmodal binding through neural coherence: implications for multisensory processing.Trends Neurosci2008;31:401-9

[29] Stein BE.Multisensory integration: current issues from the perspective of the single neuron.Nat Rev Neurosci2008;9:255-66

[30] Fetsch CR,Angelaki DE.Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons.Nat Rev Neurosci2013;14:429-42 PMCID:PMC3820118

[31] Wang J,Cai P.Artificial sense technology: emulating and extending biological senses.ACS Nano2021;15:18671-8

[32] Li G.A multisensory tactile system for robotic hands to recognize objects.Adv Mater Technol2019;4:1900602

[33] Wang M,Wang T.Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors.Nat Electron2020;3:563-70

[34] Baltrušaitis T,Morency L.Challenges and applications in multimodal machine learning.The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition2018;2:17-48

[35] Zhou F.Near-sensor and in-sensor computing.Nat Electron2020;3:664-71

[36] Wang M,Luo Y.Fusing stretchable sensing technology with machine learning for human-machine interfaces.Adv Funct Mater2021;31:2008807

[37] Carleton A,Simon SA.Coding in the mammalian gustatory system.Trends Neurosci2010;33:326-34 PMCID:PMC2902637

[38] Svechtarova MI,Toebes BJ,Anton N.Sensor devices inspired by the five senses: a review.Electroanalysis2016;28:1201-41

[39] Keat J,Reid RC.Predicting every spike: a model for the responses of visual neurons.Neuron2001;30:803-17

[40] Bean BP.The action potential in mammalian central neurons.Nat Rev Neurosci2007;8:451-65

[41] Fries P.A mechanism for cognitive dynamics: neuronal communication through neuronal coherence.Trends Cogn Sci2005;9:474-80

[42] Womelsdorf T,Oostenveld R.Modulation of neuronal interactions through neuronal synchronization.Science2007;316:1609-12

[43] Atilgan H,Wood KC.Integration of visual information in auditory cortex promotes auditory scene analysis through multisensory binding.Neuron2018;97:640-655.e4 PMCID:PMC5814679

[44] Beauchamp MS.See me, hear me, touch me: multisensory integration in lateral occipital-temporal cortex.Curr Opin Neurobiol2005;15:145-53

[45] Kayser C,Augath M.Integration of touch and sound in auditory cortex.Neuron2005;48:373-84

[46] Talsma D,Soto-Faraco S.The multifaceted interplay between attention and multisensory integration.Trends Cogn Sci2010;14:400-10 PMCID:PMC3306770

[47] Arnal LH.Cortical oscillations and sensory predictions.Trends Cogn Sci2012;16:390-8

[48] Atiani S,Elgueda D.Emergent selectivity for task-relevant stimuli in higher-order auditory cortex.Neuron2014;82:486-99 PMCID:PMC4048815

[49] Makino H.Learning enhances the relative impact of top-down processing in the visual cortex.Nat Neurosci2015;18:1116-22 PMCID:PMC4523093

[50] Manita S,Homma C.A top-down cortical circuit for accurate sensory perception.Neuron2015;86:1304-16

[51] Someya T.Toward a new generation of smart skins.Nat Biotechnol2019;37:382-8

[52] Hua Q,Liu H.Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing.Nat Commun2018;9:244 PMCID:PMC5770430

[53] You I,Matsuhisa N.Artificial multimodal receptors based on ion relaxation dynamics.Science2020;370:961-5

[54] Ho DH,Kim SY,Kim DH.Stretchable and multimodal all graphene electronic skin.Adv Mater2016;28:2601-8

[55] Yamamoto Y,Yamamoto D.Printed multifunctional flexible device with an integrated motion sensor for health care monitoring.Sci Adv2016;2:e1601473 PMCID:PMC5262446

[56] Yang S,Nicolini L.“Cut-and-paste” manufacture of multiparametric epidermal sensor systems.Adv Mater2015;27:6423-30

[57] Kim SY,Park HW,Jeong Y.Highly sensitive and multimodal all-carbon skin sensors capable of simultaneously detecting tactile and biological stimuli.Adv Mater2015;27:4178-85

[58] Gao W,Nyein HYY.Fully integrated wearable sensor arrays for multiplexed in situ perspiration analysis.Nature2016;529:509-14 PMCID:PMC4996079

[59] Wang C,Zhang M,Zhang Y.An all-silk-derived dual-mode e-skin for simultaneous temperature-pressure detection.ACS Appl Mater Interfaces2017;9:39484-92

[60] Zhao S.Electronic skin with multifunction sensors based on thermosensation.Adv Mater2017;29:1606151

[61] Yu Y,Xu C.Biofuel-powered soft electronic skin with multiplexed and wireless sensing for human-machine interfaces.Sci Robot2020;5:eaaz7946 PMCID:PMC7326328

[62] Li L,Ran W.Dual sensing signal decoupling based on tellurium anisotropy for VR interaction and neuro-reflex system application.Nat Commun2022;13:5975 PMCID:PMC9550802

[63] Chun KY,Han CS.A wearable all-gel multimodal cutaneous sensor enabling simultaneous single-site monitoring of cardiac-related biophysical signals.Adv Mater2022;34:e2110082

[64] Yao S,Malhotra A.A wearable hydration sensor with conformal nanowire electrodes.Adv Healthc Mater2017;6:1601159

[65] Yang R,Tiwari N,Li T.Multimodal sensors with decoupled sensing mechanisms.Adv Sci2022;9:e2202470 PMCID:PMC9475538

[66] Gui Q,Gao N,Wang Y.A skin-inspired integrated sensor for synchronous monitoring of multiparameter signals.Adv Funct Mater2017;27:1702050

[67] Scaffaro R,Citarrella MC.Ionic tactile sensors as promising biomaterials for artificial skin: review of latest advances and future perspectives.Eur Polym J2021;151:110421

[68] Yan Y,Yang Z.Soft magnetic skin for super-resolution tactile sensing with force self-decoupling.Sci Robot2021;6:eabc8801

[69] Shin YE,Ghosh SK,Park J.Ultrasensitive multimodal tactile sensors with skin-inspired microstructures through localized ferroelectric polarization.Adv Sci2022;9:e2105423 PMCID:PMC8948547

[70] Zhang H,Lee J.Bioinspired chromotropic ionic skin with in-plane strain/temperature/pressure multimodal sensing and ultrahigh stimuli discriminability.Adv Funct Mater2022;32:2208362

[71] Wan C,Guo X.An artificial sensory neuron with visual-haptic fusion.Nat Commun2020;11:4602 PMCID:PMC7490423

[72] Liu M,Wang J.A star-nose-like tactile-olfactory bionic sensing array for robust object recognition in non-visual environments.Nat Commun2022;13:79 PMCID:PMC8748716

[73] Ehatisham-ul-haq M,Azam MA.Robust human activity recognition using multimodal feature-level fusion.IEEE Access2019;7:60736-51

[74] Ahmad Z.Human action recognition using deep multilevel multimodal (M²) fusion of depth and inertial sensors.IEEE Sensors J2020;20:1445-55

[75] Dawar N,Kehtarnavaz N.Data augmentation in deep learning-based fusion of depth and inertial sensing for action recognition.IEEE Sens Lett2019;3:1-4

[76] Dawar N.Action detection and recognition in continuous action streams by deep learning-based sensing fusion.IEEE Sensors J2018;18:9660-8

[77] Hall D.An introduction to multisensor data fusion.Proc IEEE1997;85:6-23

[78] Xu J,Zhang Y.Learning multimodal attention LSTM networks for video captioning. Proceedings of the 25th ACM international conference on Multimedia; 2017. p. 537-45.

[79] Shon S,Glass J.Noise-tolerant audio-visual online person verification using an attention-based neural network fusion. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2019. p. 3995-9.

[80] Shyu M,Chen M.Video semantic event/concept detection using a subspace-based multimedia data mining framework.IEEE Trans Multimedia2008;10:252-9

[81] Yang Z,Lu Z,Gong Z.Semi-supervised multimodal clustering algorithm integrating label signals for social event detection. 2015 IEEE International Conference on Multimedia Big Data; 2015 Apr 20-22; Beijing, China. IEEE; 2015. p. 32-9.

[82] Zhou P,Chen W,Jia J.Modality attention for end-to-end audio-visual speech recognition. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2019. p. 6565-9.