Artificial visual-tactile perception array for enhanced memory and neuromorphic computations

Abstract

The emulation of human multisensory functions to construct artificial perception systems is an intriguing challenge for developing humanoid robotics and cross-modal human–machine interfaces. Inspired by human multisensory signal generation and neuroplasticity-based signal processing, here, an artificial perceptual neuro array with visual-tactile sensing, processing, learning, and memory is demonstrated. The neuromorphic bimodal perception array compactly combines an artificial photoelectric synapse network and an integrated mechanoluminescent layer, endowing individual and synergistic plastic modulation of optical and mechanical information, including short-term memory, long-term memory, paired pulse facilitation, and “learning-experience” behavior. Sequential or superimposed visual and tactile stimuli inputs can efficiently simulate the associative learning process of “Pavlov's dog”. The fusion of visual and tactile modulation enables enhanced memory of the stimulation image during the learning process. A machine-learning algorithm is coupled with an artificial neural network for pattern recognition, achieving a recognition accuracy of 70% for bimodal training, which is higher than that obtained by unimodal training. In addition, the artificial perceptual neuron has a low energy consumption of ~20 pJ. With its mechanical compliance and simple architecture, the neuromorphic bimodal perception array has promising applications in large-scale cross-modal interactions and high-throughput intelligent perceptions.

Keywords

artificial intelligent systems / mechanoluminescence / neuromorphic computing / optoelectronic synapse / visual-tactile perception

References

[1] Kim S, Roe DG, Choi YY, et al. Artificial stimulus-response system capable of conscious response. Sci Adv. 2021;7(15):eabe3996.

[2] Jung YH, Park B, Kim JU, Kim TI. Bioinspired electronics for artificial sensory systems. Adv Mater. 2019;31(34):1803637.

[3] Sun FQ, Lu QF, Hao MM, et al. An artificial neuromorphic somatosensory system with spatio-temporal tactile perception and feedback functions. npj Flexible Electron. 2022;6(1):72.

[4] Wei HH, Shi RC, Sun L, et al. Mimicking efferent nerves using a graphdiyne-based artificial synapse with multiple ion diffusion dynamics. Nat Commun. 2021;12(1):1068.

[5] Kim J, Song S, Lee JM, et al. Metal-oxide heterojunction optoelectronic synapse and multilevel memory devices enabled by broad spectral photocarrier modulation. Small. 2023;35:2301186.

[6] Chen S, Zhang T, Tappertzhofen S, Yang Y, Valov I. Electrochemical memristor-based artificial neurons and synapses—fundamentals, applications, and challenges. Adv Mater. 2023;35(37):e2301924.

[7] Baek JH, Kwak KJ, Kim SJ, et al. Two-terminal lithium-mediated artificial synapses with enhanced weight modulation for feasible hardware neural networks. Nano-Micro Lett. 2023;15(1):69.

[8] Jiang Z, Chen N, Yi ZG, et al. A 1.3-micrometre-thick elastic conductor for seamless on-skin and implantable sensors. Nat Electron. 2022;5(11):784-793.

[9] Oh S, Cho JI, Lee BH, et al. Flexible artificial Si-In-Zn-O/ion gel synapse and its application to sensory-neuromorphic system for sign language translation. Sci Adv. 2021;7(44):eabg9450.

[10] Zhu QB, Li B, Yang DD, et al. A flexible ultrasensitive optoelectronic sensor array for neuromorphic vision systems. Nat Commun. 2021;12(1):1798.

[11] Han X, Xu ZS, Wu WQ, et al. Recent progress in optoelectronic synapses for artificial visual-perception system. Small Struct. 2020;1(3):2000029.

[12] Wen XN, Wu WZ, Pan CF, et al. Development and progress in piezotronics. Nano Energy. 2015;14:276-295.

[13] Chun S, Kim JS, Yoo Y, et al. An artificial neural tactile sensing system. Nat Electron. 2021;4(6):429-438.

[14] Wang CF, Dong L, Peng DF, et al. Tactile sensors for advanced intelligent systems. Adv Intell Sys. 2019;1(8):1900090.

[15] Wang M, Luo YF, Wang T, et al. Artificial skin perception. Adv Mater. 2021;33(19):2003014.

[16] Zhang JH, Yao HM, Mo JY, et al. Finger-inspired rigid-soft hybrid tactile sensor with superior sensitivity at high frequency. Nat Commun. 2022;13(1):5076.

[17] Liu JH, Zhang ZC, Qiao S, et al. Lateral bipolar photoresistance effect in the CIGS heterojunction and its application in position sensitive detector and memory device. Sci Bull. 2020;65(6):477-485.

[18] Jia MM, Guo PW, Wang W, et al. Tactile tribotronic reconfigurable p-n junctions for artificial synapses. Sci Bull. 2022;67(8):803-812.

[19] Liu QH, Yin L, Zhao C, et al. All-in-one metal-oxide heterojunction artificial synapses for visual sensory and neuromorphic computing systems. Nano Energy. 2022;97:107171.

[20] Sarkar T, Lieberth K, Pavlou A, et al. An organic artificial spiking neuron for in situ neuromorphic sensing and biointerfacing. Nat Electron. 2022;5(11):774-783.

[21] Sarwat SG, Kersting B, Moraitis T, Jonnalagadda VP, Sebastian A. Phase-change memtransistive synapses for mixed-plasticity neural computations. Nat Nanotechnol. 2022;17(5):507-513.

[22] Shi JL, Jie JS, Deng W, et al. A fully solution-printed photosynaptic transistor array with ultralow energy consumption for artificial-vision neural networks. Adv Mater. 2022;34(18):2200380.

[23] Liu L, Xu WL, Ni Y, et al. Stretchable neuromorphic transistor that combines multisensing and information processing for epidermal gesture recognition. ACS Nano. 2022;16(2):2282-2291.

[24] Wan HC, Zhao JY, Lo LW, et al. Multimodal artificial neurological sensory-memory system based on flexible carbon nanotube synaptic transistor. ACS Nano. 2021;15(9):14587-14597.

[25] Yu JR, Gao GY, Huang JR, et al. Contact-electrification-activated artificial afferents at femtojoule energy. Nat Commun. 2021;12(1):1581.

[26] Yu JR, Yang XX, Gao GY, et al. Bioinspired mechano-photonic artificial synapse based on graphene/MoS₂ heterostructure. Sci Adv. 2021;7(12):eabd9117.

[27] Kim Y, Chortos A, Xu WT, et al. A bioinspired flexible organic artificial afferent nerve. Science. 2018;360(6392):998-1003.

[28] Wan CJ, Chen G, Fu YM, et al. An artificial sensory neuron with tactile perceptual learning. Adv Mater. 2018;30(30):1801291.

[29] Wan CJ, Cai PQ, Guo XT, et al. An artificial sensory neuron with visual-haptic fusion. Nat Commun. 2020;11(1):4602.

[30] Tan HW, Tao QZ, Pande I, et al. Tactile sensory coding and learning with bio-inspired optoelectronic spiking afferent nerves. Nat Commun. 2020;11(1):1369.

[31] Tan HW, Zhou YF, Tao QZ, et al. Bioinspired multisensory neural network with crossmodal integration and recognition. Nat Commun. 2021;12(1):1120.

[32] Wang W, Jiang Y, Zhong D, et al. Neuromorphic sensorimotor loop embodied by monolithically integrated, low-voltage, soft e-skin. Science. 2023;380(6646):735-742.

[33] Yu RJ, Zhang XH, Gao CS, et al. Low-voltage solution-processed artificial optoelectronic hybrid-integrated neuron based on 2D MXene for multi-task spiking neural network. Nano Energy. 2022;99:107418.

[34] Zhu Y, Wu C, Xu Z, et al. Light-emitting memristors for optoelectronic artificial efferent nerve. Nano Lett. 2021;21(14):6087-6094.

[35] Hou B, Yi LY, Li C, et al. An interactive mouthguard based on mechanoluminescence-powered optical fibre sensors for bite-controlled device operation. Nat Electron. 2022;5(10):682-693.

[36] Ma X, Wang C, Wei R, et al. Bimodal tactile sensor without signal fusion for user-interactive applications. ACS Nano. 2022;16(2):2789-2797.

[37] Wang CF, Ma RH, Peng DF, et al. Mechanoluminescent hybrids from a natural resource for energy-related applications. InfoMat. 2021;3(11):1272-1284.

[38] Wei RL, He JQ, Ge SP, et al. Self-powered all-optical tactile sensing platform for user-interactive interface. Adv Mater Technol. 2023;8(1):202200757.

[39] Yang F, Wu X, Cui H, et al. Palette of rechargeable mechanoluminescent fluids produced by a biomineral-inspired suppressed dissolution approach. J Am Chem Soc. 2022;144(40):18406-18418.

[40] Zhuang YX, Xie RJ. Mechanoluminescence rebrightening the prospects of stress sensing: A review. Adv Mater. 2021;33(50):2005925.

[41] Noh H-K, Chang KJ, Ryu B, Lee WJ. Electronic structure of oxygen-vacancy defects in amorphous In-Ga-Zn-O semiconductors. Phys Rev B. 2011;84(11):115205.

[42] Lee M, Lee W, Choi S, et al. Brain-inspired photonic neuromorphic devices using photodynamic amorphous oxide semiconductors and their persistent photoconductivity. Adv Mater. 2017;29(28):1700951.

[43] Kwon SM, Cho SW, Kim M, Heo JS, Kim YH, Park SK. Environment-adaptable artificial visual perception behaviors using a light-adjustable optoelectronic neuromorphic device array. Adv Mater. 2019;31(52):1906433.

[44] Kim M-K, Lee J-S. Synergistic improvement of long-term plasticity in photonic synapses using ferroelectric polarization in hafnia-based oxide-semiconductor transistors. Adv Mater. 2020;32(12):1907826.

[45] Abbott LF, Regehr WG. Synaptic computation. Nature. 2004;431(7010):796-803.

[46] Wu W, Wang X, Han X, et al. Flexible photodetector arrays based on patterned CH₃NH₃PbI_3−xCl_x perovskite film for real-time photosensing and imaging. Adv Mater. 2019;31(3):1805913.

[47] Dou L, Yang Y, You J, et al. Solution-processed hybrid perovskite photodetectors with high detectivity. Nat Commun. 2014;5(1):5404.

[48] Sun ZH, Li JH, Liu CM, et al. Trap-assisted charge storage in titania nanocrystals toward optoelectronic nonvolatile memory. Nano Lett. 2021;21(1):723-730.

[49] Wei SL, Wang F, Zou XM, et al. Flexible quasi-2D perovskite/IGZO phototransistors for ultrasensitive and broadband photodetection. Adv Mater. 2020;32(6):1907527.

[50] Peng DF, Jiang Y, Huang BL, et al. A ZnS/CaZnOS heterojunction for efficient mechanical-to-optical energy conversion by conduction band offset. Adv Mater. 2020;32(16):1907747.

[51] Yoneyama M, Fukushima Y, Tsukada M, Aihara T. Spatiotemporal characteristics of synaptic EPSP summation on the dendritic trees of hippocampal CA₁ pyramidal neurons as revealed by laser uncaging stimulation. Cogn Neurodyn. 2011;5(4):333-342.

RIGHTS & PERMISSIONS

2023 2023 The Authors. InfoMat published by UESTC and John Wiley & Sons Australia, Ltd.