Flexible retinomorphic vision sensors with scotopic and photopic adaptation for a fully flexible neuromorphic machine vision system

Abstract

Bioinspired neuromorphic machine vision system (NMVS) that integrates retinomorphic sensing and neuromorphic computing into one monolithic system is regarded as the most promising architecture for visual perception. However, the large intensity range of natural lights and complex illumination conditions in actual scenarios always require the NMVS to dynamically adjust its sensitivity according to the environmental conditions, just like the visual adaptation function of the human retina. Although some opto-sensors with scotopic or photopic adaption have been developed, NMVSs, especially fully flexible NMVSs, with both scotopic and photopic adaptation functions are rarely reported. Here we propose an ion-modulation strategy to dynamically adjust the photosensitivity and time-varying activation/inhibition characteristics depending on the illumination conditions, and develop a flexible ion-modulated phototransistor array based on MoS₂/graphdiyne heterostructure, which can execute both retinomorphic sensing and neuromorphic computing. By controlling the intercalated Li⁺ ions in graphdiyne, both scotopic and photopic adaptation functions are demonstrated successfully. A fully flexible NMVS consisting of front-end retinomorphic vision sensors and a back-end convolutional neural network is constructed based on the as-fabricated 28 × 28 device array, demonstrating quite high recognition accuracies for both dim and bright images and robust flexibility. This effort for fully flexible and monolithic NMVS paves the way for its applications in wearable scenarios.

Keywords

flexible neuromorphic machine vision system / in-sensor computing / neuromorphic computing / retinomorphic vision sensors / visual adaptation

References

[1] Mennel L, Symonowicz J, Wachter S, Polyushkin DK, Molina-Mendoza AJ. Mueller T. Ultrafast machine vision with 2D material neural network image sensors. Nature. 2020; 579(7797): 62-66.

[2] Chai Y. In-sensor computing for machine vision. Nature. 2020; 579(7797): 32-33.

[3] Zhou F, Chai Y. Near-sensor and in-sensor computing. Nat Electron. 2020; 3(11): 664-671.

[4] Wang C-Y, Liang S-J, Wang S, et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci Adv. 2020; 6(26): eaba6173.

[5] Cui B, Fan Z, Li W, et al. Ferroelectric photosensor network: an advanced hardware solution to real-time machine vision. Nat Commun. 2022; 13(1): 1707.

[6] Zhou F, Zhou Z, Chen J, et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat Nanotechnol. 2019; 14(8): 776-782.

[7] Dang B, Liu K, Wu X, et al. One-phototransistor-one-memristor array with high-linearity light-tunable weight for optic neuromorphic computing. Adv Mater. 2022; 35(37): 2204844.

[8] Fu X, Li T, Cai B, et al. Graphene/MoS_2-xO_x/graphene photomemristor with tunable non-volatile responsivities for neuromorphic vision processing. Light: Sci Appl. 2023; 12(1): 39.

[9] Zhang Z-C, Chen X-D, Lu T-B. Recent progress in neuromorphic and memory devices based on graphdiyne. Sci Technol Adv Mater. 2023; 24(1): 2196240.

[10] Wan T, Shao B, Ma S, et al. In-sensor computing: materials, devices, and integration technologies. Adv Mater. 2022; 35(37): 2203830.

[11] Lee D, Park M, Baek Y, Bae B, Heo J, Lee K. In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing. Nat Commun. 2022; 13(1): 5223.

[12] Wang S, Wang C-Y, Wang P, et al. Networking retinomorphic sensor with memristive crossbar for brain-inspired visual perception. Natl Sci Rev. 2021; 8(2): nwaa172.

[13] Feng G, Zhang X, Tian B, Duan C. Retinomorphic hardware for in-sensor computing. InfoMat. 2023; 5(9): e12473.

[14] Pan X, Shi J, Wang P, et al. Parallel perception of visual motion using light-tunable memory matrix. Sci Adv. 2023; 9(39): eadi4083.

[15] Hou Y, Li J, Yoon J, et al. Retina-inspired narrowband perovskite sensor array for panchromatic imaging. Sci Adv. 2023; 9(15): eade2338.

[16] Pi L, Wang P, Liang S-J, et al. Broadband convolutional processing using band-alignment-tunable heterostructures. Nat Electron. 2022; 5(4): 248-254.

[17] Zhang G-X, Zhang Z-C, Chen X-D. et al. Broadband sensory networks with locally stored responsivities for neuromorphic machine vision. Sci Adv. 2023; 9(37): eadi5104.

[18] Wu G, Zhang X, Feng G, et al. Ferroelectric-defined reconfigurable homojunctions for in-memory sensing and computing. Nat Mater. 2023; 22: 1499-1506.

[19] Li T, Miao J, Fu X, et al. Reconfigurable, non-volatile neuromorphi. photovoltaics. Nat Nanotechnol. 2023; 18: 1303-1310.

[20] Huang P-Y, Jiang B-Y, Chen H-J. et al. Neuro-inspired optical sensor array for high-accuracy static image recognition and dynamic trace extraction. Nat Commun. 2023; 14(1): 6736.

[21] Liao F, Zhou Z, Kim BJ, et al. Bioinspired in-sensor visual adaptation for accurate perception. Nat Electron. 2022; 5(2): 84-91.

[22] Smirnakis SM, Berry MJ, Warland DK, Bialek W, Meister M. Adaptation of retinal processing to image contrast and spatial scale. Nature. 1997; 386(6620): 69-73.

[23] Dunn FA, Lankheet MJ, Rieke F. Light adaptation in cone vision involves switching between receptor and post-receptor sites. Nature. 2007; 449(7162): 603-606.

[24] He Z, Shen H, Ye D, et al. An organic transistor with light intensity-dependent active photoadaptation. Nat Electron. 2021; 4(7): 522-529.

[25] Hong S, Choi SH, Park J, et al. Sensory adaptation and neuromorphic phototransistors based on CsPb(Br_1-xI_x)₃ perovskite and MoS₂ hybrid structure. ACS Nano. 2020; 14(8): 9796-9806.

[26] Liu W, Yang X, Wang Z, et al. Self-powered and broadband opto-sensor with bionic visual adaptation function based on multilayer γ-InSe flakes. Light: Sci Appl. 2023; 12(1): 180.

[27] Jin C, Liu W, Xu Y, et al. Artificial vision adaption mimicked by an optoelectrical in₂o₃ transistor array. Nano Lett. 2022; 22(8): 3372-3379.

[28] Lee S-J, Lin Z, Huang J, et al. Programmable devices based on reversible solid-state doping of two-dimensional semiconductors with superionic silver iodide. Nat Electron. 2020; 3(10): 630-637.

[29] Cai Y, Wang F, Wang X, et al. Broadband visual adaption and image recognition in a monolithic neuromorphic machine vision system. Adv Funct Mater. 2023; 33(5): 2212917.

[30] Hou Y-X, Li Y, Zhang Z-C, et al. Large-scale and flexible optical synapses for neuromorphic computing and integrated visible information sensing memory processing. ACS Nano. 2021; 15(1): 1497-1508.

[31] Wang F-D, Yu M-X, Chen X-D. et al. Optically modulated dual-mode memristor arrays based on core-shell CsPbBr₃@graphdiyne nanocrystals for fully memristive neuromorphic computing hardware. SmartMat. 2023; 4(1): e1135.

[32] Wang Y, Gong Y, Huang S, et al. Memristor-based biomimetic compound eye for real-time collision detection. Nat Commun. 2021; 12(1): 5979.

[33] Seo S, Lee J-J, Lee R-G. et al. An optogenetics-inspired flexible van der Waals optoelectronic synapse and its application to a convolutional neural network. Adv Mater. 2021; 33(40): 2102980.

[34] Jin T, Gao J, Wang Y, Chen W. Flexible neuromorphic electronics based on low-dimensional materials. Sci China Mater. 2022; 65(8): 2154-2159.

[35] Zhou K, Jia Z, Ma X-Q, et al. Manufacturing of graphene based synaptic devices for optoelectronic applications. Int J Extreme Manufac. 2023; 5(4): 042006.

[36] Li Q, Wang T, Fang Y, et al. Ultralow power wearable organic ferroelectric device for optoelectronic neuromorphic computing. Nano Lett. 2022; 22(15): 6435-6443.

[37] Li Y, Wang J, Yang Q, Shen G. Flexible artificial optoelectronic synapse based on lead-free metal halide nanocrystals for neuromorphic computing and color recognition. Adv Sci. 2022; 9(22): 2202123.

[38] Li Q, Wang T, Hu X, et al. Organic optoelectronic synaptic devices for energy-efficient neuromorphic computing. IEEE Electron Device Lett. 2022; 43(7): 1089-1092.

[39] Ni Y, Yang L, Feng J, Liu J, Sun L, Xu W. Flexible optoelectronic neural transistors with broadband spectrum sensing and instant electrical processing for multimodal neuromorphic computing. SmartMat. 2023; 4(2): e1154.

[40] Xu Y, Zhang G, Liu W, et al. Flexible multiterminal photoelectronic neurotransistors based on self-assembled rubber semiconductors for spatiotemporal information processing. SmartMat. 2023; 4(2): e1162.

[41] Zhang J, Guo Z, Sun T, et al. Energy-efficient organic photoelectric synaptic transistors with environment-friendly CuInSe₂ quantum dots for broadband neuromorphic computing. SmartMat. 2024; 5(4): e1246.

[42] Li J, Zhang Z, Kong Y, et al. Synthesis of wafer-scale ultrathin graphdiyne for flexible optoelectronic memory with over 256 storage levels. Chem. 2021; 7(5): 1284-1296.

[43] Yao BW, Li J, Chen XD, et al. Non-volatile electrolyte-gated transistors based on graphdiyne/MoS₂ with robust stability for low-power neuromorphic computing and logic-in-memory. Adv Funct Mater. 2021; 31(25): 2100069.

[44] Yao P, Wu H, Gao B, et al. Fully hardware-implemented memristor convolutional neural network. Nature. 2020; 577(7792): 641-646.

[45] Posch C, Serrano-Gotarredona T. Linares-Barranco B, Delbruck T. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc IEEE. 2014; 102(10): 1470-1484.

[46] Hong X, Huang Y, Tian Q, et al. Two-dimensional perovskite-gated AlGaN/GaN high-electron-mobility-transistor for neuromorphic vision sensor. Adv Sci. 2022; 9(27): 2202019.

[47] Zhang Z, Wang S, Liu C, Xie R, Hu W, Zhou P. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat Nanotechnol. 2022; 17(1): 27-32.

[48] Gao X, Liu H, Wang D, Zhang J. Graphdiyne: synthesis, properties, and applications. Chem Soc Rev. 2019; 48(3): 908-936.

[49] Wang B, Luo H, Wang X, et al. Bifunctional NbS₂-based asymmetric heterostructure for lateral and vertical electronic devices. ACS Nano. 2020; 14(1): 175-184.

[50] Zheng X, Chen S, Li J, et al. Two-dimensional carbon graphdiyne: advances in fundamental and application research. ACS Nano. 2023; 17(15): 14309-14346.

[51] Zhu X, Li D, Liang X, Lu WD. Ionic modulation and ionic coupling effects in MoS2 devices for neuromorphic computing. Nat Mater. 2019; 18(2): 141-148.

[52] Chen P, Liu F, Lin P, et al. Open-loop analog programmable electrochemical memory array. Nat Commun. 2023; 14(1): 6184.

[53] Zhu J, Yang Y, Jia R, et al. Ion-gated synaptic transistors based on 2D van der Waals crystals with tunable diffusive dynamics. Adv Mater. 2018; 30(21): 1800195.

[54] Wen J, Tang W, Kang Z, et al. Direct charge trapping multilevel memory with graphdiyne/MoS₂ van der Waals heterostructure. Adv Sci. 2021; 8(21): 2101417.

[55] Zhang Z-C, Li Y, Wang J-J, et al. Synthesis of wafer-scale graphdiyne/graphene heterostructure for scalable neuromorphic computing and artificial visual systems. Nano Res. 2021; 14(12): 4591-4600.

[56] Wang X-H, Zhang Z-C, Wang J-J. et al. Synthesis of wafer-scale monolayer pyrenyl graphdiyne on ultrathin hexagonal boron nitride for multibit optoelectronic memory. ACS Appl Mater Interfaces. 2020; 12(29): 33069-33075.

[57] Zhang ZC, Li Y, Li J, et al. An ultrafast nonvolatile memory with low operation voltage for high-speed and low-power applications. Adv Funct Mater. 2021; 31(28): 2102571.

[58] Wang Z, Joshi S, Savel’ev S, et al. Fully memristive neural networks for pattern classification with unsupervised learning. Nat Electron. 2018; 1(2): 137-145.

[59] van de Burgt Y, Lubberman E, Fuller EJ, et al. A non-volatile organic electrochemical device as a low-voltage artificial synapse for neuromorphic computing. Nat Mater. 2017; 16(4): 414-418.

[60] Fuller EJ, Gabaly FE, Léonard F, et al. Li-ion synaptic transistor for low-power analog computing. Adv Mater. 2017; 29(4): 1604310.

[61] Zhu Y, Mao H, Zhu Y, et al. CMOS-compatible neuromorphic devices for neuromorphic perception and computing: a review. Int J Extreme Manuf. 2023; 5(4): 042010.

RIGHTS & PERMISSIONS

2024 The Authors. SmartMat published by Tianjin University and John Wiley & Sons Australia, Ltd.