Thin-film event-based vision sensors for enhanced multispectral perception beyond human vision

Abstract

Dynamic detection is crucial for intelligent vision systems, enabling applications like autonomous vehicles and advanced surveillance. Event-based sensors, which convert illumination variations into sparse event spikes, are highly effective for dynamic detection with low data redundancy. However, current event-based vision sensors with simplified photosensitive capacitor structures face limitations, particularly in their spectral response, which hinders effective information acquisition in multispectral scenes. Here, we introduce a two-terminal thin-film event-based vision sensor that innovatively integrates an inorganic oxide p–n junction with the pyro-phototronic effect, synergistically combining the photovoltaic and pyroelectric mechanisms. This innovation enables spiking signals with a tenfold increase in responsivity, a dynamic range of 110 dB, and an extended spectral response from ultraviolet (UV) to near-infrared (NIR). With a thin-film sensor array, these spiking signals accurately extract fingerprint edge features even under low-light conditions, benefiting from high sensitivity to minor luminance variations. Additionally, the sensors' broadband spiking response captures richer information, achieving 99.25% accuracy in multispectral dynamic gesture recognition while reducing data processing by over 65%. This approach effectively eliminates redundant data while minimizing information loss, offering a promising alternative to current dynamic perception technologies.

Keywords

event-based vision / in-sensor processing / motion detection / multispectral perception

References

[1] Gehrig D, Scaramuzza D. Low-latency automotive vision with event cameras. Nature. 2024; 629(8014): 1034-1040.

[2] Yang ZY, Wang TY, Lin YH, et al. A vision chip with complementary pathways for open-world sensing. Nature. 2024; 629(8014): 1027-1033.

[3] Posch C, Serrano-Gotarredona T, Linares-Barranco B, Delbruck T. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc IEEE. 2014; 102(10): 1470-1484.

[4] Lichtsteiner P, Posch C, Delbruck T. A 128 × 128 120 dB 15 μs latency asynchronous temporal contrast vision sensor. IEEE J Solid State Circuits. 2008; 43(2): 566-576.

[5] Leñero-Bardallo JA, Serrano-Gotarredona T, Linares-Barranco B. A 3.6 μs latency asynchronous frame-free event-driven dynamic-vision-sensor. IEEE J Solid State Circuits. 2011; 46(6): 1443-1455.

[6] Brückerhoff-Plückelmann F, Bente I, Becker M, et al. Event-driven adaptive optical neural network. Sci Adv. 2023; 9: 9127.

[7] Chen L, Ren M, Zhou JX, et al. Bioinspired iontronic synapse fibers for ultralow-power multiplexing neuromorphic sensorimotor textiles. Proc Natl Acad Sci U S A. 2024; 121(33): e2407971121.

[8] Zhang C, Wang CY, Li C, et al. Topological hydrogen-bonded organic frameworks (HOFs) and their electronic applications in sensor, memristor, and neuromorphic computing. Appl Phys Rev. 2024; 11(3): 11.

[9] Li Y, Qiu ZC, Kan H, et al. A human-computer interaction strategy for an FPGA platform boosted integrated “perception-memory” system based on electronic tattoos and memristors. Adv Sci. 2024; 11: 2402582.

[10] Jang H, Hinton H, Jung WB, et al. In-sensor optoelectronic computing using electrostatically doped silicon. Nat Electron. 2022; 5(8): 519-525.

[11] Mennel L, Symonowicz J, Wachter S, Polyushkin DK, Molina-Mendoza AJ, Mueller T. Ultrafast machine vision with 2D material neural network image sensors. Nature. 2020; 579(7797): 62-66.

[12] Zhou Y, Fu JW, Wan TQ, et al. A 2T2R1C vision cell with 140 dB dynamic range and event-driven characteristics for in-sensor spiking neural network. Paper presented at: International Electron Devices Meeting, 2022.

[13] Zhou Y, Fu JW, Chen ZR, et al. Computational event-driven vision sensors for in-sensor spiking neural networks. Nat Electron. 2023; 6(11): 870-878.

[14] Wu Y, Deng WJ, Li KX, et al. A spiking artificial vision architecture based on fully emulating the human vision. Adv Mater. 2024; 36: 2312094.

[15] Labram JG. Operating principles of zero-bias retinomorphic sensors. J Phys D Appl Phys. 2023; 56(6): 065105.

[16] Trujillo Herrera C, Labram JG. A perovskite retinomorphic sensor. Appl Phys Lett. 2020; 117(23): 233501.

[17] Wu SE, Zeng LH, Zhai YC, et al. Retinomorphic motion detector fabricated with organic infrared semiconductors. Adv Sci. 2023; 10: 2304688.

[18] Wu Y, Deng WJ, Chen XQ, Li JJ, Li SY, Zhang YZ. CMOS-compatible retinomorphic Si photodetector for motion detection. Sci China Inf Sci. 2023; 66: 162401.

[19] Ouyang BS, Wang JL, Zeng G, et al. Bioinspired in-sensor spectral adaptation for perceiving spectrally distinctive features. Nat Electron. 2024; 7(8): 1-9.

[20] Zhang ZF, Zhao XL, Zhang XM, et al. In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array. Nat Commun. 2022; 13(1): 6590.

[21] Bhatnagar P, Patel M, Lee K, Kim J. Self-powered transparent photodetector for subretinal visual functions of wide-field-of-view and broadband perception. InfoMat. 2023; 5(6): e12408.

[22] Yang WH, Kan H, Shen GZ, Li Y. A network intrusion detection system with broadband WO_3−x/WO_3−x-Ag/WO_3-x optoelectronic memristor. Adv Funct Mater. 2024; 34(23): 2312885.

[23] Guan XW, Yu XC, Periyanagounder D, et al. Recent progress in short-to long-wave infrared photodetection using 2D materials and heterostructures. Adv Opt Mater. 2021; 9: 2001708.

[24] Boruah BD. Zinc oxide ultraviolet photodetectors: rapid progress from conventional to self-powered photodetectors. Nanoscale Adv. 2019; 1(6): 2059-2085.

[25] You DT, Xu CX, Zhang W, Zhao J, Qin FF, Shi ZL. Photovoltaic-pyroelectric effect coupled broadband photodetector in self-powered ZnO/ZnTe core/shell nanorod arrays. Nano Energy. 2019; 62: 310-318.

[26] Özgür Ü, Hofstetter D, Morkoc H. ZnO devices and applications: a review of current status and future prospects. P IEEE. 2010; 98(7): 1255-1268.

[27] Kumar M, Patel M, Nguyen TT, Kim J, Yi J. High-performing ultrafast transparent photodetector governed by the pyro-phototronic effect. Nanoscale. 2018; 10(15): 6928-6935.

[28] Wang ZN, Yu RM, Wen XN, et al. Optimizing performance of silicon-based p-n junction photodetectors by the piezo-phototronic effect. ACS Nano. 2014; 8(12): 12866-12873.

[29] Jiang J, Zhang LF, Ming C, et al. Giant pyroelectricity in nanomembranes. Nature. 2022; 607(7919): 480-485.

[30] Wang ZN, Yu RM, Pan CF, et al. Light-induced pyroelectric effect as an effective approach for ultrafast ultraviolet nanosensing. Nat Commun. 2015; 6(1): 1-7.

[31] Zhang D, Wu HT, Bowen CR, Yang Y. Recent advances in pyroelectric materials and applications. Small. 2021; 17(51): 2103960.

[32] Sun B, Chen YZ, Xiao M, et al. A unified capacitive-coupled memristive model for the nonpinched current-voltage hysteresis loop. Nano Lett. 2019; 19(9): 6461-6465.

[33] Wang BH, Huang W, Chi LF, Al-Hashimi M, Marks TJ, Facchetti A. High-k gate dielectrics for emerging flexible and stretchable electronics. Chem Rev. 2018; 118(11): 5690-5754.

[34] Houssa M, Pantisano L, Ragnarsson L-Å, et al. Electrical properties of high-κ gate dielectrics: challenges, current issues, and possible solutions. Mater Sci Eng R. 2006; 51(4-6): 37-85.

[35] Cadena C, Carlone L, Carrillo H, et al. Past, present, and future of simultaneous localization and mapping: toward the robust-perception age. IEEE T Robot. 2016; 32(6): 1309-1332.

[36] Gallego G, Delbrück T, Orchard G, et al. Event-based vision: a survey. IEEE T Pattern Anal. 2020; 44(1): 154-180.

[37] Trujillo Herrera C, Labram JG. An organic retinomorphic sensor. ACS Appl Electron Mater. 2021; 4(1): 92-98.

[38] Zhang XQ, Labram JG. Role of blend ratio in bulk heterojunction organic retinomorphic sensors. J Mater Chem C. 2022; 10(36): 12998-13004.

[39] Berry MJ, Brivanlou IH, Jordan TA, Meister M. Anticipation of moving stimuli by the retina. Nature. 1999; 398(6725): 334-338.

[40] Yang YK, Pan C, Li YX, et al. In-sensor dynamic computing for intelligent machine vision. Nat Electron. 2024; 7(3): 225-233.

[41] Shannon CE. A mathematical theory of communication. Bell Syst Tech J. 1948; 27(1): 379-423.

[42] Dong J, Wu G, Yang T, Li Y. The improved image scrambling algorithm for the wireless image transmission systems of UAVs. Sensors. 2018; 18(10): 3430.

[43] Camuñas-Mesa LA, Serrano-Gotarredona T, Linares-Barranco B. Event-driven sensing and processing for high-speed robotic vision. Paper presented at: IEEE Biomedical Circuits and Systems Conference, 2014.

[44] Fang W, Yu ZF, Chen YQ, Masquelier T, Huang TJ, Tian YH. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. Paper presented at: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

[45] Wang XT, Zhang YX, Zhang YZ. MT-SNN: Enhance Spiking Neural Network With Multiple Thresholds. arXiv preprint arXiv:2303.11127. 2023.

RIGHTS & PERMISSIONS

2025 The Author(s). InfoMat published by UESTC and John Wiley & Sons Australia, Ltd.