Unsupervised lightweight 3D convolutional network for enhanced infrared imaging in wearable devices

Abstract

With the increasing frequency of natural disasters and health emergencies, wearable infrared thermal imaging devices are becoming more prevalent in fire protection and medical fields. However, these devices often face imaging performance challenges such as insufficient contrast, dark areas and blurred edges, which significantly limit their practical effectiveness. To tackle these challenges, we propose a novel unsupervised lightweight 3D convolutional network (UL3DCN) specifically designed for enhancing infrared images on wearable devices. In this framework, the task of infrared image enhancement is conceptualized as generating high dynamic range infrared images from the corresponding temperature sequences during thermal equilibrium. To achieve this, we first design a learnable dynamic filtering module tailored for simulating a series of infrared image sequences under varying temperature differences. This module extends a single image from the spatial domain into the spatio-temporal domain. Subsequently, we employ a lightweight 3D convolution module to effectively extract spatio-temporal information from the image sequence. Finally, inspired by Zero-DCE, we utilize the extracted information to estimate pixel values and high-order curves, thereby enhancing the infrared images. Comprehensive experimental results demonstrate that our method achieves outstanding performance and real-time capabilities. Additionally, the proposed UL3DCN model has been successfully integrated into a wearable infrared firefighting mask.

Graphical abstract

Keywords

infrared image enhancement / HDR / dynamic filtering / wearable devices

References

[1] Kruse F A, Lefkoff A B, Dietz J B. Expert system-based mineral mapping in northern death valley, California/Nevada, using the airborne visible/infrared imaging spectrometer (AVIRIS). Remote Sensing of Environment, 1993, 44(2−3): 309−336

[2] Siboni S, Shabtai A, Tippenhauer N O, Lee J, Elovici Y . Advanced security testbed framework for wearable IoT devices. ACM Transactions on Internet Technology (TOIT), 2016, 16( 4): 1–25

[3] Guglielmi C L, Banschbach S, Dort J, Ferla B, Simon R, Groah L . Hand-held communication devices: friend or foe?. AORN Journal, 2013, 98( 3): 294–303

[4] Gaber R, Ali A, Ahmed K. Performance evaluation of infrared image enhancement techniques. 2022, arXiv preprint arXiv:2202.03427

[5] Wang Q, Jin P, Wu Y, Zhou L, Shen T . Infrared image enhancement: a review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024, 18: 3281–3299

[6] Lin C L . An approach to adaptive infrared image enhancement for long-range surveillance. Infrared Physics & Technology, 2011, 54( 2): 84–91

[7] Khidse S, Nagori M . A comparative study of image enhancement techniques. International Journal of Computer Applications, 2013, 81( 15): 28–32

[8] Janani V, Dinakaran M. Infrared image enhancement techniques—a review. In: Proceedings of the 2nd International Conference on Current Trends in Engineering and Technology-ICCTET 2014. 2014, 167−173

[9] Singh G, Mittal A . Various image enhancement techniques-a critical review. International Journal of Innovation and Scientific Research, 2014, 10( 2): 267–274

[10] LeCun Y, Bengio Y, Hinton G . Deep learning. Nature, 2015, 521( 7553): 436–444

[11] Hu H, Gu J, Zhang Z, Dai J, Wei Y. Relation networks for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 3588−3597

[12] Zou Z, Chen K, Shi Z, Guo Y, Ye J . Object detection in 20 years: a survey. Proceedings of the IEEE, 2023, 111( 3): 257–276

[13] Chen E, Chen S, Ye T, Liu Y . Degradation-adaptive neural network for jointly single image dehazing and desnowing. Frontiers of Computer Science, 2024, 18(2): 182707

[14] Zhang J, He F, Duan Y, Yang S . AIDEDNet: Anti-interference and detail enhancement dehazing network for real-world scenes. Frontiers of Computer Science, 2023, 17( 2): 172703

[15] Li C, Guo C, Han L, Jiang J, Cheng M M, Gu J, Loy C C . Low-light image and video enhancement using deep learning: a survey. IEEE transactions on pattern analysis and machine intelligence, 2022, 44( 12): 9396–9416

[16] Hu M, Bai L, Fan J, Zhao S, Chen E . Vehicle color recognition based on smooth modulation neural network with multi-scale feature fusion. Frontiers of Computer Science, 2023, 17( 3): 173321

[17] Huang J, Liu B, Miao C, Zhang X, Liu J, Su L, Liu Z, Gu Y . PhyFinAtt: An undetectable attack framework against PHY layer fingerprint-based WiFi authentication. IEEE Transactions on Mobile Computing, 2024, 23( 7): 7753–7770

[18] Zhao S, Tang H, Liu S, Zhang Y, Wang H, Xu T, Chen E, Guan C . ME-PLAN: A deep prototypical learning with local attention network for dynamic micro-expression recognition. Neural Networks, 2022, 153: 427–443

[19] Wang L, Yoon K J . Deep learning for HDR imaging: State-of-the-art and future trends. IEEE transactions on pattern analysis and machine intelligence, 2022, 44( 12): 8874–8895

[20] Kuk J G, Cho N I, Lee S U. High dynamic range (HDR) imaging by gradient domain fusion. In: Proceedings of 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2011, 1461−1464

[21] Guo L, Yi H. Research on multi-sensor high dynamic range imaging technology and application. In: Proceedings of SPIE 11850, First Optics Frontier Conference. 2021, 118500B

[22] Guo C, Li C, Guo J, Loy C C, Hou J, Kwong S, Cong R. Zero-reference deep curve estimation for low-light image enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 1777−1786

[23] Mittal A, Moorthy A K, Bovik A C . No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing, 2012, 21( 12): 4695–4708

[24] Mittal A, Soundararajan R, Bovik A C . Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters, 2013, 20( 3): 209–212

[25] De Brabandere B, Jia X, Tuytelaars T, Van Gool L. Dynamic filter networks. Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 667−675

[26] Li G, Luo W, Li P, Lv H. Fusion enhancement of color image based on global histogram equalization. In: Proceedings of the International Conference on Computer Science and Software Engineering. 2008, 205−208

[27] Cheng H D, Shi X J . A simple and effective histogram equalization approach to image enhancement. Digital Signal Processing, 2004, 14( 2): 158–170

[28] Zhu H, Chan F H Y, Lam F K . Image contrast enhancement by constrained local histogram equalization. Computer Vision and Image Understanding, 1999, 73( 2): 281–290

[29] Stark J A . Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Transactions on Image Processing, 2000, 9( 5): 889–896

[30] Zuiderveld K. Contrast limited adaptive histogram equalization. In: Heckbert P S, ed. Graphics Gems IV. San Diego: Academic Press, 1994, 474−485

[31] Reza A M . Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement. Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology, 2004, 38( 1): 35–44

[32] Jadiya S, Goyal A, Jain V. Independent histogram equalization using optimal threshold for contrast enhancement and brightness preservation. In: Proceedings of the 4th International Conference on Computer and Communication Technology (ICCCT). 2013, 54−59

[33] Ashiba H, Awadalla K, El-Halfawy S, Abd El-Samie F . Homomorphic enhancement of infrared images using the additive wavelet transform. Progress in Electromagnetics Research C, 2008, 1: 123–130

[34] Sheet D, Garud H, Suveer A, Mahadevappa M, Chatterjee J . Brightness preserving dynamic fuzzy histogram equalization. IEEE Transactions on Consumer Electronics, 2010, 56( 4): 2475–2480

[35] D.J. Jobson, Z.Rahman, G. Woodell, Properties and performance of a center/surround retinex, IEEE Trans. Image Process, 1997, 6(3): 451−462.

[36] Z Rahman, D.J Jobson, G.Woodell, Multi-scale retinex for color image enhancement, In: Proceedings IEEE International Conference on Image Processing, 1996, 3: 1003–1006.

[37] Wang C, Gao S, Tang X. The research of infrared image enhancement algorithm based on human vision. In: Proceedings of SPIE 9301, International Symposium on Optoelectronic Technology and Application 2014: Image Processing and Pattern Recognition. 2014, 93013C

[38] Choi Y, Kim N, Hwang S, Kweon I S. Thermal image enhancement using convolutional neural network. In: Proceedings of 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2016, 223−230

[39] Wang Z, Shen H, Men C, Sun Q, Huang K. Thermal infrared image inpainting via edge-aware guidance. In: Proceedings of ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2023, 1−5

[40] Kuang X, Sui X, Liu Y, Chen Q, Gu G. Single infrared image enhancement using a deep convolutional neural network. Neurocomputing, 2019, 332: 119−128

[41] Wang D, Lai R, Guan J . Target attention deep neural network for infrared image enhancement. Infrared Physics & Technology, 2021, 115: 103690

[42] Shen H, Zhao Z Q, Zhang W. Adaptive dynamic filtering network for image denoising. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 2227−2235

[43] Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 7132−7141

[44] Wei C, Wang W, Yang W, Liu J. Deep retinex decomposition for low-light enhancement. In: Proceedings of the British Machine Vision Conference 2018. 2018, 155

[45] Cai J, Gu S, Zhang L . Learning a deep single image contrast enhancer from multi-exposure images. IEEE Transactions on Image Processing, 2018, 27( 4): 2049–2062

[46] Bychkovsky V, Paris S, Chan E, Durand F. Learning photographic global tonal adjustment with a database of input/output image pairs. In: Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition. 2011, 97−104

[47] Zhang Q, Nie Y, Zheng W S . Dual illumination estimation for robust exposure correction. Computer Graphics Forum, 2019, 38( 7): 243–252

[48] Wang R, Zhang Q, Fu C W, Shen X, Zheng W S, Jia J. Underexposed photo enhancement using deep illumination estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 6842−6850

[49] Cui Z, Li K, Gu L, Su S, Gao P, Jiang Z, Qiao Y, Harada T. You only need 90k parameters to adapt light: a light weight transformer for image enhancement and exposure correction. In: Proceedings of the 33rd British Machine Vision Conference 2022. 2022, 238

[50] Ma L, Ma T, Liu R, Fan X, Luo Z. Toward fast, flexible, and robust low-light image enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 5627−5636

[51] Liang J, Cao J, Sun G, Zhang K, Van Gool L, Timofte R. SwinIR: Image restoration using swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, 1833−1844

[52] Jian M, Wu R, Xu W, Zhi H, Tao C, Chen H, Li X . VascuConNet: an enhanced connectivity network for vascular segmentation. Medical & Biological Engineering & Computing, 2024, 62( 11): 3453–3554

RIGHTS & PERMISSIONS

Higher Education Press