Low-light enhancement method with dual branch feature fusion and learnable regularized attention

Abstract

Restricted by the lighting conditions, the images captured at night tend to suffer from color aberration, noise, and other unfavorable factors, making it difficult for subsequent vision-based applications. To solve this problem, we propose a two-stage size-controllable low-light enhancement method, named Dual Fusion Enhancement Net (DFEN). The whole algorithm is built on a double U-Net structure, implementing brightness adjustment and detail revision respectively. A dual branch feature fusion module is adopted to enhance its ability of feature extraction and aggregation. We also design a learnable regularized attention module to balance the enhancement effect on different regions. Besides, we introduce a cosine training strategy to smooth the transition of the training target from the brightness adjustment stage to the detail revision stage during the training process. The proposed DFEN is tested on several low-light datasets, and the experimental results demonstrate that the algorithm achieves superior enhancement results with the similar parameters. It is worth noting that the lightest DFEN model reaches 11 FPS for image size of 1224×1024 in an RTX 3090 GPU.

Graphical abstract

Keywords

Power inspection / Low-light enhancement / Feature fusion / Learnable regularized attention

References

[1] Chen, J., Fu, Z., Cheng, X., Wang, F.: An method for power lines insulator defect detection with attention feedback and double spatial pyramid. Electric Power Syst. Res. 218, 109175 (2023)

[2] Tao, X., Zhang, D., Wang, Z., Liu, X., Zhang, H., Xu, D.: Detection of power line insulator defects using aerial images analyzed with convolutional neural networks. IEEE Trans. Syst. Man Cybernetics Syst. 50(4), 1486–1498 (2020)

[3] Wang, S., Zou, X., Zhu, W., Zeng, L.: Insulator defects detection for aerial photography of the power grid using you only look once algorithm. J. Electr. Eng. Technol. 18, 3287–3300 (2023)

[4] Jobson, D.J., Rahman, Z., Woodell, G.A.: Properties and performance of a center/surround retinex. IEEE Trans. Image Process. 6, 451–462 (1997)

[5] Rahman, Z., Jobson, D.J., Woodell, G.A.: Multi-scale retinex for color image enhancement. In: Proceedings of 3r. IEEE International Conference on Image Processing. pp. 1003–1006. IEEE, Lausanne, Switzerland (1996)

[6] Jobson, D.J., Rahman, Z., Woodell, G.A.: A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process. 6, 965–976 (1997)

[7] Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2017)

[8] Ying, Z., Li, G., Gao, W.: A bio-inspired multi-exposure fusion framework for low-light image enhancement (2017) arxiv.org/abs/1711.00591

[9] Ying, Z., Li, G., Ren, Y., Wang, R., Wang, W.: A new low-light image enhancement algorithm using camera response model. In: IEEE International Conference on Computer Vision Workshops (ICCVW). pp. 3015–3022. IEEE, Venice, Italy (2017)

[10] Dong, X., Wang, G., Pang, Y., Li, W., Wen, J., Meng, W., Lu, Y.: Fast efficient algorithm for enhancement of low lighting video. In: IEEE International Conference on Multimedia and Expo. pp. 1–6. IEEE, Barcelona, Spain (2011)

[11] Shen, L., Yue, Z., Feng, F., Chen, Q., Liu, S., Ma, J.: MSR-net: low-light image enhancement using deep convolutional network (2017) arxiv.org/abs/1711.02488

[12] Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. In: The British Machine Vision Conference. British Machine Vision Association, Newcastle (2018)

[13] Lore, K.G., Akintayo, A., Sarkar, S.: LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recogn. 61, 650–662 (2017)

[14] Wang, W., Wei, C., Yang, W., Liu, J.: GLADNet: low-light enhancement network with global awareness. In: 13th IEEE International Conference on Automatic Face and Gesture Recognition. pp. 751–755. IEEE, Xi’an (2018)

[15] Zeng, H., Cai, J., Li, L., Cao, Z., Zhang, L.: Learning imageadaptive 3D lookup tables for high performance photo enhancement in real-time. IEEE Trans. Pattern Analysis Mach. Intell. 44, 2058–2073 (2022)

[16] Guo, C., Li, C., Guo, J., Loy, C.C., Hou, J., Kwong, S., Cong, R.: Zero-reference deep curve estimation for low-light image enhancement. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1777–1786. IEEE, Seattle (2020)

[17] Jiang, Y., Gong, X., Liu, D., Cheng, Y., Fang, C., Shen, X., Yang, J., Zhou, P., Wang, Z.: EnlightenGAN: deep light enhancement without paired supervision. IEEE Trans. Image Process. 30, 2340–2349 (2021)

[18] Ma, L., Ma, T., Liu, R., Fan, X., Luo, Z.: Toward fast, flexible, and robust low-light image enhancement. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5627–5636. IEEE, New Orleans (2022)

[19] Zhang, Y., Zhang, J., Guo, X.: Kindling the darkness: a practical low-light image enhancer. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 1632–1640. ACM, Nice (2019)

[20] Zhang, Y., Guo, X., Ma, J., Liu, W., Zhang, J.: Beyond brightening low-light images. Int. J. Comput. Vis. 129, 1013–1037 (2021)

[21] Chen, L., Guo, L., Cheng, D., Kou, Q.: Structure-preserving and color-restoring up-sampling for single low-light image. IEEE Trans. Circuits Syst. Video Technol. 32(4), 1889–1902 (2022)

[22] Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3291–3300. IEEE, Salt Lake City (2018)

[23] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Learning enriched features for real image restoration and enhancement. In: 2020 European Conference on Computer Vision (ECCV). pp. 492–511. Springer International Publishing (2020)

[24] Wang, Y., Wan, R., Yang, W., Li, H., Chau, L.P., Kot, A.: Low-light image enhancement with normalizing flow. Proc. AAAI Conference Artif. Intell. 36, 2604–2612 (2022)

[25] Fu, H., Zheng, W., Meng, X., Wang, X., Wang, C., Ma, H.: You do not need additional priors or regularizers in retinex-based low-light image enhancement. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 18125–18134. IEEE, Vancouver (2023)

[26] Cai, Y., Bian, H., Lin, J., Wang, H., Timofte, R., Zhang, Y.: Retinexformer: one-stage retinex-based transformer for low-light image enhancement. In: IEEE/CVF International Conference on Computer Vision (ICCV). pp. 12470–12479. IEEE, Paris (2023)

[27] Yang, Q., Wu, Y., Cao, D., Luo, M., Wei, T.: A lowlight image enhancement method learning from both paired and unpaired data by adversarial training. Neurocomputing 433, 83–95 (2021)

[28] Nakamura, J. (ed.): Image sensors and signal processing for digital still cameras. Taylor & Francis, Boca Raton (2006)

[29] Yin, Y., Xu, D., Tan, C., Liu, P., Zhao, Y., Wei, Y.: CLE Diffusion: controllable light enhancement diffusion model. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 8145–8156. ACM, Ottawa (2023)

[30] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7132–7141. Salt Lake City (2018)

[31] Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., and Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention. pp. 234–241. Springer International Publishing, Munich (2015)

[32] Zheng, C., Cao, D., Hu, C.: A similarity-guided segmentation model for garbage detection under road scene. Front. Optoelectron. 15(22), 1–17 (2022)

[33] Zhang, Q., Yang, Y.: SA-net: shuffle attention for deep convolutional neural networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 2235–2239. IEEE, Toronto (2021)

[34] Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: Asilomar Conference on Signals, Systems and Computers. pp. 1398–1402. IEEE, Pacific Grove (2003)

[35] Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2414–2423. IEEE, Las Vegas (2016)

[36] Yang, W., Wang, W., Huang, H., Wang, S., Liu, J.: Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Trans. Image Process. 30, 2072–2086 (2021)

[37] Cai, J., Gu, S., Zhang, L.: Learning a deep single image contrast enhancer from multi-exposure images. IEEE Trans. Image Process. 27, 2049–2062 (2018)

[38] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization (2019)

[39] Li, C., Guo, C., Chen, C.L.: Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Trans. Pattern Analysis Mach. Intell. 44(8), 4225–4237 (2022)

[40] Wang, S., Zheng, J., Hu, H.M., Li, B.: Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Trans. Image Process. 22(9), 3538–3548 (2013)

RIGHTS & PERMISSIONS

2024 The Author(s) 2024