MCRNet: Underwater image enhancement using multi-color space residual network

Abstract

The selective attenuation and scattering of light in underwater environments cause color distortion and contrast reduction in underwater images, which can impede the ever-growing demand for underwater robot operations. To address these issues, we propose a Multi-Color space Residual Network (MCRNet) for underwater image enhancement. Our method takes advantage of the unique features of color representation in the RGB, HSV, and Lab color spaces. By utilizing the distinct feature representations of images in different color spaces, we can highlight and fuse the most informative features of the three color spaces. Our approach employs a self-attention mechanism in the multi-color space feature fusion module. Extensive experiments demonstrate that our method achieves satisfactory results in color correction and contrast improvement of underwater images, particularly in severely degraded scenes. Consequently, our method outperforms state-of-the-art methods in both subjective visual comparison and objective evaluation metrics.

Keywords

Underwater image enhancement / Deep learning / Color correction / Underwater robots

CRediT authorship contribution statement

Ningwei Qin: Writing - review & editing, Writing - original draft, Methodology, Formal analysis. Junjun Wu: Supervision, Project administration. Xilin Liu: Writing - review & editing, Validation, Methodology. Zeqin Lin: Supervision, Conceptualization. Zhifeng Wang: Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Key R&D Program of China (2022YFB4702300), in part by the National Natural Science Foundation of China(62273097), in part by the Guangdong Basic and Applied Basic Research Foundation, China (2022A1515140044, 2019A1515110304, 2020A1515110255, and 2021B1515120017), in part by the Research Foundation of Universities of Guangdong Province, China (2019KZDZX1026, 2020KCXTD015, and 2021KCXTD083), in part by the Foshan Key Area Technology Research Foundation, China (2120001011009), and in part by the Guangdong Philosophy and Social Science Program, China (GD23XTS03).

References

[1] W.-H. Lin, J.-X. Zhong, S. Liu, T. Li, G. Li,RoIMix: proposal-fusion among multiple images for underwater object detection, in: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, IEEE, 2020, pp. 2588-2592.

[2] X. Liang, P. Song, Excavating roi attention for underwater object detection, in: 2022 IEEE International Conference on Image Processing, ICIP, IEEE, 2022, pp. 2651-2655.

[3] N. Nezla, T.M. Haridas, M. Supriya, Semantic segmentation of underwater images using unet architecture based deep convolutional encoder decoder model, in: 2021 7th International Conference on Advanced Computing and Communication Systems, ICACCS, vol. 1, IEEE, 2021, pp. 28-33.

[4] S. Mittal, S. Srivastava, J.P. Jayanth, A survey of deep learning techniques for underwater image classification, IEEE Trans. Neural Netw. Learn. Syst.(2022).

[5] X. Liu, S. Wen, Z. Pan, C. Xu, J. Hu, H. Meng, Vision-IMU multi-sensor fusion semantic topological map based on ratslam, Measurement 220 (2023) 113335.

[6] S. Wen, Z. Zhang, C. Ma, Y. Wang, H. Wang, An extended Kalman filter-simultaneous localization and mapping method with Harris-scale-invariant feature transform feature recognition and laser mapping for humanoid robot navigation in unknown environment, Int. J. Adv. Robot. Syst. 14 (6)(2017) 1729881417744747.

[7] R. Liu, Z. Jiang, S. Yang, X. Fan, Twin adversarial contrastive learning for underwater image enhancement and beyond, IEEE Trans. Image Process. 31 (2022) 4922-4936.

[8] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, R. Timofte, Swinir: Image restoration using swin transformer,in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1833-1844.

[9] D. Song, Y. Wang, H. Chen, C. Xu, C. Xu, D. Tao, Addersr: Towards energy ef-ficient image super-resolution,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15648-15657.

[10] S.W. Zamir, A. Arora, S. Khan, M. Hayat, F.S. Khan, M.-H. Yang, L. Shao,Multi-stage progressive image restoration, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14821-14831.

[11] S.W. Zamir, A. Arora, S. Khan, M. Hayat, F.S. Khan, M.-H. Yang, Restormer: Efficient transformer for high-resolution image restoration,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5728-5739.

[12] S. Raveendran, M.D. Patil, G.K. Birajdar, Underwater image enhancement: A comprehensive review, recent trends, challenges and applications, Artif. Intell. Rev. 54 (7) (2021) 5413-5467.

[13] K. He, J. Sun, X. Tang, Single image haze removal using dark channel prior, IEEE Trans. Pattern Anal. Mach. Intell. 33 (12) (2010) 2341-2353.

[14] P. Drews, E. Nascimento, F. Moraes, S. Botelho, M. Campos,Transmis-sion estimation in underwater single images, in:Proceedings of the IEEE International Conference on Computer Vision Workshops, 2013, pp. 825-830.

[15] A. Galdran, D. Pardo, A. Picón, A. Alvarez-Gila, Automatic red-channel underwater image restoration J. Vis. Commun. Image Represent. 26 (2015) 132-145.

[16] Y.-T. Peng, P.C. Cosman, Underwater image restoration based on image blurriness and light absorption, IEEE Trans. Image Process. 26 (4) (2017) 1579-1594.

[17] D. Akkaynak, T. Treibitz, Sea-thru: A method for removing water from un-derwater images,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1682-1691.

[18] W. Song, Y. Wang, D. Huang, D. Tjondronegoro, A rapid scene depth esti-mation model based on underwater light attenuation prior for underwater image restoration, in: Pacific Rim Conference on Multimedia, Springer, 2018, pp. 678-688.

[19] J. Zhou, T. Yang, W. Chu, W. Zhang, Underwater image restoration via backscatter pixel prior and color compensation, Eng. Appl. Artif. Intell. 111 (2022) 104785.

[20] C. Ancuti, C.O. Ancuti, T. Haber, P. Bekaert, Enhancing underwater images and videos by fusion, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2012, pp. 81-88.

[21] C.O. Ancuti, C. Ancuti, C. De Vleeschouwer, P. Bekaert, Color balance and fusion for underwater image enhancement, IEEE Trans. Image Process. 27(1)(2017) 379-393.

[22] X. Fu, P. Zhuang, Y. Huang, Y. Liao, X.-P. Zhang, X. Ding, A retinex-based en-hancing approach for single underwater image, in: 2014 IEEE International Conference on Image Processing, ICIP, IEEE, 2014, pp. 4572-4576.

[23] P. Zhuang, C. Li, J. Wu, Bayesian retinex underwater image enhancement Eng. Appl. Artif. Intell. 101 (2021) 104171.

[24] C.-Y. Li, J.-C. Guo, R.-M. Cong, Y.-W. Pang, B. Wang, Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior, IEEE Trans. Image Process. 25 (12) (2016) 5664-5677.

[25] W. Zhang, P. Zhuang, H.-H. Sun, G. Li, S. Kwong, C. Li, Underwater image enhancement via minimal color loss and locally adaptive contrast enhancement, IEEE Trans. Image Process. 31 (2022) 3997-4010.

[26] J. Wu, X. Liu, Q. Lu, Z. Lin, N. Qin, Q. Shi, FW-GAN: Underwater image enhancement using generative adversarial network with multi-scale fusion, Signal Process., Image Commun. 109 (2022) 116855.

[27] Q. Jiang, Y. Zhang, F. Bao, X. Zhao, C. Zhang, P. Liu, Two-step domain adaptation for underwater image enhancement, Pattern Recognit. 122 (2022) 108324.

[28] H. Li, P. Zhuang, Dewaternet: A fusion adversarial real underwater image enhancement network, Signal Process., Image Commun. 95 (2021) 116248.

[29] C. Fabbri, M.J. Islam, J. Sattar, Enhancing underwater imagery using generative adversarial networks, in: 2018 IEEE International Conference on Robotics and Automation, ICRA, IEEE, 2018, pp. 7159-7165.

[30] C. Li, C. Guo, W. Ren, R. Cong, J. Hou, S. Kwong, D. Tao, An underwater image enhancement benchmark dataset and beyond, IEEE Trans. Image Process. 29 (2019) 4376-4389.

[31] C. Li, S. Anwar, F. Porikli, Underwater scene prior inspired deep underwater image and video enhancement, Pattern Recognit. 98 (2020) 107038.

[32] Z. Shen, H. Xu, T. Luo, Y. Song, Z. He, UDAformer: Underwater image enhancement based on dual attention transformer, Comput. Graph. (2023).

[33] L. Peng, C. Zhu, L. Bian, U-shape transformer for underwater image enhancement, 2021, arXiv preprint arXiv:2111.11843.

[34] K. Iqbal, M. Odetayo, A. James, R.A. Salam, A.Z.H. Talib, Enhancing the low quality images using unsupervised colour correction method, in: 2010 IEEE International Conference on Systems, Man and Cybernetics, IEEE, 2010, pp. 1703-1709.

[35] O. Deperlioglu, U. Kose, G.E. Guraksin, Underwater image enhancement with HSV and histogram equalization, Image 1 (4) (2018) 461-465.

[36] J. Hu, Q. Jiang, R. Cong, W. Gao, F. Shao, Two-branch deep neural network for underwater image enhancement in HSV color space, IEEE Signal Process. Lett. 28 (2021) 2152-2156.

[37] Z. Lyu, A. Peng, Q. Wang, D. Ding, An efficient learning-based method for underwater image enhancement, Displays 74 (2022) 102174.

[38] C. Li, S. Anwar, J. Hou, R. Cong, C. Guo, W. Ren, Underwater image enhance-ment via medium transmission-guided multi-color space embedding, IEEE Trans. Image Process. 30 (2021) 4985-5000.

[39] Y.-T. Peng, K. Cao, P.C. Cosman, Generalization of the dark channel prior for single image restoration, IEEE Trans. Image Process. 27 (6) (2018) 2856-2868.

[40] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, Y. Fu, Image super-resolution using very deep residual channel attention networks, in: Proceedings of the European Conference on Computer Vision, ECCV, 2018, pp. 286-301.

[41] X. Li, W. Wang, X. Hu, J. Yang,Selective kernel networks, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 510-519.

[42] K. He, X. Zhang, S. Ren, J. Sun,Deep residual learning for image recognition, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.

[43] B. Lim, S. Son, H. Kim, S. Nah, K. Mu Lee,Enhanced deep residual networks for single image super-resolution, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 136-144.

[44] P. Charbonnier, L. Blanc-Feraud, G. Aubert, M. Barlaud, Two deterministic half-quadratic regularization algorithms for computed imaging, in: Pro-ceedings of 1st International Conference on Image Processing, vol. 2, IEEE, 1994, pp. 168-172.

[45] G. Seif, D. Androutsos, Edge-based loss function for single image super-resolution, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, IEEE, 2018, pp. 1468-1472.

[46] N. Silberman, D. Hoiem, P. Kohli, R. Fergus, Indoor segmentation and sup-port inference from RGBD images, in: European Conference on Computer Vision, Springer, 2012, pp. 746-760.

[47] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, 2014, arXiv preprint arXiv:1409.1556.

[48] M.J. Islam, Y. Xia, J. Sattar, Fast underwater image enhancement for improved visual perception, IEEE Robot. Autom. Lett. 5 (2) (2020) 3227-3234.

[49] Z. Fu, W. Wang, Y. Huang, X. Ding, K.-K. Ma, Uncertainty inspired under-water image enhancement, in: European Conference on Computer Vision, Springer, 2022, pp. 465-482.

[50] C. Guo, R. Wu, X. Jin, L. Han, W. Zhang, Z. Chai, C. Li, Underwater ranker: Learn which is better and how to be better,in: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, (no. 1) 2023, pp. 702-709.

[51] D. Berman, D. Levy, S. Avidan, T. Treibitz, Underwater single image color restoration using haze-lines and a new quantitative dataset, IEEE Trans. Pattern Anal. Mach. Intell. 43 (8) (2020) 2822-2837.

[52] K. Panetta, C. Gao, S. Agaian, Human-visual-system-inspired underwater image quality measures, IEEE J. Ocean. Eng. 41 (3) (2015) 541-551.

[53] M. Yang, A. Sowmya, An underwater color image quality evaluation metric, IEEE Trans. Image Process. 24 (12) (2015) 6062-6071.

[54] Z. Fu, X. Fu, Y. Huang, X. Ding, Twice mixing: A rank learning based quality assessment approach for underwater image enhancement, Signal Process., Image Commun. 102 (2022) 116622.

[55] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process. 13(4) (2004) 600-612.

[56] Z. Chen, T. Jiang, Y. Tian,Quality assessment for comparing image enhance-ment algorithms, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3003-3010.

[57] D.G. Lowe, Distinctive image features from scale-invariant keypoints Int.J. Comput. Vis. 60 (2) (2004) 91-110.

[58] J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal. Mach. Intell. (6) (1986) 679-698.