Removing nonrigid refractive distortions for underwater images using an attention-based deep neural network

Abstract

Refractive distortions in underwater images usually occur when these images are captured through a dynamic refractive water surface, such as unmanned aerial vehicles capturing shallow underwater scenes from the surface of water or autonomous underwater vehicles observing floating platforms in the air. We propose an end-to-end deep neural network for learning to restore real scene images for removing refractive distortions. This network adopts an encoder-decoder architecture with a specially designed attention module. The use of the attention image and the distortion field generated by the proposed deep neural network can restore the exact distorted areas in more detail. Qualitative and quantitative experimental results show that the proposed framework effectively eliminates refractive distortions and refines image details. We also test the proposed framework in practical applications by embedding it into an NVIDIA JETSON TX2 platform, and the results demonstrate the practical value of the proposed framework.

Keywords

Underwater image / Refractive distortion / Deep neural network / Attention mechanism

References

[1] Boroujeni S, Razi A. IC-GAN: an improved conditional generative adversarial network for RGB-to-IR image translation with applications to forest fire monitoring. Expert Syst Appl, 2024, 238,

[2] Cap N, Ruiz B, Rabal H. Refraction holodiagrams and Snell’s law. Optik, 2003, 114(2): 89-94,

[3] De Greve B (2006) Reflections and refractions in ray tracing. pp 1–6. Retrieved from https://graphics.stanford.edu/courses/cs148-10-summer/docs/2006--degreve--reflection_refraction.pdf

[4] De Souza VLT, Marques BAD, Batagelo HC, Gois JP. A review on generative adversarial networks for image generation. Comput Graph-UK, 2023, 114: 13-25,

[5] Donate A, Dahme G, Ribeiro E (2006) Classification of textures distorted by waterwaves. In: 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, pp 421–424. https://doi.org/10.1109/ICPR.2006.371

[6] Donate A, Ribeiro E (2007) Improved reconstruction of images distorted by water waves. In: Braz J et al (eds) Advances in computer graphics and computer vision. Springer, Berlin, pp 264–277. https://doi.org/10.1007/978-3-540-75274-5_18

[7] Efros A, Isler V, Shi J, Visontai M (2004) Seeing through water. In: NIPS’04: Proceedings of the 17th International Conference on Neural Information Processing Systems, Vancouver, pp 393–400. https://proceedings.neurips.cc/paper_files/paper/2004

[8] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S et al (2014) Generative adversarial nets. In: NIPS’14: Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, pp 2672–2680. https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

[9] Isola P, Zhu JY, Zhou TH, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, pp 5967–5976. https://doi.org/10.1109/CVPR.2017.632

[10] James JG, Agrawal P, Rajwade A (2019) Restoration of non-rigidly distorted underwater images using a combination of compressive sensing and local polynomial image representations. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, pp 7839–7848. https://doi.org/10.1109/ICCV.2019.00793

[11] Ko K, Yeom T, Lee M. SuperstarGAN: generative adversarial networks for image-to-image translation in large-scale domains. Neural Netw, 2023, 162: 330-339,

[12] Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, pp 105–114. https://doi.org/10.1109/CVPR.2017.19

[13] Levin IM, Savchenko VV, Osadchy VJ. Correction of an image distorted by a wavy water surface: laboratory experiment. Appl Optics, 2008, 47(35): 6650-6655,

[14] Li C, Wand M (2016) Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: 14th European Conference on Computer Vision (ECCV), Amsterdam, pp 702–716. https://doi.org/10.1007/978-3-319-46487-9_43

[15] Li TY, Rong SH, Chen L, Zhou HY, He B. Underwater motion deblurring based on cascaded attention mechanism. IEEE J Ocean Eng, 2022, 49(1): 262-278,

[16] Li TY, Rong SH, Zhao WF, Chen L, Liu YB, Zhou HY, et al.. Underwater image enhancement using adaptive color restoration and dehazing. Opt Express, 2022, 30(4): 6216-6235,

[17] Li TY, Yang QQ, Rong SH, Chen L, He B. Distorted underwater image reconstruction for an autonomous underwater vehicle based on a self-attention generative adversarial network. Appl Optics, 2020, 59(32): 10049-10060,

[18] Li ZQ, Murez Z, Kriegman D, Ramamoorthi R, Chandraker M (2018) Learning to see through turbulent water. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, pp 512–520. https://doi.org/10.1109/WACV.2018.00062

[19] Niu ZY, Zhong GQ, Yu H. A review on the attention mechanism of deep learning. Neurocomputing, 2021, 452: 48-62,

[20] Peng YT, Cosman PC. Underwater image restoration based on image blurriness and light absorption. IEEE Trans Image Proc, 2017, 26(4): 1579-1594,

[21] Rao DY, Xu TY, Wu XJ. TGFuse: an infrared and visible image fusion approach based on transformer and generative adversarial network. IEEE Trans Image Proc, 2023,

[22] Rao YJ, Wu D, Han MA, Wang T, Yang Y, Lei T, et al.. AT-GAN: a generative adversarial network with attention and transition for infrared and visible image fusion. Inf Fusion, 2023, 92: 336-349,

[23] Roy AM, Bhaduri J. DenseSPH-YOLOv5: an automated damage detection model based on DenseNet and Swin-Transformer prediction head-enabled YOLOv5 with attention mechanism. Adv Eng Inform, 2023, 56,

[24] Seemakurthy K, Rajagopalan AN. Deskewing of underwater images. IEEE Trans Image Proc, 2015, 24(3): 1046-1059,

[25] Thapa S, Li N, Ye J (2020) Dynamic fluid surface reconstruction using deep neural network. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, pp 21–30. https://doi.org/10.1109/CVPR42600.2020.00010

[26] Thapa S, Li N, Ye J (2021) Learning to remove refractive distortions from underwater images. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, pp 5007–5016. https://doi.org/10.1109/ICCV48922.2021.00496

[27] Tian YD, Narasimhan SG. Globally optimal estimation of nonrigid image distortion. Int J Comput vis, 2012, 98(3): 279-302,

[28] Tian YD, Narasimhan SG. Theory and practice of hierarchical data-driven descent for optimal deformation estimation. Int J Comput vis, 2015, 115(1): 44-67,

[29] Wang XT, Yu K, Wu SX, Gu JJ, Liu YH, Dong C et al (2018) ESRGAN: enhanced super-resolution generative adversarial networks. In: 15th European Conference on Computer Vision (ECCV), Munich, pp 63–79. https://doi.org/10.1007/978-3-030-11021-5_5

[30] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc, 2004, 13(4): 600-612,

[31] Wen ZY, Fraser D, Lambert A, Li HD (2007) Reconstruction of underwater image by bispectrum. In: IEEE International Conference on Image Processing, San Antonio, pp 1–4. https://doi.org/10.1109/ICIP.2007.4379367

[32] Xie Y, Franz E, Chu M, Thuerey N. tempoGAN: a temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Trans Graph, 2018, 37(4): 95,

[33] Xiong J, Heidrich W (2021) In-the-wild single camera 3D reconstruction through moving water surfaces. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, pp 12558–12567. https://doi.org/10.1109/ICCV48922.2021.01233

[34] Zhang Z, Tang YG, Yang K. A two-stage restoration of distorted underwater images using compressive sensing and image registration. Adv Manuf, 2021, 9(2): 273-285,

[35] Zhang Z, Yang X. Reconstruction of distorted underwater images using robust registration. Opt Express, 2019, 27(7): 9996-10008,

[36] Zhong G, Ding W, Chen L, Wang Y, Yu Y. Multi-scale attention generative adversarial network for medical image enhancement. IEEE Trans Emerg Top Comput Intell, 2023, 7(4): 1113-1125,

[37] Zhou J, Gai Q, Zhang D, Lam KM, Zhang W, Fu X. IACC: cross-Illumination awareness and color correction for underwater images under mixed natural and artificial lighting. IEEE Trans Geosci Remote Sens, 2024, 62: 4201115,

[38] Zhou JC, Sun JM, Zhang WS, Lin ZF. Multi-view underwater image enhancement method via embedded fusion mechanism. Eng Appl Artif Intell, 2023, 121,