A super-resolution reconstruction algorithm for mural images based on improved generative adversarial network

Abstract

In order to solve the problem of the lack of ornamental value and research value of ancient mural paintings due to low resolution and fuzzy texture details, a super resolution (SR) method based on generative adduction network (GAN) was proposed. This method reconstructed the detail texture of mural image better. Firstly, in view of the insufficient utilization of shallow image features, information distillation blocks (IDB) were introduced to extract shallow image features and enhance the output results of the network behind. Secondly, residual dense blocks with residual scaling and feature fusion (RRDB-Fs) were used to extract deep image features, which removed the BN layer in the residual block that affected the quality of image generation, and improved the training speed of the network. Furthermore, local feature fusion and global feature fusion were applied in the generation network, and the features of different levels were merged together adaptively, so that the reconstructed image contained rich details. Finally, in calculating the perceptual loss, the brightness consistency between the reconstructed fresco and the original fresco was enhanced by using the features before activation, while avoiding artificial interference. The experimental results showed that the peak signal-to-noise ratio and structural similarity metrics were improved compared with other algorithms, with an improvement of 0.512 dB-3.016 dB in peak signal-to-noise ratio and 0.009-0.089 in structural similarity, and the proposed method had better visual effects.

Keywords

mural image / super-resolution reconstruction / generative adversarial network / information distillation block (IDB) / feature fusion

References

[1] GUO J, WU J, GUO C, et al. Image super-resolution reconstruction based on residual connection convolutional neural network. Journal of Jilin University (Engineering and Technology Edition), 2019, 49(5): 1726-1734.

[2] SU H, ZHOU J, ZHANG Z H. Survey of super-resolution image reconstruction methods. Acta Automatica Sinica, 2014, 39(8): 1202-1213.

[3] HASAN M S, HAQUE S T. Single image super-resolution using back-propagation neural networks//2017 20th International Conference of Computer and Information Technology (ICCIT), December 22-24, 2017, Dhaka, Bangladesh. New York: IEEE, 2017: 1-5.

[4] PENG Y F, GAO Y, DU T T, et al. Single image super- resolution reconstruction method for generative adversarial network. Journal of Frontiers of Computer Science and Technology, 2020, 14(9): 1612-1620.

[5] WANG Z W, LIU D, YANG J C, et al. Deep networks for image super-resolution with sparse prior//2015 IEEE International Conference on Computer Vision (ICCV), December 7-13, 2015, Santiago, Chile. New York: IEEE, 2015: 370-378.

[6] DONG C, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.

[7] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA. New York: IEEE, 2016: 1646-1654.

[8] KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 27-30, 2016, Las Vegas, NV, USA. New York: IEEE, 2016: 1637-1645.

[9] LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 2017: 5835-5843.

[10] HU S Y, WANG G D, ZHAO Y, et al. Image super-resolution network based on dense connection and squeeze module. Laser & Optoelectronics Progress, 2019, 56(20): 201005.

[11] ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(7): 2480-2495.

[12] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks. Communications of the ACM, 2020, 63(11): 139-144.

[13] LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 2017: 105-114.

[14] HU X Y, LIU X J, WANG Z C, et al. RTSRGAN: real-time super-resolution generative adversarial networks//2019 Seventh International Conference on Advanced Cloud and Big Data (CBD), September 21-22, 2019, Suzhou, China. New York: IEEE, 2019: 321-326.

[15] CHEN Z H, HU H L, YAO J M, et al. Single frame image super-resolution reconstruction based on improved generative adversarial network. Chinese Journal of Liquid Crystal and Displays, 2021, 36(5): 705-712.

[16] LI Y H, MU X, ZHU Y L, et al. Super resolution image restoration and reconstruction of deep generative countermeasure network. Journal of Xi'an University of Technology, 2021, 35(4): 1-8.

[17] LI X X, CAO Q, LIU C M. Image super-resolution based on no match generative adversarial network. Journal of Zhengzhou University (Engineering Science), 2021, 42(5): 1-6.

[18] DUAN Y X, ZHANG H X, SUN Q F, et al. Image super-resolution reconstruction algorithm based on laplacian pyramid generative adversarial network. Journal of Computer Applications, 2021, 41(4): 1020-1026.

[19] HUI Z, WANG X M, GAO X B. Fast and accurate single image super-resolution via information distillation network//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 18-23, 2018, Salt Lake City, UT, USA. New York: IEEE, 2018: 723-731.

[20] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27-30, 2016, Las Vegas, NV, USA. New York: IEEE, 2016: 770-778.