Generative adversarial mural inpainting algorithm based on structural and texture hybrid enhancement

Abstract

For the existing deep learning image restoration methods, the joint guidance of structure and texture information is not considered, which leads to structural disorder and texture blur in the restoration results. A generative adversarial mural inpainting algorithm based on structural and texture hybrid enhancement was proposed. Firstly, the structure guidance branch composed of dynamic convolution cascade was constructed to improve the expression ability of structure features, and the structure information was used to guide the encoder coding to enhance the edge contour information of the coding feature map. Then, the multi-granularity feature extraction module was designed to obtain the texture features of texture guided branches, and the multi-scale texture information was used to guide the decoder to reconstruct and repair, so as to improve the texture consistency of murals. Finally, skip connection was used to promote the feature sharing of structure and texture features, and the spectral-normalized PatchGAN discriminator was used to complete the mural restoration. The digital restoration experiment results of real Dunhuang murals showed that the proposed method was better than the comparison algorithms in both subjective and objective evaluation, and the restoration results were clearer and more natural.

Keywords

image processing / mural inpainting / structural and texture enhancement / dynamic convolution / multi-granularity feature extraction

References

[1] WANG H, LI Q Q, JIA S. A global and local feature weighted method for ancient murals inpainting. International Journal of Machine Learning and Cybernetics, 2020, 11(6): 1197-1216.

[2] SRIDEVI G, SRINIVAS KUMAR S. Image inpainting based on fractional-order nonlinear diffusion for image reconstruction. Circuits, Systems, and Signal Processing, 2019, 38(8): 3802-3817.

[3] CHEN Y T, ZHANG H P, LIU L W, et al. Research on image inpainting algorithm of improved total variation minimization method. Journal of Ambient Intelligence and Humanized Computing, 2023, 14(5): 5555-5564.

[4] ZHA Z Y, YUAN X, WEN B H, et al. Image restoration using joint patch-group-based sparse representation. IEEE Transactions on Image Processing, 2020, 29: 7735-7750.

[5] QUAN W Z, ZHANG R S, ZHANG Y, et al. Image inpainting with local and global refinement. IEEE Transactions on Image Processing, 2022, 31: 2405-2420.

[6] LAHIRI A, JAIN A K, AGRAWAL S, et al. Prior guided GAN based semantic inpainting//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 13-19, 2020, Seattle, WA, USA. New York: IEEE, 2020: 13693-13702.

[7] WANG N, ZHANG Y P, ZHANG L F. Dynamic selection network for image inpainting. IEEE Transactions on Image Processing, 2021, 30: 1784-1798.

[8] LI J Y, WANG N, ZHANG L F, et al. Recurrent feature reasoning for image inpainting//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 13-19, 2020, Seattle, WA, USA. New York: IEEE, 2020: 7757-7765.

[9] LI J Y, HE F X, ZHANG L F, et al. Progressive reconstruction of visual structure for image inpainting//2019 IEEE/CVF International Conference on Computer Vision, October 27-November 2, 2019, Seoul, Korea. New York: IEEE, 2019: 5961-5970.

[10] LIAO L, XIAO J, WANG Z, et al. Image inpainting guided by coherence priors of semantics and textures//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 20-25, 2021, Nashville, TN, USA. New York: IEEE, 2021: 6535-6544.

[11] NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: generative image inpainting with adversarial edge learning. 2019: 1901.00212.

[12] LIU H Y, JIANG B, SONG Y B, et al. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations//Computer Vision–ECCV 2020: 16th European Conference, August 23–28, 2020, Glasgow, UK, Cham: Springer International Publishing, 2020: 725-741.

[13] YANG J, QI Z Q, SHI Y. Learning to incorporate structure knowledge for image inpainting. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12605-12612.

[14] KIM J, YANG J L, TONG X. Learning high-fidelity face texture completion without complete face texture//2021 IEEE/CVF International Conference on Computer Vision, October 10-17, 2021, Montreal, QC, Canada. New York: IEEE, 2021: 13970-13979.

[15] CHEN Y P, DAI X Y, LIU M C, et al. Dynamic convolution: attention over convolution kernels//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 13-19, 2020. Seattle, WA, USA. New York: IEEE, 2020: 11027-11036.

[16] WANG Z S, WU Y Y, WANG J Y, et al. Res2Fusion: infrared and visible image fusion based on dense Res2net and double nonlocal attention models. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 5005012.

[17] YU J H, LIN Z, YANG J M, et al. Free-form image inpainting with gated convolution//2019 IEEE/CVF International Conference on Computer Vision, October 27-November 2, 2019, Seoul, Korea. New York: IEEE, 2019: 4470-4479.

[18] LI H, HU L, ZHANG J. Irregular mask image inpainting based on progressive generative adversarial networks. The Imaging Science Journal, 2023, 71(3): 299-312.

[19] SONG J, YI H, XU W, et al. ESRGAN-DP: Enhanced super-resolution generative adversarial network with adaptive dual perceptual loss. Heliyon, 2023, 9(4): e15134.

[20] Al NAJJAr Y. Comparative analysis of image quality assessment metrics: MSE, PSNR, SSIM and FSIM. International Journal of Science and Research (IJSR), 2024, 13(3): 110-114.

[21] WANG H, SONG H X, ZHANG Z. Two-stream face image restoration algorithm based on texture and structure. Journal of Test and Measurement Technology, 2024, 38(3): 274-280.