SR-AFU: super-resolution network using adaptive frequency component upsampling and multi-resolution features

Abstract

Image super-resolution (SR) is one of the classic computer vision tasks. This paper proposes a super-resolution network based on adaptive frequency component upsampling, named SR-AFU. The network is composed of multiple cascaded dilated convolution residual blocks (CDCRB) to extract multi-resolution features representing image semantics, and multiple multi-size convolutional upsampling blocks (MCUB) to adaptively upsample different frequency components using CDCRB features. The paper also defines a new loss function based on the discrete wavelet transform, making the reconstructed SR images closer to human perception. Experiments on the benchmark datasets show that SR-AFU has higher peak signal to noise ratio (PSNR), significantly faster training speed and more realistic visual effects compared with the existing methods.

Graphical abstract

Keywords

super-resolution / multi-resolution features / adaptive frequency upsampling / wavelet transformation

References

[1] Freeman W T , Pasztor E C , Carmichael O T . Learning low-level vision. International Journal of Computer Vision, 2000, 40( 1): 25– 47

[2] ShiW, CaballeroJ, LedigC, ZhuangX, BaiW, BhatiaK, deMarvao A M S M, DawesT, O’ReganD, RueckertD. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention. 2013, 9−16

[3] Zhang S , Liang G , Pan S , Zheng L . A fast medical image super resolution method based on deep learning network. IEEE Access, 2018, 7 : 12319– 12327

[4] Oh J , Lee C , Seo D C . Automated HRSI georegistration using orthoimage and SRTM: focusing KOMPSAT-2 imagery. Computers & Geosciences, 2013, 52 : 77– 84

[5] Nogueira K , Penatti O A B , Dos Santos J A . Towards better exploiting convolutional neural networks for remote sensing scene classification. Pattern Recognition, 2017, 61 : 539– 556

[6] Hu X , Ma P , Mai Z , Peng S , Yang Z , Wang L . Face hallucination from low quality images using definition-scalable inference. Pattern Recognition, 2019, 94 : 110– 121

[7] ChenL C, ZhuY, PapandreouG, SchroffF, AdamH. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 833−851

[8] LiP, WangQ, ZuoW, ZhangL. Log-Euclidean kernels for sparse representation and dictionary learning. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1601−1608

[9] LeeY, ChoeY. Neighbor embedding based single image super-resolution using hybrid feature and adaptive weight decay regularization. In: Proceedings of the 4th IEEE International Conference on Consumer Electronics Berlin. 2014, 185−187

[10] Dong C , Loy C C , He K , Tang X . Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38( 2): 295– 307

[11] KimJ, LeeJ K, LeeK M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1646−1654

[12] KimJ, LeeJ K, LeeK M. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1637−1645

[13] ZhangY, TianY, KongY, ZhongB, FuY. Residual dense network for image super-resolution. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 2472−2481

[14] ZhangY, LiK, LiK, WangL, ZhongB, FuY. Image super-resolution using very deep residual channel attention networks. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 294−310

[15] RadM S, BozorgtabarB, MartiU V, BaslerM, EkenelH K, ThiranJ P. SROBB: targeted perceptual loss for single image super-resolution. In: Proceedings of IEEE/CVF International Conference on Computer Vision. 2019, 2710−2719

[16] Ng M K , Shen H , Lam E Y , Zhang L . A total variation regularization based super-resolution reconstruction algorithm for digital video. EURASIP Journal on Advances in Signal Processing, 2007, 2007 : 074585–

[17] Jiang K , Wang Z , Yi P , Jiang J . Hierarchical dense recursive network for image super-resolution. Pattern Recognition, 2020, 107 : 107475–

[18] DongC, LoyC C, TangX. Accelerating the super-resolution convolutional neural network. In: Proceedings of the 14th European Conference on Computer Vision. 2016, 391−407

[19] HeK, ZhangX, RenS, SunJ. Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770−778

[20] LaiW S, HuangJ B, AhujaN, YangM H. Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5835−5843

[21] LedigC, TheisL, HuszárF, CaballeroJ, CunninghamA, AcostaA, AitkenA, TejaniA, TotzJ, WangZ, ShiW. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 105−114

[22] WangX, YuK, WuS, GuJ, LiuY, DongC, QiaoY, LoyC C. ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018, 63−79

[23] LimB, SonS, KimH, NahS, LeeK M. Enhanced deep residual networks for single image super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017, 1132−1140

[24] YuJ, FanY, YangJ, XuN, WangZ, WangX, HuangT. Wide activation for efficient and accurate image super-resolution. 2018, arXiv preprint arXiv: 1808.08718

[25] DaiT, CaiJ, ZhangY, XiaS T, ZhangL. Second-order attention network for single image super-resolution. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 11057−11066

[26] ZhaoH, GalloO, FrosioI, KautzJ. Loss functions for neural networks for image processing. 2018, arXiv preprint arXiv: 1511.08861

[27] TimofteR, AgustssonE, Van GoolL, YangM H, ZhangL. NTIRE 2017 challenge on single image super-resolution: methods and results. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017, 1110−1121

[28] BevilacquaM, RoumyA, GuillemotC, MorelM L A. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of British Machine Vision Conference. 2012

[29] ZeydeR, EladM, ProtterM. On single image scale-up using sparse-representations. In: Proceedings of the 7th International Conference on Curves and Surfaces. 2010, 711−730

[30] MartinD, FowlkesC, TalD, MalikJ. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the 8th IEEE International Conference on Computer Vision. 2001, 416−423

[31] HuangJ B, SinghA, AhujaN. Single image super-resolution from transformed self-exemplars. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 5197−5206

[32] KingmaD P, BaJ. Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference for Learning Representations. 2015

[33] Wang Z , Bovik A C , Sheikh H R , Simoncelli E P . Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13( 4): 600– 612

[34] DongC, LoyC C, HeK, TangX. Learning a deep convolutional network for image super-resolution. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 184−199

[35] TaiY, YangJ, LiuX, XuC. MemNet: a persistent memory network for image restoration. In: Proceedings of IEEE International Conference on Computer Vision. 2017, 4549−4557

[36] HarisM, ShakhnarovichG, UkitaN. HarisM, ShakhnarovichG, UkitaN. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 1664−1673

[37] LiZ, YangJ, LiuZ, YangX, JeonG, WuW. Feedback network for image super-resolution. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 3862−3871

[38] HuiZ, GaoX, YangY, WangX. Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM International Conference on Multimedia. 2019, 2024−2032

[39] ZhaoH, KongX, HeJ, QiaoY, DongC. Efficient image super-resolution using pixel attention. In: Proceedings of the European Conference on Computer Vision. 2020, 56−72

[40] WangC, LiZ, ShiJ. Lightweight image super-resolution with adaptive weighted learning network. 2019, arXiv preprint arXiv: 1904.02358

RIGHTS & PERMISSIONS

Higher Education Press 2021