SR-AFU: super-resolution network using adaptive frequency component upsampling and multi-resolution features

Abstract

Image super-resolution (SR) is one of the classic computer vision tasks. This paper proposes a super-resolution network based on adaptive frequency component upsampling, named SR-AFU. The network is composed of multiple cascaded dilated convolution residual blocks (CDCRB) to extract multi-resolution features representing image semantics, and multiple multi-size convolutional upsampling blocks (MCUB) to adaptively upsample different frequency components using CDCRB features. The paper also defines a new loss function based on the discrete wavelet transform, making the reconstructed SR images closer to human perception. Experiments on the benchmark datasets show that SR-AFU has higher peak signal to noise ratio (PSNR), significantly faster training speed and more realistic visual effects compared with the existing methods.

Graphical abstract

Keywords

super-resolution / multi-resolution features / adaptive frequency upsampling / wavelet transformation

Ke-Jia Chen is an associate professor in Nanjing University of Posts and Telecommunications, China. She received her PhD in Université de Technologie de Compiègne, France and her master’s degree in Nanjing University, China. She joined Jiangsu Key Laboratory of Big Data Security & Intelligent Processing, China in 2017. Her current research focuses on machine learning and its applications in complex network analysis

Mingyu Wu received the BS degree in Electronic Information Engineering from Nanjing University of Posts and Telecommunications, China in 2021. He is working toward the master degree in Signal and information processing at Nanjing University of Posts and Telecommunications, China. His current research interests include casual inference, sequence modeling and cross-modal analysis

Yibo Zhang received the BS degree in Electronic Information Engineering from Nanjing University of Posts and Telecommunications, China in 2021. His research interests include computer vision, UAV system and automatic driving

Zhiwei Chen received the BS degree in Electronic Information Engineering from Nanjing University of Posts and Telecommunications, China in 2021. He is working toward the master degree in Electronic information at the South China Normal University, China. His current research interests include machine learning, computer vision and Neuromorphic computation

References

[1] Freeman W T , Pasztor E C , Carmichael O T . Learning low-level vision. International Journal of Computer Vision, 2000, 40( 1): 25– 47

[2] ShiW, CaballeroJ, LedigC, ZhuangX, BaiW, BhatiaK, deMarvao A M S M, DawesT, O’ReganD, RueckertD. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention. 2013, 9−16

[3] Zhang S , Liang G , Pan S , Zheng L . A fast medical image super resolution method based on deep learning network. IEEE Access, 2018, 7 : 12319– 12327

[4] Oh J , Lee C , Seo D C . Automated HRSI georegistration using orthoimage and SRTM: focusing KOMPSAT-2 imagery. Computers & Geosciences, 2013, 52 : 77– 84

[5] Nogueira K , Penatti O A B , Dos Santos J A . Towards better exploiting convolutional neural networks for remote sensing scene classification. Pattern Recognition, 2017, 61 : 539– 556

[6] Hu X , Ma P , Mai Z , Peng S , Yang Z , Wang L . Face hallucination from low quality images using definition-scalable inference. Pattern Recognition, 2019, 94 : 110– 121

[7] ChenL C, ZhuY, PapandreouG, SchroffF, AdamH. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 833−851

[8] LiP, WangQ, ZuoW, ZhangL. Log-Euclidean kernels for sparse representation and dictionary learning. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1601−1608

[9] LeeY, ChoeY. Neighbor embedding based single image super-resolution using hybrid feature and adaptive weight decay regularization. In: Proceedings of the 4th IEEE International Conference on Consumer Electronics Berlin. 2014, 185−187

[10] Dong C , Loy C C , He K , Tang X . Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38( 2): 295– 307

[11] KimJ, LeeJ K, LeeK M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1646−1654

[12] KimJ, LeeJ K, LeeK M. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1637−1645

[13] ZhangY, TianY, KongY, ZhongB, FuY. Residual dense network for image super-resolution. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 2472−2481

[14] ZhangY, LiK, LiK, WangL, ZhongB, FuY. Image super-resolution using very deep residual channel attention networks. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 294−310

[15] RadM S, BozorgtabarB, MartiU V, BaslerM, EkenelH K, ThiranJ P. SROBB: targeted perceptual loss for single image super-resolution. In: Proceedings of IEEE/CVF International Conference on Computer Vision. 2019, 2710−2719

[16] Ng M K , Shen H , Lam E Y , Zhang L . A total variation regularization based super-resolution reconstruction algorithm for digital video. EURASIP Journal on Advances in Signal Processing, 2007, 2007 : 074585–

[17] Jiang K , Wang Z , Yi P , Jiang J . Hierarchical dense recursive network for image super-resolution. Pattern Recognition, 2020, 107 : 107475–

[18] DongC, LoyC C, TangX. Accelerating the super-resolution convolutional neural network. In: Proceedings of the 14th European Conference on Computer Vision. 2016, 391−407

[19] HeK, ZhangX, RenS, SunJ. Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770−778

[20] LaiW S, HuangJ B, AhujaN, YangM H. Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5835−5843

[21] LedigC, TheisL, HuszárF, CaballeroJ, CunninghamA, AcostaA, AitkenA, TejaniA, TotzJ, WangZ, ShiW. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 105−114

[22] WangX, YuK, WuS, GuJ, LiuY, DongC, QiaoY, LoyC C. ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018, 63−79

[23] LimB, SonS, KimH, NahS, LeeK M. Enhanced deep residual networks for single image super-resolution. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017, 1132−1140

[24] YuJ, FanY, YangJ, XuN, WangZ, WangX, HuangT. Wide activation for efficient and accurate image super-resolution. 2018, arXiv preprint arXiv: 1808.08718

[25] DaiT, CaiJ, ZhangY, XiaS T, ZhangL. Second-order attention network for single image super-resolution. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 11057−11066

[26] ZhaoH, GalloO, FrosioI, KautzJ. Loss functions for neural networks for image processing. 2018, arXiv preprint arXiv: 1511.08861

[27] TimofteR, AgustssonE, Van GoolL, YangM H, ZhangL. NTIRE 2017 challenge on single image super-resolution: methods and results. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017, 1110−1121

[28] BevilacquaM, RoumyA, GuillemotC, MorelM L A. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of British Machine Vision Conference. 2012

[29] ZeydeR, EladM, ProtterM. On single image scale-up using sparse-representations. In: Proceedings of the 7th International Conference on Curves and Surfaces. 2010, 711−730

[30] MartinD, FowlkesC, TalD, MalikJ. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the 8th IEEE International Conference on Computer Vision. 2001, 416−423

[31] HuangJ B, SinghA, AhujaN. Single image super-resolution from transformed self-exemplars. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 5197−5206

[32] KingmaD P, BaJ. Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference for Learning Representations. 2015

[33] Wang Z , Bovik A C , Sheikh H R , Simoncelli E P . Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13( 4): 600– 612

[34] DongC, LoyC C, HeK, TangX. Learning a deep convolutional network for image super-resolution. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 184−199

[35] TaiY, YangJ, LiuX, XuC. MemNet: a persistent memory network for image restoration. In: Proceedings of IEEE International Conference on Computer Vision. 2017, 4549−4557

[36] HarisM, ShakhnarovichG, UkitaN. HarisM, ShakhnarovichG, UkitaN. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 1664−1673

[37] LiZ, YangJ, LiuZ, YangX, JeonG, WuW. Feedback network for image super-resolution. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 3862−3871

[38] HuiZ, GaoX, YangY, WangX. Lightweight image super-resolution with information multi-distillation network. In: Proceedings of the 27th ACM International Conference on Multimedia. 2019, 2024−2032

[39] ZhaoH, KongX, HeJ, QiaoY, DongC. Efficient image super-resolution using pixel attention. In: Proceedings of the European Conference on Computer Vision. 2020, 56−72

[40] WangC, LiZ, ShiJ. Lightweight image super-resolution with adaptive weighted learning network. 2019, arXiv preprint arXiv: 1904.02358

Acknowledgements

This research was supported by the National Natural Science Foundation of China (Grant Nos. 61603197 and 61772284), Natural Science Foundation of Nanjing University of Posts and Telecommunications (NY221071).

RIGHTS & PERMISSIONS

2021 Higher Education Press 2021