Video Super-Resolution via Effective Spatio-Temporal Alignment Network

Abstract

Extracting spatio-temporal cues from neighbouring frames is challenging in video super-resolution (VSR). Although deformable alignment-based VSR methods have shown promise in aligning neighbouring frames with the reference frame, most existing methods rely on one or a few traditional convolutions to estimate motion offsets for spatio-temporal alignment, restricting receptive field size and alignment accuracy. To address these limitations, we propose an effective spatio-temporal alignment network (ESTA-Net) for VSR. The core component of our method is the group convolution-based alignment module (GCBAM), which utilises cascaded group convolutions to learn offsets across both the original and downsampled resolutions. By employing group convolutions rather than traditional convolutions, GCBAM enables the deformable alignment to achieve a wider receptive field with lower computational cost, thereby improving the accuracy of offset estimation. Additionally, the bi-scale alignment strategy within GCBAM enhances robustness to complex and large-scale motions. Furthermore, we introduce an attention-based feature enhancement module (AFEM) to refine the aligned features, focusing on critical details to improve reconstruction quality. Extensive experiments on standard benchmarks show that our ESTA-Net achieves superior VSR performance against other advanced methods, while maintaining a good equilibrium between model size and performance.

Keywords

group convolution / motion offsets / spatio-temporal alignment / video super-resolution

Acknowledgements

This study is supported by the Establishment of Key Laboratory of Shenzhen Science and Technology Innovation Committee under Grant No. ZDSYS20190902093015527 and the Shenzhen Science and Technology Innovation Committee under Grant No. JSGG20220831104402004.

Conflicts of Interest

Yong Xu is a Deputy Editor-in-Chief for the journal, and was not involved in peer review process or the decision to publish this article. The authors declare that they have no conflict of interest.

Data Availability Statement

Data available on request from the authors.

References

[1] S. Y. Kim, J. Lim, T. Na, and M. Kim, “3dsrnet: Video SuperResolution Using 3d Convolutional Neural Networks,” arXiv preprint arXiv:1812.09079 (2018).

[2] H. Liu, P. Zhao, Z. Ruan, F. Shang, Y. Liu, “Large Motion Video SuperResolution With Dual Subnet and MultiStage Communicated Upsampling,” in Proceedings of the AAAI Conference on Artificial Intelligence (2021), 2127-2135, https://doi.org/10.1609/aaai.v35i3.16310.

[3] H. Li and P. Zhang, “SpatioTemporal Fusion Network for Video SuperResolution,” in 2021 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2021), 1-9.

[4] T. Isobe, X. Jia, S. Gu, S. Li, S. Wang, and Q. Tian, “Video SuperResolution With Recurrent StructureDetail Network,” in Computer Vision-ECCV 2020: 16Th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16 (Springer, 2020), 645-660.

[5] J. Caballero, C. Ledig, A. Aitken, et al., RealTime Video SuperResolution With SpatioTemporal Networks and Motion Compensation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 4778-4787.

[6] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video Enhancement With TaskOriented Flow,” International Journal of Computer Vision 127, no. 8 (2019): 1106-1125, https://doi.org/10.1007/s11263018011442.

[7] M. S. Sajjadi, R. Vemulapalli, and M. Brown, “FrameRecurrent Video Superresolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2018), 6626-6634.

[8] Y. Tian, Y. Zhang, Y. Fu, and C. Xu, “Tdan: TemporallyDeformable Alignment Network for Video SuperResolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2020), 3360-3369.

[9] X. Ying, L. Wang, Y. Wang, W. Sheng, W. An, and Y. Guo, “Deformable 3d Convolution for Video SuperResolution,” IEEE Signal Processing Letters 27 (2020): 1500-1504, https://doi.org/10.1109/lsp.2020.3013518.

[10] K. C. Chan, X. Wang, K. Yu, C. Dong, and C. C. Loy, “Understanding Deformable Alignment in Video SuperResolution,” Proceedings of the AAAI Conference on Artificial Intelligence 35, no. 2 (2021): 973-981, https://doi.org/10.1609/aaai.v35i2.16181.

[11] J. Chen, X. Tan, C. Shan, S. Liu, and Z. Chen, “VesrNet: The Winning Solution to Youku Video Enhancement and SuperResolution Challenge,” arXiv preprint arXiv:2003.02115 (2020).

[12] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image Superresolution Using Very Deep Residual Channel Attention Networks,” in Proceedings of the European Conference on Computer Vision (ECCV), (2018), 286-301.

[13] X. Wang, J. Li, J. Li, S. Wang, L. Yan, and Y. Xu, “A Collaborative Network of Mamba and Cnn for Lightweight Image SuperResolution,” IEEE Transactions on Consumer Electronics (2025).

[14] C. Tian, X. Zhang, Q. Zhang, M. Yang, and Z. Ju, “Image SuperResolution via Dynamic Network,” CAAI Transactions on Intelligence Technology 9, no. 4 (2024): 837-849, https://doi.org/10.1049/cit2.12297.

[15] C. Tian, M. Song, X. Fan, X. Zheng, B. Zhang, and D. Zhang, “A TreeGuided Cnn for Image SuperResolution,” IEEE Transactions on Consumer Electronics (2025).

[16] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual Dense Network for Image SuperResolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), 2472-2481.

[17] H. Dou, C. Chen, X. Hu, Z. Xuan, Z. Hu, and S. Peng, “PcaSrgan: Incremental Orthogonal Projection Discrimination for Face SuperResolution,” in Proceedings of the 28th ACM International Conference on Multimedia, (2020), 1891-1899.

[18] X. Wang, S. Wang, J. Li, M. Li, J. Li, and Y. Xu, “Omnidirectional Image SuperResolution via Position Attention Network,” Neural Networks 178 (2024): 106464, https://doi.org/10.1016/j.neunet.2024.106464.

[19] X. Wang, J. Li, J. Li, S. Wang, and Y. Xu, “LatitudeOriented Hierarchical Enhancement Network for Omnidirectional Image SuperResolution,” Information Processing & Management 62, no. 6 (2025): 104217, https://doi.org/10.1016/j.ipm.2025.104217.

[20] Q. Zhang, J. Xiao, C. Tian, J. Xu, S. Zhang, and C.W. Lin, “A Parallel and Serial Denoising Network,” Expert Systems with Applications 231 (2023): 120628, https://doi.org/10.1016/j.eswa.2023.120628.

[21] C. Tian, M. Zheng, B. Li, Y. Zhang, S. Zhang, and D. Zhang, “Perceptive SelfSupervised Learning Network for Noisy Image Watermark Removal,” IEEE Transactions on Circuits and Systems for Video Technology 34, no. 8 (2024): 7069-7079, https://doi.org/10.1109/tcsvt.2024.3349678.

[22] C. Tian, X. Zhang, T. Wang, Y. Zhang, Q. Zhu, and C.W. Lin, “Adaptive Convolutional Neural Network for Image SuperResolution,” arXiv preprint arXiv:2402.15704 (2024).

[23] C. Tian, M. Zheng, C.W. Lin, Z. Li, and D. Zhang, “Heterogeneous Window Transformer for Image Denoising,” IEEE Transactions on Systems, Man, and Cybernetics: Systems (2024).

[24] C. Tian, M. Zheng, T. Jiao, W. Zuo, Y. Zhang, and C.W. Lin, “A SelfSupervised Cnn for Image Watermark Removal,” IEEE Transactions on Circuits and Systems for Video Technology 34, no. 8 (2024): 7566-7576, https://doi.org/10.1109/tcsvt.2024.3375831.

[25] X. Zhu, Z. Li, X.Y. Zhang, C. Li, Y. Liu, and Z. Xue, “Residual Invertible SpatioTemporal Network for Video SuperResolution,” Proceedings of the AAAI Conference on Artificial Intelligence 33, no. 1 (2019): 5981-5988, https://doi.org/10.1609/aaai.v33i01.33015981.

[26] Y. Jo, S. W. Oh, J. Kang, and S. J. Kim, “Deep Video SuperResolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), 3224-3232.

[27] J. Liang, J. Cao, Y. Fan, et al., Vrt: A Video Restoration Transformer,” IEEE Transactions on Image Processing (2024).

[28] R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, “Video SuperResolution via Deep DraftEnsemble Learning,” in Proceedings of the IEEE International Conference on Computer Vision, (2015), 531-539.

[29] L. Xu, J. Jia, and Y. Matsushita, “Motion Detail Preserving Optical Flow Estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence 34, no. 9 (2011): 1744-1757, https://doi.org/10.1109/TPAMI.2011.236.

[30] J. Cao, Y. Li, K. Zhang, and L. V. Gool, “Video SuperResolution Transformer,” arXiv preprint arXiv:2106.06847 (2023).

[31] X. Wang, K. C. Chan, K. Yu, C. Dong, and C. Change Loy, “Edvr: Video Restoration With Enhanced Deformable Convolutional Networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, (2019).

[32] H. Wang, X. Xiang, Y. Tian, W. Yang, and Q. Liao, “Stdan: Deformable Attention Network for SpaceTime Video SuperResolution,” IEEE Transactions on Neural Networks and Learning Systems (2023).

[33] Y.J. Choi, Y.W. Lee, and B.G. Kim, “Wavelet Attention Embedding Networks for Video SuperResolution,” in 2020 25th International Conference on Pattern Recognition (ICPR) (IEEE, 2021), 7314-7320.

[34] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An Extremely Efficient Convolutional Neural Network for Mobile Devices,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), 6848-6856.

[35] X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable Convnets v2: More Deformable, Better Results,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2019), 9308-9316.

[36] J. Hu, L. Shen, and G. Sun, “SqueezeAndExcitation Networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), 7132-7141.

[37] Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, “EcaNet: Efficient Channel Attention for Deep Convolutional Neural Networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2020), 11534-11542.

[38] C. Liu and D. Sun, “On Bayesian Adaptive Video Super Resolution,” IEEE Transactions on Pattern Analysis and Machine Intelligence 36, no. 2 (2013): 346-360, https://doi.org/10.1109/TPAMI.2013.127.

[39] S. Nah, S. Baik, S. Hong, et al., Ntire 2019 Challenge on Video Deblurring and SuperResolution: Dataset and Study,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, (2019).

[40] D. P. Kingma, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980 (2014).

[41] P. Charbonnier, L. BlancFeraud, G. Aubert, and M. Barlaud, “Two Deterministic halfquadratic Regularization Algorithms for Computed Imaging,” in Proceedings of 1st International Conference on Image Processing, Vol. 2 (IEEE, 1994), 168-172.

[42] A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, “Video SuperResolution With Convolutional Neural Networks,” IEEE Transactions on Computational Imaging 2, no. 2 (2016): 109-122, https://doi.org/10.1109/tci.2016.2532323.

[43] L. Wang, Y. Guo, Z. Lin, X. Deng, and W. An, “Learning for Video SuperResolution Through Hr Optical Flow Estimation,” in Computer Vision-ACCV 2018: 14Th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part I 14 (Springer, 2019), 514-529.

[44] P. Yi, Z. Wang, K. Jiang, J. Jiang, and J. Ma, “Progressive Fusion Video SuperResolution Network via Exploiting NonLocal SpatioTemporal Correlations,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2019), 3106-3115.

[45] M. Lu and P. Zhang, “Grouped SpatioTemporal Alignment Network for Video SuperResolution,” IEEE Signal Processing Letters 29 (2022): 2193-2197, https://doi.org/10.1109/lsp.2022.3210874.

[46] W. Wen, W. Ren, Y. Shi, Y. Nie, J. Zhang, and X. Cao, “Video SuperResolution via a SpatioTemporal Alignment Network,” IEEE Transactions on Image Processing 31 (2022): 1761-1773, https://doi.org/10.1109/tip.2022.3146625.

[47] M. Zheng and J. Luo, “SpaceTime Video SuperResolution 3d Transformer,” in International Conference on Multimedia Modeling (Springer, 2023), 374-385.

[48] X. Wang, X. Yang, H. Li, and T. Li, “FddccVSR: A Lightweight Video Superresolution Network Based on Deformable 3d Convolution and Cheap Convolution,” Visual Computer 41, no. 5 (2025): 3581-3593, https://doi.org/10.1007/s0037102403621x.

[49] K. C. Chan, S. Zhou, X. Xu, and C. C. Loy, “Investigating Tradeoffs in RealWorld Video SuperResolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2022), 5962-5971.