Multi-frame super-resolution reconstruction based on global motion estimation using a novel CNN descriptor

Abstract

In this paper, we introduce a novel feature descriptor based on deep learning that trains a model to match the patches of images on scenes captured under different viewpoints and lighting conditions for Multi-frame super-resolution. The patch matching of images capturing the same scene in varied circumstances and diverse manners is challenging. We develop a model which maps the raw image patch to a low dimensional feature vector. As our experiments show, the proposed approach is much better than state-of-the-art descriptors and can be considered as a direct replacement of SURF. The results confirm that these techniques further improve the performance of the proposed descriptor. Then we propose an improved Random Sample Consensus algorithm for removing false matching points. Finally, we show that our neural network based image descriptor for image patch matching outperforms state-of-the-art methods on a number of benchmark datasets and can be used for image registration with high quality in multi-frame super-resolution reconstruction.

References

[1] Hyde RE. SPIE, 2002, 4849: 28

[2] BayH, EssA, TuytelaarsT, Van GoolL. Computer Vision and image Understanding, 2008, 110: 346

[3] BrownM, HuaG, WinderS. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33: 43

[4] TrzcinskiT, ChristoudiasM, LepetitV. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37: 597

[5] TrzcinskiT, ChristoudiasM, FuaP, LepetitV. Boosting Binary Key-Point Descriptors, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2013, 2874

[6] RussakovskyO, DengJ, SuH, KrauseJ, SatheeshS, MaS, HuangZ, KarpathyA, KhoslaA, BernsteinM, BergA, Fei-FeiL. International Journal Of Computer Vision, 2015, 115: 211

[7] FischerP, DosovitskiyA, BroxT. Descriptor Matching with Convolutional Neural Networks: a Comparison to SIFT, 2014,

[8] Simo-SerraE, TrullsE, FerrazL, KokkinosI, FuaP, Moreno-NoguerF. Discriminative Learning of Deep Convolutional Feature Point Descriptors, IEEE International Conference on Computer Vision, 2015, 118

[9] HanX, LeungT, JiaY, SukthankarR, BergA. Matchnet: Unifying Feature and Metric Learning for Patch-Based Matching. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2015, 3279

[10] YiK, TrullsE, LepetitV, FuaP. LIFT: Learned Invariant Feature Transform. European Conference Computer Vision, 2016, 467

[11] TianY, FanB, WuF. L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2017, 6128

[12] ChenM, WangC, QinH. Computer Aided Geometric Design, 2018, 62: 192

[13] BrownL. ACM Computing Surveys, 1992, 24: 325

[14] ZitovaB, FlusserJ. Image and Vision Computing, 2003, 21: 977

[15] LucasB, KanadeT. An Iterative Image Registration Technique with an Application to Stereo Vision, The 7th International Joint Conference on Artificial Intelligence, 1981, 674

[16] HarrisC, StephensM. A Combined Corner and Edge Detector, The 4th Alvey Vision Conference, 1988, 10

[17] LoweD. International Journal of Computer Vision, 2004, 60: 91

[18] KerenD, PelegS, BradaR. Image Sequence Enhancement Using Subpixel Displacements. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1988, 742

[19] IraniM, PelegS. CVGIP: Graphical Models & Image Processing, 1991, 53: 231

[20] SchultzR, StevensonR. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 1996, 5: 996

[21] BakerS, KanadeT. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24: 1167

[22] LiaoR, TaoX, LiR. Video Super-Resolution via Deep Draft-Ensemble Learning. IEEE International Conference on Computer Vision, 2015, 531

[23] KappelerA, YooS, DaiQ, KatsaggelosA. IEEE Transactions on Computational Imaging, 2016, 2: 109

[24] CaballeroJ, LedigC, AitkenA, AcostaA, TotzJ, WangZ, ShiW. Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation. IEEE Computer Vision and Pattern Recognition, 2017, 2848

[25] TaoX, GaoH, LiaoR, WangJ, JiaJ. Detail-Revealing Deep Video Super-Resolution. IEEE International Conference on Computer Vision, 2017, 4482

[26] RenS, HeK, GirshickR, SunJ. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39: 1137

[27] FischlerM, BollesR. Communications of the ACM, 1981, 24: 381

[28] VerdieY, YiK, FuaP, LepetitV. TILDE: A Temporally Invariant Learned Detector. IEEE Conference on Computer Vision and Pattern Recognition, 2015, 5279

[29] StrechaC, HansenW, Van GoolL, FuaP, ThoennessenU. On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery. IEEE Conference on Computer Vision and Pattern Recognition, 2008, 1

[30] RubleeE, RabaudV, KonolidgeK, BradskiG. ORB: An Efficient Alternative to SIFT or SURF. International Conference on Computer Vision, 2011, 2564

[31] BalntasV, JohnsE, TangL, MikolajczykK. PN-Net: Conjoined Triple Deep Network for Learning Local Image Descriptors, 2016,

[32] HanX, LeungT, JiaY, SukthankarR, BergA. MatchNet: Unifying Feature and Metric Learning for Patch-Based Matching. IEEE Conference on Computer Vision and Pattern Recognition, 2015, 3279

[33] WangZ, BovikA, SheikhH, SimoncelliE. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2004, 13: 600