Joint view synthesis and disparity refinement for stereo matching

Abstract

Typical stereo algorithms treat disparity estimation and view synthesis as two sequential procedures. In this paper, we consider stereo matching and view synthesis as two complementary components, and present a novel iterative refinement model for joint view synthesis and disparity refinement. To achieve the mutual promotion between view synthesis and disparity refinement, we apply two key strategies, disparity maps fusion and disparity-assisted plane sweep-based rendering (DAPSR). On the one hand, the disparity maps fusion strategy is applied to generate disparity map from synthesized view and input views. This strategy is able to detect and counteract disparity errors caused by potential artifacts from synthesized view. On the other hand, the DAPSR is used for view synthesis and updating, and is able to weaken the interpolation errors caused by outliers in the disparity maps. Experiments onMiddlebury benchmarks demonstrate that by introducing the synthesized view, disparity errors due to large occluded region and large baseline are eliminated effectively and the synthesis quality is greatly improved.

Keywords

stereo matching / view synthesis / disparity refinement

References

[1] Bleyer M, Rother C, Kohli P. Surface stereo with soft segmentation. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. 2010, 1570–1577

[2] Fickel G P, Jung C R, Malzbender T, Samadani R, Culbertson B. Stereo matching and view interpolation based on image domain triangulation. IEEE Transactions on Image Processing, 2013, 22(9): 3353–3365

[3] Vogel C, Schindler K, Roth S. 3D scene flow estimation with a piecewise rigid scene model. International Journal of Computer Vision, 2015, 115(1): 1–28

[4] Wu G, Masia B, Jarabo A, Zhang Y, Wang L, Dai Q, Chai T, Liu Y. Light field image processing: an overview. IEEE Journal of Selected Topics in Signal Processing, 2017, 11(7): 926–954

[5] Wu G, Liu Y, Fang L, Dai Q, Chai T. Light field reconstruction using deep convolutional network on EPI and extended applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(7): 1681–1694

[6] Fehn C. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Proceedings of SPIE Stereoscopic Displays and Virtual Reality Systems XI. 2004, 93–104

[7] Hosni A, Rhemann C, Bleyer M, Rother C, Gelautz M. Fast costvolume filtering for visual correspondence and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2): 504–511

[8] Lee S, Lee J H, Lim J, Suh I H. Robust stereo matching using adaptive random walk with restart algorithm. Image and Vision Computing, 2015, 37(C): 1–11

[9] Sun D, Roth S, Black M J. A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision, 2014, 106(2): 115–137

[10] Kubota A, Smolic A, Magnor M, Tanimoto M, Chen T, Zhang C. Multiview imaging and 3DTV. IEEE Signal Processing Magazine, 2007, 24(6): 10–21

[11] Stefanoski N, Wang O, Lang M, Greisen P, Heinzle S, Smolic A. Automatic view synthesis by image-domain-warping. IEEE Transactions on Image Processing, 2013, 22(9): 3329–3341

[12] Tian D, Lai P L, Lopez P, Gomila C. View synthesis techniques for 3D video. In: Proceedings of Internatlbnal Society for Optics and Phontonics, Applications of Digital Image Processing XXXII. 2009, 7443: 74430T

[13] Mori Y, Fukushima N, Yendo T, Fujii T, Tanimoto M. View generation with 3D warping using depth information for FTV. Signal Processing: Image Communication, 2009, 24(1–2): 65–72

[14] Wanner S, Goldluecke B. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(3): 606–619

[15] Zhang L, Zhang Y H, Huang H. Efficient variational light field view synthesis for making stereoscopic 3D images. Computer Graphics Forum. 2015, 7(34): 183–191

[16] Zhang C, Li Z, Cheng Y, Cai R, Chao H, Rui Y. Meshstereo: a global stereo model with mesh alignment regularization for view interpolation. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 2057–2065

[17] Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics, 2016, 35(6): 193

[18] Scharstein D, Szeliski R. A taxonomy and evaluation of dense twoframe stereo correspondence algorithms. International Journal of Computer Vision, 2002, 47(1–3): 7–42

[19] Kordelas G A, Alexiadis D S, Daras P, Izquierdo E. Enhanced disparity estimation in stereo images. Image and Vision Computing, 2015, 35: 31–49

[20] Peng Y, Li G, Wang R, Wang W. Stereo matching with spaceconstrained cost aggregation and segmentation-based disparity refinement. In: Proceedings of International Society for Optics and Phoritonics, Three-Dimensional Image Processing, Measurement, and Applications. 2015, 9393: 939309

[21] Zbontar J, LeCun Y. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 2016, 17(1): 2287–2318

[22] Luo W, Schwing A G, Urtasun R. Efficient deep learning for stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 5695–5703

[23] Mattoccia S, Viti M, Ries F. Near real-time fast bilateral stereo on the GPU. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2011, 136–143

[24] Mattoccia S, Giardino S, Gambini A. Accurate and efficient cost aggregation strategy for stereo correspondence based on approximated joint bilateral filtering. In: Proceedings of Asian Conference on Computer Vision. 2009, 371–380

[25] Yoon K J, Kweon I S. Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(4): 650–656

[26] Kolmogorov V, Zabih R. Computing visual correspondence with occlusions using graph cuts. In: Proceedings of the 8th IEEE International Conference on Computer Vision. 2001, 508–515

[27] Taniai T, Matsushita Y, Naemura T. Graph cut based continuous stereo matching using locally shared labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1613–1620

[28] Yang Q, Wang L, Yang R, Stewénius H, Nistér D. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(3): 492–504

[29] Guney F, Geiger A. Displets: resolving stereo ambiguities using object knowledge. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4165–4175

[30] Psota E T, Kowalczuk J, Mittek M, Perez L C. Map disparity estimation using hidden markov trees. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 2219–2227

[31] Mozerov M G, Van de W J. Accurate stereo matching by two-step energy minimization. IEEE Transactions on Image Processing, 2015, 24(3): 1153–1163

[32] Mei X, Sun X, Zhou M, Jiao S, Wang H, Zhang X. On building an accurate stereo matching system on graphics hardware. In: Proceedings of the IEEE Conference on Computer Vision Workshops. 2011, 467–474

[33] Veksler O. Fast variable window for stereo correspondence using integral images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2003, 556–561

[34] Klaus A, Sormann M, Karner K. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proceedings of the 18th International Conference on Pattern Recognition. 2006, 15–18

[35] Vogel C, Schindler K, Roth S. 3D scene flow estimation with a rigid motion prior. In: Proceedings of the IEEE International Conference on Computer Vision. 2011, 1291–1298

[36] Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11): 1222–1239

[37] Yang Q, Yang R, Davis J, Nistér D. Spatial-depth super resolution for range images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8

[38] Boominathan V, Mitra K, Veeraraghavan A. Improving resolution and depth-of-field of light field cameras using a hybrid imaging system. In: Proceedings of the IEEE International Conference on Computational Photography. 2014, 1–10

[39] Ma Z, He K, Wei Y, Sun J, Wu E. Constant time weighted median filtering for stereo matching and beyond. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 49–56

[40] Zhang Q, Xu L, Jia J. 100+ times faster weighted median filter (WMF). In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 2830–2837

[41] Lin Z, Shum H Y. A geometric analysis of light field rendering. International Journal of Computer Vision, 2004, 58(2): 121–138

[42] Scharstein D, Szeliski R. High-accuracy stereo depth maps using structured light. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2003, 195–202

[43] Scharstein D, Hirschmüller H, Kitajima Y, Krathwohl G, Nešić N, Wang X, Westling P. High-resolution stereo datasets with subpixelaccurate ground truth. In: Proceedings of German Conference on Pattern Recognition. 2014, 31–42

RIGHTS & PERMISSIONS

Higher Education Press and Springer-Verlag GmbH Germany, part of Springer Nature