The role of prior in image based 3D modeling: a survey

Abstract

The prior knowledge is the significant supplement to image-based 3D modeling algorithms for refining the fragile consistency-based stereo. In this paper, we review the image-based 3D modeling problem according to prior categories, i.e., classical priors and specific priors. The classical priors including smoothness, silhouette and illumination are well studied for improving the accuracy and robustness of the 3D reconstruction. In recent years, various specific priors which take advantage of Manhattan rule, geometry template and trained category features have been proposed to enhance the modeling performance. The advantages and limitations of both kinds of priors are discussed and evaluated in the paper. Finally, we discuss the trend and challenges of the prior studies in the future.

Keywords

prior information / consistency-based stereo / smoothness / illumination / silhouette / specific prior

References

[1] Dyer C R. Volumetric scene reconstruction from multiple views. Foundations of Image Understanding, 2001, 628: 469–489

[2] Slabaugh G, Schafer R, Malzbender T, Culbertson B. A survey of methods for volumetric scene reconstruction from photographs. In: Proceedings of the Joint IEEE TCVG and Eurographics Workshop in Stony Brook. 2010, 81–100

[3] Seitz S M, Curless B, Diebel J, Scharstein D, Szeliski R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2006, 519–528

[4] Brenner C. Building reconstruction from images and laser scanning. International Journal of Applied Earth Observation and Geoinformation, 2005, 6(3): 187–198

[5] Newcombe R A, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison A J, Kohi P, Shotton J, Hodges S, Fitzgibbon A. KinectFusion: realtime dense surface mapping and tracking. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality. 2011, 127–136

[6] Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fitzgibbon A. Kinect- Fusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the ACM Symposium on User Interface Software and Technology. 2011, 559–568

[7] Scharstein D, Szeliski R, Hirschmüller H. Stereo. http://vision.middlebury. edu/stereo/, 2015

[8] Huang T S, Netravali A N. Motion and structure from feature correspondences: a review. Proceedings of the IEEE, 1994, 82(2): 252–268

[9] Oliensis J. A critique of structure-from-motion algorithms. Computer Vision and Image Understanding, 2000, 80(2): 172–214

[10] Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110

[11] Durrant-Whyte H, Bailey T. Simultaneous localization and mapping. IEEE Robotics & Automation Magazine, 2006, 13(2): 99–110

[12] Williams B, Klein G, Reid I. Real-time SLAMrelocalisation. In: Proceedings of the 11th IEEE International Conference on Computer Vision. 2007, 1–8

[13] Newcombe R A, Lovegrove S J, Davison A J. DTAM: dense tracking and mapping in real-time. In: Proceedings of the IEEE International Conference on Computer Vision. 2011, 2320–2327

[14] Tan W, Liu H M, Dong Z L, Zhang G F, Bao H J. Robust monocular SLAM in dynamic environments. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality. 2013, 209–218

[15] Zhang R, Tsai P S, Cryer J E, Shah M. Shape-from-shading: a survey. IEEE Transactions on Pattern Analysis andMachine Intelligence, 1999, 21(8): 690–706

[16] Woodham R J. Photometric method for determining surface orientation from multiple images. Optical Engineering, 1980, 19(1): 1–22

[17] Bartoli A, Gerard Y, Chadebecq F, Collins T, Pizarro D. The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(10): 2099–2118

[18] Laurentini A. The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1994, 16(2): 150–162

[19] Lee W, Woo W, Boyer E. Silhouette segmentation in multiple views. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(7): 1429–1441

[20] Tao H, Sawhney H S, Kumar R. A global matching framework for stereo computation. In: Proceedings of the 8th IEEE International Conference on Computer Vision. 2001, 532–539

[21] Bleyer M, Gelautz M. A layered stereo algorithm using image segmentation and global visibility constraints. In: Proceedings of the IEEE International Conference on Image Processing. 2004, 2997–3000

[22] Hong L, Chen G. Segment-based stereo matching using graph cuts. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2004, 74–81

[23] Klaus A, Sormann M, Karner K. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proceedings of the 18th IEEE International Conference on Pattern Recognition. 2006, 15–18

[24] Yang Q X, Wang L, Yang R G, Stewénius H, Nistér D. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(3): 492–504

[25] Scharstein D, Szeliski R. A taxonomy and evaluation of dense twoframe stereo correspondence algorithms. International Journal of Computer Vision, 2002, 47(1–3): 7–42

[26] Furukawa Y, Ponce J. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(8): 1362–1376

[27] Beeler T, Bickel B, Beardsley P, Sumner B, Gross M. High-quality single-shot capture of facial geometry. ACM Transactions on Graphics, 2010, 29(4): 40

[28] Kolev K, Klodt M, Brox T, Esedoglu S, Cremers D. Continuous global optimization in multiview 3D reconstruction. International Journal of Computer Vision, 2009, 84(1): 80–96

[29] Liu Y B, Cao X, Dai Q H, Xu W L. Continuous depth estimation for multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2009, 2121–2128

[30] Yao Y, Zhu H, Nie Y M, Ji X L, Cao X. Revised depth map estimation for multi-view stereo. In: Proceedings of the IEEE International Conference on 3D Imaging. 2014, 1–7

[31] Li G, Zucker S W. Surface geometric constraints for stereo in belief propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2006, 2355–2362

[32] Woodford O, Torr P, Reid I, Reid I, Fitzgibbon A. Global stereo reconstruction under second-order smoothness priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(12): 2115–2128

[33] Han X, Xu C Y, Prince J L. A topology preserving level set method for geometric deformable models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(6): 755–768

[34] Esteban C H, Schmitt F. Silhouette and stereo fusion for 3D object modeling. Computer Vision and Image Understanding, 2004, 96(3): 367–392

[35] Sorkine O, Cohen-Or D, Lipman Y, Alexa M, Rössl C, Seidel H P. Laplacian surface editing. In: Proceedings of the ACM SIGGRAPH symposium on Geometry processing. 2004, 175–184

[36] Zeng G, Paris S, Quan L, Sillion F. Progressive surface reconstruction from images using a local prior. In: Proceedings of the 10th IEEE International Conference on Computer Vision. 2005, 1230–1237

[37] Tasdizen T, Whitaker R. Higher-order nonlinear priors for surface reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(7): 878–891

[38] Li Y, Sun J, Tang C K, Shum H Y. Lazy snapping. ACM Transactions on Graphics, 2004, 23(3): 303–308

[39] Kolmogorov V, Criminisi A, Blake A, Cross G. Probabilistic fusion of stereo with color and contrast for bilayer segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(9): 1480–1492

[40] Franco J S, Boyer E. Efficient polyhedral modeling from silhouettes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(3): 414–427

[41] Matusik W, Buehler C, Raskar R, Gortler S, McMillan L. Imagebased visual hulls. In: Proceedings of the 27th ACM Annual Conference on Computer Graphics and Interactive Techniques. 2000, 369–374

[42] Miller G, Hilton A. Exact view-dependent visual hulls. In: Proceedings of the IEEE International Conference on Pattern Recognition. 2006, 107–111

[43] Vogiatzis G, Torr P H S, Cipolla R. Multi-view stereo via volumetric graph-cuts. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005, 391–398

[44] Furukawa Y, Ponce J. Carved visual hulls for image-based modeling. International Journal of Computer Vision, 2009, 81(1): 53–67

[45] Zheng Z Y, Ma L Z, Li Z, Chen Z H. Reconstruction of shape and reflectance properties based on visual hull. In: Proceedings of the ACM Computer Graphics International Conference. 2009, 29–38

[46] Sinha S N, Pollefeys M. Multi-view reconstruction using photoconsistency and exact silhouette constraints: a maximum-flow formulation. In: Proceedings of the 10th IEEE International Conference on Computer Vision. 2005, 349–356

[47] Gall J, Stoll C, De Aguiar E, Theobalt C, Rosenhahn B, Seidel H. Motion capture using joint skeleton tracking and surface estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1746–1753

[48] Vlasic D, Baran I, Matusik W, Popović J. Articulated mesh animation from multi-view silhouettes. ACM Transactions on Graphics, 2008, 27(3): 97

[49] Straka M, Hauswiesner S, Rüther M, Bischof H. Simultaneous shape and pose adaption of articulated models using linear optimization. In: Proceedings of European Conference on Computer Vision. 2012, 724–737

[50] Liu Y, Gall J, Stoll C, Dai Q, Seidel H, Theobalt C. Markerless motion capture of multiple characters using multiview image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(11): 2720–2735

[51] Horn B K P. Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. Technical Report. 1970

[52] Durou J D, Falcone M, Sagona M. Numerical methods for shapefrom- shading: a new survey with benchmarks. Computer Vision and Image Understanding, 2008, 109(1): 22–43

[53] Herbort S, Wohler C. An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods. 3D Research, 2011, 2(3): 1–17

[54] Leclerc Y G, Bobick A F. The direct computation of height from shading. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1991, 552–558

[55] Fua P, Leclerc Y G. Object-centered surface reconstruction: combining multi-image stereo and shading. International Journal of Computer Vision, 1995, 16(1): 35–56

[56] Jin H, Cremers D, Yezzi A J, Soatto S. Shedding light on stereoscopic segmentation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2004, 36–42

[57] Jin H, Yezz A, Soatto S. Stereoscopic shading: integrating multiframe shape cues in a variational framework. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2000, 169–176

[58] Jin H, Yezzi A J, Soatto S. Region-based segmentation on evolving surfaces with application to 3D reconstruction of shape and piecewise constant radiance. In: Proceedings of European Conference on Computer Vision. 2004, 114–125

[59] Yu T L, Xu N, Ahuja N. Recovering shape and reflectance model of non-lambertian objects from multiple views. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2004, 226–233

[60] Yu T L, Xu N, Ahuja N. Shape and view independent reflectance map from multiple views. International Journal of Computer Vision, 2007, 73(2): 123–138

[61] Yoon K J, Prados E, Sturm P. Joint estimation of shape and reflectance using multiple images with known illumination conditions. International Journal of Computer Vision, 2010, 86(2–3): 192–210

[62] Wu C L, Wilburn B, Matsushita Y, Theobalt C. High-quality shape from multi-view stereo and shading under general illumination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 969–976

[63] Han Y, Lee J Y, Kweon I S. High quality shape from a single RGBD image under uncalibrated natural illumination. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 1617–1624

[64] Zhang L, Curless B, Hertzmann A, Seitz S M. Shape and motion under varying illumination: unifying structure from motion, photometric stereo, and multiview stereo. In: Proceedings of the IEEE International Conference on Computer Vision. 2003, 618–625

[65] Basri R, Jacobs D, Kemelmacher I. Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 2007, 72(3): 239–257

[66] Hernndez C, Vogiatzis G, Brostow G J, Stenger B, Cipolla R. Nonrigid photometric stereo with colored lights. In: Proceedings of the 11th IEEE International Conference on Computer Vision. 2007, 1–8

[67] Brostow G J, Hernández C, Vogiatzis G, Stenger B, Cipolla R. Video normals from colored lights. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(10): 2104–2114

[68] Roth J, Tong Y, Liu X. Unconstrained 3D face reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 2606–2615

[69] Debevec P. The light stages and their applications to photoreal digital actors. SIGGRAPH Asia Technical Briefs, 2012, 2

[70] Ghosh A, Fyffe G, Tunwattanapong B, Busch J, Yu X M, Debevec P. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics, 2011, 30(6): 129

[71] Liu Y B, Dai Q H, Xu W L. A point-cloud-based multiview stereo algorithm for free-viewpoint video. IEEE Transactions on Visualization and Computer Graphics, 2010, 16(3): 407–418

[72] Wu C L, Liu Y B, Dai Q H, Bennett W. Fusing multiview and photometric stereo for 3D reconstruction under uncalibrated illumination. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(8): 1082–1095

[73] Coughlan J M, Yuille A L. Manhattan world: compass direction from a single image by Bayesian inference. In: Proceedings of the 7th IEEE International Conference on Computer Vision. 1999, 941–947

[74] Furukawa Y, Curless B, Seitz S M, Szeliski R. Manhattan-world stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1422–1429

[75] Vanegas C A, Aliaga D G, Beneš B. Building reconstruction using manhattan-world grammars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2010, 358–365

[76] Zeisl B, Zach C, Pollefeys M. Stereo reconstruction of building interiors with a vertical structure prior. In: Proceedings of the IEEE International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission. 2011, 366–373

[77] Sun J, Li Y, Kang S B, Shum H Y. Symmetric stereo matching for occlusion handling. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005, 399–406

[78] Zhang G, Jia J, Wong T T, Bao H. Consistent depth maps recovery from a video sequence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(6): 974–988

[79] Yang Q X, Wang L, Yang R G, Stewénius H, Nistér D. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(3): 492–504

[80] Sinha S N, Steedly D, Szeliski R. Piecewise planar stereo for imagebased rendering. In: Proceedings of the IEEE International Conference on Computer Vision. 2009, 1881–1888

[81] Gallup D, Frahm J M, Pollefeys M. Piecewise planar and nonplanar stereo for urban scene reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2010, 1418–1425

[82] Kim H, Xiao H, Max N. Piecewise planar scene reconstruction and optimization for multi-view stereo. In: Proceedings of Asian Conference on Computer Vision. 2013, 191–204

[83] Mathias M, Martinovic A, Weissenberg J, Van Gool L. Procedural 3D building reconstruction using shape grammars and detectors. In: Proceedings of the IEEE International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission. 2011, 304–311

[84] Zebedin L, Bauer J, Karner K, Bischof H. Fusion of feature-and areabased information for urban buildings modeling from aerial imagery. In: Proceedings of European Conference on Computer Vision. 2008, 873–886

[85] Wu C C, Agarwal S, Curless B, Seitz S M. Schematic surface reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 1498–1505

[86] Lafarge F, Keriven R, Brédif M, Hoang-Hiep V. A hybrid multiview stereo algorithm for modeling urban scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 5–17

[87] Lafarge F, Keriven R, Brédif M, Hoang-Hiep V. Hybrid multi-view reconstruction by jump-diffusion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2010, 350–357

[88] Mahabadi R K, Hane C, Pollefeys M. Segment based 3D object shape priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 2838–2846

[89] Widanagamaachchi W N, Dharmaratne A T. 3D face reconstruction from 2D images. In: Proceedings of the IEEE Digital Image Computing: Techniques and Applications. 2008, 365–371

[90] Amin S H, Gillies D. Analysis of 3D face reconstruction. In: Proceedings of the 14th IEEE International Conference on Image Analysis and Processing. 2007, 413–418

[91] Hassner T, Basri R. Example based 3D reconstruction from single 2D images. In: Proceedings of the IEEE Computer Vision and Pattern Recognition Workshop. 2006, 15

[92] Cheng C M, Lai S H. An integrated approach to 3D face model reconstruction from video. In: Proceedings of the IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. 2001, 16–22

[93] Fidaleo D, Medioni G. Model-assisted 3D face reconstruction from video. In: Proceedings of the International Workshop on Analysis and Modeling of Faces and Gestures. 2007, 124–138

[94] Tytgat D, Lievens S, Six E. A prior-based approach to 3D face reconstruction using depth images. Advances in Depth Image Analysis and Applications, 2013, 32–41

[95] Baumberger C, Reyes M, Constantinescu M, Olariu R, Aguiar E, Oliveira-Santos T. 3D face reconstruction from video using 3D morphable model and silhouette. In: Proceedings of the 27th IEEE SIBGRAPI Conference on Graphics, Patterns and Images. 2014, 1–8

[96] Kemelmacher-Shlizerman I, Seitz S M. Face reconstruction in the wild. In: Proceedings of the IEEE International Conference on Computer Vision. 2011, 1746–1753

[97] Bérard P, Bradley D, Nitti M, Beeler T, Gross M. High-quality capture of eyes. ACM Transactions on Graphics, 2014, 33(6): 1–12

[98] Bermano A, Beeler T, Kozlov Y, Bradley D, Bickel B, Gross M. Detailed spatio-temporal reconstruction of eyelids. ACM Transactions on Graphics, 2015, 34(4): 44

[99] Hu L W, Ma C Y, Luo L J, Li H. Robust hair capture using simulated examples. ACM Transactions on Graphics, 2014, 33(4): 126

[100] Luo L, Li H, Rusinkiewicz S. Structure-aware hair capture. ACM Transactions on Graphics, 2013, 32(4): 76

[101] Xiao J, Fang T, Zhao P, Lhuillier M, Quan L. Image-based street-side city modeling. ACM Transactions on Graphics, 2009, 28(5): 89–97

[102] Nan L, Sharf A, Zhang H, Cohen-Or D, Chen B. SmartBoxes for interactive urban reconstruction. ACM Transactions on Graphics, 2010, 29(4): 157–166

[103] Tan P, Zeng G, Wang J D, Kang S B, Quan L. Image-based tree modeling. ACM Transactions on Graphics, 2007, 26(3): 87

[104] Quan L, Tan P, Zeng G, Yuan L, Wang J D, Kang S B. Image-based plant modeling. ACM Transactions on Graphics, 2006, 25(3): 599–604

[105] Livny Y, Yan F, Olson M, Chen B, Zhang H, El-Sana J. Automatic reconstruction of tree skeletal structures from point clouds. ACM Transactions on Graphics, 2010, 29(6): 151

[106] Blanz V, Mehl A, Vetter T, Seidel H. A statistical method for robust 3D surface reconstruction from sparse data. In: Proceedings of the IEEE International Symposium on 3D Data Processing, Visualization and Transmission. 2004, 293–300

[107] Bao S Y, Savarese S. Semantic structure from motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 2025–2032

[108] Bao S Y, Chandraker M, Lin Y, Savarese S. Dense object reconstruction with semantic priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1264–1271

[109] Dame A, Prisacariu V A, Ren C Y, Reid I. Dense reconstruction using 3D object shape priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1288–1295

[110] Hane C, Savinov N, Pollefeys M. Class specific 3D object shape priors using surface normals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 652–659

[111] Kazhdan M, Bolitho M, Hoppe H. Poisson surface reconstruction. In: Proceedings of Eurographics Symposium on Geometry Processing. 2006, 61–70

[112] Zhang Q S, Song X, Shao X W, Zhao H J, Shibasaki R. When 3D reconstruction meets ubiquitous RGB-D images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 700–707

[113] Mehrdad V, Ebrahimnezhad H. 3D object retrieval based on histogram of local orientation using one-shot score support vector machine. Frontiers of Computer Science, 2015, 9(6): 990–1005

[114] Hane C, Ladicky L, Pollefeys M. Direction matters: depth estimation with a surface normal classifier. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 381–389

[115] Park M G, Yoon K J. Leveraging stereo matching with learning-based confidence measures. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 101–109

[116] Guney F, Geiger A. Displets: resolving stereo ambiguities using object knowledge. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4165–4175

[117] Tanskanen P, Kolev K, Meier L, Camposeco F, Saurer O, Pollefeys M. Live metric 3D reconstruction on mobile phones. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 65–72

[118] Kolev K, Tanskanen P, Speciale P, Pollefeyset M. Turning mobile phones into 3D scanners. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 3946–3953

RIGHTS & PERMISSIONS

2016 Higher Education Press and Springer-Verlag Berlin Heidelberg