Multi-view Shape-from-Shading: a Tour

Abstract

This paper re-explores the coupling between shape-from-shading (SfS) and multi-view stereo (MVS), i.e., reconstructing the geometry of a shaded surface from a set of views. We first provide intuitive evidence for the uniqueness of a solution based on epipolar geometry and Lambertian assumption arguments. Then, we explore several alternatives for approximating this solution, from the semi-Lagrangian approximation of the underlying Hamilton-Jacobi equation to the local optimisation of a neural implicit surface. We qualitatively and quantitatively evaluate the latter approach, which we denote SfS-NeuS, on several synthetic and real-world examples and show that it outperforms comparable state-of-the-art 3D reconstruction methods.

Keywords

Shape-from-shading (SfS) / Multi-view (MV) / 3D reconstruction / 65K05 / 90C46

References

[1] Blake, A.: Ambiguity of shading and stereo contour. In: Proceedings of the 2nd Alvey Vision Conference, pp. 1–10. Alvey Vision Club (1987)

[2] Blake A, Zisserman A, Knowles G. Surface descriptions from stereo and shading. Image Vis. Comput., 1985, 3(4): 183-191

[3] Brahimi, M., Haefner, B., Yenamandra, T., Goldluecke, B., Cremers, D.: SupeRVol: super-resolution shape and reflectance estimation in inverse volume rendering. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3139–3149. IEEE, Piscataway, NJ, USA (2024)

[4] Breuß M, Cristiani E, Durou J-D, Falcone M, Vogel O. Perspective shape from shading: ambiguity analysis and numerical approximations. SIAM J. Imag. Sci., 2012, 5(1): 311-342

[5] Bruckstein AM. On shape from shading. Comput. Vis. Graph. Image Process., 1988, 44(2): 139-154

[6] Brument, B., Bruneau, R., Quéau, Y., Mélou, J., Lauze, F., Durou, J.-D., Calvet, L.: RNb-NeuS: reflectance and normal-based multi-view 3D reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, California, USA (2024). https://doi.org/10.1109/CVPR52733.2024.00500

[7] Bruss AR. The eikonal equation: some results applicable to computer vision. J. Math. Phys., 1982, 23(5): 890-896

[8] Buskulic, N.: Neural network based methods for inverse problems: algorithms and guarantees. Doctoral dissertation. Université de Caen Normandie (2024)

[9] Buskulic N, Fadili J, Quéau Y. Convergence and recovery guarantees of unsupervised neural networks for inverse problems. J. Math. Imaging Vis., 2024, 66(4): 584-605

[10] Camilli F, Tozza S. A unified approach to the well-posedness of some non-Lambertian models in shape-from-shading theory. SIAM J. Imag. Sci., 2017, 10(1): 26-46

[11] Cao Z, Wang Y, Zheng W, Yin L, Tang Y, Miao W, Liu S, Yang B. The algorithm of stereo vision and shape from shading based on endoscope imaging. Biomed. Signal Process. Control, 2022, 76 103658

[12] Chambolle, A.: A uniqueness result in the theory of stereo vision: coupling shape from shading and binocular information allows unambiguous depth reconstruction. Annales de l’IHP Analyse Non Linéaire 11, 1–16 (1994)

[13] Cheng, Z., Li, H., Asano, Y., Zheng, Y., Sato, I.: Multi-view 3D reconstruction of a texture-less smooth surface of unknown generic reflectance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16226–16235. IEEE, Los Alamitos, California, USA (2021)

[14] Chow CK, Yuen SY. Recovering shape by shading and stereo under Lambertian shading model. Int. J. Comput. Vision, 2009, 85(1): 58-100

[15] Crandall M, Hitoshi I, Lions P-L. Uniqueness of viscosity solutions of Hamilton-Jacobi equations revisited. J. Math. Soc. Japan, 1987, 39(4): 581-596

[16] Cristiani, E., Falcone, M., Seghini, A.: Some remarks on perspective shape-from-shading models. In: Proceedings of the 1st International Conference on Scale Space and Variational Methods in Computer Vision, pp. 276–287. Springer, Berlin, Germany (2007)

[17] Durou, J.-D., Falcone, M., Quéau, Y., Tozza, S.: Advances in photometric 3D-reconstruction. In: Advances in Computer Vision and Pattern Recognition. Springer International Publishing, Heidelberg (2020)

[18] Durou, J.-D., Falcone, M., Sagona, M.: Numerical methods for shape-from-shading: a new survey with benchmarks. Comput. Vis. Image Underst. 109(1), 22–43 (2008)

[19] Falcone, M., Finzi Vita, S.: A scheme for the shape-from-shading model with “black shadows”. In: Falcone, M., Makridakis, C. (eds) Numerical Mathematics and Advanced Applications, pp. 503–512. World Scientific, Singapore (2003). https://doi.org/10.1142/4781

[20] Falcone M, Finzi Vita S. A finite-difference approximation of a two-layer system for growing sandpiles. SIAM J. Sci. Comput., 2006, 28(3): 1120-1132

[21] Gallardo, M., Collins, T., Bartoli, A.: Non-rigid structure-from-motion and shading. In: Durou, J.-D., Falcone, M., Quéau, Y., Tozza, S. (eds) Advances in Photometric 3D-Reconstruction, pp. 115–146. Springer, Cham, Switzerland (2020)

[22] Griwodz, C., Gasparini, S., Calvet, L., Gurdjos, P., Castan, F., Maujean, B., De Lillo, G., Lanthony, Y.: AliceVision Meshroom: an open-source 3D reconstruction pipeline. In: Proceedings of the 12th ACM Multimedia Systems Conference (2021)

[23] Hofherr, F., Haefner, B., Cremers, D.: On Neural BRDFs: a thorough comparison of state-of-the-art approaches. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1785–1794. IEEE, Piscataway, NJ, USA (2025)

[24] Horn, B.K.: Shape from shading: a method for obtaining the shape of a smooth opaque object from one view. Doctoral dissertation. Massachusetts Institute of Technology (1970)

[25] Horn BK, Brooks MJ. The variational approach to shape from shading. Comput. Vis. Graph. Image Process., 1986, 33(2): 174-208

[26] Jin H, Cremers D, Wang D, Prados E, Yezzi A, Soatto S. 3-D reconstruction of shaded objects from multiple images under unknown illumination. Int. J. Comput. Vis., 2008, 76(3): 245-256

[27] Kingma, D.P., Ba, J.: ADAM: a method for stochastic optimization (2014). arXiv:1412.6980

[28] Langguth, F., Sunkavalli, K., Hadap, S., Goesele, M.: Shading-aware multi-view stereo. In: Proceedings of the 19th European Conference on Computer Vision, pp. 469–485. Springer, Cham, Switzerland (2016)

[29] Li, Z., Müller, T., Evans, A., Taylor, R.H., Unberath, M., Liu, M.-Y., Lin, C.-H.: Neuralangelo: high-fidelity neural surface reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8456–8465. IEEE, Piscataway, NJ, USA (2023)

[30] Liang Z, Xu C, Hu J, Li Y, Meng Z. Better together: shading cues and multi-view stereo for reconstruction depth optimization. IEEE Access, 2020, 8: 112348-112356

[31] Liu B, Todd JT. Perceptual biases in the interpretation of 3D shape from shading. Vision. Res., 2004, 44(18): 2135-2145

[32] Marr D, Poggio T. Cooperative computation of stereo disparity: a cooperative algorithm is derived for extracting disparity information from stereo image pairs. Science, 1976, 194(4262): 283-287

[33] Maurer D, Ju YC, Breuß M, Bruhn A. Combining shape from shading and stereo: a joint variational method for estimating depth, illumination and albedo. Int. J. Comput. Vis., 2018, 126(12): 1342-1366

[34] Mecca R, Falcone M. Uniqueness and approximation of a photometric shape-from-shading model. SIAM J. Imag. Sci., 2013, 6(1): 616-659

[35] Mélou, J., Quéau, Y., Castan, F., Durou, J.-D.: A splitting-based algorithm for multi-view stereopsis of textureless objects. In: Proceedings of the 7th International Conference on Scale Space and Variational Methods in Computer Vision, pp. 51–63. Springer, Cham, Switzerland (2019)

[36] Pentland AP. Local shading analysis. IEEE Trans. Pattern Anal. Mach. Intell., 1984, 2: 170-187

[37] Quéau, Y., Mélou, J., Castan, F., Cremers, D., Durou, J.-D.: A variational approach to shape-from-shading under natural illumination. In: Proceedings of the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 342–357. Springer, Cham, Switzerland (2018)

[38] Romanoni, A., Matteucci, M.: Tapa-MVS: textureless-aware patchmatch multi-view stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10413–10422. IEEE, Piscataway, NJ, USA (2019)

[39] Schönberger, J., Frahm, J.-M.: Structure-from-motion revisited. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, USA (2016)

[40] Tozza S, Falcone M. Analysis and approximation of some shape-from-shading models for non-Lambertian surfaces. J. Math. Imag. Vis., 2016, 55(2): 153-178

[41] Visual Computing Lab-ISTI VCGlib. https://github.com/cnr-isti-vclab/vcglib

[42] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: NeuS: learning neural implicit surfaces by volume rendering for multi-view reconstruction. In: Proceedings of the 35th Annual Conference on Neural Information Processing Systems. Curran Associates, Inc., Red Hook, NY, USA (2021)

[43] Wang, Y., Zeng, Z., Guan, T., Yang, W., Chen, Z., Liu, W., Xu, L., Luo, Y.: Adaptive patch deformation for textureless-resilient multi-view stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1621–1630. IEEE, Piscataway, NJ, USA (2023)

[44] Woodham, R.J.: Photometric stereo: A reflectance map technique for determining surface orientation from image intensity. In: Image Understanding Systems and Industrial Applications I, vol. 155, pp. 136–143. SPIE, Bellingham, Washington, USA (1979)

[45] Xu D, Duan Q, Zheng J, Zhang J, Cai J, Cham T-J. Shading-based surface detail recovery under general unknown illumination. IEEE Trans. Pattern Anal. Mach. Intell., 2018, 40(2): 423-436

[46] Yamashita, K., Enyo, Y., Nobuhara, S., Nishino, K.: nLMVS-Net: deep non-Lambertian multi-view stereo. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3037–3046. IEEE, Los Alamitos, California, USA (2023)

Funding

Gruppo Nazionale per l’Analisi Matematica, la Probabilità e le loro Applicazioni(UP_E53C23001670001)

Rights & permissions

Shanghai University

Just Accepted

This article has successfully passed peer review and final editorial review, and will soon enter typesetting, proofreading and other publishing processes. The currently displayed version is the accepted final manuscript. The officially published version will be updated with format, DOI and citation information upon launch. We recommend that you pay attention to subsequent journal notifications and preferentially cite the officially published version. Thank you for your support and cooperation.