From dark flash images to relightable 3D scenes with photometric stereo priors

Abstract

Radiance fields, such as NeRFs, 3D Gaussians, and their variants, have emerged as the leading representations for 3D scene reconstruction due to their exceptional performance in novel view synthesis. However, their effectiveness depends on input images captured in well-lit, static environments, making dark scenes a significantly challenging case. Prior works employ low-light enhancement for low-light scenes (e.g., candlelight), but completely dark scenes remain an unsolved problem. However, this is a very common case when exploring unknown scenes, such as caves or nighttime forests, or derelict buildings. To solve the problem, we propose capturing images with a camera-mounted flashlight for exploring such scenes, which is an easily accessible setting for robots. The flashlight’s parameters are modeled and optimized in the reconstruction pipeline, including the flashlight’s angular and distance attenuation, position, rotation, and intensities. Under this setting, the captured images are under dynamic lighting conditions, i.e., lighting is changing for each image. We formulate a photometric stereo (PS) problem of input images by a grouping-and-merging strategy, leveraging its results as supervision priors. As a result, the method enables reconstruction and relighting of dark scenes. Experiments show that the method outperforms state-of-the-art approaches in decomposition, geometry, and relighting.

Graphical abstract

Keywords

3D reconstruction / dark-scene reconstruction / image relighting

References

[1] Mildenhall B, Hedman P, Martin-Brualla R, Srinivasan P P, Barron J T. NeRF in the dark: high dynamic range view synthesis from noisy raw images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, 16169−16178

[2] Li Z, Wang Y, Kot A, Wen B. From chaos to clarity: 3DGS in the dark. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024, 3010

[3] Jin X, Jiao P, Duan Z P, Yang X, Li C, Guo C L, Ren B. Lighting every darkness with 3DGS: fast training and real-time rendering for HDR view synthesis. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. 2024, 2549

[4] Xu J, Liao M, Kathirvel R P, Patel V M. Leveraging thermal modality to enhance reconstruction in low-light conditions. In: Proceedings of the 18th European Conference on Computer Vision. 2024, 321−339

[5] Zhang T, Huang K, Zhi W, Johnson-Roberson M. DarkGS: Learning neural illumination and 3D Gaussians relighting for robotic exploration in the dark. In: Proceedings of 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems. 2024, 12864−12871

[6] Huang J, Zhu P, Geng M, Ran J, Zhou X, Xing C, Wan P, Ji X. Range scaling global u-net for perceptual image enhancement on mobile devices. In: Proceedings of the European Conference on Computer Vision Workshops. 2018, 230−242

[7] He Z, Ran W, Liu S, Li K, Lu J, Xie C, Liu Y, Lu H . Low-light image enhancement with multi-scale attention and frequency-domain optimization. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34( 4): 2861–2875

[8] Mildenhall B, Srinivasan P P, Tancik M, Barron J T, Ramamoorthi R, Ng R . NeRF: representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021, 65( 1): 99–106

[9] Kerbl B, Kopanas G, Leimkuehler T, Drettakis G . 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023, 42( 4): 139

[10] Cui Z, Gu L, Sun X, Ma X, Qiao Y, Harada T. Aleth-NeRF: Low-light condition view synthesis with concealing fields. 2023, arXiv preprint arXiv: 2303.05807

[11] Cui Z, Chu X, Harada T. Luminance-GS: Adapting 3D Gaussian splatting to challenging lighting conditions with view-adaptive curve adjustment. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025, 26472−26482

[12] Ye S, Dong Z H, Hu Y, Wen Y H, Liu Y J . Gaussian in the dark: real-time view synthesis from inconsistent dark images using Gaussian splatting. Computer Graphics Forum, 2024, 43( 7): e15213

[13] Woodham R J . Photometric method for determining surface orientation from multiple images. Optical Engineering, 1980, 19( 1): 139–144

[14] Santo H, Samejima M, Sugano Y, Shi B, Matsushita Y . Deep photometric stereo networks for determining surface normal and reflectances. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44( 1): 114–128

[15] Ikehata S. Scalable, detailed and mask-free universal photometric stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 13198−13207

[16] Tiwari A, Sutariya M, Raman S. LIPIDS: Learning-based illumination planning in discretized (light) space for photometric stereo. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2025, 650−659

[17] Zhou Z, Wu Z, Tan P. Multi-view photometric stereo with spatially varying isotropic materials. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1482−1489

[18] Chen G, Han K, Shi B, Matsushita Y, Wong K Y K K. Self-calibrating deep photometric stereo networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 8731−8739

[19] Brument B, Bruneau R, Quéau Y, Mélou J, Lauze F B, Durou J D, Calvet L. RNb-NeuS: Reflectance and normal-based multi-view 3D reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 5230−5239

[20] Brahimi M, Haefner B, Ye Z, Goldluecke B, Cremers D. Sparse views, near light: a practical paradigm for uncalibrated point-light photometric stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 11862−11872

[21] Kaya B, Kumar S, Oliveira C, Ferrari V, Van Gool L. Multi-view photometric stereo revisited. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023, 3125−3134

[22] Yang W, Chen G, Chen C, Chen Z, Wong K Y K. PS-NeRF: neural inverse rendering for multi-view photometric stereo. In: Proceedings of the 17th European Conference on Computer Vision. 2022, 266−284

[23] Chen Y, Liang B, Guo H, Cheng Y, Zhao J, Weng D . PS-GS: Gaussian splatting for multi-view photometric stereo. Computers & Graphics, 2026, 136: 104561

[24] Zhu C, Liu X, Xu K, Yi R . A survey on 3D editing based on NeRF and 3DGS. Frontiers of Computer Science, 2026, 20( 4): 2004701

[25] Srinivasan P P, Deng B, Zhang X, Tancik M, Mildenhall B, Barron J T. NeRV: neural reflectance and visibility fields for relighting and view synthesis. In: Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 7495−7504

[26] Zeng C, Chen G, Dong Y, Peers P, Wu H, Tong X. Relighting neural radiance fields with shadow and highlight hints. In: Proceedings of ACM SIGGRAPH 2023 Conference. 2023, 73

[27] Zhu X, Yi R, Wen X, Zhu C, Xu K . Relighting scenes with object insertions in neural radiance fields. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35( 7): 6787–6802

[28] Yi R, Zhu C, Xu K. Weakly-supervised single-view image relighting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023, 8402−8411

[29] Zhang K, Luan F, Li Z, Snavely N. IRON: Inverse rendering by optimizing neural SDFs and materials from photometric images. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 5555−5564

[30] Liang Z, Zhang Q, Feng Y, Shan Y, Jia K. GS-IR: 3D Gaussian splatting for inverse rendering. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 21644−21653

[31] Gao J, Gu C, Lin Y, Li Z, Zhu H, Cao X, Zhang L, Yao Y. Relightable 3D Gaussians: realistic point cloud relighting with BRDF decomposition and ray tracing. In: Proceedings of the 18th European Conference on Computer Vision. 2024, 73−89

[32] Jiang Y, Tu J, Liu Y, Gao X, Long X, Wang W, Ma Y. GaussianShader: 3D Gaussian splatting with shading functions for reflective surfaces. In: Proceedings of 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 5322−5332

[33] Zhu Z L, Wang B, Yang J. GS-ROR: 3D Gaussian splatting for reflective object relighting via SDF priors. 2024, arXiv preprint arXiv: 2406.18544

[34] Kaleta J, Kania K, Trzcinski T, Kowalski M. LumiGauss: relightable Gaussian splatting in the wild. In: Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision. 2025, 1−10

[35] Xie M, Cai H, Shah S, Xu Y, Feng B Y, Huang J, Metzler C A. Flash-Splat: 3D reflection removal with flash cues and Gaussian splats. In: Proceedings of the 18th European Conference on Computer Vision. 2024, 122−139

[36] Bi Z, Zeng Y, Zeng C, Pei F, Feng X, Zhou K, Wu H. GS³: efficient relighting with triple Gaussian splatting. In: Proceedings of SIGGRAPH Asia 2024 Conference Papers. 2024, 12

[37] Huang B, Yu Z, Chen A, Geiger A, Gao S. 2D Gaussian splatting for geometrically accurate radiance fields. In: Proceedings of ACM SIGGRAPH 2024 Conference Papers. 2024, 32

[38] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P . Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13( 4): 600–612

[39] Zhang R, Isola P, Efros A A, Shechtman E, Wang O. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 586−595

RIGHTS & PERMISSIONS

Higher Education Press