RFG-GS: renderability field guided high-fidelity reconstruction for 3D buildings

Abstract

Reconstruction of large-scale 3D scenes using drone scanning is a significant research focus in computer graphics and 3D vision. To address limitations such as inadequate data coverage, low modeling accuracy, and insufficient rendering details during horizontal flight, we propose RFG-GS, a novel “structural scanning – fine reconstruction” framework based on 3D Gaussian Splatting (3D-GS). This approach guides drone scanning viewpoint planning and facilitates high-fidelity active 3D reconstruction of unknown complex building scenes, achieving full coverage through renderability fields. First, RGB images from initial drone oblique photography are used to extract color features of the underlying 3D buildings. After segmenting and clustering the structures, candidate viewpoints for a subsequent close-range scanning pass are generated at safe distances. A renderability field for this viewpoint set is then established, incorporating resolution, angular consistency, and geometric reliability metrics. Second, the renderability values of candidate viewpoints are computed to determine an optimal viewpoint set, which guides the secondary data collection. Finally, a KNN algorithm based on KD-Tree optimizes adaptive density control during gaussian sphere splitting, significantly improving splitting accuracy. Experimental results demonstrate that RFG-GS increases the structural similarity index (SSIM) of novel view synthesis images by approximately 3.0% compared with state-of-the-art (SOTA) methods. Our renderability-guided scheme provides an accurate, efficient, and robust solution for 3D building modeling.

Graphical abstract

Keywords

drone scanning / active 3D reconstruction / renderability field / 3D Gaussian Splatting / adaptive density control / gaussian sphere splitting

References

[1] Smith N, Moehrle N, Goesele M, Heidrich W . Aerial path planning for urban scene reconstruction: a continuous optimization method and benchmark. ACM Transactions on Graphics, 2018, 37( 6): 183

[2] Snavely N, Seitz S M, Szeliski R. Photo tourism: exploring photo collections in 3D. In: Proceedings of the ACM SIGGRAPH 2006 Papers. 2006, 835−846

[3] Connolly C. The determination of next best views. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). 1985, 432−435

[4] Chen X, Li Q, Wang T, Xue T, Pang J. GenNBV: generalizable next-best-view policy for active 3D reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, 16436−16445

[5] Liu Q, Jiang Y, Li Y. A sampling-based next-best-view path planner for environment exploration. In: Proceedings of the 9th International Conference on Mechatronics and Robotics Engineering (ICMRE). 2023, 128−132

[6] Danilova E V . Concept of the collage city of colin Rowe and Fred Ketter in the context of contemporary architecture theory. Urban Construction and Architecture, 2021, 11( 1): 103–109

[7] Kerbl B, Kopanas G, Leimkühler T, Drettakis G . 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 2023, 42( 4): 139

[8] Zeng R, Zhao W, Liu Y J. PC-NBV: a point cloud based deep network for efficient next best view planning. In: Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2020, 7050−7057

[9] Pan S, Hu H, Wei H . SCVP: learning one-shot view planning via set covering for unknown object reconstruction. IEEE Robotics and Automation Letters, 2022, 7( 2): 1463–1470

[10] Lee I D, Seo J H, Kim Y M, Choi J, Han S, Yoo B . Automatic pose generation for robotic 3-D scanning of mechanical parts. IEEE Transactions on Robotics, 2020, 36( 4): 1219–1238

[11] Yi Z, Xie K, Lyu J, Gong M, Huang H. Where to render: studying renderability for IBR of large-scale scenes. In: Proceedings of 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). 2023, 356−366

[12] Mildenhall B, Srinivasan P P, Tancik M, Barron J T, Ramamoorthi R, Ng R . NeRF: representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2022, 65( 1): 99–106

[13] Garbin S J, Kowalski M, Johnson M, Shotton J, Valentin J. FastNeRF: high-fidelity neural rendering at 200FPS. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021, 14326−14335

[14] Tang J, Zhou H, Chen X, Hu T, Ding E, Wang J, Zeng G. Delicate textured mesh recovery from NeRF via adaptive surface refinement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023, 17693−17703

[15] Wang Y, Funk N, Ramezani M, Papatheodorou S, Popović M, Camurri M, Leutenegger S, Fallon M. Elastic and efficient LiDAR reconstruction for large-scale exploration tasks. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). 2021, 5035−5041

[16] Seitz S M, Curless B, Diebel J, Scharstein D, Szeliski R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2006, 519−528

[17] Hahsler M, Piekenbrock M, Doran D . dbscan: fast density-based clustering with R. Journal of Statistical Software, 2019, 91( 1): 1–30

[18] Edelsbrunner H, Kirkpatrick D, Seidel R . On the shape of a set of points in the plane. IEEE Transactions on Information Theory, 1983, 29( 4): 551–559

[19] Zhang Y, Kang B, Hooi B, Yan S, Feng J . Deep long-tailed learning: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45( 9): 10795–10816

[20] Chen S E, Williams L. View interpolation for image synthesis. In: Whitton M C, ed. Seminal Graphics Papers: Pushing the Boundaries, Volume 2. New York: ACM, 2023, 45

[21] Buehler C, Bosse M, McMillan L, Gortler S, Cohen M. Unstructured lumigraph rendering. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 2001, 425−432

[22] Debevec P E, Taylor C J, Malik J. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In: Whitton M C, ed. Seminal Graphics Papers: Pushing the Boundaries, Volume 2. New York: ACM, 2023, 49

[23] Blender Online Community. Blender (version 2.8), 2019. Computer Software(查阅网上资料, 未找到本条文献信息, 请确认)

[24] Barron J T, Mildenhall B, Verbin D, Srinivasan P P, Hedman P. Mip-NeRF 360: unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2022, 5460−5469

[25] Ye Z, Li W, Liu S, Qiao P, Dou Y. AbsGS: recovering fine details in 3D Gaussian splatting. In: Proceedings of the 32nd ACM International Conference on Multimedia (ACM MM). 2024, 1053−1061

[26] Huang B, Yu Z, Chen A, Geiger A, Gao S. 2D Gaussian splatting for geometrically accurate radiance fields. In: Proceedings of the ACM SIGGRAPH 2024 Conference Papers. 2024, 32

[27] Mallick S S, Goel R, Kerbl B, Steinberger M, Carrasco F V, De La Torre F. Taming 3DGS: high-quality radiance fields with limited resources. In: Proceedings of 2024 SIGGRAPH Asia Conference Papers. 2024, 2

[28] Gao A, Guo L, Chen T, Wang Z, Tai Y, Yang J, Zhang Z. EasySplat: view-adaptive learning makes 3D Gaussian splatting easy. In: Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). 2025, 1−6

[29] Niemeyer M, Barron J T, Mildenhall B, Sajjadi M S M, Geiger A, Radwan N. RegNeRF: regularizing neural radiance fields for view synthesis from sparse inputs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, 5470−5480

[30] Avidan S, Shashua A. Novel view synthesis in tensor space. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1997, 1034−1040

[31] Liu X, Ji Z, Zhou H, Zhang Z, Tao P, Xi K, Chen L, Marcato Junior J. An object-oriented UAV 3D path planning method applied in cultural heritage documentation. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2022, V-1–2022: 33−40

[32] Zhang H, Yao Y, Xie K, Fu C W, Zhang H, Huang H . Continuous aerial path planning for 3D urban scene reconstruction. ACM Transactions on Graphics, 2021, 40( 6): 225

[33] Gong H, Huang B, Jia B, Dai H . Modeling power consumptions for multirotor UAVs. IEEE Transactions on Aerospace and Electronic Systems, 2023, 59( 6): 7409–7422

RIGHTS & PERMISSIONS

Higher Education Press