CDFNet: Cross-Modal Deep Fusion for Monocular 3D Semantic Scene Completion

Xianjing Cheng , Lintai Wu , Jie Wen , Tianhao Peng , Yong Xu , Zairong Wei

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) : 1161 -1178.

PDF (5304KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (4) :1161 -1178. DOI: 10.1049/cit2.70124
ORIGINAL RESEARCH
research-article
CDFNet: Cross-Modal Deep Fusion for Monocular 3D Semantic Scene Completion
Author information +
History +
PDF (5304KB)

Abstract

Semantic scene completion (SSC) aims to predict the semantic occupancy and geometry of 3D scenes. Recently, most studies focus on camera-based approaches due to the rich visual cues of images and the cost-effectiveness of cameras. However, these methods usually lack efficient fusion and fine-grained processing of cross-modal semantic information, resulting in sub-optimal performance. To address these issues, we propose a novel cross-modal semantic deep fusion framework. Unlike previous approaches, our method effectively integrates 2D textural, 2D spatial and 3D geometric knowledge from three different modalities to reconstruct complete 3D scenes. Specifically, we employ two encoders to extract 2D textural and 2D spatial features from RGB images and depth maps, which are then fused and lifted into 3D space via our tailored cross-modal semantic fusion module. In contrast to previous methods that encompass extensive redundant 3D voxel features, we design a lightweight voxel feature filter to efficiently eliminate these redundancies. Furthermore, 3D geometric features are extracted from the point cloud derived from the depth map. The 3D features from multiple modalities are deeply fused and further refined by a sparse-to-dense voxel completion module, which effectively enriches the semantic information. Besides, we propose a new evaluation metric more suitable for assessing the SSC task with class imbalance issues in the dataset. Extensive experiments show that our method achieves the state of the art in camera-based semantic scene completion. We will release the source code publicly.

Keywords

class imbalance / cross-modal learning / feature fusion / occupancy prediction / semantic scene completion

Cite this article

Download citation ▾
Xianjing Cheng, Lintai Wu, Jie Wen, Tianhao Peng, Yong Xu, Zairong Wei. CDFNet: Cross-Modal Deep Fusion for Monocular 3D Semantic Scene Completion. CAAI Transactions on Intelligence Technology, 2026, 11 (4) : 1161-1178 DOI:10.1049/cit2.70124

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

X. Cheng, L. Wu, J. Hou, Z. Hu, J. Wen, and Y. Xu, “ Arbitrary-Scale Point Cloud Upsampling via Enhanced Geometric Spatial Consistency,” CAAI Transactions on Intelligence Technology (2025).

[2]

Y. Zhao, W. Nie, J. Nie, Y. Zhang, and B. Wang, “ Rjan: Region-Based Joint Attention Network for 3d Shape Recognition,” CAAI Transactions on Intelligence Technology 10, no. 2 (2025): 460-473, https://doi.org/10.1049/cit2.12388.

[3]

R. Chang, Y. Ma, T. Hao, W. Wang, and W. Nie, “ 3d Shape Knowledge Graph for Cross-Domain 3d Shape Retrieval,” CAAI Transactions on Intelligence Technology 9, no. 5 (2024): 1199-1216, https://doi.org/10.1049/cit2.12326.

[4]

L. Hong, M. H. Modirrousta, M. Hossein Nasirpour, et al., “ Gan-lstm-3d: An Efficient Method for Lung Tumour 3d Reconstruction Enhanced by Attention-Based Lstm,” CAAI Transactions on Intelligence Technology (2023).

[5]

N. Yu, H. Li, Q. Xu, O. Sie, and E. Firdaous, “ 3d Reconstruction and Defect Pattern Recognition of Bonding Wire Based on Stereo Vision,” CAAI Transactions on Intelligence Technology 9, no. 2 (2024): 348-364, https://doi.org/10.1049/cit2.12240.

[6]

B. Wen, H. Xie, Z. Chen, F. Hong, and Z. Liu, “ 3d Scene Generation: A Survey,” arXiv preprint arXiv:2505.05474 (2025).

[7]

L. Roldao, R. De Charette, and A. Verroust-Blondet, “ 3d Semantic Scene Completion: A Survey,” International Journal of Computer Vision 130, no. 8 (2022): 1978-2005, https://doi.org/10.1007/s11263-021-01504-5.

[8]

L. Wu, X. Cheng, Y. Xu, H. Zeng, and J. Hou, “ Unsupervised 3d Point Cloud Completion Via Multi-View Adversarial Learning,” IEEE Transactions on Visualization and Computer Graphics 31, no. 10 (2025): 7890-7905, https://doi.org/10.1109/tvcg.2025.3559340.

[9]

X. Cheng, L. Wu, Z. Wang, J. Hou, J. Wen, and Y. Xu, “ Pvnet: Point-Voxel Interaction LiDAR Scene Upsampling via Diffusion Models,” IEEE Transactions on Image Processing 34 (2025): 6895-6910, https://doi.org/10.1109/tip.2025.3620666.

[10]

L. Wu, J. Hou, L. Song, and Y. Xu, “ 3d Shape Completion on Unseen Categories: A Weakly-Supervised Approach,” IEEE Transactions on Visualization and Computer Graphics 31, no. 9 (2025): 5547-5561, https://doi.org/10.1109/tvcg.2024.3459018.

[11]

S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser, “ Semantic Scene Completion From a Single Depth Image,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2017), 1746-1754.

[12]

L. Roldao, R. de Charette, and A. Verroust-Blondet, “ Lmscnet: Lightweight Multiscale 3d Semantic Completion,” in Proceedings of International Conference on 3D Vision (IEEE, 2020), 111-119.

[13]

T. Kühner and J. Kümmerle, “ Large-Scale Volumetric Scene Reconstruction Using Lidar,” in 2020 IEEE International Conference on Robotics and Automation (IEEE, 2020), 6261-6267.

[14]

L. Wu, Q. Zhang, J. Hou, and Y. Xu, “ Leveraging Single-View Images for Unsupervised 3d Point Cloud Completion,” IEEE Transactions on Multimedia 27 (2025): 940-953, https://doi.org/10.1109/tmm.2023.3340892.

[15]

A.-Q. Cao and R. De Charette, “ Monoscene: Monocular 3d Semantic Scene Completion,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2022), 3991-4001.

[16]

S. Popov, P. Bauszat, and V. Ferrari, “ Corenet: Coherent 3d Scene Reconstruction From a Single Rgb Image,” in Proceedings of European Conference on Computer Vision (Springer, 2020), 366-383.

[17]

Y. Li, Z. Yu, C. Choy, et al., “ Voxformer: Sparse Voxel Transformer for Camera-based 3d Semantic Scene Completion,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2023), 9087-9098.

[18]

H. Jiang, T. Cheng, N. Gao, et al., “ Symphonize 3d Semantic Scene Completion With Contextual Instance Queries,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2024), 20258-20267.

[19]

J. Mei, Y. Yang, M. Wang, et al., “ Camera-Based 3d Semantic Scene Completion With Sparse Guidance Network,” IEEE Transactions on Image Processing (2024).

[20]

Z. Yu, R. Zhang, J. Ying, et al., “ Context and Geometry Aware Voxel Transformer for Semantic Scene Completion,” arXiv preprint arXiv:2405.13675 (2024).

[21]

J. Hu, L. Shen, and G. Sun, “ Squeeze-And-Excitation Networks,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2018), 7132-7141.

[22]

X. Zhu, H. Zhou, T. Wang, et al., “ Cylindrical and Asymmetrical 3d Convolution Networks for LiDAR Segmentation,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2021), 9939-9948.

[23]

J. Behley, M. Garbade, A. Milioto, et al., “ SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences,” in Proceedings of IEEE International Conference on Computer Vision, (2019).

[24]

S. Li, C. Zou, Y. Li, X. Zhao, and Y. Gao, “ Attention-Based Multi-Modal Fusion Network for Semantic Scene Completion,” Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 7 (2020): 11402-11409, https://doi.org/10.1609/aaai.v34i07.6803.

[25]

J. Li, K. Han, P. Wang, Y. Liu, and X. Yuan, “ Anisotropic Convolutional Networks for 3d Semantic Scene Completion,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2020), 3351-3359.

[26]

S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller, “ Zoedepth: Zero-Shot Transfer by Combining Relative and Metric Depth,” arXiv preprint arXiv:2302.12288 (2023).

[27]

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “ Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2024), 10371-10381.

[28]

S. Patni, A. Agarwal, and C. Arora, “ Ecodepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2024), 28285-28295.

[29]

J. Song, C. Meng, and S. Ermon, “ Denoising Diffusion Implicit Models,” arXiv preprint arXiv:2010.02502 (2020).

[30]

J. Ho, A. Jain, and P. Abbeel, “ Denoising Diffusion Probabilistic Models,” in Proceedings of Advanced Neural Information Processing System, Vol. 33, (2020), 6840-6851.

[31]

F. Zhang, V. Prisacariu, R. Yang, and P. H. Torr, “ Ga-Net: Guided Aggregation Net for End-to-End Stereo Matching,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2019), 185-194.

[32]

H. Xu and J. Zhang, “ Aanet: Adaptive Aggregation Network for Efficient Stereo Matching,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2020), 1959-1968.

[33]

F. Shamsafar, S. Woerz, R. Rahim, and A. Zell, “ Mobilestereonet: Towards Lightweight Deep Networks for Stereo Matching,” in Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision, (2022), 2417-2426.

[34]

L. Lipson, Z. Teed, and J. Deng, “ Raft-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching,” in Proceedings of International Conference on 3D Vision (IEEE, 2021), 218-227.

[35]

K. Li, L. Wang, Y. Zhang, K. Xue, S. Zhou, and Y. Guo, “ Los: Local Structure-Guided Stereo Matching,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2024), 19746-19756.

[36]

X. Wang, G. Xu, H. Jia, and X. Yang, “ Selective-Stereo: Adaptive Frequency Information Selection for Stereo Matching,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2024), 19701-19710.

[37]

J. Zhang, H. Zhao, A. Yao, Y. Chen, L. Zhang, and H. Liao, “ Efficient Semantic Scene Completion Network With Spatial Group Convolution,” in Proceedings of European Conference on Computer Vision, (2018), 733-749.

[38]

X. Yang, H. Zou, X. Kong, et al., “ Semantic Segmentation-Assisted Scene Completion for LiDAR Point Clouds,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2021), 3555-3562.

[39]

Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon, “ Detr3d: 3d Object Detection From Multi-View Images Via 3d-to-2d Queries,” in Conference on Robot Learning (PMLR, 2022), 180-191.

[40]

Z. Chen and H. Zhang, “ Learning Implicit Fields for Generative Shape Modeling,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2019), 5939-5948.

[41]

Z. Xia, Y. Liu, X. Li, et al., “ Scpnet: Semantic Scene Completion on Point Cloud,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2023), 17642-17651.

[42]

X. Yan, J. Gao, J. Li, et al., “ Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors From Scene Completion,” Proceedings of the AAAI Conference on Artificial Intelligence 35, no. 4 (2021): 3101-3109, https://doi.org/10.1609/aaai.v35i4.16419.

[43]

Z. Li, W. Wang, H. Li, et al., “ Bevformer: Learning Bird’s-Eye-View Representation From Multi-Camera Images Via Spatiotemporal Transformers,” in Proceedings of European Conference on Computer Vision (Springer, 2022), 1-18.

[44]

Y. Huang, W. Zheng, Y. Zhang, J. Zhou, and J. Lu, “ Tri-Perspective View for Vision-Based 3d Semantic Occupancy Prediction,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2023), 9223-9232.

[45]

Y. Wang, I. Skorokhodov, and P. Wonka, “ Pet-Neus: Positional Encoding Tri-Planes for Neural Surfaces,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2023), 12598-12607.

[46]

Y. Zhang, Z. Zhu, and D. Du, “ Occformer: Dual-Path Transformer for Vision-Based 3d Semantic Occupancy Prediction,” in Proceedings of IEEE International Conference on Computer Vision, (2023), 9433-9443.

[47]

Y. Wang and C. Tong, “ H2gformer: Horizontal-To-Global Voxel Transformer for 3d Semantic Scene Completion,” Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 6 (2024): 5722-5730, https://doi.org/10.1609/aaai.v38i6.28384.

[48]

M. Tan and Q. Le, “ Efficientnet: Rethinking Model Scaling for Convolutional Neural Networks,” in Proceedings of International Conference on Machine Learning (PMLR, 2019), 6105-6114.

[49]

J. Philion and S. Fidler, “ Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3d,” in Proceedings of European Conference on Computer Vision (Springer, 2020), 194-210.

[50]

M. Ye, R. Wan, S. Xu, T. Cao, and Q. Chen, “ Efficient Point Cloud Segmentation With Geometry-Aware Sparse Networks,” in European Conference on Computer Vision (Springer, 2022), 196-212.

[51]

M. Berman, A. R. Triki, and M. B. Blaschko, “ The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2018), 4413-4421.

[52]

A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, “ Rangenet++: Fast and Accurate LiDAR Semantic Segmentation,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2019), 4213-4220.

[53]

M. Menze and A. Geiger, “ Object Scene Flow for Autonomous Vehicles,” in Proceedings of IEEE Conference Computer Vision and Pattern Recognition, (2015).

[54]

Y. Liao, J. Xie, and A. Geiger, “ Kitti-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2d and 3d,” IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

[55]

J. M. Johnson and T. M. Khoshgoftaar, “ Survey on Deep Learning With Class Imbalance,” Journal of Big Data 6, no. 1 (2019): 1-54, https://doi.org/10.1186/s40537-019-0192-5.

[56]

X. Wang, X. Wu, S. Wang, L. Kong, and Z. Zhao, “ Monocular Semantic Scene Completion via Masked Recurrent Networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2025), 24811-24822.

[57]

X. Liu, H. Xie, S. Zhang, et al., “ 2d Semantic-Guided Semantic Scene Completion,” International Journal of Computer Vision 133, no. 3 (2025): 1306-1325, https://doi.org/10.1007/s11263-024-02244-y.

[58]

Y. Huang, A. Thammatadatrakoon, W. Zheng, Y. Zhang, D. Du, and J. Lu, “ Gaussianformer-2: Probabilistic Gaussian Superposition for Efficient 3d Occupancy Prediction,” in Proceedings of the Computer Vision and Pattern Recognition Conference, (2025), 27477-27486.

[59]

Y. Xue, H. Pi, J. Zhang, et al., “ Sdformer: Vision-Based 3d Semantic Scene Completion via Sam-Assisted Dual-Channel Voxel Transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2025), 26837-26847.

[60]

J. Yao and J. Zhang, “ Depthssc: Depth-Spatial Alignment and Dynamic Voxel Resolution for Monocular 3d Semantic Scene Completion,” arXiv preprint arXiv:2311.17084 7 (2023): 16, https://doi.org/10.48550/arXiv.2311.17084.

[61]

M. Wang, H. Pi, R. Li, Y. Qin, Z. Tang, and K. Li, “ Vlscene: Vision-Language Guidance Distillation for Camera-Based 3d Semantic Scene Completion,” Proceedings of the AAAI Conference on Artificial Intelligence 39, no. 8 (2025): 7808-7816, https://doi.org/10.1609/aaai.v39i8.32841.

[62]

H.-J. Lee, Y. J. Koh, H. Kim, H. Kim, Y. Lee, and J. Lee, “ Soap: Vision-Centric 3d Semantic Scene Completion With Scene-Adaptive Decoder and Occluded Region-Aware View Projection,” in Proceedings of the Computer Vision and Pattern Recognition Conference, (2025), 17145-17154.

[63]

H. Jiang, Z. Lou, L. Ding, et al., “ Defom-Stereo: Depth Foundation Model Based Stereo Matching,” in Proceedings of the Computer Vision and Pattern Recognition Conference, (2025), 21857-21867.

PDF (5304KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉