DWT-3DRec: DeepJSCC-based wireless transmission for efficient 3D scene reconstruction using CityNeRF^✩

Abstract

The Unmanned Aerial Vehicle (UAV)-assisted sensing-transmission-computing integrated system plays a vital role in emergency rescue scenarios involving damaged infrastructure. To tackle the challenges of data transmission and enable timely rescue decision-making, we propose DWT-3DRec-an efficient wireless transmission model for 3D scene reconstruction. This model leverages MobileNetV2 to extract image and pose features, which are transmitted through a Dual-path Adaptive Noise Modulation network (DANM). Moreover, we introduce the Gumbel Channel Masking Module (GCMM), which enhances feature extraction and improves reconstruction reliability by mitigating the effects of dynamic noise. At the ground receiver, the Multi-scale Deep Source-Channel Coding for 3D Reconstruction (MDS-3DRecon) framework integrates Deep Joint Source-Channel Coding (DeepJSCC) with Cityscale Neural Radiance Fields (CityNeRF). It adopts a progressive close-view training strategy and incorporates an Adaptive Fusion Module (AFM) to achieve high-precision scene reconstruction. Experimental results demonstrate that DWT-3DRec significantly outperforms the Joint Photographic Experts Group (JPEG) standard in transmitting image and pose data, achieving an average loss as low as 0.0323 and exhibiting strong robustness across a Signal-to-Noise Ratio (SNR) range of 5-20 dB. In large-scale 3D scene reconstruction tasks, MDS-3DRecon surpasses Multum in Parvo Neural Radiance Fields (Mip-NeRF) and Bungee Neural Radiance Field (BungeeNeRF), achieving a Peak Signal-to-Noise Ratio (PSNR) of 24.921 dB and a reconstruction loss of 0.188. Ablation studies further confirm the essential roles of GCMM, DANM, and AFM in enabling high-fidelity 3D reconstruction.

Keywords

DeepJSCC / CityNeRF / Multi-scale / 3D reconstruction / Integrated sensing-transmission-computation

References

[1] S.A.H. Mohsan, N.Q.H. Othman, Y. Li, M.H. Alsharif, M.A. Khan, Unmanned aerial vehicles (UAVs): practical aspects, applications, open challenges, security issues, and future trends, Intell. Serv. Robot. 16 (1) (2023) 109-137.

[2] Z. Jia, B. Wang, C. Chen, Drone-nerf: efficient nerf based 3D scene reconstruction for large-scale drone survey, Image Vis. Comput. 143 (2024) 104920.

[3] Z. Lyu, G. Zhu, J. Xu, B. Ai, S. Cui, Semantic communications for image recovery and classification via deep joint source and channel coding, IEEE Trans. Wirel. Commun. 23 (8) (2024) 8388-8404.

[4] Y. Shao, Q. Cao, D. Gündüz, A theory of semantic communication, IEEE Trans. Mob. Comput. 23 (12) (2024) 12211-12228.

[5] H. Yuan, W. Xu, Y. Wang, X. Wang, Channel-blind joint source-channel coding for wireless image transmission, Sensors 24 (12) (2024) 4005.

[6] C. Bian, Y. Shao, D. Gündüz, Wireless point cloud transmission, in: 2024 IEEE 25th International Workshop on Signal Processing Advances in Wireless Communications, IEEE, 2024, pp. 851-855.

[7] H. Zhang, C. Xie, H. Toriya, H. Shishido, I. Kitahara, Vehicle localization in a com- pleted city-scale 3d scene using aerial images and an on-board stereo camera, Remote Sens. 15 (15) (2023) 3871.

[8] Y. Xiangli, L. Xu, X. Pan, N. Zhao, A. Rao, C. Theobalt, B. Dai, D. Lin, Bungeenerf: progressive neural radiance field for extreme multi-scale scene rendering,in: Euro- pean Conference on Computer Vision, Springer, 2022, pp. 106-122.

[9] X. Kang, B. Song, J. Guo, Z. Qin, F.R. Yu, Task-oriented image transmission for scene classification in unmanned aerial systems, IEEE Trans. Commun. 70 (8) (2022) 5181-5192.

[10] S. Yao, K. Niu, S. Wang, J. Dai, Semantic coding for text transmission: an iterative design, IEEE Trans. Cogn. Commun. Netw. 8 (4) (2022) 1594-1603.

[11] S. Wang, J. Dai, Z. Liang, K. Niu, Z. Si, C. Dong, X. Qin, P. Zhang, Wireless deep video semantic transmission, IEEE J. Sel. Areas Commun. 41 (1) (2022) 214-229.

[12] X. Zhou, D. Huang, Z. Qi, L. Zhang, T. Jiang, A multi-scale spatial-temporal network for wireless video transmission, preprint, arXiv:2411.09936.

[13] T.-Y. Tung, D. Gündüz, Deepwive: deep-learning-aided wireless video transmission, IEEE J. Sel. Areas Commun. 40 (9) (2022) 2570-2583.

[14] M. Yang, H.-S. Kim,Deep joint source-channel coding for wireless image transmis- sion with adaptive rate control, in: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2022, pp. 5193-5197.

[15] S. Wang, J. Dai, Z. Liang, K. Niu, Z. Si, C. Dong, X. Qin, P. Zhang, Wireless deep video semantic transmission, IEEE J. Sel. Areas Commun. 41 (1) (2023) 214-229.

[16] K. Yang, S. Wang, J. Dai, X. Qin, K. Niu, P. Zhang, Swinjscc: taming swin trans- former for deep joint source-channel coding, IEEE Trans. Cogn. Commun. Netw. 11 (1) (2025) 90-104.

[17] J. Xu, B. Ai, W. Chen, A. Yang, P. Sun, M. Rodrigues, Wireless image transmission using deep source channel coding with attention modules, IEEE Trans. Circuits Syst. Video Technol. 32 (4) (2021) 2315-2328.

[18] S. Xie, H. He, H. Li, S. Song, J. Zhang, Y.-J.A. Zhang, K.B. Letaief, Deep learning-based adaptive joint source-channel coding using hypernetworks, in: 2024 IEEE Interna- tional Mediterranean Conference on Communications and Networking, IEEE, 2024, pp. 191-196.

[19] E. Bourtsoulatze, D. Burth Kurka, D. Gündüz, Deep joint source-channel coding for wireless image transmission, IEEE Trans. Cogn. Commun. Netw. 5 (3) (2019) 567-579.

[20] H. Luo, J. Zhang, X. Liu, L. Zhang, J. Liu, Large-scale 3D reconstruction from multi- view imagery: a comprehensive review, Remote Sens. 16 (5) (2024) 773.

[21] N. Amir, Z. Zainuddin, Z. Tahir, 3D reconstruction with SFM-MVS method for food volume estimation, Int. J. Comput. Digit. Syst. 16 (1) (2024) 1-11.

[22] B. Mildenhall, P.P. Srinivasan, M. Tancik, J.T. Barron, R. Ramamoorthi, R. Ng, Nerf: representing scenes as neural radiance fields for view synthesis, Commun. ACM 65 (1) (2021) 99-106.

[23] S. Athar, Z. Xu, K. Sunkavalli, E. Shechtman, Z. Shu, Rignerf: fully controllable neural 3d portraits,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 2022, pp. 20364-20373.

[24] B. Yang, C. Bao, J. Zeng, H. Bao, Y. Zhang, Z. Cui, G. Zhang, Neumesh: learning disentangled neural mesh-based implicit field for geometry and texture editing,in: European Conference on Computer Vision, Springer, 2022, pp. 597-614.

[25] S. Liu, X. Zhang, Z. Zhang, R. Zhang, J.-Y. Zhu, B. Russell, Editing conditional radi-ance fields, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE/CVF, 2021, pp. 5773-5783.

[26] B. Mildenhall, P. Hedman, R. Martin-Brualla, P.P. Srinivasan, J.T. Barron, Nerf in the dark: high dynamic range view synthesis from noisy raw images,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 2022, pp. 16190-16199.

[27] X. Huang, Q. Zhang, Y. Feng, H. Li, X. Wang, Q. Wang, Hdr-nerf: high dynamic range neural radiance fields,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 2022, pp. 18398-18408.

[28] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M.R. Oswald, M. Polle-feys, Nice-slam: neural implicit scalable encoding for slam,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 2022, pp. 12786-12796.

[29] X. Yang, H. Li, H. Zhai, Y. Ming, Y. Liu, G. Zhang, Vox-fusion: dense tracking and mapping with voxel-based neural implicit representation, in: 2022 IEEE Interna- tional Symposium on Mixed and Augmented Reality, IEEE, 2022, pp. 499-507.

[30] K. Rematas, A. Liu, P.P. Srinivasan, J.T. Barron, A. Tagliasacchi, T. Funkhouser, V. Ferrari, Urban radiance fields, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 2022, pp. 12932-12942.

[31] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P.P. Srinivasan, J.T. Barron, H. Kretzschmar, Block-nerf: scalable large scene neural view synthesis,in:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 2022, pp. 8248-8258.

[32] R. Martin-Brualla, N. Radwan, M.S. Sajjadi, J.T. Barron, A. Dosovitskiy, D. Duck- worth, Nerf in the wild: neural radiance fields for unconstrained photo collections,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, IEEE/CVF, 2021, pp. 7210-7219.

[33] K. Zhang, G. Riegler, N. Snavely, V. Koltun, Nerf++: analyzing and improving neural radiance fields, preprint, arXiv:2010.07492.

[34] R. Martin-Brualla, N. Radwan, M.S. Sajjadi, J.T. Barron, A. Dosovitskiy, D. Duck- worth, Nerf in the wild: neural radiance fields for unconstrained photo collections,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, IEEE/CVF, 2021, pp. 7210-7219.

[35] W. Chen, Y. Chen, Q. Yang, C. Huang, Q. Wang, Z. Zhang,Deep joint source-channel coding for wireless image transmission with entropy-aware adaptive rate control, in: GLOBECOM 2023-2023 IEEE Global Communications Conference, IEEE, 2023, pp. 2239-2244.

[36] J.T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, P.P. Srinivasan, Mip-nerf: a multiscale representation for anti-aliasing neural radiance fields,in:Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, IEEE/CVF, 2021, pp. 5855-5864.