Content-aware robust semantic transmission of images over wireless channels with GANs^✩

Abstract

Semantic Communication (SemCom) can significantly reduce the transmitted data volume and keep robustness. Task-oriented SemCom of images aims to convey the implicit meaning of source messages correctly, rather than achieving precise bit-by-bit reconstruction. Existing image SemCom systems directly perform semantic encoding and decoding on the entire image, which has not considered the correlation between image content and downstream tasks or the adaptability to channel noise. To this end, we propose a content-aware robust SemCom framework for image transmission based on Generative Adversarial Networks (GANs). Specifically, the accurate semantics of the image are extracted by the semantic encoder, and divided into two parts for different downstream tasks: Regions of Interest (ROI) and Regions of Non-Interest (RONI). By reducing the quantization accuracy of RONI, the amount of transmitted data volume is reduced significantly. During the transmission process of semantics, a Signal-to-Noise Ratio (SNR) is randomly initialized, enabling the model to learn the average noise distribution. The experimental results demonstrate that by reducing the quantization level of RONI, transmitted data volume is reduced up to 60.53% compared to using globally consistent quantization while maintaining comparable performance to existing methods in downstream semantic segmentation tasks. Moreover, our model exhibits increased robustness with variable SNRs.

Keywords

Semantic communication / GANs / Image transmission / ROI

References

[1] W. Yang, H. Du, Z.Q. Liew, W.Y.B. Lim, Z. Xiong, D. Niyato, X. Chi, X. Shen, C. Miao, Semantic communications for future Internet: fundamentals, applications, and challenges, IEEE Commun. Surv. Tutor. 25 (1) (2023) 213-250.

[2] H. Xie, Z. Qin, A lite distributed semantic communication system for Internet of things, IEEE J. Sel. Areas Commun. 39 (2021) 142-153.

[3] H.Q. Xie, Z.J. Qin, G.Y. Li, B.H. Juang, Deep learning enabled semantic communi-cation systems, IEEE Trans. Signal Process. 69 (2021) 2663-2675.

[4] Y.N. Wang, M.Z. Chen, T. Luo, W. Saad, D. Niyato, H.V. Poor, S.G. Cui, Performance optimization for semantic communications: an attention-based reinforcement learn-ing approach, IEEE J. Sel. Areas Commun. 40 (9) (2022) 2598-2613.

[5] H. Tong, Z. Yang, S. Wang, Y. Hu, W. Saad, C. Yin, Federated learning based audio semantic communication over wireless networks, in: Proceedings of the 2021 IEEE Global Communications Conference (GLOBECOM), IEEE, 2021, pp. 1-6.

[6] E. Bourtsoulatze, D.B. Kurka, D. Gündüz, Deep joint source-channel coding for wire-less image transmission, IEEE Trans. Cogn.commun. Netw. 5 (3) (2019) 567-579.

[7] A. Li, X. Liu, G. Wang, P. Zhang, Domain knowledge driven semantic communication for image transmission over wireless channels, IEEE Wirel. Commun. Lett. 12 (1) (2023) 55-59.

[8] J. Xu, B. Ai, W. Chen, A. Yang, P. Sun, M. Rodrigues, Wireless image transmission using deep source channel coding with attention modules, IEEE Trans. Circuits Syst. Video Technol. 32 (4) (2022) 2315-2328.

[9] C. Dong, H. Liang, X. Xu, S. Han, B. Wang, P. Zhang, Semantic communication system based on semantic slice models propagation, IEEE J. Sel. Areas Commun. 41 (1) (2023) 202-213.

[10] W.F. Lo, N. Mital, H. Wu, D. Gündüz, Collaborative semantic communication for edge inference, IEEE Wirel. Commun. Lett. 12 (7) (2023) 1125-1129.

[11] D.B. Kurka, D. Gündüz, Bandwidth-agile image transmission with deep joint source-channel coding, IEEE Trans. Wirel.commun. 20 (12) (2021) 8081-8095.

[12] M. Yang, H.S. Kim, Deep joint source-channel coding for wireless image transmission with adaptive rate control, in: Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 5193-5197.

[13] T.Y. Tung, D. Gündüz, Deepwive: deep-learning-aided wireless video transmission, IEEE J. Sel. Areas Commun. 40 (9) (2022) 2570-2583.

[14] S. Wang, J. Dai, Z. Liang, K. Niu, Z. Si, C. Dong, X. Qin, P. Zhang, Wireless deep video semantic transmission, IEEE J. Sel. Areas Commun. 41 (1) (2023) 214-229.

[15] L. Galteri, M. Bertini, L. Seidenari, T. Uricchio, A. Del Bimbo, Increasing video per-ceptual quality with gans and semantic coding, in: Proceedings of the 28th ACM International Conference on Multimedia, IEEE, Association for Computing Machin-ery, 2020, pp. 862-870.

[16] P. Jiang, C.K. Wen, S. Jin, G.Y. Li, Wireless semantic communications for video con-ferencing, IEEE J. Sel. Areas Commun. 41 (1) (2023) 230-244.

[17] J. Shao, Y. Mao, J. Zhang, Learning task-oriented communication for edge infer-ence: an information bottleneck approach, IEEE J. Sel. Areas Commun. 40 (1) (2022) 197-211.

[18] J. Shao, Y. Mao, J. Zhang, Task-oriented communication for multidevice cooperative edge inference, IEEE Trans. Wirel.commun. 22 (1) (2023) 73-87.

[19] K. Yang, S. Wang, K. Tan, J. Dai, D. Zhou, K. Niu, Resolution-adaptive source-channel coding for end-to-end wireless image transmission, in: Proceedings of the 2022 IEEE Global Communications Conference (GLOBECOM), IEEE, pp. 1460-1465.

[20] C. Cai, L. Chen, X. Zhang, Z. Gao, End-to-end optimized roi image compression, IEEE Trans. Image Process. 29 (2020) 3442-3457.

[21] J. Wu, C. Wu, Y. Lin, T. Yoshinaga, L. Zhong, X. Chen, Y. Ji, Semantic segmentation-based semantic communication system for image transmission, Digit.commun. Netw. 10 (3) (2024) 519-527.

[22] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, Commun. ACM 63 (11) (2020) 139-144.

[23] X. Mao, Q. Li, H. Xie, R.Y. Lau, Z. Wang, S.P. Smolley, Least squares generative adversarial networks, in: Proceeding of the 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, 2017, pp. 2813-2821.

[24] K. Simonyan, A. Zisserman,Very deep convolutional networks for large-scale image recognition, in:Proceedings of the 3rd International Conference on Learning Repre-sentations (ICLR), 2015, pp. 1-14.

[25] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Pro-ceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 770-778.

[26] Q. He, H. Yuan, D. Feng, B. Che, Z. Chen, X.-G. Xia, Robust semantic transmission of images with generative adversarial networks, in: Proceedings of the 2022 IEEE Global Communications Conference (GLOBECOM), IEEE, 2022, pp. 3953-3958.

[27] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, B. Catanzaro, High-resolution image synthesis and semantic manipulation with conditional gans, in: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2018, pp. 8798-8807.

[28] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, B. Schiele, The cityscapes dataset for semantic urban scene understanding, in: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recog-nition (CVPR), IEEE, 2016, pp. 3213-3223.

[29] D. Huang, F. Gao, X. Tao, Q. Du, J. Lu, Toward semantic communications: deep learning-based image semantic coding, IEEE J. Sel. Areas Commun. 41 (1) (2023) 55-71.

[30] H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia, Pyramid scene parsing network, in: Pro-ceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 2881-2890.