Dynamic adversarial jamming-based reinforcement learning for designing constellations

Abstract

To resist various types of jamming in wireless channels, appropriate constellation modulation is used in wireless communication to ensure a low bit error rate. Due to the complexity and variability of the channel environment, a simple preset constellation is difficult to adapt to all scenarios, so the online constellation optimization method based on Reinforcement Learning (RL) shows its potential. However, the existing RL technology is difficult to ensure the optimal convergence efficiency. Therefore, in this paper, Dynamic Adversarial Interference (DAJ) waveforms are introduced and the DAJ-RL method is proposed by referring to adversarial training in Deep Learning (DL). The algorithm can converge to the optimal state quickly by self-adaptive power and probability direction of dynamic strong adversary of DAJ. In this paper, a rigorous theoretical proof of the symbol error rate is given and it is shown that the method approaches the mathematical limit. Also, numerical and hardware experiments show that the constellations generated by DAJ-RL have the best error rate at all noise levels. In the end, the proposed DAJ-RL method effectively improves the RL-based anti-jamming modulation for cognitive electronic warfare.

Keywords

Wireless communication / Constellation design / Reinforcement learning / Adversarial jamming

References

[1] J.G. Proakis, M. Salehi, Digital Communications, 4th edition, McGraw-Hill, New York, 2001.

[2] B. Kernighan, S. Lin, Heuristic solution of a signal design optimization problem, Bell Syst. Tech. J. 52 (7) (1973) 1145-1159.

[3] G. Foschini, R. Gitlin, S. Weinstein, Optimization of two-dimensional signal con-stellations in the presence of Gaussian noise, IEEE Trans. Commun. 22 (1) (1974) 28-38.

[4] Y. Li, S. Xu, H. Yang, Design of signal constellations in the presence of phase noise, in: 2008 IEEE 68th Vehicular Technology Conference, IEEE, 2008, pp. 1-5.

[5] R. Krishnan, A.G. i Amat, T. Eriksson, G. Colavolpe, Constellation optimization in the presence of strong phase noise, IEEE Trans. Commun. 61 (12) (2013) 5056-5066.

[6] T.J. O’Shea, K. Karra, T.C. Clancy, Learning to communicate: channel auto-encoders, domain specific regularizers and attention, in: 2016 IEEE International Sym-posium on Signal Processing and Information Technology (ISSPIT), IEEE, 2016, pp. 223-228.

[7] T. O’shea, J. Hoydis, An introduction to deep learning for the physical layer, IEEE Trans. Cognit. Commun. Netw. 3(4) (2017) 563-575.

[8] F. Alberge, Deep learning constellation design for the awgn channel with additive radar interference, IEEE Trans. Commun. 67 (2) (2018) 1413-1423.

[9] X. Fu, B.F. Da Silva, D. Le Ruyet, Multidimensional codebook design using deep learning techniques for Rayleigh fading channels, IEEE Wirel. Commun. Lett. 10 (9)(2021) 1974-1978.

[10] T.J. O’Shea, T. Erpek, T.C. Clancy, Physical layer deep learning of encodings for the mimo fading channel, in: 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton), IEEE, 2017, pp. 76-80.

[11] S. Cammerer, F.A. Aoudia, S. Dörner, M. Stark, J. Hoydis, S. Ten Brink, Train-able communication systems: concepts and prototype, IEEE Trans. Commun. 68 (9)(2020) 5489-5503.

[12] T.J. O’Shea, T. Roy, N. West, Approximating the void: learning stochastic channel models from observation with variational generative adversarial networks, in: 2019 International Conference on Computing, Networking and Communications (ICNC), IEEE, 2019, pp. 681-686.

[13] B. Karanov, M. Chagnon, V. Aref, D. Lavery, P. Bayvel, L. Schmalen, Concept and experimental demonstration of optical im/dd end-to-end system optimization using a generative model, in: 2020 Optical Fiber Communications Conference and Exhibi-tion (OFC), IEEE, 2020, pp. 1-3.

[14] F.A. Aoudia, J. Hoydis, Model-free training of end-to-end communication systems, IEEE J. Sel. Areas Commun. 37 (11) (2019) 2503-2516.

[15] C. de Vrieze, S. Barratt, D. Tsai, A. Sahai, Cooperative multi-agent reinforcement learning for low-level wireless communication, arXiv preprint, arXiv :1801.04541, 2018.

[16] R. Bruno, A. Masaracchia, A. Passarella,Robust adaptive modulation and coding (amc) selection in lte systems using reinforcement learning, in: 2014 IEEE 80th Vehicular Technology Conference (VTC2014-Fall), IEEE, 2014, pp. 1-6.

[17] N. Abuzainab, T. Erpek, K. Davaslioglu, Y.E. Sagduyu, Y. Shi, S.J. Mackey, M. Patel, F. Panettieri, M.A. Qureshi, V. Isler, et al., Qos and jamming-aware wireless net-working using deep reinforcement learning, in: MILCOM 2019-2019 IEEE Military Communications Conference (MILCOM), IEEE, 2019, pp. 610-615.

[18] Y.S. Nasir, D. Guo, Multi-agent deep reinforcement learning for dynamic power allo-cation in wireless networks, IEEE J. Sel. Areas Commun. 37 (10) (2019) 2239-2250.

[19] F. Tang, Y. Zhou, N. Kato, Deep reinforcement learning for dynamic uplink/down-link resource allocation in high mobility 5g hetnet, IEEE J. Sel. Areas Commun. 38 (12) (2020) 2773-2782.

[20] T.-Y. Tung, S. Kobus, J.R. Pujol, D. Gunduz, Effective communications: a joint learn-ing and communication framework for multi-agent reinforcement learning over noisy channels, arXiv preprint, arXiv :2101.10369, 2021.

[21] S. Park, O. Simeone, J. Kang, End-to-end fast training of communication links with-out a channel model via online meta-learning, in: 2020 IEEE 21st International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), IEEE, 2020, pp. 1-5.

[22] V. Raj, S. Kalyani, Backpropagating through the air: deep learning at physical layer without channel models, IEEE Commun. Lett. 22 (11) (2018) 2278-2281.

[23] H. Xie, X. Xiang, N. Liu, B. Dong, Blind adversarial training: balance accuracy and robustness, arXiv preprint, arXiv :2004.05914, 2020.

[24] I.J. Goodfellow, J. Shlens, C. Szegedy,Explaining and harnessing adversarial exam-ples, arXiv preprint, arXiv :1412.6572, 2014.

[25] T. Erpek, Y.E. Sagduyu, Y. Shi, Deep learning for launching and mitigating wireless jamming attacks, IEEE Trans. Cognit. Commun. Netw. 5(1) (2018) 2-14.

[26] Y. Xu, H. Xie, N. Ji, Y. Chen, N. Liu, X. Xiang, Toward anti-jamming constella-tions via adversarial reinforcement learning, in: 3rd URSI Atlantic/Asia-Pacific Radio Science Meeting, URSI, 2022 [Online], available at https://www.ursi.org/proceedings/procAT22/ATAPRASC2022-papers/B8BWB6X7JV.pdf.

[27] J.A. Sills,Maximum-likelihood modulation classification for psk/qam, in: MIL-COM 1999-1999 IEEE Military Communications Conference (MILCOM), IEEE, 1999, pp. 217-220.

[28] T.J. Rouphael, RF and Digital Signal Processing for Software-defined Radio: a Multi-Standard Multi-Mode Approach, Newnes, 2009.

[29] F. Gray,Pulse code communication, United States Patent Number 2632058, 1953.

[30] R.W. Hamming, Error detecting and error correcting codes, Bell Syst. Tech. J. 29 (2)(1950) 147-160.

[31] H. Xie, Y. Xu, Y. Chen, N. Ji, S. Yuan, N. Liu, X. Xiang, Adversarial jamming for a more effective constellation attack, in: 2021 13th International Symposium on Antennas, Propagation and EM Theory (ISAPE), IEEE, 2021, pp. 01-03.

[32] T.C. Hales, A proof of the Kepler conjecture, Ann. Math. (2005) 1065-1185.

[33] S. Kravitz, Packing cylinders into cylindrical containers, Math. Mag. 40 (2) (1967) 65-71.

[34] M. Locatelli, F. Schoen, Efficient algorithms for large scale global optimization: Lennard-Jones clusters, Comput. Optim. Appl. 26 (2) (2003) 173-190.

[35] R. Peikert, D. Würtz, M. Monagan, C. de Groot, Packing circles in a square: a review and new results,in:System Modelling and Optimization: Proceedings of the 15th IFIP Conference, Springer, 1992, pp. 45-54.

[36] B.D. Lubachevsky, R.L. Graham, Curved hexagonal packings of equal disks in a cir-cle, Discrete Comput. Geom. 18 (2) (1997) 179-194.

[37] B. Addis, M. Locatelli, F. Schoen, Disk packing in a square: a new global optimization approach, INFORMS J. Comput. 20 (4) (2008) 516-524.

[38] I. Castillo, F.J. Kampas, J.D. Pintér, Solving circle packing problems by global op-timization: numerical results and industrial applications, Eur. J. Oper. Res. 191 (3)(2008) 786-802.

[39] Specht,Packomania website, http://www.packomania.com/, 2023. (Accessed 11 March 2023).

[40] A. Demir, A. Mehrotra, J. Roychowdhury, Phase noise in oscillators: a unifying theory and numerical methods for characterization, IEEE Trans. Circuits Syst. I, Fundam. Theory Appl. 47 (5) (2000) 655-674.

[41] A. Lapidoth, On phase noise channels at high snr, in: Proceedings of the IEEE Infor-mation Theory Workshop, IEEE, 2002, pp. 1-4.

[42] USRP X310, https://kb.ettus.com/X300/X310, 2023. (Accessed 11 March 2023).

[43] Ettus Research UHD, https://kb.ettus.com/UHD, 2023. (Accessed 11 March 2023).