Evaluation of Reinforcement Learning-Based Adaptive Modulation in Shallow Sea Acoustic Communication

Abstract

While reinforcement learning-based underwater acoustic adaptive modulation shows promise for enabling environment-adaptive communication as supported by extensive simulation-based research, its practical performance remains underexplored in field investigations. To evaluate the practical applicability of this emerging technique in adverse shallow sea channels, a field experiment was conducted using three communication modes: orthogonal frequency division multiplexing (OFDM), M-ary frequency-shift keying (MFSK), and direct sequence spread spectrum (DSSS) for reinforcement learning-driven adaptive modulation. Specifically, a Q-learning method is used to select the optimal modulation mode according to the channel quality quantified by signal-to-noise ratio, multipath spread length, and Doppler frequency offset. Experimental results demonstrate that the reinforcement learning-based adaptive modulation scheme outperformed fixed threshold detection in terms of total throughput and average bit error rate, surpassing conventional adaptive modulation strategies.

References

[1] Abdallah S, Kaisers M. Addressing environment non-stationarity by repeating Q-learning updates. Journal of Machine Learning Research, 2016, 17(46): 1-31

[2] Barua S, Rong Y, Nordholm S, Chen P. Real-time adaptive modulation schemes for underwater acoustic OFDM communication. Sensors, 2022, 22(9): 3436

[3] Benson A, Proakis J, Stojanovic M. Towards robust adaptive acoustic communications. OCEANS 2000 MTS/IEEE Conference and Exhibition. Conference Proceedings (Cat. No. 00CH37158), 2000, Piscataway, IEEE 1243-1249

[4] Bhopale P, Kazi F, Singh N. Reinforcement learning based obstacle avoidance for autonomous underwater vehicle. Journal of Marine Science and Application, 2019, 18: 228-238

[5] Bulut S, Ergin S. Effects of temperature, salinity, and fluid type on acoustic characteristics of turbulent flow around circular cylinder. Journal of Marine Science and Application, 2021, 20: 213-228

[6] Fan C, Wang Z. Adaptive switching for multimodal underwater acoustic communications based on reinforcement learning. The 15th International Conference on Underwater Networks & Systems, 2021, Shenzhen, ACM 1-2

[7] Fu Q, Song A. Adaptive modulation for underwater acoustic communications based on reinforcement learning. OCEANS 2018 MTS/IEEE Charleston, 2018, Piscataway, IEEE 1-8

[8] Gussen CMG, Diniz PSR, Campos MLR, Martins WA, Costa FM, Gois JN. A survey of underwater wireless communication technologies. Journal of Communication and Information Systems, 2016, 31(1): 242-255

[9] Huang L, Wang Y, Zhang Q, Han J, Tan W, Tian Z. Machine learning for underwater acoustic communications. IEEE Wireless Communications, 2022, 29(3): 102-108

[10] Huang J, Diamant R. Adaptive modulation for long-range underwater acoustic communication. IEEE Transactions on Wireless Communications, 2020, 19(10): 6844-6857

[11] Huang JG, Wang H, He CB, Zhang QF, Jing LY. Underwater acoustic communication and the general performance evaluation criteria. Frontiers of Information Technology & Electronic Engineering, 2018, 19: 951-971

[12] Li B, Zheng S, Tong F. Bit-error rate based Doppler estimation for shallow water acoustic OFDM communication. Ocean Engineering, 2019, 182: 203-210

[13] Li G, Wu J, Tang T, Chen Z, Chen J, Liu H. Underwater acoustic time delay estimation based on envelope differences of correlation functions. Sensors, 2019, 19(5): 1259

[14] Mani S, Duman TM, Hursky P. Adaptive coding/modulation for shallow-water UWA communications. Journal of the Acoustical Society of America, 2008, 123: 3749-3749

[15] Radosevic A, Ahmed R, Duman TM, Proaki JG, Stojanovic M. Adaptive OFDM modulation for underwater acoustic communications: design considerations and experimental results. IEEE Journal of Oceanic Engineering, 2014, 39(2): 357-370

[16] Stojanovic M, Preisig J. Underwater acoustic communication channels: Propagation models and statistical characterization. IEEE Communications Magazine, 2009, 47(1): 84-89

[17] Su W, Lin J, Chen K, Xiao L, Em C. Reinforcement learning-based adaptive modulation and coding for efficient underwater communications. IEEE Access, 2019, 7: 67539-67550

[18] Su Y, Liu Y, Fan R, et al.. A cooperative jamming scheme based on node authentication for underwater acoustic sensor networks. Journal of Marine Science and Application, 2022, 21: 197-209

[19] Tang N, Zeng Q, Luo D, Xu Q, Hu H. Research on development and application of underwater acoustic communication system. Journal of Physics: Conference Series, 2020, 1617: 012036

[20] Sweta T, Ruthrapriya S, Sneka J, John SRA, Rohith G, Mangal D. Reinforcement learning-based automated modulation switching algorithm for an enhanced underwater acoustic communication. Results in Engineering, 2024, 23: 102791

[21] Wan L, Wang Z, Zhou S, Yang TC, Shi Z. Performance comparison of doppler scale estimation methods for underwater acoustic OFDM. Journal of Electrical and Computer Engineering, 2012, 2012: 1-11

[22] Wan L, Zhou H, Xu X, Huang Y, Zhou S, Shi Z. Adaptive modulation and coding for underwater acoustic OFDM. IEEE Journal of Oceanic Engineering, 2015, 40(2): 327-336

[23] Wang C, Wang Z, Sun W, Fuhrmann DR. Reinforcement learning-based adaptive transmission in time-varying underwater acoustic channels. IEEE Access, 2018, 6: 2541-2558

[24] Zhang Y, Zhang Z, Chen L, Wang X. Reinforcement Learning-Based Opportunistic Routing Protocol for Underwater Acoustic Sensor Networks. IEEE Transactions on Vehicular Technology, 2021, 70(3): 2756-2770

[25] Zhu Z, Tong F, Zhou Y, Zhang Z, Zhang F. Deep learning prediction of time-varying underwater acoustic channel based on LSTM with attention mechanism. Journal of Marine Science and Application, 2023, 22: 650-658

RIGHTS & PERMISSIONS

Harbin Engineering University and Springer-Verlag GmbH Germany, part of Springer Nature