Evaluation of Reinforcement Learning-Based Adaptive Modulation in Shallow Sea Acoustic Communication

Abstract

While reinforcement learning-based underwater acoustic adaptive modulation shows promise for enabling environment-adaptive communication as supported by extensive simulation-based research, its practical performance remains underexplored in field investigations. To evaluate the practical applicability of this emerging technique in adverse shallow sea channels, a field experiment was conducted using three communication modes: orthogonal frequency division multiplexing (OFDM), M-ary frequency-shift keying (MFSK), and direct sequence spread spectrum (DSSS) for reinforcement learning-driven adaptive modulation. Specifically, a Q-learning method is used to select the optimal modulation mode according to the channel quality quantified by signal-to-noise ratio, multipath spread length, and Doppler frequency offset. Experimental results demonstrate that the reinforcement learning-based adaptive modulation scheme outperformed fixed threshold detection in terms of total throughput and average bit error rate, surpassing conventional adaptive modulation strategies.

References

[] Abdallah S, Kaisers M. Addressing environment non-stationarity by repeating Q-learning updates. Journal of Machine Learning Research, 2016, 17(46): 1-31

[] Barua S, Rong Y, Nordholm S, Chen P. Real-time adaptive modulation schemes for underwater acoustic OFDM communication. Sensors, 2022, 22(9): 3436

[] Benson A, Proakis J, Stojanovic M. Towards robust adaptive acoustic communications. OCEANS 2000 MTS/IEEE Conference and Exhibition. Conference Proceedings (Cat. No. 00CH37158), 2000, Piscataway, IEEE 1243-1249

[] Bhopale P, Kazi F, Singh N. Reinforcement learning based obstacle avoidance for autonomous underwater vehicle. Journal of Marine Science and Application, 2019, 18: 228-238

[] Bulut S, Ergin S. Effects of temperature, salinity, and fluid type on acoustic characteristics of turbulent flow around circular cylinder. Journal of Marine Science and Application, 2021, 20: 213-228

[] Fan C, Wang Z. Adaptive switching for multimodal underwater acoustic communications based on reinforcement learning. The 15th International Conference on Underwater Networks & Systems, 2021, Shenzhen, ACM 1-2

[] Fu Q, Song A. Adaptive modulation for underwater acoustic communications based on reinforcement learning. OCEANS 2018 MTS/IEEE Charleston, 2018, Piscataway, IEEE 1-8

[] Gussen CMG, Diniz PSR, Campos MLR, Martins WA, Costa FM, Gois JN. A survey of underwater wireless communication technologies. Journal of Communication and Information Systems, 2016, 31(1): 242-255

[] Huang L, Wang Y, Zhang Q, Han J, Tan W, Tian Z. Machine learning for underwater acoustic communications. IEEE Wireless Communications, 2022, 29(3): 102-108

[] Huang J, Diamant R. Adaptive modulation for long-range underwater acoustic communication. IEEE Transactions on Wireless Communications, 2020, 19(10): 6844-6857

[] Huang JG, Wang H, He CB, Zhang QF, Jing LY. Underwater acoustic communication and the general performance evaluation criteria. Frontiers of Information Technology & Electronic Engineering, 2018, 19: 951-971

[] Li B, Zheng S, Tong F. Bit-error rate based Doppler estimation for shallow water acoustic OFDM communication. Ocean Engineering, 2019, 182: 203-210

[] Li G, Wu J, Tang T, Chen Z, Chen J, Liu H. Underwater acoustic time delay estimation based on envelope differences of correlation functions. Sensors, 2019, 19(5): 1259

[] Mani S, Duman TM, Hursky P. Adaptive coding/modulation for shallow-water UWA communications. Journal of the Acoustical Society of America, 2008, 123: 3749-3749

[] Radosevic A, Ahmed R, Duman TM, Proaki JG, Stojanovic M. Adaptive OFDM modulation for underwater acoustic communications: design considerations and experimental results. IEEE Journal of Oceanic Engineering, 2014, 39(2): 357-370

[] Stojanovic M, Preisig J. Underwater acoustic communication channels: Propagation models and statistical characterization. IEEE Communications Magazine, 2009, 47(1): 84-89

[] Su W, Lin J, Chen K, Xiao L, Em C. Reinforcement learning-based adaptive modulation and coding for efficient underwater communications. IEEE Access, 2019, 7: 67539-67550

[] Su Y, Liu Y, Fan R, et al.. A cooperative jamming scheme based on node authentication for underwater acoustic sensor networks. Journal of Marine Science and Application, 2022, 21: 197-209

[] Tang N, Zeng Q, Luo D, Xu Q, Hu H. Research on development and application of underwater acoustic communication system. Journal of Physics: Conference Series, 2020, 1617: 012036

[] Sweta T, Ruthrapriya S, Sneka J, John SRA, Rohith G, Mangal D. Reinforcement learning-based automated modulation switching algorithm for an enhanced underwater acoustic communication. Results in Engineering, 2024, 23: 102791

[] Wan L, Wang Z, Zhou S, Yang TC, Shi Z. Performance comparison of doppler scale estimation methods for underwater acoustic OFDM. Journal of Electrical and Computer Engineering, 2012, 2012: 1-11

[] Wan L, Zhou H, Xu X, Huang Y, Zhou S, Shi Z. Adaptive modulation and coding for underwater acoustic OFDM. IEEE Journal of Oceanic Engineering, 2015, 40(2): 327-336

[] Wang C, Wang Z, Sun W, Fuhrmann DR. Reinforcement learning-based adaptive transmission in time-varying underwater acoustic channels. IEEE Access, 2018, 6: 2541-2558

[] Zhang Y, Zhang Z, Chen L, Wang X. Reinforcement Learning-Based Opportunistic Routing Protocol for Underwater Acoustic Sensor Networks. IEEE Transactions on Vehicular Technology, 2021, 70(3): 2756-2770

[] Zhu Z, Tong F, Zhou Y, Zhang Z, Zhang F. Deep learning prediction of time-varying underwater acoustic channel based on LSTM with attention mechanism. Journal of Marine Science and Application, 2023, 22: 650-658