Dynamic Clustering Method for Underwater Wireless Sensor Networks based on Deep Reinforcement Learning

Abstract

Underwater wireless sensor networks (UWSNs) have emerged as a new paradigm of real-time organized systems, which are utilized in a diverse array of scenarios to manage the underwater environment surrounding them. One of the major challenges that these systems confront is topology control via clustering, which reduces the overload of wireless communications within a network and ensures low energy consumption and good scalability. This study aimed to present a clustering technique in which the clustering process and cluster head (CH) selection are performed based on the Markov decision process and deep reinforcement learning (DRL). DRL algorithm selects the CH by maximizing the defined reward function. Subsequently, the sensed data are collected by the CHs and then sent to the autonomous underwater vehicles. In the final phase, the consumed energy by each sensor is calculated, and its residual energy is updated. Then, the autonomous underwater vehicle performs all clustering and CH selection operations. This procedure persists until the point of cessation when the sensor’s power has been reduced to such an extent that no node can become a CH. Through analysis of the findings from this investigation and their comparison with alternative frameworks, the implementation of this method can be used to control the cluster size and the number of CHs, which ultimately augments the energy usage of nodes and prolongs the lifespan of the network. Our simulation results illustrate that the suggested methodology surpasses the conventional low-energy adaptive clustering hierarchy, the distance- and energy-constrained K-means clustering scheme, and the vector-based forward protocol and is viable for deployment in an actual operational environment.

Keywords

Underwater wireless sensor network / Clustering / Cluster head selection / Deep reinforcement learning

References

[1] AkyildizIF, SuW, SankarasubramaniamY, CayirciE. Wireless sensor networks: a survey. Computer Networks, 2002, 38(4): 393-422

[2] AlsalmanL, AlotaibiE. A balanced routing protocol based on machine learning for underwater sensor networks. IEEE Access, 2021, 9: 152082-152097

[3] ArulkumaranK, DeisenrothMP, BrundageM, BharathAA. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 2017, 34(6): 26-38

[4] BalharaS, GuptaN, AlkhayyatA, BhartiI, MalikRQ, MahmoodSN, AbediF. A survey on deep reinforcement learning architectures, applications and emerging trends. IET Communications, 2022, 2022: 1-16

[5] BhattacharjyaK, AlamS, DeD. CUWSN: energy efficient routing protocol selection for cluster based underwater wireless sensor network. Microsystem Technologies, 2019, 28(2): 1-17

[6] BholowaliaP, KumarA. EBK-means: A clustering technique based on elbow method and k-means in WSN. International Journal of Computer Applications, 2014, 105(9): 17-24

[7] DasD, AmeerPM. Energy efficient geographic clustered multi-hop routing for underwater sensor networks. TENCON 2017–2017 IEEE Region 10 Conference, 2017409414

[8] FattahS, GaniA, AhmedyI, IdrisMYI, Targio HashemIA. A survey on underwater wireless sensor networks: requirements, taxonomy, recent advances, and open research challenges. Sensors, 2020, 20185393

[9] GuptaS, SinghNP. Underwater wireless sensor networks: a review of routing protocols, taxonomy, and future directions. The Journal of Supercomputing, 2024, 80(4): 5163-5196

[10] HanG, ShenS, SongH, YangT, ZhangW. A stratification-based data collection scheme in underwater acoustic sensor networks. IEEE Transactions on Vehicular Technology, 2018, 67(11): 10671-10682

[11] HeinzelmanWB, ChandrakasanAP, BalakrishnanH. An application-specific protocol architecture for wireless microsensor networks. IEEE Transactions on Wireless Communications, 2002, 1(4): 660-670

[12] HuT, FeiY. QELAR: A machine-learning-based adaptive routing protocol for energy-efficient and lifetime-extended underwater sensor networks. IEEE Transactions on Mobile Computing, 2010, 9(6): 796-809

[13] JavaidN, HafeezT, WadudZ, AlrajehN, AlabedMS, GuizaniN. Establishing a cooperation-based and void node avoiding energy-efficient underwater WSN for a cloud. IEEE Access, 2017, 5: 11582-11593

[14] JawharI, MohamedN, Al-JaroodiJ, ZhangS. An architecture for using autonomous underwater vehicles in wireless sensor networks for underwater pipeline monitoring. IEEE Transactions on Industrial Informatics, 2018, 15(3): 1329-1340

[15] KhalidM, UllahZ, AhmadN, ArshadM, JanB, CaoY, AdnanA. A survey of routing issues and associated protocols in underwater wireless sensor networks. Journal of Sensors, 2017, 7539751: 1-17

[16] KhanA, JavaidN, LatifG, JattaL, FatimaA, KhanW. Cluster based and adaptive power controlled routing protocol for underwater wireless sensor networks. 21st Saudi Computer Society National Computer Conference (NCC), 201816

[17] KhanG, GolaKK, AliW. Energy efficient routing algorithm for UWSN-A clustering approach. Second International Conference on Advances in Computing and Communication Engineering, 2015150155

[18] KhanW, WangH, AnwarMS, AyazM, AhmadS, UllahI. A multi-layer cluster based energy efficient routing scheme for UWSNs. IEEE Access, 2019, 7: 77398-77410

[19] KhisaS, MohS. Survey on recent advancements in energy-efficient routing protocols for underwater wireless sensor networks. IEEE Access, 2021, 9: 55045-55062

[20] KozlowskiM, McConvilleR, Santos-RodriguezR, PiechockiREnergy efficiency in reinforcement learning for wireless sensor networks, 2018

[21] LiYDeep reinforcement learning: An overview, 2017

[22] LuY, HeR, ChenX, LinB, YuC. Energy-efficient depth-based opportunistic routing with q-learning for underwater wireless sensor networks. Sensors, 2020, 2041025

[23] LuoJ, ChenY, WuM, YangY. A survey of routing protocols for underwater wireless sensor networks. IEEE Communications Surveys & Tutorials, 2021, 23(1): 137-160

[24] MajidA, AzamI, WaheedA, Zain-ul-AbidinM, HafeezT, KhanZA, QasimU, JavaidN. An energy efficient and balanced energy consumption cluster based routing protocol for underwater wireless sensor networks. IEEE 30th International Conference on Advanced Information Networking and Applications (AINA), 2016324333

[25] MazinaniSM, YousefiH, MirzaieM. A vector-based routing protocol in underwater wireless sensor networks. Wireless Personal Communications, 2018, 100(4): 1569-1583

[26] NasirYS, GuoD. Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks. IEEE Journal on Selected Areas in Communications, 2019, 37(10): 2239-2250

[27] NazarethP, ChandavarkarBR. Cluster-based multi-attribute routing protocol for underwater acoustic sensor networks. Wireless Personal Communications, 2024, 134(2): 781-808

[28] NguyenNT, LeTTT, NguyenHH, VoznakM. Energy-efficient clustering multi-hop routing protocol in a UWSN. Sensors, 2021, 212627

[29] OmekeKG, MollelMS, OzturkM, AnsariS, ZhangL, AbbasiQH, ImranMA. DEKCS: A dynamic clustering protocol to prolong underwater sensor networks. IEEE Sensors Journal, 2021, 21(7): 9457-9464

[30] RahmaniM, DehghaniMJ, XiaoP, BasharM, DebbahM. Multi-agent reinforcement learning-based pilot assignment for cell-free massive MIMO systems. IEEE Access, 2022, 10: 120492-120502

[31] RowshanradS, KeshtgaryM, JavidanR. MBC: A multi-hop balanced clustering routing protocol for wireless sensor networks. International Journal of Artificial Intelligence and Mechatronics, 2014, 2(6): 164-170

[32] ShelarPA, MahallePN, ShindeGR, BhapkarHR, TeferaMA. Performance-aware green algorithm for clustering of underwater wireless sensor network based on optical signal-to-noise ratio. Mathematical Problems in Engineering, 2022, 2022: 1-18

[33] SouikiS, HadjilaM, FehamM. Fuzzy based clustering and energy efficient routing for underwater wireless sensor networks. Int. J. of Comp. Netw. & Commun. (IJCNC), 2015, 7(2): 33-44

[34] StojanovicM, PreisigJ. Underwater acoustic communication channels: Propagation models and statistical characterization. IEEE Communications Magazine, 2009, 47(1): 84-89

[35] SunY, ZhengM, HanX, LiS, YinJ. Adaptive clustering routing protocol for underwater sensor networks. Ad Hoc Networks, 2022, 136102953

[36] SuttonRS, BartoAGReinforcement learning: An introduction, 2018

[37] TaheriR, ParsaeiMR, JavidanR. A new method for optimizing energy consumption in wireless sensor networks using enhanced LEACH protocol. Journal of Engineering and Applied Sciences, 2016, 100(3): 576-581

[38] WangC, ShenX, WangH, ZhangH, MeiH. Reinforcement learning-based opportunistic routing protocol using depth information for energy-efficient underwater wireless sensor networks. IEEE Sensors Journal, 2023, 23(15): 17771-17783

[39] WangK, GaoH, XuX, JiangJ, YueD. An energy-efficient reliable data transmission scheme for complex environmental monitoring in underwater acoustic sensor networks. IEEE Sensors Journal, 2015, 16(11): 4051-4062

[40] WangM, ChenY, SunX, XiaoF, XuX. Node energy consumption balanced multi-hop transmission for underwater acoustic sensor networks based on clustering algorithm. IEEE Access, 2020, 8: 191231-191241

[41] ZhuF, WeiJ. An energy efficient routing protocol based on layers and unequal clusters in underwater wireless sensor networks. Journal of Sensors, 2018, 2018: 1-10

RIGHTS & PERMISSIONS

Harbin Engineering University and Springer-Verlag GmbH Germany, part of Springer Nature