Deep reinforcement learning-based resilience optimization for infrastructure networks restoration with multiple crews

Abstract

Restoration of infrastructure networks (INs) following large disruptions has received much attention lately due to examples of massive localized attacks. Within this challenge are two complex but critical problems: repair route identification and optimizing the sequence of the repair actions for resilience improvement. Existing approaches have not, however, given due consideration to globally optimal enhancement in resilience, especially with multiple repair crews that have uneven capacities. To address this gap, this paper focuses on a resilience optimization (RO) strategy for coordinating multiple crews. The objective is to determine the optimal routes for each crew and the best sequence of repairs for damaged nodes and links. Given the two-layered decision-making required—coordinating between multiple crews and optimizing each crew’s actions—this study develops a deep reinforcement learning (DRL) framework. The framework leverages an actor-critic neural network that processes IN damage data and guides Monte Carlo tree search (MCTS) to identify optimal repair routes and actions for each crew. A case study based on the 228-node power grid, simulated using Python, demonstrates that the proposed DRL approach effectively supports restoration decision-making.

Graphical abstract

Keywords

Infrastructure network restoration / Deep reinforcement learning / Resilience optimization / Repair routing / Multiple crews

References

[1] Ahmadian N, Lim G J, Cho J, Bora S, (2020). A quantitative approach for assessment and improvement of network resilience. Reliability Engineering & System Safety, 200: 106977

[2] Allal A, Sahnoun M, Adjoudj R, Benslimane S M, Mazar M, (2021). Multi-agent based simulation-optimization of maintenance routing in offshore wind farms. Computers & Industrial Engineering, 157: 107342

[3] Arulkumaran K, Deisenroth M P, Brundage M, Bharath A A, (2017). Deep reinforcement learning a brief survey. IEEE Signal Processing Magazine, 34( 6): 26–38

[4] Bai G H, Wang H, Zheng X Q, Dui H Y, Xie M, (2021). Improved resilience measure for component recovery priority in power grids. Frontiers of Engineering Management, 8( 4): 545–556

[5] Browne C B, Powley E, Whitehouse D, Lucas S M, Cowling P I, Rohlfshagen P, Tavener S, Perez D, Samothrakis S, Colton S, (2012). A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games, 4( 1): 1–43

[6] Carnevali L, Tarani F, Vicario E, (2020). Performability evaluation of water distribution systems during maintenance procedures. IEEE Transactions on Systems, Man, and Cybernetics. Systems, 50( 5): 1704–1720

[7] Das L, Munikoti S, Natarajan B, Srinivasan B, (2020). Measuring smart grid resilience: Methods, challenges and opportunities. Renewable & Sustainable Energy Reviews, 130: 109918

[8] Díaz-Ramírez J, Huertas J I, Trigos F, (2014). Aircraft maintenance, routing, and crew scheduling planning for airlines with a single fleet and a single maintenance and crew base. Computers & Industrial Engineering, 75: 68–78

[9] Du Y, Wu D, (2022). Deep reinforcement learning from demonstrations to assist service restoration in islanded microgrids. IEEE Transactions on Sustainable Energy, 13( 2): 1062–1072

[10] Dui H Y, Dong X H, Wu X M, Chen L W, Bai G H, (2024a). IoT-enabled risk warning and maintenance strategy optimization for tunnel-induced ground settlement. IEEE Internet of Things Journal, 11( 13): 22966–22981

[11] Dui H Y, Liu M, Song J Y, Wu S M, (2023). Importance measure-based resilience management: Review, methodology and perspectives on maintenance. Reliability Engineering & System Safety, 237: 109383

[12] DuiH YWuX MWuS MXieM (2024b). Importance measure-based maintenance strategy optimization: Fundamentals, applications and future directions in AI and IoT. Frontiers of Engineering Management, 11(3): 542–567

[13] Dui H Y, Zhang H Q, Dong X H, Zhang S R, (2024c). Cascading failure and resilience optimization of unmanned vehicle distribution networks in IoT. Reliability Engineering & System Safety, 246: 110071

[14] Dui H Y, Zhu Y W, Tao J Y, (2024d). Multi-phased resilience methodology of urban sewage treatment network based on the phase and node recovery importance in IoT. Reliability Engineering & System Safety, 247: 110130

[15] Fan X D, Zhang X J, Yu X, (2022). A graph convolution network-deep reinforcement learning model for resilient water distribution network repair decisions. Computer-Aided Civil and Infrastructure Engineering, 37( 12): 1547–1565

[16] Feng Q, Bi X, Zhao X J, Chen Y R, Sun B, (2017). Heuristic hybrid game approach for fleet condition-based maintenance planning. Reliability Engineering & System Safety, 157: 166–176

[17] Fu J F, Nunez A, De Schutter B, (2022). Real-time UAV routing strategy for monitoring and inspection for postdisaster restoration of distribution networks. IEEE Transactions on Industrial Informatics, 18( 4): 2582–2592

[18] Ganganath N, Wang J V, Xu X, Cheng C, Tse C K, (2018). Agglomerative clustering-based network partitioning for parallel power system restoration. IEEE Transactions on Industrial Informatics, 14( 8): 3325–3333

[19] HeK MZhangX YRenS QSunJ (2016). Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Conference on Computer Vision and Pattern Recognition. Seattle, CA: IEEE, 770–778

[20] Liu Z X, Wang L F, (2021). Leveraging network topology optimization to strengthen power grid resilience against cyber-physical attacks. IEEE Transactions on Smart Grid, 12( 2): 1552–1564

[21] Lu L Q, Wang X, Ouyang Y F, Roningen J, Myers N, Calfas G, (2018). Vulnerability of interdependent urban infrastructure networks: equilibrium after failure propagation and cascading impacts. Computer-Aided Civil and Infrastructure Engineering, 33( 4): 300–315

[22] Molnar F, Nishikawa T, Motter A E, (2021). Asymmetry underlies stability in power grids. Nature Communications, 12( 1): 1457

[23] Ogunfowora O, Najjaran H, (2023). Reinforcement and deep reinforcement learning-based solutions for machine maintenance planning, scheduling policies, and optimization. Journal of Manufacturing Systems, 70: 244–263

[24] Pinciroli L, Baraldi P, Ballabio G, Compare M, Zio E, (2021). Deep reinforcement learning based on proximal policy optimization for the maintenance of a wind farm with multiple crews. Energies, 14( 20): 6743

[25] Rosin C D, (2011). Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, 61( 3): 203–230

[26] Ruan J H, Wang Z X, Chan F T S, Patnaik S, Tiwari M K, (2021). A reinforcement learning-based algorithm for the aircraft maintenance routing problem. Expert Systems with Applications, 169: 114399

[27] Si S B, Zhao J B, Cai Z Q, Dui H Y, (2020). Recent advances in system reliability optimization driven by importance measures. Frontiers of Engineering Management, 7( 3): 335–358

[28] Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y T, Lillicrap T, Hui F, Sifre L, van den Driessche G, Graepel T, Hassabis D, (2017). Mastering the game of Go without human knowledge. Nature, 550( 7676): 354–359

[29] Wang P J, Wen G H, Yu X H, Yu W W, Wan Y, (2021). Synchronization of resilient complex networks under attacks. IEEE Transactions on Systems, Man, and Cybernetics. Systems, 51( 2): 1116–1127

[30] Wu Q L, Feng Q, Ren Y, Xia Q, Wang Z L, Cai B P, (2021). An intelligent preventive maintenance method based on reinforcement learning for battery energy storage systems. IEEE Transactions on Industrial Informatics, 17( 12): 8254–8264

[31] Wu Z R, Xu W W, Li C C, Meng X F, (2022). A new approach for generator startup sequence online decision making with a heuristic search algorithm and graph theory. Energy Reports, 8: 678–686

[32] Xie H N, Jiang M H, Zhang D D, Goh H H, Ahmad T, Liu H, Liu T H, Wang S Y, Wu T, (2023). IntelliSense technology in the new power systems. Renewable & Sustainable Energy Reviews, 177: 113229

[33] Xu M, Ouyang M, Hong L, Mao Z J, Xu X L, (2022). Resilience-driven repair sequencing decision under uncertainty for critical infrastructure systems. Reliability Engineering & System Safety, 221: 108378

[34] Xu Z Z, Chopra S S, (2022). Network-based assessment of metro infrastructure with a spatial-temporal resilience cycle framework. Reliability Engineering & System Safety, 223: 108434

[35] Yu S, Zhang L, Lu H H, Fernando T, Wong K P, (2017). A DSE-based power system frequency restoration strategy for PV-integrated power systems considering solar irradiance variations. IEEE Transactions on Industrial Informatics, 13( 5): 2511–2518

[36] Zhang Z Y, Ji T T, Wei H H, (2022). Dynamic emergency inspection routing and restoration scheduling to enhance the post-earthquake resilience of a highway-bridge network. Reliability Engineering & System Safety, 220: 108282

[37] Zuo M J, (2021). System reliability and system resilience. Frontiers of Engineering Management, 8( 4): 615–619

RIGHTS & PERMISSIONS

Higher Education Press