Multi-Agent Reinforcement Learning for Optimal Operation of PV-ES-EV Microgrids

Abstract

Aiming at the difficulty in balancing economic efficiency and islanding autonomy security during grid-connected operation of microgrids, as well as the limitation of fixed weights in traditional multiobjective optimization, this paper proposes a grid-connected interactive optimization strategy considering dynamic autonomy weights. A microgrid autonomy index is defined to quantify islanding preparedness, and a lightweight prediction network is designed to generate online weights for the three objectives of economy, security, and autonomy, so as to realize adaptive adjustment of the optimization focus. Furthermore, the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm is adopted to coordinate photovoltaics, energy storage, electric vehicle chargers, various loads, as well as power purchasing and selling, enabling decentralized decision-making. Results show that the proposed strategy achieves economic performance close to that of economic-only optimization (i.e., disregarding islanding preparedness) under grid-connected conditions without external faults, while shortening the interruption duration of critical loads by more than 72% during islanding transition caused by external grid faults. Meanwhile, the state of charge (SOC) remains strictly within the operational safety band of 20-90% throughout all dispatch cycles, complying with industry norms for battery cycle life preservation. The dynamic weights for economy, security, and autonomy are generated online by a lightweight neural network based solely on real-time system states rather than being fixed a priori, verifying the effectiveness of the proposed mechanism in achieving a context-aware trade-off among conflicting objectives.

Keywords

Microgrid / Dynamic autonomy weight / Multi-agent reinforcement learning / Clipped proximal policy optimization algorithm / PV-ES-EV

Acknowledgments

The authors would like to thank all those who provided support and constructive feedback during the preparation of this work.

Author Contributions

Conceptualization, Y.S.; Writing—Original Draft Preparation, Y.S.; Writing—Review, Y.X.; Supervision else, X.H., Z.W., C.W., W.W., T.Z., C.Z., W.D. and H.C.

Ethics Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Funding

This research was supported by the Sichuan Natural Science Foundation Project (2026NSFSCZY0091) and Sichuan University Student Innovation and Entrepreneur-ship Training Program (X2026106100509) and Institutional Research Fund from Sichuan University (0-1 Innovation Research Project) under Grant 2023SCUH0002.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

[1] Petrollese M, Valverde L, Cocco D, Cau G, Guerra J. Real-Time Integration of Optimal Generation Scheduling with MPC for the Energy Management of a Renewable Hydrogen-Based Microgrid. Appl. Energy 2016, 166, 96-106. DOI:10.1016/j.apenergy.2016.01.014

[2] Parisio A, Rikos E, Glielmo L. A Model Predictive Control Approach to Microgrid Operation Optimization. IEEE Trans. Control Syst. Technol. 2014, 22, 1813-1827. DOI:10.1109/TCST.2013.2295737

[3] Hossain MA, Pota HR, Squartini S, Abdou AF. Modified PSO Algorithm for Real-Time Energy Management in Grid-Connected Microgrids. Renew. Energy 2019, 136, 746-757. DOI:10.1016/j.renene.2019.01.005

[4] Torkan R, Ilinca A, Ghorbanzadeh M. A Genetic Algorithm Optimization Approach for Smart Energy Management of Microgrids. Renew. Energy 2022, 197, 852-863. DOI:10.1016/j.renene.2022.07.055

[5] Liu X, Chen S, Huang H, Ma L, Kong X. Energy Management Strategy for Smart Microgrids Based on Proximal Policy Optimization. Sci. China Inf. Sci. 2026, 56, 937-953. DOI:10.1360/SSI-2025-0102

[6] Shen JJ, Yang S, Chen YH, Guo F. Physics-Constrained and Gradient-Guided Reinforcement Learning for Secure Energy Dispatch in Microgrids. Control Decis. 2026, 42, 1179-1188. DOI:10.13195/j.kzyjc.2025.1231(InChinese)

[7] Yan JY.Research on Dispatch Strategy of Photovoltaic Microgrid Based on Deep Reinforcement Learning. Ph.D. Dissertation, University of Electronic Science and Technology of China, Chengdu, China, 2025.

[8] Zhao ZH, Ni H. Research on Economic Optimization Dispatch of Multi-Microgrid Based on Improved SAC Algorithm. Acta Energ. Sol. Sin. 2026, 47, 355-364. DOI:10.19912/j.0254-0096.tynxb.2024-1835(InChinese)

[9] Lei Q, Wu PR, Li ZW. Optimal Dispatch of Microgrid Based on Improved SAC Algorithm. J. Electr. Eng. 2025, 20, 108-116. DOI:10.19457/j.1001-2095.dqcd26592(InChinese)

[10] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O.Proximal Policy Optimization Algorithms. arXiv 2017, arXiv:1707.06347. DOI:10.48550/arXiv.1707.06347

[11] Fang X, Hong P, He S, Zhang Y, Tan D. Multi-Layer Energy Management and Strategy Learning for Microgrids: A Proximal Policy Optimization Approach. Energies 2024, 17, 3990. DOI:10.3390/en17163990

[12] Lu YH, Fan PX, Yang J, Li R. Intelligent Frequency Coordination Strategy for Islanded Microgrid with Electric Vehicles Based on Proximal Policy Optimization Algorithm. Electr. Power Autom. Equip. 2025, 45, 135-143. DOI:10.16081/j.epae.202503021(InChinese)

[13] Lowe R, Wu YI, Tamar A, Harb J, Pieter Abbeel O, Mordatch I. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017), Long Beach, CA, USA, 4-9 December 2017; pp. 6379-6390.

[14] Yu C, Velu A, Vinitsky E, Gao J, Wang Y, Bayen A, et al. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. In Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022), New Orleans, LA, USA, 28 November-9 December 2022; pp. 24611-24624.

[15] Li SC.Research on Optimal Operation Strategy of Microgrid Based on Multi-Agent. Ph.D. Dissertation, University of Electronic Science and Technology of China, Chengdu, China, 2024.

[16] Zheng S, Zhang Y, Li Y, Chen YH. Two-layer reinforcement learning optimization and regulation strategy for rural power grid supported by multi-microgrid collaboration. Distrib. Util. 2025, 42, 41-51+57. DOI:10.19421/j.cnki.1006-6357.2025.12.005(InChinese)

[17] Jiang L, Han D. Hierarchical Low-Carbon Optimal Operation Strategy for Distribution-Microgrid System with Heterogeneous Agents Based on TD3 Algorithm. Proc. CSU-EPSA 2026, 38, 102-111. DOI:10.19635/j.cnki.csuepsa.001811(InChinese)

[18] Zhou RJ.Research on Multi-Modal Adaptive Scheduling Optimization of Microgrid Based on Deep Reinforcement Learning. Ph.D. Dissertation, Huazhong University of Science and Technology, Wuhan, China, 2024.

[19] Xiang Y, Shao KW, Yang YJ, Tang ZY, Li ZB, Li YC, et al. Review of AI-Driven Distribution System Planning Enhancement Methods. Sci. Sin. Technol. 2026, 56, 55-78. Avaliable online: accessed on 1 April 2026). (In Chinese)

[20] Zhu YD, Xiao Q, Jia HJ, Lu WB, Mu YF, Jin Y. Adaptive Robust Optimization Strategy for AC/DC Distribution Networks Considering Autonomous Operation of Microgrids. J. Shanghai Jiaotong Univ. 2026, 60, 317-328. DOI:10.16183/j.cnki.jsjtu.2025.119(InChinese)