Attention-driven electrical-computational cooperative scheduling approach via hierarchical multi-agent reinforcement learning

Abstract

The surge of data center loads and high renewable penetration threatens power system stability, requiring joint power-computing scheduling. This paper presents an attention-driven hierarchical multi-agent proximal policy optimization (H-MAPPO) method to solve the coordination issue. A Transformer-based fine-grained task-computing-power coupling model is built to map diverse computing tasks to grid loads. The grid-computing interaction is formulated as a Stackelberg game via bi-level Markov decision process (MDP). The upper agent conducts global dispatch with proximal policy optimization (PPO), and lower agents use attention-based coordination algorithm PPO (ACA-PPO) for distributed task scheduling. Tests on German European network of transmission system operators for electricity (ENTSO-E) grid data and Google cluster traces reveal H-MAPPO attains 98.36% profit of the optimal mixed integer quadratic programming (MIQP), shortens solving time from 342.6 s to 2.5 s, and lifts task completion rate by 21.6% versus hierarchical independent proximal policy optimization (H-IPPO). The hierarchical attention-aided framework supports scalable real-time power-computing collaborative scheduling for low-carbon grids.

Keywords

computing and power collaboration / adjustable resources / double-layer optimization / optimize scheduling

References

[1] VERSTRAETEN T, DAEMS P J, BARGIACCHI E, et al. Scalable optimization for wind farm control using coordination graphs. arXiv Preprint, arXiv:2101.07844, 2021.

[2] BAHI A, OURICI A . Deep reinforcement learning for real-time green energy integration in data centers. arXiv Preprint, arXiv:2507.21153, 2025.

[3] ZHOU Y C, WEI F C, LI S X, et al. Data center load modeling through optimal energy consumption characteristics: A path to simultaneously enhance energy efficiency and demand response quality. Applied Energy, 2025, 393(C): Article 126095.

[4] GUO H J, YU H, W M, et al. Integrated management of workloads and energy system for data centers. Energy, 2025, 327: Article 136400.

[5] GUO C Y, WANG X, ZHENG Y H, et al. Optimal energy management of multi-microgrids connected to distribution system based on deep reinforcement learning. International Journal of Electrical Power & Energy Systems, 2021, 131: Article 107048.

[6] XU W K, CHU Z D, TENG F . LAPSO: A unified optimization view for learning-augmented power system operations. arXiv Preprint, arXiv:2505.05203, 2025.

[7] LI J H, ZHANG R C, WANG H, et al. Deep reinforcement learning for optimal power flow with renewables using spatial-temporal graph information. arXiv Preprint, arXiv:2112.11461, 2021.

[8] YU C, VELU A, VINITSKY E, et al. The surprising effectiveness of PPO in cooperative, multi-agent games. Advances in Neural Information Processing Systems, 2022, 35: 24611-24624.

[9] WANG S Y, ZHUGE C X, SHAO C F, et al. Short-term electric vehicle charging demand prediction: A deep learning approach. Applied Energy, 2023, 340: Article 121032.

[10] BOX G E P, JENKINS G M, REINSEL G C, et al. Time series analysis: forecasting and control. New York, NY, USA: John Wiley & Sons, 2015.

[11] KOOT M, WIJNHOVEN F . Usage impact on data center electricity needs: A system dynamic forecasting model. Applied Energy, 2021, 291: Article 116798.

[12] MYTTON D, ASHTINE M . Sources of data center energy estimates: A comprehensive review. Joule, 2022, 6(9): 2032-2056.

[13] SETYO Z G M, RIJAL H B, AQILAH N, et al. Energy efficiency measurement method and thermal environment in data centers: A literature review. Energies, 2025, 18(14): Article 3689.

[14] XU Y C, CHEN L L, LU Z H, et al. An adaptive mechanism for dynamically collaborative computing power and task scheduling in edge environment. IEEE Internet of Things Journal, 2023, 10(4): 3118-3129.

[15] GANDHI A, LEE D, LIU Z H, et al. Metrics for sustainability in data centers. ACM SIGENERGY Energy Informatics Review, 2023, 3(3): 40-46.

[16] ZHAO D M, ZHOU J T, ZHAI J d, et al. A reinforcement learning based framework for holistic energy optimization of sustainable cloud data centers. IEEE Transactions on Services Computing, 2025, 18(1): 15-28.

[17] RADOVANOVIC A, CHEN B, TALUKDAR S, et al. Power modeling for effective datacenter planning and compute management. IEEE Transactions on Smart Grid, 2022, 13(2): 1611-1621.

[18] AL-JUMAILI A H A, MUNIYANDI R C, HASAN M K, et al. Big data analytics using cloud computing based frameworks for power management systems: Status, constraints, and future recommendations. Sensors, 2023, 23(6): Article 2952.

[19] LATIF I, NEWKIRK A C, CARBONE M R, et al. Empirical measurements of AI training power demand on a GPU-accelerated node. arXiv Preprint, arXiv:2412.08602, 2024.

[20] WU J J, ZENG Z, XU S Y, et al. Unified multi-party representation model based power dispatch mechanism for distributed renewable energy systems. The Journal of China Universities of Posts and Telecommunications, 2025, 32(6): 13-31.

[21] YANG D F, WANG X Y, SHEN R D, et al. Global optimization strategy of prosumer data center system operation based on multi-agent deep reinforcement learning. Journal of Building Engineering, 2024, 91: Article 109519.

[22] MOHINI C, SHUBHAM G, DHANASHREE J. Optimizing power system observability using GAMS. Proceedings of the IEEE PES 14th Asia-Pacific Power and Energy Engineering Conference (APPEEC'22), 2022, Nov 20-23, Melbourne, Australia. Piscataway, NJ, USA: IEEE, 2022: 6p.

[23] KAPOOR A, FREED B, CHOSET H, et al. Assigning credit with partial reward decoupling in multi-agent proximal policy optimization. arXiv Preprint, arXiv:2408.04295, 2024.

[24] KONG H, XING Q L, WANG Q, et al. ADAC: Actor-double-attention-critic for multi-agent cooperation in mixed cooperative-competitive environments. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(7): 9579-9592.

[25] ZHANG Y B, CHEN X, GONG S M, et al. Collective large-scale wind farm multivariate power output control based on hierarchical communication multi-agent proximal policy optimization. Renewable Energy, 2023, 219: Article 119479.

[26] KUMAR H, KOPPEL A, RIBEIRO A . On the sample complexity of actor-critic method for reinforcement learning with function approximation. Machine Learning, 2023, 112(7): 2433-2467.

[27] DUAN J L, GUAN Y, LI S E, et al. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on neural networks and learning systems, 2022, 33(11): 6584-6598.

[28] GU Y, CHENG Y H, CHEN C L P, et al. Proximal policy optimization with policy feedback. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022, 52(7): 4600-4610.

[29] JUNG S W, AN Y Y, SUH B K, et al. Multi-agent deep reinforcement learning for scheduling of energy storage system in microgrids. Mathematics, 2025, 13(12): Article 1999.

[30] OROOJLOOY A, HAJINEZHAD D . A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 2023, 53(11): 13677-13722.

[31] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms. arXiv Preprint, arXiv:1707.06347, 2017.

[32] XIAO B C, RAMASUBRAMANIAN B, POOVENDRAN R . Agent-temporal attention for reward redistribution in episodic multi-agent reinforcement learning. Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS'22), 2022, May 9-13, Online. 2022.

[33] ZHAN X Y, ZHU X Y, CHENG P, et al. Data center cooling system optimization using offline reinforcement learning. arXiv Preprint, arXiv:2501.15085, 2025.

[34] HIRTH L, MÜHLENPFORDT J, BULKELEY M. The ENTSO-E transparency platform: A review of Europe's most ambitious electricity data platform. Applied energy, 2018, 225: 1054-1067.

[35] REISS C, WILKES J, HELLERSTEIN J L. Google cluster-usage traces: format + schema. White Paper. Google Inc, 2011.