Multi-Objective Optimization Design of An Integrated Energy System with Ice Storage Based on Deep Reinforcement Learning

Abstract

This study proposes an integrated energy system that combines photovoltaic power, wind power, battery storage, and ice storage to meet the electricity and cooling demands of buildings. The model for the ice storage tank incorporates the nonlinear ice-melting characteristics. An improved Multi-Objective Proximal Policy Optimization algorithm is employed for multi-objective optimization. In a case study of an office building in Shanghai, the optimization results demonstrate that the proposed method reduces daily operating costs by 6.52% and improves the CO₂ emission reduction rate by 9.54%. The results demonstrate that the synergistic operation of electrical and ice storage effectively maintains supply-demand balance across different seasons. Sensitivity analysis further reveals that a 40% reduction in the unit cost of ice storage leads to a 5.7% decrease in battery capacity and a significant drop in grid dependency from 28.9% to 15.3%, highlighting the critical role of reducing ice storage costs in improving the system’s economic viability and renewable energy integration capability.

Keywords

Integrated energy system / MOPPO / Ice storage / Multi-objective optimization

References

[1] IEA. Buildings-Energy System. Available online: https://www.iea.org/energy-system/buildingsaccessed on 1 April 2025).

[2] Xiong J, Ye Y, Wang Q, Dong X, Lu T, Ma D . A comprehensive review on distributed energy cooperative control and optimization method for energy interconnection system. Electr. Power Syst. Res. 2024, 237, 111007. DOI: 10.1016/j.epsr.2024.111007

[3] Wang M, Yu H, Yang Y, Lin X, Guo H, Li C, et al. Unlocking emerging impacts of carbon tax on integrated energy systems through supply and demand co-optimization. Appl. Energy 2021, 302, 117579. DOI: 10.1016/j.apenergy.2021.117579

[4] Sehar F, Pipattanasomporn M, Rahman S . An energy management model to study energy and peak power savings from PV and storage in demand responsive buildings. Appl. Energy 2016, 173, 406-417. DOI: 10.1016/j.apenergy.2016.04.039

[5] Ullah K, Ali S, Khan TA, Khan I, Jan S, Shah IA, et al. An Optimal Energy Optimization Strategy for Smart Grid Integrated with Renewable Energy Sources and Demand Response Programs. Energies 2020, 13, 5718. DOI: 10.3390/en13215718

[6] Karimi H. A tri-objectives scheduling model for renewable-hydrogen-based microgrid system considering hydrogen storage system and demand-side management. Int. J. Hydrogen Energy 2024, 68, 1412-1422. DOI: 10.1016/j.ijhydene.2024.04.330

[7] Zhu X, Gui P, Zhang X, Han Z, Li Y . Multi-objective optimization of a hybrid energy system integrated with solar-wind-PEMFC and energy storage. J. Energy Storage 2023, 72, 108562. DOI: 10.1016/j.est.2023.108562

[8] Hua LG, Bilal M, Hafeez G, Ali S, Alghamdi B, Alsafran AS, et al. An energy-efficient system with demand response, distributed generation, and storage batteries for energy optimization in smart grids. J. Energy Storage 2025, 117, 115491. DOI: 10.1016/j.est.2025.115491

[9] Sabareesh SU, Aravind KSN, Chowdary KB, Syama S, Devi VSK . LSTM Based 24 hours Ahead Forecasting of Solar PV System for Standalone Household System. Procedia Comput. Sci. 2023, 218, 1304-1313. DOI: 10.1016/j.procs.2023.01.109

[10] Cui X, Zhu J, Jia L, Wang J, Wu Y . A novel heat load prediction model of district heating system based on hybrid whale optimization algorithm (WOA) and CNN-LSTM with attention mechanism. Energy 2024, 312, 133536. DOI: 10.1016/j.energy.2024.133536

[11] Liu B, Xie Y, Wang K, Yu L, Zhou Y, Lv X . Short-Term Multi-Step Wind Direction Prediction Based on OVMD Quadratic Decomposition and LSTM. Sustainability 2023, 15, 11746. DOI: 10.3390/su151511746

[12] Wang L, Mao M, Xie J, Liao Z, Zhang H, Li H . Accurate solar PV power prediction interval method based on frequency-domain decomposition and LSTM model. Energy 2023, 262, 125592. DOI: 10.1016/j.energy.2022.125592

[13] Chen B, Li Z, Li S, Zhao Q, Liu X . A Wind Power Prediction Framework for Distributed Power Grids. Energy 2024, 121, 1291-1307. DOI: 10.32604/ee.2024.046374

[14] Ma Y. Multi-objective optimization model and algorithm implementation of the distributed power generation system for renewable energy in China and Russia. Unconv. Resour. 2025, 7, 100201. DOI: 10.1016/j.uncres.2025.100201

[15] Wang C, Jiao S, Zhang Y, Li Y, Li Y, Zhang Q . A new transfer evolutionary multi-task optimization algorithm for bi-level optimal configuration of distributed generations and energy storage systems considering uncertainties. J. Energy Storage 2025, 113, 115556. DOI: 10.1016/j.est.2025.115556

[16] Yau YH, Rismanchi B . A review on cool thermal storage technologies and operating strategies. Renew. Sustain. Energy Rev. 2012, 16, 787-797. DOI: 10.1016/j.rser.2011.09.004

[17] Curto D, Favuzza S, Franzitta V, Guercio A, Amparo Navarro Navia M, Telaretti E, et al. Grid Stability Improvement Using Synthetic Inertia by Battery Energy Storage Systems in Small Islands. Energy 2022, 254, 124456. DOI: 10.1016/j.energy.2022.124456

[18] Saffari M, de Gracia A, Fernández C, Belusko M, Boer D, Cabeza LF . Optimized demand side management (DSM) of peak electricity demand by coupling low temperature thermal energy storage (TES) and solar PV. Appl. Energy 2018, 211, 604-616. DOI: 10.1016/j.apenergy.2017.11.063

[19] Al-Aali I, Narayanaswamy A, Modi V . Design of chiller system with thermal and battery storage for enhanced integration with on-site PV. J. Build. Eng. 2024, 87, 109072. DOI: 10.1016/j.jobe.2024.109072

[20] Al-Aali I, Modi V. Decarbonizing the electricity sector in Qatar using PV combined with ice thermal and battery storage. Energy Strategy Rev. 2022, 44, 101014. DOI: 10.1016/j.esr.2022.101014

[21] Odufuwa OY, Kusakana K, Numbi BP, Tartibu LK . Optimal energy management of grid-connected PV for HVAC cooling with ice thermal storage system. J. Energy Storage 2024, 77, 109844. DOI: 10.1016/j.est.2023.109844

[22] Luerssen C, Gandhi O, Reindl T, Sekhar C, Cheong D . Life cycle cost analysis (LCCA) of PV-powered cooling systems with thermal energy and battery storage for off-grid applications. Appl. Energy 2020, 273, 115145. DOI: 10.1016/j.apenergy.2020.115145

[23] Arévalo P, Benavides D, Lata-García J, Jurado F . Techno-economic evaluation of renewable energy systems combining PV-WT-HKT sources: Effects of energy management under Ecuadorian conditions. Int. Trans. Electr. Energy Syst. 2020, 30, e12567. DOI: 10.1002/2050-7038.12567

[24] Qiao Q, Yunusa-Kaltungo A, Edwards RE . Feature selection strategy for machine learning methods in building energy consumption prediction. Energy Rep. 2022, 8, 13621-13654. DOI: 10.1016/j.egyr.2022.10.125

[25] Cha JH, Park J, Yeon SH, Yoon Y, Lee KH . Particle Swarm Optimization for multi-chiller system: Capacity configuration and load distribution. J. Build. Eng. 2024, 98, 110953. DOI: 10.1016/j.jobe.2024.110953

[26] Chen C, Duan S, Cai T, Liu B, Hu G . Optimal Allocation and Economic Analysis of Energy Storage System in Microgrids. IEEE Trans. Power Electron. 2011, 26, 2762-2773. DOI: 10.1109/TPEL.2011.2116808

[27] Benfatma H, Khouidmi H, Bessedik B . Neural network and ACO algorithm-tuned PI controller for MPPT in a hybrid battery-supercapacitor energy storage system within DC micro-grid photovoltaic installations. J. Energy Storage 2025, 120, 116499. DOI: 10.1016/j.est.2025.116499

[28] Zakaria NF, Sulaiman MH, Mustaffa Z . Feature optimization with metaheuristics for Artificial Neural Network-based chiller power prediction. J. Build. Eng. 2025, 105, 112561. DOI: 10.1016/j.jobe.2025.112561

[29] Chen B, Zeng W, Nie H, Deng Z, Yang W, Yan B . Optimal load distribution control for airport terminal chiller units based on deep reinforcement learning. J. Build. Eng. 2024, 97, 110787. DOI: 10.1016/j.jobe.2024.110787

[30] Biemann M, Scheller F, Liu X, Huang L . Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control. Appl. Energy 2021, 298, 117164. DOI: 10.1016/j.apenergy.2021.117164

[31] Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529-533. DOI: 10.1038/nature14236

[32] Peng H, Yuan D, Jiang K, Xu W. PPO-based fluctuation-consumption co-optimized energy management for wind-disturbed fuel cell hybrid electric flying car. J. Power Sources 2025, 647, 237349. DOI: 10.1016/j.jpowsour.2025.237349

[33] Ning D, Chen X, Chen J, Meng T, Xu B, Zhang H . PPO-MixClip: An energy scheduling algorithm for low-carbon parks. Energy Rep. 2024, 12, 4195-4207. DOI: 10.1016/j.egyr.2024.09.042

[34] Singh A, Panigrahi BK . A HEN-PPO strategy for home energy management system with reduce EV anxieties. e-Prime Adv. Electr. Eng. Electron. Energy 2024, 10, 100871. DOI: 10.1016/j.prime.2024.100871

[35] Bai L, Tan Z . Optimizing energy efficiency, thermal comfort, and indoor air quality in HVAC systems using a robust DRL algorithm. J. Build. Eng. 2024, 98, 111493. DOI: 10.1016/j.jobe.2024.111493

[36] Ge Y, Xie J, Chang J, Feng S . A multi-objective deep reinforcement learning method for intelligent scheduling of wind-solar-hydro-battery complementary generation systems. Int. J. Electr. Power Energy Syst. 2025, 167, 110635. DOI: 10.1016/j.ijepes.2025.110635

[37] Long Short-Term Memory-An Overview|ScienceDirect Topics. Available online: https://www.sciencedirect.com/topics/engineering/long-short-term-memoryaccessed on 17 May 2025).

[38] Diniz APM, Ciarelli PM, Salles EOT, Coco KF . Use of deep neural networks for clogging detection in the Submerged Entry Nozzle of the continuous casting. Expert Syst. Appl. 2024, 238, 121963. DOI: 10.1016/j.eswa.2023.121963

[39] Tang C, Li N, Bao L . Predictive Control Modeling of Regional Cooling Systems Incorporating Ice Storage Technology. Buildings 2024, 14, 2488. DOI: 10.3390/buildings14082488

[40] Kang J, Wang J, Liu C, Ye S, Yang M . Coordinated optimization of configuration and operation of a photovoltaic integrated building cooling system with electricity and ice storages under source-load uncertainties. Energy Build. 2024, 320, 114600. DOI: 10.1016/j.enbuild.2024.114600

[41] Chen Q, Wei W, Li N . Techno-economic control strategy optimization for water-source heat pump coupled with ice storage district cooling system. Int. J. Refrig. 2022, 138, 148-158. DOI: 10.1016/j.ijrefrig.2022.03.010

[42] Ren X, Han Z, Ma J, Xue K, Chong D, Wang J, et al. Life-cycle-based multi-objective optimal design and analysis of distributed multi-energy systems for data centers. Energy 2024, 288, 129679. DOI: 10.1016/j.energy.2023.129679

[43] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O . Proximal Policy Optimization Algorithms. arXiv 2017, arXiv:1707.06347. DOI: 10.48550/arXiv.1707.06347

[44] Dingcheng H, Metwally ASM, Ali S, Sillanpaa M, Yassen W, Sobhani B . Sustainable and environmentally-friendly multi-generation system of power, cooling, and hydrogen; Multi-objective optimization using particle swarm algorithm. Appl. Therm. Eng. 2023, 225, 120093. DOI: 10.1016/j.applthermaleng.2023.120093

[45] He W, Xu Q, Guo R, Liu S, Wang Y, Guo H, et al. Optimization analysis of different distributed energy configurations for data centers. Sustain. Mater. Technol. 2024, 41, e01005. DOI: 10.1016/j.susmat.2024.e01005

[46] DKA Solar Centre. Available online: https://dkasolarcentre.com.au/accessed on 3 June 2025).

[47] Chen Y, Xu J . Solar and wind power data from the Chinese State Grid Renewable Energy Generation Forecasting Competition. Sci. Data 2022, 9, 577. Available online: https://www.nature.com/articles/s41597-022-01696-6accessed on 3 June 2025).

[48] Yan R, Zhao T, Rezgui Y, Kubicki S, Li Y. Transferability and robustness of a data-driven model built on a large number of buildings. J. Build. Eng. 2023, 80, 108127. DOI: 10.1016/j.jobe.2023.108127

[49] Yan R, Chen Z, Zhang X, Zhao T, Rezgui Y, Li Y . A novel model ensemble method based on self-adaptive weight for building energy transfer learning. J. Build. Eng. 2025, 109, 113024. DOI: 10.1016/j.jobe.2025.113024