While mobile battery energy storage systems (MBESSs) are typically used to improve the stability of power systems, their ability to move also creates good opportunities for businesses to earn money through energy arbitrage. This profit depends heavily on decisions about timing and location, and is affected by uncertain conditions like fluctuating electricity prices and traffic. However, finding the best real-time control strategy that considers long-term profit and these uncertainties requires significant computing power. To tackle this issue, this paper presents a deep reinforcement learning framework for MBESSs designed to get the most profit from market arbitrage. Within this framework, we introduce the Knowledge-Assisted Deep Deterministic Policy Gradient (KA-DDPG) algorithm to learn the best policy more efficiently. The core novelty of KA-DDPG lies in its probabilistic hybrid action selection mechanism that unifies the agent's learned policy, offline expert criteria, and random exploration to manage the complex hybrid action space. Additionally, a two-phase guidance strategy is implemented to transition from offline-based to real-time-based criteria actions, ensuring both learning acceleration and policy robustness under computational constraints. Our rigorous statistical evaluations demonstrate that the proposed KA-DDPG approach leads to a 3%–7% improvement in average profits over the state-of-the-art Soft Actor-Critic baseline. Furthermore, it achieves exceptional policy stability, exhibiting a variance reduction of over 60% compared to standard DRL baselines and over 92% compared to the deterministic closed-loop MPC. The KA-DDPG algorithm also substantially speeds up the learning phase, validating its efficacy for real-time MBESS control under high uncertainty.
| [1] |
M. Eskandari, A. Rajabi, A. V. Savkin, M. H. Moradi, and Z. Y. Dong, “Battery Energy Storage Systems (BESSs) and the Economy-Dynamics of Microgrids: Review, Analysis, and Classification for Standardization of BESSs Applications,” Journal of Energy Storage 55 (2022): 105627.
|
| [2] |
H. Saboori and S. Jadid, “Mobile and Self-Powered Battery Energy Storage System in Distribution Networks-Modeling, Operation Optimization, and Comparison With Stationary Counterpart,” Journal of Energy Storage 42 (2021): 103068.
|
| [3] |
I. U. Khan and M. Jamil, “Energy Arbitrage Analysis for Market-Selection of a Battery Energy Storage System-Based Venture,” Energies 18, no. 16 (2025): 4245.
|
| [4] |
B. Faessler, “Stationary, Second Use Battery Energy Storage Systems and Their Applications: A Research Review,” Energies 14, no. 8 (2021): 2335.
|
| [5] |
C. Zhao, P. B. Andersen, C. Træholt, and S. Hashemi, “Grid-Connected Battery Energy Storage System: A Review on Application and Integration,” Renewable and Sustainable Energy Reviews 182 (2023): 113400.
|
| [6] |
N. Kharlamova, S. Hashemi, and C. Træholt, “Data-Driven Approaches for Cyber Defense of Battery Energy Storage Systems,” Energy and AI 5 (2021): 100095.
|
| [7] |
T. Wan, Y. Tao, J. Qiu, and S. Lai, “Data-Driven Hierarchical Optimal Allocation of Battery Energy Storage System,” IEEE Transactions on Sustainable Energy 12, no. 4 (2021): 2097–2109.
|
| [8] |
T. Nguyen-Duc, L. Hoang-Tuan, H. Ta-Xuan, L. Do-Van, and H. Takano, “A Mixed-Integer Programming Approach for Unit Commitment in Micro-Grid With Incentive-Based Demand Response and Battery Energy Storage System,” Energies 15, no. 19 (2022): 7192.
|
| [9] |
F. Garcia-Torres, C. Bordons, J. Tobajas, R. Real-Calvo, I. Santiago, and S. Grieu, “Stochastic Optimization of Microgrids With Hybrid Energy Storage Systems for Grid Flexibility Services Considering Energy Forecast Uncertainties,” IEEE Transactions on Power Systems 36, no. 6 (2021): 5537–5547.
|
| [10] |
J. Xu, X. Mei, H. Wang, H. Shi, Z. Sun, and Z. Zou, “A Model Based Balancing System for Battery Energy Storage Systems,” Journal of Energy Storage 49 (2022): 104114.
|
| [11] |
M. Bahramipanah, D. Torregrossa, R. Cherkaoui, and M. Paolone, “A Decentralized Adaptive Model-Based Real-Time Control for Active Distribution Networks Using Battery Energy Storage Systems,” IEEE Transactions on Smart Grid 9, no. 4 (2016): 3406–3418.
|
| [12] |
Z. Ni and S. Paul, “A Multistage Game in Smart Grid Security: A Reinforcement Learning Solution,” IEEE Transactions on Neural Networks and Learning Systems 30, no. 9 (2019): 2684–2695.
|
| [13] |
Y. Li, “Deep reinforcement learning: An overview,” arXiv preprint arXiv:1701.07274 (2017).
|
| [14] |
X. Wang, S. Wang, X. Liang, et al., “Deep Reinforcement Learning: A Survey,” IEEE Transactions on Neural Networks and Learning Systems 35, no. 4 (2022): 5064–5078.
|
| [15] |
P. Ladosz, L. Weng, M. Kim, and H. Oh, “Exploration in Deep Reinforcement Learning: A Survey,” Information Fusion 85 (2022): 1–22.
|
| [16] |
Q. Huang, R. Huang, W. Hao, J. Tan, R. Fan, and Z. Huang, “Adaptive Power System Emergency Control Using Deep Reinforcement Learning,” IEEE Transactions on Smart Grid 11, no. 2 (2019): 1171–1182.
|
| [17] |
T. Yang, L. Zhao, W. Li, and A. Y. Zomaya, “Dynamic Energy Dispatch Strategy for Integrated Energy System Based on Improved Deep Reinforcement Learning,” Energy 235 (2021): 121377.
|
| [18] |
T. Chen, X. Xu, H. Wang, and Z. Yan, “Routing and Scheduling of Mobile Energy Storage System for Electricity Arbitrage Based on Two-Layer Deep Reinforcement Learning,” IEEE Transactions on Transportation Electrification 9, no. 1 (2022): 1087–1102.
|
| [19] |
Y. Ding, X. Chen, and J. Wang, “Deep Reinforcement Learning-Based Method for Joint Optimization of Mobile Energy Storage Systems and Power Grid With High Renewable Energy Sources,” Batteries 9, no. 4 (2023): 219.
|
| [20] |
Y. Shang, W. Wu, J. Guo, et al., “Stochastic Dispatch of Energy Storage in Microgrids: An Augmented Reinforcement Learning Approach,” Applied Energy 261 (2020): 114423.
|
| [21] |
G. Xu, J. Shi, J. Wu, et al., “An Optimal Solutions-Guided Deep Reinforcement Learning Approach for Online Energy Storage Control,” Applied Energy 361 (2024): 122915.
|
| [22] |
H. Van Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learning With Double q-Learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1 (2016).
|
| [23] |
H. Zhao, J. Zhao, J. Qiu, G. Liang, and Z. Y. Dong, “Cooperative Wind Farm Control With Deep Reinforcement Learning and Knowledge-Assisted Learning,” IEEE Transactions on Industrial Informatics 16, no. 11 (2020): 6912–6921.
|
| [24] |
Y. Zheng, D. J. Hill, K. Meng, and S. Hui, “Critical Bus Voltage Support in Distribution Systems With Electric Springs and Responsibility Sharing,” IEEE Transactions on Power Systems 32, no. 5 (2016): 3584–3593.
|
| [25] |
J. Long, W. Tan, W. Szeto, and Y. Li, “Ride-Sharing With Travel Time Uncertainty,” Transportation Research Part B: Methodological 118 (2018): 143–171.
|
| [26] |
X. Liu, Y. Wang, K. Jiang, Z. Zhou, K. Nam, and C. Yin, “Interactive Trajectory Prediction Using a Driving Risk Map-Integrated Deep Learning Method for Surrounding Vehicles on Highways,” IEEE Transactions on Intelligent Transportation Systems 23, no. 10 (2022): 19076–19087.
|
| [27] |
L. Timilsina, P. R. Badr, P. H. Hoang, G. Ozkan, B. Papari, and C. S. Edrington, “Battery Degradation in Electric and Hybrid Electric Vehicles: A Survey Study,” IEEE Access 11 (2023): 42431–42462.
|
| [28] |
H. Zhao, J. Zhao, J. Qiu, et al., “Data-Driven Risk Preference Analysis in Day-Ahead Electricity Market,” IEEE Transactions on Smart Grid 12, no. 3 (2020): 2508–2517.
|
| [29] |
Transportation Networks for Research Core Team. “Transportation Networks for Research,” [Online]. https://github.com/bstabler/TransportationNetworks (2023).
|
| [30] |
Y. Wang, D. Qiu, and G. Strbac, “Multi-Agent Deep Reinforcement Learning for Resilience-Driven Routing and Scheduling of Mobile Energy Storage Systems,” Applied Energy 310 (2022): 118575.
|
| [31] |
S. Yao, P. Wang, X. Liu, H. Zhang, and T. Zhao, “Rolling Optimization of Mobile Energy Storage Fleets for Resilient Service Restoration,” IEEE Transactions on Smart Grid 11, no. 2 (2019): 1030–1043.
|
RIGHTS & PERMISSIONS
2026 The Author(s). Battery Energy published by Xijing University and John Wiley & Sons Australia, Ltd.
PDF
(3239KB)
