Active battery balancing is essential for maximizing the performance and safety of lithium-ion battery packs in electric vehicles and energy storage systems, yet traditional control methods struggle with nonlinear dynamics. This paper investigates the critical role of state-space design in tabular Q-learning for controlling switches of a buck-boost converter in a four-cell pack, addressing a key gap in the application of reinforcement learning to battery management systems. We propose and compare three novel discrete state representations: a coarse 11-state pairwise comparison, an intermediate 27-state hierarchical relational model, and a fine-grained 81-state individual deviation model. Through simulations across 1000 training episodes and 24 test scenarios, the 27-state model achieves superior convergence, with an average balancing time of around 41 timesteps and the lowest performance variance (σ = 12.28). Statistical analysis and state-transition graphs reveal that this optimal granularity enables hierarchical control strategies, balancing informational richness with learnability to avoid perceptual aliasing and the curse of dimensionality. These findings provide a blueprint for designing efficient RL policies in BMS, which has implications for scalable and real-time implementations in high-voltage applications.
| [1] |
A. Masias, J. Marcicki, and W. A. Paxton, “Opportunities and Challenges of Lithium Ion Batteries in Automotive Applications,” ACS Energy Letters 6, no. 2 (2021): 621-630.
|
| [2] |
G. Zubi, R. Dufo-López, M. Carvalho, and G. Pasaoglu, “The Lithium-Ion Battery: State of the Art and Future Perspectives,” Renewable and Sustainable Energy Reviews 89 (2018): 292-308.
|
| [3] |
T. Wulandari, D. Fawcett, S. B. Majumder, and G. E. J. Poinern, “Lithium-Based Batteries, History, Current Status, Challenges, and Future Perspectives,” Battery Energy 2, no. 6 (2023): 20230030, https://doi.org/10.1002/bte2.20230030.
|
| [4] |
T. Kim, W. Song, D.-Y. Son, L. K. Ono, and Y. Qi, “Lithium-Ion Batteries: Outlook on Present, Future, and Hybridized Technologies,” Journal of Materials Chemistry A 7, no. 7 (2019): 2942-2964.
|
| [5] |
S. Castano-Solis, D. Serrano-Jimenez, L. Gauchia, and J. Sanz, “The Influence of BMSs on the Characterization and Modeling of Series and Parallel Li-Ion Packs,” Energies 10, no. 3 (2017): 273.
|
| [6] |
X. Gong, R. Xiong, and C. C. Mi, “Study of the Characteristics of Battery Packs in Electric Vehicles With Parallel-Connected Lithium-Ion Battery Cells,” IEEE Transactions on Industry Applications 51, no. 2 (2015): 1872-1879.
|
| [7] |
J. Tian, Y. Fan, T. Pan, X. Zhang, J. Yin, and Q. Zhang, “A Critical Review on Inconsistency Mechanism, Evaluation Methods and Improvement Measures for Lithium-Ion Battery Energy Storage Systems,” Renewable and Sustainable Energy Reviews 189 (2024): 113978.
|
| [8] |
M. S. Ramkumar, C. Reddy, A. Ramakrishnan, et al., “Review on Li-Ion Battery With Battery Management System in Electrical Vehicle,” Advances in Materials Science and Engineering 2022, no. 1 (2022): 1-8.
|
| [9] |
J. P. D. Miranda, L. A. M. Barros, and J. G. Pinto, “A Review on Power Electronic Converters for Modular BMS With Active Balancing,” Energies 16, no. 7 (2023): 3255.
|
| [10] |
A. Zabetian-Hosseini, A. Ghazanfari, and B. Boulet, “A Finite-State Machine-Based Control Design for Thermal and State-of-Charge Balancing of Lithium Iron Phosphate Battery Using Flyback Converters,” Battery Energy 3, no. 4 (2024): 20230055.
|
| [11] |
T. Duraisamy and D. Kaliyaperumal, “Adaptive Passive Balancing in Battery Management System for e-Mobility,” International Journal of Energy Research 45, no. 7 (2021): 10752-10764.
|
| [12] |
S. Hemavathi, “Overview of Cell Balancing Methods for Li-Ion Battery Technology,” Energy Storage 203 (2020): 1-12.
|
| [13] |
U. Krishnamoorthy, G. S. Satheesh Kumar, S. Barua, and H. H. Fayek, “Investigation of Active Cell Balancing Performance for Series Connected Lithium-Ion Cells in Electric Vehicle Applications,” IET Power Electronics 16, no. 15 (2023): 2492-2503.
|
| [14] |
Z. Zhang, L. Zhang, L. Hu, and C. Huang, “Active Cell Balancing of Lithium-Ion Battery Pack Based on Average State of Charge,” International Journal of Energy Research 44, no. 4 (2020): 2535-2548.
|
| [15] |
A. Alvarez-Diazcomas, A. A. Estévez-Bén, J. Rodríguez-Reséndiz, M.-A. Martínez-Prado, R. V. Carrillo-Serrano, and S. Thenozhi, “A Review of Battery Equalizer Circuits for Electric Vehicle Applications,” Energies 13, no. 21 (2020): 5688.
|
| [16] |
Z. Zhao, H. Hu, Z. He, H. H.-C. Iu, P. Davari, and F. Blaabjerg, “Power Electronics-Based Safety Enhancement Technologies for Lithium-Ion Batteries: An Overview From Battery Management Perspective,” IEEE Transactions on Power Electronics 38, no. 7 (2023): 8922-8955.
|
| [17] |
J. Cao, B. Xia, and J. Zhou, “An Active Equalization Method for Lithium-Ion Batteries Based on Flyback Transformer and Variable Step Size Generalized Predictive Control,” Energies 14, no. 1 (2021): 207.
|
| [18] |
L. Zheng, J. Zhu, G. Wang, D. D.-C. Lu, P. McLean, and T. He, “Model Predictive Control Based Balancing Strategy for Series-Connected Lithium-ion Battery Packs.” 2017 19th European Conference on Power Electronics and Applications (EPE'17 ECCE Europe) (IEEE, 2017).
|
| [19] |
T. T. Ngoc Nguyen, H. G. Yoo, S. K. Oruganti, and F. Bien, “Neuro-Fuzzy Controller for Battery Equalisation in Serially Connected Lithium Battery Pack,” IET Power Electronics 8, no. 3 (2015): 458-466.
|
| [20] |
R. L. Velho, J. Pombo, J. B. L. Fermeiro, M. Calado, and S. Mariano, “Lookup Table Based Intelligent Charging and Balancing Algorithm for Li-Ion Battery Packs.” 2018 International Conference on Intelligent Systems (IS) (IEEE, 2018), 209-214.
|
| [21] |
Y. Tavakol-Moghaddam, M. Boroushaki, and M. Astaneh, “Reinforcement Learning for Battery Energy Management: A New Balancing Approach for Li-Ion Battery Packs,” Results in Engineering 23 (2024): 102532.
|
| [22] |
S. Harippriya, E. E. Vigneswaran, and S. Jayanthy, “Battery Management System to Estimate Battery Aging Using Deep Learning and Machine Learning Algorithms,” Journal of Physics: Conference Series 2325, no. 1 (2022): 012004-020041.
|
| [23] |
R. R. Ardeshiri, B. Balagopal, A. Alsabbagh, C. Ma, and M.-Y. Chow, “Machine Learning Approaches in Battery Management Systems: State of the Art: Remaining Useful Life and Fault Detection.” 2020 2nd IEEE International Conference on Industrial Electronics for Sustainable Energy Systems (IESES) (IEEE, 2020). 1, 61-66.
|
| [24] |
L. Buşoniu, T. De Bruin, D. Tolić, J. Kober, and I. Palunko, “Reinforcement Learning for Control: Performance, Stability, and Deep Approximators,” Annual Reviews in Control 46 (2018): 8-28.
|
| [25] |
B. Jiang, J. Tang, Y. Liu, and L. Boscaglia, “Active Balancing of Reconfigurable Batteries Using Reinforcement Learning Algorithms.” 2023 IEEE Transportation Electrification Conference & Expo (ITEC) (IEEE, 2023), 1-6.
|
| [26] |
B. Xia, F. Ding, S. Yue, and Y. Li, “An Intelligent Active Equalization Control Strategy Based on Deep Reinforcement Learning for Lithium-Ion Battery Pack,” Journal of Energy Storage 86 (2024): 111255.
|
| [27] |
B. Jang, M. Kim, G. Harerimana, and J. W. Kim, “Q-Learning Algorithms: A Comprehensive Classification and Applications,” IEEE Access 7 (2019): 133653-133667.
|
| [28] |
P. Suanpang and P. Jamjuntr, “Optimal Electric Vehicle Battery Management Using Q-Learning for Sustainability,” Sustainability 16, no. 16 (2024): 7180.
|
| [29] |
D. Karnehm, S. Pohlmann, and A. Neve, “State-of-Charge (SoC) Balancing of Battery Modular Multilevel Management (BM3) Converter Using Q-Learning.” 2023 IEEE Green Technologies Conference (GreenTech) (IEEE, 2023), 107-111.
|
| [30] |
C. Lu, J. Chen, C. Chen, Y. Huang, and D. Xuan, “An Active Equalization Method for Redundant Battery Based on Deep Reinforcement Learning,” Measurement 210 (2023): 112507.
|
| [31] |
“Battery, Generic Battery Model,” accessed May 15, 2025, https://ch.mathworks.com/help/sps/powersys/ref/battery.html.
|
| [32] |
A. V. D. B. Ali Farzan Moghaddam, “An Active Cell Equalization Technique for Lithium Ion Batteries Based on Inductor Balancing,” Presented at the 2018 9th International Conference on Mechanical and Aerospace Engineering, 2018.
|
| [33] |
C. Colas, O. Sigaud, and P.-Y. Oudeyer, “How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments,” arXiv Preprint arXiv:1806.08295, 2018.
|
RIGHTS & PERMISSIONS
2026 The Author(s). Battery Energy published by Xijing University and John Wiley & Sons Australia, Ltd.
PDF
