Supervisory control of the hybrid off-highway vehicle for fuel economy improvement using predictive double Q-learning with backup models

Abstract

This paper studied a supervisory control system for a hybrid off-highway electric vehicle under the charge-sustaining (CS) condition. A new predictive double Q-learning with backup models (PDQL) scheme is proposed to optimize the engine fuel in real-world driving and improve energy efficiency with a faster and more robust learning process. Unlike the existing “model-free” methods, which solely follow on-policy and off-policy to update knowledge bases (Q-tables), the PDQL is developed with the capability to merge both on-policy and off-policy learning by introducing a backup model (Q-table). Experimental evaluations are conducted based on software-in-the-loop (SiL) and hardware-in-the-loop (HiL) test platforms based on real-time modelling of the studied vehicle. Compared to the standard double Q-learning (SDQL), the PDQL only needs half of the learning iterations to achieve better energy efficiency than the SDQL at the end learning process. In the SiL under 35 rounds of learning, the results show that the PDQL can improve the vehicle energy efficiency by 1.75% higher than SDQL. By implementing the PDQL in HiL under four predefined real-world conditions, the PDQL can robustly save more than 5.03% energy than the SDQL scheme.

Keywords

supervisory charge-sustaining control / hybrid electric vehicle / reinforcement learning / predictive double Q-learning

References

[1] TranM K, BhattiA, VrolykR, et al.. A review of range extenders in battery electric vehicles: Current progress and future perspectives [J]. World Electric Vehicle Journal, 2021, 12(2): 54

[2] GönülÖ, DumanA, GülerÖ. Electric vehicles and charging infrastructure in Turkey: An overview [J]. Renewable and Sustainable Energy Reviews, 2021, 143: 110913

[3] European Commission. Proposal for post — 2020 CO₂ targets for cars and vans Climate action [R]. 2017.

[4] GREGOR E. EU legislation in progress CO₂ emission standards for heavy-duty vehicles [R]. December 2018.

[5] APC. Roadmap 2020 heavy goods >3.5 t and off-highway vehicle [R]. 2020.

[6] ZhouQ, ZhangY, LiZ, et al.. Cyber-physical energy-saving control for hybrid aircraft-towing tractor based on online swarm intelligent programming [J]. IEEE Trans Ind Informatics, 2018, 1494149-4158

[7] PérezL V, BossioG R, MoitreD, et al.. Optimization of power management in an hybrid electric vehicle using dynamic programming [J]. Mathematics and Computers in Simulation, 2006, 73(1–4): 244-254

[8] ZhouQ, ZhaoD, ShuaiB, et al.. Knowledge implementation and transfer with an adaptive learning network for real-time power management of the plug-in hybrid vehicle [J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(12): 5298-5308

[9] ZhouQ, LiY, ZhaoD, et al.. Transferable representation modelling for real-time energy management of the plug-in hybrid vehicle based on k-fold fuzzy learning and Gaussian process regression [J]. Applied Energy, 2022, 305117853

[10] ZhouQ, ZhangW, CashS, et al.. Intelligent sizing of a series hybrid electric power-train system based on chaos-enhanced accelerated particle swarm optimization [J]. Applied Energy, 2017, 189588-601

[11] ZhouQ, HeY, ZhaoD, et al.. Modified particle swarm optimization with chaotic attraction strategy for modular design of hybrid powertrains [J]. IEEE Transactions on Transportation Electrification, 2021, 7(2): 616-625

[12] LiJ, ZhouQ, HeY, et al.. Dual-loop online intelligent programming for driver-oriented predict energy management of plug-in hybrid electric vehicles [J]. Applied Energy, 2019, 253: 113617

[13] ZhouQ, GuoS, XuL, et al.. Global optimization of the hydraulic-electromagnetic energy-harvesting shock absorber for road vehicles with human-knowledge-integrated particle swarm optimization scheme [J]. IEEE/ASME Transactions on Mechatronics, 2021, 26(3): 1225-1235

[14] PourabdollahM, EgardtB, MurgovskiN, et al.. Convex optimization methods for powertrain sizing of electrified vehicles by using different levels of modeling details [J]. IEEE Transactions on Vehicular Technology, 2018, 67(3): 1881-1893

[15] SHUAI B, ZHOU Q, WILLIAMS H, et al. Impact of exploration-to-exploitation ratio on energy-saving potential of plug-in hybrid vehicles controlled, 2021.

[16] LuC, WangH, LvC, et al.. Learning driver-specific behavior for overtaking: A combined learning framework [J]. IEEE Transactions on Vehicular Technology, 2018, 67(8): 6788-6802

[17] ApolinarskaA A, PacherM, LiH, et al.. Robotic assembly of timber joints using reinforcement learning [J]. Automation in Construction, 2021, 125103569

[18] LIU Ruo-ze, GUO Hai-feng, JI Xiao-zhong, et al. Efficient reinforcement learning for StarCraft by abstract forward models and transfer learning [J]. IEEE Transactions on Games, 2021: 1. DOI: https://doi.org/10.1109/tg.2021.3071162.

[19] XingY, LvC, WangH, et al.. An ensemble deep learning approach for driver lane change intention inference [J]. Transportation Research Part C: Emerging Technologies, 2020, 115102615

[20] ZHU Zhao-xuan, PIVARO N, GUPTA S, et al. Safe model-based off-policy reinforcement learning for eco-driving in connected and automated hybrid electric vehicles [J]. IEEE Transactions on Intelligent Vehicles, 2022: 1. DOI: https://doi.org/10.1109/tiv.2022.3150668.

[21] ZHAO Dong-bin, WANG Hai-tao, KUN Shao, et al. Deep reinforcement learning with experience replay based on SARSA [C]//2016 IEEE Symposium Series on Computational Intelligence. December 6–9, 2016, Athens. IEEE, 2016: 1–6. DOI: https://doi.org/10.1109/SSCI.2016.7849837.

[22] LLORENTE F, MARTINO L, READ J, et al. A survey of Monte Carlo methods for noisy and costly densities with application to reinforcement learning [EB/OL]. 2021: arXiv: 2108.00490[cs.LG]. https://arxiv.org/abs/2108.00490.

[23] TanH, ZhangH, PengJ, et al.. Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space [J]. Energy Conversion and Management, 2019, 195548-560

[24] LUO W, TANG Q, FU C, et al. Deep-sarsa based multi-uav path planning and obstacle avoidance in a dynamic environment [J]. Lect Notes Comput Sci, 2018: 102–111. DOI: https://doi.org/10.1007/978-3-319-93818-9_10.

[25] LIU T, TANG X, CHEN J, et al. Transferred energy management strategies for hybrid electric vehicles based on driving conditions recognition [C]//2020 IEEE Veh Power Propuls Conf. VPPC 2020-Proc, 2020.

[26] KOUCHE-BIYOUKI S A, NASERI-JAVARESHK S M A, NOORI A et al. Power management strategy of hybrid vehicles using Sarsa method [C]//Electrical Engineering (ICEE), Iranian Conference on. 2018: 946–950.

[27] ZHU Z, GUPTA S, GUPTA A, et al. A deep reinforcement learning framework for eco-driving in connected and automated hybrid electric vehicles [OL]. arXiv: 2101.05372v2.

[28] LiuT, ZouY, LiuD, et al.. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle [J]. IEEE Trans Ind Electron, 2015, 62(12): 7837-7846

[29] ZhouQ, LiJ, ShuaiB, et al.. Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle [J]. Applied Energy, 2019, 255: 113755

[30] van HASSELT H. Double Q-learning [R]. Multi-agent and Adaptive Computation Group, Centrum Wiskunde & Information, 2010: 1–9.

[31] HanX, HeH, WuJ, et al.. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle [J]. Applied Energy, 2019, 254113708

[32] ZhangJ, JiaoX, YangC. A double-deep Q-network-based energy management strategy for hybrid electric vehicles under variable driving cycles [J]. Energy Technology, 2021, 9(2): 2000770

[33] YuY, CaiZ, LiuY. Double deep Q-learning coordinated control of hybrid energy storage system in island micro-grid [J]. International Journal of Energy Research, 2021, 4523315-3326

[34] ShuaiB, ZhouQ, LiJ, et al.. Heuristic action execution for energy efficient charge-sustaining control of connected hybrid vehicles with model-free double Q-learning [J]. Applied Energy, 2020, 267114900

[35] REN Z, ZHU G, HU H, et al. On the estimation bias in double Q-learning [OL]. arXiv:2109.14419.

[36] HuX, MurgovskiN, JohannessonL, et al.. Energy efficiency analysis of a series plug-in hybrid electric bus with different energy management strategies and battery sizes [J]. Applied Energy, 2013, 1111001-1009

[37] SchilperoortJ, MakI, DruganM M, et al.. Learning to play pac-xon with Q-learning and two double Q-learning variants [C]. 2018 IEEE Symposium Series on Computational Intelligence, 2018, Bangalore, India, IEEE, 11511158