A multi-objective optimization approach for the virtual coupling train set driving strategy

Abstract

This paper presents an improved virtual coupling train set (VCTS) operation control framework to deal with the lack of optimization of speed curves in the traditional techniques. The framework takes into account the temporary speed limit on the railway line and the communication delay between trains, and it uses a VCTS consisting of three trains as an experimental object. It creates the virtual coupling train tracking and control process by improving the driving strategy of the leader train and using the leader–follower model. The follower train uses the improved speed curve of the leader train as its speed reference curve through knowledge migration, and this completes the multi-objective optimization of the driving strategy for the VCTS. The experimental results confirm that the deep reinforcement learning algorithm effectively achieves the optimization goal of the train driving strategy. They also reveal that the intrinsic curiosity module prioritized experience replay dueling double deep Q-network (ICM-PER-D3QN) algorithm outperforms the deep Q-network (DQN) algorithm in optimizing the driving strategy of the leader train. The ICM-PER-D3QN algorithm enhances the leader train driving strategy by an average of 57% when compared to the DQN algorithm. Furthermore, the particle swarm optimization (PSO)-based model predictive control (MPC) algorithm has also demonstrated tracking accuracy and further improved safety during VCTS operation, with an average increase of 37.7% in tracking accuracy compared to the traditional MPC algorithm.

Keywords

High-speed trains / Virtual coupling / Multi-objective optimization / Deep reinforcement learning / Knowledge transfer / Model predictive control

References

[1.] Liu H, Lang Y, Zhang L, et al. A cooperative control based reference curve generating method for virtually coupled train sets. Sci Technol Rev, 2023, 41(1):62-72.

[2.] Cao Y, Wen J, Ma L. Tracking and collision avoidance of virtual coupling train control system. Future Gener Comp Sys, 2021, 120: 76-90.

[3.] Luo X, Tang T, Lin BY, et al. A robust model predictive control approach for reducing following distance between virtually coupled unit trains. J Chin Railway Soc, 2023, 45(8):68-76.

[4.] Lin JT, Ni MJ. Trigger model predictive control based on extended state observers for virtual coupling. J Transp Syst Eng Inf Technol, 2023, 23(4):134-146.

[5.] Su S, Liu W, Zhu Q, et al. A cooperative collision-avoidance control methodology for virtual coupling trains. Accident Anal Prev, 2022, 173: 106703.

[6.] Xi W, Hu M, Wang H, et al. Formation control for virtual coupling trains with parametric uncertainty and unknown disturbances. IEEE T Circuits-II, 2023, 70(9):3429-3433.

[7.] Yang AA, Sun JY, Wang B, et al. Optimization of virtual-coupling-orientated train operation plan based on full-length and short-turn routing. J Beijing Jiaotong University, 2022, 46(4):9-14 (in Chinese)

[8.] Luo X, Tang T, Li K, et al. Computation-efficient distributed MPC for dynamic coupling of virtually coupled train set. Control Eng Pract, 2024, 145: 105846.

[9.] Cao Y, Wang Z, Liu F, et al. Bio-inspired speed curve optimization and sliding mode tracking control for subway trains. IEEE T Ven Technol, 2019, 68(7):6331-6342.

[10.] Su S, Zhu Q, Liu J, et al. A data-driven iterative learning approach for optimizing the train control strategy. IEEE Trans Ind Inform, 2023, 19(7):7885-7893.

[11.] Anh TTTA, Quyến N. Optimal speed profile determination with fixed trip time in the electric train operation of the Cat Linh-Ha Dong metro line based on Pontryagin’s maximum principle. Eng Technol Appl Sci, 2020, 10(6):6488-6493.

[12.] Tan Z, Lu S, Bao K, et al. Adaptive partial train speed trajectory optimization. Energies, 2018, 11(12):3302.

[13.] Ying P, Zeng X, Song H, et al. Energy-efficient train operation with steep track and speed limits: A novel Pontryagin’s maximum principle-based approach for adjoint variable discontinuity cases. IET Intell Transp Sys, 2021, 15(9):1183-1202.

[14.] Wei S, Yan X, Cai B, et al. Multiobjective optimization for train speed trajectory in CTCS high-speed railway with hybrid evolutionary algorithm. IEEE T Intell Transp, 2015, 16(4):2215-2225.

[15.] Mo P, Yang L, Gao Z. Energy-efficient train operation strategy with speed profiles selection for an urban metro line. Transport Res Rec, 2019, 2673(4):348-360.

[16.] Liu S, Cao F, Xun J et al (2015) Energy-efficient operation of single train based on the control strategy of ATO. In: 2015 IEEE 18th International Conference on Intelligent Transportation Systems (ITSC), Gran Canaria, pp 2580–2586

[17.] Fernández PM, Font Torres JB, Sanchís IV, et al. Multi-objective ant colony optimization to obtain efficient metro speed profiles. Proc Inst Mech Eng F J Rail Rapid Transit, 2023, 237(2):232-242.

[18.] Zhang Y, Zuo T, Zhu M, et al. Research on multi-train energy saving optimization based on cooperative multi-objective particle swarm optimization algorithm. Int J Energy Res, 2021, 45(2):2644-2667.

[19.] Yin J, Chen D, Li L. Intelligent train operation algorithms for subway by expert system and reinforcement learning. IEEE Trans Intell Transp syst, 2014, 15(6):2561-2571.

[20.] Ning L, Zhou M, Hou Z, et al. Deep deterministic policy gradient for high-speed train trajectory optimization. IEEE Trans Intell Transp, 2021, 23(8):11562-11574.

[21.] Lin X, Liang Z, Shen L, et al. Reinforcement learning method for the multi-objective speed trajectory optimization of a freight train. Control Eng Pract, 2023, 138: 105605.

[22.] Meng Z, Tang T, Wei G, et al. Digital twin based comfort scenario modeling of ATO controlled train. J Phys Conf Ser, 2020, 1654(1): 012071.

[23.] Mehta P, Meyn S (2009) Q-learning and Pontryagin’s minimum principle. In: Proceedings of the 48h IEEE conference on decision and control (CDC) held jointly with 2009 28th Chinese control conference (CCC), Shanghai, pp 3598–3605

[24.] Liu Y, Halev A, Liu X (2021) Policy learning with constraints in model-free reinforcement learning: A survey. In: The 30th international joint conference on artificial intelligence (IJCAI), Montreal, pp 4508–4515

[25.] Hing MM, Harten AV, Schuur PC, et al. Reinforcement learning versus heuristics for order acceptance on a single resource. J Heuristics, 2007, 13: 167-187.

[26.] Liu M, Zhao F, Yin J, et al. Reinforcement-tracking: an effective trajectory tracking and navigation method for autonomous urban driving. IEEE T Intell Transp, 2021, 23(7):6991-7007.

[27.] Fu P, Gao S, Dong H et al (2018) Speed tracking error and rate driven event-triggered PID control design method for automatic train operation system. In: 2018 Chinese Automation Congress (CAC), Xi’an, pp 2889–2894

[28.] Pu Q, Zhu X, Zhang R, et al. Speed profile tracking by an adaptive controller for subway train based on neural network and PID algorithm. IEEE Trans Veh Technol, 2020, 69(10):10656-10667.

[29.] Wang L, Wang X, Sheng Z, et al. Model predictive controller based on online obtaining of softness factor and fusion velocity for automatic train operation. Sensors, 2020, 20(6):1719.

[30.] Bersani C, Cardano M, Lavaggi S, et al. Stochastic linear quadratic optimal control of speed and position of multiple trains on a single-track line. IEEE Tran Intell Transp syst, 2023, 24(9):9110-9120.

[31.] Chen Y, Huang D, Li Y, et al. A novel iterative learning approach for tracking control of high-speed trains subject to unknown time-varying delay. IEEE Trans Autom Sci Eng, 2020, 19(1):113-121.

[32.] Huang Z, Wang P, Zhou F, et al. Cooperative tracking control of the multiple-high-speed trains system using a tunable artificial potential function. J Adv Transport, 2022, 2022: 3639586.

[33.] Cai Q, Luo X, Gao C, et al. A machine learning-based model predictive control method for pumped storage systems. Front Energy Res, 2021, 9: 757507.

[34.] Bujarbaruah M, Zhang X, Rosolia U et al (2018) Adaptive MPC for iterative tasks. In: 2018 IEEE Conference on Decision and Control (CDC). Miami, pp 6322–6327

[35.] Liu X, Xun J, Gao S, et al. Robust self-triggered model predictive control for accurate stopping of high-speed trains. Acta Automatica Sinica, 2022, 48(1):171-181.

[36.] Yin L (2022) High speed train modeling and nonlinear speed tracking control based on disturbance observer. Dissertation, Jilin University (in Chinese)

[37.] Meng X (2021) Energy-efficient train operation control of automatic driving based on Q learning and deep Q learning. Dissertation, Beijing Jiaotong University (in Chinese)

[38.] Pathak D, Agrawal P, Efros AA et al (2017) Curiosity-driven exploration by self-supervised prediction. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, pp 2778–2787

[39.] Schaul T, Quan J, Antonoglou I et al (2015) Prioritized experience replay. In: 4th International Conference on Learning Representations, San Juan, May 2–4

[40.] Wang Z, Schaul T, Hessel M et al (2016) Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning (ICML), New York, pp 1995–2003

[41.] Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning. Proc of the AAAI conf on artificial intell (AAAI-16), Phoenix, 2016, 30(1):2094-2100.

[42.] Yang Y, Huang P, Peng Q, et al. Statistical delay distribution analysis on high-speed railway trains. J Mod Transp, 2019, 27(3):188-197.

[43.] González-Gil A, Palacin R, Batty P. Sustainable urban rail systems: Strategies and technologies for optimal management of regenerative braking energy. Energ Convers Manage, 2013, 75: 374-388.

[44.] Zhang J, Zhu A. Optimization method of automatic train operation speed curve based on genetic algorithm and particle swarm optimization. J Comput Appl, 2022, 42(2):599-605.

[45.] Luo X, Tang T, Yin J, et al. A robust MPC approach with controller tuning for close following operation of virtually coupled train set. Transport Res Part C Emerg Technol, 2023, 151: 104116.

[46.] Wang JN, Teng F, Li J, et al. Intelligent vehicle lane change trajectory control algorithm based on weight coefficient adaptive adjustment. Adv Mech Eng, 2021, 13(3):16878140211003393.

[47.] Sasfi A, Zeilinger MN, Köhler J. Robust adaptive MPC using control contraction metrics. Automatica, 2023, 155: 111169.

[48.] Liu H, Yang L, Yang H. Cooperative optimal control of the following operation of high-speed trains. IEEE Trans Intell Transp Syst, 2022, 23(10):17744-17755.

[49.] Vaquero-Serrano MA, Felez J. A decentralized robust control approach for virtually coupled train sets. Comput Aided Civ Infrastruct Eng, 2023, 38(14):1896-1915.

[50.] Ma Y (2022) Research on the integration of high-speed train operation adjustment and energy-saving control under the condition of road network. Dissertation, Lanzhou Jiaotong University (in Chinese)

[51.] Long SH (2021) Models and algorithms for the integrated optimization of train rescheduling and train control for high-speed railway. Dissertation, Beijing Jiaotong University (in Chinese)

[52.] Parise R, Dittus H, Winter J, et al. Reasoning functional requirements for virtually coupled train sets: Communication. IEEE Commun Mag, 2019, 57(9):12-17.

[53.] Guo Y, Pei X, Luo X, et al. A particle swarm optimization-based online optimization approach for virtual coupling trains with communication delay. IEEE Intell Transp Syst, 2023, 15(6):49-63.

Funding

National Natural Science Foundation of China http://dx.doi.org/10.13039/501100001809(52162050)