Reinforcement learning-based control algorithm for connected and automated vehicles on a two-lane highway section with a moving bottleneck

Abstract

To address the problem of control optimization for connected and automated vehicles (CAVs) on a highway section with two lanes in the same direction (hereafter referred to as a two-lane highway section) under a moving bottleneck scenario and to improve their traffic efficiency, this study proposes a control model based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The model fully incorporates the state information of the ego vehicle and surrounding vehicles as input to generate continuous control actions as output, and utilizes a multi-objective reward function to guide the CAV’s behavior comprehensively. To further enhance training efficiency and driving safety, a collision avoidance strategy based on the inverse Time-to-Collision criterion is integrated into the framework, which effectively mitigates the risk of collision caused by inappropriate acceleration decisions. In the simulation experiments, the proposed method is evaluated under various CAV entering probabilities and compared against a conventional heuristic rules-based algorithm (HRA). The results demonstrate that the TD3-based control strategy outperforms the HRA across multiple performance metrics. In particular, under high-density conditions, the overtaking throughput is improved by approximately 40%, significantly enhancing traffic efficiency in the presence of moving bottlenecks.

Keywords

Reinforcement learning / Moving bottleneck / Control of connected and automated vehicles / Two-lane highway section

References

[1] Ding J, Li L, Peng H, et al. . A rule-based cooperative merging strategy for connected and automated vehicles. IEEE Trans Intell Transp Syst, 2020, 21(8): 3436-3446.

[2] Shladover SE, Su D, Lu XY. Impacts of cooperative adaptive cruise control on freeway traffic flow. Transp Res Rec, 2012, 2324(1): 63-70.

[3] Milanes V, Shladover SE, Spring J, et al. . Cooperative adaptive cruise control in real traffic situations. IEEE Trans Intell Transp Syst, 2014, 15(1): 296-305.

[4] Zheng Z, Ahn S, Monsere CM. Impact of traffic oscillations on freeway crash occurrences. Accid Anal Prev, 2010, 42(2): 626-636.

[5] Memarian A, Rosenberger JM, Mattingly SP, et al. . An optimization-based traffic diversion model during construction closures. Computer-Aided Civil and Infrastructure Engineering, 2019, 34(12): 1087-1099.

[6] Han Y, Ramezani M, Hegyi A, et al. . Hierarchical ramp metering in freeways: an aggregated modeling and control approach. Transp Res Part C Emerg Technol, 2020, 110: 1-19.

[7] Jing D, Yao E, Chen R. Moving characteristics analysis of mixed traffic flow of CAVs and HVs around accident zones. Physica A Stat Mech Appl, 2023, 626129085

[8] Du Y, Makridis MA, Tampère CMJ, et al. . Adaptive control with moving actuators at motorway bottlenecks with connected and automated vehicles. Transportation Research Part C: Emerging Technologies, 2023, 156104319

[9] Li Y, Pan B, Chen Z, et al. . Developing a dynamic speed control system for mixed traffic flow to reduce collision risks near freeway bottlenecks. IEEE Trans Intell Transp Syst, 2023, 24(11): 1-22.

[10] Wu Y, Tan H, Qin L, et al. . Differential variable speed limits control for freeway recurrent bottlenecks via deep actor-critic algorithm. Transp Res Part C Emerg Technol, 2020, 117102649

[11] Guo Q, Angah O, Liu Z, et al. . Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors. Transp Res Part C Emerg Technol, 2021, 124102980

[12] Wu C, Kreidieh AR, Parvate K, et al. . Flow: a modular learning framework for mixed autonomy traffic. IEEE Trans Robot, 2022, 38(2): 1270-1286.

[13] Elmorshedy L, Smirnov I, Abdulhai B. Freeway congestion management on multiple consecutive bottlenecks with RL-based headway control of autonomous vehicles. IET Intell Transp Syst, 2024, 18(6): 1137-1163.

[14] He X, Huang W, Lv C. Trustworthy autonomous driving via defense-aware robust reinforcement learning against worst-case observational perturbations. Transp Res C Emerg Technol, 2024, 163104632

[15] Kreidieh AR, Farid Y, Oguchi K. Lateral flow control of connected vehicles through deep reinforcement learning. 34th IEEE Intelligent Vehicles Symposium (IV), 2023, USA. Anchorage1-7

[16] Yang J, Wang P, Ju Y. Variable speed limit intelligent decision-making control strategy based on deep reinforcement learning under emergencies. Sustainability, 2024, 16(3): 965.

[17] Lu W, Yi Z, Gu Y, et al. . TD3LVSL: a lane-level variable speed limit approach based on twin delayed deep deterministic policy gradient in a connected automated vehicle environment. Transportation Research Part C: Emerging Technologies, 2023, 153104221

[18] Li D, Lasenby J. Imagination-augmented reinforcement learning framework for variable speed limit control. IEEE Trans Intell Transp Syst, 2024, 25(2): 1-10.

[19] Pan T, Guo R, Lam WHK, et al. . Integrated optimal control strategies for freeway traffic mixed with connected automated vehicles: a model-based reinforcement learning approach. Transp Res C Emerg Technol, 2021, 123102987

[20] Juran I, Prashker JN, Bekhor S, et al. . A dynamic traffic assignment model for the assessment of moving bottlenecks. Transportation Research Part C: Emerging Technologies, 2009, 17(3): 240-258.

[21] Gazis DC, Herman R. The moving and “phantom” bottlenecks. Transp Sci, 1992, 26(3): 223-229.

[22] Newell GF. A moving bottleneck. Transp Res B Methodol, 1998, 32(8): 531-537.

[23] Simoni MD, Claudel CG. A fast simulation algorithm for multiple moving bottlenecks and applications in urban freight traffic management. Transp Res B Methodol, 2017, 104: 238-255.

[24] Ou H, Tang T. Impacts of moving bottlenecks on traffic flow. Physica A Stat Mech Appl, 2018, 500: 131-138.

[25] Hu X, Lin C, Hao X, et al. . Influence of tidal lane on traffic breakdown and spatiotemporal congested patterns at moving bottleneck in the framework of Kerner’s three-phase traffic theory. Physica A Stat Mech Appl, 2021, 584126335

[26] Wang X, Zeng J, Qian Y, et al. . Heterogeneous traffic flow of expressway with level 2 autonomous vehicles considering moving bottlenecks. Physica A Stat Mech Appl, 2024, 650129991

[27] Wu Y, Lin Y, Hu R. Modeling and simulation of traffic congestion for mixed traffic flow with connected automated vehicles: a cell transmission model approach. J Adv Transp, 2022, 2022(Pt.8): 1.1-1.20

[28] Piacentini G, Ferrara A, Papamichail I et al (2019) Highway traffic control with moving bottlenecks of connected and automated vehicles for travel time reduction. In: 58th IEEE Conference on Decision and Control (CDC), Nice, France, pp 3140–3145

[29] Chen R, Zhang T, Levin MW. Effects of variable speed limit on energy consumption with autonomous vehicles on urban roads using modified cell-transmission model. Journal of Transportation Engineering, Part A: Systems, 2020, 146(74020049

[30] Liu H, Jiang R. Efficient control of connected and automated vehicles on a two-lane highway with a moving bottleneck. Chin Phys B, 2023, 325): 54501-54601.

[31] Ni H, Yu G, Chen P, et al. . An integrated framework of lateral and longitudinal behavior decision-making for autonomous driving using reinforcement learning. IEEE Trans Veh Technol, 2024, 73(7): 9706-9720.

[32] Wang S, Wang Z, Jiang R, et al. . A multi-agent reinforcement learning-based longitudinal and lateral control of CAVs to improve traffic efficiency in a mandatory lane change scenario. Transportation Research Part C: Emerging Technologies, 2024, 158104445

[33] Luo Y, Xiang Y, Cao K, et al. . A dynamic automated lane change maneuver based on vehicle-to-vehicle communication. Transportation Research Part C: Emerging Technologies, 2016, 62: 87-102.

[34] Jia D, Ngoduy D. Platoon based cooperative driving model with consideration of realistic inter-vehicle communication. Transp Res C Emerg Technol, 2016, 68: 245-264.

[35] Wang M. Infrastructure assisted adaptive driving to stabilise heterogeneous vehicle strings. Transp Res C Emerg Technol, 2018, 91: 276-295.

[36] Hu X, Sun J. Trajectory optimization of connected and autonomous vehicles at a multilane freeway merging area. Transp Res C Emerg Technol, 2019, 101: 111-125.

[37] Wang Y, Wei L, Chen P. Trajectory reconstruction for freeway traffic mixed with human-driven vehicles and connected and automated vehicles. Transp Res C Emerg Technol, 2020, 111: 135-155.

[38] Liu H, Jiang R. Improving comfort level in traffic flow of CACC vehicles at lane drop on two-lane highways. Physica A Stat Mech Appl, 2021, 575126055

Funding

State Key Lab of Intelligent Transportation System(2025-A001)

National Natural Science Foundation of China(W2411064)

China National Postdoctoral Program for Innovative Talents(BX20240033)

Beijing Natural Science Foundation(9242013)

RIGHTS & PERMISSIONS

The Author(s)