Adaptive traffic signal control optimization using a novel road partition and multi-channel state representation method

Abstract

This study proposes a novel adaptive traffic signal control method leveraging a Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) to optimize signal timing by integrating variable cell length and multi-channel state representation. A road partition formula consisting of the sum of logarithmic and linear functions was proposed. The state variables are a vector composed of three channels: the number of vehicles, the average speed, and space occupancy. The set of available signal phases constitutes the action space, and the selected phase is executed with a fixed green time. The reward function is formulated using the absolute values of key traffic state metrics—waiting time, speed, and fuel consumption. Each metric is normalized by a typical maximum value and assigned a weight that reflects its priority and optimization direction. The simulation results, using Sumo-TensorFlow-Python, demonstrate a cross-range transferability evaluation and show that the proposed variable cell length and multi-channel state representation method excels compared to fixed cell length in optimization performance.

Keywords

Traffic signal control / Road partition / Variable cell length / Multi-channel state representation / Deep Q network / Proximal policy optimization

References

[1] Haydari A, Yilmaz Y. Deep reinforcement learning for intelligent transportation systems: a survey. IEEE Trans Intell Transp Syst, 2022, 23(1): 11-32

[2] Genders W, Razavi S (2016) Using a deep reinforcement learning agent for traffic signal control. arXiv:1611.01142 [cs.LG]

[3] Fan L, Yang Y, Ji H, Xiong S. Research on cooperative control of traffic signals based on deep reinforcement learning. IEEE 12th Data Driven Control and Learning Systems Conference, 2023, Xiang Tan. IEEE: 1608-1612

[4] Jamil A, Ganguly K, Nower N. Adaptive traffic signal control system using composite reward architecture based deep reinforcement learning. IEEE Trans Intell Transp Syst, 2021, 14(14): 2030-2041

[5] Wang C, Hwang M. Value-based deep reinforcement learning for adaptive isolated intersection signal control. IEEE Trans Intell Transp Syst, 2018, 12(9): 1005-1010

[6] Kodama N, Harada T, Miyazaki K. Traffic signal control system using deep reinforcement learning with emphasis on reinforcing successful experiences. IEEE Access, 2022,

[7] Liu J, Qin S, Su M, Luo Y, Zhang S, Wang Y, Yang S. Traffic signal control using reinforcement learning based on the teacher-student framework. Expert Syst Appl, 2023, 228 120458

[8] Savithramma R, Sumathi R, Sudhira H. Reinforcement learning based traffic signal controller with state reduction. J Eng Res, 2023, 11(1): 100017

[9] Guo M, Wang P, Chan C, Askary S (2019) A reinforcement learning approach for intelligent traffic signal control at urban intersections. IEEE Intelligent Transportation Systems Conference (ITSC). Auckland, pp 4242–4247

[10] Li L, Lv Y, Wang F. Traffic signal timing via deep reinforcement learning. IEEE CAA J Autom Sin, 2016, 3(3): 247-254

[11] Ma D, Zhou B, Song X, Dai H. A deep reinforcement learning approach to traffic signal control with temporal traffic pattern mining. IEEE Trans Intell Transp Syst, 2022, 23(8): 11789-11890

[12] Wang P, Ni W. An enhanced dueling double deep q-network with convolutional block attention module for traffic signal optimization in deep reinforcement learning. IEEE Access, 2024, 12: 44224-44232

[13] Han G, Liu X, Wang H, Dong C, Han Y. An attention reinforcement learning-based strategy for large-scale adaptive traffic signal control system. J Transp Eng Part A Syst, 2024, 150(3): 04024001

[14] Shabestary S, Abdulhai B. Adaptive traffic signal control with deep reinforcement learning and high dimensional sensory inputs: case study and comprehensive sensitivity analyses. IEEE Trans Intell Transp Syst, 2022, 23(11): 20021-20035

[15] Shi Y, Wang Z, LaClair T, Wang C, Shao Y, Yuan J. A novel deep reinforcement learning approach to traffic signal control with connected vehicles. Appl Sci, 2023, 13(2750): 2750

[16] Luo J, Li X, Zheng Y (2020) Researches on intelligent traffic signal control based on deep reinforcement learning. 16th International Conference on Mobility, Sensing and Networking (MSN). Tokyo, pp 729–734

[17] Thorpe T, Anderson C (1996) Traffic light control using SARSA with three state representations. IBM Corporation

[18] Abdulhai B, Pringle R, Karakoulas G. Reinforcement learning for true adaptive traffic signal control. J Transp Eng Part A Syst, 2003, 129(3): 278-285

[19] Wen K, Qu S, Zhang Y. A stochastic adaptive control model for isolated intersections, 2007, Sanya. IEEE Int. Conf. on Robotics and Biomimetics: 2256-2260

[20] Lu S, Liu X, Dai S. Q-Learning for Adaptive Traffic Signal Control Based on Delay Minimization Strategy, 2008, Sensing and Control, Sanya. IEEE Int. Conf. on Networking: 687-691

[21] Touhbi S, Babram M, Nguyen-Huu T, Marilleau N, Hbid M, Cambier C, Stinckwich S. Adaptive traffic signal control: exploring reward definition for reinforcement learning. Procedia Comput Sci, 2017, 109: 513-520

[22] Bouktif S, Cheniki A, Ouni A, El-Sayed H. Deep reinforcement learning for traffic signal control with consistent state and reward design approach. Knowl Based Syst, 2023, 267 110440

[23] Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Communications of the ACM vo, 2017, 60(6): 84-90

RIGHTS & PERMISSIONS

The Author(s)