Fairness-aware multi-agent reinforcement learning and visual perception for adaptive traffic signal control

Abstract

The majority of multi-agent reinforcement learning (MARL) methods for solving adaptive traffic signal control (ATSC) problems are dedicated to maximizing the throughput while ignoring fairness, resulting in a bad situation where some vehicles keep waiting. For this reason, this paper models the ATSC problem as a partially observable Markov game (POMG), in which a value function that combines throughput and fairness is elaborated. On this basis, we propose a new cooperative MARL method of fairness-aware multi-agent proximity policy optimization (FA-MAPPO). In addition, the FA-MAPPO uses graph attention neural networks to efficiently extract state representations from traffic data acquired through visual perception in multi-intersection scenarios. Experimental results in Jinan and synthetic scenarios confirm that the FA-MAPPO improves fairness while guaranteeing passage efficiency compared to the state-of-the-art (SOTA) methods.

References

[1] WEI H, ZHENG G, GAYAH V, et al. A survey on traffic signal control methods[EB/OL]. (2019-04-17) [2023-08-13]. https://arxiv.org/abs/1904.08117.

[2] Roess R P. Traffic engineering[M], 1990, Washington: National Academy of Sciences

[3] Varaiya P. The max-pressure controller for arbitrary networks of signalized intersections[M]. UKKSURI S, OZBAY K, UKKUSURI S V. Advances in dynamic network modeling in complex transportation systems, 2013, Berlin, Heidelberg: Springer 27-66

[4] Zhang J, Liu H, Ying X, et al. Coordinated underwater dark channel prior for artifact removal of challenging image enhancement[J]. Optoelectronics letters, 2023, 19(7): 416-424

[5] Wei H, Zheng G, Gayah V, et al. Recent advances in reinforcement learning for traffic signal control: a survey of models and evaluation[J]. ACM SIGKDD explorations newsletter, 2021, 22(2): 12-18

[6] Guo Q, Li L, Ban X J. Urban traffic signal control with connected and automated vehicles: a survey[J]. Transportation research part C: emerging technologies, 2019, 101: 313-334

[7] CHEN R, FANG F, SADEH N. The real deal: a review of challenges and opportunities in moving reinforcement learning-based traffic signal control systems towards reality[EB/OL]. (2022-06-23) [2023-08-13]. https://arxiv.org/abs/2206.11996.

[8] Zhang C, Tian Y, Zhang Z, et al. Neighborhood cooperative multiagent reinforcement learning for adaptive traffic signal control in epidemic regions[J]. IEEE transactions on intelligent transportation systems, 2022, 23(12): 25157-25168

[9] Van Der Pol E, Oliehoek F A. Coordinated deep reinforcement learners for traffic light control[J]. Proceedings of learning, inference and control of multi-agent systems, 2016, 8: 21-38

[10] Chu T, Wang J, Codecà L, et al. Multi-agent deep reinforcement learning for large-scale traffic signal control[J]. IEEE transactions on intelligent transportation systems, 2019, 21(3): 1086-1095

[11] Ma J, Wu F. Feudal multi-agent deep reinforcement learning for traffic signal control[C]. Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, May 9–13, 2020, Auckland, New Zealand, 2020, South Carolina: AAMAS 816-824

[12] Xu B, Wang Y, Wang Z, et al. Hierarchically and cooperatively learning traffic signal control[C]. Proceedings of the 35th AAAI Conference on Artificial Intelligence, February 2–9, 2021, Vancouver, Canada, 2021, Washington: AAAI 669-677 35 (1)

[13] Wei H, Xu N, Zhang H, et al. Colight: learning network-level cooperation for traffic signal control[C]. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, November 3–7, 2019, Beijing, China, 2019, Maryland: CIKM 1913-1922

[14] Wang Y, Xu T, Niu X, et al. STMARL: a spatio-temporal multi-agent reinforcement learning approach for cooperative traffic light control[J]. IEEE transactions on mobile computing, 2020, 21(6): 2228-2242

[15] Huang H, Hu Z, Lu Z, et al. Network-scale traffic signal control via multiagent reinforcement learning with deep spatiotemporal attentive network[J]. IEEE transactions on cybernetics, 2021, 53(1): 262-274

[16] Wang M, Wu L, Li M, et al. Meta-learning based spatial-temporal graph attention network for traffic signal control[J]. Knowledge-based systems, 2022, 250: 109166

[17] Yan L, Zhu L, Song K, et al. Graph cooperation deep reinforcement learning for ecological urban traffic signal control[J]. Applied intelligence, 2023, 53: 6248-6265

[18] Chen C, Wei H, Xu N, et al. Toward a thousand lights: decentralized deep reinforcement learning for large-scale traffic signal control[C]. Proceedings of the 34th AAAI Conference on Artificial Intelligence, February 7–12, 2020, New York, USA, 2020, Washington: AAAI 3414-3421 34 (4)

[19] Wei H, Chen C, Zheng G, et al. Presslight: learning max pressure control to coordinate traffic signals in arterial network[C]. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, August 4–8, 2019, Anchorage, AK, USA, 2019, New York: ACM 1290-1298

[20] Wang X, Ke L, Qiao Z, et al. Large-scale traffic signal control using a novel multiagent reinforcement learning[J]. IEEE transactions on cybernetics, 2020, 51(1): 174-187

[21] Raeis M, Leon-Garcia A. A deep reinforcement learning approach for fair traffic signal control[C]. 2021 IEEE International Intelligent Transportation Systems Conference, September 19–22, 2021, Indianapolis, IN, USA, 2021, New York: IEEE 2512-2518

[22] Chen J, Zhang Z, Feng J, et al. FIT: fairness-aware intelligent traffic signal control with deep reinforcement learning[C]. 2021 IEEE 23rd International Conference on High Performance Computing & Communications; 7th International Conference on Data Science & Systems; 19th International Conference on Smart City; 7th International Conference on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), December 20–22, 2021, Haikou, Hainan, China, 2021, New York: IEEE 846-852

[23] Yu C, Velu A, Vinitsky E, et al. The surprising effectiveness of PPO in cooperative multi-agent games[J]. Advances in neural information processing systems, 2022, 35: 24611-24624

[24] Webster F V. Traffic signal settings[R], 1958 39

[25] Varaiya P. Max pressure control of a network of signalized intersections[J]. Transportation research part C: emerging technologies, 2013, 36: 177-195

[26] JIANG J, DUN C, HUANG T, et al. Graph convolutional reinforcement learning[EB/OL]. (2018-10-22) [2023-08-13]. https://arxiv.org/abs/1810.09202.