Collision-free parking recommendation based on multi-agent reinforcement learning in vehicular crowdsensing

Abstract

The recent proliferation of Fifth-Generation (5G) networks and Sixth-Generation (6G) networks has given rise to Vehicular Crowd Sensing (VCS) systems which solve parking collisions by effectively incentivizing vehicle participation. However, instead of being an isolated module, the incentive mechanism usually interacts with other modules. Based on this, we capture this synergy and propose a Collision-free Parking Recommendation (CPR), a novel VCS system framework that integrates an incentive mechanism, a non-cooperative VCS game, and a multi-agent reinforcement learning algorithm, to derive an optimal parking strategy in real time. Specifically, we utilize an LSTM method to predict parking areas roughly for recommendations accurately. Its incentive mechanism is designed to motivate vehicle participation by considering dynamically priced parking tasks and social network effects. In order to cope with stochastic parking collisions, its non-cooperative VCS game further analyzes the uncertain interactions between vehicles in parking decision-making. Then its multi-agent reinforcement learning algorithm models the VCS campaign as a multi-agent Markov decision process that not only derives the optimal collision-free parking strategy for each vehicle independently, but also proves that the optimal parking strategy for each vehicle is Pareto-optimal. Finally, numerical results demonstrate that CPR can accomplish parking tasks at a 99.7% accuracy compared with other baselines, efficiently recommending parking spaces.

Keywords

Incentive mechanism / Non-cooperative VCS game / Multi-agent reinforcement learning / Collision-free parking strategy / Vehicular crowdsensing

References

[1] M. Khalid, K. Wang, N. Aslam, Y. Cao, N. Ahmad, M.K. Khan, From smart parking towards autonomous valet parking: a survey, challenges and future works, Int. J. Comput. Netw. Appl. 175 (2021) 102935.

[2] T. do Vale Saraiva, C.A.V. Campos, R. dos Reis Fontes, C.E. Rothenberg, S. Sorour, S. Valaee, An application-driven framework for intelligent transportation systems using 5g network slicing, IEEE Trans. Intell. Transp. Syst. 22 (8) (2021) 5247-5260.

[3] B. Sliwa, R. Adam, C. Wietfeld, Client-based intelligence for resource efficient ve-hicular big data transfer in future 6g networks, IEEE Trans. Veh. Technol. 70 (6)(2021) 5332-5346.

[4] D. Asprone, S. Di Martino, P. Festa, L.L.L. Starace, Vehicular crowd-sensing: a para-metric routing algorithm to increase spatio-temporal road network coverage, Int. J. Geogr. Inf. Sci. 35 (9) (2021) 1876-1904.

[5] C.H. Liu, Z. Dai, H. Yang, J. Tang, Multi-task-oriented vehicular crowdsensing: a deep learning approach, in: IEEE INFOCOM 2020-IEEE Conference on Computer Communications, IEEE, 2020, pp. 1123-1132.

[6] L. Liu, L. Wang, Z. Lu, Y. Liu, W. Jing, X. Wen, Cost-and-quality aware data col-lection for edge-assisted vehicular crowdsensing, IEEE Trans. Veh. Technol. 71 (5)(2022) 5371-5386.

[7] Y. Liu, L. Kong, G. Chen, Data-oriented mobile crowdsensing: a comprehensive sur-vey, IEEE Commun. Surv. Tutor. 21 (3) (2019) 2849-2885.

[8] X. Zhu, Y. Luo, A. Liu, W. Tang, M.Z.A. Bhuiyan, A deep learning-based mobile crowdsensing scheme by predicting vehicle mobility, IEEE Trans. Intell. Transp. Syst. 22 (7) (2020) 4648-4659.

[9] A.O. Kotb, Y.-C. Shen, X. Zhu, Y. Huang, Iparker—a new smart car-parking system based on dynamic resource allocation and pricing, IEEE Trans. Intell. Transp. Syst. 17 (9) (2016) 2637-2647.

[10] F. Shi, D. Wu, D.I. Arkhipov, Q. Liu, A.C. Regan, J.A. McCann, Parkcrowd: reliable crowdsensing for aggregation and dissemination of parking space information, IEEE Trans. Intell. Transp. Syst. 20 (11) (2018) 4032-4044.

[11] H. Jin, L. Su, K. Nahrstedt, Theseus: incentivizing truth discovery in mobile crowd sensing systems,in: Proceedings of the 18th ACM International Symposium on Mo-bile Ad Hoc Networking and Computing, 2017, pp. 1-10.

[12] H. Jin, L. Su, D. Chen, K. Nahrstedt, J. Xu,Quality of information aware incentive mechanisms for mobile crowd sensing systems, in:Proceedings of the 16th ACM International Symposium on Mobile Ad Hoc Networking and Computing, 2015, pp. 167-176.

[13] Y. Zhao, C.H. Liu, Social-aware incentive mechanism for vehicular crowdsensing by deep reinforcement learning, IEEE Trans. Intell. Transp. Syst. 22 (4) (2020) 2314-2325.

[14] C. Lin, S. Wei, J. Deng, M.S. Obaidat, H. Song, L. Wang, G. Wu, Gtccs: a game theoretical collaborative charging scheduling for on-demand charging architecture, IEEE Trans. Veh. Technol. 67 (12) (2018) 12124-12136.

[15] A.H. Salem, I.W. Damaj, H.T. Mouftah, Vehicle as a computational resource: opti-mizing quality of experience for connected vehicles in a smart city, Veh. Commun. 33 (2022) 100432.

[16] H. Jin, L. Su, H. Xiao, K. Nahrstedt, Inception: incentivizing privacy-preserving data aggregation for mobile crowd sensing systems,in: Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing, 2016, pp. 341-350.

[17] S. Smaldone, L. Han, P. Shankar, L. Iftode, Roadspeak: enabling voice chat on road-ways using vehicular social networks,in: Proceedings of the 1st Workshop on Social Network Systems, 2008, pp. 43-48.

[18] K. Tan, D. Bremner, J. Le Kernec, L. Zhang, M. Imran, Machine learning in vehicular networking: an overview, Digit. Commun. Netw. 8(1) (2022) 18-24.

[19] H. Park, Y. Lim, Reinforcement learning for energy optimization with 5g communi-cations in vehicular social networks, Sensors 20 (8) (2020) 2361.

[20] Y. Guan, Y. Ren, S.E. Li, Q. Sun, L. Luo, K. Li, Centralized cooperation for con-nected and automated vehicles at intersections by proximal policy optimization, IEEE Trans. Veh. Technol. 69 (11) (2020) 12597-12608.

[21] W. Tan, L. Zhao, B. Li, L. Xu, Y. Yang, Multiple cooperative task allocation in group-oriented social mobile crowdsensing, IEEE Trans. Serv. Comput. 15 (6) (2021) 3387-3401.

[22] Y. Kang, S. Liu, H. Zhang, Z. Han, S. Osher, H.V. Poor, Task selection and collision-free route planning for mobile crowdsensing using multi-population mean-field games, IEEE Trans. Green Commun. Netw. 5(4) (2021) 1947-1960.

[23] V. Paidi, H. Fleyeh, J. Håkansson, R.G. Nyberg, Smart parking sensors, technologies and applications for open parking lots: a review, IET Intell. Transp. Syst. 12 (8)(2018) 735-741.

[24] K. Banti, M. Louta, G. Karetsos, Parkcar: a smart roadside parking application ex-ploiting the mobile crowdsensing paradigm, in: 2017 8th International Conference on Information, Intelligence, Systems & Applications (IISA), IEEE, 2017, pp. 1-6.

[25] F. Bock, S. Di Martino, A. Origlia, Smart parking: using a crowd of taxis to sense on-street parking space availability, IEEE Trans. Intell. Transp. Syst. 21 (2) (2019) 496-508.

[26] D. Yang, G. Xue, X. Fang, J. Tang, Crowdsourcing to smartphones: incentive mech-anism design for mobile phone sensing,in: Proceedings of the 18th Annual Interna-tional Conference on Mobile Computing and Networking, 2012, pp. 173-184.

[27] Y. Zhan, C.H. Liu, Y. Zhao, J. Zhang, J. Tang, Free market of multi-leader multi-follower mobile crowdsensing: an incentive mechanism design by deep reinforce-ment learning, IEEE Trans. Mob. Comput. 19 (10) (2019) 2316-2329.

[28] Z. Zhou, H. Liao, B. Gu, K.M.S. Huq, S. Mumtaz, J. Rodriguez, Robust mobile crowd sensing: when deep learning meets edge computing, IEEE Netw. 32 (4) (2018) 54-60.

[29] N. Deo, M.M. Trivedi,Convolutional social pooling for vehicle trajectory prediction, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 1468-1476.

[30] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, S. Savarese, Social lstm: human trajectory prediction in crowded spaces, in: Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition, 2016, pp. 961-971.

[31] N. Lee, W. Choi, P. Vernaza, C.B. Choy, P.H. Torr, M. Chandraker, Desire: distant future prediction in dynamic scenes with interacting agents,in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 336-345.

[32] M. Zhang, J. Chen, L. Yang, J. Zhang, Dynamic pricing for privacy-preserving mo-bile crowdsensing: a reinforcement learning approach, IEEE Netw. 33 (2) (2019) 160-165.

[33] R. Lowe, Y.I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, I. Mordatch, Multi-agent actor-critic for mixed cooperative-competitive environments, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, ACM, 2017, pp. 6382-6393.

[34] C.H. Liu, Z. Chen, J. Tang, J. Xu, C. Piao, Energy-efficient uav control for effective and fair communication coverage: a deep reinforcement learning approach, IEEE J. Sel. Areas Commun. 36 (9) (2018) 2059-2070.

[35] C.H. Liu, X. Ma, X. Gao, J. Tang, Distributed energy-efficient multi-uav navigation for long-term communication coverage by deep reinforcement learning, IEEE Trans. Mob. Comput. 19 (6) (2019) 1274-1285.

[36] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy opti-mization algorithms, arXiv preprint, arXiv :1707.06347.