Evaluating end-to-end autonomous driving architectures: a proximal policy optimization approach in simulated environments

Abstract

Autonomous driving systems (ADS) are at the forefront of technological innovation, promising enhanced safety, efficiency, and convenience in transportation. This study investigates the potential of end-to-end reinforcement learning (RL) architectures for ADS, specifically focusing on a Go-To-Point task involving lane-keeping and navigation through basic urban environments. The study uses the Proximal Policy Optimization (PPO) algorithm within the CARLA simulation environment. Traditional modular systems, which separate driving tasks into perception, decision-making, and control, provide interpretability and reliability in controlled scenarios but struggle with adaptability to dynamic, real-world conditions. In contrast, end-to-end systems offer a more integrated approach, potentially enhancing flexibility and decision-making cohesion.

This research introduces CARLA-GymDrive, a novel framework integrating the CARLA simulator with the Gymnasium API, enabling seamless RL experimentation with both discrete and continuous action spaces. Through a two-phase training regimen, the study evaluates the efficacy of PPO in an end-to-end ADS focused on basic tasks like lane-keeping and waypoint navigation. A comparative analysis with modular architectures is also provided. The findings highlight the strengths of PPO in managing continuous control tasks, achieving smoother and more adaptable driving behaviors than value-based algorithms like Deep Q-Networks. However, challenges remain in generalization and computational demands, with end-to-end systems requiring extensive training time.

While the study underscores the potential of end-to-end architectures, it also identifies limitations in scalability and real-world applicability, suggesting that modular systems may currently be more feasible for practical ADS deployment. Nonetheless, the CARLA-GymDrive framework and the insights gained from PPO-based ADS contribute significantly to the field, laying a foundation for future advancements in AD.

Keywords

Autonomous Driving Systems (ADS) / End-to-End Architecture / Software System Architecture / Proximal Policy Optimization (PPO) / Real-Time Embedded Systems / Simulation Framework

References

[1] KutelaB., SalumJ.H., SeifS.R., DasS., KidandoE.. Navigating the blame game: investigating automated vehicle fault in collisions under mixed traffic conditions. Robot. Auton. Syst., 2024, 182. 104831

[2] Morgado, GonalvesC., PomboN.. Enhancing autonomous vehicles: an experimental analysis of path planning and decision-making processes through simulink-based architecture. 2023 IEEE 17th International Conference on Application of Information and Communication Technologies (AICT), 202319.

[3] ShaoH., WangL., ChenR., WaslanderS.L., LiH., LiuY.. Reasonnet: end-to-end driving with temporal and global reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20231372313733

[4] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, H. Li, End-to-end autonomous driving: challenges and frontiers (2024). arXiv:2306.16927

[5] P.N. Morgado, CARLA-GymDrive: autonomous driving episode generation for the Carla simulator in a gym environment (2024). Contact: [e-mail]. https://github.com/angelomorgado/CARLA-GymDrive

[6] M. Towers, A. Kwiatkowski, J. Terry, J.U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, et al., Gymnasium: a standard interface for reinforcement learning environments (2024). arXiv preprint. arXiv:2407.17032

[7] AtakishiyevS., SalamehM., YaoH., GoebelR.. Explainable artificial intelligence for autonomous driving: a comprehensive overview and field guide for future research directions. IEEE Access, 2024, 12: 101603-101625.

[8] KabirM.M., JimJ.R., IstenesZ.. Terrain detection and segmentation for autonomous vehicle navigation: a state-of-the-art systematic review. Inf. Fusion, 2025, 113. 102644

[9] WangJ., HuangH., LiK., LiJ.. Towards the unified principles for level 5 autonomous vehicles. Engineering, 2021, 7(9): 1313-1325.

[10] ChenL., WuP., ChittaK., JaegerB., GeigerA., LiH.. End-to-end autonomous driving: challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024120.

[11] BicerY., AlizadehA., UreN.K., ErdoganA., KizilirmakO.. Sample efficient interactive end-to-end deep learning for self-driving cars with selective multi-class safe dataset aggregation. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 201926292634

[12] VaishR., DwivediU.D., TewariS., TripathiS.M.. Machine learning applications in power system fault diagnosis: research advancements and perspectives. Eng. Appl. Artif. Intell., 2021, 106. 104504

[13] J. Hossain, Autonomous Driving with Deep Reinforcement Learning in CARLA Simulation (2023)

[14] WuY., LiaoS., LiuX., LiZ., LuR.. Deep reinforcement learning on autonomous driving policy with auxiliary critic network. IEEE Trans. Neural Netw. Learn. Syst., 2023, 34(7): 3680-3690.

[15] Pérez-Gil, R. Barea, E. López-Guillén, L. Bergasa, C. Gómez-Huélamo, R. Gutiérrez, A. Díaz, Deep reinforcement learning based control for autonomous vehicles in carla. Multimed. Tools Appl. 81 (2022). https://doi.org/10.1007/s11042-021-11437-3

[16] QinY., ZhangC., ChenC., XieS., LewisF.L.. Control policy learning design for vehicle urban positioning via beidou navigation. J. Syst. Sci. Complex., 2024, 37(1): 114-135.

[17] DengQ., LiuG., LiR., HuQ., ZhaoY., LiR.LuoB., ChengL., WuZ.-G., LiH., LiC.. End-to-end urban autonomous navigation with decision hindsight. Neural Information Processing, 2024, Singapore. Springer. 6783

[18] MoniR., Gyires-TthB.. Self-supervised domain transfer for reinforcement learning-based autonomous driving agent. Expert Syst. Appl., 2025, 284. 127809

[19] WangZ., LiP., WangZ., LiZ.. Highway autonomous vehicle decision-making method based on prior knowledge and improved experience replay reinforcement learning algorithm. Expert Syst. Appl., 2025, 284. 127927

[20] T. Li, J. Ruan, K. Zhang, The investigation of reinforcement learning-based end-to-end decision-making algorithms for autonomous driving on the road with consecutive sharp turns. Green Energy Intell. Transp., 100288 (2025). https://doi.org/10.1016/j.geits.2025.100288

[21] ZhaoZ.. Control of autonomous vehicles based on ppo algorithm. Proceedings of the 7th International Conference on Information Technologies and Electrical Engineering. ICITEE ’24, 2025, New York. Association for Computing Machinery. 386392.

[22] WangZ., YanH., WeiC., WangJ., BoS., XiaoM.. Research on autonomous driving decision-making strategies based deep reinforcement learning. Proceedings of the 2024 4th International Conference on Internet of Things and Machine Learning. IoTML ’24, 2024, New York. Association for Computing Machinery. 211215.

[23] DosovitskiyA., RosG., CodevillaF., LopezA., KoltunV.. CARLA: an open urban driving simulator. Proceedings of the 1st Annual Conference on Robot Learning, 2017116

[24] RaffinA., HillA., GleaveA., KanervistoA., ErnestusM., DormannN.. Stable-baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res., 2021, 22(268): 1-8

[25] KoprinkovaP., PetrovaM.. Data-scaling problems in neural-network training. Eng. Appl. Artif. Intell., 1999, 12(3): 281-296.

Funding

Agência Regional para o Desenvolvimento da Investigação, Tecnologia e Inovação(UIDB/50008/2020)

RIGHTS & PERMISSIONS

The Author(s)