End-to-end replay-based trajectory planning for autonomous vehicles under multi-weather scenarios

Abstract

Autonomous driving systems face challenges from perception degradation and kinematic coupling in adverse weather. This paper introduces an end-to-end trajectory prediction framework integrating multi-weather continual learning with kinematic constraint optimization. Traditional weather-specific models suffer from fragmented experience and catastrophic forgetting, impacting control in low-visibility, high-curvature scenarios. We propose a multi-weather adaptive replay mechanism (MWARM) with entropy-weighted sampling for cross-weather knowledge transfer, paired with a bird’s eye view (BEV)-based perception-planning architecture using multi-objective model predictive control (MO-MPC) to adjust weights based on real-time curvature and weather data. Evaluated in CARLA with a multi-weather dataset, the framework provides a robust solution for complex conditions.

Keywords

Autonomous driving vehicles / Dynamic weather conditions / Continual learning / Memory replay mechanism / Path planning

CRediT authorship contribution statement

Jinjun Dun: Writing – original draft. Yuenan Zhao: Writing – review & editing. Xiaoyu Xu: Methodology. Zhenguo Chen: Data curation. Hui Xie: Funding acquisition.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

Supplementary material related to this article can be found online at https://doi.org/10.1016/j.birob.2026.100275.

References

[1] Ma Jin, Mingcheng Qu, Qingyang Gao, Zhuo Huang, Tonghua Su, Zhongchao Liang, Advanced trajectory planning and control for autonomous vehicles with quintic polynomials, Sensors, 1424-8220, 24 (24), (2024), 10.3390/s24247928,URL https://www.mdpi.com/1424-8220/24/24/7928.

[2] Almalioglu Yasin, Turan Mehmet, Trigoni Niki, Markham Andrew, Deep learning-based robust positioning for all-weather autonomous driving, Nat. Mach. Intell., (9) (2022), p. 4.

[3] S. Fortunato, Community detection in graphs, Phys. Rep. 486 (2010) 75-174.

[4] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, Andreas Geiger, TransFuser: Imitation with transformer-based sensor fusion for autonomous driving, IEEE Trans. Pattern Anal. Mach. Intell. 45 (11) (2023) 12878-12895, https://doi.org/10.1109/TPAMI.2022.3200245.

[5] Aditya Prakash, Kashyap Chitta, Andreas Geiger, Multi-Modal Fusion Transformer for End-to-End Autonomous Driving, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2021, pp. 7077-7087.

[6] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, Yu Qiao, Trajectory-guided control prediction for end-to-end autonomous driving: a simple yet strong baseline, Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, 9781713871088, Curran Associates Inc., Red Hook, NY, USA, (2022).

[7] Prashant W. Patil, Sunil Gupta, Santu Rana, Svetha Venkatesh, Subrahmanyam Murala, Multi-weather image restoration via domain translation, 2023 IEEE/CVF International Conference on Computer Vision, ICCV, 2023, pp. 21639-21648, https://doi.org/10.1109/ICCV51070.2023.01983.

[8] Z. Zhu, S. Li, Y. Zhang, Deep Q-learning based multi-constraint gliding trajectory planning for autonomous vehicles, IEEE Trans. Intell. Transp. Syst. 22 (11) (2021) 6885-6895, https://doi.org/10.1109/TITS.2021.3071234.

[9] Binyu Wang, Zhe Liu, Qingbiao Li, Amanda Prorok, Mobile robot path planning in dynamic environments through globally guided reinforcement learning, IEEE Robot. Autom. Lett. 5 (4) (2020) 6932-6939, https://doi.org/10.1109/LRA.2020.3026638.

[10] J. Li, H. Zhang, L. Wang, Continuous advantage learning for minimum-time trajectory planning of autonomous vehicles, IEEE Trans. Control Syst. Technol. 32 (2) (2024) 567-578, https://doi.org/10.1109/TCST.2023.3145678.

[11] Jimeng Shi, Azam Shirali, Bowen Jin, Sizhe Zhou, Wei Hu, Rahuul Rangaraj, Shaowen Wang, Jiawei Han, Zhaonan Wang, Upmanu Lall, Yanzhao Wu, Leonardo Bobadilla, Giri Narasimhan, Deep learning and foundation models for weather prediction: A survey, (2025),URL https://arxiv.org/abs/2501.06907.

[12] G. Dong, X. Wei, J. Bao, G. Brochard, Z. Lin, W. Tang, Deep learning based surrogate models for first-principles global simulations of fusion plasmas, Nucl. Fusion, 1741-4326 61 (12) (2021) 126061, https://doi.org/10.1088/1741-4326/ac32f1.

[13] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, (2018),URL http://proceedings.mlr.press/v80/haarnoja18b.html.

[14] Jinlong Li, Runsheng Xu, Xinyu Liu, Jin Ma, Baolu Li, Qin Zou, Jiaqi Ma, Hongkai Yu, Domain adaptation based object detection for autonomous driving in foggy and rainy weather, IEEE Trans. Intell. Veh., 2024, pp. 1-12, https://doi.org/10.1109/TIV.2024.3419689.

[15] Myung-Wook Park, Sangwoo Lee, Wooyong Han, Development of steering control system for autonomous vehicle using geometry-based path tracking algorithm, ETRI J., 37 (2015),URL https://api.semanticscholar.org/CorpusID:62733057.

[16] Fei Lai, Chao Qun Huang, Seventh-degree polynomial-based single lane change trajectory planning and four-wheel steering model predictive tracking control for intelligent vehicles, Vehicles, (2024),URL https://api.semanticscholar.org/CorpusID:275030190.

[17] Jian Zhou, Björn Olofsson, Erik Frisk, Interaction-aware motion planning for autonomous vehicles with multi-modal obstacle uncertainty predictions, IEEE Trans. Intell. Veh. 9 (1) (2024) 1305-1319, https://doi.org/10.1109/TIV.2023.3314709.

[18] Yeongseok Lee, Keun Ha Choi, Kyung-Soo Kim, GPU-enabled parallel trajectory optimization framework for safe motion planning of autonomous vehicles, IEEE Robot. Autom. Lett. 9 (11) (2024) 10407-10414, https://doi.org/10.1109/LRA.2024.3471452.

[19] Razvan C. Rafaila, Gheorghe Livint, Nonlinear model predictive control of autonomous vehicle steering, 2015 19th International Conference on System Theory, Control and Computing, ICSTCC, 2015, pp. 466-471, https://doi.org/10.1109/ICSTCC.2015.7321337.

[20] Kristian Ceder, Ze Zhang, Adam Burman, Ilya Kuangaliyev, Krister Mattsson, Gabriel Nyman, Arvid Petersén, Lukas Wisell, Knut Åkesson, Bird’s-eye-view trajectory planning of multiple robots using continuous deep reinforcement learning and model predictive control, 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2024, pp. 8002-8008, https://doi.org/10.1109/IROS58592.2024.10801434.

[21] Xiaoli Ren, Xiaoyong Li, Kaijun Ren, Junqiang Song, Zichen Xu, Kefeng Deng, Xiang Wang, Deep learning-based weather prediction: A survey, Big Data Res., 2214-5796 23 (2021) 100178, https://doi.org/10.1016/j.bdr.2020.100178,URL https://www.sciencedirect.com/science/article/pii/S2214579620300460.

[22] Tim Brophy, Darragh Mullins, Ashkan Parsi, Jonathan Horgan, Enda Ward, Patrick Denny, Ciarán Eising, Brian Deegan, Martin Glavin, Edward Jones, A review of the impact of rain on camera-based perception in automated driving systems, IEEE Access 11 (2023) 67040-67057, https://doi.org/1010.1109/ACCESS.2023.3290143.

[23] Xiaobai Ma, Jiachen Li, Mykel J. Kochenderfer, David Isele, Kikuo Fujimura, Reinforcement learning for autonomous driving with latent state inference and spatial-temporal relationships, 2021 IEEE International Conference on Robotics and Automation, ICRA, (2021), pp. 6064-6071,URL https://doi.org/10.1109/ICRA48506.2021.9562006.

[24] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, Fabien Moutarde, GRI: General reinforced imitation and its application to vision-based autonomous driving, Robotics 12 (5) (2023) 127,URL https://doi.org/10.3390/robotics12050127.

[25] Dian Chen, Vladlen Koltun, Philipp Krähenbühl, Learning to drive from a world on rails, 2021 IEEE/CVF International Conference on Computer Vision, ICCV, 2021, pp. 15570-15579, https://doi.org/10.1109/ICCV48922.2021.01530.

[26] Felipe Codevilla, Eder Santana, Antonio Lopez, Adrien Gaidon, Exploring the limitations of behavior cloning for autonomous driving, 2019 IEEE/CVF International Conference on Computer Vision, ICCV, 2019, pp. 9328-9337, https://doi.org/10.1109/ICCV.2019.00942.

[27] Dian Chen, Brady Zhou, Vladlen Koltun, Philipp Krähenbühl, Learning by Cheating, in: Proceedings of the Conference on Robot Learning, Vol. 100, 2020, pp. 66-75.

[28] Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, Jamie Shotton, Model-based imitation learning for urban driving, Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, 9781713871088, Curran Associates Inc., Red Hook, NY, USA, (2022).

[29] Dian Chen, Philipp Krähenbühl, Learning from all vehicles, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2022, pp. 17201-17210, https://doi.org/10.1109/CVPR52688.2022.01671.

[30] Hengli Wang, Peide Cai, Yuxiang Sun, Lujia Wang, Ming Liu, Learning interpretable end-to-end vision-based motion planning for autonomous driving with optical flow distillation, 2021 IEEE International Conference on Robotics and Automation, ICRA, IEEE Press, 2021, pp. 13731-13737, https://doi.org/10.1109/ICRA48506.2021.9561334.

[31] Sergio Casas, Cole Gulino, Renjie Liao, Raquel Urtasun, SpAGNN: Spatially-aware graph neural networks for relational behavior forecasting from sensor data, 2020 IEEE International Conference on Robotics and Automation, ICRA, 2020, pp. 9491-9497, https://doi.org/10.1109/ICRA40945.2020.9196697.

[32] Sen Han, Lingxiao Yan, Jiahao Sun, Shifeng Ding, Fang Li, Li Zhou, Automatic unberthing for underactuated unmanned surface vehicle: Model-based planning and control approaches in constricted harbors, Ocean Eng., 0029-8018 312 (2024) 119059, https://doi.org/10.1016/j.oceaneng.2024.119059,URL https://www.sciencedirect.com/science/article/pii/S0029801824023977.

[33] Hongbo Li, Bowen Li, Hongjiu Yang, Chaoxu Mu, Nonlinear model predictive control for time-optimal turning around of an autonomous vehicle under steering lag, IEEE/ASME Trans. Mechatronics 30 (1) (2025) 577-586, https://doi.org/10.1109/TMECH.2024.3400839.

[34] Zhuo Li, Yunlong Guo, Gang Wang, Wei Chen, Level curve tracking via robust RL-guided model predictive control, IEEE/CAA J. Autom. Sin. 11 (12) (2024) 2512-2514, https://doi.org/10.1109/JAS.2024.124350.

[35] Yuming Liu, Shitao Chen, Jiamin Shi, Nanning Zheng, The optimal horizon model predictive control planning for autonomous vehicles in dynamic environments, 2024 IEEE Intelligent Vehicles Symposium, IV, 2024, pp. 2421-2428, https://doi.org/10.1109/IV55156.2024.10588705.

[36] Qiping Chen, Binghao Yu, Shilong Min, Lu Gan, Chagen Luo, Dequan Zeng, Yiming Hu, Qin Liu, Study on intelligent vehicle trajectory planning and tracking control based on improved APF and MPC, Int. J. Automot. Technol., (2024),URL https://api.semanticscholar.org/CorpusID:274298645.

[37] Manel Ammour, Rodolfo Orjuela, Michel Basset, A MPC combined decision making and trajectory planning for autonomous vehicle collision avoidance, IEEE Trans. Intell. Transp. Syst. 23 (12) (2022) 24805-24817, https://doi.org/10.1109/TITS.2022.3210276.

[38] Yifan Liu, You Wang, Guang Li, SOMTP: A self-supervised learning-based optimizer for MPC-based safe trajectory planning problems in robotics, IEEE Robot. Autom. Lett. 9 (11) (2024) 9247-9254, https://doi.org/10.1109/LRA.2024.3456508.

[39] Johannes Fischer, Marlon Steiner, Ömer Şahin Taş, Christoph Stiller, Safety reinforced model predictive control (SRMPC): Improving MPC with reinforcement learning for motion planning in autonomous driving, 2023 IEEE 26th International Conference on Intelligent Transportation Systems, ITSC, 2023, pp. 2811-2818, https://doi.org/10.1109/ITSC57777.2023.10422605.

[40] Li Haoran, Chaozhong Wu, Duanfeng Chu, Liping Lu, Cheng Ken, Joint trajectory planning and tracking for autonomous vehicles considering driving style, IEEE Access 9 (2021) 9453-9463, https://doi.org/1010.1109/ACCESS.2021.3050005.