A survey of testing automated driving system

Abstract

Automated driving systems (ADSs) have made significant strides in recent years through the combined efforts of academia and industry. A typical ADS is composed of various complex modules, including perception, planning, and control. As emerging and complex computer programs, ADSs inevitably contain flaws, making it crucial to ensure their safety since any unsafe behavior can result in catastrophic outcomes. Testing is widely recognized as a key approach to ensuring ADS safety by uncovering unsafe behaviors. However, designing effective testing techniques for ADSs is exceptionally challenging due to the high complexity and multidisciplinary nature of these systems. Although an extensive body of literature focuses on ADS testing and several surveys summarizing technical advancements have been published, most concentrate on system-level testing performed within software simulators. Consequently, they often overlook the distinct characteristics, testing requirements, and datasets associated with various ADS modules. In this paper, we present a comprehensive survey of existing ADS testing literature. We begin by investigating the testing infrastructure for ADSs, including available datasets and tools, detailing their capabilities and characteristics. We then survey testing techniques for individual ADS modules (e.g., AI-based modules and firmware) and the integrated system, highlighting technical differences between validation layers. Finally, based on our findings, we identify key challenges and outline potential research opportunities in the field.

Graphical abstract

Keywords

ADS testing / module-level testing / system-level testing / firmware testing / dataset

References

[1] Zhong Z, Tang Y, Zhou Y, de Oliveira Neves V, Liu Y, Ray B. A survey on scenario-based testing for automated driving systems in high-fidelity simulation. 2021, arXiv preprint arXiv: 2112.00964

[2] Hartwich F, Beggiato M, Krems J F . Driving comfort, enjoyment and acceptance of automated driving – effects of drivers’ age and driving style familiarity. Ergonomics, 2018, 61( 8): 1017–1032

[3] Checkoway S, McCoy D, Kantor B, Anderson D, Shacham H, Savage S, Koscher K, Czeskis A, Roesner F, Kohno T. Comprehensive experimental analyses of automotive attack surfaces. In: Proceedings of the 20th USENIX Conference on Security. 2011, 6

[4] Wang S, Li Z, Wang Y, Zhao W, Liu T . Evidence of automated vehicle safety’s influence on people’s acceptance of the automated driving technology. Accident Analysis & Prevention, 2024, 195: 107381

[5] NTSB . Collision between a sport utility vehicle operating with partial driving automation and a crash attenuator. NTSB/HAR-20/01. Mountain View: NTSB, 2018

[6] Sportillo D, Paljic A, Ojeda L. On-road evaluation of autonomous driving training. In: Proceedings of ACM/IEEE International Conference on Human-Robot Interaction. 2019, 182–190

[7] Zhao X, Robu V, Flynn D, Salako K, Strigini L. Assessing the safety and reliability of autonomous vehicles from road testing. In: Proceedings of the 30th IEEE International Symposium on Software Reliability Engineering. 2019, 13–23

[8] Rong G, Shin B H, Tabatabaee H, Lu Q, Lemke S, Možeiko M, Boise E, Uhm G, Gerow M, Mehta S, Agafonov E, Kim T H, Sterner E, Ushiroda K, Reyes M, Zelenkovsky D, Kim S. LGSVL simulator: a high fidelity simulator for autonomous driving. In: Proceedings of the 23rd IEEE International Conference on Intelligent Transportation Systems. 2020, 1–6

[9] Baltodano S, Sibi S, Martelaro N, Gowda N, Ju W. The RRADS platform: a real road autonomous driving simulator. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 2015, 281–288

[10] Duy Son T, Bhave A, Van der Auweraer H. Simulation-based testing framework for autonomous driving development. In: Proceedings of IEEE International Conference on Mechatronics. 2019, 576–583

[11] Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V. CARLA: an open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning. 2017, 1–16

[12] Kalra N, Paddock S M . Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability?. Transportation Research Part A: Policy and Practice, 2016, 94: 182–193

[13] Wang S, Sheng Z, Xu J, Chen T, Zhu J, Zhang S, Yao Y, Ma X. ADEPT: a testing platform for simulated autonomous driving. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 2022, 150

[14] Huai Y, Chen Y, Almanee S, Ngo T, Liao X, Wan Z, Chen Q A, Garcia J. Doppelgänger test generation for revealing bugs in autonomous driving software. In: Proceedings of IEEE/ACM International Conference on Software Engineering. 2023, 2591–2603

[15] Li Z, Pan M, Zhang T, Li X. Testing DNN-based autonomous driving systems under critical environmental conditions. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 6471–6482

[16] Yang Z, Huang S, Zheng C, Wang X, Wang Y, Xia C . MetaLiDAR: automated metamorphic testing of LiDAR-based autonomous driving systems. Journal of Software: Evolution and Process, 2024, 36( 7): e2644

[17] Huybrechts T, Vanommeslaeghe Y, Blontrock D, Van Barel G, Hellinckx P. Automatic reverse engineering of CAN bus data using machine learning techniques. In: Proceedings of the 12th International Conference on P2P. 2017, 751–761

[18] Fowler D S, Bryans J, Cheah M, Wooderson P, Shaikh S A. A method for constructing automotive cybersecurity tests, a CAN fuzz testing example. In: Proceedings of the 19th IEEE International Conference on Software Quality, Reliability and Security Companion. 2019, 1–8

[19] Zaddach J, Bruno L, Francillon A, Balzarotti D. AVATAR: a framework to support dynamic security analysis of embedded systems’ firmwares. In: Proceedings of the 21st Annual Network and Distributed System Security Symposium. 2014

[20] Feng B, Mera A, Lu L. P²IM: scalable and hardware-independent firmware testing via automatic peripheral interface modeling. In: Proceedings of the 29th USENIX Security Symposium. 2020, 1237–1254

[21] Davidson D, Moench B, Jha S, Ristenpart T. FIE on firmware: finding vulnerabilities in embedded systems using symbolic execution. In: Proceedings of the 22nd USENIX Security Symposium. 2013, 463–478

[22] Chen D D, Egele M, Woo M, Brumley D. Towards automated dynamic analysis for Linux-based embedded firmware. In: Proceedings of the 23rd Annual Network and Distributed System Security Symposium. 2016

[23] Zheng Y, Davanian A, Yin H, Song C, Zhu H, Sun L. FIRM-AFL: High-Throughput greybox fuzzing of IoT firmware via augmented process emulation. In: Proceedings of the 28th USENIX Security Symposium. 2019, 1099–1114

[24] Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3354–3361

[25] Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B. The cityscapes dataset for semantic urban scene understanding. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 3213–3223

[26] Dai J, Gao B, Luo M, Huang Z, Li Z, Zhang Y, Yang M. SCTRANS: constructing a large public scenario dataset for simulation testing of autonomous driving systems. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 2024, 591–603

[27] Yurtsever E, Lambert J, Carballo A, Takeda K . A survey of autonomous driving: common practices and emerging technologies. IEEE Access, 2020, 8: 58443–58469

[28] Lou G, Deng Y, Zheng X, Zhang M, Zhang T. Testing of autonomous driving systems: where are we and where should we go? In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2022, 31–43

[29] Dai J, Li Z, Zhang W, Zhang Y, Yang M . Simulation-based fuzzing for autonomous driving systems: landscapes, challenges and prospects. Journal of Computer Research and Development, 2023, 60( 7): 1433–1447

[30] Tahir M, Alexander R. Coverage based testing for V&V and safety assurance of self-driving autonomous vehicles: a systematic literature review. In: Proceedings of 2020 IEEE International Conference on Artificial Intelligence Testing. 2021, 23–30

[31] Yang Z J, Jia X S, Li H Y, Yan J C. LLM4Drive: A Survey of Large Language Models for Autonomous Driving. 2024, arXiv preprint arXiv:2311.01043

[32] Alemayehu H, Sargolzaei A . Testing and verification of connected and autonomous vehicles: a review. Electronics, 2025, 14( 3): 600

[33] Song H, Stocco A, Chowdhury S. Concepts in testing of autonomous systems: academic literature and industry practice. In: Proceedings of IEEE International Conference on Software Testing, Verification and Validation Workshops. 2021, 66–73

[34] Tang S, Zhang Z, Zhang Y, Zhou J, Guo Y, Liu S, Guo S, Li Y F, Ma L, Xue Y, Liu Y . A survey on automated driving system testing: landscapes and trends. ACM Transactions on Software Engineering and Methodology, 2023, 32( 5): 124

[35] Zhu X, Wang H, You H, Zhang W, Zhang Y, Liu S, Chen J, Wang Z, Li K . Survey on testing of intelligent systems in autonomous vehicles. Journal of Software, 2021, 32( 7): 2056–2077

[36] SAE International. SAE J3016:2014 Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems. Warrendale, PA: SAE International, 2014

[37] China Intelligent Transportation Systems Association. T/ITS 0131-2019 Commercial vehicle — Safety specifications for automatic small and medium bus. Beijing: China Intelligent Transportation Systems Association, 2019

[38] State Administration for Market Regulation, National Standardization Administration. GB/T 34590.1-2022 Road vehicles — Functional safety — Part 1: Vocabulary. Beijing: Standards Press of China, 2022

[39] State Administration for Market Regulation, National Standardization Administration. GB/T 40429-2021 Taxonomy of driving automation for vehicles. Beijing: Standards Press of China, 2021

[40] China Society of Automotive Engineers. T/CSAE 212-2021 Requirements and methods for scenario data image annotation of intelligent connected vehicles. Beijing: China Society of Automotive Engineers, 2021

[41] State Administration for Market Regulation, National Standardization Administration. GB/T 41796-2022 Performance requirements and test methods for lane keeping assist system of commercial vehicles. Beijing: Standards Press of China, 2022

[42] State Administration for Market Regulation, National Standardization Administration. GB/T 41797-2022 Performance requirements and test methods for driver attention monitoring system. Beijing: Standards Press of China, 2022

[43] State Administration for Market Regulation, National Standardization Administration. GB/T 41798-2022 Intelligent and connected vehicles - track testing methods and requirements for automated driving functions. Beijing: Standards Press of China, 2022

[44] State Administration for Market Regulation, National Standardization Administration. GB/T 44721-2024 Intelligent and connected vehicle—General technical requirements for automated driving system. Beijing: Standards Press of China, 2024

[45] State Administration for Market Regulation, National Standardization Administration. GB 44495-2024 Technical requirements for vehicle cybersecurity. Beijing: Standards Press of China, 2024

[46] State Administration for Market Regulation, National Standardization Administration. GB 44496-2024 General technical requirements for software update of vehicles. Beijing: Standards Press of China, 2024

[47] State Administration for Market Regulation, National Standardization Administration. GB 44497-2024 Intelligent and connected vehicle - data storage system for automated driving. Beijing: Standards Press of China, 2024

[48] Beijing Municipal People’s Congress Standing Committee. Regulations on the Administration of Road Testing and Demonstration Application of Intelligent Connected Vehicles in Beijing. Beijing, 2024

[49] China Quality Certification Center. Implementation rules for compulsory product certification-motor vehicles: CQC-C1101-2020. Beijing: China Quality Certification Center, 2020

[50] U.S. House of Representatives. H.R. 3388 - Safely Ensuring Lives Future Deployment and Research In Vehicle Evolution Act (SELF DRIVE Act). Washington, D.C.: U.S. Government Publishing Office, 2017

[51] U.S. Department of Transportation, National Highway Traffic Safety Administration. Automated Driving Systems 2.0: A Vision for Safety. Washington, D.C.: U.S. Department of Transportation, 2017

[52] U.S. Senate. S. 680 - Security and Privacy in Your Car Act of 2017. Washington, D.C.: U.S. Government Publishing Office, 2017

[53] National Science Technology Council, U.S. Department of Transportation. Ensuring American Leadership in Automated Vehicle Technologies: Automated Vehicles 4.0. Washington, D.C.: U.S. Department of Transportation, 2020

[54] National Highway Traffic Safety Administration. ADS-equipped vehicle safety, transparency, and evaluation program. 2024

[55] United Nations Economic Commission for Europe. UN Regulation No. 155 - Uniform provisions concerning the approval of vehicles with regards to cyber security and cyber security management system. Geneva: United Nations, 2021

[56] United Nations Economic Commission for Europe. UN Regulation No. 156 - Uniform provisions concerning the approval of vehicles with regards to software update and software updates management system. Geneva: United Nations, 2021

[57] United Nations Economic Commission for Europe. UN Regulation No. 157 - Uniform provisions concerning the approval of vehicles with regard to Automated Lane Keeping Systems. Geneva: United Nations, 2021

[58] Federal Republic of Germany. Act to Amend the Road Traffic Act and the Compulsory Insurance Act – Autonomous Driving. Bonn: Federal Law Gazette (Bundesgesetzblatt), 2021

[59] European Commission. Commission Implementing Regulation (EU) 2022/1426 of 5 August 2022 laying down rules for the application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type-approval of the automated driving system (ADS) of fully automated vehicles. Luxembourg: Publications Office of the European Union, 2022

[60] European Parliament, Council of the European Union. Regulation (EU) 2023/988 of the European Parliament and of the Council of 10 May 2023 on general product safety, amending Regulation (EU) No 1025/2012 of the European Parliament and of the Council and Directive (EU) 2020/1828 of the European Parliament and of the Council, and repealing Directive 2001/95/EC of the European Parliament and of the Council and Council Directive 87/357/EEC. Luxembourg: Publications Office of the European Union, 2023

[61] European Parliament, Council of the European Union. Directive (EU) 2022/2555 of the European Parliament and of the Council of 14 December 2022 on measures for a high common level of cybersecurity across the Union, amending Regulation (EU) No 910/2014 and Directive (EU) 2018/1972, and repealing Directive (EU) 2016/1148 (NIS 2 Directive). Luxembourg: Publications Office of the European Union, 2022

[62] United Nations Economic Commission for Europe. UN Regulation No. 171 - Uniform provisions concerning the approval of vehicles with regard to Driver Control Assistance Systems (DCAS). Geneva: United Nations, 2024

[63] UK Parliament. Automated vehicles act 2024. See legislation.gov.uk/ukpga/2024/10#:~:text=An%20Act%20to%20regulate%20the%20use%20of%20automated,in%20relation%20to%20vehicle%20automation.%20%5B20th%20May%202024%5D website, 2024

[64] International Organization for Standardization. ISO 26262:2018 Road vehicles — Functional safety. Geneva: International Organization for Standardization, 2018

[65] International Organization for Standardization. ISO/SAE 21434:2021 Road vehicles — Cybersecurity engineering. Geneva: International Organization for Standardization, 2021

[66] International Organization for Standardization. ISO/SAE PAS 22736:2021 Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Geneva: International Organization for Standardization, 2021

[67] International Organization for Standardization. ISO 22737:2021 Intelligent transport systems — Low-speed automated driving (LSAD) systems for predefined routes — Performance requirements, system requirements and performance test procedures. Geneva: International Organization for Standardization, 2021

[68] International Organization for Standardization. ISO 21448:2022 Road vehicles — Safety of the intended functionality. Geneva: International Organization for Standardization, 2022

[69] International Organization for Standardization. ISO 34501:2022 Road vehicles — Test scenarios for automated driving systems — Vocabulary. Geneva: International Organization for Standardization, 2022

[70] International Organization for Standardization. ISO 34502:2022 Road vehicles — Test scenarios for automated driving systems — Scenario based safety evaluation framework. Geneva: International Organization for Standardization, 2022

[71] International Organization for Standardization. ISO 34503:2023 Road vehicles — Test scenarios for automated driving systems — Specification for operational design domain. Geneva: International Organization for Standardization, 2023

[72] United Nations Economic Commission for Europe. Framework Document on Automated/Autonomous Vehicles (ECE/TRANS/WP.29/2019/34/Rev.2). Geneva: United Nations, 2022

[73] International Organization for Standardization. ISO/PAS 8800:2024 Road vehicles — Safety and artificial intelligence. Geneva: International Organization for Standardization, 2024

[74] Guo A, Feng Y, Chen Z. LiRTest: augmenting LiDAR point clouds for automated testing of autonomous driving systems. In: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 2022, 480–492

[75] Tu J, Ren M, Manivasagam S, Liang M, Yang B, Du R, Cheng F, Urtasun R. Physically realizable adversarial examples for LiDAR object detection. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 13713–13722

[76] Gao X, Wang Z, Feng Y, Ma L, Chen Z, Xu B. Benchmarking robustness of AI-enabled multi-sensor fusion systems: challenges and opportunities. In: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2023, 871–882

[77] Gao X, Wang Z, Feng Y, Ma L, Chen Z, Xu B. MultiTest: physical-aware object insertion for testing multi-sensor fusion perception systems. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 2024, 139

[78] Tu J, Li H, Yan X, Ren M, Chen Y, Liang M, Bitar E, Yumer E, Urtasun R. Exploring adversarial robustness of multi-sensor perception systems in self driving. In: Proceedings of the 5th Conference on Robot Learning. 2022, 1013–1024

[79] Zhou H, Li W, Kong Z, Guo J, Zhang Y, Yu B, Zhang L, Liu C. DeepBillboard: systematic physical-world testing of autonomous driving systems. In: Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering. 2020, 347–358

[80] Xie R, Cui Z, Chen X, Zheng L . IATG: interpretation-analysis-based testing method for autonomous driving software. Journal of Software, 2024, 35( 6): 2753–2774

[81] Woodlief T, Elbaum S, Sullivan K. Semantic image fuzzing of AI perception systems. In: Proceedings of the 44th International Conference on Software Engineering. 2022, 1958–1969

[82] Zhang X, Cai Y. Building critical testing scenarios for autonomous driving from real accidents. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2023, 462–474

[83] Udacity/self-driving-car . The Udacity open source self-driving car project. See github.com/udacity/self-driving-car website, 2021

[84] Pei K, Cao Y, Yang J, Jana S. DeepXplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles. 2017, 1–18

[85] Tian Y, Pei K, Jana S, Ray B. DeepTest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering. 2018, 303–314

[86] Zhang M, Zhang Y, Zhang L, Liu C, Khurshid S. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering. 2018, 132–142

[87] Kim J, Feldt R, Yoo S . Evaluating surprise adequacy for deep learning system testing. ACM Transactions on Software Engineering and Methodology, 2023, 32( 2): 42

[88] Kalaee A, Parsa S . Metamorphic testing of deep neural network-based autonomous driving systems using behavioural domain adequacy. Neural Computing and Applications, 2025, 37( 9): 6677–6724

[89] Sakaridis C, Dai D, Van Gool L . Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 2018, 126( 9): 973–992

[90] Behley J, Garbade M, Milioto A, Quenzel J, Behnke S, Stachniss C, Gall J. SemanticKITTI: a dataset for semantic scene understanding of LiDAR sequences. In: Proceedings of IEEE/CVF International Conference on Computer Vision. 2019, 9296–9306

[91] Christian G, Woodlief T, Elbaum S. Generating realistic and diverse tests for LiDAR-based perception systems. In: Proceedings of IEEE/ACM 45th International Conference on Software Engineering. 2023, 2604–2616

[92] Caesar H, Bankiti V, Lang A H, Vora S, Liong V E, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O. nuScenes: a multimodal dataset for autonomous driving. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 11618–11628

[93] Zhang Q, Hu S, Sun J, Chen Q A, Mao Z M. On adversarial robustness of trajectory prediction for autonomous vehicles. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 15138–15147

[94] Sun P, Kretzschmar H, Dotiwalla X, Chouard A, Patnaik V, Tsui P, Guo J, Zhou Y, Chai Y, Caine B, Vasudevan V, Han W, Ngiam J, Zhao H, Timofeev A, Ettinger S, Krivokon M, Gao A, Joshi A, Zhang Y, Shlens J, Chen Z, Anguelov D. Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 2443–2451

[95] Yu F, Chen H, Wang X, Xian W, Chen Y, Liu F, Madhavan V, Darrell T. BDD100K: a diverse driving dataset for heterogeneous multitask learning. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 2633–2642

[96] Jia Y, Lu Y, Shen J, Chen Q A, Chen H, Zhong Z, Wei T. Fooling detection alone is not enough: Adversarial attack against multiple object tracking. In: Proceedings of International Conference on Learning Representations. 2020

[97] Luu Q H, Liu H, Chen T Y, Vu H L. A sequential metamorphic testing framework for understanding autonomous vehicle’s decisions. IEEE Transactions on Intelligent Vehicles, 2024

[98] Geyer J, Kassahun Y, Mahmudi M, Ricou X, Durgesh R, Chung A S, Hauswald L, Pham V H, Mühlegg M, Dorn S, Fernandez T, Jänicke M, Mirashi S, Savani C, Sturm M, Vorobiov O, Oelker M, Garreis S, Schuberth P. A2D2: Audi autonomous driving dataset. 2020, arXiv preprint arXiv: 2004.06320

[99] Deng Y, Zheng X, Zhang T, Liu H, Lou G, Kim M, Chen T Y . A declarative metamorphic testing framework for autonomous driving. IEEE Transactions on Software Engineering, 2023, 49( 4): 1964–1982

[100] Diaz-Ruiz C A, Xia Y, You Y, Nino J, Chen J, Monica J, Chen X, Luo K, Wang Y, Emond M, Chao W L, Hariharan B, Weinberger K Q, Campbell M. Ithaca365: dataset and driving perception under repeated and challenging weather conditions. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 21351–21360

[101] Gambi A, Huynh T, Fraser G. Generating effective test cases for self-driving cars from police reports. In: Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2019, 257–267

[102] Queiroz R, Sharma D, Caldas R, Czarnecki K, García S, Berger T, Pelliccione P . A driver-vehicle model for ADS scenario-based testing. IEEE Transactions on Intelligent Transportation Systems, 2024, 25( 8): 8641–8654

[103] Guo A, Zhou Y, Tian H, Fang C, Sun Y, Sun W, Gao X, Luu A T, Liu Y, Chen Z. SoVAR: building generalizable scenarios from accident reports for autonomous driving testing. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering. 2024, 268–280

[104] Zhong Z, Kaiser G, Ray B . Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles. IEEE Transactions on Software Engineering, 2023, 49( 4): 1860–1875

[105] Zhang K, Feng F, Katare D, Zhang Y, Huang Z, Keroglou C, Wang Z, Huang H H, Shi H, Krishnan S. ChatScene: knowledge-enabled safety-critical scenario generation for autonomous vehicles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, 14890–14899

[106] Lu C, Yue T, Ali S. DeepScenario: an open driving scenario dataset for autonomous driving system testing. In: Proceedings of the 20th IEEE/ACM International Conference on Mining Software Repositories. 2023, 52–56

[107] Gong L, Zhang Y, Xia Y, Zhang Y, Ji J. SDAC: a multimodal synthetic dataset for anomaly and corner case detection in autonomous driving. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 1914–1922

[108] von Stein M, Shriver D, Elbaum S . DeepManeuver: adversarial test generation for trajectory manipulation of autonomous vehicles. IEEE Transactions on Software Engineering, 2023, 49( 10): 4496–4509

[109] Gambi A, Mueller M, Fraser G. Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2019, 318–328

[110] Luo Y, Zhang X Y, Arcaini P, Jin Z, Zhao H, Ishikawa F, Wu R, Xie T. Targeting requirements violations of autonomous driving systems by dynamic evolutionary search. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering. 2021, 279–291

[111] Birchler C, Khatiri S, Derakhshanfar P, Panichella S, Panichella A . Single and multi-objective test cases prioritization for self-driving cars in virtual environments. ACM Transactions on Software Engineering and Methodology, 2023, 32( 2): 28

[112] Tang S, Zhang Z, Zhou J, Zhou Y, Li Y F, Xue Y. EvoScenario: integrating road structures into critical scenario generation for autonomous driving system testing. In: Proceedings of the 34th IEEE International Symposium on Software Reliability Engineering. 2023, 309–320

[113] Hamdi A, Mueller M, Ghanem B. SADA: semantic adversarial diagnostic attacks for autonomous applications. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 10901–10908

[114] Haq F U, Shin D, Briand L. Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization. In: Proceedings of the 44th IEEE/ACM International Conference on Software Engineering. 2022, 811–822

[115] Wang T, Gu T, Deng H, Li H, Kuang X, Zhao G. Dance of the ADS: orchestrating failures through historically-informed scenario fuzzing. In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024, 1086–1098

[116] Crespo-Rodriguez V, Neelofar , Aleti A. PAFOT: a position-based approach for finding optimal tests of autonomous vehicles. In: Proceedings of 2024 IEEE/ACM International Conference on Automation of Software Test. 2024, 159–170

[117] Kim S, Liu M, Rhee J J, Jeon Y, Kwon Y, Kim C H. DriveFuzz: discovering autonomous driving bugs through driving quality-guided fuzzing. In: Proceedings of ACM SIGSAC Conference on Computer and Communications Security. 2022, 1753–1767

[118] Lin S, Chen F, Xi L, Wang G, Xi R, Sun Y, Zhu H . TM-fuzzer: fuzzing autonomous driving systems through traffic management. Automated Software Engineering, 2024, 31( 2): 61

[119] Li C, Sifakis J, Yan R, Zhang J. Rigorous simulation-based testing for autonomous driving systems − targeting the Achilles’ heel of four open autopilots. 2024, arXiv preprint arXiv: 2405.16914

[120] Yang Y, Kujanpää K, Babadi I A, Pajarinen J, Ilin A. Suicidal pedestrian: generation of safety-critical scenarios for autonomous vehicles. In: Proceedings of the 26th IEEE International Conference on Intelligent Transportation Systems. 2023, 1983–1988

[121] Underwood R, Luu Q H, Liu H. A metamorphic testing framework and toolkit for modular automated driving systems. In: Proceedings of the 8th IEEE/ACM International Workshop on Metamorphic Testing. 2023, 17–24

[122] Expósito Jiménez V J, Macher G, Watzenig D, Brenner E . Safety of the intended functionality validation for automated driving systems by using perception performance insufficiencies injection. Vehicles, 2024, 6( 3): 1164–1184

[123] Lu Y, Tian Y, Bi Y, Chen B, Peng X. DiaVio: LLM-empowered diagnosis of safety violations in ADS simulation testing. In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024, 376–388

[124] Zhong Z, Hu Z, Guo S, Zhang X, Zhong Z, Ray B. Detecting multi-sensor fusion errors in advanced driver-assistance systems. In: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 2022, 493–505

[125] Li G, Li Y, Jha S, Tsai T, Sullivan M, Hari S K S, Kalbarczyk Z, Iyer R. AV-FUZZER: finding safety violations in autonomous driving systems. In: Proceedings of the 31st IEEE International Symposium on Software Reliability Engineering. 2020, 25–36

[126] Hu Z, Guo S, Zhong Z, Li K. Coverage-based scene fuzzing for virtual autonomous driving testing. 2021, arXiv preprint arXiv: 2106.00873

[127] Li C, Cheng C H, Sun T, Chen Y, Yan R. ComOpT: combination and optimization for testing autonomous driving systems. In: Proceedings of International Conference on Robotics and Automation. 2022, 7738–7744

[128] Tang Y, Zhou Y, Zhang T, Wu F, Liu Y, Wang G. Systematic testing of autonomous driving systems using map topology-based scenario classification. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering. 2021, 1342–1346

[129] Zhou J, Tang S, Guo Y, Li Y F, Xue Y. From collision to verdict: responsibility attribution for autonomous driving systems testing. In: Proceedings of the 34th IEEE International Symposium on Software Reliability Engineering. 2023, 321–332

[130] Sun Y, Poskitt C M, Sun J, Chen Y, Yang Z. LawBreaker: an approach for specifying traffic laws and fuzzing autonomous vehicles. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 2022, 62

[131] Li C, Sifakis J, Wang Q, Yan R, Zhang J. Simulation-based validation for autonomous driving systems. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2023, 842–853

[132] Zhang X, Zhao W, Sun Y, Sun J, Shen Y, Dong X, Yang Z. Testing automated driving systems by breaking many laws efficiently. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2023, 942–953

[133] Li Z, Dai J, Huang Z, You N, Zhang Y, Yang M. VioHawk: detecting traffic violations of autonomous driving systems through criticality-guided simulation testing. In: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024, 844–855

[134] Lu C, Shi Y, Zhang H, Zhang M, Wang T, Yue T, Ali S . Learning configurations of operating environment of autonomous vehicles to maximize their collisions. IEEE Transactions on Software Engineering, 2023, 49( 1): 384–402

[135] Huai Y, Almanee S, Chen Y, Wu X, Chen Q A, Garcia J . scenoRITA: generating diverse, fully mutable, test scenarios for autonomous vehicle planning. IEEE Transactions on Software Engineering, 2023, 49( 10): 4656–4676

[136] Yang Z, Huang S, Wang X, Bai T, Wang Y . MT-Nod: metamorphic testing for detecting non-optimal decisions of autonomous driving systems in interactive scenarios. Information and Software Technology, 2025, 180: 107659

[137] Cheng M, Zhou Y, Xie X, Wang J, Meng G, Yang K. Evaluating decision optimality of autonomous driving via metamorphic testing. 2024, arXiv preprint arXiv: 2402.18393

[138] Iqbal M, Han J C, Zhou Z Q, Towey D, Chen T Y . Metamorphic testing of Advanced Driver-Assistance System (ADAS) simulation platforms: Lane Keeping Assist System (LKAS) case studies. Information and Software Technology, 2023, 155: 107104

[139] Abdessalem R B, Nejati S, Briand L C, Stifter T. Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of International Conference on Software Engineering. 2018, 1016–1026

[140] Kochanthara S, Singh T, Forrai A, Cleophas L . Safety of perception systems for automated driving: a case study on Apollo. ACM Transactions on Software Engineering and Methodology, 2024, 33( 3): 64

[141] Ma L, Zhang F, Sun J, Xue M, Li B, Chen C, Liu T, Zhao J, Liu Y. DeepMutation: mutation testing of deep learning systems. In: Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE). 2018, 100–111

[142] Riccio V, Jahangirova G, Stocco A, Humbatova N, Weiss M, Tonella P. Testing machine learning based systems: a systematic mapping. Empirical Software Engineering. 2020, 25(6): 5193–5254

[143] Feng Y, Shi Q, Gao X, Wan J, Fang C, Chen Z. DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 2020, 177–188

[144] Weng S, Feng Y, Yin Y, Liu J. Prioritizing testing instances to enhance the robustness of object detection systems. In: Proceedings of the 14th Asia-Pacific Symposium on Internetware. 2023, 194–204

[145] Xie X, Ma L, Juefei-Xu F, Xue M, Chen H, Liu Y, Zhao J, Li B, Yin J, See S. DeepHunter: a coverage-guided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 2019, 146–157

[146] Michaelis C, Mitzkus B, Geirhos R, Rusak E, Bringmann O, Ecker A S, Bethge M, Brendel W. Benchmarking robustness in object detection. 2019, arXiv preprint arXiv: 1907.07484

[147] Stocco A, Pulfer B, Tonella P . Mind the gap! A study on the transferability of virtual versus physical-world testing of autonomous driving systems. IEEE Transactions on Software Engineering, 2023, 49( 4): 1928–1940

[148] Li X, Chen Z, Zhao Y, Tong Z, Zhao Y, Lim A, Zhou J T. PointBA: Towards backdoor attacks in 3D point cloud. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, 16492–16501

[149] Hanselmann N, Renz K, Chitta K, Bhattacharyya A, Geiger A. KING: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. In: Proceedings of the European Conference on Computer Vision. 2022, 335–352

[150] Stocco A, Nunes P J, D’Amorim M, Tonella P. ThirdEye: attention maps for safe autonomous driving systems. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 2022, 102

[151] Lee H, Choi K, Chung K, Kim J, Yim K. Fuzzing CAN packets into automobiles. In: Proceedings of IEEE International Conference on Advanced Information Networking and Applications. 2015, 817–821

[152] Markovitz M, Wool A . Field classification, modeling and anomaly detection in unknown CAN bus networks. Vehicular Communications, 2017, 9: 43–52

[153] Marchetti M, Stabili D . READ: reverse engineering of automotive data frames. IEEE Transactions on Information Forensics and Security, 2019, 14( 4): 1083–1097

[154] Pesé M D, Stacer T, Campos C A, Newberry E, Chen D, Shin K G. LibreCAN: automated CAN message translator. In: Proceedings of ACM SIGSAC Conference on Computer and Communications Security. 2019, 2283–2300

[155] Verma M, Bridges R, Hollifield S. ACTT: automotive CAN tokenization and translation. In: Proceedings of International Conference on Computational Science and Computational Intelligence. 2018, 278–283

[156] Young C, Svoboda J, Zambreno J. Towards reverse engineering controller area network messages using machine learning. In: Proceedings of IEEE World Forum on Internet of Things. 2020, 1–6

[157] Buscemi A, Turcanu I, Castignani G, Crunelle R, Engel T . CANMatch: a fully automated tool for CAN bus reverse engineering based on frame matching. IEEE Transactions on Vehicular Technology, 2021, 70( 12): 12358–12373

[158] Bi Z, Xu G, Xu G, Wang C, Zhang S . Bit-level automotive controller area network message reverse framework based on linear regression. Sensors, 2022, 22( 3): 981

[159] Bi Z, Xu G, Wang C, Xu G, Zhang S . A method for translating automotive body-related CAN messages based on labeled bits. Applied Sciences, 2023, 13( 3): 1942

[160] Kim H, Jeong Y, Choi W, Lee D H, Jo H J . Efficient ECU analysis technology through structure-aware CAN fuzzing. IEEE Access, 2022, 10: 23259–23271

[161] Li Z, Jiang W, Liu X, Tan K, Jin X, Yang M . GAN model using field fuzz mutation for in-vehicle CAN bus intrusion detection. Mathematical Biosciences and Engineering, 2022, 19( 7): 6996–7018

[162] Park H B, Kim Y, Jeon J, Moon H S, Woo S . Practical methodology for in-vehicle CAN security evaluation. Journal of Internet Services and Information Security, 2019, 9( 2): 42–56

[163] Studnia I, Nicomette V, Alata E, Deswarte Y, Kaâniche M, Laarouchi Y. Security of embedded automotive networks: state of the art and a research proposal. In: Proceedings of the 32nd International Conference on Computer Safety, Reliability and Security. 2013

[164] Song H M, Kim H R, Kim H K. Intrusion detection system based on the analysis of time intervals of CAN messages for in-vehicle network. In: Proceedings of International Conference on Information Networking. 2016, 63–68

[165] Marchetti M, Stabili D, Guido A, Colajanni M. Evaluation of anomaly detection for in-vehicle networks through information-theoretic algorithms. In: Proceedings of IEEE International Forum on Research and Technologies for Society and Industry Leveraging a Better Tomorrow. 2016, 1–6

[166] Tian D, Li Y, Wang Y, Duan X, Wang C, Wang W, Hui R, Guo P. An intrusion detection system based on machine learning for CAN-bus. In: Proceedings of the 3rd International Conference on Industrial Networks and Intelligent Systems. 2017, 285–2948

[167] Kang M J, Kang J W . Intrusion detection system using deep neural network for in-vehicle network security. PLoS One, 2016, 11( 6): e0155781

[168] Zhou A, Li Z, Shen Y . Anomaly detection of CAN bus messages using a deep neural network for autonomous vehicles. Applied Sciences, 2019, 9( 15): 3174

[169] Taylor A, Leblanc S, Japkowicz N. Anomaly detection in automobile control network data with long short-term memory networks. In: Proceedings of IEEE International Conference on Data Science and Advanced Analytics. 2016, 130–139

[170] Kwak B I, Han M L, Kim H K . Cosine similarity based anomaly detection methodology for the CAN bus. Expert Systems with Applications, 2021, 166: 114066

[171] Aksu D, Aydin M A . MGA-IDS: optimal feature subset selection for anomaly detection framework on in-vehicle networks-CAN bus based on genetic algorithm and intrusion detection approach. Computers & Security, 2022, 118: 102717

[172] Zhang G, Liu Q, Cao C, Li J, Li Y . Bit scanner: anomaly detection for in-vehicle CAN bus using binary sequence whitelisting. Computers & Security, 2023, 134: 103436

[173] Tang Z, Serag K, Zonouz S, Celik Z B, Xu D, Beyah R. ERACAN: defending against an emerging CAN threat model. In: Proceedings of ACM SIGSAC Conference on Computer and Communications Security. 2024, 1894–1908

[174] Chen C Y, Shin K G, Dadras S. Context-aware anomaly detection using vehicle dynamics. In: Proceedings of International Symposium on Research in Attacks, Intrusions and Defenses. 2024, 531–545

[175] Jing P, Cai Z, Cao Y, Yu L, Du Y, Zhang W, Qian C, Luo X, Nie S, Wu S. Revisiting automotive attack surfaces: a practitioners’ perspective. In: Proceedings of IEEE Symposium on Security and Privacy. 2024, 2348–2365

[176] Corteggiani N, Camurati G, Francillon A. Inception: system-wide security testing of real-world embedded systems software. In: Proceedings of USENIX Security Symposium. 2018, 309–326

[177] Gustafson E, Muench M, Spensky C, Redini N, Machiry A, Fratantonio Y, Francillon A, Balzarotti D, Choe Y R, Kruegel C, Vigna G. Toward the analysis of embedded firmware through automated re-hosting. In: Proceedings of International Symposium on Research in Attacks, Intrusions and Defenses. 2019, 135–150

[178] Mera A, Feng B, Lu L, Kirda E. DICE: automatic emulation of DMA input channels for dynamic firmware analysis. In: Proceedings of IEEE Symposium on Security and Privacy. 2021, 1938–1954

[179] Scharnowski T, Bars N, Schloegel M, Gustafson E, Muench M, Vigna G, Kruegel C, Holz T, Abbasi A. Fuzzware: using precise MMIO modeling for effective firmware fuzzing. In: Proceedings of USENIX Security Symposium. 2022, 1239–1256

[180] Feng B, Luo M, Liu C, Lu L, Kirda E . AIM: automatic interrupt modeling for dynamic firmware analysis. IEEE Transactions on Dependable and Secure Computing, 2024, 21( 4): 3866–3882

[181] Cao C, Guan L, Ming J, Liu P. Device-agnostic firmware execution is possible: a concolic execution approach for peripheral emulation. In: Proceedings of the 36th Annual Computer Security Applications Conference. 2020, 746–759

[182] Li W, Shi J, Li F, Lin J, Wang W, Guan L. μAFL: non-intrusive feedback-driven fuzzing for microcontroller firmware. In: Proceedings of the 44th International Conference on Software Engineering. 2022, 1–12

[183] Mera A, Liu C, Sun R, Kirda E, Lu L. SHiFT: semi-hosted fuzz testing for embedded applications. In: Proceedings of the 33rd USENIX Conference on Security Symposium. 2024, 5323–5340

[184] Liu C, Mera A, Kirda E, Xu M, Lu L. CO3: concolic co-execution for firmware. In: Proceedings of the 33rd USENIX Security Symposium. 2024, 5591–5608

[185] Lei C, Ling Z, Zhang Y, Yang Y, Luo J, Fu X. A friend’s eye is a good mirror: synthesizing MCU peripheral models from peripheral drivers. In: Proceedings of USENIX Security Symposium. 2024, 396

[186] Winner H, Lemmer K, Form T, Mazzega J. PEGASUS—first steps for the safe introduction of automated driving. In: Meyer G, Beiker S, eds. Road Vehicle Automation 5. Cham: Springer, 2019, 185–195

[187] Fremont D J, Dreossi T, Ghosh S, Yue X, Sangiovanni-Vincentelli A L, Seshia S A. Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation. 2019, 63–78

[188] Majumdar R, Mathur A, Pirron M, Stegner L, Zufferey D. Paracosm: a language and tool for testing autonomous driving systems. 2021, arXiv preprint arXiv: 1902.01084

[189] Queiroz R, Berger T, Czarnecki K. GeoScenario: an open DSL for autonomous driving scenario representation. In: Proceedings of IEEE Intelligent Vehicles Symposium. 2019, 287–294

[190] Li B, Du D, Chen S, Wei M, Zheng C, Zhang X. SML4ADS: an open DSML for autonomous driving scenario representation and generation. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 2022, 145

[191] Tan S, Ivanovic B, Weng X, Pavone M, Kraehenbuehl P. Language conditioned traffic generation. In: Proceedings of the 7th Annual Conference on Robot Learning. 2023

[192] Guo A, Feng Y, Cheng Y, Chen Z . Semantic-guided fuzzing for virtual testing of autonomous driving systems. Journal of Systems and Software, 2024, 212: 112017

[193] Dodoiu T, da Costa A A B, Khastgir S, Jennings P. Incorporating human factors into scenario languages for automated driving systems. In: Proceedings of IEEE Intelligent Vehicles Symposium. 2024, 3009–3016

[194] Tang Y, da Costa A A B, Irvine P, Dodoiu T, Zhang Y, Zhao X, Khastgir S, Jennings P. Augmenting scenario description languages for intelligence testing of automated driving systems. In: Proceedings of IEEE Intelligent Vehicles Symposium. 2024, 1112–1118

[195] Gambi A, Nguyen V, Ahmed J, Fraser G. Generating critical driving scenarios from accident sketches. In: Proceedings of IEEE International Conference on Artificial Intelligence Testing. 2022, 95–102

[196] Klischat M, Althoff M. Generating critical test scenarios for automated vehicles with evolutionary algorithms. In: Proceedings of IEEE Intelligent Vehicles Symposium. 2019, 2352–2358

[197] Feng S, Sun H, Yan X, Zhu H, Zou Z, Shen S, Liu H X . Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 2023, 615( 7953): 620–627

[198] Almendros-Jiménez J M, Becerra-Terón A, Merayo M G, Núñez M . Using metamorphic testing to improve the quality of tags in OpenStreetMap. IEEE Transactions on Software Engineering, 2023, 49( 2): 549–563

[199] Song R, Ozmen M O, Kim H, Muller R, Celik Z B, Bianchi A. Discovering adversarial driving maneuvers against autonomous vehicles. In: Proceedings of USENIX Security Symposium. 2023, 2957–2974

[200] Menzel T, Bagschik G, Maurer M. Scenarios for development, test and validation of automated vehicles. In: Proceedings of IEEE Intelligent Vehicles Symposium. 2018, 1821–1827

[201] Majumdar R, Mathur A S, Pirron M, Stegner L, Zufferey D. Paracosm: a test framework for autonomous driving simulations. In: Proceedings of the 24th International Conference on Fundamental Approaches to Software Engineering (FASE). 2021, 172–195

[202] Najm W G, Smith J D, Yanagisawa M. Pre-crash scenario typology for crash avoidance research. DOT-VNTSC-NHTSA-06-02. Cambridge: Volpe National Transportation Systems Center, 2007

[203] Bühler O, Wegener J. Automatic testing of an autonomous parking system using evolutionary computation. SAE Technical Paper 2004-01-0459. Warrendale, PA: SAE International, 2004

[204] Buehler O, Wegener J. Evolutionary functional testing of a vehicle brake assistant system. In: Proceedings of the 6th Metaheuristics International Conference. 2005, 89–98

RIGHTS & PERMISSIONS

Higher Education Press