Deep learning for autonomous driving systems: technological innovations, strategic implementations, and business implications - a comprehensive review

Abstract

The rapid advancements in deep learning have significantly transformed the landscape of autonomous driving, with profound technological, strategic, and business implications. Autonomous driving systems, which rely on deep learning to enhance real-time perception, decision-making, and control, are poised to revolutionize transportation by improving safety, efficiency, and mobility. Despite this progress, numerous challenges remain, such as real-time data processing, decision-making under uncertainty, and navigating complex environments. This comprehensive review explores the state-of-the-art deep learning methodologies, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks, Long Short-Term Memory networks, and transformers that are central to autonomous driving tasks such as object detection, scene understanding, and path planning. Additionally, the review examines strategic implementations, focusing on the integration of deep learning into the automotive sector, the scalability of artificial intelligence-driven systems, and their alignment with regulatory and safety standards. Furthermore, the study highlights the business implications of deep learning adoption, including its influence on operational efficiency, competitive dynamics, and workforce requirements. The literature also identifies gaps, particularly in achieving full autonomy (Level 5), improving sensor fusion, and addressing the long-term costs and regulatory challenges. By addressing these issues, deep learning has the potential to redefine the future of mobility, enabling safer, more efficient, and fully autonomous driving systems. This review aims to provide insights for stakeholders, including automotive manufacturers, artificial intelligence developers, and policymakers, to navigate the complexities of integrating deep learning into autonomous driving.

Keywords

Deep learning / autonomous driving / object detection / Advanced Driver Assistance Systems (ADAS) / artificial intelligence in transportation / convolutional neural networks (CNNs)

References

[1] Singh S. Critical reasons for crash investigation in the national motor vehicle crash causation survey. (Traffic safety facts crash•stats. report No. DOT HS 812 506). Washington, DC: National Highway Traffic Safety Administration. 2018. Available from: https://crashstats.nhtsa.dot.gov/Api/Public/Publication/812506. [Last accessed on 12 Feb 2025]

[2] Lana I,Velez M.Road traffic forecasting: recent advances and new challenges.IEEE Intell Transport Syst Mag2018;10:93-109

[3] Crayton TJ.Autonomous vehicles: developing a public health research agenda to frame the future of transportation policy.J Transp Health2017;6:245-52

[4] Goldfain B,You C.AutoRally: an open platform for aggressive autonomous driving.IEEE Control Syst2019;39:26-55

[5] First internationally valid system approval for conditionally automated driving. Mercedes-Benz Group. Available from: https://group.mercedes-benz.com/innovation/product-innovation/autonomous-driving/system-approval-for-conditionally-automated-driving.html. [Last accessed on 12 Feb 2025]

[6] Khanum A,Yang C.Involvement of deep learning for vision sensor-based autonomous driving control: a review.IEEE Sensors J2023;23:15321-41

[7] Lecun Y,Bengio Y.Gradient-based learning applied to document recognition.Proc IEEE86:2278-324

[8] HUBEL DH.Shape and arrangement of columns in cat’s striate cortex.J Physiol1963;165:559-68 PMCID:PMC1359325

[9] Song JG.CNN-based object detection and distance prediction for autonomous driving using stereo images.Int J Automot Technol2023;24:773-86

[10] Lee D.End-to-end deep learning of lane detection and path prediction for real-time autonomous driving.SIViP2023;17:199-205

[11] Li Q,Wang Y. Hdmapnet: an online HD map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA); 2022 May 23-27. Philadelphia, PA, USA. IEEE; 2022. pp. 4628-34. Available from: https://ieeexplore.ieee.org/abstract/document/9812383. [Last accessed on 12 Feb 2025]

[12] Ghintab S.CNN-based visual localization for autonomous vehicles under different weather conditions.ETJ2022;41:1-12

[13] Hoque S,Maiti A,Arafat MY.Deep learning for 6D pose estimation of objects - a case study for autonomous driving.Expert Syst Appl2023;223:119838

[14] Yang K,Qiu S,Wei Z.Towards robust decision-making for autonomous driving on highway.IEEE Trans Veh Technol2023;72:11251-63

[15] He K,Dollár P.Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision; 2017. pp. 2961-9.

[16] Radwan N,Burgard W.Vlocnet++: deep multitask learning for semantic visual localization and odometry.IEEE Robot Autom Lett2018;3:4407-14

[17] Chen C,Kornhauser A.Deepdriving: learning affordance for direct perception in autonomous driving. In Proceedings of the IEEE international conference on computer vision; 2015. pp. 2722-30.

[18] Hu Y,Chen L. Planning-oriented autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2023. pp. 17853-62. Available from: https://openaccess.thecvf.com/content/CVPR2023/html/Hu_Planning-Oriented_Autonomous_Driving_CVPR_2023_paper.html. [Last accessed on 12 Feb 2025]

[19] Redmon J,Girshick R.You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016. pp. 779-88.

[20] Liu W,Erhan D.SSD: single shot MultiBox detector. In: Leibe B, Matas J, Sebe N, Welling M, editors. Computer Vision - ECCV 2016. Cham: Springer International Publishing; 2016. pp. 21-37.

[21] Law H.Deng J. CornerNet: detecting objects as paired keypoints. In: Proceedings of the European conference on computer vision (ECCV); 2018, pp. 734-50.

[22] Lin G,Shen C.RefineNet: multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017. pp. 1925-34.

[23] Ren S,Girshick R.Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.IEEE Trans Pattern Anal Mach Intell2017;39:1137-49

[24] Dai J,He K.R-FCN: object detection via region-based fully convolutional networks.Adv Neural Inf Process Syst2016;29:Available from: https://proceedings.neurips.cc/paper/2016/hash/577ef1154f3240ad5b9b413aa7346a1e. [Last accessed on 12 Feb 2025]

[25] Wang R,Xu Z.A real-time object detector for autonomous vehicles based on YOLOv4.Comput Intell Neurosci2021;2021:9218137

[26] Badrinarayanan V,Cipolla R.SegNet: a deep convolutional encoder-decoder architecture for image segmentation.IEEE Trans Pattern Anal Mach Intell2017;39:2481-95

[27] Valada A,Dhall A.Adapnet: adaptive semantic segmentation in adverse environmental conditions. In: 2017 IEEE International Conference on Robotics and Automation (ICRA); 2017 May 29-Jun 3; pp. 4644-51.

[28] Guan L.Instance segmentation model evaluation and rapid deployment for autonomous driving using domain differences.IEEE Trans Intell Transport Syst2023;24:4050-9

[29] Vora S,Helou B.PointPainting: sequential fusion for 3d object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2020. pp. 4604-12.

[30] Wang C,Zhu M. PointAugmenting: cross-modal augmentation for 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021. pp. 11794-803. Available from: https://openaccess.thecvf.com/content/CVPR2021/html/Wang_PointAugmenting_Cross-Modal_Augmentation_for_3D_Object_Detection_CVPR_2021_paper.html?utm_campaign=Akira%27s+Machine+Learning+News+++&utm_medium=email&utm_source=Revue+newsletter. [Last accessed on 12 Feb 2025]

[31] Yin T,Krähenbühl P.Multimodal virtual point 3d detection.Adv Neural Inf Process Syst2021;34:pp.16494-507Available from: https://proceedings.neurips.cc/paper/2021/hash/895daa408f494ad58006c47a30f51c1f. [Last accessed on 12 Feb 2025]

[32] Liu Z,Amini A.BEVFusion: multi-task multi-sensor fusion with unified bird’s-eye view representation. In: 2023 IEEE International Conference on Robotics and Automation (ICRA); 2023 29 May-2 Jun. London, United Kingdom. IEEE; 2023. pp. 2774-81.

[33] Liu Z,Cao Y.Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision; 2021. pp. 10012-22.

[34] Kendall A,Cipolla R.PoseNet: a convolutional network for real-time 6-dof camera relocalization. In: Proceedings of the IEEE international conference on computer vision; 2015. pp. 2938-46.

[35] Sarlin PE,Dymczyk M,Cadena C. Leveraging deep visual descriptors for hierarchical efficient localization. In: Conference on Robot Learning; 2018. pp. 456-65. Available from: https://proceedings.mlr.press/v87/sarlin18a.html. [Last accessed on 12 Feb 2025]

[36] Charroud A,Palade V.XDLL: explained deep learning LiDAR-based localization and mapping method for self-driving vehicles.Electronics2023;12:567

[37] Thrun S. Robotic mapping: a survey. 2002. Available from: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=cbe046f24f31aace8b61b36e41392a93225029e0. [Last accessed on 12 Feb 2025]

[38] Stojcheski J,Ulrich M,Gläser C.Self-supervised occupancy grid map completion for automated driving. In: 2023 IEEE Intelligent Vehicles Symposium (IV); 2023, pp. 1-7.

[39] Shalev-Shwartz S,Shashua A. Safe, multi-agent, reinforcement learning for autonomous driving. 2016. Available from: https://arxiv.org/abs/1610.03295. [Last accessed on 12 Feb 2025]

[40] Liao B. Maptrv2: an end-to-end framework for online vectorized hd map construction. 2024. Available from: https://arxiv.org/abs/2308.05736. [Last accessed on 18 Feb 2025]

[41] Liao B,Wang X. Maptr: structured modeling and learning for online vectorized hd map construction. 2022. Available from: https://arxiv.org/abs/2208.14437. [Last accessed on 12 Feb 2025]

[42] Varshney KR.Engineering safety in machine learning. In: 2016 Information Theory and Applications Workshop (ITA); 2016 Jan 31-Feb 5. La Jolla, CA, USA. IEEE; 2016. pp. 1-5.

[43] Amodei D,Steinhardt J,Schulman J. Concrete problems in AI safety. 2016. Available from: https://arxiv.org/abs/1606.06565. [Last accessed on 12 Feb 2025]

[44] Baheri A.Safe reinforcement learning with mixture density network, with application to autonomous driving.Results Control Optim2022;6:100095

[45] Luo Z,Xiang H.Road object detection for HD map: full-element survey, analysis and perspectives.ISPRS JPhotogramm Remote Sens2023;197:122-44

[46] Yuan T,Wang Y,Zhao H. StreamMapNet: streaming mapping network for vectorized online HD map construction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision; 2024, pp. 7356-65. Available from: https://openaccess.thecvf.com/content/WACV2024/html/Yuan_StreamMapNet_Streaming_Mapping_Network_for_Vectorized_Online_HD_Map_Construction_WACV_2024_paper.html. [Last accessed on 12 Feb 2025]

[47] Liu Y,Wang Y,Zhao H. Vectormapnet: end-to-end vectorized hd map learning. In: International Conference on Machine Learning; 2023. pp. 22352-69. Available from: https://proceedings.mlr.press/v202/liu23ax.html. [Last accessed on 12 Feb 2025]

[48] Wang S,Wang B.UrbanPose: a new benchmark for VRU pose estimation in urban traffic scenes. In: 2021 IEEE Intelligent Vehicles Symposium (IV); 2021 July 11-17. Nagoya, Japan. IEEE 2021. pp. 1537-44.

[49] Shao H,Chen R,Liu Y. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In: Proceedings of The 6th Conference on Robot Learning; 2023. Available from: https://proceedings.mlr.press/v205/shao23a.html. [Last accessed on 18 Feb 2025]

[50] Li Z,Wang S. Hydra-MDP: end-to-end multimodal planning with multi-target hydra-distillation. Available from: https://arxiv.org/abs/2406.06978. [Last accessed on 18 Feb 2025]

[51] R-Car-V3H: SoC optimized for automotive application in stereo front cameras | Renesas. 2022. Available from: https://www.renesas.com/us/en/products/automotive-products/automotive-system-chips-socs/r-car-v3h-system-chip-soc-designed-intelligent-camera-deep-learning-capabilities. [Last accessed on 12 Feb 2025]

[52] Caesar H,Lang AH.nuScenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2020. pp. 11621-31.

[53] Wilson B,Agarwal T. Argoverse 2: next generation datasets for self-driving perception and forecasting. 2023, Available from: https://api.semanticscholar.org/CorpusID:244906596. [Last accessed on 12 Feb 2025]

[54] Veiga A, Astakhova LV, Botha A, Herselman M. Defining organisational information security culture-perspectives from academia and industry.Comput Secur2020;92:101713