Enhancing autonomous vehicle safety in rain: a data centric approach for clear vision

i

This article has been updated. A Correction to this article was published on .

i

The Original Article was published on

i

The Original Article was published on

i

This article has been updated

i

This article was retracted on

i

This article has been updated

Abstract

Autonomous vehicles (AV) face significant challenges in navigating adverse weather, particularly rain, due to the visual impairment of camera-based systems. In this study, we leveraged contemporary deep learning techniques to mitigate these challenges, aiming to develop a vision model that processes live vehicle camera feeds to eliminate rain-induced visual hindrances, yielding visuals closely resembling clear, rain-free scenes. Using the Car Learning to Act (CARLA) simulation environment, we generated a comprehensive dataset of clear and rainy images for model training and testing. In our model, we employed a classic encoder-decoder architecture with skip connections and concatenation operations. It was trained using novel batching schemes designed to effectively distinguish high-frequency rain patterns from low-frequency scene features across successive image frames. To evaluate the model’s performance, we integrated it with a steering module that processes front-view images as input. The results demonstrated notable improvements in steering accuracy, underscoring the model’s potential to enhance navigation safety and reliability in rainy weather conditions.

Keywords

autonomous vehicles / vehicle safety / adverse weather navigation / deep learning / computer vision / image deraining / batching schemes / CARLA simulator / steering performance

CHANGE HISTORY

A Correction to this paper has been published:
https://doi.org

The Retraction Note was updated to reflect reception of statements of disagreement from all authors.

This article has been retracted.Please see the Retraction Notice for more detail:
https://doi.org

References

[1] P. Vincent, H. Larochelle, Y. Bengio, P. A. Manzagol, Extracting and composing robust features with denoising autoencoders, Proceedings of the 25th International Conference on Machine Learning, 2008, 1096-1103. https://doi.org/10.1145/1390156.1390294

[2] A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv: 1511.06434. https://arxiv.org/abs/1511.06434

[3] O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for biomedical image segmentation, In: Medical image computing and computer-assisted intervention-MICCAI 2015, Cham: Springer, 2015, 234-241. https://doi.org/10.1007/978-3-319-24574-4_28

[4] T. Karras, T. Aila, S. Laine, J. Lehtinen, Progressive growing of gans for improved quality, stability, and variation, arXiv: 1710.10196. https://doi.org/10.48550/arxiv.1710.10196

[5] A. Brock, J. Donahue, K. Simonyan, Large scale GAN training for high fidelity natural image synthesis, arXiv: 1809.11096. https://doi.org/10.48550/arXiv.1809.11096

[6] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, et al., Stackgan: text to photo-realistic image synthesis with stacked generative adversarial networks, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, 5907-5915.

[7] T. Karras, S. Laine, T. Aila, A style-based generator architecture for generative adversarial networks, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 4396-4405. https://doi.org/10.1109/CVPR.2019.00453

[8] W. Xu, C. Long, R. Wang, G. Wang, Drb-gan: a dynamic resblock generative adversarial network for artistic style transfer, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, 6383-6392.

[9] H. Zhen, Y. Shi, J. Yang, J. Vehni, Co-supervised learning paradigm with conditional generative adversarial networks for sample-efficient classification, Appl. Comput. Intell., 3 (2023), 13-26. https://doi.org/10.3934/aci.2023002

[10] S. Motamed, P. Rogalla, F. Khalvati, Data augmentation using generative adversarial networks (GANs) for GAN-based detection of Pneumonia and COVID-19 in chest X-ray images, Informatics in Medicine Unlocked, 27 (2021), 100779. https://doi.org/10.1016/j.imu.2021.100779

[11] A. Jadli, M. Hain, A. Chergui, A. Jaize, DCGAN-based data augmentation for document classification, Proceedings of IEEE 2nd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS), 2020, 1-5. https://doi.org/10.1109/icecocs50124.2020.9314379

[12] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, et al., Zero-shot text-to-image generation, Proceedings of the 38th International Conference on Machine Learning, 2021, 8821-8831.

[13] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, High-resolution image synthesis with latent diffusion models, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 10684-10695. https://doi.org/10.1109/CVPR52688.2022.01042

[14] B. Xia, Y. Zhang, S. Wang, Y. Wang, X. Wu, Y. Tian, et al., Diffir: efficient diffusion model for image restoration, Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), 2023, 13049-13059. https://doi.org/10.1109/ICCV51070.2023.01204

[15] D. Ren, W. Zuo, Q. Hu, P. Zhu, D. Meng, Progressive image deraining networks: a better and simpler baseline, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 3937-3946. https://doi.org/10.1109/CVPR.2019.00406

[16] M. Bojarski, D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, et al., End to end learning for self-driving cars, arXiv: 1604.07316. https://doi.org/10.48550/arXiv.1604.07316

[17] H. Zhang, V. M. Patel, Density-aware single image de-raining using a multi-stream dense network, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 695-704. https://doi.org/10.1109/CVPR.2018.00079

[18] W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, S. Yan, Deep joint rain detection and removal from a single image, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, 1357-1366. https://doi.org/10.1109/CVPR.2017.183

[19] X. Fu, J. Huang, X. Ding, Y. Liao, J. Paisley, Clearing the skies: a deep network architecture for single-image rain removal, IEEE Trans. Image Process., 26 (2017), 2944-2956. https://doi.org/10.1109/tip.2017.2691802

[20] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, et al., Backpropagation applied to handwritten zip code recognition, Neural Comput., 1 (1989), 541-551. https://doi.org/10.1162/neco.1989.1.4.541

[21] S. Zamir, A. Arora, S. Khan, M. Hayat, F. Khan, M. Yang, Restormer: efficient transformer for high-resolution image restoration, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 5728-5739. https://doi.org/10.1109/cvpr52688.2022.00564

[22] Y. Zhang, D. Li, X. Shi, D. He, K. Song, X. Wang, et al., Kbnet: kernel basis network for image restoration, arXiv: 2303.02881. https://doi.org/10.48550/arXiv.2303.02881

[23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al., Generative adversarial networks, Commun. ACM, 63 (2020), 139-144. https://doi.org/10.1145/3422622

[24] Y. Wei, Z. Zhang, Y. Wang, M. Xu, Y. Yang, S. Yan, et al., Deraincyclegan: rain attentive cyclegan for single image deraining and rainmaking, IEEE Trans. Image Process., 30 (2021), 4788-4801. https://doi.org/10.1109/TIP.2021.3074804

[25] H. Zhang, V. Sindagi, V. M. Patel, Image de-raining using a conditional generative adversarial network, IEEE Trans. Circ. Syst. Vid., 30 (2020), 3943-3956. https://doi.org/10.1109/tcsvt.2019.2920407

[26] C. Wang, C. Xu, C. Wang, D. Tao, Perceptual adversarial networks for image-to-image transformation, IEEE Trans. Image Process., 27 (2018), 4066-4079. https://doi.org/10.1109/TIP.2018.2836316

[27] P. Xiang, L. Wang, F. Wu, J. Cheng, M. Zhou, Single-image de-raining with feature-supervised generative adversarial network, IEEE Signal Proc. Let., 26 (2019), 650-654. https://doi.org/10.1109/LSP.2019.2903874

[28] Y. Ren, M. Nie, S. Li, C. Li, Single image de-raining via improved generative adversarial nets, Sensors, 20 (2020), 1591. https://doi.org/10.3390/s20061591

[29] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, V. Koltun, CARLA: an open urban driving simulator, arXiv: 1711.03938. https://doi.org/10.48550/arXiv.1711.03938

[30] S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift, Proceedings of the 32nd International Conference on International Conference on Machine Learning, 2015, 448-456.

[31] J. Springenberg, A. Dosovitskiy, T. Brox, M. A. Riedmiller, Striving for simplicity: the all convolutional net, arXiv: 1412.6806. https://doi.org/10.48550/arxiv.1412.6806

[32] D. Kingma, J. Ba, Adam: a method for stochastic optimization, arXiv: 1412.6980. https://doi.org/10.48550/arXiv.1412.6980

[33] J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020, 6840-6851.