Event-level abnormal driving behavior detection in mining trucks based on temporal-aware vision-language models

Abstract

Unsafe driving behaviors significantly contribute to accidents in open-pit mining operations. Conventional monitoring systems often fail to capture the temporal continuity and semantic ambiguity of such behaviors. To address these challenges, an event-level abnormal driving behavior recognition method based on a vision-language model (VLM) is proposed. The method integrates Temporal-Aware Low-Rank Adaptation (T-LoRA) with event-level supervised fine-tuning to enable efficient vertical-domain customization of the VLM toward complex open-pit operating environments. By explicitly enhancing temporal modeling of visual sequences, the proposed method enables accurate event-level recognition of abnormal driving behaviors, precise localization of their onset and duration, and the generation of interpretable semantic descriptions. Experiments conducted on a real-world mining truck dataset demonstrate that the proposed approach achieves an event-level F1-score of 0.923, while maintaining a mean absolute error for temporal localization below 0.8 s. These results indicate that the proposed method effectively captures continuous behavioral patterns and improves both recognition accuracy and temporal precision, providing a reliable solution for intelligent safety monitoring. Furthermore, the system supports proactive intervention through event-level early warnings, contributing to safer and more efficient mining operations.

Keywords

intelligent mining / mining large models / abnormal driving behavior detection / mining truck safety / intelligent driver monitoring

References

[1] Chen Y, Lu CW, Tian SC, et al. . Monitoring and detecting coal miners’ fatigue status using MPA-LSSVM in the vision of smart mine. Process. Saf. Environ. Prot., 2023, 179774

[2] Chan TK, Chin CS, Chen H, Zhong XH. A comprehensive review of driver behavior analysis utilizing smart-phones. IEEE Trans. Intell. Transp. Syst., 2020, 21104444

[3] Nasr Azadani M, Boukerche A. Driving behavior analysis guidelines for intelligent transportation systems. IEEE Trans. Intell. Transp. Syst., 2022, 2376027

[4] Zhu MK, Chen JF, Li HB, Liang FJ, Han L, Zhang ZT. Vehicle driver drowsiness detection method using wearable EEG based on convolution neural network. Neural Comput. Appl., 2021, 332013965

[5] Wang GF, Xu YX, Ren HW. Intelligent and ecological coal mining as well as clean utilization technology in China: Review and prospects. Int. J. Min. Sci. Technol., 2019, 292161

[6] E. Michelaraki, C. Katrakazas, S. Kaiser, T. Brijs, and G. Yannis, Real-time monitoring of driver distraction: State-of-the-art and future insights, Accid. Anal. Prev., 192(2023), art. No. 107241.

[7] M. Awais, N. Badruddin, and M. Drieberg, A hybrid approach to detect driver drowsiness utilizing physiological signals to improve system performance and wearability, Sensors, 17(2017), No. 9, art. No. 1991.

[8] Wang AA, Cao S, Yilmaz E, Cao H. Research on the visualization method of lithology intelligent recognition based on deep learning using mine tunnel images. Int. J. Miner. Metall. Mater., 2026, 331141

[9] Nakano K, Chakraborty B. Real-time distraction detection from driving data based personal driving model using deep learning. Int. J. Intell. Transp. Syst. Res., 2022, 201238

[10] S.K. Chen, L.Y. Pan, K.L. Xu, et al., Real-time monitoring and prediction of remote operator fatigue in plateau deep mining based on dynamic Bayesian networks, Sci. Rep., 15(2025), art. No. 1063.

[11] Jiang XD, Zhong SH, Liang JL. Simulation of unsafe behavior in mine operation based on the SMAPP model. Processes, 2023, 1161732

[12] Seo H, Shin J, Kim KH, Lim C, Bae J. Driving risk assessment using non-negative matrix factorization with driving behavior records. IEEE Trans. Intell. Transp. Syst., 2022, 231120398

[13] M.I.B. Ahmed, H. Alabdulkarem, F. Alomair, et al., A deep-learning approach to driver drowsiness detection, Safety, 9(2023), No. 3, art. No. 65.

[14] Wang JY, Chai WH, Venkatachalapathy A, et al. . A survey on driver behavior analysis from in-vehicle cameras. IEEE Trans. Intell. Transp. Syst., 2022, 23810186

[15] Abbas Q, Ibrahim MEA, Khan S, Baig AR. Hypo-driver: A multiview driver fatigue and distraction level detection system. Comput. Mater. Continua, 2022, 7111999

[16] Kong LJ, Xie K, Niu KX, He JB, Zhang W. Remote photoplethysmography and motion tracking convolutional neural network with bidirectional long short-term memory: Non-invasive fatigue detection method based on multi-modal fusion. Sensors, 2024, 242455

[17] F.M. Qu, N. Dang, B. Furht, and M. Nojoumian, Comprehensive study of driver behavior monitoring systems using computer vision and machine learning techniques, J. Big Data, 11(2024), No. 1, art. No. 32.

[18] M.U. Hossain, M.A. Rahman, M.M. Islam, A. Akhter, M.A. Uddin, and B.K. Paul, Automatic driver distraction detection using deep convolutional neural networks, Intell. Syst. Appl., 14(2022), art. No. 200075.

[19] M.H. Lv, Y.J. Liu, Z.H. Zha, et al., Si-CA MobileNet: A lightweight and efficient convolutional neural network for distracted driver detection, Neurocomputing, 654(2025), art. No. 131281.

[20] Yang WX, Tan CH, Chen YX, et al. . BiRSwinT: Bilinear full-scale residual swin-transformer for fine-grained driver behavior recognition. J. Franklin Inst., 2023, 36021166

[21] Zhang JY, Huang JX, Jin S, Lu SJ. Vision-language models for vision tasks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 2024, 4685625

[22] Zhao R, Yuan QR, Li JY, Fan YZ, Li Y, Gao F. DriveLLaVA: Human-level behavior decisions via vision language model. Sensors, 2024, 24134113

[23] Cao YK, Xu XH, Cheng YQ, et al. . Personalizing vision-language models with hybrid prompts for zero-shot anomaly detection. IEEE Trans. Cybern., 2025, 5541917

[24] Hasan MZ, Chen JJ, Wang JY, et al. . Vision-language models can identify distracted driver behavior from naturalistic videos. IEEE Trans. Intell. Transp. Syst., 2024, 25911602

[25] C.J. Fan, S.F. Huang, S.X. Lin, D.Y. Xu, Y. Peng, and S.G. Yi, Types, risk factors, consequences, and detection methods of train driver fatigue and distraction, Comput. Intell. Neurosci., 2022(2022), art. No. 8328077.

[26] Li RJ, Yu CD, Qin XR, et al. . YOLO-SGC: A dangerous driving behavior detection method with multiscale spatial-channel feature aggregation. IEEE Sens. J., 2024, 242136044

[27] N. Moslemi, M. Soryani, and R. Azmi, Computer vision-based recognition of driver distraction: A review, Concurr. Comput. Pract. Exp., 33(2021), No. 24, art. No. e6475.

[28] L. Li, B.X. Zhong, C. Hutmacher Jr, Y.L. Liang, W.J. Horrey, and X. Xu, Detection of driver manual distraction via image-based hand and ear recognition, Accid. Anal. Prev., 137(2020), art. No. 105432.

[29] H. Yadav and A. Thakkar, NOA-LSTM: An efficient LSTM cell architecture for time series forecasting, Expert Syst. Appl., 238(2024), art. No. 122333.

[30] Florez R, Palomino-Quispe F, Coaquira-Castillo RJ, Herrera-Levano JC, Paixão T, Alvarez AB. A CNN-based approach for driver drowsiness detection by real-time eye state identification. Appl. Sci., 2023, 13137849

[31] Radford A, Kim JW, Hallacy C, et al. . Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, 2021, 1398748

[32] Hu E, Shen YL, Wallis P, et al. . LoRA: Low-rank adaptation of large language models. The Tenth International Conference on Learning Representations (ICLR 2022). 2022

[33] Dong G, Yuan H, Lu K, et al. . How abilities in large language models are affected by supervised fine-tuning data composition. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024177

[34] Y.W. Zheng, R.C. Zhang, J.H. Zhang, et al., LlamaFactory: Unified efficient fine-tuning of 100+ language models, 2024 [2025-05-14], arXiv: 2403.13372.

[35] A. Yang, A.F. Li, B.S. Yang, et al., Qwen3 technical report, 2025 [2025-05-14], arXiv: 2505.09388.

[36] J. Li, S.H. Xie, X.Y. Zhou, L. Zhang, and X.G. Li, Real-time detection of coal mine safety helmet based on improved YOLOv8, J. Real Time Image Process., 22(2024), No. 1, art. No. 26.

[37] S. Bai, K.Q. Chen, X.J. Liu, et al., Qwen2.5-VL technical report, 2025 [2025-02-19], arXiv: 2502.13923.

RIGHTS & PERMISSIONS

University of Science and Technology Beijing