AI-Powered Anomaly Detection for Secure Internet of Things (IoT): Optimising XGBoost and Deep Learning With Bayesian Optimisation

Abstract

Intelligent and adaptive defence systems that can quickly thwart changing cyberthreats are becoming more and more necessary in the dynamic and data-intensive Internet of things (IoT) environment. Using the NSL-KDD benchmark dataset, this paper presents an improved anomaly detection system that combines an optimised sequential neural network (OSNN) with an XGBoost model optimised using Bayesian approaches. Key drawbacks of conventional intrusion detection systems (IDSs), including manual hyperparameter tuning, extensive preprocessing and ineffective optimisation techniques, are successfully addressed by the suggested solution. By automating the adjustment of key parameters, such as learning rate, tree depth and regularisation strength, Bayesian optimisation enhances model convergence, prediction accuracy and generalisation. Superior binary classification performance was attained by the optimised XGBoost, which had 99.98% accuracy, 99.94% F1 score and 99.95% MCC. In a similar vein, the Bayesian-optimised OSNN obtained flawless precision and recall for the normal and U2R classes, achieving 99.92% overall accuracy in multiclass identification. Across all attack types, its CNN-based architecture showed excellent sensitivity and specificity, attaining almost flawless AUC values. The suggested Bayesian-optimised framework offers a highly accurate, scalable and self-adaptive intrusion detection solution for actual IoT network environments, outperforming traditional machine and deep learning models by a large margin overall.

Keywords

classification / computational complexity / deep neural networks / forensic science

Acknowledgements

The authors extend their appreciation to the Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia for funding this research work through the project number MoE-IF-UJ-R2-22-20211-1.

Funding

The study was supported by Research and Innovation, Ministry of Education in Saudi Arabia MoE-IF-UJ-R2-22-20211-1.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

Publicly available dataset is utilised.

References

[1] V. P. Gandi, N. S. L. Jatla, G. Sadhineni, S. Geddamuri, G. K. Chaithanya, and A. K. Velmurugan, “A Comparative Study of AI Algorithms for Anomaly-Based Intrusion Detection,” in 2023 7th International Conference on Computing Methodologies and Communication (IEEE, 2023), 530-534.

[2] A. Ali, A. W. Septyanto, I. Chaudhary, H. Al Hamadi, H. M. Alzoubi, and Z. F. Khan, “Applied Artificial Intelligence as Event Horizon of Cyber Security,” in 19th International Conference on Smart Business Technologies. (IEEE, 2022), 1-7.

[3] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash, “Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications,” IEEE Communications Surveys & Tutorials 17, no. 4 (2015): 2347-2376, https://doi.org/10.1109/comst.2015.2444095.

[4] I. Makhdoom, M. Abolhasan, J. Lipman, R. P. Liu, and W. Ni, “Anatomy of Threats to the Internet of Things,” IEEE Communications Surveys & Tutorials 21, no. 2 (2019): 1636-1675, https://doi.org/10.1109/comst.2018.2874978.

[5] B. B. Zarpelão, R. S. Miani, C. T. Kawakani, and S. C. de Alvarenga, “A Survey of Intrusion Detection in Internet of Things,” Journal of Network and Computer Applications 84 (2017): 25-37, https://doi.org/10.1016/j.jnca.2017.02.009.

[6] I. Stellios, P. Kotzanikolaou, M. Psarakis, C. Alcaraz, and J. Lopez, “A Survey of IoT-Enabled Cyberattacks: Assessing Attack Paths to Critical Infrastructures and Services,” IEEE Communications Surveys & Tutorials 20, no. 4 (2018): 3453-3495, https://doi.org/10.1109/comst.2018.2855563.

[7] M. Hu, X. Feng, Z. Ji, K. Yan, and S. Zhou, “A Novel Computational Approach for Discord Search With Local Recurrence Rates in Multivariate Time Series,” Information Scientist 477 (2019): 220-233, https://doi.org/10.1016/j.ins.2018.10.047.

[8] Z. Ji, Y. Wang, K. Yan, X. Xie, Y. Xiang, and J. Huang, “A Space-Embedding Strategy for Anomaly Detection in Multivariate Time Series,” Expert Systems With Applications 206 (2022): 117892, https://doi.org/10.1016/j.eswa.2022.117892.

[9] M. Zheng, J. Man, D. Wang, Y. Chen, Q. Li, and Y. Liu, “Semi-Supervised Multivariate Time Series Anomaly Detection for Wind Turbines Using Generator SCADA Data,” Reliability Engineering & System Safety 235 (2023): 109235, https://doi.org/10.1016/j.ress.2023.109235.

[10] Y. Lian, Y. Geng, and T. Tian, “Anomaly Detection Method for Multivariate Time Series Data of Oil and Gas Stations Based on Digital Twin and MTAD-GAN,” Applied Sciences 13, no. 3 (2023): 1891, https://doi.org/10.3390/app13031891.

[11] C. Tang, L. Xu, B. Yang, Y. Tang, and D. Zhao, “GRU-Based Interpretable Multivariate Time Series Anomaly Detection in Industrial Control System,” Computers & Security 127 (2023): 103094, https://doi.org/10.1016/j.cose.2023.103094.

[12] T. T. Khoei, W. C. Hu, and N. Kaabouch, “Residual Convolutional Network for Detecting Attacks on Intrusion Detection Systems in Smart Grid,” in 2022 IEEE International Conference on Electro Information Technology. (IEEE, 2022), 231-237.

[13] A. Shaikh and P. Gupta, “Real-Time Intrusion Detection Based on Residual Learning Through Resnet Algorithm,” International Journal of System Assurance Engineering and Management 2022 (2022): 1-15, https://doi.org/10.1007/s13198-021-01558-1.

[14] O. Alkadi, N. Moustafa, B. Turnbull, and K.-K. R. Choo, “A Deep Blockchain Framework-Enabled Collaborative Intrusion Detection for Protecting Iot and Cloud Networks,” IEEE Internet of Things Journal 8, no. 12 (2021): 9463-9472, https://doi.org/10.1109/jiot.2020.2996590.

[15] A. K. Silivery, R. M. Rao Kovvur, R. Solleti, L. S. Kumar, and B. Madhu , “A Model for Multi-Attack Classification to Improve Intrusion Detection Performance Using Deep Learning Approaches,” Meas Sensors 30 (2023): 100924, https://doi.org/10.1016/j.measen.2023.100924.

[16] D. Yin, Z. Wu, K. Yokota, K. Matsumoto, and S. Shibayama, “Identify Novel Elements of Knowledge With Word Embedding,” PLoS One 18, no. 6 (2023): e0284567, https://doi.org/10.1371/journal.pone.0284567.

[17] S. S. Bamber, A. V. R. Katkuri, S. Sharma, and M. Angurala , “A Hybrid CNN-LSTM Approach for Intelligent Cyber Intrusion Detection System,” Computers & Security 148 (2025): 104146, https://doi.org/10.1016/j.cose.2024.104146.

[18] R. Amin, G. El-Taweel, A. F. Ali, and M. Tahoun, “A Hybrid Extreme Gradient Boosting and Long Short-Term Memory Algorithm for Cyber Threats Detection,” MENDEL 29, no. 2 (2023): 307-322, https://doi.org/10.13164/mendel.2023.2.307.

[19] S. Sharma, Y. Gigras, R. Chhikara, and A. Dhull, “Analysis of NSL KDD Dataset Using Classification Algorithms for Intrusion Detection System,” Recent Patents on Engineering 13, no. 2 (2019): 142-147, https://doi.org/10.2174/1872212112666180402122150.

[20] N. K. Gyimah, J. Mwakalonge, G. Comert, et al., An AutoML-based Approach for Network Intrusion Detection,” in SoutheastCon 2025 (IEEE, 2025).

[21] Y. Xiao and X. Xiao, “An Intrusion Detection System Based on a Simplified Residual Network,” Information 10, no. 11 (2019): 356, https://doi.org/10.3390/info10110356.

[22] E. Jaw and X. Wang, “Feature Selection and Ensemble-Based Intrusion Detection System: An Efficient and Comprehensive Approach,” Symmetry 13, no. 10 (2021): 1764, https://doi.org/10.3390/sym13101764.

[23] D. Upadhyay, J. Manero, M. Zaman, and S. Sampalli, “Gradient Boosting Feature Selection With Machine Learning Classifiers for Intrusion Detection on Power Grids,” IEEE Transactions on Network and Service Management 18, no. 1 (2021): 1104-1116, https://doi.org/10.1109/tnsm.2020.3032618.

[24] X. Ding, C. Feng, P. Yu, K. Li, and X. Chen, “Gradient Boosting Decision Tree in the Prediction of NOx Emission of Waste Incineration,” Energy 264 (2023): 126174, https://doi.org/10.1016/j.energy.2022.126174.

[25] M. E. Aguilar and N. Antonio, “Short-Term Electricity Load Forecasting With Machine Learning,” Information 12, no. 2 (2021): 50, https://doi.org/10.3390/info12020050.

[26] S. Ahmed, B. Raza, L. Hussain, et al., The Deep Learning ResNet101 and Ensemble XGBoost Algorithm With Hyperparameters Optimization Accurately Predict the Lung Cancer,” Applied Artificial Intelligence 37, no. 1 (2023): 2166222, https://doi.org/10.1080/08839514.2023.2166222.

[27] K. Budholiya, S. K. Shrivastava, and V. Sharma, “An Optimized XGBoost Based Diagnostic System for Effective Prediction of Heart Disease,” Journal of King Saud University-Computer and Information Sciences 34, no. 7 (2022): 4514-4523, https://doi.org/10.1016/j.jksuci.2020.10.013.

[28] A. Hjort, J. Pensar, I. Scheel, and D. E. Sommervoll, “House Price Prediction With Gradient Boosted Trees Under Different Loss Functions,” Journal of Property Research 39, no. 4 (2022): 338-364, https://doi.org/10.1080/09599916.2022.2070525.

[29] Y. Qiu, J. Zhou, M. Khandelwal, H. Yang, P. Yang, and C. Li, “Performance Evaluation of Hybrid WOA-XGBoost, GWO-XGBoost and BO-XGBoost Models to Predict Blast-Induced Ground Vibration,” supplement, Engineering Computers 38, no. S5 (2022): 4145-4162, https://doi.org/10.1007/s00366-021-01393-9.

[30] I. Ullah, A. Rios, V. Gala, and S. Mckeever, “Explaining Deep Learning Models for Tabular Data Using Layer-Wise Relevance Propagation,” Applied Sciences 12, no. 1 (2021): 136, https://doi.org/10.3390/app12010136.

[31] N. Divyashree and K. Nandini Prasad, “Improved Clinical Diagnosis Using Predictive Analytics,” IEEE Access 10 (2022): 75158-75175, https://doi.org/10.1109/ACCESS.2022.3190416.

[32] C. V. Priscilla and D. P. Prabha, “Influence of Optimizing XGBoost to Handle Class Imbalance in Credit Card Fraud Detection,” in 2020 Third International Conference on Smart Systems and Inventive Technology (IEEE, 2020), 1309-1315.

[33] T. Chen and C. Guestrin, “XGBoost,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Min (ACM, 2016), 785-794.

[34] B. Kaur, A. Chaudhary, S. Bano, R. Yashmita, S. R. N. Reddy, and R. Anand, “Fostering Inclusivity Through Effective Communication: Real-Time Sign Language to Speech Conversion System for the Deaf and Hard-of-Hearing Community,” Multimedia Tools and Applications 83, no. 15 (2024): 45859-45880, https://doi.org/10.1007/s11042-023-17372-9.

[35] S. Pan, Z. Zheng, Z. Guo, and H. Luo, “An Optimized XGBoost Method for Predicting Reservoir Porosity Using Petrophysical Logs,” Journal of Petroleum Science & Engineering 208 (2022): 109520, https://doi.org/10.1016/j.petrol.2021.109520.

[36] L. Bloch and C. M. Friedrich, “Using Bayesian Optimization to Effectively Tune Random Forest and XGBoost Hyperparameters for Early Alzheimer’s Disease Diagnosis,” in International Conference on Wireless Mobile Communication and Healthcare (Springer International Publishing, 2021), 285-299.

[37] J. Bect, D. Ginsbourger, L. Li, V. Picheny, and E. Vazquez, “Sequential Design of Computer Experiments for the Estimation of a Probability of Failure,” Statistics and Computing 22, no. 3 (2012): 773-793, https://doi.org/10.1007/s11222-011-9241-4.

[38] T. L. Lau, “Tsang EPK the Guided Genetic Algorithm and Its Application to the Generalized Assignment Problem,” in Proceedings of the Tenth IEEE International Conference on Tools with Artificial Intelligence (Cat. No.98CH36294) (IEEE, 1998), 336-343.

[39] Z. Sun, G. G. Yen, J. Wu, H. Ren, H. An, and J. Yang, “Mission Planning for Energy-Efficient Passive UAV Radar Imaging System Based on Substage Division Collaborative Search,” IEEE Transactions on Cybernetics 53, no. 1 (2023): 275-288, https://doi.org/10.1109/tcyb.2021.3090662.

[40] S. Hosseini, “A New Machine Learning Method Consisting of GA-LR and ANN for Attack Detection,” Wireless Networks 26, no. 6 (2020): 4149-4162, https://doi.org/10.1007/s11276-020-02321-3.

[41] X. Gao, C. Shan, C. Hu, Z. Niu, and Z. Liu, “An Adaptive Ensemble Machine Learning Model for Intrusion Detection,” IEEE Access 7 (2019): 82512-82521, https://doi.org/10.1109/access.2019.2923640.

[42] T. Su, H. Sun, J. Zhu, S. Wang, and Y. Li, “BAT: Deep Learning Methods on Network Intrusion Detection Using NSL-KDD Dataset,” IEEE Access 8 (2020): 29575-29585, https://doi.org/10.1109/access.2020.2972627.

[43] T. A. Tang, L. Mhamdi, D. McLernon, S. A. R. Zaidi, and M. Ghogho, “Deep Learning Approach for Network Intrusion Detection in Software Defined Networking,” in 2016 International Conference on Wireless Networks and Mobile Communications (IEEE, 2016), 258-263.

[44] H. Bostani and M. Sheikhan, “Hybrid of Binary Gravitational Search Algorithm and Mutual Information for Feature Selection in Intrusion Detection Systems,” Soft Computing 21, no. 9 (2017): 2307-2324, https://doi.org/10.1007/s00500-015-1942-8.

[45] C. Zhang, F. Ruan, L. Yin, X. Chen, L. Zhai, and F. Liu, “A Deep Learning Approach for Network Intrusion Detection Based on NSL-KDD Dataset,” in 2019 IEEE 13th International Conference on Anti-counterfeiting, Security, and Identification (IEEE, 2019), 41-45.

[46] Z. Hu, L. Wang, L. Qi, Y. Li, and W. Yang, “A Novel Wireless Network Intrusion Detection Method Based on Adaptive Synthetic Sampling and an Improved Convolutional Neural Network,” IEEE Access 8 (2020): 195741-195751, https://doi.org/10.1109/access.2020.3034015.

[47] G. Pardeshi N and V. Patil D , “ Binary and Multiclass Classification Intrusion Detection System Using Benchmark NSL-KDD and Machine Learning Models,” in 2024 International Conference on Data Science and Network Security (IEEE, 2024), 1-7.

[48] N. Iftikhar, M. U. Rehman, M. A. Shah, M. J. F. Alenazi, and J. Ali, “Intrusion Detection in NSL-KDD Dataset Using Hybrid Self-Organizing Map Model,” Computer Modeling in Engineering and Sciences 143, no. 1 (2025): 639-671, https://doi.org/10.32604/cmes.2025.062788.

[49] P. Mehrotra and U. Dwivedi, “An Efficient Deep Learning Framework for Classification and Detection of Anomaly Based Network Intrusion Using NSL-KDD Dataset,” in 2025 3rd International Conference on Disruptive Technologies (ICDT) (IEEE, 2025), 170-177.

[50] Z. Hassan, W. A. Bukhari, A. Sikander, U. Bashir, and H. Malik, “Intrusion Detection and Classification Using Deep Belief Networks With Feature Reduction,” Cluster Computing 28, no. 14 (2025): 890, https://doi.org/10.1007/s10586-025-05659-5.

[51] F. H. Almasoudy, W. L. Al-Yaseen, and A. K. Idrees, “Differential Evolution Wrapper Feature Selection for Intrusion Detection System,” Procedia Computer Science 167 (2020): 1230-1239, https://doi.org/10.1016/j.procs.2020.03.438.

[52] O. A. Sarumi, A. O. Adetunmbi, and F. A. Adetoye, “Discovering Computer Networks Intrusion Using Data Analytics and Machine Intelligence,” Scientific African 9 (2020): e00500, https://doi.org/10.1016/j.sciaf.2020.e00500.