ExplainableDetector: Exploring transformer-based language modeling approach for SMS spam detection with explainability analysis^✩

Abstract

Short Message Service (SMS) is a widely used and cost-effective communication medium that has unfortunately become a frequent target for unsolicited messages - commonly known as SMS spam. With the rapid adoption of smartphones and increased Internet connectivity, SMS spam has emerged as a prevalent threat. Spammers have recognized the critical role SMS plays in today’s modern communication, making it a prime target for abuse. As cybersecurity threats continue to evolve, the volume of SMS spam has increased substantially in recent years. Moreover, the unstructured format of SMS data creates significant challenges for SMS spam detection, making it more difficult to successfully combat spam attacks. In this paper, we present an optimized and fine-tuned transformer-based Language Model to address the problem of SMS spam detection. We use a benchmark SMS spam dataset to analyze this spam detection model. Additionally, we utilize pre-processing techniques to obtain clean and noise-free data and address class imbalance problem by leveraging text augmentation techniques. The overall experiment showed that our optimized fine-tuned BERT (Bidirectional Encoder Representations from Transformers) variant model RoBERTa obtained high accuracy with 99.84%. To further enhance model transparency, we incorporate Explainable Artificial Intelligence (XAI) techniques that compute positive and negative coefficient scores, offering insight into the model’s decision-making process. Additionally, we evaluate the performance of traditional machine learning models as a baseline for comparison. This comprehensive analysis demonstrates the significant impact language models can have on addressing complex text-based challenges within the cybersecurity landscape.

Keywords

Cybersecurity / Machine learning / Large language model / Spam detection / Text analytics / Explainable AI / Fine-tuning / Transformer

References

[1] X. Liu, H. Lu, A. Nayak, A spam transformer model for sms spam detection, IEEE Access 9 (2021) 80253-80263.

[2] C. Oswald, S.E. Simon, A. Bhattacharya, Spotspam: intention analysis-driven sms spam detection using bert embeddings, ACM Trans. Web 16 (3) (2022) 1-27.

[3] S.M.M. Hossain, A. Sen, K. Deb, Detecting spam sms using self attention mechanism, in: International Conference on Intelligent Computing & Optimization, Springer, 2022, pp. 175-184.

[4] F. Wei, T. Nguyen, A lightweight deep neural model for sms spam detection, in: 2020 International Symposium on Networks, Computers and Communications (ISNCC), IEEE, 2020, pp. 1-6.

[5] I.H. Sarker, AI-Driven Cybersecurity and Threat Intelligence:Cyber Automation, In- telligent Decision-Making and Explainability, Springer Nature, 2024.

[6] W.X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al., A survey of large language models, arXiv preprint arXiv:2303.18223.

[7] I.H. Sarker, Llm potentiality and awareness: a position paper from the perspective of trustworthy and responsible ai modeling, Discov. Artif. Intell. 4 (1) (2024) 40.

[8] Y. Yao, J. Duan, K. Xu, Y. Cai, E. Sun, Y. Zhang, A survey on large language model (llm) security and privacy: the good, the bad, and the ugly, arXiv preprint arXiv: 2312.02003.

[9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Adv. Neural Inf. Process. Syst. 30 (2017).

[10] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, Y. Wang, Transformer in transformer, Adv. Neural Inf. Process. Syst. 34 (2021) 15908-15919.

[11] S. Jamal, H. Wimmer,I. Sarker, An improved transformer-based model for detecting phishing, spam, and ham: a large language model approach, arXiv preprint arXiv: 2311.04913.

[12] M. Koroteev,Bert: a review of applications in natural language processing and un- derstanding, arXiv preprint arXiv:2103.11943.

[13] A.H. Mohammed, A.H. Ali, Survey of bert (bidirectional encoder representation transformer) types, J. Phys. Conf. Ser. 1963 ( 2021) 012173.

[14] J. Devlin, M.-W. Chang, K. Lee,K. Toutanova, Bert: pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805.

[15] R. Anil, B. Ghazi, V. Gupta, R. Kumar, P. Manurangsi, Large-scale differentially pri- vate bert, arXiv preprint arXiv:2108.01624.

[16] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma,R. Soricut, Albert: a lite bert for self-supervised learning of language representations, arXiv preprint arXiv:1909. 11942.

[17] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer,V. Stoyanov, Roberta: a robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692.

[18] V. Sanh, L. Debut, J. Chaumond,T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108.

[19] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang,Q. Liu, Tinybert: distilling bert for natural language understanding, arXiv preprint arXiv:1909.10351.

[20] F. Xu, H. Uszkoreit, Y. Du, W. Fan, D. Zhao, J. Zhu, Explainable ai: a brief survey on history, research areas, approaches and challenges,in: Natural Language Processing and Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9-14, 2019, Proceedings, Springer, 2019, pp. 563-574.

[21] P. Linardatos, V. Papastefanopoulos, S. Kotsiantis, Explainable ai: a review of ma- chine learning interpretability methods, Entropy 23 (1) (2020) 18.

[22] I.H. Sarker, H. Janicke, A. Mohsin, A. Gill, L. Maglaras,Explainable ai for cyberse- curity automation, intelligence and trustworthiness in digital twin: methods, taxon- omy, challenges and prospects, ICT Express.

[23] M.A. Abid, S. Ullah, M.A. Siddique, M.F. Mushtaq, W. Aljedaani, F. Rustam, Spam sms filtering based on text features and supervised machine learning techniques, Multimed. Tools Appl. 81 (28) (2022) 39853-39871.

[24] S. Mishra, D. Soni, Dsmishsms-a system to detect smishing sms, Neural Comput. Appl. 35 (7) (2023) 4975-4992.

[25] G. Sonowal, Detecting phishing sms based on multiple correlation algorithms, SN Comput. Sci. 1 (6) (2020) 361.

[26] S.M. Nagare, P.P. Dapke, S.A. Quadri, S.B. Bandal, M.R. Baheti, Short message service (sms) mobile spam detection using naïve Bayes, in: 2024 5th International Con- ference on Mobile Computing and Sustainable Informatics (ICMCSI), IEEE, 2024, pp. 67-70.

[27] S. Mishra, D. Soni, Smishing detector: a security model to detect smishing through sms content analysis and url behavior analysis, Future Gener. Comput. Syst. 108 (2020) 803-815.

[28] J.W. Joo, S.Y. Moon, S. Singh, J.H. Park, S-detector: an enhanced security model for detecting smishing attack for mobile computing, Telecommun. Syst. 66 (2017) 29-38.

[29] T. Xia, X. Chen, A discrete hidden Markov model for sms spam detection, Appl. Sci. 10 (14) (2020) 5011.

[30] A. Ghourabi, M.A. Mahmood, Q.M. Alzubi, A hybrid cnn-lstm model for sms spam detection in Arabic and English messages, Future Internet 12 (9) (2020) 156.

[31] P.K. Roy, J.P. Singh, S. Banerjee, Deep learning to filter sms spam, Future Gener. Comput. Syst. 102 (2020) 524-533.

[32] O. Abayomi-Alli, S. Misra, A. Abayomi-Alli, A deep learning method for automatic sms spam classification: performance of learning algorithms on indigenous dataset, Concurr. Comput., Pract. Exp. 34 (17) (2022) e6989.

[33] T. Sahmoud, D.M. Mikki, Spam detection using bert, arXiv preprint arXiv:2206. 02443.

[34] V.S. Tida, S. Hsu, Universal spam detection using transfer learning of bert model, arXiv preprint arXiv:2202.03480.

[35] T. Chen, C. Guestrin, Xgboost: a scalable tree boosting system,in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 785-794.

[36] L. Wang, Support Vector Machines: Theory and Applications, vol. 177, Springer Sci- ence & Business Media, 2005.

[37] G. Guo, H. Wang, D. Bell, Y. Bi, K. Greer, Knn model-based approach in classifi- cation, in: On the Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003. Proceedings, Springer, 2003, pp. 986-996.

[38] L. Breiman, Random forests, Mach. Learn. 45 (2001) 5-32.

[39] A.F. Adoma, N.-M. Henry, W. Chen, Comparative analyses of bert roberta, distilbert, and xlnet for text-based emotion recognition, in: 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (IC- CWAMTIP), IEEE, 2020, pp. 117-121.

[40] K.L. Tan, C.P. Lee, K.S.M. Anbananthen, K.M. Lim, Roberta-lstm: a hybrid model for sentiment analysis with transformer and recurrent neural network, IEEE Access 10 (2022) 21517-21525.

[41] P.M. Radiuk, Impact of training set batch size on the performance of convolutional neural networks for diverse datasets, Inf. Technol. Manag. Sci. 20 (1) (2017) 20-24.

[42] R. Lin, Analysis on the selection of the appropriate batch size in cnn neural network, in: 2022 International Conference on Machine Learning and Knowledge Engineering (MLKE), IEEE, 2022, pp. 106-109.

[43] B. Schmeiser, Batch size effects in the analysis of simulation output, Oper. Res. 30 (3) (1982) 556-568.

[44] I. Loshchilov,F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv: 1711.05101.

[45] Z. Zhuang, M. Liu, A. Cutkosky, F. Orabona, Understanding adamw through proximal methods and scale-freeness, arXiv preprint arXiv:2202.00089.

[46] Y. Zhang, K. Song, Y. Sun, S. Tan, M. Udell, “Why should you trust my explanation?” Understanding uncertainty in lime explanations, arXiv preprint arXiv:1904.12991.

[47] A. Holzinger, A. Saranti, C. Molnar, P. Biecek, W. Samek, Explainable ai methods- a brief overview, in: International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers, Springer, 2022, pp. 13-38.

[48] R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan, et al., Explainable ai (xai): core ideas, techniques, and solutions, ACM Comput. Surv. 55 (9) (2023) 1-33.

[49] M.T. Ribeiro, S. Singh, C. Guestrin, “Why should I trust you?” Explaining the pre- dictions of any classifier,in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135-1144.

[50] C. Molnar, Interpretable Machine Learning, Lulu.com, 2020.