A novel zero-day ransomware detection approach based on CVAE and 1D-CNN

Abstract

Ransomware has emerged as one of the most prevalent and destructive cyber attacks confronting global organizations. By locking critical devices or encrypting essential data and then demanding payment for restoration, ransomware attacks disrupt operations, result in significant financial losses, and damage organizational reputations. In particular, zero-day ransomware attacks, which attempt to exploit previously unknown vulnerabilities, pose a severe threat to existing cyber security solutions. Due to the lack of training data, detection of zero-day ransomware attacks remains a significant challenge. This paper proposes a novel zero-day ransomware detection framework that integrates a refined Conditional Variational Autoencoder (CVAE) with a 1D Convolutional Neural Network (1D-CNN). The encoder of the CVAE model comprises a posterior network and a parallel prior network. Using variational coding, the posterior network maps behavioral features of software samples from known families into a latent space, represented by a fixed multivariate Gaussian distribution with a diagonal covariance matrix. Simultaneously, the prior network eliminates dependency on class labels while maintaining distributional consistency with the posterior network via Kullback-Leibler (KL) divergence minimization. This dual-network structure enables unified latent space mapping for both labeled and unlabeled samples, effectively narrowing distributional discrepancies between software samples from known and unknown families. The harmonized latent representations subsequently enhance the discriminative capability of the 1D-CNN classifier in detecting zero-day ransomware. The comprehensive experimental results have verified that the proposed method can effectively detect zero-day ransomware attacks.

Keywords

Attack detection / Zero-day ransomware / CVAE / 1D-CNN

CRediT authorship contribution statement

Bohan Cui: Validation, Methodology. Yan Hu: Writing - original draft, Conceptualization. Tianheng Qu: Writing - review & editing, Methodology. Yunhua He: Writing - review & editing, Conceptualization. Limin Sun: Funding acquisition, Project administration.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors appreciate the reviewers for their helpful comments and suggestions for the improvement of this paper. This work is supported by the Industrial Foundation Reconstruction and High-Quality Development of Manufacturing Industry Special Project (0747-2361SCCZA193), the National Natural Science Foundation of China (62272007) and Interdisciplinary Research Project for Young Teachers of USTB (Fundamental Research Funds for the Central Universities , FRF-IDRY-24-015).

References

[1] I. Kara, M. Aydos, The rise of ransomware: Forensic analysis for windows based ransomware attacks, Expert Syst. Appl. 190 (2022) 116198.

[2] N. Kshetri, J. Voas, Do crypto-currencies fuel ransomware? IT Prof. 19 (5) (2017) 11-15.

[3] Zimba. A, M. Chishimba, On the economic impact of crypto-ransomware attacks: The state of the art on enterprise systems, Eur. J. Secur. Res. 4 (1) (2019) 3-31.

[4] N.K. Sreelaja, Ant colony optimization based light weight binary search for efficient signature matching to filter ransomware, Appl. Soft Comput. 111 (2021) 107635.

[5] K.S. Sangher, A. Singh, H.M. Pandey, Signature based ransomware detection based on optimizations approaches using RandomClassifier and CNN algorithms, Int. J. Syst. Assur. Eng. Manag. 15 (5) (2024) 1687-1703.

[6] N.K. Sreelaja, Ant colony optimization based light weight binary search for efficient signature matching to filter ransomware, Appl. Soft Comput. 111 (2021) 107635.

[7] K. Begovic, A. Al-Ali, Q. Malluhi, Cryptographic ransomware encryption detection: Survey, Comput. Secur. 132 (2023) 103349.

[8] S. Lee, N. Jho, D. Chung, Y. Kang, M. Kim, Rcryptect: Real-time detection of cryptographic function in the user-space filesystem, Comput. Secur. 112 (2022) 102512.

[9] D. Yu, H. Zhang, Y. Huang, Z. Xie, Data distribution inference attack in federated learning via reinforcement learning support, High-Confid. Comput. 5 (1) (2025) 100235.

[10] X. Wang, Y. Zhao, L. Zhang, M. Xie, Y. Yu, H. Li, Linkable group signatures against malicious regulators for regulated privacy-preserving cryptocurrencies, High-Confid. Comput. (2025) 100318.

[11] J. Hwang, J. Kim, S. Lee, K. Kim, Two-stage ransomware detection using dynamic analysis and machine learning techniques, Wirel. Pers. Commun. 112 (4) (2020) 2597-2609.

[12] A. Tauhid, L. Xu, M. Rahman, E. Tomai, A survey on security analysis of machine learning-oriented hardware and software intellectual property, High-Confid. Comput. 3 (2) (2023) 100114.

[13] H. Zhang, X. Xiao, F. Mercaldo, S. Ni, F. Martinelli, A. Sangaiah, Classification of ransomware families with machine learning based onN-gram of opcodes, Future Gener. Comput. Syst. 90 (2019) 211-221.

[14] Moreira.Caio. C, Moreira.Davi. C, S.de.Sales.Jr.Claudomiro. de, F. Martinelli, A. Sangaiah, Improving ransomware detection based on portable executable header using xception convolutional neural network, Comput. Secur. 130 (2023) 103265.

[15] J. Zhu, J. Jang-Jaccard, A. Singh, I. Welch, H. Al-Sahaf, S. Camtepe, A few-shot meta-learning based siamese neural network using entropy features for ransomware classification, Comput. Secur. 117 (2022) 102691.

[16] H. Öz, A. Aris, A. Levi, A.S. Uluagac, A survey on ransomware: Evolution, Taxon. Déf. Solutions. Surv. 55 (9) (2022) 1-36.

[17] D. Vidyarthi, C. Kumar, S. Rakshit, S. Chansarkar, Static malware analysis to identify ransomware properties, Int. J. Comput. Sci. Issues 16 (3) (2019) 10-17.

[18] F. Manavi, A. Hamzeh, S. Chansarkar, A novel approach for ransomware detection based on PE header using graph embeddingt, J. Comput. Virol. Hacking Tech. 18 (4) (2022) 285-296.

[19] E. Karbab, M. Debbabi, A. Derhab, Swiftr: Cross-platform ransomware fingerprinting using hierarchical neural networks on hybrid features, Expert Syst. Appl. 225 (2022) 120017.

[20] S. Gülmez, A.G. Kakisim, I. Sogukpinar, Xran: Explainable deep learning-based ransomware detection using dynamic analysis, Comput. Secur. 139 (2024) 103703.

[21] J. Baldwin, A. Dehghantanha, Leveraging support vector machine for opcode density based detection of crypto-ransomware, Cyber Threat. Intell. (2018) 107-136.

[22] S. Sharma, S. Singh, Texture-based automated classification of ransomware, J. Inst. Eng. (India): Ser. 102 (1) (2021) 131-142.

[23] S. Lee, N. Jho, D. Chung, Kang. Y, M. Kim, Rcryptect: Real-time detection of cryptographic function in the user-space filesystem, Comput. Secur. 112 (2022) 102512.

[24] S. Homayoun, A. Dehghantanha, M. Ahmadzadeh, S. Hashemi, R. Khayami, Know abnormal, find evil: frequent pattern mining for ransomware threat hunting and intelligence, IEEE Trans. Emerg. Top. Comput. 8 (2) (2017) 341-351.

[25] B. Jethva, I. Traoré, A. Ghaleb, K. Ganame, S. Ahmed, Multilayer ransomware detection using grouped registry key operations, File Entropy File Signat. Monit. J. Comput. Secur. 28 (3) (2020) 337-373.

[26] R.M.A. Molina, S. Torabi, K. Sarieddine, E. Bou-Harb, N. Bouguila, C. Assi, On ransomware family attribution using pre-attack paranoia activities, IEEE Trans. Netw. Serv. Manag. 19 (1) (2021) 19-36.

[27] H. Zhang, L. Zhao, A. Yu, L. Cai, D. Meng, Ranker: Early ransomware detection through kernel-level behavioral analysis, IEEE Trans. Inf. Forensics Secur. 19 (2024) 6113-6127.

[28] M. Cen, X. Deng, F. Jiang, R. Doss, Zero-Ran Sniff: A zero-day ransomware early detection method based on zero-shot learning, Comput. Secur. 142 (2024) 103849.

[29] Y. Brinkley, D. Thompson, N. Simmons, Machine learning-based intrusion detection for zero-day ransomware in unseen data, Authorea (2024) 1-8, http://dx.doi.org/10.22541/au.172685266.62026194/v1.

[30] U. Zahoora, M. Rajarajan, Z. Pan, A. Khan, Zero-day ransomware attack detection using deep contractive autoencoder and voting based ensemble classifier, Appl. Intell. 52 (12) (2022) 1396-13941.

[31] R. Rana, R. Singhal, Chi-square test and its application in hypothesis testing, J. Pr. Cardiovasc. Sci. 1 (1) (2015) 69-71.

[32] D. Sgandurra, L. Muñoz-González, R. Mohsen, E.C. Lupu, Automated dynamic analysis of ransomware: Benefits, limitations and use for detection, 2016, arXiv preprint arXiv:1609.03020.

[33] Y. Yang, K. Zheng, C. Wu, Y. Yang, Improving the classification effectiveness of intrusion detection by using improved conditional variational autoencoder and deep neural network, Sensors 19 (11) (2019) 2528.

[34] M. Greenacre, P.J. Groenen, T. Hastie, A.I. d’Enza, A. Markos, E. Tuzhilina, Principal component analysis, Nat. Rev. Methods Prim. 2 (1) (2022) 100.

[35] Z. Zhang, P. Qi, W. Wang, Dynamic malware analysis with feature engineering and feature learning, Proc. the AAAI Conf. Artif. Intell. 34 (1) (2020) 1210-1217.

[36] P.H. Barros, E.T.C. Chagas, L.B. Oliveira, F. Queiroz, H.S. Ramos, Malware-SMELL: A zero-shot learning strategy for detecting zero-day vulnerabilities, Comput. Secur. 120 (2022) 102785.