IDL-LTSOJ: Research and implementation of an intelligent online judge system utilizing DNN for defect localization

Abstract

The evolution of artificial intelligence has thrust the Online Judge (OJ) systems into the forefront of research, particularly within programming education, with a focus on enhancing performance and efficiency. Addressing the shortcomings of the current OJ systems in coarse defect localization granularity and heavy task scheduling architecture, this paper introduces an innovative Integrated Intelligent Defect Localization and Lightweight Task Scheduling Online Judge (IDL-LTSOJ) system. Firstly, to achieve token-level fine-grained defect localization, a Deep Fine-Grained Defect Localization (Deep-FGDL) deep neural network model is developed. By integrating Bidirectional Long Short-Term Memory (BiLSTM) and Bidirectional Gated Recurrent Unit (BiGRU), this model extracts fine-grained information from the abstract syntax tree (AST) of code, enabling more accurate defect localization. Subsequently, we propose a lightweight task scheduling architecture to tackle issues, such as limited concurrency in task evaluation and high equipment costs. This architecture integrates a Kafka messaging system with an optimized task distribution strategy to enable concurrent execution of evaluation tasks, substantially enhancing system evaluation efficiency. The experimental results demonstrate that the Deep-FGDL model improves the accuracy by 35.9% in the Top-20 rank compared to traditional machine learning benchmark methods for fine-grained defect localization tasks. Moreover, the lightweight task scheduling strategy notably reduces response time by nearly 6000ms when handling 120 task volumes, which represents a significant improvement in evaluation efficiency over centralized evaluation methods.

Keywords

Online Judge (OJ) system / Fine-grained defect localization / Deep neural network / Task scheduling

CRediT authorship contribution statement

Lihua Song: Writing - review & editing, Supervision, Resources. Ying Han: Writing - original draft. Yufei Guo: Writing - review & editing, Supervision. Chenying Cai: Writing - review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the Beijing Natural Science Foundation (4212018) and the National Key R&D Program of China (2024YFE200500), awarded to Lihua Song.

References

[1] S.I. Rahman, et al., Unsupervised machine learning approach for tailor-ing educational content to individual student weaknesses, High Confid. Comput. (2024) 100228, http://dx.doi.org/10.1016/j.hcc.2024.100228.

[2] S. Wasik, M. Antczak, J. Badura, A. Laskowski, T. Sternal, A survey on online judge systems and their applications, ACM Comput. Surv. 51 (1) (2018) 3:1-3:34, http://dx.doi.org/10.1145/3143560.

[3] Z.J. Liu, N. Wu, F.Q. Huang, Y. Song, Hybrid programming task recommen-dation model based on knowledge graph and collaborative, Filter. Online Judg. Comput. Sci. 50 (2) (2023) 106-114.

[4] M. Zakeri-Nasrabadi, S. Parsa, M. Ramezani, C. Roy, M. Ekhtiarzadeh, A systematic literature review on source code similarity measurement and clone detection: Techniques, applications, and challenges, J. Syst. Softw. 204 (2023) 111796, http://dx.doi.org/10.1016/j.jss.2023.111796.

[5] H. Cheers, Y. Lin, S.P. Smith, Academic source code plagiarism detec-tion by measuring program behavioral similarity, IEEE Access 9 (2021) 50391-50412, http://dx.doi.org/10.1109/ACCESS.2021.3069367.

[6] W. Zhang, Z. Li, Q. Wang, J. Li, FineLocator: A novel approach to method-level fine-grained bug localization by query expansion, Inf. Softw. Technol. 110 (2019) 121-135, http://dx.doi.org/10.1016/j.infsof.2019.03.001.

[7] S. Jeon, H.K. Kim, AutoVAS: An automated vulnerability analysis system with a deep learning approach, Comput. Secur. 106 (2021) 102308, http://dx.doi.org/10.1016/j.cose.2021.102308.

[8] L. Wartschinski, Y. Noller, T. Vogel, T. Kehrer, L. Grunske, VUDENC: Vulnerability detection with deep learning on a natural codebase for python, Inf. Softw. Technol. 144 (2022) 106809, http://dx.doi.org/10.1016/j.infsof.2021.106809.

[9] Z. Zhou, L. Wang, X. Li, Automatic defect repair and validation approach for C/C++ programs, Ruan Jian Xue Bao J. Softw. 30 (5) (2019) 1243-1255, http://dx.doi.org/10.13328/j.cnki.jos.005729.

[10] D. Ghosh, J. Singh, Spectrum-based multi-fault localization using chaotic genetic algorithm, Inf. Softw. Technol. 133 (2021) 106512, http://dx.doi.org/10.1016/j.infsof.2021.106512.

[11] M. Wang, W. Han, W. Chen, MetaOJ: A massive distributed online judge system, Tsinghua Sci. Technol. 26 (4) (2021) 548-557, http://dx.doi.org/10.26599/TST.2020.9010016.

[12] S.S. Hajam, S.A. Sofi, Spider monkey optimization based resource allocation and scheduling in fog computing environment, High Confid. Comput. 3 (3)(2023) 100149, http://dx.doi.org/10.1016/j.hcc.2023.100149.

[13] Q. Xia, W. Ye, Z. Tao, J. Wu, Q. Li, A survey of federated learning for edge computing: Research problems and solutions, High Confid. Comput. 1 (1)(2021) 100008, http://dx.doi.org/10.1016/j.hcc.2021.100008.

[14] B. Liu, X. Xu, L. Qi, Q. Ni, W. Dou, Task scheduling with precedence and placement constraints for resource utilization improvement in multi-user MEC environment, J. Syst. Archit. 114 (2021) 101970, http://dx.doi.org/10.1016/j.sysarc.2020.101970.

[15] X. Ma, G. Rao, H. Xu, Research on task scheduling in cloud computing, Comput. Sci. 46 (3) (2019) 1-8.

[16] J. Meng, H. Tan, X.-Y. Li, Z. Han, B. Li, Online deadline-aware task dispatch-ing and scheduling in edge computing, IEEE Trans. Parallel Distrib. Syst. 31 (6) (2020) 1270-1286, http://dx.doi.org/10.1109/TPDS.2019.2961905.

[17] N. Verba, K.-M. Chao, J. Lewandowski, N. Shah, A. James, F. Tian, Modeling industry 4.0 based fog computing environments for application analysis and deployment,Future Gener. Comput. Syst. 91 (2019) 48-60, http://dx.doi.org/10.1016/j.future.2018.08.043.

[18] N. Rizvi, R. Dharavath, D.R. Edla, Cost and makespan aware workflow scheduling in iaas clouds using hybrid spider monkey optimization, Simul. Model. Pract. Theory 110 (2021) 102328, http://dx.doi.org/10.1016/j.simpat.2021.102328.

[19] N. Mansouri, M.M. Javidi, Cost-based job scheduling strategy in cloud com-puting environments, Distrib. Parallel Databases 38 (2) (2020) 365-400, http://dx.doi.org/10.1007/s10619-019-07273-y.

[20] C. Li, Y. Zhang, Y. Luo, Neighborhood search-based job scheduling for IoT big data real-time processing in distributed edge-cloud computing environment, J. Supercomput. 77 (2) (2021) 1853-1878, http://dx.doi.org/10.1007/s11227-020-03343-6.

[21] G. Zheng, H. Zhang, Y. Li, L. Xi, 5G network-oriented hierarchical dis-tributed cloud computing system resource optimization scheduling and allocation, Comput. Commun. 164 (2020) 88-99, http://dx.doi.org/10.1016/j.comcom.2020.10.005.

[22] K. Mishra, R. Pradhan, S.K. Majhi, Quantum-inspired binary chaotic salp swarm algorithm (QBCSSA)-based dynamic task scheduling for mul-tiprocessor cloud computing systems, J. Supercomput. 77 (9) (2021) 10377-10423, http://dx.doi.org/10.1007/s11227-021-03695-7.

[23] H. Hanif, M.H.N. Md Nasir, M.F. Ab Razak, A. Firdaus, N.B. Anuar, The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches, J. Netw. Comput. Appl. 179 (2021) 103009, http://dx.doi.org/10.1016/j.jnca.2021.103009.

[24] O. Chaparro, J.M. Florez, A. Marcus, Using bug descriptions to reformulate queries during text-retrieval-based bug localization, Empir Software Eng 24 (5) (2019) 2947-3007, http://dx.doi.org/10.1007/s10664-018-9672-z.

[25] J. Wang, Z. Huang, H. Xiao, Y. Xiao, JFinder: A novel architecture for java vulnerability identification based quad self-attention and pre-training mechanism, High Confid. Comput. 3 (4) (2023) 100148, http://dx.doi.org/10.1016/j.hcc.2023.100148.

[26] X. Huo, F. Thung, M. Li, D. Lo, S.-T. Shi, Deep transfer bug localization, IEEE Trans. Softw. Eng. 47 (7) (2021) 1368-1380, http://dx.doi.org/10.1109/TSE.2019.2920771.

[27] Z. Li, D. Zou, S. Xu, Z. Chen, Y. Zhu, H. Jin, VulDeeLocator: A deep learning-based fine-grained vulnerability detector, IEEE Trans. Dependable Secure Comput. 19 (4) (2022) 2821-2837, http://dx.doi.org/10.1109/TDSC.2021.3076142.

[28] G. Lin, S. Wen, Q.-L. Han, J. Zhang, Y. Xiang, Software vulnerability detection using deep neural networks: A survey, Proc. IEEE 108 (10) (2020) 1825-1848, http://dx.doi.org/10.1109/JPROC.2020.2993293.

[29] S. Cao, X. Sun, L. Bo, Y. Wei, B. Li, BGNN4VD: Constructing bidirectional graph neural-network for vulnerability detection, Inf. Softw. Technol. 136 (2021) 106576, http://dx.doi.org/10.1016/j.infsof.2021.106576.

[30] G. Giray, K.E. Bennin, Ö. Köksal, Ö. Babur, B. Tekinerdogan, On the use of deep learning in software defect prediction, J. Syst. Softw. 195 (2023) 111537, http://dx.doi.org/10.1016/j.jss.2022.111537.

[31] Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, Z. Chen, SySeVR: A framework for using deep learning to detect software vulnerabilities, IEEE Trans. Dependable Secure Comput. 19 (4) (2022) 2244-2258, http://dx.doi.org/10.1109/TDSC.2021.3051525.

[32] H.S. Munir, S. Ren, M. Mustafa, C.N. Siddique, S. Qayyum, Attention based GRU-LSTM for software defect prediction, PLoS One 16 (3) (2021) e0247444, http://dx.doi.org/10.1371/journal.pone.0247444.

[33] J. Deng, L. Lu, S. Qiu, Y. Ou, A suitable AST node granularity and multi-kernel transfer convolutional neural network for cross-project defect prediction, IEEE Access 8 (2020) 66647-66661, http://dx.doi.org/10.1109/ACCESS.2020.2985780.

[34] L. Zhao, Z. Shang, L. Zhao, T. Zhang, Y.Y. Tang, Software defect prediction via cost-sensitive siamese parallel fully-connected neural networks, Neuro-computing 352 (2019) 64-74, http://dx.doi.org/10.1016/j.neucom.2019.03.076.

[35] W. Zheng, Y. Jiang, X. Su, Vu1SPG: Vulnerability detection based on slice property graph representation learning, in: 2021 IEEE 32nd Interna-tional Symposium on Software Reliability Engineering, (ISSRE), 2021, pp. 457-467, http://dx.doi.org/10.1109/ISSRE52982.2021.00054.

[36] H. Wu, Research proposal: Reliability evaluation of the Apache Kafka streaming system, in: 2019 IEEE International Symposium on Software Reliability Engineering Workshops, (ISSREW), 2019, pp. 112-113, http://dx.doi.org/10.1109/ISSREW.2019.00055.

[37] H. Chen, X. Cui, Y. Wang, Summary of task scheduling algorithms based on multiple cloud environments, Appl. Res. Comput. 40 (10) (2023) 2889-2895, http://dx.doi.org/10.19734/j.issn.1001-3695.2023.02.0076.

[38] X. Cheng, H. Wang, J. Hua, G. Xu, Y. Sui, DeepWukong: Statically detecting software vulnerabilities using deep graph neural network, ACM Trans. Softw. Eng. Methodol. 30 (3) (2021) 38:1-38:33, http://dx.doi.org/10.1145/3436877.

[39] C. Tantithamthavorn, S. Lemma Abebe, A.E. Hassan, A. Ihara, K. Matsumoto, The impact of IR-based classifier configuration on the performance and the effort of method-level bug localization, Inf. Softw. Technol. 102 (2018) 160-174, http://dx.doi.org/10.1016/j.infsof.2018.06.001.

[40] S. Liu, et al., CD-VulD: Cross-domain vulnerability discovery based on deep domain adaptation, IEEE Trans. Dependable Secure Comput. 19 (1) (2022) 438-451, http://dx.doi.org/10.1109/TDSC.2020.2984505.

[41] A. Majd, M. Vahidi-Asl, A. Khalilian, P. Poorsarvi-Tehrani, H. Haghighi, SLDeep: Statement-level software defect prediction using deep-learning model on static code features, Expert Syst. Appl. 147 (2020) 113156, http://dx.doi.org/10.1016/j.eswa.2019.113156.

[42] Y. Zhang, J. Zhou, J. Hu, Research on defect location method of c language code based on deep learning, in: 2021 16th International Conference on Computer Science & Education, (ICCSE), 2021, pp. 360-365, http://dx.doi.org/10.1109/ICCSE51940.2021.9569383.

[43] Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, Y. Zhang, A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly, High Confid. Comput. 4 (2) (2024) 100211, http://dx.doi.org/10.1016/j.hcc.2024.100211.