From Ambiguous Queries to Verifiable Insights: A Task-Driven Framework for LLM-Powered SOC Analysis

Abstract

Security operations centre (SOC) analysts must investigate alerts, correlate threat intelligence and interpret heterogeneous telemetry under tight timing constraints. Although large language models (LLMs) offer strong understanding capabilities, directly applying them to SOC environments remains challenging due to semantic ambiguity in analyst queries, fragmented multisource event data, limited domain-specific reasoning and reliability concerns associated with unconstrained query generation. We present a task-driven knowledge-augmented framework designed to produce verifiable and contextually grounded responses for SOC workflows. The framework integrates four components: (i) contrastive context task recognition that mitigates semantic ambiguity by mapping analyst queries to predefined SOC task types; (ii) expert-guided knowledge augmentation that fuses dense and sparse retrieval to bridge the semantic gap; (iii) schema-aligned event retrieval combined with entity-centric evidence profiling to ensure reliable and secure access to heterogeneous telemetry and (iv) verifiable task-aware generation that constrains model outputs to retrieved knowledge and security events. To assess the framework, we construct a benchmark of 12,500 validated question–answer pairs derived through semiautomated synthesis over more than 34 million real SOC records. Experiments across multiple foundation models demonstrate consistent improvements in relevance and grounding quality. Our results indicate that the four proposed components substantially enhance LLMs' reliability in practical SOC analysis.

Keywords

heterogeneous security telemetry / knowledge-augmented retrieval / large language models / security operations centre / task recognition

Acknowledgements

This study was funded by the Shenzhen Science and Technology Program (Grant No. KJZD20240903103811016), the Science and Technology Development Fund, Macao, SAR, under Grant 0007/2024/AKP, the Major Key Project of PCL (Grant No. PCL2024A05) and the Science and Technology Development Fund (Macao, SAR, Grant No. 0007/2024/AKP).

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

To ensure reproducibility whilst preserving organisational data confidentiality, we publicly release all nonsensitive research artefacts. For the internal SOC dataset, we provide the complete set of 1250 question templates, schema definitions and aggregated metadata. Furthermore, we release a fully open-source benchmark based on the Linux-APT-2024 dataset, encompassing the complete database, 260 verified question- answer pairs. The benchmark repository is publicly available at https://github.com/Hanaezh/soc-benchmark. Researchers interested in accessing the raw internal operational data may contact the first author or corresponding author to request access, subject to institutional verification and appropriate nondisclosure agreements (NDAs).

References

[1] M. Simoni, A. Saracino, V. P, and M. Conti, “Morse:Bridging the Gap in Cybersecurity Expertise With Retrieval Augmented Generation,” in Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, (2025), 1213-1222, https://doi.org/10.1145/3672608.3707898.

[2] S. Rajapaksha, R. Rani, and E. Karafili, “A Rag-Based Question-Answering Solution for Cyber-Attack Investigation and Attribution,” in European Symposium on Research in Computer Security, (2024), 238-256, https://doi.org/10.1007/978-3-031-82362-6_15.

[3] C. Zhao, G. Agrawal, T. Kumarage, et al., “Ontology-Aware Rag for Improved Question-Answering in Cybersecurity Education,” preprint arXiv:2412.14191 (2024).

[4] X. Zhao, X. Zhou, and G. Li, “Chat2data:An Interactive Data Analysis System With Rag, Vector Databases and Llms,” Proceedings of the VLDB Endowment 17, no. 12 (2024): 4481-4484, https://doi.org/10.14778/3685800.3685905.

[5] X. Zhou, Z. Sun, and G. Li, “Db-gpt: Large Language Model Meets Database,” Data Science and Engineering 9, no. 1 (2024): 102-111, https://doi.org/10.1007/s41019-023-00235-6.

[6] B. Wang, C. Ren, J. Yang, et al., “MAC-SQL:A Multi-Agent Collaborative Framework for Text-to-sql,” in Proceedings of the 31st International Conference on Computational Linguistics, (2025), 540-557,

[7] H. Li, J. Zhang, H. Liu, et al., “Codes: Towards Building Open-Source Language Models for Text-to-sql,” Proceedings of the ACM on Management of Data 2, no. 3 (2024): 1-28, https://doi.org/10.1145/3654930.

[8] M. Pourreza and D. Rafiei, “Din-Sql: Decomposed in-Context Learning of Text-to-sql With Self-Correction,” Advances in Neural Information Processing Systems 36 (2023): 36339-36348,

[9] D. Gao, H. Wang, Y. Li, et al., “Text-to-Sql Empowered by Large Language Models: A Benchmark Evaluation,” Proceedings of the VLDB Endowment 17, no. 5 (2024): 1132-1145, https://doi.org/10.14778/3641204.3641221.

[10] H. Li, Z. Wang, C. Lan, P. Wu, and N. Zeng, “A Novel Dynamic Multiobjective Optimization Algorithm With Hierarchical Response System,” IEEE Transactions on Computational Social Systems 11, no. 2 (2023): 2494-2512, https://doi.org/10.1109/tcss.2023.3293331.

[11] H. Li, Z. Wang, N. Zeng, P. Wu, and Y. Li, “Promoting Objective Knowledge Transfer: A Cascaded Fuzzy System for Solving Dynamic Multiobjective Optimization Problems,” IEEE Transactions on Fuzzy Systems 32, no. 11 (2024): 6199-6213, https://doi.org/10.1109/tfuzz.2024.3443207.

[12] F. Y. Loumachi, M. C. Ghanem, and M. A. Ferrag, “Advancing Cyber Incident Timeline Analysis Through Retrieval-Augmented Generation and Large Language Models,” Computer Times 14, no. 2 (2025): 1-42, https://doi.org/10.3390/computers14020067.

[13] R. Kurnia, F. Widyatama, I. M. Wibawa, et al., “Enhancing Security Operations Center: Wazuh Security Event Response With Retrieval-Augmented-Generation-Driven Copilot,” Sens 25 (2025): 870, https://doi.org/10.3390/s25030870.

[14] J. Zhang, H. Bu, H. Wen, et al., “When Llms Meet Cybersecurity: A Systematic Literature Review,” Cybersecurity 8 (2025): 1-41, https://doi.org/10.1186/s42400-025-00361-w.

[15] R. Fayyazi, R. Taghdimi, and S. J. Yang, “Advancing Ttp Analysis:Harnessing the Power of Large Language Models With Retrieval Augmented Generation,” in 2024 Annual Computer Security Applications Conference Workshops (ACSAC Workshops), (2024), 255-261, https://doi.org/10.1109/ACSACW65225.2024.00036.

[16] Y. Liu, Y. Xue, D. Wu, et al., “Propertygpt:Llm-Driven Formal Verification of Smart Contracts Through Retrieval-Augmented Property Generation,” in 32nd Annual Network and Distributed System Security Symposium, (2025), 1-17.

[17] A. Alhuzali, “Llm-Powered Threat Intelligence: A Retrieval-Augmented Generation Approach for Cyber Attack Investigation,” PeerJ Computer Science 11 (2025): e3371, https://doi.org/10.7717/peerj-cs.3371.

[18] F. Blefari, C. Cosentino, F. A. Pironti, A. Furfaro, and F. Marozzo, “Cyberrag: An Agentic Rag Cyber Attack Classification and Reporting Tool,” Future Generation Computer Systems 176 (2025): 108186, https://doi.org/10.1016/j.future.2025.108186.

[19] Z. Hong, Z. Yuan, Q. Zhang, et al., “Next-Generation Database Interfaces: A Survey of Llm-Based Text-to-sql,” IEEE Transactions on Knowledge and Data Engineering (2025).

[20] S. Talaei, M. Pourreza, Y.-C. Chang, A. Mirhoseini, and A. Saberi, “Chess: Contextual Harnessing for Efficient Sql Synthesis,” preprint arXiv:2405.16755 (2024).

[21] M. Pourreza, H. Li, R. Sun, et al., “CHASE-SQL:Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL,” in The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, (April 2025).

[22] U. N. A. Perera, S. Rathnayaka, N. Perera, W. Madushanka, and A. N. Senarathne, “The Next Gen Security Operation Center,” in 2021 6th International Conference for Convergence in Technology (I2CT), (2021), 1-9.

[23] M. Mărmureanu and C. Oprişa, “ Mitre Tactics Inference From Splunk Queries,” in 2023 IEEE 19th International Conference on Intelligent Computer Communication and Processing (ICCP), (2023), 277-283, https://doi.org/10.1109/ICCP60212.2023.10398612.

[24] S. Ndichu, T. Ban, T. Takahashi, and D. Inoue, “A Machine Learning Approach to Detection of Critical Alerts From Imbalanced Multi-Appliance Threat Alert Logs,” in 2021 IEEE International Conference on Big Data (Big Data), (2021), 2119-2127, https://doi.org/10.1109/BIGDATA52589.2021.9671956.

[25] R. Vaarandi and A. Guerra-Manzanares, “Stream Clustering Guided Supervised Learning for Classifying Nids Alerts,” Future Generation Computer Systems 155 (2024): 231-244, https://doi.org/10.1016/j.future.2024.01.032.

[26] J.-y. Kim and H.-Y. Kwon, “Threat Classification Model for Security Information Event Management Focusing on Model Efficiency,” Computers & Security 120 (2022): 102789, https://doi.org/10.1016/j.cose.2022.102789.

[27] Z. T. Sworna, C. Islam, and M. A. Babar, “Apiro: A Framework for Automated Security Tools Api Recommendation,” ACM Transactions on Software Engineering and Methodology 32 (2023): 1-42, https://doi.org/10.1145/3512768.

[28] Z. T. Sworna, M. A. Babar, and A. Sreekumar, “Irp2api: Automated Mapping of Cyber Security Incident Response Plan to Security Tools’ Apis,” in IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), (2023), 546-557, https://doi.org/10.1109/SANER56733.2023.00057.

[29] C. Islam, M. A. Babar, R. Croft, and H. Janicke, “Smartvalidator: A Framework for Automatic Identification and Classification of Cyber Threat Data,” Journal of Network and Computer Applications 202 (2022): 103370, https://doi.org/10.1016/j.jnca.2022.103370.

[30] S. Freitas, J. Kalajdjieski, A. Gharib, and R. McCann, “Ai-Driven Guided Response for Security Operation Centers With Microsoft Copilot for Security,” in Companion Proceedings of the ACM on Web Conference, (2025), 191-200, https://doi.org/10.1145/3701716.3715209.

[31] P. Tseng, Z. Yeh, X. Dai, and P. Liu, “Using Llms to Automate Threat Intelligence Analysis Workflows in Security Operation Centers,” preprint arXiv:2407.13093 (2024).

[32] Y. Hu, F. Zou, J. Han, X. Sun, and Y. Wang, “Llm-Tikg: Threat Intelligence Knowledge Graph Construction Utilizing Large Language Model,” Computers & Security 145 (2024): 103999, https://doi.org/10.1016/j.cose.2024.103999.

[33] J. Ye, Z. Wu, J. Feng, T. Yu, and L. Kong, “Compositional Exemplars for in-Context Learning,” in International Conference on Machine Learning (PMLR, 2023), 39818-39833.

[34] S. Min, K. Krishna, X. Lyu, et al., Factscore: Fine-Grained Atomic Evaluation of Factual Precision in Long Form Text Generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, (2023), 12076-12100.

[35] J. Wei, C. Yang, X. Song, et al., Long-Form Factuality in Large Language Models,” Advances in Neural Information Processing Systems 37 (2024): 80756-80827, https://doi.org/10.52202/079017-2567.

[36] M. T. Alam, D. Bhusal, L. Nguyen, and N. Rastogi, “Ctibench: A Benchmark for Evaluating Llms in Cyber Threat Intelligence,” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems, Vol. 37, (2024), 50805-50825, http://papers.nips.cc/paper_files/paper/2024/hash/5acd3c628aa1819fbf07c39ef73e7285-Abstract-Datasets_and_Benchmarks_Track.html.

[37] P. Jing, M. Tang, X. Shi, et al., Secbench: A Comprehensive Multi-Dimensional Benchmarking Dataset for Llms in Cybersecurity,” preprint arXiv:2412.20787 (2024).

[38] G. C. Mandiant, “M-Trends 2024: Special Report,” (2024), https://services.google.com/fh/files/misc/m-trends-2024-executive-edition.pdf.

[39] B. Tostes, L. Ventura, E. Lovat, M. Martins, and D. Menasché, “Learning When to Say Goodbye: What Should Be the Shelf Life of an Indicator of Compromise?,” in IEEE International Conference on Cyber Security and Resilience (CSR), (2023), 503-510, https://doi.org/10.1109/CSR57506.2023.10224937.

[40] A. Liu, B. Feng, B. Xue, et al., Deepseek-v3 Technical Report,” preprint arXiv:2412 (2024): 19437.

[41] J. Achiam, S. Adler, S. Agarwal, et al., Gpt-4 Technical Report,” preprint arXiv:2303.08774 (2023).

[42] A. Grattafiori, A. Dubey, A. Jauhri, et al., The Llama 3 Herd of Models,” preprint arXiv:2407.21783 (2024).

[43] B. Hui, J. Yang, Z. Cui, et al., Qwen2.5-Coder Technical Report,” preprint arXiv:2409.12186 (2024).

[44] Y.-C. Yu, T.-H. Chiang, C.-W. Tsai, C.-M. Huang, and W.-K. Tsao, “Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity Llm Training,” preprint arXiv:2502.11191 (2025).

[45] S. Clouditera, “A Network Security Large Language Model,” (2023), https://github.com/Clouditera/secgpt.

[46] S. Es, J. James, L. E. Anke, and S. Schockaert, “Ragas: Automated Evaluation of Retrieval Augmented Generation,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, (2024), 150-158, https://aclanthology.org/2024.eacl-demo.16.

[47] L. Zheng, W.-L. Chiang, Y. Sheng, et al., Judging Llm-as-a-Judge With mt-Bench and Chatbot Arena,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, Vol. 36, (2023), 46595-46623.

[48] S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J.-Y. Nie, “C-Pack: Packed Resources for General Chinese Embeddings,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, (2024), 641-649.

[49] L. Wang, N. Yang, X. Huang, et al., Text Embeddings by Weakly-Supervised Contrastive Pre-Training,” preprint arXiv:2212.03533 (2022).

[50] S. S. Karim, M. Afzal, W. Iqbal, and D. Al Abri, “Advanced Persistent Threat (Apt) and Intrusion Detection Evaluation Dataset for Linux Systems 2024,” Data in Brief 54 (2024): 110290, https://doi.org/10.1016/j.dib.2024.110290.