Actionable Warning Identification in the Era of LLMs: An Empirical Study
Mengyao Zhang , Lingeng Ma , Xuan Xie , Haolong Huang , Ya Pan , Xiuting Ge , Chunrong Fang
Actionable Warning Identification (AWI) is crucial for improving the usability of Static Code Analyzers (SCAs). Owing to their strong capability in understanding code semantics, Large Language Models (LLMs) have recently been preliminarily applied to AWI. Existing studies indicate that both prompt design and LLM selection can substantially influence the performance of LLM-based AWI. However, the effects of different prompt templates and LLM choices have not been systematically investigated, which limits our understanding of when and why LLM-based AWI works well and hinders its effective and reliable adoption in practice.
To bridge the above gap, we perform the first comprehensive empirical study to investigate the performance of LLM-based AWI on four typical LLMs, eight elaborate prompt templates, four widely used SCAs, and three programming languages (C, C++, and Java). The experimental results on 12K+ warnings show that the precision of LLM-based AWI achieves 24.14%. Besides, the combination of DeepSeek-R1 and the self-heuristic prompt template consistently outperforms other configurations. Furthermore, we investigate the root cause of underperformance in LLM-based AWI (e.g., insufficient warning context extraction). Based on these findings, we provide several implications for future research (e.g., warning context refinement).
Actionable Warning Identification / Large Language Model / Empirical Study / Prompt Engineering
Higher Education Press 2026
/
| 〈 |
|
〉 |
PDF
(19339KB)
