SEA-SQL: semantic-enhanced text-to-SQL with adaptive refinement

Abstract

Recent advancements in large language models (LLMs) have significantly contributed to the progress of the Text-to-SQL task. A common requirement in many of these works is the post-correction of SQL queries. However, the majority of this process entails analyzing error cases to develop prompts with rules that eliminate model bias. And there is a weakness of execution verification for SQL queries. In addition, the prevalent techniques primarily depend on GPT-4 and few-shot prompts, resulting in expensive costs. To investigate the effective methods for SQL refinement in a cost-efficient manner, we introduce Semantic-Enhanced Text-to-SQL with Adaptive Refinement (SEA-SQL), which includes Adaptive Bias Elimination and Dynamic Execution Adjustment, aims to improve performance while minimizing resource expenditure with zero-shot prompts. Specifically, SEA-SQL employs a semantic-enhanced schema to augment database information and optimize SQL queries. During the SQL query generation, a fine-tuned adaptive bias eliminator is applied to mitigate inherent biases caused by the LLM. The dynamic execution adjustment is utilized to guarantee the executability of the bias eliminated SQL query. We conduct experiments on the Spider and BIRD datasets to demonstrate the effectiveness of this framework. The results demonstrate that SEA-SQL achieves state-of-the-art performance in the GPT-3.5 scenario with 9%–58% of the generation cost. Furthermore, SEA-SQL is comparable to GPT-4 with only 0.9%–5.3% of the generation cost. Our code is available at the website of github.com/545999961/SEA-SQL.

Graphical abstract

Keywords

Text-to-SQL / adaptive bias elimination / dynamic execution adjustment / economize

References

[1] Cai R, Xu B, Zhang Z, Yang X, Li Z, Liang Z. An encoder-decoder framework translating natural language to database queries. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018, 3977–3983

[2] Rai D, Wang B, Zhou Y, Yao Z. Improving generalization in language model-based text-to-SQL semantic parsing: two simple semantic boundary-based techniques. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2023, 150–160

[3] Li H, Zhang J, Li C, Chen H. RESDSQL: decoupling schema linking and skeleton parsing for text-to-SQL. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 13067–13075

[4] Zeng L, Parthasarathi S H K, Hakkani-Tur D. N-best hypotheses reranking for text-to-SQL systems. In: Proceedings of 2022 IEEE Spoken Language Technology Workshop (SLT). 2023, 663–670

[5] Liu Q, Yang D, Zhang J, Guo J, Zhou B, Lou J G. Awakening latent grounding from pretrained language models for semantic parsing. In: Proceedings of Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021, 1174–1189

[6] Hui B, Shi X, Geng R, Li B, Li Y, Sun J, Zhu X. Improving text-to-SQL with schema dependency learning. 2021, arXiv preprint arXiv: 2103.04399

[7] Yu T, Yasunaga M, Yang K, Zhang R, Wang D, Li Z, Radev D. SyntaxSQLNet: syntax tree networks for complex and cross-DomainText-to-SQL task. 2018, arXiv preprint arXiv: 1810.05237

[8] Wang B, Shin R, Liu X, Polozov O, Richardson M. RAT-SQL: relation-aware schema encoding and linking for text-to-SQL parsers. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 7567–7578

[9] Gan Y, Chen X, Xie J, Purver M, Woodward J R, Drake J, Zhang Q. Natural SQL: making SQL easier to infer from natural language specifications. In: Proceedings of Findings of the Association for Computational Linguistics: EMNLP 2021. 2021, 2030–2042

[10] Yin P, Neubig G. A syntactic neural model for general-purpose code generation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017, 440–450

[11] Scholak T, Schucher N, Bahdanau D. PICARD: parsing incrementally for constrained auto-regressive decoding from language models. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 9895–9901

[12] Suhr A, Chang M W, Shaw P, Lee K. Exploring unexplored generalization challenges for cross-database semantic parsing. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 8372–8388

[13] Li J, Hui B, Qu G, Yang J, Li B, Li B, Wang B, Qin B, Geng R, Huo N, Zhou X, Ma C, Li G, Chang K C C, Huang F, Cheng R, Li Y. Can LLM already serve as A database interface? A big bench for large-scale database grounded text-to-SQLs. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 1835

[14] Liu A, Hu X, Wen L, Yu P S. A comprehensive evaluation of ChatGPT’s zero-shot text-to-SQL capability. 2023, arXiv preprint arXiv: 2303.13547

[15] Chang S, Fosler-Lussier E. How to prompt LLMS for text-to-SQL: a study in zero-shot, single-domain, and cross-domain settings. 2023, arXiv preprint arXiv: 2305.11853

[16] Dong X, Zhang C, Ge Y, Mao Y, Gao Y, Chen L, Lin J, Lou D. C3: zero-shot text-to-SQL with ChatGPT. 2023, arXiv preprint arXiv: 2307.07306

[17] Pourreza M, Rafiei D. DIN-SQL: decomposed in-context learning of text-to-SQL with self-correction. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 1577

[18] Gao D, Wang H, Li Y, Sun X, Qian Y, Ding B, Zhou J . Text-to-SQL empowered by large language models: a benchmark evaluation. Proceedings of the VLDB Endowment, 2024, 17( 5): 1132–1145

[19] Zhou X, Sun Z, Li G . DB-GPT: large language model meets database. Data Science and Engineering, 2024, 9( 1): 102–111

[20] Yao S, Zhao J, Yu D, Du N, Shafran I, Narasimhan K R, Cao Y. ReAct: synergizing reasoning and acting in language models. In: Proceedings of the 11th International Conference on Learning Representations. 2023, 1–33

[21] Wang B, Ren C, Yang J, Liang X, Bai J, Zhang Q W, Yan Z, Li Z. MAC-SQL: a multi-agent collaborative framework for text-to-SQL. 2023, arXiv preprint arXiv: 2312.11242

[22] Wang X, Wei J, Schuurmans D, Le Q V, Chi E H, Narang S, Chowdhery A, Zhou D. Self-consistency improves chain of thought reasoning in language models. In: Proceedings of the 11th International Conference on Learning Representations. 2023, 1–24

[23] Lin X V, Socher R, Xiong C. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In: Proceedings of Findings of the Association for Computational Linguistics: EMNLP 2020. 2020, 4870–4888

[24] Wan X, Han X . Efficient top-k frequent itemset mining on massive data. Data Science and Engineering, 2024, 9( 2): 177–203

[25] Li H, Zhang J, Liu H, Fan J, Zhang X, Zhu J, Wei R, Pan H, Li C, Chen H . CodeS: towards building open-source language models for text-to-SQL. Proceedings of the ACM on Management of Data, 2024, 2( 3): 127

[26] Wang J, Zhao W, Tu X, He T . A novel dense retrieval framework for long document retrieval. Frontiers of Computer Science, 2023, 17( 4): 174609

[27] Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi E H, Le Q V, Zhou D. Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2024, 1800

[28] Yu T, Zhang R, Yang K, Yasunaga M, Wang D, Li Z, Ma J, Li I, Yao Q, Roman S, Zhang Z, Radev D R. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In: Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing. 2018, 3911–3921

[29] Deng X, Awadallah A H, Meek C, Polozov O, Sun H, Richardson M. Structure-grounded pretraining for text-to-SQL. In: Proceedings of 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021, 1337–1350

[30] Zhong R, Yu T, Klein D. Semantic evaluation for text-to-SQL with distilled test suites. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020, 396–411

[31] Xu Y, Xie L, Gu X, Chen X, Chang H, Zhang H, Chen Z, Zhang X, Tian Q. QA-LoRA: quantization-aware low-rank adaptation of large language models. In: Proceedings of the 12th International Conference on Learning Representations, 2024

[32] Aminabadi R Y, Rajbhandari S, Awan A A, Li C, Li D, Zheng E, Ruwase O, Smith S, Zhang M, Rasley J, He Y. DeepSpeed- inference: enabling efficient inference of transformer models at unprecedented scale. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 2022, 1–15

[33] Dao T, Fu D Y, Ermon S, Rudra A, Ré C. FLASHATTENTION: fast and memory-efficient exact attention with IO-awareness. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1189

[34] Jiang W, Ning B, Li G, Bai M, Jia X, Wei F . Graph-decomposed k-NN searching algorithm on road network. Frontiers of Computer Science, 2024, 18( 3): 183609

[35] Jin P, Chu Z, Liu G, Luo Y, Wan S . Optimizing B⁺-tree for hybrid memory with in-node hotspot cache and eADR awareness. Frontiers of Computer Science, 2024, 18( 5): 185606

[36] Li F, Zhang T, Cui S, Liu H, Ren Z, Di D, Wang X, Zhang P, Yu G . A sampling method based on forecasting and combinatorial optimization for high performance A/B testing. Frontiers of Computer Science, 2023, 17( 6): 176616

[37] Yang A, Yang B, Hui B, Zheng B, Yu B, Zhou C, Li C, Li C, Liu D, Huang F, Dong G, Wei H, Lin H, Tang J, Wang J, Yang J, Tu J, Zhang J, Ma J, Yang J, Xu J, Zhou J, Bai J, He J, Lin J, Dang K, Lu K, Chen K, Yang K, Li M, Xue M, Ni N, Zhang P, Wang P, Peng R, Men R, Gao R, Lin R, Wang S, Bai S, Tan S, Zhu T, Li T, Liu T, Ge W, Deng X, Zhou X, Ren X, Zhang X, Wei X, Ren X, Liu X, Fan Y, Yao Y, Zhang Y, Wan Y, Chu Y, Liu Y, Cui Z, Zhang Z, Guo Z, Fan Z . Qwen2 technical report. 2024, arXiv preprint arXiv: 2407, 1067, 1: 2024

[38] Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, et al. The Llama 3 herd of models. 2024, arXiv preprint arXiv: 2407.21783

[39] Rivière M, Pathak S, Sessa P G, Hardin C, Bhupatiraju S, , . Gemma 2: improving open language models at a practical size. 2024, arXiv preprint arXiv: 2408.00118

[40] Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S, Bikel D, Blecher L, Canton Ferrer C, Chen M, Cucurull G, Esiobu D, Fernandes J, Fu J, Fu W, Fuller B, Gao C, Goswami V, Goyal N, Hartshorn A, Hosseini S, Hou R, Inan H, Kardas M, Kerkez V, Khabsa M, Kloumann I, Korenev A, Koura P S, Lachaux M A, Lavril T, Lee J, Liskovich D, Lu Y, Mao Y, Martinet X, Mihaylov T, Mishra P, Molybog I, Nie Y, Poulton A, Reizenstein J, Rungta R, Saladi K, Schelten A, Silva R, Smith E M, Subramanian R, Tan X E, Tang B, Taylor R, Williams A, Kuan J X, Xu P, Yan Z, Zarov I, Zhang Y, Fan A, Kambadur M, Narang S, Rodriguez A, Stojnic R, Edunov S, Scialom T. Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307.09288

[41] Zhang H, Dong Y, Xiao C, Oyamada M. Large language models as data preprocessors. In: Proceedings of Workshops at the 50th International Conference on Very Large Data Bases. 2024, 1–4

RIGHTS & PERMISSIONS

Higher Education Press