Large language models for generative information extraction: a survey

Abstract

Information Extraction (IE) aims to extract structural knowledge from plain natural language texts. Recently, generative Large Language Models (LLMs) have demonstrated remarkable capabilities in text understanding and generation. As a result, numerous works have been proposed to integrate LLMs for IE tasks based on a generative paradigm. To conduct a comprehensive systematic review and exploration of LLM efforts for IE tasks, in this study, we survey the most recent advancements in this field. We first present an extensive overview by categorizing these works in terms of various IE subtasks and techniques, and then we empirically analyze the most advanced methods and discover the emerging trend of IE tasks with LLMs. Based on a thorough review conducted, we identify several insights in technique and promising research directions that deserve further exploration in future studies. We maintain a public repository and consistently update related works and resources on GitHub (LLM4IE repository).

Graphical abstract

Keywords

information extraction / large language models / review

References

[1] Zhong L, Wu J, Li Q, Peng H, Wu X. A comprehensive survey on automatic knowledge graph construction. ACM Computing Surveys, 2024, 56( 4): 94

[2] Fu C, Chen T, Qu M, Jin W, Ren X. Collaborative policy learning for open knowledge graph reasoning. In: Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019, 2672−2681

[3] Srihari R K, Li W. Information extraction supported question answering. In: Proceedings of the 8th Text REtrieval Conference. 1999

[4] Lu Y, Liu Q, Dai D, Xiao X, Lin H, Han X, Sun L, Wu H. Unified structure generation for universal information extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 5755−5772

[5] Wang X, Zhou W, Zu C, Xia H, Chen T, Zhang Y, Zheng R, Ye J, Zhang Q, Gui T, Kang J, Yang J, Li S, Du C. InstructUIE: multi-task instruction tuning for unified information extraction. 2023, arXiv preprint arXiv: 2304.08085

[6] Guo Y, Li Z, Jin X, Liu Y, Zeng Y, Liu W, Li X, Yang P, Bai L, Guo J, Cheng X. Retrieval-augmented code generation for universal information extraction. 2023, arXiv preprint arXiv: 2311.02962

[7] Zhong Y, Xu T, Luo P. Contextualized hybrid prompt-tuning for generation-based event extraction. In: Proceedings of the 16th International Conference on Knowledge Science, Engineering and Management. 2023, 374−386

[8] Zhou S, Yu B, Sun A, Long C, Li J, Sun J. A survey on neural open information extraction: current status and future directions. In: Proceedings of the 31st International Joint Conference on Artificial Intelligence. 2022, 5694−5701

[9] OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, , . GPT-4 technical report. 2023, arXiv preprint arXiv: 2303.08774

[10] Liu Q, He Y, Lian D, Zheng Z, Xu T, Liu C, Chen E. UniMEL: a unified framework for multimodal entity linking with large language models. 2024, arXiv preprint arXiv: 2407.16160

[11] Peng W, Li G, Jiang Y, Wang Z, Ou D, Zeng X, Xu D, Xu T, Chen E. Large language model based long-tail query rewriting in Taobao search. In: Companion Proceedings of the ACM Web Conference 2024. 2024, 20−28

[12] Zhang C, Zhang H, Wu S, Wu D, Xu T, Gao Y, Hu Y, Chen E. NoteLLM-2: multimodal large representation models for recommendation. 2024, arXiv preprint arXiv: 2405.16789

[13] Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 2023, 55( 9): 195

[14] Lyu Y, Li Z, Niu S, Xiong F, Tang B, Wang W, Wu H, Liu H, Xu T, Chen E. CRUD-RAG: a comprehensive Chinese benchmark for retrieval-augmented generation of large language models. 2024, arXiv preprint arXiv: 2401.17043

[15] Lyu Y, Niu Z, Xie Z, Zhang C, Xu T, Wang Y, Chen E. Retrieve-plan-generation: an iterative planning and answering framework for knowledge-intensive LLM generation. 2024, arXiv preprint arXiv: 2406.14979

[16] Jia P, Liu Y, Zhao X, Li X, Hao C, Wang S, Yin D. MILL: mutual verification with large language models for zero-shot query expansion. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 2498−2518

[17] Wang M, Zhao Y, Liu J, Chen J, Zhuang C, Gu J, Guo R, Zhao X. Large multimodal model compression via iterative efficient pruning and distillation. In: Companion Proceedings of the ACM Web Conference 2024. 2024, 235−244

[18] Fu Z, Li X, Wu C, Wang Y, Dong K, Zhao X, Zhao M, Guo H, Tang R. A unified framework for multi-domain CTR prediction via large language models. 2023, arXiv preprint arXiv: 2312.10743

[19] Jia P, Liu Y, Li X, Zhao X, Wang Y, Du Y, Han X, Wei X, Wang S, Yin D. G3: an effective and adaptive framework for worldwide geolocalization using large multi-modality models. 2024, arXiv preprint arXiv: 2405.14702

[20] Zhang C, Wu S, Zhang H, Xu T, Gao Y, Hu Y, Chen E. NoteLLM: a retrievable large language model for note recommendation. In: Companion Proceedings of the ACM Web Conference 2024. 2024, 170−179

[21] Wang X, Chen Z, Xie Z, Xu T, He Y, Chen E. In-context former: lightning-fast compressing context for large language model. 2024, arXiv preprint arXiv: 2406.13618

[22] Zhu J, Liu S, Yu Y, Tang B, Yan Y, Li Z, Xiong F, Xu T, Blaschko M B. FastMem: fast memorization of prompt improves context awareness of large language models. 2024, arXiv preprint arXiv: 2406.16069

[23] Wang L, Ma C, Feng X, Zhang Z, Yang H, Zhang J, Chen Z, Tang J, Chen X, Lin Y, Zhao W X, Wei Z, Wen J. A survey on large language model based autonomous agents. Frontiers of Computer Science, 2024, 18( 6): 186345

[24] Guan Z, Wu L, Zhao H, He M, Fan J. Enhancing collaborative semantics of language model-driven recommendations via graph-aware learning. 2024, arXiv preprint arXiv: 2406.13235

[25] Huang J, She Q, Jiang W, Wu H, Hao Y, Xu T, Wu F. QDMR-based planning-and-solving prompting for complex reasoning tasks. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 13395−13406

[26] Fu C, Dai Y, Luo Y, Li L, Ren S, Zhang R, Wang Z, Zhou C, Shen Y, Zhang M, Chen P, Li Y, Lin S, Zhao S, Li K, Xu T, Zheng X, Chen E, Ji R, Sun X. Video-MME: the first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. 2024, arXiv preprint arXiv: 2405.21075

[27] Li X, Su L, Jia P, Zhao X, Cheng S, Wang J, Yin D. Agent4Ranking: semantic robust ranking via personalized query rewriting using multi-agent LLM. 2023, arXiv preprint arXiv: 2312.15450

[28] Qi J, Zhang C, Wang X, Zeng K, Yu J, Liu J, Hou L, Li J, Bin X. Preserving knowledge invariance: rethinking robustness evaluation of open information extraction. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 5876−5890

[29] Chen W, Zhao L, Luo P, Xu T, Zheng Y, Chen E. HEProto: a hierarchical enhancing ProtoNet based on multi-task learning for few-shot named entity recognition. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023, 296−305

[30] Lou J, Lu Y, Dai D, Jia W, Lin H, Han X, Sun L, Wu H. Universal information extraction as unified semantic matching. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 13318−13326

[31] Josifoski M, De Cao N, Peyrard M, Petroni F, West R. GenIE: generative information extraction. In: Proceedings of 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022, 4626−4643

[32] Sainz O, García-Ferrero I, Agerri R, de Lacalle O L, Rigau G, Agirre E. GoLLIE: annotation guidelines improve zero-shot information-extraction. In: Proceedings of the ICLR 2024. 2024

[33] Paolini G, Athiwaratkun B, Krone J, Ma J, Achille A, Anubhai R, dos Santos C N, Xiang B, Soatto S. Structured prediction as translation between augmented natural languages. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[34] Gan C, Zhang Q, Mori T. GIELLM: Japanese general information extraction large language model utilizing mutual reinforcement effect. 2023, arXiv preprint arXiv: 2311.06838

[35] Fei H, Wu S, Li J, Li B, Li F, Qin L, Zhang M, Zhang M, Chua T S. LasUIE: unifying information extraction with latent adaptive structure-aware generative language model. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1125

[36] Li P, Sun T, Tang Q, Yan H, Wu Y, Huang X, Qiu X. CodeIE: large code generation models are better few-shot information extractors. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 15339−15353

[37] Yan H, Gui T, Dai J, Guo Q, Zhang Z, Qiu X. A unified generative framework for various NER subtasks. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021, 5808−5822

[38] Huang K H, Tang S, Peng N. Document-level entity-based extraction as template generation. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 5257−5269

[39] Cabot P L H, Navigli R. REBEL: relation extraction by end-to-end language generation. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2021. 2021, 2370−2381

[40] Wei X, Cui X, Cheng N, Wang X, Zhang X, Huang S, Xie P, Xu J, Chen Y, Zhang M, Jiang Y, Han W. ChatIE: zero-shot information extraction via chatting with ChatGPT. 2023, arXiv preprint arXiv: 2302.10205

[41] Wang X, Li S, Ji H. Code4Struct: code generation for few-shot event structure prediction. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 3640−3663

[42] Wang S, Sun X, Li X, Ouyang R, Wu F, Zhang T, Li J, Wang G. GPT-NER: named entity recognition via large language models. 2023, arXiv preprint arXiv: 2304.10428

[43] Ashok D, Lipton Z C. PromptNER: prompting for named entity recognition. 2023, arXiv preprint arXiv: 2305.15444

[44] Xu X, Zhu Y, Wang X, Zhang N. How to unleash the power of large language models for few-shot relation extraction? In: Proceedings of the 4th Workshop on Simple and Efficient Natural Language Processing. 2023, 190−200

[45] Nasar Z, Jaffry S W, Malik M K. Named entity recognition and relation extraction: state-of-the-art. ACM Computing Surveys, 2022, 54( 1): 20

[46] Ye H, Zhang N, Chen H, Chen H. Generative knowledge graph construction: a review. In: Proceedings of 2022 Conference on Empirical Methods in Natural Language Processing. 2022, 1−17

[47] Foppiano L, Lambard G, Amagasa T, Ishii M. Mining experimental data from materials science literature with large language models: an evaluation study. Science and Technology of Advanced Materials: Methods, 2024, 4( 1): 2356506

[48] Liu H, Xue W, Chen Y, Chen D, Zhao X, Wang K, Hou L, Li R, Peng W. A survey on hallucination in large vision-language models. 2024, arXiv preprint arXiv: 2402.00253

[49] Sahoo P, Singh A K, Saha S, Jain V, Mondal S, Chadha A. A systematic survey of prompt engineering in large language models: techniques and applications. 2024, arXiv preprint arXiv: 2402.07927

[50] Xu D, Zhang Z, Zhu Z, Lin Z, Liu Q, Wu X, Xu T, Wang W, Ye Y, Zhao X, Chen E, Zheng Y. Editing factual knowledge and explanatory ability of medical large language models. 2024, arXiv preprint arXiv: 2402.18099

[51] Yuan S, Yang D, Liang J, Li Z, Liu J, Huang J, Xiao Y. Generative entity typing with curriculum learning. In: Proceedings of 2022 Conference on Empirical Methods in Natural Language Processing. 2022, 3061−3073

[52] Feng Y, Pratapa A, Mortensen D. Calibrated seq2seq models for efficient and generalizable ultra-fine entity typing. In: Proceedings of the Findings of the Association for Computational Linguistics. 2023, 15550−15560

[53] Cui L, Wu Y, Liu J, Yang S, Zhang Y. Template-based named entity recognition using BART. In: Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021, 1835−1845

[54] Zhang S, Shen Y, Tan Z, Wu Y, Lu W. De-bias for generative extraction in unified NER task. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 808−818

[55] Wang L, Li R, Yan Y, Yan Y, Wang S, Wu W, Xu W. InstructionNER: a multi-task instruction-based generative framework for few-shot NER. 2022, arXiv preprint arXiv: 2203.03903

[56] Xia Y, Zhao Y, Wu W, Li S. Debiasing generative named entity recognition by calibrating sequence likelihood. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 1137−1148

[57] Cai C, Wang Q, Liang B, Qin B, Yang M, Wong K F, Xu R. In-context learning for few-shot multimodal named entity recognition. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 2969−2979

[58] Hu X, Jiang Y, Liu A, Huang Z, Xie P, Huang F, Wen L, Yu P S. Entity-to-text based data augmentation for various named entity recognition tasks. In: Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023. 2023, 9072−9087

[59] Amalvy A, Labatut V, Dufour R. Learning to rank context for named entity recognition using a synthetic dataset. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 10372−10382

[60] Chen X, Li L, Qiao S, Zhang N, Tan C, Jiang Y, Huang F, Chen H. One model for all domains: collaborative domain-prefix tuning for cross-domain NER. In: Proceedings of the 32nd International Joint Conference on Artificial Intelligence. 2023, 559

[61] Zhang R, Li Y, Ma Y, Zhou M, Zou L. LLMaAA: making large language models as active annotators. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 13088−13103

[62] Ma Y, Cao Y, Hong Y, Sun A. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 10572−10601

[63] Xie T, Li Q, Zhang Y, Liu Z, Wang H. Self-improving for zero-shot named entity recognition with large language models. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 583−593

[64] Zhou W, Zhang S, Gu Y, Chen M, Poon H. UniversalNER: targeted distillation from large language models for open named entity recognition. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[65] Zhang X, Tan M, Zhang J, Zhu W. NAG-NER: a unified non-autoregressive generation framework for various NER tasks. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 676−686

[66] Su J, Yu H. Unified named entity recognition as multi-label sequence generation. In: Proceedings of 2023 International Joint Conference on Neural Networks. 2023, 1−8

[67] Ding Y, Li J, Wang P, Tang Z, Bowen Y, Zhang M. Rethinking negative instances for generative named entity recognition. In: Proceedings of the Findings of the Association for Computational Linguistics ACL 2024. 2024, 3461−3475

[68] Bogdanov S, Constantin A, Bernard T, Crabbé B, Bernard E. NuNER: entity recognition encoder pre-training via LLM-annotated data. 2024, arXiv preprint arXiv: 2402.15343

[69] Chen J, Lu Y, Lin H, Lou J, Jia W, Dai D, Wu H, Cao B, Han X, Sun L. Learning in-context learning for named entity recognition. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 13661−13675

[70] Zhang Z, Zhao Y, Gao H, Hu M. LinkNER: linking local named entity recognition models to large language models using uncertainty. In: Proceedings of the ACM Web Conference 2024. 2024, 4047−4058

[71] Tang X, Wang J, Su Q. Small language model is a good guide for large language model in Chinese entity relation extraction. 2024, arXiv preprint arXiv: 2402.14373

[72] Popovič N, Färber M. Embedded named entity recognition using probing classifiers. 2024, arXiv preprint arXiv: 2403.11747

[73] Heng Y, Deng C, Li Y, Yu Y, Li Y, Zhang R, Zhang C. ProgGen: generating named entity recognition datasets step-by-step with self-reflexive large language models. In: Proceedings of the Findings of the Association for Computational Linguistics ACL 2024. 2024, 15992−16030

[74] Mo Y, Yang J, Liu J, Zhang S, Wang J, Li Z. C-ICL: contrastive in-context learning for information extraction. 2024, arXiv preprint arXiv: 2402.11254

[75] Keloth V K, Hu Y, Xie Q, Peng X, Wang Y, Zheng A, Selek M, Raja K, Wei C H, Jin Q, Lu Z, Chen Q, Xu H. Advancing entity recognition in biomedicine via instruction tuning of large language models. Bioinformatics, 2024, 40( 4): btae163

[76] Kim S, Seo K, Chae H, Yeo J, Lee D. VerifiNER: verification-augmented NER via knowledge-grounded reasoning with large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 2441−2461

[77] Li Y, Ramprasad R, Zhang C. A simple but effective approach to improve structured language model output for information extraction. 2024, arXiv preprint arXiv: 2402.13364

[78] Oliveira V, Nogueira G, Faleiros T, Marcacini R. Combining prompt-based language models and weak supervision for labeling named entity recognition on legal documents. Artificial Intelligence and Law, 2024: 1-21

[79] Lu J, Yang Z, Wang Y, Liu X, Namee B M, Huang C. PaDeLLM-NER: parallel decoding in large language models for named entity recognition. 2024, arXiv preprint arXiv: 2402.04838

[80] Bölücü N, Rybinski M, Wan S. Impact of sample selection on in-context learning for entity extraction from scientific writing. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 5090−5107

[81] Liu J, Wang J, Huang H, Zhang R, Yang M, Zhao T. Improving LLM-based health information extraction with in-context learning. In: Proceedings of the 9th China Health Information Processing Conference. 2024, 49−59

[82] Wu C, Ke W, Wang P, Luo Z, Li G, Chen W. ConsistNER: towards instructive NER demonstrations for LLMs with the consistency of ontology and context. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 19234−19242

[83] Naguib M, Tannier X, Névéol A. Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting. 2024, arXiv preprint arXiv: 2402.12801

[84] Zaratiana U, Tomeh N, Holat P, Charnois T. GliNER: generalist model for named entity recognition using bidirectional transformer. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 5364−5376

[85] Munnangi M, Feldman S, Wallace B, Amir S, Hope T, Naik A. On-the-fly definition augmentation of LLMs for biomedical NER. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 3833−3854

[86] Zhang M, Wang B, Fei H, Zhang M. In-context learning for few-shot nested named entity recognition. 2024, arXiv preprint arXiv: 2402.01182

[87] Yan F, Yu P, Chen X. LTNER: Large language model tagging for named entity recognition with contextualized entity marking. 2024, arXiv preprint arXiv: 2404.05624

[88] Jiang G, Luo Z, Shi Y, Wang D, Liang J, Yang D. ToNER: type-oriented named entity recognition with generative language model. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 16251−16262

[89] Nunes R O, Spritzer A S, Dal Sasso Freitas C, Balreira D S. Out of sesame street: a study of portuguese legal named entity recognition through in-context learning. In: Proceedings of the 26th International Conference on Enterprise Information Systems. 2024

[90] Hou W, Zhao W, Liu X, Guo W. Knowledge-enriched prompt for low-resource named entity recognition. ACM Transactions on Asian and Low-Resource Language Information Processing, 2024, 23( 5): 72

[91] Li M, Zhou H, Yang H, Zhang R. RT: a retrieving and chain-of-thought framework for few-shot medical named entity recognition. Journal of the American Medical Informatics Association, 2024, 13( 9): 1929–1938

[92] Jiang G, Ding Z, Shi Y, Yang D. P-ICL: point in-context learning for named entity recognition with large language models. 2024, arXiv preprint arXiv: 2405.04960

[93] Xie T, Zhang J, Zhang Y, Liang Y, Li Q, Wang H. Retrieval augmented instruction tuning for open ner with large language models. 2024, arXiv preprint arXiv:2406.17305

[94] Li J, Li H, Sun D, Wang J, Zhang W, Wang Z, Pan G. LLMs as bridges: reformulating grounded multimodal named entity recognition. In: Proceedings of the Findings of the Association for Computational Linguistics ACL 2024. 2024, 1302−1318

[95] Ye J, Xu N, Wang Y, Zhou J, Zhang Q, Gui T, Huang X. LLM-DA: data augmentation via large language models for few-shot named entity recognition. 2024, arXiv preprint arXiv: 2402.14568

[96] Li G, Wang P, Ke W. Revisiting large language models as zero-shot relation extractors. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 6877−6892

[97] Pang C, Cao Y, Ding Q, Luo P. Guideline learning for in-context information extraction. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 15372−15389

[98] Zhang K, Gutierrez B J, Su Y. Aligning instruction tasks unlocks large language models as zero-shot relation extractors. In: Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023. 2023, 794−812

[99] Wan Z, Cheng F, Mao Z, Liu Q, Song H, Li J, Kurohashi S. GPT-RE: in-context learning for relation extraction using large language models. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 3534−3547

[100] Ma M D, Wang X, Kung P N, Brantingham P J, Peng N, Wang W. STAR: boosting low-resource information extraction by structure-to-text data generation with large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 18751−18759

[101] Wang Q, Zhou K, Qiao Q, Li Y, Li Q. Improving unsupervised relation extraction by augmenting diverse sentence pairs. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 12136−12147

[102] Li B, Yu D, Ye W, Zhang J, Zhang S. Sequence generation with label augmentation for relation extraction. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 13043−13050

[103] Guo Q, Yang Y, Yan H, Qiu X, Zhang Z. DORE: document ordered relation extraction based on generative framework. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022. 2022, 3463−3474

[104] Ma X, Li J, Zhang M. Chain of thought with explicit evidence reasoning for few-shot relation extraction. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 2334−2352

[105] Zhou S, Meng Y, Jin B, Han J. Grasping the essentials: tailoring large language models for zero-shot relation extraction. 2024, arXiv preprint arXiv: 2402.11142

[106] Qi J, Ji K, Wang X, Yu J, Zeng K, Hou L, Li J, Xu B. Mastering the task of open information extraction with large language models and consistent reasoning environment. 2023, arXiv preprint arXiv: 2310.10590

[107] Li G, Wang P, Liu J, Guo Y, Ji K, Shang Z, Xu Z. Meta in-context learning makes large language models better zero and few-shot relation extractors. In: Proceedings of the 33rd International Joint Conference on Artificial Intelligence. 2024

[108] Otto W, Upadhyaya S, Dietze S. Enhancing software-related information extraction via single-choice question answering with large language models. 2024, arXiv preprint arXiv: 2404.05587

[109] Shi Z, Luo H. CRE-LLM: a domain-specific Chinese relation extraction framework with fine-tuned large language model. 2024, arXiv preprint arXiv: 2404.18085

[110] Li G, Wang P, Ke W, Guo Y, Ji K, Shang Z, Liu J, Xu Z. Recall, retrieve and reason: towards better in-context relation extraction. In: Proceedings of the 33rd International Joint Conference on Artificial Intelligence. 2024

[111] Li G, Xu Z, Shang Z, Liu J, Ji K, Guo Y. Empirical analysis of dialogue relation extraction with large language models. In: Proceedings of the 33rd International Joint Conference on Artificial Intelligence. 2024

[112] Efeoglu S, Paschke A. Retrieval-augmented generation-based relation extraction. 2024, arXiv preprint arXiv: 2404.13397

[113] Li Y, Peng X, Li J, Zuo X, Peng S, Pei D, Tao C, Xu H, Hong N. Relation extraction using large language models: a case study on acupuncture point locations. Journal of the American Medical Informatics Association, 2024: ocae233

[114] Fan Z, He S. Efficient data learning for open information extraction with pre-trained language models. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 13056−13063

[115] Kwak A S, Jeong C, Forte G, Bambauer D, Morrison C, Surdeanu M. Information extraction from legal wills: how well does GPT-4 do? In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 4336−4353

[116] Sun Q, Huang K, Yang X, Tong R, Zhang K, Poria S. Consistency guided knowledge retrieval and denoising in LLMs for zero-shot document-level relation triplet extraction. In: Proceedings of the ACM Web Conference 2024. 2024, 4407−4416

[117] Ozyurt Y, Feuerriegel S, Zhang C. In-context few-shot relation extraction via pre-trained language models. 2023, arXiv preprint arXiv: 2310.11085

[118] Xue L, Zhang D, Dong Y, Tang J. AutoRE: document-level relation extraction with large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 211−220

[119] Liu Y, Peng X, Du T, Yin J, Liu W, Zhang X. ERA-CoT: improving chain-of-thought through entity relationship analysis. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 8780−8794

[120] Li G, Ke W, Wang P, Xu Z, Ji K, Liu J, Shang Z, Luo Q. Unlocking instructive in-context learning with tabular prompting for relational triple extraction. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024

[121] Ding Z, Huang W, Liang J, Xiao Y, Yang D. Improving recall of large language models: a model collaboration approach for relational triple extraction. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 8890−8901

[122] Ni X, Li P, Li H. Unified text structuralization with instruction-tuned language models. 2023, arXiv preprint arXiv: 2303.14956

[123] Zaratiana U, Tomeh N, Holat P, Charnois T. An autoregressive text-to-graph framework for joint entity and relation extraction. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 19477−19487

[124] Peng L, Wang Z, Yao F, Wang Z, Shang J. MetaIE: distilling a meta model from LLM for all kinds of information extraction tasks. 2024, arXiv preprint arXiv: 2404.00457

[125] Atuhurra J, Dujohn S C, Kamigaito H, Shindo H, Watanabe T. Distilling named entity recognition models for endangered species from large language models. 2024, arXiv preprint arXiv: 2403.15430

[126] Tang X, Su Q, Wang J, Deng Z. CHisIEC: an information extraction corpus for ancient Chinese history. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 3192−3202

[127] Ben Veyseh A P, Lai V, Dernoncourt F, Nguyen T H. Unleash GPT-2 power for event detection. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021, 6271−6282

[128] Xia N, Yu H, Wang Y, Xuan J, Luo X. DAFS: a domain aware few shot generative model for event detection. Machine Learning, 2023, 112( 3): 1011–1031

[129] Cai Z, Kung P N, Suvarna A, Ma M, Bansal H, Chang B, Brantingham P J, Wang W, Peng N. Improving event definition following for zero-shot event detection. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024

[130] Li S, Ji H, Han J. Document-level event argument extraction by conditional generation. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021, 894−908

[131] Lu Y, Lin H, Xu J, Han X, Tang J, Li A, Sun L, Liao M, Chen S. Text2Event: controllable sequence-to-structure generation for end-to-end event extraction. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021, 2795−2806

[132] Zhou Y, Shen T, Geng X, Long G, Jiang D. ClarET: pre-training a correlation-aware context-to-event transformer for event-centric generation and classification. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 2559−2575

[133] Huang K H, Hsu I, Natarajan P, Chang K W, Peng N. Multilingual generative language models for zero-shot cross-lingual event argument extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 4633−4646

[134] Ma Y, Wang Z, Cao Y, Li M, Chen M, Wang K, Shao J. Prompt for extraction? PAIE: prompting argument interaction for event argument extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 6759−6774

[135] Liu X, Huang H, Shi G, Wang B. Dynamic prefix-tuning for generative template-based event extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 5216−5228

[136] Cai E, O’Connor B. A Monte Carlo language model pipeline for zero-shot sociopolitical event extraction. In: Proceedings of the NeurIPS 2023. 2023

[137] Luo L, Xu Y. Context-aware prompt for generation-based event argument extraction with diffusion models. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023, 1717−1725

[138] Lu D, Ran S, Tetreault J, Jaimes A. Event extraction as question generation and answering. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 1666−1688

[139] van Nguyen C, Man H, Nguyen T H. Contextualized soft prompts for extraction of event arguments. In: Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023. 2023, 4352−4361

[140] Hsu I H, Xie Z, Huang K, Natarajan P, Peng N. AMPERE: AMR-aware prefix for generation-based event argument extraction model. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 10976−10993

[141] Duan J, Liao X, An Y, Wang J. KeyEE: enhancing low-resource generative event extraction with auxiliary keyword sub-prompt. Big Data Mining and Analytics, 2024, 7( 2): 547–560

[142] Lin Z, Zhang H, Song Y. Global constraints with prompting for zero-shot event argument classification. In: Proceedings of the Findings of the Association for Computational Linguistics: EACL 2023. 2023, 2482−2493

[143] Liu W, Zhou L, Zeng D, Xiao Y, Cheng S, Zhang C, Lee G, Zhang M, Chen W. Beyond single-event extraction: towards efficient document-level multi-event argument extraction. In: Proceedings of the Findings of the Association for Computational Linguistics ACL 2024. 2024, 9470−9487

[144] Zhang X F, Blum C, Choji T, Shah S, Vempala A. ULTRA: unleash LLMs’ potential for event argument extraction through hierarchical modeling and pair-wise self-refinement. In: Proceedings of the Findings of the Association for Computational Linguistics ACL 2024. 2024

[145] Sun Z, Pergola G, Wallace B, He Y. Leveraging ChatGPT in pharmacovigilance event extraction: an empirical study. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics. 2024, 344−357

[146] Zhou H, Qian J, Feng Z, Hui L, Zhu Z, Mao K. LLMs learn task heuristics from demonstrations: a heuristic-driven prompting strategy for document-level event argument extraction. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 11972−11990

[147] Hsu I H, Huang K H, Boschee E, Miller S, Natarajan P, Chang K W, Peng N. DEGREE: a data-efficient generation-based event extraction model. In: Proceedings of 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022, 1890−1908

[148] Zhao G, Gong X, Yang X, Dong G, Lu S, Li S. DemoSG: demonstration-enhanced schema-guided generation for low-resource event extraction. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 1805−1816

[149] Gao J, Zhao H, Wang W, Yu C, Xu R. EventRL: enhancing event extraction with outcome supervision for large language models. 2024, arXiv preprint arXiv: 2402.11430

[150] Huang K H, Hsu I H, Parekh T, Xie Z, Zhang Z, Natarajan P, Chang K W, Peng N, Ji H. TextEE: benchmark, reevaluation, reflections, and future challenges in event extraction. In: Proceedings of the Findings of the Association for Computational Linguistics ACL 2024. 2024, 12804−12825

[151] Wang C, Liu X, Chen Z, Hong H, Tang J, Song D. DeepStruct: pretraining of language models for structure prediction. In: Proceedings of the Findings of the Association for Computational Linguistics: ACL 2022. 2022, 803−823

[152] Li J, Zhang Y, Liang B, Wong K F, Xu R. Set learning for generative information extraction. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 13043−13052

[153] Wei X, Chen Y, Cheng N, Cui X, Xu J, Han W. CollabKG: a learnable human-machine-cooperative information extraction toolkit for (event) knowledge graph construction. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024

[154] Wang J, Chang Y, Li Z, An N, Ma Q, Hei L, Luo H, Lu Y, Ren F. TechGPT-2.0: a large language model project to solve the task of knowledge graph construction. 2024, arXiv preprint arXiv: 2401.04507

[155] Xiao X, Wang Y, Xu N, Wang Y, Yang H, Wang M, Luo Y, Wang L, Mao W, Zeng D. YAYI-UIE: a chat-enhanced instruction tuning framework for universal information extraction. 2023, arXiv preprint arXiv: 2312.15548

[156] Xu J, Sun M, Zhang Z, Zhou J. ChatUIE: exploring chat-based unified information extraction using large language models. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 3146−3152

[157] Gui H, Yuan L, Ye H, Zhang N, Sun M, Liang L, Chen H. IEPile: unearthing large scale schema-conditioned information extraction corpus. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 127−146

[158] Guo Q, Guo Y, Zhao J. Diluie: constructing diverse demonstrations of in-context learning with large language model for unified information extraction. Neural Computing and Applications, 2024, 36( 22): 13491–13512

[159] Bi Z, Chen J, Jiang Y, Xiong F, Guo W, Chen H, Zhang N. CodeKGC: code language model for generative knowledge graph construction. ACM Transactions on Asian and Low-Resource Language Information Processing, 2024, 23( 3): 45

[160] Li Z, Zeng Y, Zuo Y, Ren W, Liu W, Su M, Guo Y, Liu Y, Lixiang L, Hu Z, Bai L, Li W, Liu Y, Yang P, Jin X, Guo J, Cheng X. KnowCoder: coding structured knowledge into LLMs for universal information extraction. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 8758−8779

[161] Li J, Jia Z, Zheng Z. Semi-automatic data enhancement for document-level relation extraction with distant supervision from large language models. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 5495−5505

[162] Tang R, Han X, Jiang X, Hu X. Does synthetic data generation of LLMs help clinical text mining? 2023, arXiv preprint arXiv: 2303.04360

[163] Meoni S, De la Clergerie E, Ryffel T. Large language models as instructors: a study on multilingual clinical entity extraction. In: Proceedings of the 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. 2023, 178−190

[164] Evans J, Sadruddin S, D’Souza J. Astro-NER–astronomy named entity recognition: is GPT a good domain expert annotator? 2024, arXiv preprint arXiv: 2405.02602

[165] Naraki Y, Yamaki R, Ikeda Y, Horie T, Naganuma H. Augmenting NER datasets with LLMs: towards automated and refined annotation. 2024, arXiv preprint arXiv: 2404.01334

[166] Chen F, Feng Y. Chain-of-thought prompt distillation for multimodal named entity recognition and multimodal relation extraction. 2023, arXiv preprint arXiv: 2306.14122

[167] Li J, Li H, Pan Z, Sun D, Wang J, Zhang W, Pan G. Prompting ChatGPT in MNER: enhanced multimodal named entity recognition with auxiliary refined knowledge. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 2787−2802

[168] Josifoski M, Sakota M, Peyrard M, West R. Exploiting asymmetry for synthetic training data generation: synthIE and the case of information extraction. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 1555−1574

[169] Wadhwa S, Amir S, Wallace B. Revisiting relation extraction in the era of large language models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 15566−15589

[170] Yuan C, Xie Q, Ananiadou S. Zero-shot temporal relation extraction with ChatGPT. In: Proceedings of the 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. 2023, 92−102

[171] Bian J, Zheng J, Zhang Y, Zhu S. Inspire the large language model by external knowledge on biomedical named entity recognition. 2023, arXiv preprint arXiv: 2309.12278

[172] Hu Y, Chen Q, Du J, Peng X, Keloth V K, Zuo X, Zhou Y, Li Z, Jiang X, Lu Z, Roberts K, Xu H. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association, 2024, 31( 9): 1812–1820

[173] Shao W, Zhang R, Ji P, Fan D, Hu Y, Yan X, Cui C, Tao Y, Mi L, Chen L. Astronomical knowledge entity extraction in astrophysics journal articles via large language models. Research in Astronomy and Astrophysics, 2024, 24( 6): 065012

[174] Geng S, Josifosky M, Peyrard M, West R. Flexible grammar-based constrained decoding for language models. 2023, arXiv preprint arXiv: 2305.13971

[175] Liu T, Jiang Y E, Monath N, Cotterell R, Sachan M. Autoregressive structured prediction with language models. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022. 2022, 993−1005

[176] Chen X, Li L, Deng S, Tan C, Xu C, Huang F, Si L, Chen H, Zhang N. LightNER: a lightweight tuning paradigm for low-resource NER via pluggable prompting. In: Proceedings of the 29th International Conference on Computational Linguistics. 2022, 2374−2387

[177] Nie B, Shao Y, Wang Y. Know-adapter: towards knowledge-aware parameter-efficient transfer learning for few-shot named entity recognition. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 9777−9786

[178] Zhang J, Liu X, Lai X, Gao Y, Wang S, Hu Y, Lin Y. 2INER: instructive and in-context learning on few-shot named entity recognition. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 3940−3951

[179] Monajatipoor M, Yang J, Stremmel J, Emami M, Mohaghegh F, Rouhsedaghat M, Chang K W. LLMs in biomedicine: a study on clinical named entity recognition. 2024, arXiv preprint arXiv: 2404.07376

[180] Dunn A, Dagdelen J, Walker N, Lee S, Rosen A S, Ceder G, Persson K, Jain A. Structured information extraction from complex scientific text with fine-tuned large language models. 2022, arXiv preprint arXiv: 2212.05238

[181] Cheung J, Zhuang Y, Li Y, Shetty P, Zhao W, Grampurohit S, Ramprasad R, Zhang C. POLYIE: a dataset of information extraction from polymer material scientific literature. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024

[182] Dagdelen J, Dunn A, Lee S, Walker N, Rosen A S, Ceder G, Persson K A, Jain A. Structured information extraction from scientific text with large language models. Nature Communications, 2024, 15( 1): 1418

[183] Ma M D, Taylor A, Wang W, Peng N. DICE: data-efficient clinical event extraction with generative models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 15898−15917

[184] Hu Y, Ameer I, Zuo X, Peng X, Zhou Y, Li Z, Li Y, Li J, Jiang X, Xu H. Zero-shot clinical entity recognition using ChatGPT. 2023, arXiv preprint arXiv: 2303.16416

[185] Agrawal M, Hegselmann S, Lang H, Kim Y, Sontag D. Large language models are few-shot clinical information extractors. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022, 1998−2022

[186] Labrak Y, Rouvier M, Dufour R. A zero-shot and few-shot study of instruction-finetuned large language models applied to clinical and biomedical tasks. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024

[187] Gutiérrez B J, McNeal N, Washington C, Chen Y, Li L, Sun H, Su Y. Thinking about GPT-3 in-context learning for biomedical IE? Think again. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022. 2022, 4497−4512

[188] Biana J, Zhai W, Huang X, Zheng J, Zhu S. VANER: leveraging large language model for versatile and adaptive biomedical named entity recognition. 2024, arXiv preprint arXiv: 2404.17835

[189] González-Gallardo C E, Boros E, Girdhar N, Hamdi A, Moreno J G, Doucet A. yes but.. can ChatGPT identify entities in historical documents? In: Proceedings of 2023 ACM/IEEE Joint Conference on Digital Libraries. 2023, 184−189

[190] Xie T, Li Q, Zhang J, Zhang Y, Liu Z, Wang H. Empirical study of zero-shot NER with ChatGPT. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 7935−7956

[191] Gao J, Zhao H, Yu C, Xu R. Exploring the feasibility of ChatGPT for event extraction. 2023, arXiv preprint arXiv: 2303.03836

[192] Gui H, Zhang J, Ye H, Zhang N. InstructIE: a Chinese instruction-based information extraction dataset. 2023, arXiv preprint arXiv: 2305.11527

[193] Han R, Peng T, Yang C, Wang B, Liu L, Wan X. Is information extraction solved by ChatGPT? an analysis of performance, evaluation criteria, robustness and errors. 2023, arXiv preprint arXiv: 2305.14450

[194] Katz U, Vetzler M, Cohen A, Goldberg Y. NERetrieve: dataset for next generation named entity recognition and retrieval. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 3340−3354

[195] Li B, Fang G, Yang Y, Wang Q, Ye W, Zhao W, Zhang S. Evaluating ChatGPT’s information extraction capabilities: an assessment of performance, explainability, calibration, and faithfulness. 2023, arXiv preprint arXiv: 2304.11633

[196] Fei H, Zhang M, Zhang M, Chua T S. XNLP: an interactive demonstration system for universal structured NLP. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024

[197] Liu C, Zhao F, Kang Y, Zhang J, Zhou X, Sun C, Kuang K, Wu F. RexUIE: a recursive method with explicit schema instructor for universal information extraction. In: Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. 2023, 15342−15359

[198] Zhu T, Ren J, Yu Z, Wu M, Zhang G, Qu X, Chen W, Wang Z, Huai B, Zhang M. Mirror: a universal framework for various information extraction tasks. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 8861−8876

[199] Bhagavatula C, Le Bras R, Malaviya C, Sakaguchi K, Holtzman A, Rashkin H, Downey D, Yih S W T, Choi Y. Abductive commonsense reasoning. In: Proceedings of the 8th International Conference on Learning Representations. 2020

[200] OpenAI. Introduce ChatGPT. See openai.com/index/chatgpt/ website, 2023

[201] Whitehouse C, Choudhury M, Aji A F. LLM-powered data augmentation for enhanced cross-lingual performance. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023

[202] Wu L, Zheng Z, Qiu Z, Wang H, Gu H, Shen T, Qin C, Zhu C, Zhu H, Liu Q, Xiong H, Chen E. A survey on large language models for recommendation. World Wide Web, 2024, 27( 5): 60

[203] Chen Y, Wang Q, Wu S, Gao Y, Xu T, Hu Y. TOMGPT: reliable text-only training approach for cost-effective multi-modal large language model. ACM Transactions on Knowledge Discovery from Data, 2024, 18( 7): 171

[204] Luo P, Xu T, Liu C, Zhang S, Xu L, Li M, Chen E. Bridging gaps in content and knowledge for multimodal entity linking. In: Proceedings of the ACM Multimedia 2024. 2024

[205] Yang H, Zhao X, Huang S, Li Q, Xu G. LATEX-GCL: large language models (LLMs)-based data augmentation for text-attributed graph contrastive learning. 2024, arXiv preprint arXiv: 2409.01145

[206] Gao Y, Xiong Y, Gao X, Jia K, Pan J, Bi Y, Dai Y, Sun J, Wang M, Wang H. Retrieval-augmented generation for large language models: a survey. 2023, arXiv preprint arXiv: 2312.10997

[207] Gao L, Biderman S, Black S, Golding L, Hoppe T, Foster C, Phang J, He H, Thite A, Nabeshima N, Presser S, Leahy C. The pile: an 800GB dataset of diverse text for language modeling. 2020, arXiv preprint arXiv: 2101.00027

[208] Marvin G, Hellen N, Jjingo D, Nakatumba-Nabende J. Prompt engineering in large language models. In: Jacob I J, Piramuthu S, Falkowski-Gilski P. Data Intelligence and Cognitive Informatics. Singapore: Springer, 2024, 387−402

[209] Zhao H, Zheng S, Wu L, Yu B, Wang J. LANE: logic alignment of non-tuning large language models and online recommendation systems for explainable reason generation. 2024, arXiv preprint arXiv: 2407.02833

[210] Zheng Z, Qiu Z, Hu X, Wu L, Zhu H, Xiong H. Generative job recommendations with large language model. 2023, arXiv preprint arXiv: 2307.02157

[211] Wu L, Qiu Z, Zheng Z, Zhu H, Chen E. Exploring large language model for graph data understanding in online job recommendations. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 9178−9186

[212] Zheng Z, Chao W, Qiu Z, Zhu H, Xiong H. Harnessing large language models for text-rich sequential recommendation. In: Proceedings of the ACM Web Conference 2024. 2024, 3207−3216

[213] Chen B, Zhang Z, Langrené N, Zhu S. Unleashing the potential of prompt engineering in large language models: a comprehensive review. 2023, arXiv preprint arXiv: 2310.14735

[214] Zhao Z, Lin F, Zhu X, Zheng Z, Xu T, Shen S, Li X, Yin Z, Chen E. DynLLM: when large language models meet dynamic graph recommendation. 2024, arXiv preprint arXiv: 2405.07580

[215] Wang J, Shi E, Yu S, Wu Z, Ma C, Dai H, Yang Q, Kang Y, Wu J, Hu H, Yue C, Zhang H, Liu Y, Pan Y, Liu Z, Sun L, Li X, Ge B, Jiang X, Zhu D, Yuan Y, Shen D, Liu T, Zhang S. Prompt engineering for healthcare: methodologies and applications. 2023, arXiv preprint arXiv: 2304.14670

[216] Xu D, Zhang Z, Lin Z, Wu X, Zhu Z, Xu T, Zhao X, Zheng Y, Chen E. Multi-perspective improvement of knowledge graph completion with large language models. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 11956−11968

[217] Li X, Zhou J, Chen W, Xu D, Xu T, Chen E. Visualization recommendation with prompt-based reprogramming of large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 13250−13262

[218] Liu C, Xie Z, Zhao S, Zhou J, Xu T, Li M, Chen E. Speak from heart: an emotion-guided LLM-based multimodal method for emotional dialogue generation. In: Proceedings of 2024 International Conference on Multimedia Retrieval. 2024, 533−542

[219] Peng W, Xu D, Xu T, Zhang J, Chen E. Are GPT embeddings useful for ads and recommendation? In: Proceedings of the 16th International Conference on Knowledge Science, Engineering and Management. 2023, 151−162

[220] Peng W, Yi J, Wu F, Wu S, Zhu B B, Lyu L, Jiao B, Xu T, Sun G, Xie X. Are you copying my model? Protecting the copyright of large language models for EaaS via backdoor watermark. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 7653−7668

[221] Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi E H, Le Q V, Zhou D. Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1800

[222] Chu Z, Chen J, Chen Q, Yu W, He T, Wang H, Peng W, Liu M, Qin B, Liu T. A survey of chain of thought reasoning: advances, frontiers and future. 2023, arXiv preprint arXiv: 2309.15402

[223] Kojima T, Gu S S, Reid M, Matsuo Y, IwasawaY. Large language models are zero-shot reasoners. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 22199−22213

[224] Yin S, Fu C, Zhao S, Li K, Sun X, Xu T, Chen E. A survey on multimodal large language models. 2023, arXiv preprint arXiv: 2306.13549

[225] Willard B T, Louf R. Efficient guided generation for large language models. 2023, arXiv preprint arXiv: 2307.09702

[226] Beurer-Kellner L, Müller M N, Fischer M, Vechev M. Prompt sketching for large language models. 2023, arXiv preprint arXiv: 2311.04954

[227] Zheng L, Yin L, Xie Z, Huang J, Sun C, Yu C H, Cao S, Kozyrakis C, Stoica I, Gonzalez J E, Barrett C, Sheng Y. Efficiently programming large language models using SGLang. 2023, arXiv preprint arXiv: 2312.07104

[228] Huang J, Li C, Subudhi K, Jose D, Balakrishnan S, Chen W, Peng B, Gao J, Han J. Few-shot named entity recognition: an empirical baseline study. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 10408−10423

[229] Liu Z, Wu L, He M, Guan Z, Zhao H, Feng N. Dr.E bridges graphs with large language models through words. 2024, arXiv preprint arXiv: 2406.15504

[230] Guan Z, Zhao H, Wu L, He M, Fan J. LangTopo: aligning language descriptions of graphs with tokenized topological modeling. 2024, arXiv preprint arXiv: 2406.13250

[231] Zha R, Zhang L, Li S, Zhou J, Xu T, Xiong H, Chen E. Scaling up multivariate time series pre-training with decoupled spatial-temporal representations. In: Proceedings of the 40th IEEE International Conference on Data Engineering. 2024, 667−678

[232] Zhao L, Liu Q, Yue L, Chen W, Chen L, Sun R, Song C. COMI: COrrect and mitigate shortcut learning behavior in deep neural networks. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024, 218−228

[233] Lin F, Zhao Z, Zhu X, Zhang D, Shen S, Li X, Xu T, Zhang S, Chen E. When box meets graph neural network in tag-aware recommendation. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024, 1770−1780

[234] Liu Q, Wu X, Zhao X, Zhu Y, Xu D, Tian F, Zheng Y. When MOE meets LLMs: parameter efficient fine-tuning for multi-task medical applications. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024, 1104−1114

[235] Liu Q, Wu X, Zhao X, Zhu Y, Zhang Z, Tian F, Zheng Y. Large language model distilling medication recommendation model. 2024, arXiv preprint arXiv: 2402.02803

[236] Wang Y, Wang Y, Fu Z, Li X, Zhao X, Guo H, Tang R. LLM4MSR: an LLM-enhanced paradigm for multi-scenario recommendation. 2024, arXiv preprint arXiv: 2406.12529

[237] Zhao Z, Fan W, Li J, Liu Y, Mei X, Wang Y Q. Recommender systems in the era of large language models (LLMs). IEEE Transactions on Knowledge and Data Engineering, 2024, 36( 11): 6889–6907

[238] Qiao S, Ou Y, Zhang N, Chen X, Yao Y, Deng S, Tan C, Huang F, Chen H. Reasoning with language model prompting: a survey. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 5368−5393

[239] Sang E F T K, De Meulder F. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In: Proceedings of the 7th Conference on Natural Language Learning. 2003, 142−147

[240] Roth D, Yih W T. A linear programming formulation for global inference in natural language tasks. In: Proceedings of the 8th Conference on Computational Natural Language Learning. 2004, 1−8

[241] Walker C, Strassel S, Medero J, Maeda K. Ace 2005 multilingual training corpus-linguistic data consortium. See catalog.ldc.upenn.edu/LDC2006T06 website, 2005

[242] Doddington G R, Mitchell A, Przybocki M A, Ramshaw L A, Strassel S M, Weischedel R M. The automatic content extraction (ACE) program - tasks, data, and evaluation. In: Proceedings of the 4th International Conference on Language Resources and Evaluation. 2004, 837−840

[243] Li J, Sun Y, Johnson R J, Sciaky D, Wei C H, Leaman R, Davis A P, Mattingly C J, Wiegers T C, Lu Z. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016, 2016: baw068

[244] Derczynski L, Bontcheva K, Roberts I. Broad twitter corpus: a diverse named entity recognition resource. In: Proceedings of the 26th International Conference on Computational Linguistics. 2016, 1169−1179

[245] Karimi S, Metke-Jimenez A, Kemp M, Wang C. C_ADEC: a corpus of adverse drug event annotations. Journal of Biomedical Informatics, 2015, 55: 73–81

[246] Wang Z, Shang J, Liu L, Lu L, Liu J, Han J. CrossWeigh: training named entity tagger from imperfect annotations. In: Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019, 5153−5162

[247] Liu Z, Xu Y, Yu T, Dai W, Ji Z, Cahyawijaya S, Madotto A, Fung P. CrossNER: evaluating cross-domain named entity recognition. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 13452−13460

[248] Kumar A, Starly B. ”FabNER”: information extraction from manufacturing process science domain literature using named entity recognition. Journal of Intelligent Manufacturing, 2022, 33( 8): 2393–2407

[249] Ding N, Xu G, Chen Y, Wang X, Han X, Xie P, Zheng H, Liu Z. Few-NERD: a few-shot named entity recognition dataset. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021, 3198−3213

[250] Guan R, Man K L, Chen F, Yao S, Hu R, Zhu X, Smith J, Lim E G, Yue Y. FindVehicle and VehicleFinder: a NER dataset for natural language-based vehicle retrieval and a keyword-based cross-modal vehicle retrieval system. Multimedia Tools and Applications, 2024, 83: 24841–24874

[251] Kim J D, Ohta T, Tateisi Y, Tsujii J. GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics, 2003, 19( S1): i180–i182

[252] Chen P, Xu H, Zhang C, Huang R. Crossroads, buildings and neighborhoods: a dataset for fine-grained location recognition. In: Proceedings of 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022, 3329−3339

[253] Liu J, Pasupat P, Cyphers S, Glass J. Asgard: a portable architecture for multilingual dialogue systems. In: Proceedings of 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013, 8386−8390

[254] Tedeschi S, Navigli R. MultiNERD: a multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation). In: Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2022. 2022, 801−812

[255] Doğan R I, Leaman R, Lu Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. Journal of Biomedical Informatics, 2014, 47: 1–10

[256] Pradhan S, Moschitti A, Xue N, Ng H T, Björkelund A, Uryupina O, Zhang Y, Zhong Z. Towards robust linguistic analysis using OntoNotes. In: Proceedings of the 17th Conference on Computational Natural Language Learning. 2013, 143−152

[257] Pradhan S, Elhadad N, South B R, Martínez D, Christensen L, Vogel A, Suominen H, Chapman W W, Savova G. Task 1: ShARe/CLEF eHealth evaluation lab 2013. In: Proceedings of the Working Notes for CLEF 2013 Conference. 2013

[258] Mowery D L, Velupillai S, South B R, Christensen L, Martínez D, Kelly L, Goeuriot L, Elhadad N, Pradhan S, Savova G, Chapman W W. Task 2: ShARe/CLEF eHealth evaluation lab 2014. In: Proceedings of the Working Notes for CLEF 2014 Conference. 2014, 31−42

[259] Lu D, Neves L, Carvalho V, Zhang N, Ji H. Visual attention model for name tagging in multimodal social media. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018, 1990−1999

[260] Rijhwani S, Preotiuc-Pietro D. Temporally-informed analysis of named entity recognition. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 7605−7617

[261] Jiang H, Hua Y, Beeferman D, Roy D. Annotating the tweebank corpus on named entity recognition and building NLP models for social media analysis. In: Proceedings of the 13th Language Resources and Evaluation Conference. 2022, 7199−7208

[262] Zhang Q, Fu J, Liu X, Huang X. Adaptive co-attention network for named entity recognition in tweets. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2018, 5674−5681

[263] Ushio A, Barbieri F, Silva V, Neves L, Camacho-Collados J. Named entity recognition in twitter: a dataset and analysis on short-term temporal shifts. In: Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. 2022, 309−319

[264] Wang X, Tian J, Gui M, Li Z, Wang R, Yan M, Chen L, Xiao Y. WikiDiverse: a multimodal entity linking dataset with diversified contextual topics and entity types. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 4785−4797

[265] Derczynski L, Nichols E, van Erp M, Limsopatham N. Results of the WNUT2017 shared task on novel and emerging entity recognition. In: Proceedings of the 3rd Workshop on Noisy User-Generated Text. 2017, 140−147

[266] Gurulingappa H, Rajput A M, Roberts A, Fluck J, Hofmann-Apitius M, Toldo L. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedical Informatics, 2012, 45( 5): 885–892

[267] Yao Y, Ye D, Li P, Han X, Lin Y, Liu Z, Liu Z, Huang L, Zhou J, Sun M. DocRED: a large-scale document-level relation extraction dataset. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019, 764−777

[268] Zheng C, Wu Z, Feng J, Fu Z, Cai Y. MNRE: a challenge multimodal dataset for neural relation extraction with visual evidence in social media posts. In: Proceedings of 2021 IEEE International Conference on Multimedia and Expo. 2021, 1−6

[269] Riedel S, Yao L, McCallum A. Modeling relations and their mentions without labeled text. In: Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases. 2010, 148−163

[270] Stoica G, Platanios E A, Poczos B. Re-TACRED: addressing shortcomings of the TACRED dataset. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 13843−13850

[271] Luan Y, He L, Ostendorf M, Hajishirzi H. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In: Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing. 2018, 3219−3232

[272] Hendrickx I, Kim S N, Kozareva Z, NakovP, Séaghdha D Ó Padó S, Pennacchiotti M, Romano L, Szpakowicz S. SemEval-2010 task 8: multi-way classification of semantic relations between pairs of nominals. In: Proceedings of the 5th International Workshop on Semantic Evaluation. 2010, 33−38

[273] Zhang Y, Zhong V, Chen D, Angeli G, Manning C D. Position-aware attention and supervised data improve slot filling. In: Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing. 2017, 35−45

[274] Alt C, Gabryszak A, Hennig L. TACRED revisited: a thorough evaluation of the TACRED relation extraction task. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 1558−1569

[275] Satyapanich T, Ferraro F, Finin T. CASIE: extracting cybersecurity event information from text. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 8749−8757

[276] Kim J D, Wang Y, Takagi T, Yonezawa A. Overview of genia event task in BioNLP shared task 2011. In: Proceedings of BioNLP Shared Task 2011 Workshop. 2011, 7−15

[277] Kim J D, Wang Y, Yamamoto Y. The genia event extraction shared task, 2013 edition -overview. In: Proceedings of BioNLP Shared Task 2013 Workshop. 2013, 8−15

[278] Sun Z, Li J, Pergola G, Wallace B, John B, Greene N, Kim J, He Y. PHEE: a dataset for pharmacovigilance event extraction from text. In: Proceedings of 2022 Conference on Empirical Methods in Natural Language Processing. 2022, 5571−5587

[279] Ebner S, Xia P, Culkin R, Rawlins K, Van Durme B. Multi-sentence argument linking. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 8057−8077

[280] Zamai A, Zugarini A, Rigutini L, Ernandes M, Maggini M. Show less, instruct more: Enriching prompts with definitions and guidelines for zero-shot ner. 2024, arXiv preprint arXiv:2407.01272

[281] Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020, 7871−7880

[282] Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu P J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020, 21( 1): 140

[283] Xue L, Constant N, Roberts A, Kale M, Al-Rfou R, Siddhant A, Barua A, Raffel C. mT5: a massively multilingual pre-trained text-to-text transformer. In: Proceedings of 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021, 483−498

[284] Chung H W, Hou L, Longpre S, Zoph B, Tay Y, Fedus W, Li Y, Wang X, Dehghani M, Brahma S, Webson A, Gu S S, Dai Z, Suzgun M, Chen X, Chowdhery A, Castro-Ros A, Pellat M, Robinson K, Valter D, Narang S, Mishra G, Yu A, Zhao V, Huang Y, Dai A, Yu H, Petrov S, Chi E H, Dean J, Devlin J, Roberts A, Zhou D, Le Q V, Wei J. Scaling instruction-finetuned language models. 2022, arXiv preprint arXiv: 2210.11416

[285] Du Z, Qian Y, Liu X, Ding M, Qiu J, Yang Z, Tang J. GLM: general language model pretraining with autoregressive blank infilling. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 320−335

[286] Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Lacroix T, Rozière B, Goyal N, Hambro E, Azhar F, Rodriguez A, Joulin A, Grave E, Lample G. LLaMA: open and efficient foundation language models. 2023, arXiv preprint arXiv: 2302.13971

[287] Taori R, Gulrajani I, Zhang T, Dubois Y, Li X. Stanford alpaca: An instruction-following llama model. See github. com/tatsu-lab/stanford_alpaca website, 2023

[288] Chiang W L, Li Z, Lin Z, Sheng Y, Wu Z, Zhang H, Zheng L, Zhuang S, Zhuang Y, Gonzalez J E, Stoica I, Xing E P. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. See vicuna. lmsys. org websit, 2023

[289] Hugo T, Louis M, Kevin S, Albert P, Almahairi A, , . Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307.09288

[290] Rozière B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan X E, Adi Y, Liu J, Sauvestre R, Remez T, Rapin J, Kozhevnikov A, Evtimov I, Bitton J, Bhatt M, FerrerC C, Grattafiori A, Xiong W, Défossez A, Copet J, Azhar F, Touvron H, Martin L, Usunier N, Scialom T, Synnaeve G. Code llama: open foundation models for code. 2023, arXiv preprint arXiv: 2308.12950

[291] Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog, 2019, 1( 8): 9

[292] Brown T B, Mann B, Ryder N, Subbiah M, KaplanJ D, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler D M, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D. Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 159

[293] Wang B. Mesh-Transformer-JAX: model-parallel implementation of Transformer language model with JAX. See github.com/kingoflolz/mesh-transformer-jax website, 2021

[294] Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C L, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A, Schulman J, Hilton J, Kelton F, Miller L, Simens M, Askell A, Welinder P, Christiano P, Leike J, Lowe R. Training language models to follow instructions with human feedback. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 27730−27744

RIGHTS & PERMISSIONS

The Author(s) 2024. This article is published with open access at link.springer.com and journal.hep.com.cn