Extracting Named Entity Using Entity Labeling in Geological Text Using Deep Learning Approach

Abstract

Artificial intelligence (AI) is the key to mining and enhancing the value of big data, and knowledge graph is one of the important cornerstones of artificial intelligence, which is the core foundation for the integration of statistical and physical representations. Named entity recognition is a fundamental research task for building knowledge graphs, which needs to be supported by a high-quality corpus, and currently there is a lack of high-quality named entity recognition corpus in the field of geology, especially in Chinese. In this paper, based on the conceptual structure of geological ontology and the analysis of the characteristics of geological texts, a classification system of geological named entity types is designed with the guidance and participation of geological experts, a corresponding annotation specification is formulated, an annotation tool is developed, and the first named entity recognition corpus for the geological domain is annotated based on real geological reports. The total number of words annotated was 698 512 and the number of entities was 23 345. The paper also explores the feasibility of a model pre-annotation strategy and presents a statistical analysis of the distribution of technical and term categories across genres and the consistency of corpus annotation. Based on this corpus, a Lite Bi-directional Encoder Representations from Transformers (ALBERT)- Bi-directional Long Short-Term Memory (BiLSTM)-Conditional Random Fields (CRF) and ALBERT-BiLSTM models are selected for experiments, and the results show that the F1-scores of the recognition performance of the two models reach 0.75 and 0.65 respectively, providing a corpus basis and technical support for information extraction in the field of geology.

Keywords

ontology / geological reports / named entity recognition / geological corpus construction / semi-automated annotation platforms / deep learning

References

[1] Aone, C., Halverson, L., Hampton, T., et al., 1998. SRA: Description of the IE2 System Used for MUC-7. Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, April 29–May 1, Virginia

[2] Artstein R, Poesio M. Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 2008, 34(4): 555-596.

[3] Beniest A, Schellart W P A. Geological Map of the Scotia Sea Area Constrained by Bathymetry, Geological Data, Geophysical Data and Seismic Tomography Models from the Deep Mantle. Earth-Science Reviews, 2020, 22: 103391

[4] Bikel D M, Schwartz R, Weischedel R M. An Algorithm That Learns What’s in a Name. Machine Learning, 1999, 34(1): 211-231.

[5] Black, W. J., Rinaldi, F., Mowatt, D., 1998. FACILE: Description of the NE System Used for MUC-7. The Seventh Message Understanding Conference (MUC-7), April 29–May 1, Virginia

[6] Borthwick A E. A Maximum Entropy Approach to Named Entity Recognition, 1999, New York: New York University

[7] Cao Y, Zhu Q M, Peifeng Li P F. A Method for Constructing a Corpus of Factual Information about Chinese Events. Journal of Chinese Information Processing, 2013, 27(6): 38-44

[8] Carletta, J., 1996. Assessing Agreement on Classification Tasks: The Kappa Statistic. arXiv: cmp-lg/9602004. https://arxiv.org/abs/cmp-lg/9602004

[9] Chen, W., Zhang, Y., Isahara, H., 2006. Chinese Named Entity Recognition with Conditional Random Fields. The Fifth SIGHAN Workshop on Chinese Language Processing. 22–23 July 2006, Sydney

[10] Chu D P, Wan B, Li H, . Geological Entity Recognition Based on ELMO-CNN-BiLSTM-CRF Model. Earth Science, 2021, 46(8): 3039-3048. (in Chinese with English Abstract)

[11] Collobert R, Weston J, Bottou L, . Natural Language Processing (almost) from Scratch. arXiv: 1103.0398. Journal of Machine Learning Research, 2011, 12 2493-2537.

[12] Devlin, J., Chang, M. W., Lee, K., et al., 2018. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv: 1810.04805. https://arxiv.org/abs/1810.04805

[13] Enkhsaikhan M, Liu W, Holden E J, . Auto-Labelling Entities in Low-Resource Text: A Geological Case Study. Knowledge and Information Systems, 2021, 63(3): 695-715.

[14] Fan J, Shen S, Erwin D H, . A High-Resolution Summary of Cambrian to Early Triassic Marine Invertebrate Biodiversity. Science, 2020, 367(6475): 272-277.

[15] Feng L L, Li J H, Li P F. Corpus Construction Method of Technology and Terminology for National Defense Science and Technology. Journal of Chinese Information Processing, 2020, 34(8): 41-50

[16] Fu G H, Luke K K. Chinese Named Entity Recognition Using Lexicalized HMMS. ACM SIGKDD Explorations Newsletter, 2005, 7(1): 19-25.

[17] Hochreiter S, Schmidhuber J. Long Short-Term Memory. Neural Computation, 1997, 9(8): 1735-1780.

[18] Hou L L, Zhang J, Wu O, . Method and Dataset Entity Mining in Scientific Literature: A CNN + BiLSTM Model with Self-Attention. Knowledge-Based Systems, 2022, 235: 107621

[19] Hripcsak G, Rothschild A S. Agreement, the F-Measure, and Reliability in Information Retrieval. Journal of the American Medical Informatics Association, 2005, 12(3): 296-298.

[20] Huang, M. S., Lai, P. T., Tsai, R. T. H., et al., 2019. Revised JNLPBA Corpus: A Revised Version of Biomedical NER Corpus for Relation Extraction Task. arXiv: 1901.10219. https://doi.org/10.1093/bib/bbaa054

[21] Humphreys, K., Gaizauskas, R., Azzam, S., et al., 1998. University of Sheffield: Description of the LaSIE-II System as Used for MUC-7. Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29–May 1, 1998

[22] Isozaki, H., Kazawa, H., 2002. Efficient Support Vector Classifiers for Named Entity Recognition. Proceedings of the 19th International Conference on Computational Linguistics-Volume 1. 24 August–1 September, 2002, Taipei. https://doi.org/10.3115/1072228.1072282

[23] Jin Y P, Wanvarie D, Le P T V. Learning from Noisy Out-of-Domain Corpus Using Dataless Classification. Natural Language Engineering, 2022, 28(1): 39-69.

[24] Krupka, G., IsoQuest, K., 2005. Description of the Nerowl Extractor System as Used for muc-7. Proceedings of the 7th Message Understanding Conference, Virginia

[25] Lan, Z. Z., Chen, M. D., Goodman, S., et al., 2019. ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations. arXiv: 1909.11942. https://arxiv.org/abs/1909.11942

[26] Li R, Mo T J, Yang J X, . Bridge Inspection Named Entity Recognition via BERT and Lexicon Augmented Machine Reading Comprehension Neural Model. Advanced Engineering Informatics, 2021, 50 101416

[27] Liu P, Guo Y M, Wang F L, . Chinese Named Entity Recognition: The State of the Art. Neurocomputing, 2022, 473 37-53.

[28] Liu W C, Chunju Z, Chen W, . Geological Time Information Extraction from Chinese Text Based on BiLSTM-CRF. Advances in Earth Science, 2021, 36(2): 211-220. (in Chinese with English Abstract)

[29] Ma K, Tian M, Tan Y J, . What is this Article About? Generative Summarization with the BERT Model in the Geosciences Domain. Earth Science Informatics, 2022, 15(1): 21-36.

[30] Ma X G, Ma C, Wang C B. A New Structure for Representing and Tracking Version Information in a Deep Time Knowledge Graph. Computers & Geosciences, 2020, 145: 104620

[31] Ma, X. Z., Hovy, E., 2016. End-to-End Sequence Labeling via Bi-Directional LSTM-CNNS-CRF. arXiv: 1603.01354. https://arxiv.org/abs/1603.01354

[32] McCallum A, Li W. Early Results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons. Proceedings of the Seventh Conference on Natural Language lLearning at HLT-NAACL 2003, 2003, Morristown: Association for Computational Linguistics Edmonton

[33] Mo T J, Li R, Yang J X, . Construction of Named Entity Recognition Corpus in the Field of Periodic Inspection of Highways and Bridges. Computer Applications, 2020, 40(S1): 103-108. (in Chinese with English Abstract)

[34] Nieh E H, Schottdorf M, Freeman N W, . Geometry of Abstract Learned Knowledge in the Hippocampus. Nature, 2021, 595(7865): 80-84.

[35] Ogren P V, Savova G, Chute C. Constructing Evaluation Corpora for Automated Clinical Named Entity Recognition. LREC, 2008, 8 3143-3150.

[36] Peters, M. E., Neumann, M., Iyyer, M., et al., 2018. Deep Contextualized Word Representations. arXiv: 1802.05365. https://arxiv.org/abs/1802.05365

[37] Qiu Q J, Xie Z, Wu L A, . GNER: A Generative Model for Geological Named Entity Recognition without Labeled Data Using Deep Learning. Earth and Space Science, 2019, 6(6): 931-946.

[38] Qiu Q J, Xie Z, Wu L, . BiLSTM-CRF for Geological Named Entity Recognition from the Geoscience Literature. Earth Science Informatics, 2019, 12(4): 565-579.

[39] Qiu Q J, Xie Z, Wu L, . Geoscience Keyphrase Extraction Algorithm Using Enhanced Word Embedding. Expert Systems with Applications, 2019, 125: 157-169.

[40] Qu C Y, Yi G, Yang J F, . Construction of Annotated Corpus of Named Entities in Chinese Electronic Medical Records. High Technology Letters, 2015, 25(2): 143-150. (in Chinese with English Abstract)

[41] Schiffries, C. M., Wang, C., Hazen, R., et al., 2020. The Deep-Time Digital Earth Program: Data Driven Discovery in the Geosciences. AGU Fall Meeting 2020, 1–17 December, online

[42] Shan Y D, Wang H J, Huang H, . Research on Named Entity Recognition Model Based on Attention Mechanism. Frontiers in Chemistry, 2019, 11: 958002

[43] Verhagen, M., Saurí, R., Caselli, T., et al., 2010. SemEval-2010 Task 13: TempEval-2. Proceedings of the 5th International Workshop on Semantic Evaluation. July 15–16, 2010, Los Angeles. https://doi.org/10.5555/1859664.1859674

[44] Wang C S, Hazen R M, Cheng Q M, . The Deep-Time Digital Earth Program: Data-Driven Discovery in Geosciences. National Science Review, 2021, 8(9): nwab027

[45] Wang J B, Lu F, Wu S, . Construction Method of Geographic Entity Relationship Corpus Based on Automatic Return Bid. Journal of Geo-Information Science, 2018, 20 7 871-879

[46] Wu L A, Xue L, Li C L, . A Knowledge-Driven Geospatially Enabled Framework for Geological Big Data. ISPRS International Journal of Geo-Information, 2017, 6(6): 166

[47] Wu T, Li M Y, Kong F. Construction of a Corpus Based on Synonymous Reasoning of Subordinate Relations of Text-Level Entities. Journal of Chinese Information Processing, 2020, 34 4 38-46

[48] Xing D, Rao G Q, Xun E D, . Construction of a Collocation Library of Preposition Structure Based on a Large-Scale Corpus. Journal of Chinese Information Processing, 2020, 34(11): 1-8. (in Chinese with English Abstract)

[49] Xu J J, He H F, Sun X, . Cross-Domain and Semisupervised Named Entity Recognition in Chinese Social Media: A Unified Model. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018, 26(11): 2142-2152.

[50] Yang J F, Guan Y, He B, . Construction of a Corpus of Named Entities and Entity Relationships in Chinese Electronic Medical Records. Journal of Software, 2016, 27(11): 2725-2746. (in Chinese with English Abstract)

[51] Zaitouny A, Small M, Hill J, . Fast Automatic Detection of Geological Boundaries from Multivariate Log Data Using Recurrence. Computers & Geosciences, 2020, 135 104362

[52] Zhang K L, Zhao X, Guan T F, . Construction and Application of Entity and Relationship Labeling Platform for Medical Text. Journal of Chinese Information Processing, 2020, 34(6): 36-44. (in Chinese with English Abstract)

[53] Zhang N X, Li F, Xu G L, . Chinese NER Using Dynamic Meta-Embeddings. IEEE Access, 2019, 7 64450-64459.

[54] Zhang Q, Sun Y, Zhang L L, . Named Entity Recognition Method in Health Preserving Field Based on BERT. Procedia Computer Science, 2021, 183: 212-220.

[55] Zhang X Y, Ye P, Wang S, . Recognition Method of Geological Entities Based on Deep Belief Network. Acta Petrologica Sinica, 2018, 34(2): 343-351. (in Chinese with English Abstract)

[56] Zhang, Y., Yang, J., 2018. Chinese NER Using Lattice LSTM. arXiv: 1805.02023. https://arxiv.org/abs/1805.02023

[57] Zhou C H, Wang H, Wang C S, . Geoscience Knowledge Graph in the Big Data Era. Science China Earth Sciences, 2021, 64(7): 1105-1114.

[58] Zhou G D, Su J. Named Entity Recognition Using an HMM-Based Chunk Tagger. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, 2002, New York: ACM, 473-480