Ontology-Based BERT Model for Automated Information Extraction from Geological Hazard Reports

Abstract

Geological knowledge can provide support for knowledge discovery, knowledge inference and mineralization predictions of geological big data. Entity identification and relationship extraction from geological data description text are the key links for constructing knowledge graphs. Given the lack of publicly annotated datasets in the geology domain, this paper illustrates the construction process of geological entity datasets, defines the types of entities and interconceptual relationships by using the geological entity concept system, and completes the construction of the geological corpus. To address the shortcomings of existing language models (such as Word2vec and Glove) that cannot solve polysemous words and have a poor ability to fuse contexts, we propose a geological named entity recognition and relationship extraction model jointly with Bidirectional Encoder Representation from Transformers (BERT) pretrained language model. To effectively represent the text features, we construct a BERT- bidirectional gated recurrent unit network (BiGRU)-conditional random field (CRF)-based architecture to extract the named entities and the BERT-BiGRU-Attention-based architecture to extract the entity relations. The results show that the F1-score of the BERT-BiGRU-CRF named entity recognition model is 0.91 and the F1-score of the BERT-BiGRU-Attention relationship extraction model is 0.84, which are significant performance improvements when compared to classic language models (e.g., word2vec and Embedding from Language Models (ELMo)).

Keywords

ontology / BERT model / name entity recognition / relation extraction / knowledge graph

References

[1] Bengio Y, Ducharme R, Vincent P. A Neural Probabilistic Language Model. Journal of Machine Learning Research, 2003, 3: 1137-1155.

[2] Bojanowski P, Grave E, Joulin A, . Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 2017, 5: 135-146.

[3] Bouvrie, J., 2006. Notes on Convolutional Neural Networks, Neural Nets. http://web.mit.edu/jvb/www/papers/cnn_tutorial.pdf

[4] Cao, P., Chen, Y., Liu, K., et al., 2018. Adversarial Transfer Learning for Chinese Named Entity Recognition with Self-Attention Mechanism. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, October 31–November 4, Brusssels

[5] Chiticariu, L., Krishnamurthy, R., Li, Y. Y., et al., 2010. Domain Adaptation of Rule-Based Annotators for Named-Entity Recognition Tasks. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. October 9–11, 2010, Cambridge. 1002–1012. https://doi.org/10.5555/1870658.1870756

[6] Chiu J P, Nichols E. Named Entity Recognition with Bidirectional LSTM-CNNS. Transactions of the Association for Computational Linguistics, 2016, 4: 357-370.

[7] Deng, C., Jia, Y. T., Xu, H., et al., 2021. GAKG: A Multimodal Geoscience Academic Knowledge Graph. Proceedings of the 30th ACM International Conference on Information & Knowledge Management. November 1–5, 2021, Virtual Event, New York. 4445–4454. https://doi.org/10.1145/3459637.3482003

[8] Devlin, J., Chang, M. W., Lee, K., et al., 2018. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv: 1810.04805. https://arxiv.org/abs/1810.04805

[9] Enkhsaikhan M, Liu W, Holden E J, . Auto-Labelling Entities in Low-Resource Text: A Geological Case Study. Knowledge and Information Systems, 2021, 63(3): 695-715.

[10] Fan J, Shen S, Erwin D H, . A High-Resolution Summary of Cambrian to Early Triassic Marine Invertebrate Biodiversity. Science, 2020, 367(6475): 272-277.

[11] Fang W L, Ma L, Love P E D, . Knowledge Graph for Identifying Hazards on Construction Sites: Integrating Computer Vision with Ontology. Automation in Construction, 2020, 119: 103310

[12] Gayen, V., Sarkar, K., 2014. An HMM Based Named Entity Recognition System for Indian Languages: The JU System at ICON 2013. arXiv: 1405.7397. https://arxiv.org/abs/1405.7397

[13] Gers F A, Schmidhuber J, Cummins F. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 2000, 12(10): 2451-2471.

[14] Ghahabi O, Hernando J. Restricted Boltzmann Machines for Vector Representation of Speech in Speaker Recognition. Computer Speech & Language, 2018, 47 16-29.

[15] Hashimoto, K., Miwa, M., Tsuruoka, Y., et al., 2013. Simple Customization of Recursive Neural Networks for Semantic Relation Classification. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 18–21

[16] Hochreiter S, Schmidhuber J. Long Short-Term Memory. Neural Computation, 1997, 9(8): 1735-1780.

[17] Jauregi Unanue I, Zare Borzeshi E, Piccardi M. Recurrent Neural Networks with Specialized Word Embeddings for Health-Domain Named-Entity Recognition. Journal of Biomedical Informatics, 2017, 76: 102-109.

[18] Lai, T., Ji, H., Zhai, C. X., et al., 2021. Joint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective Inference. arXiv: 2105.13456. https://arxiv.org/abs/2105.13456

[19] Lawley C J M, Raimondo S, Chen T Y, . Geoscience Language Models and Their Intrinsic Evaluation. Applied Computing and Geosciences, 2022, 14: 100084

[20] Li P F, Mao K Z. Knowledge-Oriented Convolutional Neural Network for Causal Relation Extraction from Natural Language Texts. Expert Systems with Applications, 2019, 115 512-523.

[21] Lin, Y., Shen, S., Liu, Z., et al., 2016. Neural Relation Extraction with Selective Attention Over Instances. The 54th Annual Meeting of the Association for Computational Linguistics, August 7–12, Berlin

[22] Liu Z J, Yang M, Wang X L, . Entity Recognition from Clinical Texts via Recurrent Neural Network. BMC Medical Informatics and Decision Making, 2017, 17(Suppl2): 67

[23] Lü X, Xie Z, Xu D, . Chinese Named Entity Recognition in the Geoscience Domain Based on BERT. Earth and Space Science, 2022, 9 3 e2021EA002166

[24] Ma K, Tan Y J, Tian M, . Extraction of Temporal Information from Social Media Messages Using the BERT Model. Earth Science Informatics, 2022, 15 1 573-584.

[25] Ma K, Tan Y J, Xie Z, . Chinese Toponym Recognition with Variant Neural Structures from Social Media Messages Based on BERT Methods. Journal of Geographical Systems, 2022, 24(2): 143-169.

[26] Ma K, Tian M, Tan Y J, . What is This Article About? Generative Summarization with the BERT Model in the Geosciences Domain. Earth Science Informatics, 2022, 15(1): 21-36.

[27] Ma X G. Geo-Data Science: Leveraging Geoscience Research with Geoinformatics, Semantics and Open Data. Acta Geologica Sinica-English Edition, 2019, 93(S3): 44-47.

[28] Ma X G, Ma C, Wang C B. A New Structure for Representing and Tracking Version Information in a Deep Time Knowledge Graph. Computers & Geosciences, 2020, 145: 104620

[29] Mikolov, T., Chen, K., Corrado, G., et al., 2013. Efficient Estimation of Word Representations in Vector Space. arXiv: 1301.3781. https://arxiv.org/abs/1301.3781

[30] Miwa, M., Bansal, M., 2016. End-to-End Relation Extraction Using LSTMS on Sequences and Tree Structures. arXiv: 1601.00770. https://arxiv.org/abs/1601.00770

[31] Nguyen T H, Grishman R. Relation Extraction: Perspective from Convolutional Neural Networks. Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, 2015, Denver: Association for Computational Linguistics

[32] Nieh E H, Schottdorf M, Freeman N W, . Geometry of Abstract Learned Knowledge in the Hippocampus. Nature, 2021, 595(7865): 80-84.

[33] Oramas S, Ostuni V C, Di Noia T, . Sound and Music Recommendation with Knowledge Graphs. ACM Transactions on Intelligent Systems and Technology, 2017, 8(2): 1-21.

[34] Palumbo E, Monti D, Rizzo G, . Entity2rec: Property-Specific Knowledge Graph Embeddings for Item Recommendation. Expert Systems with Applications, 2020, 151 113235

[35] Peng, N. Y., Dredze, M., 2016. Improving Named Entity Recognition for Chinese Social Media with Word Segmentation Representation Learning. arXiv: 1603.00786. https://arxiv.org/abs/1603.00786

[36] Pennington J, Socher R, Manning C. Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, Doha: Association for Computational Linguistics

[37] Peters, M. E., Neumann, M., Iyyer, M., et al., 2018. Deep Contextualized Word Representations. arXiv: 1802.05365. https://arxiv.org/abs/1802.05365

[38] Qiu Q J, Xie Z, Wu L, . DGeoSegmenter: A Dictionary-Based Chinese Word Segmenter for the Geoscience Domain. Computers & Geosciences, 2018, 121 1-11.

[39] Qiu Q J, Xie Z, Wu L, . BiLSTM-CRF for Geological Named Entity Recognition from the Geoscience Literature. Earth Science Informatics, 2019, 12(4): 565-579.

[40] Qiu Q J, Xie Z, Wu L, . Geoscience Keyphrase Extraction Algorithm Using Enhanced Word Embedding. Expert Systems With Applications, 2019, 125: 157-169.

[41] Qiu Q J, Xie Z, Wu L A, . GNER: A Generative Model for Geological Named Entity Recognition without Labeled Data Using Deep Learning. Earth and Space Science, 2019, 6(6): 931-946.

[42] Qiu Q J, Xie Z, Wu L, . Automatic Spatiotemporal and Semantic Information Extraction from Unstructured Geoscience Reports Using Text Mining Techniques. Earth Science Informatics, 2020, 13(4): 1393-1410.

[43] Qiu Q J, Xie Z, Wu L A, . Dictionary-Based Automated Information Extraction from Geological Documents Using a Deep Learning Algorithm. Earth and Space Science, 2020, 7(3): e2019EA000993

[44] Qu J F, Ouyang D T, Hua W, . Distant Supervision for Neural Relation Extraction Integrated with Word Attention and Property Features. Neural Networks, 2018, 100: 59-69.

[45] Radford, A., Narasimhan, K., 2018. Improving Language Understanding by Generative Pre-Training, preprint. https://gwern.net/doc/www/s3-us-west-2.amazonaws.com/d73fdc5ffa8627bce44dcda2fc012da638ffb158.pdf

[46] Radford A, Wu J, Child R, . Language Models are Unsupervised Multitask Learners. OpenAI Blog, 2019, 1(8): 9

[47] Santos R, Murrieta-Flores P, Calado P, . Toponym Matching through Deep Neural Networks. International Journal of Geographical Information Science, 2018, 32 2 324-348.

[48] Singhal, A., 2012. Introducing the Knowledge Graph: Things, not Strings. Google Blog. https://www.blog.google/products/search/introducing-knowledge-graph-things-not/

[49] Sun C, Yang Z H, Wang L, . Biomedical Named Entity Recognition Using BERT in the Machine Reading Comprehension Framework. Journal of Biomedical Informatics, 2021, 118 103799

[50] Surdeanu, M., Tibshirani, J., Nallapati, R., et al., 2012. Multi-Instance Multi-Label Learning for Relation Extraction. The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. July 12–14, 2012, Jeju Island. https://doi.org/10.5555/2390948.2391003

[51] Vaswani A, Shazeer N, Parmar N, . Attention is All You Need. Advances in Neural Information Processing Systems, 2017, 30 5998-6008.

[52] Vincent, P., Larochelle, H., Bengio, Y., et al., 2008. Extracting and Composing Robust Features with Denoising Autoencoders. The 25th International conference on Machine Learning. July 5–9, 2008, Helsinki. https://doi.org/10.1145/1390156.1390294

[53] Wang C, Hazen R M, Cheng Q, . The Deep-Time Digital Earth Program: Data-Driven Discovery in Geosciences. National Science Review, 2021, 8(9): nwab027

[54] Wu, S., Song, X. N., Feng, Z. H., 2021. MECT: Multi-Metadata Embedding Based Cross-Transformer for Chinese Named Entity Recognition. arXiv: 2107.05418. https://arxiv.org/abs/2107.05418

[55] Xu, Y., Mou, L. L., Li, G., et al., 2015. Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path. arXiv: 1508.03720. https://arxiv.org/abs/1508.03720

[56] Yao L, Zhang Y, Chen Q F, . Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data. Engineering Applications of Artificial Intelligence, 2017, 64: 432-439.

[57] Zeng, D., Liu, K., Lai, S., et al., 2014. Relation Classification via Convolutional Deep Neural Network. The 25th International Conference on Computational Linguistics: Technical Papers, March 25–28, Tokyo

[58] Zhang W, Du Y H, Yoshida T, . DeepRec: A Deep Neural Network Approach to Recommendation with Item Embedding and Weighted Loss Function. Information Sciences, 2019, 470 121-140.

[59] Zhang X Y, Ye P, Wang S, . Geological Entity Recognition Method Based on Deep Belief Networks. Acta Petrologica Sinica, 2018, 34(2): 343-351

[60] Zheng S C, Hao Y X, Lu D Y, . Joint Entity and Relation Extraction Based on a Hybrid Neural Network. Neurocomputing, 2017, 257: 59-66.

[61] Zheng, S. C., Wang, F., Bao, H. Y., et al., 2017b. Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. arXiv: 1706.05075. https://arxiv.org/abs/1706.05075

[62] Zhou C H, Wang H, Wang C S, . Geoscience Knowledge Graph in the Big Data Era. Science China Earth Sciences, 2021, 64(7): 1105-1114.

[63] Zhou P, El-Gohary N. Ontology-Based Automated Information Extraction from Building Energy Conservation Codes. Automation in Construction, 2017, 74: 103-117.

[64] Zhou P, Xu J, Qi Z, . Distant Supervision for Relation Extraction with Hierarchical Selective Attention. Neural Networks, 2018, 108 240