Chinese named entity recognition with multi-network fusion of multi-scale lexical information

Abstract

Named entity recognition (NER) is an important part in knowledge extraction and one of the main tasks in constructing knowledge graphs. In today's Chinese named entity recognition (CNER) task, the BERT-BiLSTM-CRF model is widely used and often yields notable results. However, recognizing each entity with high accuracy remains challenging. Many entities do not appear as single words but as part of complex phrases, making it difficult to achieve accurate recognition using word embedding information alone because the intricate lexical structure often impacts the performance. To address this issue, we propose an improved Bidirectional Encoder Representations from Transformers (BERT) character word conditional random field (CRF) (BCWC) model. It incorporates a pre-trained word embedding model using the skip-gram with negative sampling (SGNS) method, alongside traditional BERT embeddings. By comparing datasets with different word segmentation tools, we obtain enhanced word embedding features for segmented data. These features are then processed using the multi-scale convolution and iterated dilated convolutional neural networks (IDCNNs) with varying expansion rates to capture features at multiple scales and extract diverse contextual information. Additionally, a multi-attention mechanism is employed to fuse word and character embeddings. Finally, CRFs are applied to learn sequence constraints and optimize entity label annotations. A series of experiments are conducted on three public datasets, demonstrating that the proposed method outperforms the recent advanced baselines. BCWC is capable to address the challenge of recognizing complex entities by combining character-level and word-level embedding information, thereby improving the accuracy of CNER. Such a model is potential to the applications of more precise knowledge extraction such as knowledge graph construction and information retrieval, particularly in domain-specific natural language processing tasks that require high entity recognition precision.

Keywords

Bi-directional long short-term memory (BiLSTM) / Chinese named entity recognition (CNER) / Iterated dilated convolutional neural network (IDCNN) / Multi-network integration / Multi-scale lexical features

Funding

This research was funded by the International Research Center of Big Data for Sustainable Development Goals under Grant No. CBAS2022GSP05; the Open Fund of State Key Laboratory of Remote Sensing Science under Grant No. 6142A01210404; the Hubei Key Laboratory of Intelligent Geo-Information Processing under Grant No. KLIGIP-2022-B03.

Declaration of competing interest

No conflict of interest exits in the submission of this manuscript, and manuscript is approved by all authors for publication. I would like to declare on behalf of my co-authors that the work described was original research that has not been published previously, and not under consideration for publication elsewhere, in whole or in part. All the authors listed have approved the manuscript that is enclosed.

APPENDIX A.

There are some details about the experimental process using systems such as BERT, embedding layer, and multi-head attention that require fine-tuning of the hyperparameters, and some details about this process are given in Table A1 and Table A2. Some hardware and software environments about the experiments are given in Table A3.

Table A1. Some common hyperparameter settings about the experiments.

Hyperparameter	Value
RNN dimension	64
BERT learning rate	2 ​× ​10−5
BERT dropout rate	0.35
RNN(CNN) learning rate	1 ​× ​10−3
Kernel size	3
Boundary embedding dimension	16
BERT model	BERT-base-Chinese
Epoch	30
Optimizer	AdamW
Multi-head attention head	8

Table A2. Some hyperparameter settings about the experiments on the three datasets.

Dataset	max_seq_len	Batch size	max_word_len
CLUENER	128	32	25
Weibo	64	16	20
Youku	128	16	20

Table A3. Some hardware and software environments about the experiments.

Environment	Value
OS	Windows 11
Processor	12th Gen Intel(R) Core(TM) i5-12600KF
RAM	32.0 ​GB
GPU	NVIDIA GeForce RTX 3070Ti GPU 8 ​GB
Python version	3.8.17
PyTorch version	2.0.0

References

[1] G. Popovski, B.K. Seljak, T. Eftimov, A survey of named-entity recognition methods for food information extraction, IEEE Access 8 (Feb. 2020) 31586-31594.

[2] D.-D. Yin, S.-Y. Cheng, B.-X. Pan, Y.-Y. Qiao, W. Zhao, D.-Y. Wang, Chinese named entity recognition based on knowledge based question answering system, Appl. Sci. 12 (11) (May 2022) 5373:1-19.

[3] J. Li, A.-X. Sun, J.-L. Han, C.-L. Li, A survey on deep learning for named entity recognition, IEEE T. Knowl. Data En. 34 (1) (Jan. 2020) 50-70.

[4] H.-Y. Li, Z. Shi, C.-W. Pan, D. Zhao, N. Sun, Cybersecurity knowledge graphs construction and quality assessment, Complex Intell. Syst. 10 (1) (Feb. 2024) 1201-1217.

[5] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9 (8) (Nov. 1997) 1735-1780.

[6] M. Schuster, K.K. Paliwal, Bidirectional recurrent neural networks, IEEE T. Signal Proces. 45 (11) (Nov. 1997) 2673-2681.

[7] Y. An, X.-Y. Xia, X.-L. Chen, F.-X. Wu, J.-X. Wang, Chinese clinical named entity recognition via multi-head self-attention based BiLSTM-CRF, Artif. Intell. Med. 127 (May 2022) 102282:1-12.

[8] M. Gridach, Character-level neural network for biomedical named entity recognition, J. Biomed. Inform. 70 (Jun. 2017) 85-91.

[9] X. Seti, A. Wumaier, T. Yibulayin, D. Paerhati, L.-L. Wang, A. Saimaiti, Named-entity recognition in sports field based on a character-level graph convolutional network, Information 11 (1) (Jan. 2020) 30:1-16.

[10] Z.-R. Deng, Y. Tao, R.-S. Lan, R. Yang, X.-Y. Wang, Kcr-FLAT: A Chinese-named entity recognition model with enhanced semantic information, Sensors 23 (4) (Feb. 2023) 1771:1-15.

[11] X.-T. Tian, X.-X. Bu, L. He, Multi-task learning with helpful word selection for lexicon-enhanced Chinese NER, Appl. Intell. 53 (16) (Feb. 2023) 19028-19043.

[12] H.-T. Liu, J.-H. Song, W.-M. Peng, J.-B. Sun, X.-W. Xin, TFM: A triple fusion module for integrating lexicon information in Chinese named entity recognition, Neural Process. Lett. 54 (4) (Apr. 2022) 3425-3442.

[13] X.-L. Wang, X.-R. Xu, D.-G. Huang, T. Zhang, Multi-task label-wise transformer for Chinese named entity recognition, ACM T. Asian Low-Reso. 22 (4) (Apr. 2023) 118:1-15.

[14] Y.-B. Yan, P. Zhu, D.-W. Cheng, F.-Z. Yang, Y.-F. Luo, Adversarial multi-task learning for efficient Chinese named entity recognition, ACM T. Asian Low-Reso. 22 (7) (Jul. 2023) 193:1-19.

[15] Q.-P. Liu, L.-L. Zhang, G. Ren, B.-J. Zou, Research on named entity recognition of Traditional Chinese Medicine chest discomfort cases incorporating domain vocabulary features, Comput. Biol. Med. 166 (Nov. 2023) 107466:1-6.

[16] Z. Nasar, S.W. Jaffry, M.K. Malik, Named entity recognition and relation extraction: State-of-the-art, ACM Comput. Surv. 54 (1) (Jan. 2022) 20:1-39.

[17] J. Lafferty, A. McCallum, F.C.N. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, in: Proc. of the 18th Intl. Conf. on Machine Learning, (2001), pp. 282-289. Williamstown, USA.

[18] W. Li, D.-Z. Zhao, B. Li, X.-M. Peng, J.-R. Liu, Combining CRF and rule based medical named entity recognition, Appl. Res. Comput. 32 (4) (Apr. 2015) 1082-1086.

[19] N.V. Sobhana, P. Mitra, S.K. Ghosh, Conditional random field based named entity recognition in geological text, Int. J. Comput. Appl. 1 (3) (2010) 119-125.

[20] E. Strubell, P. Verga, D. Belanger, A. McCallum, Fast and accurate entity recognition with iterated dilated convolutions, in: Proc. of the Conf. on Empirical Methods in Natural Language Processing, (2017), pp. 2670-2680. Copenhagen, Denmark.

[21] T.-Y. Chen, S. Feng, Research on named entity recognition method and model stability of electronic medical record based on IDCNN + CRF and attention mechanism, China Digital Medicine 17 (11) (Nov. 2022) 1-5.

[22] H.-Y. Hu, C.-P. Zhao, L. Ma, H.-Z. Jiang, J. Zhang, W.-G. Zhu, Study on Chinese medical named entity recognition based on the dilated convolutional neural network, Journal of Medical Informatics 42 (9) (Oct. 2021) 39-44.

[23] J.-Q. Zhao, W.-T. Zhu, C. Chen, Named entity recognition based on duplex convolution neural network model, Computer Technology and Development 33 (1) (Jan. 2023) 187-192.

[24] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proc. of the Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (2019), pp. 4171-4186. Minneapolis, USA.

[25] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, Improving Language Understanding by Generative Pre-training, Openai Blog, 2018 [Online]. Available, https://gwern.net/doc/www/s3-us-west-2.amazonaws.com/d73fdc5ffa8627bce44dcda2fc012da638ffb158.pdf

[26] Z.-J. Dai, X.-T. Wang, P. Ni, Y.-M. Li, G.-M. Li, X.-M. Bai, Named entity recognition using BERT BiLSTM CRF for Chinese electronic health records, in: Proc. of the 12th Intl. Congress on Image and Signal Processing, BioMedical Engineering and Informatics, (2019), pp. 1-5. Suzhou, China.

[27] Y.-Q. Yu, Y.-Z. Wang, J.-Q. Mu, et al., Chinese mineral named entity recognition based on BERT model, Expert Syst. Appl. 206 (Nov) (2022) 117727:1–9.

[28] S.-H. Wang, X.-F. Sun, X.-Y. Li, et al., GPT-NER: Named entity recognition via large language models (October 2023) [Online]. Available, https://arxiv.org/abs/2304.10428

[29] H. Yan, Y. Li, Q.-X. Lei, X. Wang, Chinese named entity recognition method based on lexical information features, Journal of Chinese Computer Systems 45 (7) (Jul. 2024) 1622-1628.

[30] Z.-W. Yang, J. Ma, H.-C. Chen, J.-W. Zhang, Y. Chang, Context-aware attentive multilevel feature fusion for named entity recognition, IEEE T. Neur. Net. Lear. 35 (1) (Jan. 2024) 973-984.

[31] K.-F. Long, H. Zhao, Z.-Z. Shao, et al., Deep neural network with embedding fusion for Chinese named entity recognition, ACM T. Asian Low-Reso. 91 (3) (Mar. 2023) 22:1-16.

[32] Y. Wang, Z. Wang, H. Yu, G.-Y. Wang, D.-J. Lei, The interactive fusion of characters and lexical information for Chinese named entity recognition, Artif. Intell. Rev. 57 (10) (Aug. 2024) 258:1-21.

[33] M.-J. Li, R.-Q. Huang, X.-X. Qi, Chinese clinical named entity recognition using multi-feature fusion and multi-scale local context enhancement, Comput. Mater. Con. 80 (2) (Aug. 2024) 2283-2299.

[34] Q.-J. Qiu, M. Tian, Z. Huang, et al., Chinese engineering geological named entity recognition by fusing multi-features and data enhancement using deep learning, Expert Syst. Appl. 238 (Mar. 2024) 121925:1-17.

[35] L. Xu, Y. Tong, Q.-Q. Dong, et al., CLUENER2020: Fine-grained named entity recognition dataset and benchmark for Chinese (January 2020) [Online]. Available, https://arxiv.org/abs/2001.04351

[36] N.-Y. Peng, M. Dredze, Named entity recognition for Chinese social media with jointly trained embeddings, in: Proc. of the Conf. on Empirical Methods in Natural Language Processing, (2015), pp. 548-554. Lisbon, Portugal.

[37] Z.-M. Jie, P.-J. Xie, W. Lu, R.-X. Ding, L.-L. Li, Better modeling of incomplete annotations for named entity recognition, in: Proc. of the Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (2019), pp. 729-734. Minneapolis, USA.

[38] S. Li, Z. Zhao, R.-F. Hu, W.-S. Li, T. Liu, X.-Y. Du, Analogical reasoning on Chinese morphological and semantic relations, in: Proc. of the 56th Annual Meeting of the Association for Computational Linguistics, (2018), pp. 138-143. Melbourne, Australia.

[39] T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in: Proc. of the 26th Intl. Conf. on Neural Information Processing Systems, Lake Tahoe, USA, (2013), pp. 3111-3119.

[40] Y. Goldberg, O. Levy, Levy word2vec Explained: Deriving Mikolov et al.’s negative-sampling word-embedding method (February 2014) [Online]. Available, https://arxiv.org/abs/1402.3722

[41] X.-W. Yin, S. Zheng, Q.-M. Wang, Fine-grained Chinese named entity recognition based on RoBERTa-WWM-BiLSTM-CRF model, in: Proc. of the 6th Intl. Conf. on Image, Vision and Computing, Qingdao, China, (2021), pp. 408-413.

[42] S.-Q. Li, J.-J. Zeng, J.-H. Zhang, T. Peng, L. Yang, H.-F. Lin, ALBERT-BiLSTM for sequential metaphor detection, in: Proc. of the Second Workshop on Figurative Language Processing, Virtual Event, (2020), pp. 110-115.

[43] Y.-H. Zhai, S.-W. Tian, L. Yu, et al., Chinese Characters and Pinyin: A Model with Two Parallel Feature Extractors for Chinese Entity Recognition (November 2022) [Online]. Available, https://www.researchsquare.com/article/rs-2282745/v1

[44] Z.-Y. Xuan, R. Bao, S.-Y. Jiang, FGN: fusion glyph network for Chinese named entity recognition, in: Proc. of the Knowledge Graph and Semantic Computing: Knowledge Graph and Cognitive Intelligence: 5th China Conf., (2021), pp. 28-40. Nanchang, China.

[45] Y. Zhang, J. Yang, Chinese NER using lattice LSTM, in: Proc. of the 56th Annual Meeting of the Association for Computational Linguistics, (2018), pp. 1554-1564. Melbourne, Australia.

[46] L.-Z. Li, S. Zheng, Q.-M. Wang, RoBERTa and stacked bidirectional GRU for fine-grained Chinese named entity recognition, in: Proc. of the 6th Intl. Conf. on Mathematics and Artificial Intelligence, (2021), pp. 95-100. Chengdu, China.

[47] T. Gui, R.-T. Ma, Q. Zhang, L.-J. Zhao, Y.-G. Jiang, X.-J. Huang, CNN-based Chinese NER with lexicon rethinking, in: Proc. of the 28th Intl. Joint Conf. on Artificial Intelligence,, (2019), pp. 4982-4988. Macao, China.

[48] T. Gui, Y.-C. Zou, Q. Zhang, et al., A lexicon-based graph neural network for Chinese NER, in: Proc. of the Conf. on Empirical Methods in Natural Language Processing and the 9th Intl. Joint Conf. on Natural Language Processing, (2019), pp. 1040-1050. Hong Kong, China.

[49] M. Zhang, L. Lu, A local information perception enhancement–based method for Chinese NER, Appl. Sci. 13 (17) (Sept. 2023) 9948:1-19.

[50] Z.-R. Deng, Z. Huang, S.-W. Wei, J.-L. Zhang, KCB-FLAT: Enhancing Chinese named entity recognition with syntactic information and boundary smoothing techniques, Mathematics 12 (17) (Aug. 2024) 2714:1-18.

[51] Y.-H. Yuan, Q.-H. Zhang, X. Zhou, M. Gao, A Chinese named entity recognition model: Integrating label knowledge and lexicon information, Int. J. Mach. Learn. Cyb (May 2024), https://doi.org/10.1007/s13042-024-02207-2.

[52] E.-W. Zhu, Y.-Y. Liu, J.-P. Li, Deep span representations for named entity recognition, in: Proc. of the Findings of the Association for Computational Linguistics, (2023), pp. 10565-10582. Toronto, Canada.