Knowledge enhanced graph inference network based entity-relation extraction and knowledge graph construction for industrial domain

Abstract

With the escalating complexity in production scenarios, vast amounts of production information are retained within enterprises in the industrial domain. Probing questions of how to meticulously excavate value from complex document information and establish coherent information links arise. In this work, we present a framework for knowledge graph construction in the industrial domain, predicated on knowledge-enhanced document-level entity and relation extraction. This approach alleviates the shortage of annotated data in the industrial domain and models the interplay of industrial documents. To augment the accuracy of named entity recognition, domain-specific knowledge is incorporated into the initialization of the word embedding matrix within the bidirectional long short-term memory conditional random field (BiLSTM-CRF) framework. For relation extraction, this paper introduces the knowledge-enhanced graph inference (KEGI) network, a pioneering method designed for long paragraphs in the industrial domain. This method discerns intricate interactions among entities by constructing a document graph and innovatively integrates knowledge representation into both node construction and path inference through TransR. On the application stratum, BiLSTM-CRF and KEGI are utilized to craft a knowledge graph from a knowledge representation model and Chinese fault reports for a steel production line, specifically SPOnto and SPFRDoc. The F1 value for entity and relation extraction has been enhanced by 2% to 6%. The quality of the extracted knowledge graph complies with the requirements of real-world production environment applications. The results demonstrate that KEGI can profoundly delve into production reports, extracting a wealth of knowledge and patterns, thereby providing a comprehensive solution for production management.

Graphical abstract

Keywords

knowledge graph construction / industrial / BiLSTM-CRF / document-level relation extraction / graph inference

References

[1] BahdanauDChoKBengioY (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint. arXiv:1409.0473

[2] CaoPChenYLiuKZhaoJLiuS (2018). Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Brussels: ACL, 182–192

[3] ChristopoulouFMiwaMAnaniadouS (2019). Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing / 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong: ACL, 4925–4936

[4] Deng, J Wang, T Wang, Z Zhou, J Cheng, L (2022). Research on event logic knowledge graph construction method of robot transmission system fault diagnosis. IEEE Access, 10: 17656–17673

[5] Dong, J Wang, J Chen, S (2021). Knowledge graph construction based on knowledge enhanced word embedding model in manufacturing domain. Journal of Intelligent & Fuzzy Systems, 41( 2): 3603–3613

[6] Gui, W Zeng, Z Chen, X Xie, Y Sun, Y (2020). Knowledge-driven process industry smart manufacturing. Scientia Sinica Informationis, 50( 9): 1345–1360 in Chinese)

[7] GuoZZhangYLuW (2019). Attention guided graph convolutional networks for relation extraction. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: ACL, 241–251

[8] HanXGaoTLinYPengHYangYXiaoCLiuZLiPZhouJSunM (2020). More data, more relations, more context and more openness: A review and outlook for relation extraction. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. Suzhou: ACL, 745–758

[9] Hu, L Wu, G Xing, Y Wang, F (2020). Things2Vec: Semantic modeling in the Internet of Things with graph representation learning. IEEE Internet of Things Journal, 7( 3): 1939–1948

[10] HuangZXuWYuK (2015). Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint. arXiv:1508.01991

[11] Huet, A Pinquié, R Véron, P Mallet, A Segonds, F (2021). CACDA: A knowledge graph for a context-aware cognitive design assistant. Computers in Industry, 125: 103377

[12] Kamble, S Gunasekaran, A Gawankar, S (2018). Sustainable Industry 4.0 framework: A systematic literature review identifying the current trends and future perspectives. Process Safety and Environmental Protection, 117: 408–425

[13] KipfT NWellingM (2016). Semi-supervised classification with graph convolutional networks. arXiv preprint. arXiv:1609.02907

[14] LaffertyJMcCallumAPereiraF (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the 18th International Conference on Machine Learning. San Francisco, CA: Morgan Kaufmann Publishers Inc., 282–289

[15] Li, G Pan, R Mao, J Cao, Y (2020). Entity recognition of Chinese electronic medical records based on BiLSTM-CRF network and dictionary resources. Journal of Modern Information, 40( 4): 3–12, 58 (in Chinese)

[16] Li, J Sun, A Han, J Li, C (2022). A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 34( 1): 50–70

[17] Lin, H Liu, Y Wang, W Yue, Y Lin, Z (2017). Learning entity and relation embeddings for knowledge resolution. Procedia Computer Science, 108: 345–354

[18] LinYTsaiTChouWWuKSungTHsuW (2004). A maximum entropy approach to biomedical named entity recognition. In: Proceedings of the 4th International Conference on Data Mining in Bioinformatics. Seattle WA: Springer-Verlag, 56–61

[19] Liu, M Li, X Li, J Liu, Y Zhou, B Bao, J (2022). A knowledge graph-based data representation approach for IIoT-enabled cognitive manufacturing. Advanced Engineering Informatics, 51: 101515

[20] LiuWZhouPZhaoZWangZJuQDengHWangP (2020). K-BERT: Enabling language representation with knowledge graph. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York, NY: AAAI, 2901–2908

[21] Lyu, Z Wu, Y Lai, J Yang, M Li, C Zhou, W (2023). Knowledge enhanced graph neural networks for explainable recommendation. IEEE Transactions on Knowledge and Data Engineering, 35( 5): 4954–4968

[22] NanGGuoZSekulicILuW (2020). Reasoning with latent structure refinement for document-level relation extraction. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Online: ACL, 1546–1557

[23] QiASinTFathullahMLeeC (2017). The impact of fit manufacturing on green manufacturing: A review. In: Proceedings of the 3rd Electronic and Green Materials International Conference (EGM). Krabi: AIP, 020083

[24] Ren, H Chen, Z Jiang, Z Yang, C Gui, W (2021). An industrial multilevel knowledge graph-based local–global monitoring for plant-wide processes. IEEE Transactions on Instrumentation and Measurement, 70: 1–15

[25] SahuSChristopoulouFMiwaMAnaniadouS (2019). Inter-sentence relation extraction with document-level graph convolutional neural network. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Florence: ACL, 4309–4316

[26] Shi, H Huang, D Wang, L Wu, M Xu, Y Zeng, B Pang, C (2021). An information integration approach to spacecraft fault diagnosis. Enterprise Information Systems, 15( 8): 1128–1161

[27] StrubellEVergaPBelangerDMcCallumA (2017). Fast and accurate entity recognition with iterated dilated convolutions. arXiv preprint. arXiv:1702.02098

[28] Wan, Z Ge, P Zhang, X Yin, G (2018). Research on equipment manufacturing industry upgrading under intelligent manufacturing. World Sci-Tech R & D, 40( 3): 316–327 (in Chinese)

[29] Wang, B Yi, B Liu, Z Zhou, Y Zhou, Y (2021). Evolution and state-of-the-art of intelligent manufacturing from HCPS perspective. Computer Integrated Manufacturing Systems, 27( 10): 2749–2761 (in Chinese)

[30] Wang, B Zang, J Qu, X Dong, J Zhou, Y (2018). Research on new-generation intelligent manufacturing based on human-cyber-physical systems. Strategic Study of CAE, 20( 4): 29–34

[31] WangHFockeCSylvesterRMishraNWangW (2019). Fine-tune BERT for DocRED with two-step process. arXiv preprint. arXiv:1909.11898

[32] XiaoCYaoYXieRHanXLiuZSunMLinFLinL (2020). Denoising relation extraction from document-level distant supervision. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: ACL, 3683–3688

[33] XuBWangQLyuYZhuYMaoZ (2021). Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. In: Proceedings of the AAAI Conference on Artificial Intelligence. Online: AAAI, 14149–14157

[34] Xu, Z Dang, Y Zhang, Z Chen, J (2020). Typical short-term remedy knowledge mining for product quality problem-solving based on bipartite graph clustering. Computers in Industry, 122: 103277

[35] YaoYYeDLiPHanXLinYLiuZLiuZHuangLZhouJSunM (2019). DocRED: A large-scale document-level relation extraction dataset. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Florence: ACL, 764–777

[36] ZengSXuRChangBLiL (2020). Double graph based reasoning for document-level relation extraction. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: ACL, 1630–1640

[37] Zhang, D Liu, Z Jia, W Liu, H Tan, J (2021). A review on knowledge graph and its application prospects to intelligent manufacturing. Journal of Mechanical Engineering, 57( 5): 90–113 in Chinese)

[38] ZhangZHanXLiuZJiangXSunMLiuQ (2019). ERNIE: Enhanced language representation with informative entities. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Florence: ACL, 1441–1451

[39] Zheng, P Xia, L Li, C Li, X Liu, B (2021). Towards Self-X cognitive manufacturing network: An industrial knowledge graph-based multi-agent reinforcement learning approach. Journal of Manufacturing Systems, 61: 16–26

[40] Zhou, J Zhou, Y Wang, B Zang, J (2019). Human-Cyber-Physical Systems (HCPSs) in the context of new-generation intelligent manufacturing. Engineering, 5( 4): 624–636

[41] ZhouWHuangKMaTHuangJ (2021). Document-level relation extraction with adaptive thresholding and localized context pooling. In: Proceedings of the AAAI Conference on Artificial Intelligence. Online: AAAI, 14612–14620

[42] Zhou, Y Huang, H Liu, H Hao, Z (2022). Survey on document-level relation extraction. Journal of South China University of Technology (Natural Science Edition), 50( 4): 10–25 (in Chinese)

Competing Interests

The authors declare that they have no competing interests.

Data Availability Statements

The datasets generated and analyzed within the current study are available from China Baowu Steel Group Corporation Limited. Restrictions apply to the availability of steel production data, which were used under license for this study. Steel production data are available from the corresponding author with the permission of China Baowu Steel Group Corporation Limited.

RIGHTS & PERMISSIONS

2024 Higher Education Press