A knowledge graph attention network for the cold-start problem in intelligent manufacturing: Interpretability and accuracy improvement

Abstract

In the rolling production of steel, predicting the performance of new products is challenging due to the low variety of data distributions resulting from standardized manufacturing processes and fixed product categories. This scenario poses a significant hurdle for machine learning models, leading to what is commonly known as the “cold-start problem”. To address this issue, we propose a knowledge graph attention neural network for steel manufacturing (SteelKGAT). By leveraging expert knowledge and a multi-head attention mechanism, SteelKGAT aims to enhance prediction accuracy. Our experimental results demonstrate that the SteelKGAT model outperforms existing methods when generalizing to previously unseen products. Only the SteelKGAT model accurately captures the feature trend, thereby offering correct guidance in product tuning, which is of practical significance for new product development (NPD). Additionally, we employ the Integrated Gradients (IG) method to shed light on the model's predictions, revealing the relative importance of each feature within the knowledge graph. Notably, this work represents the first application of knowledge graph attention neural networks to address the cold-start problem in steel rolling production. By combining domain expertise and interpretable predictions, our knowledge-informed SteelKGAT model provides accurate insights into the mechanical properties of products even in cold-start scenarios.

Keywords

attention mechanisms / cold-start problem / graph neural network / interpretable machine learning / knowledge graph / materials design / mechanical performance

References

[1] Chen M, Mao S, Liu Y. Big data: a survey. Mobile Network Appl. 2014; 19(2): 171-209.

[2] Cui C, Cao G, Li X, Gao Z, Zhou X, Liu Z. The coupling machine learning for microstructural evolution and rolling force during hot strip rolling of steels. J Mater Process Technol. 2022; 309:117736.

[3] Cui C, Cao G, Li X, Gao Z, Liu J, Liu Z. A strategy combining machine learning and physical metallurgical principles to predict mechanical properties for hot rolled Ti micro-alloyed steels. J Mater Process Technol. 2023; 311:117810.

[4] Cooper RG. New products—what separates the winners from the losers and what drives success. J Prod Innovat Manag. 2013; 3:34.

[5] Unger DW, Eppinger SD. Comparing product development processes and managing risk. Int J Prod Dev. 2009; 8(4): 382-402.

[6] Bobadilla J, Ortega F, Hernando A, Bernal J. A collaborative filtering approach to mitigate the new user cold start problem. Knowl Base Syst. 2012; 26: 225-238.

[7] Smirnov PS, Sudakov VA. Forecasting new product demand using machine learning. J Phys Conf Ser. 2021; 1925(1):012033.

[8] Van Steenbergen RM, Mes MR. Forecasting demand profiles of new products. Decis Support Syst. 2020; 139:113401.

[9] Kim HJ, Mahfouf M, Yang YY. Modelling of hot strip rolling process using a hybrid neural network approach. J Mater Process Technol. 2008; 201(1-3): 101-105.

[10] Jiang X, Jia B, Zhang G, et al. A strategy combining machine learning and multiscale calculation to predict tensile strength for pearlitic steel wires with industrial data. Scr Mater. 2020; 186: 272-277.

[11] Xie Q, Suvarna M, Li J, Zhu X, Cai J, Wang X. Online prediction of mechanical properties of hot rolled steel plate using machine learning. Mater Des. 2021; 197:109201.

[12] Sun J, Deng J, Peng W, Zhang D. Strip crown prediction in hot rolling process using random forest. Int J Precis Eng Manuf. 2021; 22(2): 301-311.

[13] Li H, Li Y, Huang J, et al. Physical metallurgy guided industrial big data analysis system with data classification and property prediction. Steel Res Int. 2022; 93(8):2100820.

[14] Geng X, Wang F, Wu HH, et al. Data-driven and artificial intelligence accelerated steel material research and intelligent manufacturing technology. Mat Gen Eng Adv. 2023; 1: e10.

[15] Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A. A survey on bias and fairness in machine learning. ACM Comput Surv. 2021; 54(6): 1-35.

[16] Wei X, van der Zwaag S, Jia Z, Wang C, Xu W. On the use of transfer modeling to design new steels with excellent rotating bending fatigue resistance even in the case of very small calibration datasets. Acta Mater. 2022; 235:118103.

[17] Shen C, Wang C, Wei X, Li Y, van der Zwaag S, Xu W. Physical metallurgy-guided machine learning and artificial intelligent design of ultrahigh-strength stainless steel. Acta Mater. 2019; 179: 201-214.

[18] Ren D, Wang C, Wei X, Lai Q, Xu W. Building a quantitative composition-microstructure-property relationship of dual-phase steels via multimodal data mining. Acta Mater. 2023; 252:118954.

[19] Zhang Z, Yang Z, Wang C, Xu W. Accelerating ultrashort pulse laser micromachining process comprehensive optimization using a machine learning cycle design strategy integrated with a physical model. J Intell Manuf. 2024; 35(1): 449-465.

[20] Han S, Wang C, Zhang Y, Xu W, Di H. Employing deep learning in non-parametric inverse visualization of elastic–plastic mechanisms in dual-phase steels. Mat Gen Eng Adv. 2024; 2(1):e29.

[21] Nussberger AM, Luo L, Celis LE, Crockett MJ. Public attitudes value interpretability but prioritize accuracy in artificial intelligence. Nat Commun. 2022; 13(1):5821.

[22] Zhang J, Ma C, Zhong C, Zhao P, Mu X. Combining feature importance and neighbor node interactions for cold start recommendation. Eng Appl Artif Intell. 2022; 112:104864.

[23] Liu S, Ounis I, Macdonald C, Meng Z. A heterogeneous graph neural model for cold-start recommendation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval; 2020: 2029-2032.

[24] Wang H, Zhang F, Wang J, et al. RippleNet: propagating user preferences on the knowledge graph for recommender systems. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management; 2018: 417-426.

[25] Qian T, Liang Y, Li Q, Xiong H. Attribute graph neural networks for strict cold start recommendation. IEEE Trans Knowl Data Eng. 2020; 34(8): 3597-3610.

[26] Zhu J, Li B, Wang J, Li D, Liu Y, Zhang Z. BGCL: Bi-subgraph network based on graph contrastive learning for cold-start QoS prediction. Knowl-Based Syst. 2023; 263:110296.

[27] Velickovic P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y. Graph attention networks. In: 6th International Conference on Learning Representations; 2018.

[28] Deng Z, Liang Y, Pan J, Liao J, Hao Y, Wen X. Fast 3D face reconstruction from a single image combining attention mechanism and graph convolutional network. Vis Comput. 2023; 39(11): 5547-5561.

[29] Zheng C, Fan X, Wang C, Qi J. GMAN: a graph multi-attention network for traffic prediction. Proc AAAI Conf Artif Intell. 2020; 34(01): 1234-1241.

[30] Kou J, Jia P, Liu J, Dai J, Luo H. Identify influential nodes in social networks with graph multi-head attention regression model. Neurocomputing. 2023; 530: 23-36.

[31] Wang X, He X, Cao Y, Liu M, Chua TS. KGAT: knowledge graph attention network for recommendation. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; 2019: 950-958.

[32] Brody S, Alon U, Yahav E. How attentive are graph attention networks? In: 10th International Conference on Learning Representations; 2022.

[33] Wang Q, Cui H, Zhang J, Du Y, Zhou Y, Lu X. Neighbor-augmented knowledge graph attention network for recommendation. Neural Process Lett. 2023; 55(6): 1-17.

[34] Xu K, Hu W, Leskovec J, Jegelka S. How powerful are graph neural networks? In: 7th International Conference on Learning Representations; 2019.

[35] Duvenaud DK, Maclaurin D, Iparraguirre J, et al. Convolutional networks on graphs for learning molecular fingerprints. Adv Neural Inf Process Syst. 2015; 28: 2224-2232.

[36] Sundararajan M, Taly A, Yan Q. Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning; 2017: 3319-3328.

[37] Jain S, Wallace BC. Attention is not explanation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; 2019: 3543-3556.

[38] Serrano S, Smith NA. Is attention interpretable? In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; 2019: 2931-2951.

[39] Wiegreffe S, Pinter Y. Attention is not explanation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing; 2019: 11-20.

[40] Pruthi D, Gupta M, Dhingra B, Neubig G, Lipton ZC. Learning to deceive with attention-based explanations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 2019: 4782-4793.

[41] Strout J, Zhang Y, Mooney RJ. Do human rationales improve machine explanations? In: Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP; 2019: 56-62.

[42] Bastings J, Filippova K. The elephant in the interpretability room: why use attention as explanation when we have saliency methods? In: Proceedings of the 2020 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP; 2020: 149-155.

[43] Lu Q, Xu W, van der Zwaag S. Designing new corrosion resistant ferritic heat resistant steel based on optimal solid solution strengthening and minimisation of undesirable microstructural components. Comput Mater Sci. 2014; 84: 198-205.

[44] Zheng H, Ye XN, Li JD, et al. Effect of carbon content on microstructure and mechanical properties of hot-rolled low carbon 12Cr–Ni stainless steel. Mater Sci Eng A. 2010; 527(27-28): 7407-7412.

[45] Cuddy LJ, Bauwin JJ, Raley JC. Recrystallization of austenite. Metall Trans A. 1980; 11(3): 381-386.

[46] Zeng W, Zhou M, Yang M, et al. Influence of manganese content on microstructure, mechanical properties and high temperature oxidation resistance of reduced activation ferritic/martensitic steels. J Mater Eng Perform. 2023; 32(21): 9765-9776.

[47] Drumond J, Girina O, da Silva Filho JF, Fonstein N, de Oliveira CS. Effect of silicon content on the microstructure and mechanical properties of dual-phase steels. Metallogr Microstruct Anal. 2012; 1(5): 217-223.

RIGHTS & PERMISSIONS

2025 The Author(s). Materials Genome Engineering Advances published by Wiley-VCH GmbH on behalf of University of Science and Technology Beijing.