Large language models enabled intelligent microstructure optimization and defects classification of welded titanium alloys

Abstract

The quick developments of artificial intelligence have brought tremendous attractive opportunities and changes to smart welding technology. In the present work, a novel model, ConvNeXt, which incorporates the advantages of convolutional neural networks (CNNs) and vision transformers (ViTs), has been designed to identify welding defects. The classification accuracy of the pre-trained ConvNeXt based on transfer learning method reaches as high as 99.52% after 500 iterations of training, while traditional CNNs of MobileNetV2 and ResNet34 achieve 85.94% and 93.41%, respectively. Moreover, the classification performance can be further improved through dataset optimization based on t-distributed stochastic neighbor embedding (t-SNE). In addition, arc geometrical features are added as input parameters for building a back propagation neural network to predict the formation of the weld seam, which has led to a reduction in the maximum prediction error for weld seam thickness from 0.8 to 0.6 mm. Furthermore, out of 28 sets of experimental parameters, only four sets result in errors exceeding 0.2 mm. It is worth noting that large language models (LLMs) are utilized to facilitate the automated programming for welding defect recognition, including ChatGPT 3.5, Bing Copilot, Claude3, and ERNIE Bot. LLM-aided automated programming technology is applied to develop image stitching programs, achieving unsupervised automatic stitching of multiple welding tissue images and obtaining clear and wide-field weld ones. These case studies of deep learning technologies and automated programming based on LLMs set up a solidified building block for smart welding defect recognition during non-equilibrium solidification.

Keywords

Welding defect recognition / convolutional neural networks / back propagation neural network / large language models / automated programming

References

[1] An Y,Zhao Y.Nonequilibrium solidification velocity, recalescence degree and grain refinement of highly undercooled Ni-based single-phase alloys.J Alloys Compd2021;881:160658

[2] Cheng Y,Qiu Z.Multi-physics simulation of non-equilibrium solidification in Ti-Nb alloy during selective laser melting.Acta Mater2024;272:119923

[3] Peng Z,Liu L,Wang G.Effect of high-speed ultrasonic vibration cutting on the microstructure, surface integrity, and wear behavior of titanium alloy.J Mater Res Technol2023;24:3870-88

[4] Williams W. Development of structural titanium alloys for marine applications.Ocean Eng1969;1:375-83

[5] Auwal ST,Yusof F.A review on laser beam welding of titanium alloys.Int J Adv Manuf Technol2018;97:1071-98

[6] Su Y,Wang D.Composition- and temperature-dependence of β to ω phase transformation in Ti-Nb alloys.J Mater Inf2023;3:14

[7] Li P,Wang WY.Coupling effects of high magnetic field and annealing on the microstructure evolution and mechanical properties of additive manufactured Ti–6Al–4V.Mater Sci Eng A2021;824:141815

[8] Park H.Estimation of weld bead size in CO₂ laser welding by using multiple regression and neural network.J Laser Appl1999;11:143-50

[9] Wang WY,Chai Z.Big data-assisted digital twins for the smart design and manufacturing of advanced materials: from atoms to products.J Mater Inf2022;

[10] Vasan V,Balasundaram RJ.Ensemble-based deep learning model for welding defect detection and classification.Eng Appl Artif Intell2024;136:108961

[11] Wang B,Sun L.Intelligent welding system technologies: state-of-the-art review and perspectives.J Manuf Syst2020;56:373-91

[12] Feng Y,Wang D,Feng Z.DeepWelding: A deep learning enhanced approach to GTAW using multisource sensing images.IEEE Trans Ind Inf2020;16:465-74

[13] Wang S,Wen S.An accurate state-of-charge estimation of lithium-ion batteries based on improved particle swarm optimization-adaptive square root cubature kalman filter.J Power Sources2024;624:235594

[14] Ma D,Zhou Q,Jiang P.Online porosity defect detection based on convolutional neural network for Al alloy laser welding.J Phys Conf Ser2021;1884:012008

[15] Zhang Y,Gao X.Online monitoring of welding status based on a DBN model during laser welding.Engineering2019;5:671-8

[16] Huang J,Qin R.Interpretable real-time monitoring of pipeline weld crack leakage based on wavelet multi-kernel network.J Manuf Syst2024;72:93-103

[17] Cheng Y,Zhou Q,Yuan W.Real-time sensing of gas metal arc welding process - a literature review and analysis.J Manuf Process2021;70:452-69

[18] Hossain R,Moore AL.In situ infrared temperature sensing for real-time defect detection in additive manufacturing.Addit Manuf2021;47:102328

[19] Chen C,Chen S.Welding penetration monitoring for pulsed GTAW using visual sensor based on AAM and random forests.J Manuf Process2021;63:152-62

[20] Liu T,Huang X,Bao J.3DSMDA-Net: an improved 3DCNN with separable structure and multi-dimensional attention for welding status recognition.J Manuf Syst2022;62:811-22

[21] Bacioiu D,Papaelias M.Automated defect classification of aluminium 5083 TIG welding using HDR camera and neural networks.J Manuf Process2019;45:603-13

[22] Yang J,Wang Z,Wang J.Using deep learning to detect defects in manufacturing: a comprehensive survey and current challenges.Materials2020;13:5755 PMCID:PMC7766692

[23] Dosovitskiy A,Kolesnikov A. An image is worth 16 × 16 words: transformers for image recognition at scale. arXiv 2021, arXiv.2010.11929. Available online: https://doi.org/10.48550/arXiv.2010.11929 (accessed 26 Dec 2024)

[24] Springenberg M,Wenzel M,Ma J.From modern CNNs to vision transformers: assessing the performance, robustness, and classification strategies of deep learning models in histopathology.Med Image Anal2023;87:102809

[25] Mehta S. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv 2021, arXiv.2110.02178. Available online: https://doi.org/10.48550/arXiv.2110.02178 (accessed 26 Dec 2024)

[26] Liu Z,Wu CY,Darrell T.A convnet for the 2020s. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2022 Jun 18-24; New Orleans, USA. IEEE; 2022. pp. 11976-86.

[27] Hou Q,Cheng MM.Conv2Former: a simple transformer-style ConvNet for visual recognition.IEEE Trans Pattern Anal Mach Intell2024;46:8274-83

[28] Lin M,Meng J,Wu J.Screening of retired batteries with gramian angular difference fields and ConvNeXt.Eng Appl Artif Intell2023;123:106397

[29] Lei Z,Wang Q.Real-time weld geometry prediction based on multi-information using neural network optimized by PCA and GA during thin-plate laser welding.J Manuf Process2019;43:207-17

[30] Moon H.A neuro-fuzzy approach to select welding conditions for welding quality improvement in horizontal fillet welding.J Manuf Syst1996;15:392-403

[31] Ai Y,Jiang P,Liu Y.Process modeling and parameter optimization using radial basis function neural network and genetic algorithm for laser welding of dissimilar materials.Appl Phys A2015;121:555-69

[32] Yu R,Peng Y.Monitoring of butt weld penetration based on infrared sensing and improved histograms of oriented gradients.J Mater Res Technol2023;22:3280-93

[33] Yang L,Peng J.A novel system for off-line 3D seam extraction and path planning based on point cloud segmentation for arc welding robot.Robot Cim Int Manuf2020;64:101929

[34] Zhang K,Huang T,Li Z.3D reconstruction of complex spatial weld seam for autonomous welding by laser structured light scanning.J Manuf Process2019;39:200-7

[35] Liu T,Bao J.Deep learning-based welding image recognition: a comprehensive review.J Manuf Syst2023;68:601-25

[36] Sahu PK.Multi-response optimization of process parameters in friction stir welded AM20 magnesium alloy by Taguchi grey relational analysis.J Magnes Alloy2015;3:36-46

[37] Kulal S,Chandra K. SPoC: search-based pseudocode to code. arXiv 2019, arXiv.1906.04908. Available online: https://doi.org/10.48550/arXiv.1906.04908 (accessed 26 Dec 2024)

[38] Fedorenko E,Dhamala R.The language of programming: a cognitive perspective.Trends Cogn Sci2019;23:525-8

[39] Bobrow DG.Perspectives on artificial intelligence programming.Science1986;231:951-7

[40] Ma J,Dong S.MLMD: a programming-free AI platform to predict and design materials.npj Comput Mater2024;10:1243

[41] Nadeem M,Javed L,Saudagar AKJ.Vision-enabled large language and deep learning models for image-based emotion recognition.Cogn Comput2024;16:2566-79

[42] Pei Z,Neugebauer J.Towards the holistic design of alloys with large language models.Nat Rev Mater2024;9:840-1

[43] Mouret JB.Large language models help computer programs to evolve.Nature2024;625:452-3

[44] Vaswani A,Parmar N.Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems; Long Beach, USA. Curran Associates Inc.; 2017, pp. 6000-10.

[45] Wong MF,Hang CN,Tan CW.Natural language generation and understanding of big code for AI-assisted programming: a review.Entropy2023;25:888 PMCID:PMC10297336

[46] Chiarello F,Spada I,Fantoni G.Future applications of generative large language models: a data-driven case study on ChatGPT.Technovation2024;133:103002

[47] Fernandes LC.Programming computational electromagnetic applications assisted by large language models [Em Programmer’s Notebook].IEEE Antennas Propag Mag2024;66:63-71

[48] Peng D,Liu D.Large-language models facilitate discovery of the molecular signatures regulating sleep and activity.Nat Commun2024;15:3685 PMCID:PMC11063160

[49] Karnalim O,Johan MC.Detecting AI assisted submissions in introductory programming via code anomaly.Educ Inf Technol2024;29:16841-66

[50] Zheng Y.Optimization of computer programming based on mathematical models of artificial intelligence algorithms.Comput Electr Eng2023;110:108834

[51] Wang WY,Li G.Artificial intelligence enabled smart design and manufacturing of advanced materials: the endless Frontier in AI⁺ era.MGE Advances2024;2:e56

[52] Chen L,Yao X.High-entropy alloy catalysts: high-throughput and machine learning-driven design.J Mater Inf2022;2:19

[53] Song G,Lv X.TIG and laser–TIG hybrid filler wire welding of casting and wrought dissimilar magnesium alloy.J Manuf Process2018;34:204-14

[54] Weman K. 1 - Introduction to welding. In: Welding processes handbook (Second Edition). Woodhead Publishing; 2012. p.1-12. Available from: https://shop.elsevier.com/books/welding-processes-handbook/weman/978-0-85709-510-7. [Last accessed on 26 Dec 2024]

[55] Zhang D,Liu R.Characterization of corrosion behavior of TA2 titanium alloy welded joints in seawater environment.Front Chem2022;10:950768 PMCID:PMC9354461

[56] Lei T,Rong Y.The development of tube-to-tubesheet welding from automation to digitization.Int J Adv Manuf Technol2021;116:779-802

[57] Jin Z,Zhang C,Gao H.Online welding path detection in automatic tube-to-tubesheet welding using passive vision.Int J Adv Manuf Technol2017;90:3075-84

[58] Mu H,Yuan L,Ding D.A digital shadow approach for enhancing process monitoring in wire arc additive manufacturing using sensor fusion.J Ind Inf Integr2024;40:100609

[59] Pires J, Smith JS, Balfour C. Real-time top-face vision based control of weld pool size.Ind Robot2005;32:334-40

[60] Ugender S.Influence of tool pin profile and rotational speed on the formation of friction stir welding zone in AZ31 magnesium alloy.J Magnes Alloy2018;6:205-13

[61] Lin Q,Yang R.Transistor modeling based on LM-BPNN and CG-BPNN for the GaAs pHEMT.Int J Numerl Model EL2024;37:e3268

[62] Wang X,Bai H,Wang B.SCARA mechanical fault identification based on WPM-SE+BPNN method.Meas Sci Technol2022;33:085007

[63] Lee S,Lieu QX.CNN-based image recognition for topology optimization.Knowl Based Syst2020;198:105887

[64] Zhang Y,Gao X,Gao PP.Welding defects detection based on deep learning with multiple optical sensors during disk laser welding of thick plates.J Manuf Syst2019;51:87-94

[65] Liu T,Zheng P.An expert knowledge-empowered CNN approach for welding radiographic image recognition.Adv Eng Inform2023;56:101963

[66] Wang Z,Liu J.Hyperspectral image classification via spatial shuffle-based convolutional neural network.Remote Sens2023;15:3960

[67] Dai F,Hu Y.A deep neural network potential model for transition metal diborides.J Mater Inf2024;4:10

[68] He K,Ren S.Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 Jun 27-30; Las Vegas, USA. IEEE; 2016. pp. 770-8.

[69] Zhu H,Fu H,Zhang J.Training a seismogram discriminator based on ResNet.IEEE Trans Geosci Remote Sens2021;59:pp.7076-85

[70] Howard AG,Chen B. MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv 2017, arXiv.1704.04861. Available online: https://doi.org/10.48550/arXiv.1704.04861 (accessed 26 Dec 2024)

[71] Liu Z,Shen Z,Cheng KT.Joint multi-dimension pruning via numerical gradient update.IEEE Trans Image Process2021;30:8034-45

[72] Ma S,Li T.Basic motion behavior recognition of single dairy cow based on improved Rexnet 3D network.Comput Electron Agr2022;194:106772

[73] Han K,Chen H.A survey on vision transformer.IEEE Trans Pattern Anal Mach Intell2023;45:87-110

[74] Sun W,Deng H.Vicinity vision transformer.IEEE Trans Pattern Anal Mach Intell2023;45:12635-49

[75] Lv P,Zhang Q.An improved lightweight ConvNeXt for rice classification.Alex Eng J2025;112:84-97

[76] Zhu J,Liu Q.An improved ConvNeXt with multimodal transformer for physiological signal classification. IEEE Access 2024.

[77] Hendrycks D.Gaussian error linear units (GELUs).arXiv2016, arXiv:1606.08415. Available online: accessed 26 Dec 2024)

[78] Lee M.Mathematical analysis and performance evaluation of the GELU activation function in deep learning.J Math2023;2023:1-13

[79] Qiao Y,Qi Y,Yang L.A waste classification model in low-illumination scenes based on ConvNeXt.Resour Conserv Recy2023;199:107274

[80] Jablonka KM,Al-Feghali A.14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon.Digit Discov2023;2:1233-50

[81] Choi J.Accelerating materials language processing with large language models.Commun Mater2024;5:449

[82] Zang Y,Han J,Loy CC.Contextual object detection with multimodal large language models. Int J Comput Vis 2024.

[83] Mahowald K,Blank IA,Tenenbaum JB.Dissociating language and thought in large language models.Trends Cogn Sci2024;28:517-40 PMCID:PMC11416727

[84] Wang H,Wu H,Sun Y.Pre-trained language models and their applications.Engineering2023;25:51-65

[85] Hu L,Zhao Z,Nie L.A survey of knowledge enhanced pre-trained language models.IEEE Trans Knowl Data Eng2024;36:1413-30

[86] Lai Z,Fei X.BERT4ST:: fine-tuning pre-trained large language model for wind power forecasting.Energ Convers Manage2024;307:118331

[87] Cook A.LLM-commentator: novel fine-tuning strategies of large language models for automatic commentary generation using football event data.Knowl Based Syst2024;300:112219

[88] Zhang Z,Chen S.Weld image deep learning-based on-line defects detection using convolutional neural networks for Al alloy in robotic arc welding.J Manuf Process2019;45:208-16

[89] Yang X,Lv W.Image recognition of wind turbine blade damage based on a deep learning model with transfer learning and an ensemble learning classifier.Renew Energ2021;163:386-97

[90] Balado J,Díaz-Vilariño L.Transfer learning in urban object classification: online images to recognize point clouds.Automat Constr2020;111:103058

[91] Deng J,Socher R,Li K.Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition; 2009 Jun 20-25; Miami, USA. IEEE; 2009. pp. 248-55.

[92] Husnain M,Mumtaz S,Coustaty M.Visualization of high-dimensional data by pairwise fusion matrices using t-SNE.Symmetry2019;11:107

[93] Moon H.Optimum design based on mathematical model and neural network to predict weld parameters for fillet joints.J Manuf Syst1997;16:13-23

[94] Gao Y,Xu L.Microstructure homogeneity and mechanical properties of laser-arc hybrid welded AZ31B magnesium alloy.J Magnes Alloy2024;12:1986-95

[95] Geng P,Wang M.Dissimilar linear friction welding of Ni-based superalloys.Int J Mach Tool Manu2023;191:104062

[96] Kim I,Yang Y.Sensitivity analysis for process parameters in GMA welding processes using a factorial design method.Int J Mach Tool Manu2003;43:763-9

[97] Cook GE.Robotic arc welding: research in sensory feedback control.IEEE Trans Ind Electron1983;IE-30:252-68

[98] Nie L,Liao K,Zhao Y.Unsupervised deep image stitching: reconstructing stitched features to images.IEEE Trans Image Process2021;30:6184-97