Real-time K-TIG welding penetration prediction on embedded system using a segmentation-LSTM model

Abstract

Keyhole tungsten inert gas (K-TIG) welding is capable of realizing single-sided welding and double-sided forming and has been widely used in medium and thick plate welding. In order to improve the accuracy of automatic weld identification and weld penetration prediction of robot in the process of large workpiece welding, a two-stage model is proposed in this paper, which can monitor the K-TIG welding penetration state in real time on the embedded system, called segmentation-LSTM model. The proposed system extracts 9 weld pool geometric features with segmentation network, and then extracts the weld gap using a traditional algorithm. Then these 10-dimensional features are input into the LSTM model to predict the penetration state, including under penetration, partial penetration, good penetration and over penetration. The recognition accuracy of the proposed system can reach 95.2%. In this system, to solve the difficulty of labeling data and lack of segmentation accuracy, an improved LabelMe capable of live-wire annotation tool and a novel loss function were proposed, respectively. The latter was also called focal dice loss, which enabled the network to achieve a performance of 0.933 mIoU on the testing set. Finally, an improved slimming strategy compresses the network, making the segmentation network achieve real-time on the embedded system (RK3399pro).

Keywords

Keyhole tungsten inert gas (K-TIG) welding / Penetration state prediction / Segmentation-LSTM model / Embedded system / Focal dice loss / Improved LabelMe

References

[1] Liu ZM, Chen SY, Liu S, et al. Keyhole dynamic thermal behaviour in K-TIG welding process. Int J Heat Mass Tran, 2018, 123: 54-66.

[2] Jarvis BL (2001) Keyhole gas tungsten arc welding: a new process variant. Dissertation, University of Wollongong

[3] Fang ZJ, Weng WW, Wang WJ, et al. A vision-based robotic laser welding system for insulated mugs with fuzzy seam tracking control. Symmetry, 2019, 11(11): 1385.

[4] Zhu T, Shi YH, Cui SW, et al. Recognition of weld penetration during K-TIG welding based on acoustic and visual sensing. Sensing and Imaging, 2019, 20(1): 1-21.

[5] Wang ZS, Shi YH, Hong XB, et al. Weld pool and keyhole geometric feature extraction in K-TIG welding with a gradual gap based on an improved HDR algorithm. J Manuf Process, 2022, 73: 409-427.

[6] Zhan AW, Shi YH, Chen JR (2021) The effect of butt gap on the molten pool and keyhole of K-TIG welding 304 stainless steel. Hot Working Technology 50(23):139–145

[7] Liu YK, Zhang YM. Control of 3D weld pool surface. Control Eng Pract, 2013, 21(11): 1469-1480.

[8] Richardson RW, Gutow DA, Rao SH (1982) A vision based system for arc weld pool size control. Measurement and control for batch manufacturing pp 65–75

[9] Zhan YM, Kovacevic R, Li L. Characterization and real-time measurement of geometrical appearance of the weld pool. Int J Mach Tool Manu, 1996, 36(7): 799-816.

[10] Shi FH, Huang XX, Duan Y, et al. Part-based model for visual detection and localization of gas tungsten arc weld pool. Int J Adv Manuf Tech, 2010, 47(9): 1097-1104.

[11] Wu D, Chen HB, Huang YM et al (2016) Weld penetration identification for UPPAW based on keyhole features and extreme learning machine. In 2016 IEEE workshop on advanced robotics and its social impacts (ARSO) pp 96–99. https://doi.org/10.1109/arso.2016.7736263

[12] Wu D, Chen HB, Huang YM, et al. Monitoring of weld joint penetration during variable polarity plasma arc welding based on the keyhole characteristics and pso-anfis. J Mater Process Tech, 2017, 239: 113-124.

[13] Wu D, Chen JS, Liu HB, et al. Weld penetration in situ prediction from keyhole dynamic behavior under time-varying vppaw pools via the os-elm model. Int J Adv Manuf Tech, 2019, 104(9): 3929-3941.

[14] Liu XF, Wu CS, Jia CB, et al. Visual sensing of the weld pool geometry from the topside view in keyhole plasma arc welding. J Manuf Process, 2017, 26: 74-83.

[15] Liu XF, Jia CB, Wu CS, et al. Measurement of the keyhole entrance and topside weld pool geometries in keyhole plasma arc welding with dual ccd cameras. J Mater Process Tech, 2017, 248: 39-48.

[16] Li YF, Tian SS, Wu CS, et al. Experimental sensing of molten flow velocity, weld pool and keyhole geometries in ultrasonic-assisted plasma arc welding. J Manuf Process, 2021, 64: 1412-1419.

[17] Chen ZQ, Gao XD. Detection of weld pool width using infrared imaging during high-power fiber laser welding of type 304 austenitic stainless steel. Int J Adv Manuf Tech, 2014, 74(9/12): 1247-1254.

[18] Gao XD, Zhang YX. Prediction model of weld width during high-power disk laser welding of 304 austenitic stainless steel. Int J Precis Eng Man, 2014, 15(3): 399-405.

[19] Luo M, Yung CS. Vision-based weld pool boundary extraction and width measure- ment during keyhole fiber laser welding. Opt Laser Eng, 2015, 64: 59-70.

[20] Kotecki DJ, Cheever DL, Howden DG, et al. Mechanism of ripple formation during weld solidification. WELD J, 1972, 51(8): 368

[21] Zhang WJ, Liu YK, Zhang YM (2012) Real-time measurement of three dimensional weld pool surface in GTAW. In Welding Processes. https://doi.org/10.5772/53753

[22] Zhang K, Zhang YM, Chen JS et al (2017) Welding pool oscillation behaviors for pulsed GTA welding based on laser dot matrix sensing. In 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pp 355–358. https://doi.org/10.1109/cyber.2017.8446232

[23] Dong H, Cong M, Zhang YM, et al. Modeling and real-time prediction for complex welding process based on weld pool. Int J Adv Manuf Tech, 2018, 96(5/8): 2495-2508.

[24] Huang JK, Pan W, Chen JS, et al. The transient behaviours of free surface in a fully penetrated weld pool in gas tungsten arc welding. J Manuf Process, 2018, 36: 405-416.

[25] Li CK, Shi Y, Gu YF, et al. Monitoring weld pool oscillation using reflected laser pattern in gas tungsten arc welding. J Mater Process Tech, 2018, 255: 876-885.

[26] Liang ZM, Chang HX, Wang QY, et al. 3d reconstruction of weld pool surface in pulsed gmaw by passive biprism stereo vision. IEEE Robot Autom Lett, 2019, 4(3): 3091-3097.

[27] Gu ZN, Chen J, Wu CS. Three-dimensional reconstruction of welding pool surface by binocular vision. Chin J Mech Eng-EN, 2021, 34(1): 1-13.

[28] Jiao WH, Wang QY, Cheng YC, et al. End-to-end prediction of weld penetration: A deep learning and transfer learning based method. J Manuf Process, 2021, 63: 191-197.

[29] Chen C, Xiao RQ, Chen HB, et al. Prediction of welding quality characteristics during pulsed gtaw process of aluminum alloy by multisensory fusion and hybrid network model. J Manuf Process, 2021, 68: 209-224.

[30] Knaak C, Kolter G, Schulze F, et al. Deep learning-based semantic segmentation for in-process monitoring in laser welding applications. Applications of Machine Learning, 2019, 11139: 1113905

[31] wkentaro (2022) Labelme: image polygonal annotation with python (polygon, rectangle, circle, line, point and image-level flag annotation. https://github.com/wkentaro/labelme/ Accessed 6 March 2022

[32] Yu CQ, Gao CX, Wang JB, et al. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. Int J Comput Vision, 2021, 129: 3051-3068.

[33] Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440. https://doi.org/10.1109/CVPR.2015.7298965

[34] Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: international conference on medical image computing and computer-assisted intervention, pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

[35] Paszke A, Chaurasia A, Kim S et al (2016) ENet: a deep neural network architecture for real-time semantic segmentation. arXiv preprint. https://doi.org/10.48550/arXiv.1606.02147

[36] Romera E, Alvarez J, Bergasa LM, et al. Erfnet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE T Intell Transp, 2017, 19(1): 263-272.

[37] He KM, Zhang XY, Ren SQ et al (2016) Deep residual learning for image recognition. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 770–778. https://doi.org/10.1109/cvpr.2016.90

[38] Cordts M, Omran M, Ramos S et al (2016) The cityscapes dataset for semantic urban scene understanding. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 3213–3223. https://doi.org/10.1109/CVPR.2016.350

[39] Jadon S (2020) A survey of loss functions for semantic segmentation. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) pp 1–7. https://doi.org/10.1109/cibcb48159.2020.9277638

[40] Ma YD, Liu Q, and Qian ZB (2004) Automated image segmentation using improved PCNN model based on cross-entropy. In Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing pp 743–746. https://doi.org/10.1109/isimp.2004.1434171

[41] Lin TY, Goyal P, Girshick R et al (2017) Focal loss for dense object detection. In: proceedings of the IEEE international conference on computer vision pp 2980–2988. https://doi.org/10.1109/iccv.2017.324

[42] Sudre CH, Li WQ, Vercauteren T et al (2017) Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: deep learning in medical image analysis and multimodal learning for clinical decision support pp 240–248. https://doi.org/10.1007/978-3-319-67558-9_28

[43] Berman M, Triki AR, Matthew BB (2018) The Lovasz-Softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 4413–4421. https://doi.org/10.1109/cvpr.2018.00464

[44] Yu JQ, Blaschko MB (2015) The Lovász hinge: A convex surrogate for submodular losses. Stat 1050:24. https://doi.org/10.1109/tpami.2018.2883039

[45] Fujishige S. Submodular functions and optimization, 2005, Amsterdam: Elsevier.

[46] Shrivastava A, Gupta A, Girshick R (2016) Training region-based object detectors with online hard example mining. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 761–769. https://doi.org/10.1109/cvpr.2016.89

[47] Firefly (2022) Firefly: make technology simpler, make life smarter. https://www.t-firefly.com/doc/download/65.html/ Accessed 6 March 2022

[48] Liu Z, Li JG, Shen ZQ et al (2017) Learning efficient convolutional networks through network slimming. In: proceedings of the IEEE international conference on computer vision pp 2736–2744. https://doi.org/10.1109/iccv.2017.298

[49] Krishnamoorthi R (2018) Quantizing deep convolutional networks for efficient inference: a whitepaper. arXiv preprint. https://doi.org/10.48550/arXiv.1806.08342

[50] Hecht-Nielsen R (1992) Theory of the backpropagation neural network. In: neural networks for perception pp 65–93. https://doi.org/10.1016/B978-0-12-741252-8.50010-8

[51] Niranjan M. Support vector machines: a tutorial overview and critical appraisal, 1999 IEE Colloquium on Applied Statistical Pattern Recognition

[52] Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput, 1997, 9(8): 1735-1780.

[53] Converti J (1981) Plasma-jets in arc welding. Dissertation, Massachusetts Institute of Technology

[54] Wang B, Zhu XM, Zhang HC, et al. Characteristics of welding and arc pressure in the plasma–Tig coupled arc welding process. Metals, 2018, 8(7): 512.

Funding

the Key Research and Development Program of Guangdong Province(2020B090928003)

National Natural Science Foundation of Guangdong Province(2020A1515011050)