Detecting large-scale underwater cracks based on remote operated vehicle and graph convolutional neural network

Abstract

It is of great significance to quickly detect underwater cracks as they can seriously threaten the safety of underwater structures. Research to date has mainly focused on the detection of above-water-level cracks and hasn’t considered the large scale cracks. In this paper, a large-scale underwater crack examination method is proposed based on image stitching and segmentation. In addition, a purpose of this paper is to design a new convolution method to segment underwater images. An improved As-Projective-As-Possible (APAP) algorithm was designed to extract and stitch keyframes from videos. The graph convolutional neural network (GCN) was used to segment the stitched image. The GCN’s m-IOU is 24.02% higher than Fully convolutional networks (FCN), proving that GCN has great potential of application in image segmentation and underwater image processing. The result shows that the improved APAP algorithm and GCN can adapt to complex underwater environments and perform well in different study areas.

Graphical abstract

Keywords

underwater cracks / remote operated vehicle / image stitching / image segmentation / graph convolutional neural network

References

[1] Li W, Chen G, Ge J, Yin X, Li K. High sensitivity rotating alternating current field measurement for arbitrary-angle underwater cracks. NDT & E International, 2016, 79: 123–131

[2] FangHDuanM. Off-shore Operation Facilities: Equipment and Procedures. Boston: Gulf Professional Publishing, 2014, 537–686

[3] Sun J, Xue C, Yu Y. Research on feature-based underwater image mosaic technology. Ship Electronic Engineering, 2017, 37(6): 118–121

[4] Rabczuk T, Belytschko T. Cracking particles: A simplified meshfree method for arbitrary evolving cracks. International Journal for Numerical Methods in Engineering, 2004, 61(13): 2316–2343

[5] Zhang Y M, Yang X Q, Wang X Y, Zhuang X Y. A micropolar peridynamic model with non-uniform horizon for static damage of solids considering different nonlocal enhancements. Theoretical and Applied Fracture Mechanics, 2021, 113: 102930

[6] Zhang Y, Mang H A. Global cracking elements: A novel tool for Galerkin-based approaches simulating quasi-brittle fracture. International Journal for Numerical Methods in Engineering, 2020, 121(11): 2462–2480

[7] Martinez J, Rey J, Hidalgo M C, Garrido J, Rojas D. Influence of measurement conditions on the resolution of electrical resistivity imaging: The example of abandoned mining dams in the La Carolina District (Southern Spain). International Journal of Mineral Processing, 2014, 133: 67–72

[8] Xue X, Yang X. Earthquake safety assessment of an arch dam using an anisotropic damage model for mass concrete. Computers and Concrete, 2014, 13(5): 633–648

[9] Zhang Y M, Zhuang X Y. Cracking elements method for dynamic brittle fracture. Theoretical and Applied Fracture Mechanics, 2019, 102: 1–9

[10] Zhang Y M, Zhuang X Y. Cracking elements: A self-propagating Strong Discontinuity embedded Approach for quasi-brittle fracture. Finite Elements in Analysis and Design, 2018, 144: 84–100

[11] Rezaiee-Pajand M, Tavakoli F H. Crack detection in concrete gravity dams using a genetic algorithm. Proceedings of the Institution of Civil Engineers. Structures and Buildings, 2015, 168(3): 192–209

[12] Su H, Li J, Hu J, Wen Z. Analysis and back-analysis for temperature field of concrete arch dam during construction period based on temperature data measured by DTS. IEEE Sensors Journal, 2013, 13(5): 1403–1412

[13] Lai S L, Lee D H, Wu J H, Dong Y M. Detecting the cracks and seepage line associated with an earthquake in an earth dam using the nondestructive testing technologies. Journal of the Chinese Institute of Engineers, 2014, 37(4): 428–437

[14] Luo D, Yue Y, Li P, Ma J X, Zhang L L, Ibrahim Z, Ismail Z. Concrete beam crack detection using tapered polymer optical fiber sensors. Measurement, 2016, 88: 96–103

[15] Shi P, Fan X, Ni J, Khan Z, Li M. A novel underwater dam crack detection and classification approach based on sonar images. PLoS One, 2017, 12(6): e0179627

[16] Xiong P, Xingu Z, Chao Z, Anhua C, Tianyu Z. A UAV-based machine vision method for bridge crack recognition and width quantification through hybrid feature learning. Construction and Building Materials, 2021, 299: 123896

[17] Bang S, Park S, Kim H, Kim H. Encoder–decoder network for pixel-level road crack detection in black-box images. Computer-Aided Civil and Infrastructure Engineering, 2019, 34(8): 713–727

[18] XuG. Research and implementation of concrete apparent crack detection algorithm based on deep learning. Thesis for the Master’s Degree. Shanghai: Shanghai Jiao Tong University, 2020 (in Chinese)

[19] Belytschko T, Lu Y Y, Gu L. Element-free Galerkin methods. International Journal for Numerical Methods in Engineering, 1994, 37(2): 229–256

[20] Ukai M. Development of image processing technique for detection of tunnel wall deformation using continuously scanned image. Quarterly Report of RTRI, 2000, 41(3): 120–126

[21] Ukai M. Advanced inspection system of tunnel wall deformation using image processing. Quarterly Report of RTRI, 2007, 48(2): 94–98

[22] LuG FZhaoQ CLiaoJ GHeY B. Pavement crack identification based on automatic threshold iterative method. In: Seventh International Conference on Electronics and Information Engineering. Nanjing: International Society for Optics and Photonics, 2017

[23] Talab A M A, Huang Z C, Xi F, Liu H M. Detection crack in image using Otsu method and multiple filtering in image processing techniques. Optik (Stuttgart), 2016, 127(3): 1030–1033

[24] Xiao Y, Li J. Crack detection algorithm based on the fusion of percolation theory and adaptive canny operator. In: 2018 37th Chinese Control Conference (CCC). Wuhan: IEEE, 2018, 4295–4299

[25] Feng C C, Zhang H, Wang H R, Wang S, Li Y L. Automatic pixel-level crack detection on dam surface using deep convolutional network. Sensors (Basel), 2020, 20(7): 2069

[26] Nguyen-Thanh V M, Anitescu C, Alajlan N, Rabczuk T, Zhuang X Y. Parametric deep energy approach for elasticity accounting for strain gradient effects. Computer Methods in Applied Mechanics and Engineering, 2021, 386: 114096

[27] Nguyen-Thanh V M, Zhuang X Y, Rabczuk T. A deep energy method for finite deformation hyperelasticity. European Journal of Mechanics. A, Solids, 2020, 80: 103874

[28] Guo H W, Zhuang X Y, Rabczuk T. A deep collocation method for the bending analysis of Kirchhoff plate. Computers, Materials & Continua, 2019, 59(2): 433–456

[29] Zhuang X Y, Guo H W, Alajlan N, Zhu H, Rabczuk T. Deep autoencoder based energy method for the bending, vibration, and buckling analysis of Kirchhoff plates with transfer learning. European Journal of Mechanics. A, Solids, 2021, 87: 104225

[30] Guo H W, Zhuang X Y, Chen P W, Alajlan N, Rabczuk T. Stochastic deep collocation method based on neural architecture search and transfer learning for heterogeneous porous media. Engineering with Computers, 2022, 1–26

[31] Samaniego E, Anitescu C, Goswami S, Nguyen-Thanh V M, Guo H, Hamdia K, Zhuang X, Rabczuk T. An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications. Computer Methods in Applied Mechanics and Engineering, 2020, 362: 112790

[32] Anitescu C, Atroshchenko E, Alajlan N, Rabczuk T. Artificial neural network methods for the solution of second order boundary value problems. Computers, Materials & Continua, 2019, 59(1): 345–359

[33] Kang F, Wu Y R, Li J J, Li H J. Dynamic parameter inverse analysis of concrete dams based on Jaya algorithm with Gaussian processes surrogate model. Advanced Engineering Informatics, 2021, 49: 101348

[34] Kang F, Liu X, Li J J. Temperature effect modeling in structural health monitoring of concrete dams using kernel extreme learning machines. Structural Health Monitoring, 2020, 19(4): 987–1002

[35] Cha Y, Choi W, Büyüköztürk O. Deep learning-based crack damage detection using convolutional neural networks. Computer-Aided Civil and Infrastructure Engineering, 2017, 32(5): 361–378

[36] Kim B, Cho S. Automated vision-based detection of cracks on concrete surfaces using a deep learning technique. Sensors (Basel), 2018, 18(10): 3452

[37] LongJShelhamerEDarrellT. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3431–3440

[38] Dung C V, Anh L D. Autonomous concrete crack detection using deep fully convolutional neural network. Automation in Construction, 2019, 99: 52–58

[39] Zhang X X, Rajan D, Story B. Concrete crack detection using context-aware deep semantic segmentation network. Computer-Aided Civil and Infrastructure Engineering, 2019, 34(11): 951–971

[40] Zhang C B, Chang C C, Jamshidi M. Concrete bridge surface damage detection using a single-stage detector. Computer-Aided Civil and Infrastructure Engineering, 2020, 35(4): 389–409

[41] Liu J W, Yang X, Lau S, Wang X, Luo S, Lee V C S, Ding L. Automated pavement crack detection and segmentation based on two-step convolutional neural network. Computer-Aided Civil and Infrastructure Engineering, 2020, 35(11): 1291–1305

[42] Zhang Y, Yuen K V. Crack detection using fusion features-based broad learning system and image processing. Computer-Aided Civil and Infrastructure Engineering, 2021, 36(12): 1568–1584

[43] Wang L L, Spencer B F, Li J J, Hu P. A fast image-stitching algorithm for characterization of cracks in large-scale structures. Smart Structures and Systems, 2021, 27(4): 593–605

[44] WuL JLinXChenZ CLinP JChengS Y. Surface crack detection based on image stitching and transfer learning with pretrained convolutional neural network. Structural Control and Health Monitoring, 2021, 28(8): e2766

[45] Burt P J, Adelson E H. The laplacian pyramid as a compact image code. IRE Transactions on Communications Systems, 1983, 31(4): 532–540

[46] Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110

[47] Bay H, Ess A, Tuytelaars T, Van Gool L. Speeded up robust features (SURF). Computer Vision and Image Understanding, 2008, 110(3): 346–359

[48] RubleeERabaudVKonoligeKBradskiG. ORB: An efficient alternative to SIFT or SURF. In: IEEE International Conference on Computer Vision. 2016, 2564–2571

[49] Zhu Z, German S, Brilakis I J. Detection of large-scale concrete columns for automated bridge inspection. Automation in Construction, 2010, 19(8): 1047–1055

[50] Won J, Park J W, Shim C, Park M W. Bridge-surface panoramic-image generation for automated bridge-inspection using deepmatching. Structural Health Monitoring, 2021, 20(4): 1689–1703

[51] Zhang R W, He D H, Li Y P, Huang L, Bao X J. Synthetic imaging through wavy water surface with centroid evolution. Optics Express, 2018, 26(20): 26009–26019

[52] QinG Q. Research on underwater photogrammetry for surface easurement of satellite antenna in simulated zero-gravity conditions. Dissertation for the Doctoral Degree. Zhengzhou: The PLA Information Engineering University, 2011 (in Chinese)

[53] Zaragoza J, Chin T J, Tran Q H, Brown M S, Suter D. As-projective-as-possible image stitching with moving DLT. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1285–1298

[54] LiuQ. Research on aerial image stitching technology based on improved SURF and APAP. Thesis for the Master’s Degree. Dalian: Dalian University of Technology, 2021 (in Chinese)

[55] Landrieu L, Simonovsky M. Large-scale point cloud semantic segmentation with superpoint graphs. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT: IEEE, 2018, 4558–4567

[56] YuanY HWangJ D. Ocnet: Object context network for scene parsing. 2018, arXiv:1809.00916

[57] Ding K M, Chen S P, Meng F. A novel perceptual hash algorithm for multispectral image authentication. Algorithms, 2018, 11(1): 1–14

[58] ZhangLLiX TArnabAYangK YTongHYP H STorr. Dual graph convolutional network for semantic segmentation. 2019, arXiv:1909.06121

[59] Simo-Serra E, Trulls E, Ferraz L, Kokkinos I, Fua P, Moreno-Noguer F. Discriminative learning of deep convolutional feature point descriptors. In: 2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015, 118–126

[60] Yang J X, Zhang Y B, Huang L H, Guo D C, Yang Y K. A novel diamond search algorithm for fast block motion estimation. In: International Conference on Image Processing and Pattern Recognition in Industrial Engineering. SPIE, 2010, 737–743

[61] Zhao X H, Wang Y, Du Z S, Ye X F. Research on the image enhancement technology of underwater image of super cavitation vehicle. In: 2019 IEEE International Conference on Mechatronics and Automation (ICMA). Tianjin: IEEE, 2019, 1520–1524

[62] Zhao X W, Jin T, Chi H, Qu S. Modeling and simulation of the background light in underwater imaging under different illumination conditions. Acta Physica Sinica, 2015, 64(10): 104201

[63] BahetiBInnaniSGajreSTalbarS. Eff-unet: A novel architecture for semantic segmentation in unstructured environment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle, WA: IEEE, 2020

[64] YuanY HChenX LWangJ D. Object-contextual representations for semantic segmentation. In: European Conference on Computer Vision. Springer, 2019

Acknowledgements

Thanks to South to North Water Diversion Central Route Information Technology Co., Ltd. for providing the underwater videos of the water conveyance tunnels for research purposes. Thanks to CISPDR Corporation for providing the underwater video of the dam for research purposes. This work was supported by the National Natural Science Foundation of China (Grant Nos. 51979027, 52079022, 51769033 and 51779035).

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

RIGHTS & PERMISSIONS

2022 The Author(s). This article is published with open access at link.springer.com and journal.hep.com.cn