Domain adversarial training for classification of cracking in images of concrete surfaces

Abstract

The development of automatic methods to recognize cracks in surfaces of concrete has been under focus in recent years, firstly through computer vision methods and more recently focusing on convolutional neural networks that are delivering promising results. Challenges are still persisting in crack recognition, namely due to the confusion added by the myriad of elements commonly found on concrete surfaces. The robustness of these methods would deal with these elements if access to correspondingly heterogeneous datasets was possible. Even so, this would be a cumbersome methodology, since training would be needed for each particular case and models would be case dependent. Thus, efforts from the scientific community are focusing on generalizing neural network models to achieve high performance in images from different domains, slightly different from those in which they were effectively trained. The generalization of networks can be achieved by domain adaptation techniques at the training stage. Domain adaptation enables finding a feature space in which features from both domains are invariant, and thus, classes become separable. The work presented here proposes the DA-Crack method, which is a domain adversarial training method, to generalize a neural network for recognizing cracks in images of concrete surfaces. The domain adversarial method uses a convolutional extractor followed by a classifier and a discriminator, and relies on two datasets: a source labeled dataset and a target unlabeled small dataset. The classifier is responsible for the classification of images randomly chosen, while the discriminator is dedicated to uncovering to which dataset each image belongs. Backpropagation from the discriminator reverses the gradient used to update the extractor. This enables fighting the convergence promoted by the updating backpropagated from the classifier, and thus generalizing the extractor enabling it for crack recognition of images from both source and target datasets. Results show that the DA-Crack training method improved accuracy in crack classification of images from the target dataset in 54 percentage points, while accuracy on the source dataset remains unaffected.

Keywords

DA-Crack method / Domain-adaptation / Adversarial training network / Crack detection / Concrete surfaces / Computer vision

References

[1] Abdel-QaderI, Pashaie-RadS, AbudayyehO, YehiaS. PCA based algorithm for unsupervised bridge crack detection. Advances in Engineering Software, 2006, 37(12):771-778

[2] ACI Committee. (2008). Guide for conducting a visual inspection of concrete in service. American Concrete Institute.

[3] AliL, AlnajjarF, JassmiHA, GochooM, KhanW, SerhaniMA. Performance evaluation of deep CNN-based crack detection and localization techniques for concrete structures. Sensors, 2021, 21(5):1-22

[4] Ben-DavidS, BlitzerJ, CrammerK, KuleszaA, PereiraF, VaughanJW. A theory of learning from different domains. Machine Learning, 2010, 79(1–2):151-175

[5] ChaY-J, ChoiW, BüyüköztürkO. Deep Learning-based crack damage detection using convolutional neural networks. Computer-Aided Civil and Infrastructure Engineering, 2017, 32(5):361-378

[6] ChaYJ, ChoiW, SuhG, MahmoudkhaniS, BüyüköztürkO. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil and Infrastructure Engineering, 2018, 33(9):731-747

[7] ChenFC, JahanshahiMR. NB-CNN: deep learning-based crack detection using convolutional neural network and Naïve Bayes data fusion. IEEE Transactions on Industrial Electronics, 2018, 65(5):4392-4400

[8] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: Constructing a large-scale image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248–265). https://doi.org/10.1167/9.8.1037

[9] DengJ, LuY, LeeVCS. Concrete crack detection with handwriting script interferences using faster region-based convolutional neural network. Computer-Aided Civil and Infrastructure Engineering, 2019, 35(4):373-388

[10] DengJ, LuY, LeeVCS. Imaging-based crack detection on concrete surfaces using You Only Look Once network. Structural Health Monitoring, 2021, 20(2):484-499

[11] DongCZ, CatbasFN. A review of computer vision-based structural health monitoring at local and global levels. Structural Health Monitoring, 2020

[12] Fujita, Y., & Hamamoto, Y. (2009). A robust method for automatically detecting cracks on noisy concrete surfaces. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5579 LNAI, 76–85. https://doi.org/10.1007/978-3-642-02568-6_8

[13] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Lempitsky, V. et al. (2016). Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17 (1), 2096–2130. arXiv:1505.07818. https://doi.org/10.1007/978-3-319-58347-1_10

[14] HamishebaharY, GuanH, SoS, JoJ. A comprehensive review of deep learning-based crack detection approaches. Applied Sciences (Switzerland), 2022, 12(3):1374

[15] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 770–778). arXiv:1512.03385. https://doi.org/10.1109/CVPR.2016.90

[16] ICOLD. (2018). Dam Surveillance Guide. Bulletin 158 (Tech. Rep.). International Commission on Large Dams. Retrieved from https://www.icoldcigb.org/

[17] Jang, K.-Y., Kim, B., Cho, S., & An, Y.-K. (2018). Deep learning-based concrete crack detection using hybrid images (p. 36). https://doi.org/10.1117/12.2294959

[18] JiangY, PangD, LiC. A deep learning approach for fast detection and classification of concrete damage. Automation in Construction, 2021, 128(May):103785

[19] KimB, ChoS. Automated vision-based detection of cracks on concrete surfaces using a deep learning technique. Sensors, 2018, 18(10):3452

[20] KrizhevskyA, SutskeverI, HintonGE. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 25: 1097-1105

[21] Liu, H., Yang, C., Li, A., Ge, Y., Huang, S., Feng, X., & Ruan, Z. (2021). Deep domain adaptation for pavement crack detection (pp. 1–12). Retrieved from arXiv:2111.10101.

[22] Oliveira SantosB, ValençaJ, JúlioE. Automatic mapping of cracking patterns on concrete surfaces with biological stains using hyperspectral images processing. Structural Control and Health Monitoring, 2019, 26(e2320

[23] Ozgenel, Ç. F. (2018). Concrete Crack Images for Classification.

[24] Özgenel, F., & Gönenç Sorguç, A. (2018). Performance comparison of pretrained convolutional neural networks on crack detection in buildings. In ISARC 2018-35th International Symposium on Automation and Robotics in Construction and International AEC/FM Hackathon: The Future of Building Things(ISARC). https://doi.org/10.22260/isarc2018/0094

[25] ParkSE, EemSH, JeonH. Concrete crack detection and quantification using deep learning and structured light. Construction and Building Materials, 2020, 252

[26] Pinheiro, P. O. (2018). Unsupervised domain adaptation with similarity learning. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition(i) (pp. 8004–8013). arXiv:1711.08995. https://doi.org/10.1109/CVPR.2018.00835

[27] QuZ, GuoY, JuF-R, LiuL, LinL-D. The algorithm of accelerated cracks detection and extracting skeleton by direction chain code in concrete surface image. The Imaging Science Journal, 2016, 64 3):119-130

[28] QuZ, MeiJ, LiuL, ZhouDY. Crack detection of concrete pavement with cross-entropy loss function and improved VGG16 network model. IEEE Access, 2020, 8: 54564-54573

[29] SaleemMR, ParkJW, LeeJH, JungHJ, SarwarMZ. Instant bridge visual inspection using an unmanned aerial vehicle by image capturing and geo-tagging system and deep convolutional neural network. Structural Health Monitoring, 2021, 20(4):1760-1777

[30] da SilvaWRL, de LucenaDS. Concrete cracks detection based on deep learning image classification. Proceedings-Mdpi, 2018, 2(8):498

[31] ValençaJ, Dias-da-CostaD, JúlioE. Characterisation of concrete cracking during laboratorial tests using image processing. Construction and Building Materials, 2012, 28(1):607-615

[32] ValençaJ, JúlioE. MCrack-Dam: The scale-up of a method to assess cracks on concrete dams by image processing. The case study of Itaipu Dam, at the Brazil-Paraguay border. Journal of Civil Structural Health Monitoring, 2018, 8(5):857-866

[33] Van Der Maaten, L. J. P., & Hinton, G. E. (2008). Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9, 2579–2605. arXiv:1307.1662. https://doi.org/10.1007/s10479-011-0841-3

[34] Wang, M., & Deng, W. (2018). Deep visual domain adaptation: a survey. Neurocomputing, 312, 135–153. arXiv:1802.03601. https://doi.org/10.1016/j.neucom.2018.05.083

[35] YamaguchiT, HashimotoS. Practical image measurement of crack width for real concrete structure. Electronics and Communications in Japan, 2009, 92(10):605-614

[36] Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road crack detection using deep convolutional neural network. In International conference on image processing (ICIP). https://doi.org/10.1109/ICIP.2016.7533052

Funding

Funda??o para a Ciência e a Tecnologia(CEECIND/04463/2017)