Fusion of thermal and RGB images for automated deep learning based crack detection in civil infrastructure

Abstract

Research has been continually growing toward the development of image-based structural health monitoring tools that can leverage deep learning models to automate damage detection in civil infrastructure. However, these tools are typically based on RGB images, which work well under ideal lighting conditions, but often have degrading performance in poor and low-light scenes. On the other hand, thermal images, while lacking in crispness of details, do not show the same degradation of performance in changing lighting conditions. The potential to enhance automated damage detection by fusing RGB and thermal images together within a deep learning network has yet to be explored. In this paper, RGB and thermal images are fused in a ResNET-based semantic segmentation model for vision-based inspections. A convolutional neural network is then employed to automatically identify damage defects in concrete. The model uses a thermal and RGB encoder to combine the features detected from both spectrums to improve its performance of the model, and a single decoder to predict the classes. The results suggest that this RGB-thermal fusion network outperforms the RGB-only network in the detection of cracks using the Intersection Over Union (IOU) performance metric. The RGB-thermal fusion model not only detected damage at a higher performance rate, but it also performed much better in differentiating the types of damage.

Keywords

Infrared thermography / Structural health monitoring / Image fusion / Automated crack detection

References

[1] AlexanderQG, LundermanCV. Thermal camera reliability study: Flir One Pro, 2021 US Army Engineer Research and Development Center

[2] Alexander, Q. G., Hoskere, V., Spencer Jr., B. F., & Smith, D. M. (2019). Towards the application of image based monitoring of USACE Large Civil Infrastructure. International Workshop for Structural Health Monitoring. Palo Alto, CA.

[3] An, Y.-K., Jang, K.-Y., Kim, B., & Cho, S. (2018). Deep learning-based concrete crack detection using hybrid images. Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2018. Denver.

[4] ASCE. (2020). Changing the infrastructure equation: Using aset management to optimize investments. Retrieved March 20, 2021, from http://preprod.asce.org/-/media/asce-images-and-files/advocacy/documents/changing-infrastructure-equation-report.pdf

[5] ASTM International. (2013). D4788-03(2013) standard test method for detecting delaminations in bridge decks using infrared thermography. West Conshohocken: ASTM International.

[6] AvciO, AbdeliaberO, KiranyazS, HusseinM, GabbouiM, InmanDJ. A review of vibration-based damage detection in civil structures: From traditional methods to machine learning and deep learning applications. Mechanical Systems and Signal Processing, 2021, 147: 10707

[7] AvdelidisNP, MoropoulouA. Applications of infrared thermography for the investigation of historic structures. Journal of Cultural Heritage, 2004, 5(1):119-127

[8] BaoY, LiH. Machine learning paradigm for structural health monitoring. Structural Health Monitoring, 2020, 20: 1353-13723

[9] BrynjolfssonE, McAfeeA. The second machine age: Work, progress, and prosperity in a time of brilliant technologies, 2014 W.W. Norton & Company

[10] Commons, W. (2015). File:Hendys Law.jpg. Retrieved April 18, 2020, from https://commons.wikimedia.org/wiki/File:Hendys_Law.jpg

[11] DongC-Z, CatbasF. A review of computer vision-based structural health monitoring at local and global levels. Structural Health Monitoring, 2020, 20(2):692-743

[12] Fluke. (2021). What does infrared mean? Retrieved October 31, 2021, from https://www.fluke.com/en-us/learn/blog/thermal-imaging/how-thermal-cameras-use-infrared-thermography

[13] HessM, VanoniD, PetrovicV, KuesterF. High-resolution thermal imaging methodology for non-destructive evaluation of historic structures. Infrared Physics and Technology, 2015, 73: 219-225

[14] HoskereV, FouadA, FriedelD, YangW, TangY, NarazakiY, et al.. InstaDam: Open-source platform for rapid semantic segmentation of structural damage. Applied Sciences, 2021, 11(2):520

[15] HoskereV, NarazakiY, HoangTA, SpencerBF Jr. MaDnet: Multi task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. Journal of Civil Structural Health Monitoring, 2020, 10: 757-773

[16] KochC, DoychevaK, KasiV, AkinciB, FieguthP. A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Advanced Engineering Informatics, 2015, 29(2):196-210

[17] Liu, J., Zhang, S., Wang, S., & Metaxas, D. N. (2016). Multispectral deep neural networks for pedestrian detection. https://arxiv.org/abs/1611.02644.

[18] Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 3431–3440). Boston.

[19] LucasHC. The search for survival: Lessons from disruptive technologies, 2012 ABC-CLIO LLC

[20] Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016). ENet: Deep neural network architecture for real-time semantic. arXiv preprint arXiv:1606.02147.

[21] RaoY, PrathapaniN, NagabhooshanamE. Application of normalized cross correlation to image registration. International Journal of Research in Engineering and Technology, 2014, 3(5):12-15

[22] Yasrab, R., Gu, N., & Zhang, X. (2017). An encoder-decoder based convolution neural network (CNN) for future advanced driver assistance system (ADAS). Applied Sciences, 7(4).

[23] Shivakumar, S. S., Rodrigues, N., & Zhou, A. (2019). PST900: RGB-thermal calibration, dataset and segmentation network. Retrieved from http://arxiv.org/abs/1909.10980

[24] SpencerBF Jr, HoskereV, NarazakiY. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering, 2019, 5(2):199-222

[25] SunY, ZuoW, LiuM. RTFNet: RGB-thermal fusion network for semantic segmentation of urban scenes. IEEE Robotics and Automation Letters, 2019, 4(3):2576-2583

[26] WasherG, FenwickR, NelsonS, RumbayanR. Guidelines for thermographic inspection of concrete bridge components in shaded conditions. Transportation Research Record: Journal of the Transportation Research Board, 2013, 2360(1):13-20

[27] YeXW, JinR, YuncCB. A review on deep learning-based structural health monitoring of civil infrastructures. Smart Structures and Systems, 2019, 24(5):567-575

Funding

U.S. Army Corps of Engineers