Extensible portal frame bridge synthetic dataset for structural semantic segmentation

Abstract

A number of bridges have collapsed around the world over the past years, with detrimental consequences on safety and traffic. To a large extend, such failures can be prevented by regular bridge inspections and maintenance, tasks that fall in the general category of structural health monitoring (SHM). Those procedures are time and labor consuming, which partly accounts for their neglect. Computer vision and artificial intelligence (AI) methods have the potential to ease this burden, by fully or partially automating bridge monitoring. A critical step in this automation is the identification of a bridge’s structural components. In this work, we propose an extensible synthetic dataset for structural component semantic segmentation of portal frame bridges (PFBridge). We first create a 3 dimensional (3D) generic mesh representing the bridge geometry, while respecting a set of rules. The definition of new, or the extension of the existing rules can adjust the dataset to specific needs. We then add textures and other realistic elements to the model, and create an automatically annotated synthetic dataset. The synthetic dataset is used in order to train a deep semantic segmentation model to identify bridge components on bridge images. The amount of available real images is not sufficient to entirely train such a model, but is used to refined the model trained on the synthetic data. We evaluate the contribution of the dataset to semantic segmentation by training several segmentation models on almost 2,000 synthetic images and then finetuning with 88 real images. The results show an increase of 28% on the F1-score when the synthetic dataset is used. To demonstrate a potential use case, the model is integrated in a 3D point cloud capturing system, producing an annotated point cloud where each point is associated with a semantic category (structural component). Such a point cloud can then be used in order to facilitate the generation of a bridge’s digital twin.

References

[1] AdministrationFFH. National bridge inspection standards. Federal Register, 2004, 69(239): 74419-39

[2] BaoY, LiJ, NagayamaT, XuY, SpencerBF Jr, LiH. The 1st international project competition for structural health monitoring (ipc-shm, 2020): A summary and benchmark problem. Structural Health Monitoring, 2021, 20(4): 2229-2239

[3] BianchiE, AbbottAL, TokekarP, HebdonM. Coco-bridge: Structural detail data set for bridge inspections. Journal of Computing in Civil Engineering, 2021, 35(3): 04021003

[4] BianchiE, HebdonM. Visual structural inspection datasets. Automation in Construction, 2022, 139: 104299

[5] Community, B.O. (2018). Blender - a 3D Modelling and Rendering Package. Blender Foundation, Stichting Blender Foundation, Amsterdam. Blender Foundation. http://www.blender.org.

[6] Dang, N., & Shim, C. (2020). Bim-based innovative bridge maintenance system using augmented reality technology. In: CIGOS 2019, Innovation for Sustainable Infrastructure: Proceedings of the 5th International Conference on Geotechnics, Civil Engineering Works and Structures. 1217–1222. Springer.

[7] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248–255.

[8] FischlerMA, BollesRC. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6): 381-395

[9] GaoY, MosalamKM. Deep transfer learning for image-based structural damage recognition. Computer-Aided Civil and Infrastructure Engineering, 2018, 33(9): 748-768

[10] HamY, HanKK, LinJJ, Golparvar-FardM. Visual monitoring of civil infrastructure systems via camera-equipped unmanned aerial vehicles (uavs): A review of related works. Visualization in Engineering, 2016, 4(1): 1-8

[11] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.

[12] HiasaS, CatbasFN, MatsumotoM, MitaniK. Monitoring concrete bridge decks using infrared thermography with high speed vehicles. Structural Monitoring and Maintenance, 2016, 3(3): 277-296

[13] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

[14] Hüthwohl, P., Lu, R., & Brilakis, I. (2016). Challenges of bridge maintenance inspection. DOI:10.17863/CAM.26634.

[15] KilicG, CanerA. Augmented reality for bridge condition assessment using advanced non-destructive techniques. Structure and Infrastructure Engineering, 2021, 17(7): 977-989

[16] Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

[17] Le Khac, V., Millan, A.L., Faure, E., & Gilcart, J.P. (1992). Ponts - cadres et portiques, guide de conception. Technical report, Centre des Techniques des ouvrages d’art (IQOA).

[18] LeeJS, ParkJ, RyuY-M. Semantic segmentation of bridge components based on hierarchical point cloud model. Automation in Construction, 2021, 130: 103847

[19] LiangX. Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization. Computer-Aided Civil and Infrastructure Engineering, 2019, 34(5): 415-430

[20] LinY-C, HabibA. Semantic segmentation of bridge components and road infrastructure from mobile lidar data. ISPRS Open Journal of Photogrammetry and Remote Sensing, 2022, 6: 100023

[21] Matsumoto, M., Mitani, K., & Catbas, F.N. (2013). Bridge assessment methods using image processing and infrared thermography technology. In: Proc., 92nd Annual Meeting, Transportation Research Board, Washington, DC.

[22] Narazaki, Y., Hoskere, V., Hoang, T .A., & Spencer Jr, B. F. (2018). Automated bridge component recognition using video data. arXiv preprint arXiv:1806.06820.

[23] NarazakiY, HoskereV, HoangTA, FujinoY, SakuraiA, SpencerBF Jr. Vision-based automated bridge component recognition with high-level scene consistency. Computer-Aided Civil and Infrastructure Engineering, 2020, 35(5): 465-482

[24] NarazakiY, HoskereV, YoshidaK, SpencerBF, FujinoY. Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts. Mechanical Systems and Signal Processing, 2021, 160: 107850

[25] Quattrocchi, C., Di Mauro, D., Furnari, A., Lopes, A., Moltisanti, M., & Farinella, G.M. (2023). Put your ppe on: A tool for synthetic data generation and related benchmark in construction site scenarios. In: VISIGRAPP (4: VISAPP). 656–663.

[26] RashidiM, GhodratM, SamaliB, KendallB, ZhangC. Remedial modelling of steel bridges through application of analytical hierarchy process (ahp). Applied Sciences, 2017, 7(2): 168

[27] Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. 234–241. Springer.

[28] Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A.M. (2016). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3234–3243.

[29] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4510–4520.

[30] SETRA: Iqoa — image de la qualité des ouvrages d’art. (1996). Technical report.

[31] SongG, WangC, WangB. Structural health monitoring (SHM) of civil structures. Applied Sciences, 2017

[32] VardanegaPJ, WebbGT, FidlerPRA, HuseynovF, KariyawasamKKGKD, MiddletonCR. 32 - bridge monitoring. In: Innovative bridge design, 2022 New York Butterworth-Heinemann 893-932

[33] Wavefront .OBJ. Retrieved July 03, 2023, from https://en.wikipedia.org/wiki/Wavefront_.obj_file.

[34] YangL, LinY-C, CaiH, HabibA. From scans to parametric bim: An enhanced framework using synthetic data augmentation and parametric modeling for highway bridges. Journal of Computing in Civil Engineering, 2024, 38(3): 04024008

[35] YangX, Rey CastilloE, ZouY, WotherspoonL, TanY. Automated semantic segmentation of bridge components from large-scale point clouds using a weighted superpoint graph. Automation in Construction, 2022, 142: 104519

[36] YuW, NishioM. Multilevel structural components detection and segmentation toward computer vision-based bridge inspection. Sensors, 2022, 22(9): 3502

Funding

French government

RIGHTS & PERMISSIONS

The Author(s)