A brief introductory review to deep generative models for civil structural health monitoring

Abstract

The use of deep generative models (DGMs) such as variational autoencoders, autoregressive models, flow-based models, energy-based models, generative adversarial networks, and diffusion models has been advantageous in various disciplines due to their high data generative skills. Using DGMs has become one of the most trending research topics in Artificial Intelligence in recent years. On the other hand, the research and development endeavors in the civil structural health monitoring (SHM) area have also been very progressive owing to the increasing use of Machine Learning techniques. As such, some of the DGMs have also been used in the civil SHM field lately. This short review communication paper aims to assist researchers in the civil SHM field in understanding the fundamentals of DGMs and, consequently, to help initiate their use for current and possible future engineering applications. On this basis, this study briefly introduces the concept and mechanism of different DGMs in a comparative fashion. While preparing this short review communication, it was observed that some DGMs had not been utilized or exploited fully in the SHM area. Accordingly, some representative studies presented in the civil SHM field that use DGMs are briefly overviewed. The study also presents a short comparative discussion on DGMs, their link to the SHM, and research directions.

Keywords

Deep generative models / Structural health monitoring / Generative adversarial networks / Diffusion models / Energy-based models / Flow-based models

References

[1] AckleyDH, HintonGE, SejnowskiTJ. A learning algorithm for Boltzmann machines. Cognitive Science, 1985, 9(1):147-169

[2] AnaissiA, ZandaviSM, SuleimanB, et al.. Multi-objective variational autoencoder: An application for smart infrastructure maintenance. Applied Intelligence, 2023, 53: 12047-12062

[3] Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research 70:214–223.

[4] AvciO, AbdeljaberO, KiranyazS, et al.. A review of vibration-based damage detection in civil structures: From traditional methods to machine learning and deep learning applications. Mechanical Systems and Signal Processing, 2021, 147

[5] AzimiM, EslamlouA, PekcanG. Data-driven structural health monitoring and damage detection through deep learning: state-of-the-art review. Sensors, 2020, 20: 2778

[6] BaoY, LiH. Machine learning paradigm for structural health monitoring. Structural Health Monitoring, 2021, 20: 1353-1372

[7] Bond-TaylorS, LeachA, LongY, WillcocksCG. Deep generative modelling: A comparative review of VAEs, GANs, normalizing flows, energy-based and autoregressive models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44: 7327-7347

[8] BoxGEP. Time series analysis; Forecasting and control, 1970 Holden-Day

[9] CatbasFN, Kijewski-CorreaT, AktanAE. Structural identification of constructed systems, 2013 American Society of Civil Engineers

[10] CatbasFN, LuleciF, ZakariaM, et al.. Extended reality (XR) for condition assessment of civil engineering structures: A literature review. Sensors, 2022, 22: 9560

[11] Chahal, K.S., He, M., Gao, A. et al. (2020). Energy-based models. https://atcold.github.io/pytorch-Deep-Learning/

[12] Dhariwal, P., & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. https://arxiv.org/abs/2105.05233

[13] Dinh, L., Krueger, D., & Bengio, Y. (2014). NICE: Non-linear independent components estimation. International Conference on Learning Representations. https://arxiv.org/abs/1410.8516

[14] Dinh, L., Krueger, D., & Bengio, Y. (2015). NICE: Non-linear independent components estimation. In: ICLR 2015.

[15] Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016). Density estimation using Real NVP. Published as a conference paper at ICLR 2017.

[16] Du, Y., & Mordatch, I. (2019). Implicit generation and modeling with energy-based models. In: 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

[17] Durkan, C., Bekasov, A., Murray, I., & Papamakarios, G. (2019). Neural spline flows. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

[18] EntezamiA, SarmadiH, SalarM, et al.. A novel data-driven method for structural health monitoring under ambient vibration and high-dimensional features by robust multidimensional scaling. Structural Health Monitoring, 2021

[19] FanG, HeZ, LiJ. Structural dynamic response reconstruction using self-attention enhanced generative adversarial networks. Engineering Structures, 2023, 276

[20] Goodfellow, I. (2016). NIPS 2016 Tutorial: Generative adversarial networks. http://arxiv.org/abs/1701.00160

[21] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al. (2014). Generative adversarial networks. Proceedings of the 27th International Conference on Neural Information Processing Systems (Vol 2, pp 2672–2680). https://dl.acm.org/doi/https://doi.org/10.5555/2969033.2969125

[22] Grathwohl, W., Chen, R.T.Q., Bettencourt, J., et al. (2018). FFJORD: Free-form continuous dynamics for scalable reversible generative models. Published as a conference paper at ICLR 2019.

[23] Gray, R. M. (2010). Linear predictive coding and the internet protocol: A survey of LPC and a history of realtime digital speech on packet networks. Foundations and Trends.

[24] GulM, CatbasFN. Statistical pattern recognition for Structural Health Monitoring using time series modeling: Theory and experimental verifications. Mechanical Systems and Signal Processing, 2009, 23: 2192-2204

[25] Gulrajani, I., Ahmed, F., Arjovsky, M., et al. (2017). Improved training of Wasserstein GANs. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17) (pp. 5769–5779). Curran Associates Inc., Red Hook, NY, USA.

[26] Ho, J., Chan, W., Saharia, C., et al. (2022). Imagen video: High definition video generation with diffusion models. https://doi.org/10.48550/arXiv.2210.02303

[27] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.

[28] HopfieldJJ. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 1982, 79(8):2554-2558

[29] JiangH, WanC, YangK, et al.. Continuous missing data imputation with incomplete dataset by generative adversarial networks–based unsupervised learning for long-term bridge health monitoring. Structural Health Monitoring, 2022, 21: 1093-1109

[30] KarrasT, LaineS, AilaT. A style-based generator architecture for generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(Dec. 2021):4217-4228

[31] Kingma, D.P., & Dhariwal. P. (2018). Glow: Generative flow with invertible 1x1 convolutions. 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.

[32] Kingma, D.P., & Welling, M. (2013). Auto-encoding variational Bayes. https://arxiv.org/abs/1312.6114

[33] KingmaDP, WellingM. An introduction to variational autoencoders. Foundations and Trends in Machine Learning., 2019, 12(4):307-392

[34] LeCun, Y., Chopra, S., & Hadsell, R. (2006). A tutorial on energy-based learning. In: Predicting Structured Data. MIT Press.

[35] LeiX, SunL, XiaY. Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks. Structural Health Monitoring, 2021, 20: 2069-2087

[36] LiuA, WangL, BornnL, FarrarC. Robust structural health monitoring under environmental and operational uncertainty with switching state-space autoregressive models. Structural Health Monitoring, 2019, 18: 435-453

[37] LiuJ, WeiY, BergésM, et al.. WangK-W, SohnH, HuangH, LynchJP, et al.. Detecting anomalies in longitudinal elevation of track geometry using train dynamic responses via a variational autoencoder. Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems, 2019 SPIE 49

[38] LuleciF, AvciO, CatbasFN. Improved undamaged-to-damaged acceleration response translation for structural health monitoring. Engineering Applications of Artificial Intelligence, 2023, 122

[39] Luleci, F., & Catbas, F.N. (2022). Structural state translation: Condition transfer between civil structures using domain-generalization for structural health monitoring. https://doi.org/10.48550/arXiv.2212.14048

[400] Luleci, F., & Catbas, F.N. (2023). Condition transfer between prestressed bridges using structural state translation for structural health monitoring. AI in Civil Engineering. https://doi.org/10.1007/s43503-023-00016-0

[40] LuleciF, CatbasFN, AvciO. Generative adversarial networks for labeled acceleration data augmentation for structural damage detection. J Civ Struct Health Monit, 2021

[41] LuleciF, CatbasFN, AvciO. A literature review: Generative adversarial networks for civil structural health monitoring. Front Built Environ, 2022

[42] LuleciF, CatbasFN, AvciO. CycleGAN for undamaged-to-damaged domain translation for structural health monitoring and damage detection. Mechanical Systems Signal Processing, 2023

[43] LuleciF, CatbasFN, AvciO. Generative adversarial networks for labeled acceleration data augmentation for structural damage detection. Journal of Civil Structural Health Monitoring, 2023, 13: 181-198

[44] MaX, LinY, NieZ, MaH. Structural damage identification based on unsupervised feature-extraction via variational auto-encoder. Measurement, 2020, 160

[45] MalekzadehM, AtiaG, CatbasFN. Performance-based structural health monitoring through an innovative hybrid data interpretation framework. Journal of Civil Structural Health Monitoring, 2015, 5: 287-305

[46] Mittal, M., & Behl, H.S. (2018). Variational autoencoders: A brief survey. https://mayankm96.github.io/assets/documents/projects/cs698-report.pdf

[47] Pollastro, A., Testa, G., Bilotta, A., & Prevete, R. (2022). Semi-supervised detection of structural damage using variational autoencoder and a one-class support vector machine. IEEE, https://doi.org/10.1109/ACCESS.2023.3291674

[48] Psathas, A.P., Iliadis, L., Achillopoulou, D., et al. (2022). Autoregressive deep learning models for bridge strain prediction (pp 150–164).

[49] RajeevA, PamwaniL, OjhaS, ShelkeA. Adaptive autoregressive modelling based structural health monitoring of RC beam-column joint subjected to shock loading. Structural Health Monitoring, 2022

[50] Ramesh, A., Dhariwal, P., Nichol, A., et al. (2022). Hierarchical text-conditional image generation with CLIP latents. https://doi.org/10.48550/arXiv.2204.06125

[51] Rezende, D. J., & Mohamed, S. (2015). Variational inference with normalizing flows. In: Proceedings of the 32nd International Conference on Machine Learning. JMLR: W&CP vol 37, Lille, France.

[52] RuthottoL, HaberE. An introduction to deep generative modeling. GAMM-Mitteilungen, 2021

[53] Saharia, C., Chan, W., Saxena, S., et al. (2022). Photorealistic text-to-image diffusion models with deep language understanding. https://doi.org/10.48550/arXiv.2205.11487

[54] SajediS, LiangX. Deep generative Bayesian optimization for sensor placement in structural health monitoring. Computer-Aided Civil and Infrastructure Engineering, 2022, 37: 1109-1127

[55] Salimans, T., Goodfellow, I., Zaremba, W., et al. (2016). Improved techniques for training GANs. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS'16) (pp 234–2242). Curran Associates Inc., Red Hook, NY, USA.

[56] Singer, U., Polyak, A., Hayes, T., et al. (2022). Make-a-video: Text-to-video generation without text-video data. Published as a conference paper at ICLR 2023. https://arxiv.org/abs/2209.14792

[57] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., & Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015 JMLR: W&CP volume 37.

[58] Soleimani-BabakamaliMH, SepasdarR, NasrollahzadehK, et al.. Toward a general unsupervised novelty detection framework in structural health monitoring. Computer-Aided Civil and Infrastructure Engineering, 2022, 37: 1128-1145

[59] Soleimani-BabakamaliMH, ZakerEsteghamatiM. Estimating seismic demand models of a building inventory from nonlinear static analysis using deep learning methods. Engineering Structures, 2022, 266

[60] Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

[61] TomczakJM. Deep Generative Modeling, 2022 1 Springer

[62] Ulhaq, A., Akhtar, N., & Pogrebna, G. (2022). Efficient diffusion models for vision: A survey. https://arxiv.org/abs/2210.09292

[300] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. https://arxiv.org/abs/1706.03762?context=cs

[63] van den Oord, A., Dieleman, S., Zen, H., et al. (2016b). WaveNet: A generative model for raw audio. https://arxiv.org/abs/1609.03499

[64] van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016a). Pixel recurrent neural networks. In: Proceedings of the 33rd International Conference on Machine Learning. JMLR.org

[65] WangK, ZhangX, HaoQ, et al.. Application of improved least-square generative adversarial networks for rail crack detection by AE technique. Neurocomputing, 2019, 332: 236-248

[66] Wang, X., Yu, K., Wu, S., et al. (2018). ESRGAN: Enhanced super-resolution generative adversarial networks. In: L. Leal-Taixé, S. Roth (Eds), Computer Vision—ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science (vol 11133). Springer, Cham. https://doi.org/10.1007/978-3-030-11021-5_5

[67] Wang, Z., Zheng, H., He, P., et al. (2022). Diffusion-GAN: Training GANs with diffusion. https://arxiv.org/abs/2206.02262

[68] Weng L. (2021). What are diffusion models? Lil’Log. In: https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.

[69] XuY, LuX, CetinerB, TacirogluE. Real-time regional seismic damage assessment framework based on long short-term memory neural network. Computer-Aided Civil and Infrastructure Engineering, 2021, 36: 504-521

[70] Xu, Y., Tian, Y., Zhang, Y., & Li, H. (2021b). Deep-learning-based bridge condition assessment by probability density distribution reconstruction of girder vertical deflection and cable tension using unsupervised image transformation model. pp 35–45.

[71] YuanZ, ZhuS, ChangC, et al.. An unsupervised method based on convolutional variational auto-encoder and anomaly detection algorithms for light rail squat localization. Construction and Building Materials, 2021, 313

[72] Zhai, S., Cheng, Y., & Lu, W. (2016). Deep structured energy based models for anomaly detection. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48, New York

[73] Zhang, Q., & Chen, Y. (2021). Diffusion s. In: 35th Conference on Neural Information Processing Systems.

[74] Zhao, J., Mathieu, M., & LeCun, Y. (2017). Energy-based generative adversarial networks. In: Published as a conference paper at ICLR 2017.

[75] ZhouY, ShuX, BaoT, et al.. Dam safety assessment through data-level anomaly detection and information fusion. Structural Health Monitoring, 2022

[76] Zhu, J.-Y., Park, T., Isola, P., & Efros, A.A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. IEEE International Conference on Computer Vision (ICCV), Venice, Italy (pp. 2242–2251). doi: https://doi.org/10.1109/ICCV.2017.244

Funding

National Aeronautics and Space Administration(80NSSC20K0326)