Development of an automatic and knowledge-infused framework for structural health monitoring based on prompt engineering

Abstract

Collecting and analyzing vibration signals from structures under time-varying excitations is a non-destructive structural health monitoring approach that can provide meaningful information about the structures’ safety without interrupting their normal operations. This paper develops a novel framework using prompt engineering for seamlessly integrating users’ domain knowledge about vibration signals with the advanced inference ability of well-trained large language models (LLMs) to accurately identify the actual states of structures. The proposed framework involves formulating collected data into a standardized form, utilizing various prompts to gain useful insights into the dynamic characteristics of vibration signals, and implementing an in-house program with the help of LLMs to perform damage detection. The advantages, as well as limitations, of the proposed method are qualitatively and quantitatively assessed through two realistic case studies from literature, demonstrating that the present method is a new way to quickly construct practical and reliable structural health monitoring applications without requiring advanced programming/mathematical skills or obscure specialized programs.

Graphical abstract

Keywords

structural health monitoring / vibration / large language model / signal processing / prompt engineering

References

[1] Christ M , Braun N , Neuffer J , Kempa-Liehr A W . Time series feature extraction on basis of scalable hypothesis tests (TSfresh––A python package). Neurocomputing, 2018, 307: 72–77

[2] Dang H V , Raza M , Tran-Ngoc H , Bui-Tien T , Nguyen H X . Connection stiffness reduction analysis in steel bridge via deep CNN and modal experimental data. Structural Engineering and Mechanics. International Journal, 2021, 77(4): 495–508

[3] Dang H , Nguyen T T . Robust vibration output-only structural health monitoring framework based on multi-modal feature fusion and self-learning. Periodica Polytechnica. Civil Engineering, 2023, 67(2): 416–430

[4] Dang V H , Pham H A . Vibration-based building health monitoring using spatio-temporal learning model. Engineering Applications of Artificial Intelligence, 2023, 126: 106858

[5] Lin S , Zheng H , Han B , Li Y , Han C , Li W . Comparative performance of eight ensemble learning approaches for the development of models of slope stability prediction. Acta Geotechnica, 2022, 17(4): 1477–1502

[6] GuoHZhuangXRabczukT. A deep collocation method for the bending analysis of Kirchhoff plate. 2021, arXiv: 2102.02617

[7] Zhuang X , Guo H , Alajlan N , Zhu H , Rabczuk T . Deep autoencoder based energy method for the bending, vibration, and buckling analysis of Kirchhoff plates with transfer learning. European Journal of Mechanics. A, Solids, 2021, 87: 104225

[8] Samaniego E , Anitescu C , Goswami S , Nguyen-Thanh V M , Guo H , Hamdia K , Zhuang X , Rabczuk T . An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications. Computer Methods in Applied Mechanics and Engineering, 2020, 362: 112790

[9] Guo H , Zhuang X , Chen P , Alajlan N , Rabczuk T . Stochastic deep collocation method based on neural architecture search and transfer learning for heterogeneous porous media. Engineering with Computers, 2022, 38(6): 5173–5198

[10] Katz D M , Bommarito M J , Gao S , Arredondo P . GPT-4 passes the bar exam. Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences, 2024, 382: 257572753

[11] Strobelt H , Webson A , Sanh V , Hoover B , Beyer J , Pfster H , Rush A M . Interactive and visual prompt engineering for AD-HOC task adaptation with large language models. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 1146–1156

[12] Yong G , Jeon K , Gil D , Lee G . Prompt engineering for zero-shot and few-shot defect detection and classifcation using a visual-language pretrained model. Computer-Aided Civil and Infrastructure Engineering, 2023, 38(11): 1536–1554

[13] Lubiana T , Lopes R , Medeiros P , Silva J C , Goncalves A N A , Maracaja-Coutinho V , Nakaya H I . Ten quick tips for harnessing the power of ChatGPT in computational biology. PLOS Computational Biology, 2023, 19(9): e1011319

[14] BuschKRochlitzerASolaDLeopoldH. Just tell me: Prompt engineering in business process management. In: Proceedings of International Conference on Business Process Modeling, Development and Support. Cham: Springer Cham, 2023, 3–11

[15] Zhou K , Yang J , Loy C C , Liu Z . Learning to prompt for visionlanguage models. International Journal of Computer Vision, 2022, 130(9): 2337–2348

[16] PolakM PMorganD. Extracting accurate materials data from research papers with conversational language models and prompt engineering––Example of ChatGPT. 2023, arXiv: 2303.05352

[17] LoA WSinghM. From ELIZA to ChatGPT: The Evolution of NLP and Financial Applications. Cambridge, MA: MIT Libraries, 2023

[18] WangJLiuZZhaoLWuZMaCYuSDaiHYangQLiuYZhangS, . Review of large vision models and visual prompt engineering. 2023, arXiv: 2307.00855

[19] Hatakeyama-Sato K , Yamane N , Igarashi Y , Nabae Y , Hayakawa T . Prompt engineering of GPT-4 for chemical research: What can/cannot be done. Science and Technology of Advanced Materials: Methods, 2023, 3(1): 226030

[20] HestonT F. Prompt engineering for students of medicine and their teachers. 2023, arXiv: 2308.11628

[21] Zhu J J , Jiang J , Yang M , Ren Z J . ChatGPT and environmental research. Environmental Science & Technology, 2023, 57(46): 17667–17670

[22] NeaguA. How can large language models and prompt engineering be leveraged in computer science education? Thesis for the Master’s Degree. Delft: Delft University of Technology, 2023

[23] Peres R , Schreier M , Schweidel D , Sorescu A . On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice. International Journal of Research in Marketing, 2023, 40(2): 269–275

[24] XiaQMaekawaTHaraT. Unsupervised human activity recognition through two-stage prompting with ChatGPT. 2023, arXiv: 2306.02140

[25] ChopraA K. Dynamics of Structures: Theory and Applications to Earthquake Engineering. Upper Saddle River, NJ: Prentice Hall, 2006

[26] Wei J , Wang X , Schuurmans D , Bosma M , Ichter B , Xia F , Chi E H , Le Q V , Zhou D . Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022, 35: 24824–24837

[27] VaswaniAShazeerNParmarNUszkoreitJJonesLGomezA NKaiserLPolosukhinI. Attention is all you need. Advances in Neural Information Processing Systems, 2017: 6000–6010

[28] ChungH WHouLLongpreSZophBTayYFedusWLiEWangXDehghaniMBrahmaS, . Scaling instruction-finetuned language models. 2022, arXiv: 2210.11416

[29] RadfordANarasimhanKSalimansTSutskeverI. Improving language understanding by generative pre-training. 2018. Available at the website of OpenAI

[30] TouvronHMartinLStoneKAlbertPAlmahairiABabaeiYBashlykovNBatraSBhargavaPBhosaleS, . Llama 2: Open foundation and fne-tuned chat models. 2023, arXiv: 2307.09288

[31] BernalSBeckJVenturaC. An experimental benchmark problem in structural health monitoring. In: Proceedings of the Third International Workshop on Structural Health Monitoring. Stanford, CA: IWSHM, 2001

[32] Ye X , Cao Y , Liu A , Wang X , Zhao Y , Hu N . Parallel convolutional neural network toward high efficiency and robust structural damage identifcation. Structural Health Monitoring, 2023, 22(6): 3805–3826

[33] Chi Y , Cai C , Ren J , Xue Y , Zhang N . Damage location diagnosis of frame structure based on wavelet denoising and convolution neural network implanted with inception module and LSTM. Structural Health Monitoring, 2024, 23(1): 57–76

[34] DykeS JBernalDBeckJVenturaC. Experimental phase II of the structural health monitoring benchmark problem. In: Proceedings of the 16th ASCE Engineering Mechanics Conference. Reston, VA: ASCE, 2003

[35] FigueiredoEParkGFigueirasJFarrarCWordenK. Structural Health Monitoring Algorithm Comparisons Using Standard Data Sets. Technical Report LA-14393. 2009

[36] Hung D V , Hung H M , Anh P H , Thang N T . Structural damage detection using hybrid deep learning algorithm. Journal of Science and Technology in Civil Engineering, 2020, 14(2): 53–64

[37] Das S , Saha P , Patro S . Vibration-based damage detection techniques used for health monitoring of structures: A review. Journal of Civil Structural Health Monitoring, 2016, 6(3): 477–507

[38] KirillovAMintunERaviNMaoHRollandCGustafsonLXiaoTWhiteheadSBergA CLoW Y, . Segment anything. In: Proceedings of 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Paris: IEEE, 2023

[39] HorawalavithanaSAytonESharmaSHowlandSSubramanianMVasquezSCosbeyRGlenskiMVolkovaS. Foundation models of scientific knowledge for chemistry: Opportunities, challenges and lessons learned. In: Proceedings of BigScience Episode #5––Workshop on Challenges & Perspectives in Creating Large Language Models. Dublin: Association for Computational Linguistics, 2022, 160–172

[40] Si X , Wu X , Sheng H , Zhu J , Li Z . SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1–13

Acknowledgements

The research group of Development and Application of Advanced Materials and Modern Technologies in Construction would like to thank the Hanoi University of Civil Engineering for funding this research.

Competing interests

The authors declare that they have no competing interests.

RIGHTS & PERMISSIONS

2024 Higher Education Press