Multi-patch attention Transformer for multivariate long-term time series forecasting of TBM excavation parameters

Abstract

To address the research gap in multivariable long-term time series forecasting in the field of tunnel boring machine (TBM) and provide long-term insights for decision-making in TBM construction, this paper studies a novel Transformer-based forecasting model. Leveraging a multi-patch attention mechanism, the newly developed multi-patch attention Transformer (MPAT) model is designed to predict long-term trends of multiple TBM operation parameters. The innovation lies in finding the most relevant time delay series of the input series through autocorrelation calculation, and designing a multi-patch attention mechanism to replace the traditional attention mechanism of Transformer, so that the model can capture local and global information of the series and improve the accuracy of long-term prediction of high-frequency and weakly periodic TBM data. Experimental results have shown that MPAT model has a significant effect on capturing TBM data in terms of temporal dependencies. In a case study, we applied MPAT to the Rongjiang Guanbu Water Diversion Project in Guangdong Province and predicted four excavation parameters. The experimental results show that MPAT exhibits accurate predictive ability when the input length is 36 and the outputs are 12, 24, 48, and 72, respectively. In comparison with some state-of-the-art models, MPAT outperforms MSE by 19.1%, 23.6%, 36.4%, and 48.3%, respectively. We also discussed the impact of input length and the number of patches on performance, and found that each prediction length has the best input length corresponding to it, and longer inputs don’t represent more accurate predictions. The determination of the number of patches should also depend on the input length, as too many or too few patches can affect the capture of local information in the sequence.

Keywords

Tunnel boring machine / Transformer / Attention mechanism / Time series / Long-term forecasting / Excavation parameters

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

CRediT authorship contribution statement

Mingjun Liu: Writing - original draft, Visualization, Validation, Conceptualization. Jianqin Liu: Writing - review & editing, Writing - original draft, Validation, Methodology, Investigation, Data curation, Conceptualization. Wei Guo: Methodology, Investigation, Data curation. Hongxu Liu: Visualization, Methodology, Investigation, Data curation. Xiao Guo: Investigation, Formal analysis.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

The authors wish to thank the State Key Laboratory of Shield Machine and Boring Technology for sharing their data on the water conveyance tunnel. This project is supported by the National Natural Science Foundation of China (Grant No. 52075370).

References

[1] Abdulrahman, M. L., Ibrahim, K. M., Gital, A. Y., Zambuk, F. U., Ja'afaru, B., Yakubu, Z. I., & Ibrahim, A. (2021). A review on deep learning with focus on deep recurrent neural network for electricity forecasting in residential building. Procedia Computer Science, 193, 141-154.

[2] Ates, U., & Copur, H. (2023). Investigation of parameters affecting vibration patterns generated during excavation by EPB TBMs. Tunnelling and Underground Space Technology, 138, 105185.

[3] Balta, G. K., Dikmen, I., & Birgonul, M. T. (2021). Bayesian network based decision support for predicting and mitigating delay risk in TBM tunnel projects. Automation in Construction, 129, 103819.

[4] Bani-Almarjeh, M., & Kurdy, M. B. (2023). Arabic abstractive text summarization using RNN-based and transformer-based architectures. Information Processing and Management, 60(2), 103227.

[5] Berne, B. J., Boon, J. P., & Rice, S. A. (1966). On the calculation of autocorrelation functions of dynamical variables. The Journal of Chemical Physics, 45(4), 1086-1096.

[6] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186).

[7] Du, Y., Pei, B., Zhao, X., & Ji, J. (2020). Deep scaled dot-product attention based domain adaptation model for biomedical question answering. Methods, 173, 69-74.

[8] Duhamel, P., & Vetterli, M. (1990). Fast Fourier transforms: a tutorial review and a state of the art. Signal Processing, 19(4), 259-299.

[9] Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179-211.

[10] Fu, X., & Zhang, L. (2021). Spatio-temporal feature fusion for real-time prediction of TBM operating parameters: A deep learning approach. Automation in Construction, 132, 103937.

[11] Guo, X., Guo, W., Liu, J., Qiao, J., & Hu, G. (2023). Prediction of the geological indicators in TBM tunnel based on optimized proportion of surrounding rock grades. Underground Space, 11, 204-217.

[12] Hou, S., & Liu, Y. (2022). Early warning of tunnel collapse based on adam-optimised long short-term memory network and TBM operation parameters. Engineering Applications of Artificial Intelligence, 112, 104842.

[13] Krause, H. (1976). Geologische erfahrungen beim einsatz von tunnelvortriebsmaschinen in Baden-Württemberg. In Neue Erkenntnisse im Hohlraumbau-Fundierungen im Fels/Latest Findings in the Construction of Underground Excavations-Rock Foundations: Vorträge des 24. Geomechanik-Kolloquiums der Österreichischen Gesellschaft für Geomechanik/Contributions to the 24th Geomechanical Colloquium of the Austrian Society for Geomechanics Salzburg, 2. und 3. Oktober 1975 (pp. 49-60). Springer Vienna.

[14] Lee, H. L., Song, K. I., Qi, C., Kim, J. S., & Kim, K. S. (2021). Real-time prediction of operating parameter of TBM during tunneling. Applied Sciences, 11(7), 2967.

[15] Liu, B., Wang, J., Wang, R., Wang, Y., & Zhao, G. (2023). Intelligent decision-making method of TBM operating parameters based on multiple constraints and objective optimization. Journal of Rock Mechanics and Geotechnical Engineering, 15(11), 2842-2856.

[16] Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., & Dustdar, S. (2022). Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. The Tenth International Conference on Learning Representations..

[17] Ma, M., Fu, L., Zhai, Z., & Sun, R. B. (2024). Transformer based Kalman Filter with EM algorithm for time series prediction and anomaly detection of complex systems. Measurement, 229, 114378.

[18] Mahdevari, S., Shahriar, K., Yagiz, S., & Shirazi, M. A. (2014). A support vector regression model for predicting tunnel boring machine penetration rates. International Journal of Rock Mechanics and Mining Sciences, 72, 214-229.

[19] Makridakis, S. (1994). Time series prediction: Forecasting the future and understanding the past. International Journal of Forecasting, 10(3), 463-466.

[20] Manju, B. R., & Sneha, M. R. (2020). ECG denoising using wiener filter and kalman filter. Procedia Computer Science, 171, 273-281.

[21] Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2022). A time series is worth 64 words: Long-term forecasting with transformers. ar Xiv preprint ar Xiv:2211.14730.

[22] Pan, Y., Fu, X., & Zhang, L. (2022). Data-driven multi-output prediction for TBM performance during tunnel excavation: An attention-based graph convolutional network approach. Automation in Construction, 141, 104386.

[23] Rashmi, C. R., & Shantala, C. P. (2024). Evaluating deep learning with different feature scaling techniques for EEG-based music entrainment brain computer interface. e-prime-advances in electrical engineering. Electronics and Energy, 7, 100448.

[24] Ren, J., Wang, Z., Pang, Y., & Yuan, Y. (2022). Genetic algorithmassisted an improved AdaBoost double-layer for oil temperature prediction of TBM. Advanced Engineering Informatics, 52, 101563.

[25] Schmidhuber, J., & Hochreiter, S. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.

[26] Shan, F., He, X., Armaghani, D. J., & Sheng, D. (2024). Effects of data smoothing and recurrent neural network (RNN) algorithms for realtime forecasting of tunnel boring machine (TBM) performance. Journal of Rock Mechanics and Geotechnical Engineering, 16(5), 1538-1551.

[27] Shan, F., He, X., Armaghani, D. J., Zhang, P., & Sheng, D. (2022). Success and challenges in predicting TBM penetration rate using recurrent neural networks. Tunnelling and Underground Space Technology, 130, 104728.

[28] Sun, W., Shi, M., Zhang, C., Zhao, J., & Song, X. (2018). Dynamic load prediction of tunnel boring machine (TBM) based on heterogeneous in-situ data. Automation in Construction, 92, 23-34.

[29] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (pp.30).

[30] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 22419-22430.

[31] Xue, Y., Wang, J., Zhou, M., Liu, J., Guo, Y., & Wang, J. (2022). Prediction of optimum TBM penetration strategy with minimum energy consumption in hard rocks. Computers and Geotechnics, 148, 104844.

[32] Zhang, D. (2019). Wavelet transform. In Fundamentals of image data mining:Analysis, Features, Classification and Retrieval (pp.35-44). Cham: Springer International Publishing.

[33] Zhang, P., Wu, H. N., Chen, R. P., Dai, T., Meng, F. Y., & Wang, H. B. (2020). A critical evaluation of machine learning and deep learning in shield-ground interaction prediction. Tunnelling and Underground Space Technology, 106, 103593.

[34] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021, May). Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 12, pp. 11106-11115).

[35] Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., & Jin, R. (2022). Fedformer: Frequency enhanced decomposed transformer for longterm series forecasting. International conference on machine learning (pp.27268-27286). PMLR.