CNN-LSTM based incremental attention mechanism enabled phase-space reconstruction for chaotic time series prediction

Abstract

To improve the prediction accuracy of chaotic time series and reconstruct a more reasonable phase space structure of the prediction network, we propose a convolutional neural network (CNN)-long short-term memory (LSTM) prediction model based on the incremental attention mechanism. Firstly, a traversal search is conducted through the traversal layer for finite parameters in the phase space. Then, an incremental attention layer is utilized for parameter judgment based on the dimension weight criteria (DWC). The phase space parameters that best meet DWC are selected and fed into the input layer. Finally, the constructed CNN-LSTM network extracts spatiotemporal features and provides the final prediction results. The model is verified using Logistic, Lorenz, and sunspot chaotic time series, and the performance is compared from the two dimensions of prediction accuracy and network phase space structure. Additionally, the CNN-LSTM network based on incremental attention is compared with LSTM, CNN, recurrent neural network (RNN), and support vector regression (SVR) for prediction accuracy. The experiment results indicate that the proposed composite network model possesses enhanced capability in extracting temporal features and achieves higher prediction accuracy. Also, the algorithm to estimate the phase space parameter is compared with the traditional CAO, false nearest neighbor, and C-C, three typical methods for determining the chaotic phase space parameters. The experiments reveal that the phase space parameter estimation algorithm based on the incremental attention mechanism is superior in prediction accuracy compared with the traditional phase space reconstruction method in five networks, including CNN-LSTM, LSTM, CNN, RNN, and SVR.

Key words： Chaotic time series; Incremental attention mechanism; Phase-space reconstruction; Chaotic time series; Incremental attention mechanism; Phase-space reconstruction

References

[1]	M. Han, S.-B. Feng, C.L.P. Chen, M.-L. Xu, T. Qiu. Structured manifold broad learning system: A manifold perspective for large-scale chaotic time series analysis and prediction. IEEE T. Knowl. Data En., 31 (9) (2019), pp. 1809-1821.
[2]	Y.-N. Wang. Automatic Classification of Sunspots Using Convolutional Neural Networks. M.S. Thesis. Harbin Institute of Technology, Harbin, China (2020).
[3]	X.-J. Li. Nonlinear Characteristics Identification and Analysis for River Runoff Time Series, Ph.D, Dissertation, Wuhan University, Wuhan, China (2013).
[4]	J. Gao, L.-L. Jiang, Z.-Y. Xu. Chaos game representation walk model for the protein sequences. Chin. Phys. B, 18 (10) (2009), p. 4571.
[5]	G.E.P. Box, G.M. Jenkins, G.C. Reinsel, G.M. Ljung. Time Series Analysis: Forecasting and Control (fifth ed.), John Wiley & Sons, Hoboken USA (2015).
[6]	W.-D. Yang, J.-Z. Wang, T. Niu, P. Du. A novel system for multi-step electricity price forecasting for electricity market management. Appl. Soft Comput., 88 (2020), Article 106029. View articleGoogle Scholar.
[7]	D.E. Rumelhart, G.E. Hinton, R.J. Williams. Learning representations by back-propagating errors. Nature, 323 (6088) (1986), pp. 533-536.
[8]	M. Han, J.-H. Xi, S.-G. Xu, F.-L. Yin. Prediction of chaotic time series based on the recurrent predictor neural network. IEEE T. Signal Processing, 52 (12) (2004), pp. 3409-3416.
[9]	L.J. Herrera, H. Pomares, I. Rojas, A. Guillén, A. Prieto, O. Valenzuela. Recursive prediction for long term time series forecasting using advanced models. Neurocomputing, 70 (16/18) (2007), pp. 2870-2880. View articleGoogle Scholar.
[10]	W.-J. Wang, C.-Q. Men, W.-Z. Lu. Online prediction model based on support vector machine. Neurocomputing, 71 (4/6) (2008), pp. 550-558. View articleGoogle Scholar.
[11]	M. Sangiorgio, F. Dercole. Robustness of LSTM neural networks for multi-step forecasting of chaotic time series, Chaos Soliton. Fractals, 139 (2020), Article 110045. View articleGoogle Scholar.
[12]	S.-X. Li, X.-K. Sun, L. Yin, S.-S. Zhang. A GPS height time series prediction method based on chaos theory and LSTM. J. Navigation Positioning, 8 (1) (2020), pp. 65-73.
[13]	Y.-X. Zhang, X.-Y. Zhao, Y.-F. Yang. Chaotic characteristic analysis of daily irrigation return flow time series. Proc. of 8th Intl. Conf. on Intelligent Computing and Signal Processing, Xi’an, China (2023), pp. 258-261.
[14]	J.-T. Zeng, L. Jin, Y.-J. Zhou. Sea-surface target detection based on neural network prediction. Proc. of IEEE 3rd Intl. Conf. on Software Engineering and Artificial Intelligence, Xiamen, China (2023), pp. 199-203.
[15]	L.-Y. Cao. Practical method for determining the minimum embedding dimension of a scalar time series. Physica D, 110 (1/2) (1997), pp. 43-50. View articleGoogle Scholar.
[16]	M.B. Kennel, R. Brown, H.D.I. Abarbanel. Determining embedding dimension for phase-space reconstruction using a geometrical construction. Phys. Rev. A, 45 (6) (1992), pp. 3403-3411.
[17]	H.S. Kim, R. Eykholt, J.D. Salas. Nonlinear dynamics, delay times,embedding windows. Physical D, 127 (1/2) (1999), pp. 48-60. View articleGoogle Scholar.
[18]	H.-H. Li, K. Li, J.-H. Guo, Y. Yang, Y. Zhou. NPSR: Neural network enabled phase-space reconstruction for wireless channel prediction. Proc. of IEEE Globecom Workshops, Rio de Janeiro, Brazil (2022), pp. 1729-1735.
[19]	N.H. Packard, J.P. Crutchfield, J.D. Farmer, R.S. Shaw. Geometry from a time series. Phys. Rev. Lett., 45 (9) (Sept.1980), pp. 712-716.
[20]	F. Takens.Detecting strange attractors in turbulence. Proc. of Dynamical Systems and Turbulence, Warwick, UK(1981), pp. 366-381.
[21]	S.-M. Wang, H. Shuai, Q.-S. Liu. Phase space reconstruction driven spatio-temporal feature learning for dynamic facial expression recognition, IEEE T. Affect. Computing, 13 (3) (2022), pp. 1466-1476.
[22]	V. Mnih, N. Heess, A. Graves, K. Kavukcuoglu.Recurrent models of visual attention. Proc. of 27th Intl. Conf. on Neural Information Processing Systems, Montreal, Canada(2014), pp. 2204-2212.
[23]	Y. LeCun, B. Boser, J.S. Denker, et al. Backpropagation applied to handwritten zip code recognition. Neural Comput., 1 (4) (1989), pp. 541-551.
[24]	W.-J. Huang, Y.-T. Li, Y. Huang. Prediction of chaotic time series using hybrid neural network and attention mechanism. Acta Phys. Sin., 70 (1) (2021), pp. 1-9. 010501.
[25]	X.-L. Ma, Z.-M. Tao, Y.-H. Wang, H.-Y. Yu, Y.-P. Wang. Long short-term memory neural network for traffic speed prediction using remote microwave sensor data, Transport. Res. C: Emer., 54 (2015), pp. 187-197. View articleGoogle Scholar.
[26]	SILSO, World data center--sunspot number and long-term solar observations, royal observatory of Belgium, online sunspot number catalogue [Online]. Available, https://www.sidc.be/sunspot-data/SIDCpub.php.