Adapting to the stream: an instance-attention GNN method for irregular multivariate time series data

Abstract

Multivariate time series (MTS) data are vital for various applications, particularly in machine learning tasks. However, challenges such as sensor failures can result in irregular and misaligned data with missing values, thereby complicating their analysis. While recent advancements use graph neural networks (GNNs) to manage these Irregular Multivariate Time Series (IMTS) data, they generally require a reliable graph structure, either pre-existing or inferred from adequate data to properly capture node correlations. This poses a challenge in applications where IMTS data are often streamed and waiting for future data to estimate a suitable graph structure becomes impractical. To overcome this, we introduce a dynamic GNN model suited for streaming characteristics of IMTS data, incorporating an instance-attention mechanism that dynamically learns and updates graph edge weights for real-time analysis. We also tailor strategies for high-frequency and low-frequency data to enhance prediction accuracy. Empirical results on real-world datasets demonstrate the superiority of our proposed model in both classification and imputation tasks.

Graphical abstract

Keywords

multivariate time series / irregular multivariate time series / graph neural networks

References

[1] Fawaz H I, Forestier G, Weber J, Idoumghar L, Muller P A . Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 2019, 33( 4): 917–963

[2] Schirmer M, Eltayeb M, Lessmann S, Rudolph M. Modeling irregular time series with continuous recurrent units. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 19388−19405

[3] Sezer O B, Gudelek M U, Ozbayoglu A M . Financial time series forecasting with deep learning: a systematic literature review: 2005–2019. Applied Soft Computing, 2020, 90: 106181

[4] Feng C, Tian P. Time series anomaly detection for cyber-physical systems via neural system identification and Bayesian filtering. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 2858−2867

[5] Mathur A P, Tippenhauer N O. SWaT: a water treatment testbed for research and training on ICS security. In: Proceedings of 2016 International Workshop on Cyber-physical Systems for Smart Water Networks. 2016, 31−36

[6] Ahmed C M, Palleti V R, Mathur A P. WADI: a water distribution testbed for research in the design of secure cyber physical systems. In: Proceedings of the 3rd International Workshop on Cyber-Physical Systems for Smart Water Networks. 2017, 25−28

[7] Cao D, Wang Y, Duan J, Zhang C, Zhu X, Huang C, Tong Y, Xu B, Bai J, Tong J, Zhang Q. Spectral temporal graph neural network for multivariate time-series forecasting. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1491

[8] Yehuda Y, Freedman D, Radinsky K. Self-supervised classification of clinical multivariate time series using time series dynamics. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 5416−5427

[9] Li G, Choi B, Xu J, Bhowmick S S, Chun K P, Wong G L H. ShapeNet: a shapelet-neural network approach for multivariate time series classification. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 8375−8383

[10] Tan Q, Ye M, Yang B, Liu S, Ma A J, Yip T C F, Wong G L H, Yuen P C. DATA-GRU: dual-attention time-aware gated recurrent unit for irregular multivariate time series. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 930−937

[11] Kidger P, Morrill J, Foster J, Lyons T. Neural controlled differential equations for irregular time series. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 562

[12] Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W. Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 11106−11115

[13] Nie Y, Nguyen N H, Sinthong P, Kalagnanam J. A time series is worth 64 words: long-term forecasting with transformers. In: Proceedings of the 11th International Conference on Learning Representations. 2023

[14] Bahadori M T, Liu Y. Granger causality analysis in irregular time series. In: Proceedings of the 12th SIAM International Conference on Data Mining. 2012, 660−671

[15] Schulz M, Stattegger K . Spectrum: spectral analysis of unevenly spaced paleoclimatic time series. Computers & Geosciences, 1997, 23( 9): 929–945

[16] Rehfeld K, Marwan N, Heitzig J, Kurths J . Comparison of correlation analysis techniques for irregularly sampled time series. Nonlinear Processes in Geophysics, 2011, 18( 3): 389–404

[17] Babu P, Stoica P . Spectral analysis of nonuniformly sampled data - a review. Digital Signal Processing, 2010, 20( 2): 359–378

[18] Stoica P, Li J, He H . Spectral analysis of nonuniformly sampled data: a new approach versus the periodogram. IEEE Transactions on Signal Processing, 2009, 57( 3): 843–858

[19] Li S C X, Marlin B. Classification of sparse and irregularly sampled time series with mixtures of expected Gaussian kernels and random features. In: Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence. 2015, 484−493

[20] Li S C X, Marlin B. A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 1804−1812

[21] Che Z, Purushotham S, Cho K, Sontag D, Liu Y . Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 2018, 8( 1): 6085

[22] Rubanova Y, Chen R T Q, Duvenaud D. Latent ODEs for irregularly-sampled time series. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 478

[23] Wang Z, Zhang Y, Jiang A, Zhang J, Li Z, Gao J, Li K, Lu C, Ren Z. Improving irregularly sampled time series learning with time-aware dual-attention memory-augmented networks. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021, 3523−3527

[24] Shukla S N, Marlin B M. Multi-time attention networks for irregularly sampled time series. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[25] Tipirneni S, Reddy C K . Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD), 2022, 16( 6): 105

[26] Zhang J, Zheng S, Cao W, Bian J, Li J. Warpformer: a multi-scale modeling approach for irregular clinical time series. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 3273−3285

[27] Chen Y, Ren K, Wang Y, Fang Y, Sun W, Li D. ContiFormer: continuous-time transformer for irregular time series modeling. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 2042

[28] Sun C, Hong S, Song M, Chou Y H, Sun Y, Cai D, Li H. TE-ESN: time encoding echo state network for prediction based on irregularly sampled time series data. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence. 2021, 3010−3016

[29] Cini A, Marisca I, Alippi C. Filling the G_ap_s: multivariate time series imputation by graph neural networks. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[30] Wei Y, Peng J, He T, Xu C, Zhang J, Pan S, Chen S. Compatible transformer for irregularly sampled multivariate time series. In: Proceedings of 2023 IEEE International Conference on Data Mining. 2023, 1409−1414

[31] Zhang X, Zeman M, Tsiligkaridis T, Zitnik M. Graph-guided network for irregularly sampled multivariate time series. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[32] Wang Z, Jiang T, Xu Z, Zhang J, Gao J . Irregularly sampled multivariate time series classification: a graph learning approach. IEEE Intelligent Systems, 2023, 38( 3): 3–11

[33] Wang Z, Jiang T, Xu Z, Gao J, Wu O, Yan K, Zhang J. Uncovering multivariate structural dependency for analyzing irregularly sampled time series. In: Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases: Research Track. 2023, 238−254

[34] Liu Y, Liu Q, Zhang J W, Feng H, Wang Z, Zhou Z, Chen W. Multivariate time-series forecasting with temporal polynomial graph neural networks. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2024, 1411

[35] Rodrigues de Barros M, Rissi T L, Cabrera E F, Tannuri E A, Gomi E S, Barreira R A, Costa A H R. Embracing data irregularities in multivariate time series with recurrent and graph neural networks. In: Proceedings of the 12th Brazilian Conference on Intelligent Systems. 2023, 3−17

[36] Cao W, Wang D, Li J, Zhou H, Li L, Li Y. BRITS: bidirectional recurrent imputation for time series. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 6776−6786

[37] Luo Y, Cai X, Zhang Y, Xu J, Yuan X. Multivariate time series imputation with generative adversarial networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 1603−1614

[38] Deng A, Hooi B. Graph neural network-based anomaly detection in multivariate time series. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 4027−4035

[39] Cui Y, Zheng K, Cui D, Xie J, Deng L, Huang F, Zhou X . METRO: a generic graph neural network framework for multivariate time series forecasting. Proceedings of the VLDB Endowment, 2021, 15( 2): 224–236

[40] Ma T, Ferber P, Huo S, Chen J, Katz M. Online planner selection with graph neural networks and adaptive scheduling. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 5077−5084

[41] Luo W, Zhang H, Yang X, Bo L, Yang X, Li Z, Qie X, Ye J. Dynamic heterogeneous graph neural network for real-time event prediction. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 3213−3223

[42] Wu L, Lin H, Tan C, Gao Z, Li S Z . Self-supervised learning on graphs: contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 4): 4216–4235

[43] Wang Y, Xu Y, Yang J, Wu M, Li X, Xie L, Chen Z. Graph-aware contrasting for multivariate time-series classification. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 15725−15734

[44] Cheng J, Li M, Li J, Tsung F. Wiener graph deconvolutional network improves graph self-supervised learning. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 7131−7139

[45] Ji J, Wang J, Huang C, Wu J, Xu B, Wu Z, Zhang J, Zheng Y. Spatio-temporal self-supervised learning for traffic flow prediction. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 4356−4364

[46] Xia L, Huang C, Huang C, Lin K, Yu T, Kao B. Automated self-supervised learning for recommendation. In: Proceedings of the ACM Web Conference 2023. 2023, 992−1002

[47] Wei C, Liang J, Liu D, Wang F. Contrastive graph structure learning via information bottleneck for recommendation. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1484

[48] Zheng Y, Yi X, Li M, Li R, Shan Z, Chang E, Li T. Forecasting fine-grained air quality based on big data. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2015, 2267−2276

[49] Yi X, Zheng Y, Zhang J, Li T. ST-MVL: filling missing values in geo-sensory time series data. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence. 2016, 2704−2710

[50] Lai G, Chang W C, Yang Y, Liu H. Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018, 95−104

[51] Goldberger A L, Amaral L A, Glass L, Hausdorff J M, Ivanov P C, Mark R G, Mietus J E, Moody G B, Peng C K, Stanley H E . Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation, 2000, 101( 23): e215–e220

[52] Reyna M A, Josef C S, Jeter R, Shashikumar S P, Westover M B, Nemati S, Clifford G D, Sharma A . Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. Critical Care Medicine, 2020, 48( 2): 210–217

[53] Reiss A, Stricker D. Introducing a new benchmarked dataset for activity monitoring. In: Proceedings of the 16th International Symposium on Wearable Computers. 2012, 108−109

[54] Shokoohi-Yekta M, Hu B, Jin H, Wang J, Keogh E . Generalizing DTW to the multi-dimensional case requires an adaptive approach. Data Mining and Knowledge Discovery, 2017, 31( 1): 1–31

[55] Bagnall A, Dau H A, Lines J, Flynn M, Large J, Bostrom A, Southam P, Keogh E. The UEA multivariate time series classification archive, 2018. 2018, arXiv preprint arXiv: 1811.00075

[56] Hand D J, Till R J . A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 2001, 45( 2): 171–186

[57] Qi Q, Luo Y, Xu Z, Ji S, Yang T. Stochastic optimization of areas under precision-recall curves with provable convergence. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 135

RIGHTS & PERMISSIONS

The Author(s) 2024. This article is published with open access at link.springer.com and journal.hep.com.cn