Adapting to the stream: an instance-attention GNN method for irregular multivariate time series data

Abstract

Multivariate time series (MTS) data are vital for various applications, particularly in machine learning tasks. However, challenges such as sensor failures can result in irregular and misaligned data with missing values, thereby complicating their analysis. While recent advancements use graph neural networks (GNNs) to manage these Irregular Multivariate Time Series (IMTS) data, they generally require a reliable graph structure, either pre-existing or inferred from adequate data to properly capture node correlations. This poses a challenge in applications where IMTS data are often streamed and waiting for future data to estimate a suitable graph structure becomes impractical. To overcome this, we introduce a dynamic GNN model suited for streaming characteristics of IMTS data, incorporating an instance-attention mechanism that dynamically learns and updates graph edge weights for real-time analysis. We also tailor strategies for high-frequency and low-frequency data to enhance prediction accuracy. Empirical results on real-world datasets demonstrate the superiority of our proposed model in both classification and imputation tasks.

Graphical abstract

Keywords

multivariate time series / irregular multivariate time series / graph neural networks

Kun Han received his Master’s in Computer Science from the University of Queensland, Australia in 2021. He is currently in his second year as a PhD student. His research interests lie in machine learning, time series analysis, and weakly supervised learning

Abigail M Y Koay is currently a lecturer at the University of the Sunshine Coast, focusing on Networking and the Internet-of-Things. She earned her PhD from Victoria University of Wellington, New Zealand in 2019, focusing on cybersecurity and applied machine learning. She previously worked as Research Fellow in Cybersecurity at both the University of Queensland, Australia and the University of Waikato, New Zealand

Ryan K L Ko received the bachelor of engineering (Computer Engineering) (Hons.) and PhD degrees from Nanyang Technological University, Singapore in 2005 and 2011, respectively. He is currently a professor with the School of Electrical Engineering and Computer Science, and chair and director of UQ Cyber Security with the University of Queensland, Australia. His research interests include cyber security, critical infrastructure security, cloud user data control, data provenance and privacy preservation

Weitong Chen is a lecturer at the University of Adelaide (UoA), Australia and a researcher at the Australian Institute for Machine Learning (AIML), having previously served as an Associate Lecturer and Post-Doc Research Fellow at the University of Queensland, Australia. He earned his PhD from the University of Queensland in 2020, after completing both his Master’s and Bachelor’s degrees at the University of Queensland and at Griffith University respectively. His research primarily focuses on Machine Learning with a special interest in its applications in medical data

Miao Xu is a senior lecturer in the School of Electrical Engineering and Computer Science at the University of Queensland, Australia. She was awarded the Australian Research Council Discovery Early Career Researcher Award (DECRA) in 2023. Dr Xu specializes in machine learning and data mining, particularly focusing on the challenges of learning from imperfect information. Dr. Xu earned a PhD from Nanjing University, where research efforts led to notable recognitions including the CAAI Outstanding Doctoral Dissertation Award

References

[1] Fawaz H I, Forestier G, Weber J, Idoumghar L, Muller P A . Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 2019, 33( 4): 917–963

[2] Schirmer M, Eltayeb M, Lessmann S, Rudolph M. Modeling irregular time series with continuous recurrent units. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 19388−19405

[3] Sezer O B, Gudelek M U, Ozbayoglu A M . Financial time series forecasting with deep learning: a systematic literature review: 2005–2019. Applied Soft Computing, 2020, 90: 106181

[4] Feng C, Tian P. Time series anomaly detection for cyber-physical systems via neural system identification and Bayesian filtering. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 2858−2867

[5] Mathur A P, Tippenhauer N O. SWaT: a water treatment testbed for research and training on ICS security. In: Proceedings of 2016 International Workshop on Cyber-physical Systems for Smart Water Networks. 2016, 31−36

[6] Ahmed C M, Palleti V R, Mathur A P. WADI: a water distribution testbed for research in the design of secure cyber physical systems. In: Proceedings of the 3rd International Workshop on Cyber-Physical Systems for Smart Water Networks. 2017, 25−28

[7] Cao D, Wang Y, Duan J, Zhang C, Zhu X, Huang C, Tong Y, Xu B, Bai J, Tong J, Zhang Q. Spectral temporal graph neural network for multivariate time-series forecasting. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1491

[8] Yehuda Y, Freedman D, Radinsky K. Self-supervised classification of clinical multivariate time series using time series dynamics. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 5416−5427

[9] Li G, Choi B, Xu J, Bhowmick S S, Chun K P, Wong G L H. ShapeNet: a shapelet-neural network approach for multivariate time series classification. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 8375−8383

[10] Tan Q, Ye M, Yang B, Liu S, Ma A J, Yip T C F, Wong G L H, Yuen P C. DATA-GRU: dual-attention time-aware gated recurrent unit for irregular multivariate time series. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 930−937

[11] Kidger P, Morrill J, Foster J, Lyons T. Neural controlled differential equations for irregular time series. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 562

[12] Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W. Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 11106−11115

[13] Nie Y, Nguyen N H, Sinthong P, Kalagnanam J. A time series is worth 64 words: long-term forecasting with transformers. In: Proceedings of the 11th International Conference on Learning Representations. 2023

[14] Bahadori M T, Liu Y. Granger causality analysis in irregular time series. In: Proceedings of the 12th SIAM International Conference on Data Mining. 2012, 660−671

[15] Schulz M, Stattegger K . Spectrum: spectral analysis of unevenly spaced paleoclimatic time series. Computers & Geosciences, 1997, 23( 9): 929–945

[16] Rehfeld K, Marwan N, Heitzig J, Kurths J . Comparison of correlation analysis techniques for irregularly sampled time series. Nonlinear Processes in Geophysics, 2011, 18( 3): 389–404

[17] Babu P, Stoica P . Spectral analysis of nonuniformly sampled data - a review. Digital Signal Processing, 2010, 20( 2): 359–378

[18] Stoica P, Li J, He H . Spectral analysis of nonuniformly sampled data: a new approach versus the periodogram. IEEE Transactions on Signal Processing, 2009, 57( 3): 843–858

[19] Li S C X, Marlin B. Classification of sparse and irregularly sampled time series with mixtures of expected Gaussian kernels and random features. In: Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence. 2015, 484−493

[20] Li S C X, Marlin B. A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 1804−1812

[21] Che Z, Purushotham S, Cho K, Sontag D, Liu Y . Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 2018, 8( 1): 6085

[22] Rubanova Y, Chen R T Q, Duvenaud D. Latent ODEs for irregularly-sampled time series. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 478

[23] Wang Z, Zhang Y, Jiang A, Zhang J, Li Z, Gao J, Li K, Lu C, Ren Z. Improving irregularly sampled time series learning with time-aware dual-attention memory-augmented networks. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021, 3523−3527

[24] Shukla S N, Marlin B M. Multi-time attention networks for irregularly sampled time series. In: Proceedings of the 9th International Conference on Learning Representations. 2021

[25] Tipirneni S, Reddy C K . Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD), 2022, 16( 6): 105

[26] Zhang J, Zheng S, Cao W, Bian J, Li J. Warpformer: a multi-scale modeling approach for irregular clinical time series. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 3273−3285

[27] Chen Y, Ren K, Wang Y, Fang Y, Sun W, Li D. ContiFormer: continuous-time transformer for irregular time series modeling. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 2042

[28] Sun C, Hong S, Song M, Chou Y H, Sun Y, Cai D, Li H. TE-ESN: time encoding echo state network for prediction based on irregularly sampled time series data. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence. 2021, 3010−3016

[29] Cini A, Marisca I, Alippi C. Filling the G_ap_s: multivariate time series imputation by graph neural networks. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[30] Wei Y, Peng J, He T, Xu C, Zhang J, Pan S, Chen S. Compatible transformer for irregularly sampled multivariate time series. In: Proceedings of 2023 IEEE International Conference on Data Mining. 2023, 1409−1414

[31] Zhang X, Zeman M, Tsiligkaridis T, Zitnik M. Graph-guided network for irregularly sampled multivariate time series. In: Proceedings of the 10th International Conference on Learning Representations. 2022

[32] Wang Z, Jiang T, Xu Z, Zhang J, Gao J . Irregularly sampled multivariate time series classification: a graph learning approach. IEEE Intelligent Systems, 2023, 38( 3): 3–11

[33] Wang Z, Jiang T, Xu Z, Gao J, Wu O, Yan K, Zhang J. Uncovering multivariate structural dependency for analyzing irregularly sampled time series. In: Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases: Research Track. 2023, 238−254

[34] Liu Y, Liu Q, Zhang J W, Feng H, Wang Z, Zhou Z, Chen W. Multivariate time-series forecasting with temporal polynomial graph neural networks. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2024, 1411

[35] Rodrigues de Barros M, Rissi T L, Cabrera E F, Tannuri E A, Gomi E S, Barreira R A, Costa A H R. Embracing data irregularities in multivariate time series with recurrent and graph neural networks. In: Proceedings of the 12th Brazilian Conference on Intelligent Systems. 2023, 3−17

[36] Cao W, Wang D, Li J, Zhou H, Li L, Li Y. BRITS: bidirectional recurrent imputation for time series. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 6776−6786

[37] Luo Y, Cai X, Zhang Y, Xu J, Yuan X. Multivariate time series imputation with generative adversarial networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 1603−1614

[38] Deng A, Hooi B. Graph neural network-based anomaly detection in multivariate time series. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 4027−4035

[39] Cui Y, Zheng K, Cui D, Xie J, Deng L, Huang F, Zhou X . METRO: a generic graph neural network framework for multivariate time series forecasting. Proceedings of the VLDB Endowment, 2021, 15( 2): 224–236

[40] Ma T, Ferber P, Huo S, Chen J, Katz M. Online planner selection with graph neural networks and adaptive scheduling. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 5077−5084

[41] Luo W, Zhang H, Yang X, Bo L, Yang X, Li Z, Qie X, Ye J. Dynamic heterogeneous graph neural network for real-time event prediction. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 3213−3223

[42] Wu L, Lin H, Tan C, Gao Z, Li S Z . Self-supervised learning on graphs: contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 4): 4216–4235

[43] Wang Y, Xu Y, Yang J, Wu M, Li X, Xie L, Chen Z. Graph-aware contrasting for multivariate time-series classification. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 15725−15734

[44] Cheng J, Li M, Li J, Tsung F. Wiener graph deconvolutional network improves graph self-supervised learning. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 7131−7139

[45] Ji J, Wang J, Huang C, Wu J, Xu B, Wu Z, Zhang J, Zheng Y. Spatio-temporal self-supervised learning for traffic flow prediction. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 4356−4364

[46] Xia L, Huang C, Huang C, Lin K, Yu T, Kao B. Automated self-supervised learning for recommendation. In: Proceedings of the ACM Web Conference 2023. 2023, 992−1002

[47] Wei C, Liang J, Liu D, Wang F. Contrastive graph structure learning via information bottleneck for recommendation. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1484

[48] Zheng Y, Yi X, Li M, Li R, Shan Z, Chang E, Li T. Forecasting fine-grained air quality based on big data. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2015, 2267−2276

[49] Yi X, Zheng Y, Zhang J, Li T. ST-MVL: filling missing values in geo-sensory time series data. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence. 2016, 2704−2710

[50] Lai G, Chang W C, Yang Y, Liu H. Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018, 95−104

[51] Goldberger A L, Amaral L A, Glass L, Hausdorff J M, Ivanov P C, Mark R G, Mietus J E, Moody G B, Peng C K, Stanley H E . Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation, 2000, 101( 23): e215–e220

[52] Reyna M A, Josef C S, Jeter R, Shashikumar S P, Westover M B, Nemati S, Clifford G D, Sharma A . Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. Critical Care Medicine, 2020, 48( 2): 210–217

[53] Reiss A, Stricker D. Introducing a new benchmarked dataset for activity monitoring. In: Proceedings of the 16th International Symposium on Wearable Computers. 2012, 108−109

[54] Shokoohi-Yekta M, Hu B, Jin H, Wang J, Keogh E . Generalizing DTW to the multi-dimensional case requires an adaptive approach. Data Mining and Knowledge Discovery, 2017, 31( 1): 1–31

[55] Bagnall A, Dau H A, Lines J, Flynn M, Large J, Bostrom A, Southam P, Keogh E. The UEA multivariate time series classification archive, 2018. 2018, arXiv preprint arXiv: 1811.00075

[56] Hand D J, Till R J . A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 2001, 45( 2): 171–186

[57] Qi Q, Luo Y, Xu Z, Ji S, Yang T. Stochastic optimization of areas under precision-recall curves with provable convergence. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 135

Acknowledgements

We extend our sincere gratitude to UQ Cyber for their strategic funding, which has provided a PhD scholarship and access to essential equipment. The research work was also supported by the UoA Start-up Grant, UQ Cyber Security Seed Funding, the Australian Research Council Linkage Project (LP230200821), the Australian Research Council Early Career Industry Fellowship (IE240100275), the Australian Research Council Discovery Project (DP240103070), and the Australian Research Council Discovery Early Career Researcher Award (DE230101116).

Competing interests

The authors declare that they have no competing interests or financial conflicts to disclose.

Funding note

Open Access funding enabled and organized by CAUL and its Member Institutions.

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit creativecommons.org/licenses/by/4.0/.

RIGHTS & PERMISSIONS

2024 The Author(s) 2024. This article is published with open access at link.springer.com and journal.hep.com.cn