GraphSTGAN: Situation understanding network of slow-fast high maneuvering targets for maritime monitor services of IoT data

Abstract

With the rapid growth of the maritime Internet of Things (IoT) devices for Maritime Monitor Services (MMS), maritime traffic controllers could not handle a massive amount of data in time. For unmanned MMS, one of the key technologies is situation understanding. However, the presence of slow-fast high maneuvering targets and track breakages due to radar blind zones make modeling the dynamics of marine multi-agents difficult, and pose significant challenges to maritime situation understanding. In order to comprehend the situation accurately and thus offer unmanned MMS, it is crucial to model the complex dynamics of multi-agents using IoT big data. Nevertheless, previous methods typically rely on complex assumptions, are plagued by unstructured data, and disregard the interactions between multiple agents and the spatial-temporal correlations. A deep learning model, Graph Spatial-Temporal Generative Adversarial Network(GraphSTGAN), is proposed in this paper, which uses graph neural network to model unstructured data and uses STGAN to learn the spatial-temporal dependencies and interactions. Extensive experiments show the effectiveness and robustness of the proposed method.

Keywords

Internet of things / Multi-agents / Graph neural network / Maritime monitoring services

References

[1] IMO, Imo profile UK. https://business.un.org/en/entities/13, 2022. (Accessed 5 December 2022).

[2] S. Nižetić, P. Šolić, D.L.-d.-I. González-de, L. Patrono, et al., Internet of things (iot): opportunities, issues and challenges towards a smart and sustainable future, J. Clean. Prod. 274 (2020) 122877, https://doi.org/10.1016/j.jclepro.2020.122877.

[3] L. Chettri, R. Bera, A comprehensive survey on internet of things (iot) toward 5g wireless systems, IEEE Internet Things J. 7 (1) (2019) 16-32, https://doi.org/ 10.1109/JIOT.2019.2948888.

[4] E. Adi, A. Anwar, Z. Baig, S. Zeadally, Machine learning and data analytics for the iot, Neural Comput. Appl. 32 (20) (2020) 16205-16233, https://doi.org/10.1007/s00521-020-04874-y.

[5] L. B. Furstenau, Y. P. R. Rodrigues, M. K. Sott, P. Leivas, M. S. Dohan, J. R. López-Robles, M. J. Cobo, N. L. Bragazzi, K.-K. R. Choo, Internet of things: Conceptual network structure, main challenges and future directions, Digit commun netw.To be published, https://doi.org/10.1016/j.dcan.2022.04.027.

[6] L. Gao, Y. Zhang, Application of cloud computing in marine meteorological automatic detection system, Sh. Sci. and Technol. 38 (18) (2016) 166-168.

[7] L. Zhang, Q. Meng, Z. Xiao, X. Fu, A novel ship trajectory reconstruction approach using ais data, Ocean Eng. 159 (2018) 165-174, https://doi.org/10.1016/ J. OCEANENG.2018.03.085.

[8] S. Wang, S. Gao, W. Yang, Ship route extraction and clustering analysis based on automatic identification system data, in: 2017 Eighth International Conference on Intelligent Control and Information Processing (ICICIP), IEEE, 2017, pp. 33-38, https://doi.org/10.1109/ICICIP.2017.8113913.

[9] Y. He, D. Zhang, J. Zhang, M. Zhang, T. Li, Ship route planning using historical trajectories derived from ais data, TransNav: Int. J. Mar. Nav. Saf. Sea Trans. 13 (1), https://doi.org/10.12716/1001.13.01.06.

[10] C. Capezza, S. Coleman, A. Lepore, B. Palumbo, L. Vitiello, Ship fuel consumption monitoring and fault detection via partial least squares and control charts of navigation data, Transport. Res. Transport Environ. 67 (2019) 375-387, https://doi.org/10.1016/J.TRD.2018.11.009.

[11] J. Duan, Y. Liu, B. Lin, Y. Jiang, F. Hou, W. Li, Improved ant colony optimization algorithm for optimized nodes deployment of hap-based marine monitoring sensor networks, in:International Conference in Communications, Signal Processing, and Systems, Springer, 2018, pp. 933-941, https://doi.org/10.1007/978-981-13-6508-9_113.

[12] S. Rizal, H.-H. Choi, S.-H. Kim, D.-S. Kim, S.-H. Kim, Marine engine fault detection system using networked proximity sensors, in: 2017 IEEE International Conference on Mechatronics (ICM), IEEE, 2017, pp. 284-289, https://doi.org/10.1109/ICMECH.2017.7921118.

[13] A.L. Ellefsen, E. Bjørlykhaug, V. Æsøy, H. Zhang, An unsupervised reconstruction-based fault detection algorithm for maritime components, Access 7 (2019) 16101-16109, https://doi.org/10.1109/ACCESS.2019.2895394.

[14] C.-M. Yeoh, B.-L. Chai, H. Lim, T.-H. Kwon, K.-O. Yi, T.-H. Kim, C.-S. Lee, G.-G.- H. Kwark, Ubiquitous containerized cargo monitoring system development based on wireless sensor network technology, Int. J. Comput. Commun. Control 6 (4)(2011) 779-793, https://doi.org/10.15837/IJCCC.2011.4.2109.

[15] W. Lang, R. Jedermann, D. Mrugala, A. Jabbari, B. Krieg-Brückner, K. Schill, The “intelligent container”—a cognitive sensor network for transport management, IEEE Sensor. J. 11 (3) (2010) 688-698, https://doi.org/10.1109/ JSEN.2010.2060480.

[16] L. Ruiz-Garcia, P. Barreiro, J.I. Robla, L. Lunadei, Testing zigbee motes for monitoring refrigerated vegetable transportation under real conditions, IEEE Sensor. J. 10 (5) (2010) 4968-4982, https://doi.org/10.3390/s100504968.

[17] A. Kamolov, S.H. Park, An iot based smart berthing (parking) system for vessels and ports,in: International Conference on Mobile and Wireless Technology, Springer, 2018, pp. 129-139, https://doi.org/10.1007/978-981-13-1059-1_13.

[18] A.L. Ellefsen, S. Ushakov, V. Æsøy, H. Zhang, Validation of data-driven labeling approaches using a novel deep network structure for remaining useful life predictions, Access 7 (2019) 71563-71575, https://doi.org/10.1109/ access.2019.2920297.

[19] L. Lin, Y. Bar-Shalom, T. Kirubarajan, New assignment-based data association for tracking move-stop-move targets, IEEE Trans. Aero. Electron. Syst. 40 (2) (2004) 714-725, https://doi.org/10.1109/ICIF.2002.1020913.

[20] S. Liu, H. Li, Y. Zhang, B. Zou, Multiple hypothesis method for tracking move-stop-move target, J. Eng. 2019 (19) (2019) 6155-6159, https://doi.org/10.1049/JOE.2019.0182.

[21] Y. Xu, L. Wang, Y. Wang, Y. Fu,Adaptive trajectory prediction via transferable gnn, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 6520-6531, https://doi.org/10.1109/CVPR52688.2022.00641.

[22] L. Li, M. Pagnucco, Y. Song,Graph-based spatial transformer with memory replay for multi-future pedestrian trajectory prediction, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2231-2241, https://doi.org/10.1109/CVPR52688.2022.00227.

[23] R. Zhou, H. Zhou, H. Gao, M. Tomizuka, J. Li, Z. Xu, Grouptron: dynamic multi-scale graph convolutional networks for group-aware dense crowd trajectory forecasting, in: 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022, pp. 805-811, https://doi.org/10.1109/icra46639.2022.9811585.

[24] D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, J. Chanussot, Graph convolutional networks for hyperspectral image classification, IEEE Trans. Geosci. Rem. Sens. 59 (7) (2020) 5966-5978, https://doi.org/10.1109/TGRS.2020.3015157.

[25] Y. Ding, Z. Zhang, X. Zhao, D. Hong, W. Cai, C. Yu, N. Yang, W. Cai, Multi-feature fusion: graph neural network and cnn combining for hyperspectral image classification, Neurocomputing 501 (2022) 246-257, https://doi.org/10.1016/ j. neucom.2022.06.031.

[26] C. Yang, M. Hu, G. Zhai, X.-P. Zhang, Graph-based denoising for respiration and heart rate estimation during sleep in thermal video, IEEE Internet Things J. 9 (17)(2022) 15697-15713, https://doi.org/10.1109/JIOT.2022.3150147.

[27] X. Cao, X. Fu, C. Xu, D. Meng, Deep spatial-spectral global reasoning network for hyperspectral image denoising, IEEE Trans. Geosci. Rem. Sens. 60 (2021) 1-14, https://doi.org/10.1109/TGRS.2021.3069241.

[28] T. N. Kipf, M. Welling, Semi-supervised Classification with Graph Convolutional Networks, arXiv https://doi.org/10.48550/arXiv.1609.02907.

[29] I.J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, Y. Bengio,Generative adversarial nets, in:Conference on Neural Information Processing Systems, vol. 27, 2014.

[30] J. Amirian, J.-B. Hayet, J. Pettré, Social ways: learning multi-modal distributions of pedestrian trajectories with gans,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, https://doi.org/ 10.1109/CVPRW.2019.00359,0-0.

[31] B. Ristic, B. La Scala, M. Morelande, N. Gordon, Statistical analysis of motion patterns in ais data: anomaly detection and motion prediction,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, IEEE, 2008, pp. 1-7.

[32] M. H. Tun, G. S. Chambers, T. Tan, T. Ly,Maritime port intelligence using ais data, Rec. Adv in Secur. Technol. 33.

[33] X. Zhang, G. Liu, C. Hu, X. Ma, Wavelet analysis based hidden markov model for large ship trajectory prediction, in: 2019 Chinese Control Conference (CCC), IEEE, 2019, pp. 2913-2918, https://doi.org/10.23919/ChiCC.2019.8866006.

[34] F. Mazzarella, V.F. Arguedas, M. Vespe, Knowledge-based vessel position prediction using historical ais data, in: 2015 Sensor Data Fusion: Trends, Solutions, Applications (SDF), IEEE, 2015, pp. 1-6, https://doi.org/10.1109/ SDF.2015.7347707.

[35] A.L. Duca, C. Bacciu, A. Marchetti, A k-nearest neighbor classifier for ship route prediction, in: OCEANS 2017-Aberdeen, IEEE, 2017, pp. 1-6, https://doi.org/ 10.1109/OCEANSE.2017.8084635.

[36] P. Virjonen, P. Nevalainen, T. Pahikkala, J. Heikkonen, Ship movement prediction using k-nn method, in: 2018 Baltic Geodetic Congress (BGC Geomatics), IEEE, 2018, pp. 304-309, https://doi.org/10.1109/BGC-GEOMATICS.2018.00064.

[37] S. Gan, S. Liang, K. Li, J. Deng, T. Cheng, Ship trajectory prediction for intelligent traffic management using clustering and ann, in: 2016 UKACC 11th International Conference on Control (CONTROL), IEEE, 2016, pp. 1-6, https://doi.org/10.1109/CONTROL.2016.7737569.

[38] S. Hexeberg, Ais-based Vessel Trajectory Prediction for Asv Collision Avoidance, NTNU, 2017. Master’s thesis.

[39] B.R. Dalsnes, S. Hexeberg, A.L. Flåten, B.-O.H. Eriksen, E.F. Brekke, The neighbor course distribution method with Gaussian mixture models for ais-based vessel trajectory prediction, in: 2018 21st International Conference on Information Fusion (FUSION), IEEE, 2018, pp. 580-587, https://doi.org/10.23919/ ICIF.2018.8455607.

[40] D. Alizadeh, A.A. Alesheikh, M. Sharif, Prediction of vessels locations and maritime traffic using similarity measurement of trajectory, Spatial Sci. 27 (2) (2021) 151-162, https://doi.org/10.1080/19475683.2020.1840434.

[41] X. Liu, W. He, J. Xie, X. Chu, Predicting the trajectories of vessels using machine learning, in: 2020 5th International Conference on Control, Robotics and Cybernetics, CRC), IEEE, 2020, pp. 66-70, https://doi.org/10.1109/CRC51253.2020.9253496.

[42] J. Liu, G. Shi, K. Zhu, Vessel trajectory prediction model based on ais sensor data and adaptive chaos differential evolution support vector regression (acde-svr), Appl. Sci. 9 (15) (2019) 2983, https://doi.org/10.3390/APP9152983.

[43] D.-D. Nguyen, C. Le Van, M.I. Ali,Vessel trajectory prediction using sequence-to-sequence models over spatial grid, in:Proceedings of the 12th ACM International Conference on Distributed and Event-Based Systems, 2018, pp. 258-261, https://doi.org/10.1145/3210284.3219775.

[44] M. Gao, G. Shi, S. Li, Online prediction of ship behavior with automatic identification system sensor data using bidirectional long short-term memory recurrent neural network, Sensors 18 (12) (2018) 4211, https://doi.org/10.3390/s18124211.

[45] J. Sekhon, C. Fleming, A spatially and temporally attentive joint trajectory prediction framework for modeling vessel intent, in: Learning for Dynamics and Control, PMLR, 2020, pp. 318-327, https://doi.org/10.48550/arXiv.1912.09429.

[46] D. Nguyen, R. Fablet, Traisformer-a generative transformer for ais trajectory prediction, arXiv, https://doi.org/10.48550/arXiv.2109.03958.

[47] R. E. Kalman, A new approach to linear filtering and prediction problems, https://doi.org/10.1115/1.3662552.

[48] R.S. Bucy, K.D. Senne, Digital synthesis of non-linear filters, Automatica 7 (3) (1971) 287-298, https://doi.org/10.1016/0005-1098(71)90121-X.

[49] S.J. Julier, J.K. Uhlmann, Unscented filtering and nonlinear estimation, Proc. IEEE 92 (3) (2004) 401-422, https://doi.org/10.1109/JPROC.2003.823141.

[50] I. Arasaratnam, S. Haykin, Cubature kalman filters, IEEE Trans. Automat. Control 54 (6) (2009) 1254-1269.

[51] N.J. Gordon, D.J. Salmond, A.F. Smith,Novel approach to nonlinear/non-Gaussian bayesian state estimation, in:IEE Proceedings F (Radar and Signal Processing), vol. 140, IET, 1993, pp. 107-113, https://doi.org/10.1049/ip-f-2.1993.0015.

[52] D. Magill, Optimal adaptive estimation of sampled stochastic processes, 2019 Chin. Control Conf. (CCC) 10 (4) (1965) 434-439, https://doi.org/10.1109/ TAC.1965.1098191.

[53] H.A. Blom, Y. Bar-Shalom, The interacting multiple model algorithm for systems with markovian switching coefficients, IEEE Trans. Automat. Control 33 (8) (1988) 780-783, https://doi.org/10.1109/9.1299.

[54] X.-R. Li, Y. Bar-Shalom, Multiple-model estimation with variable structure, IEEE Trans. Automat. Control 41 (4) (1996) 478-493, https://doi.org/10.1109/ 9.489270.

[55] J. Liu, Z. Wang, M. Xu, Deepmtt: a deep learning maneuvering target-tracking algorithm based on bidirectional lstm network, Inf. Fusion 53 (2020) 289-304, https://doi.org/10.1016/J.INFFUS.2019.06.012.

[56] M.K. Al-Sharman, Y. Zweiri, M.A.K. Jaradat, R. Al-Husari, D. Gan, L.D. Seneviratne, Deep-learning-based neural network training for state estimation enhancement: application to attitude estimation, IEEE Trans. Instrum. Meas. 69 (1) (2019) 24-34, https://doi.org/10.1109/TIM.2019.2895495.

[57] X. Zhang, F. He, T. Zheng, An lstm-based trajectory estimation algorithm for non-cooperative maneuvering flight vehicles, in: 2019 Chinese Control Conference (CCC), IEEE, 2019, pp. 8821-8826, https://doi.org/10.23919/ ChiCC.2019.8866249.

[58] C. Lin, H. Wang, M. Fu, J. Yuan, J. Gu, A gated recurrent unit-based particle filter for unmanned underwater vehicle state estimation, IEEE Trans. Instrum. Meas. 70 (2020) 1-12, https://doi.org/10.1109/TIM.2020.3011789.

[59] C. Lin, Y. Cheng, X. Wang, A convolutional neural network particle filter for uuv target state estimation, IEEE Trans. Instrum. Meas. 71 (2022) 1-12, https://doi.org/ 10.1109/tim.2022.3169539.

[60] A. Milan, S.H. Rezatofighi, A. Dick, I. Reid, K. Schindler,Online multi-target tracking using recurrent neural networks, in:Thirty-First AAAI Conference on Artificial Intelligence, 2017, https://doi.org/10.1609/aaai.v31i1.11194.

[61] B. Lim, S. Zohren, S. Roberts, Recurrent neural filters: learning independent bayesian filtering steps for time series prediction, in: 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, 2020, pp. 1-8, https://doi.org/ 10.1109/IJCNN48605.2020.9206906.

[62] R. Mucci, J. Arnold, Y. Bar-Shalom, Track segment association with a distributed field of sensors, J. Acoust. Soc. Am. 78 (4) (1985) 1317-1324, https://doi.org/ 10.1121/1.392901.

[63] S.-W. Yeom, T. Kirubarajan, Y. Bar-Shalom, Track segment association, fine-step imm and initialization with Doppler for improved track performance, IEEE Trans. Aero. Electron. Syst. 40 (1) (2004) 293-309, https://doi.org/10.1109/ TAES.2004.1292161.

[64] S. Zhang, Y. Bar-Shalom, Track segment association for gmti tracks of evasive move-stop-move maneuvering targets, IEEE Trans. Aero. Electron. Syst. 47 (3) (2011) 1899-1914, https://doi.org/10.1109/TAES.2011.5937272.

[65] J. Raghu, P. Srihari, R. Tharmarasa, T. Kirubarajan, Comprehensive track segment association for improved track continuity, IEEE Trans. Aero. Electron. Syst. 54 (5)(2018) 2463-2480, https://doi.org/10.1109/TAES.2018.2820364.

[66] W. Xiong, P. Xu, Y. Cui, Z. Xiong, Y. Lv, X. Gu, Track segment association with dual contrast neural network, IEEE Trans. Aero. Electron. Syst. 58 (1) (2021) 247-261, https://doi.org/10.1109/taes.2021.3098175.

[67] W. Li, C. Zhang, J. Ma, C. Jia, Long-term vessel motion predication by modeling trajectory patterns with ais data, in: 2019 5th International Conference on Transportation Information and Safety (ICTIS), IEEE, 2019, pp. 1389-1394, https://doi.org/10.1109/ICTIS.2019.8883596.

[68] J. Hu, K. Kaur, H. Lin, X. Wang, M. M. Hassan, I. Razzak, M. Hammoudeh, Intelligent anomaly detection of trajectories for iot empowered maritime transportation systems, IEEE Trans. Intell. Transport. Syst.https://doi.org/10.110 9/tits.2022.3162491.

[69] D. Nguyen, R. Vadaine, G. Hajduch, R. Garello, R. Fablet, Geotracknet-a maritime anomaly detector using probabilistic neural network representation of ais tracks and a contrario detection, IEEE Trans. Intell. Transport. Syst.https://doi.org/10.110 9/TITS.2021.3055614.

[70] C. Yin, S. Zhang, J. Wang, N. N. Xiong, Anomaly detection based on convolutional recurrent autoencoder for iot time series, IEEE Trans. Intell. Transport. Syst. 52 (1)(2020) 112-122, https://doi.org/10.1109/tsmc.2020.2968516.

[71] S. Zhang, F. Tang, X. Jin, Y. Wu, X. LinHao, D. Yang, Trawler state and net times extraction based on data from beidou vessel monitoring system, F. Inf. St. 30 (3)(2015) 205-211.

[72] A. Mohamed, K. Qian, M. Elhoseiny, C. Claudel, Social-stgcnn: a social spatio-temporal graph convolutional neural network for human trajectory prediction,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14424-14432, https://doi.org/10.1109/cvpr42600.2020.01443.

[73] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, A. Alahi, Social gan: socially acceptable trajectories with generative adversarial networks,in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 2255-2264, https://doi.org/10.1109/CVPR.2018.00240.

[74] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, T. Aila,Analyzing and improving the image quality of stylegan, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8110-8119, https://doi.org/10.1109/cvpr42600.2020.00813.

[75] S. Ravuri, K. Lenc, M. Willson, D. Kangin, R. Lam, P. Mirowski, M. Fitzsimons, M. Athanassiadou, S. Kashem, S. Madge, et al., Skilful precipitation nowcasting using deep generative models of radar, Nature 597 (7878) (2021) 672-677, https://doi.org/10.1038/s41586-021-03854-z.