Fine-grained master data curation for time series

Abstract

Master data is commonly utilized in business scenarios, serving as the bedrock for describing core entities. Studies have been presented to manage master data in curating relational data. However, the research of master data remains untouched in the context of time series. This raises concerns regarding time series data management and application, as we have found that master data is also prevalent in time series scenarios. For example, to reduce the fuel consumption of vehicles, it relies on the complete map of engine speed and torque to fuel consumption rate, known as the engine universal characteristic map, i.e., master data. Additionally, to conduct anomaly detection and weather prediction in weather scenario, we need to understand the fine-grained relationship between temperature, wind speed, and atmospheric pressure, which also represents master data. Unfortunately, such master data is often incomplete and inconsistent. In this paper, we propose to predict the fine-grained master data by learning a model from the incomplete and inconsistent observations. A novel MasterNet is designed based on Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs). Deconvolution is employed to predict incomplete data, while the discriminator can successfully tolerate inconsistent observations. We also consider two attention features to capture more time-related information. Experiments over real-world time series datasets show that our MasterNet outperforms the existing approaches in both imputing the incomplete and repairing the inconsistent master data.

Graphical abstract

Keywords

incomplete data / inconsistent data / master data / time series

References

[1] Loshin D. Master Data Management. Boston: Morgan Kaufmann, 2009

[2] Bertino E, Atzeni P, Lee Tan K, Chen Y, Tay Y C, Fan W, Li J, Ma S, Tang N, Yu W. Towards certain fixes with editing rules and master data. Proceedings of the VLDB Endowment, 2010, 3(1−2): 173−184

[3] Mei Y, Song S, Fang C, Wei Z, Fang J, Long J. Discovering editing rules by deep reinforcement learning. In: Proceedings of the 39th IEEE International Conference on Data Engineering. 2023, 355−367

[4] Song S, Chen L . Differential dependencies: reasoning and discovery. ACM Transactions on Database Systems (TODS), 2011, 36( 3): 16

[5] Altman N S . An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 1992, 46( 3): 175–185

[6] Learning internal representations by error propagation. https://ieeexplore.ieee.org/document/6302929 David E. Rumelhart; James L. McClelland, “Learning Internal Representations by Error Propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, MIT Press, 1987, pp.318−362

[7] Song S, Li C, Zhang X. Turn waste into wealth: on simultaneous clustering and cleaning over dirty data. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2015, 1115−1124

[8] Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. In: Proceedings of the 28th International Conference on Neural Information Processing Systems. 2014, 2672−2680

[9] Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. In: Proceedings of the 4th International Conference on Learning Representations. 2016

[10] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3431−3440

[11] Ye M, Yi X, Jiao S . Energy optimization by parameter matching for a truck-mounted concrete pump. Energy Procedia, 2016, 88: 574–580

[12] Lohse D, Schnabel W. Grundlagen der Straßenverkehrstechnik und der Verkehrsplanung-Band 2 Verkehrsplanung. Berlin: Beuth Verlag, 2011

[13] Ivarsson M, Åslund J, Nielsen L . Look-ahead control— consequences of a non-linear fuel map on truck fuel consumption. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 2009, 223( 10): 1223–1238

[14] Wu L, Fu Z G, Wu W D. Research of universal characteristic curve using MATLAB. In: Proceedings of 2010 International Conference on Mechanic Automation and Control Engineering. 2010, 3555−3557

[15] Sato T, Kayano M, Matsuura S. Fuel consumption estimating unit of vehicle: 20090048770, 2008-10-03, See freepatentsonline.com/y2006/0089781.html website, 2025

[16] He S H, Yang Y Z. Modeling of universal characteristics and optimization of operating conditions of concrete pump truck based on neural network. Journal of Central South University (Science and Technology), 2010, 41: 1398−1404

[17] Liu D C, Nocedal J. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 1989, 45(1−3): 503−528

[18] Pathak D, Krähenbühl P, Donahue J, Darrell T, Efros A A. Context encoders: feature learning by inpainting. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 2536−2544

[19] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2012, 1097−1105

[20] Takács G, Pilászy I, Németh B, Tikk D. Matrix factorization and neighbor based algorithms for the netflix prize problem. In: Proceedings of 2008 ACM Conference on Recommender Systems. 2008, 267−274

[21] Wang K, Peng H, Jin Y, Sha C, Wang X . Local weighted matrix factorization for top-n recommendation with implicit feedback. Data Science and Engineering, 2016, 1( 4): 252–264

[22] Luo X, Zhou M, Leung H, Xia Y, Zhu Q, You Z, Li S . An incremental-and-static-combined scheme for matrix-factorization-based collaborative filtering. IEEE Transactions on Automation Science and Engineering, 2016, 13( 1): 333–343

[23] Scarselli F, Gori M, Tsoi A C, Hagenbuchner M, Monfardini G . The graph neural network model. IEEE Transactions on Neural Networks, 2009, 20( 1): 61–80

[24] Wu Z, Pan S, Chen F, Long G, Zhang C, Yu P S . A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32( 1): 4–24

[25] Spinelli I, Scardapane S, Uncini A . Missing data imputation with adversarially-trained graph convolutional networks. Neural Networks, 2020, 129: 249–260

[26] Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R . Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 2014, 15( 1): 1929–1958

[27] Pukelsheim F . The three sigma rule. The American Statistician, 1994, 48( 2): 88–91

[28] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations. 2015

[29] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770−778

[30] Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T. DeCAF: A deep convolutional activation feature for generic visual recognition. In: Proceedings of the 31st International Conference on Machine Learning. 2014, I-647−I-655

[31] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 818−833

[32] Sharaf S A . Beam pump dynamometer card prediction using artificial neural networks. KnE Engineering, 2018, 3( 7): 198–212

[33] Esman G. Splunk and tensorflow for security: catching the fraudster with behavior biometrics. See splunk.com/en_us/blog/security/deep-learning-with-splunk-and-tensorflow-for-security-catching-the-fraudster-in-neural-networks-with-behavioral-biometrics.html website, 2012

RIGHTS & PERMISSIONS

Higher Education Press