Log integration on large scale for global networking monitoring

Abstract

Supposing that the overall situation is dug out from the distributed monitoring nodes, there should be two critical obstacles, heterogenous schema and instance, to integrating heterogeneous data from different monitoring sensors. To tackle the challenge of heterogenous schema, an instance-based approach for schema mapping, named instance-based machine-learning (IML) approach was described. And to solve the problem of heterogenous instance, a novel approach, called statistic-based clustering (SBC) approach, which utilized clustering and statistics technologies to match large scale sources holistically, was also proposed. These two algorithms utilized the machine-leaning and clustering technology to improve the accuracy. Experimental analysis shows that the IML approach is more precise than SBC approach, reaching at least precision of 81% and recall rate of 82%. Simulation studies further show that SBC can tackle large scale sources holistically with 85% recall rate when there are 38 data sources.

Keywords

machine-learning / clustering / data integration / schema matching / instance matching

References

[1] US-CERT. Technical cyber security alerts[EB/OL]. [2005-10-04]. https://doi.org/www.us-cert.gov/cas/techalerts/.

[2] LiX., LiuG.-y., QiJ.-xun.. Fuzzy neural and chaotic searching hybrid algorithm and its application in electric customers’s credit risk evaluation[J]. Journal of Central South University of Technology, 2007, 14(1): 140-143

[3] MiaoJ.-jia.. GS-TMS: A global stream-based threat monitor system[C]. Proceedings of the 34th International Conference on Very Large Data Bases, 2008, Auckland, VLDB Endowment: 1678-1687

[4] DoanA., HalevyA. Y.. Semantic-integration research in the database community[J]. AI Magazine, 2005, 26(5): 183-194

[5] RahmE., BernsteinP. A.. A survey of approaches to automatic schema matching[J]. The VLDB Journal, 2001, 11(1): 334-350

[6] LiW., CliftonC.. SEMINT: A tool for identifying attribute correspondences in heterogeneous databases using neural networks[J]. Data and Knowledge Engineering, 2000, 3(4): 49-84

[7] DoanA., DomingosP., HalevyA. Y.. Reconciling schemas of disparate data sources: A machine-learning approach[C]. Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data, 2001, Santa Barbara, ACM Press: 509-520

[8] HaasL. M.. Clio grows up: From research prototype to industrial tool[C]. Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, 2005, Baltimore, ACM Press: 805-810

[9] MillerR. J.. The Clio project: Managing heterogeneity[J]. SIGMOD Record, 2001, 30: 78-83

[10] HeB., ChangK. C.. Making holistic schema matching robust: An ensemble approach[C]. Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, 2005, Chicago, ACM Press: 429-438

[11] HeB., ChangK. C.. Discovering complex matching across web query interfaces: A correlation mining approach[C]. Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2004, Seattle, ACM Press: 148-157

[12] YangQ., ZhangH. H., LiT.. Mining web logs for prediction models in www caching and perfecting[C]. Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2001, Chicago, ACM Press: 473-478

[13] TingK. M., WittenI. H.. Issues in stacked generalization[J]. Journal of Artif Intell Res, 1999, 10(5): 271-289

[14] DaemiA., CalmetJ.. From ontologies to trust through entropy[C]. Proceedings of the International Conference on Advances in Intelligent Systems — Theory and Applications, 2004, Luxembourg, IEEE Computer Society: 12-43

[15] BerkhinP.. A survey of clustering data mining techniques[J]. Grouping Multidimensional Data, 2006, 1(2): 25-71