Approach based on wavelet analysis for detecting and amending anomalies in dataset

Abstract

It is difficult to detect the anomalies whose matching relationship among some data attributes is very different from others’ in a dataset. Aiming at this problem, an approach based on wavelet analysis for detecting and amending anomalous samples was proposed. Taking full advantage of wavelet analysis’ properties of multi-resolution and local analysis, this approach is able to detect and amend anomalous samples effectively. To realize the rapid numeric computation of wavelet translation for a discrete sequence, a modified algorithm based on Newton-Cores formula was also proposed. The experimental result shows that the approach is feasible with good result and good practicality.

Keywords

data preprocessing / wavelet analysis / anomaly detecting / data mining

References

[1] EskinELangleyP. Anomaly detection over noisy data using learned probability distributions [C]. Proceedings of the 17th International Conference on Machine Learning, 2000, San Francisco, Morgan Kaufmann Publishers Inc: 255-262

[2] YamanishiK, TakeuchiJ I, WilliamsG. On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms[C]. Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2000, Boston, ACM Press: 320-324

[3] KnorrE M, NgR TGuptaA, ShmueliO, WidomJ. Algorithms for mining distance-based outliers in large datasets [C]. Proceedings of the 24th International Conference on Very Large Data Bases, 1998, Changsha, Morgan Kaufmann: 392-403

[4] KnorrE M, NgR TAtkinsonM P, OrlowskaM E, ValduriezP. Finding intensional knowledge of distance-based outliers[C]. Proceedings of the 25th International Conference on Very Large Data Bases, 1999, Edinburgh, Morgan Kaufmann: 211-222

[5] RamaswamyS, RastogiR, KyuseokSChenW, NaughtonJ F, BernsteinP A. Efficient algorithms for mining outliers from large data sets[C]. Proceedings of the ACM SIGMOD International Conference on Management of Data, 2000, Dallas, ACM Press: 427-438

[6] BayS D, SchwabacherM. Mining distance-based outliers in near linear time with randomization and a simple pruning rule[C]. Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2003, Washington, ACM Press: 29-38

[7] BreunigM M, KriegelH P, NgR T, et al.ZytkowJ M, RauchJ, et al.. OPTICS-OF: identifying local outliers [C]. Proceedings of the 3rd European Conference on Principles and Practice of Knowledge Discovery in Databases, 1999, Berlin, Springer: 262-270

[8] BreunigM M, KriegelH P, NgR. T, et al.ChenW, NaughtonJ F, BernsteinP A, et al.. LOF: identifying density-based local outliers[C]. Proceedings of the ACM SIGMOD International Conference on Management of Data, 2000, Dallas, ACM Press: 93-104

[9] JiangM F, TsengS S, SuC M. Two-phase clustering process for outliers detection[J]. Pattern Recognition Letters, 2001, 22(6–7): 691-700

[10] HeZeng-you, XuXiao-fei, DengSheng-chun. Discovering cluster-based local outliers[J]. Pattern Recognition Letters, 2003, 24(9–10): 1641-1650

[11] Arshad M H, Chan P K. Identifying outliers via clustering for anomaly detection[EB/OL]. [2003-06-13]. http://www.cs.fit.edu/Projects/tech-reports/cs-2003-19.pdf

[12] HeZeng-you, DengSheng-chun, XuXiao-fei. Outlier detection integrating semantic knowledge [C]. Proceeding of the 3rd International Conference on Web-Age Information Management, 2002, London, Central South University of Technology: 126-131

[13] HawkinsS, HeHong-xing, WilliamsG, et al.. Outlier detection using replicator neural networks [C]. Proceedings of the 4th International Conference and Data Warehousing and Knowledge Discovery, 2002, London, Central South University of Technology: 170-180

[14] YangFu-shengWavelet transformation’s analysis and application in engineering[M], 2000, Beijing, Science Press(in Chinese)

[15] LiQing-yan, WangNeng-chao, YiDa-yiNumerical analysis[M], 19863rd ed.Wuhan, Huazhong University of Science and Technology Press(in Chinese)