Outlier detection based on multi-dimensional clustering and local density

Abstract

Outlier detection is an important task in data mining. In fact, it is difficult to find the clustering centers in some sophisticated multidimensional datasets and to measure the deviation degree of each potential outlier. In this work, an effective outlier detection method based on multi-dimensional clustering and local density (ODBMCLD) is proposed. ODBMCLD firstly identifies the center objects by the local density peak of data objects, and clusters the whole dataset based on the center objects. Then, outlier objects belonging to different clusters will be marked as candidates of abnormal data. Finally, the top N points among these abnormal candidates are chosen as final anomaly objects with high outlier factors. The feasibility and effectiveness of the method are verified by experiments.

Keywords

data mining / outlier detection / outlier detection method based on multi-dimensional clustering and local density (ODBMCLD) algorithm / deviation degree

References

[1] KaurH, SinghG, MinhasJ. A review of machine learning based anomaly detection techniques [J]. International Journal of Computer Applications Technology and Research, 2013, 2(2): 185-187

[2] SinghK, UpadhyayaD S. Outlier detection: Applications and techniques [J]. International Journal of Computer Science Issues, 2012, 9(1): 307-323

[3] YeH, KitagawaH, XiaoJ. Continuous angle-based outlier detection on high-dimensional data streams [C]//. Proceedings of the 19th International Database Engineering & Applications Symposium, 2015162167

[4] PandaM, PatraM R. A novel classification via clustering method for anomaly based network intrusion detection system [J]. International Journal of Recent Trends in Engineering, 2009, 2(1): 211-232

[5] ShuchitaD, SinghK. Classification based outlier detection techniques [J]. International Journal of Computer Trends and Technology, 2012, 3(2): 294-298

[6] KurianM J, GladstonR S. An analysis on the performance of a classification based outlier detection system using feature selection [J]. International Journal of Computer Applications, 2015, 132(8): 15-21

[7] SugiyamaM, BorgwardtK M. Rapid distance-based outlier detection via sampling [C]//. Proceedings of the 26th International Conference on Neural Information Processing Systems, 2013467475

[8] CampelloR J G B, MoulaviD, SanderA J. Hierarchical density estimates for data clustering, visualization, and outlier detection [J]. ACM Transactions on Knowledge Discovery from Data (TKDD), 2015, 10(1): 1-5

[9] GosaviJ S, WadneV S. Unsupervised distance-based outlier detection using nearest neighbours algorithm on distributed approach: Survey [J]. International Journal of Innovative Research in Computer and Communication Engineering, 2014, 2(12): 7510-7514

[10] PavaniD, RajendraprasadK. The competent reverse nearest neighbors for outlier detection in high dimensional data [J]. International Journal of Computer Science and Technology, 2016, 7(1): 70-73

[11] UpadhyayaD S, SinghK. Nearest neighbour based outlier detection techniques [J]. International Journal of Computer Trends and Technology, 2012, 31(2): 299-303

[12] WangW, ZhangB-j, WangD, JiangY, QinS, XueLei. Anomaly detection based on probability density function with Kullback–Leibler divergence [J]. Signal Processing, 2016, 126: 12-17

[13] BhattV, DhakarM, ChaurasiaB K. Filtered clustering based on local outlier factor in data mining [J]. International Journal of Database Theory and Application, 2016, 9(5): 275-282

[14] BaiM, WangX T, XinJ C, WangG. An efficient algorithm for distributed density-based outlier detection on big data [J]. Neurocomputing, 2016, 181: 19-28

[15] MengJ-l, ShangH-k, BianLing. The application on intrusion detection based on k-means cluster algorithm [C]//. International Forum on Information Technology and Applications, 2009150152

[16] LeeM H, WeiH, LeeS H, LeeS M, ShinS S. Design of similarity measure for discrete data and application to multi-dimension [J]. Journal of Central South University, 2013, 20(4): 982-987

[17] RanjanR, SahooG. A new clustering approach for anomaly intrusion detection [J]. International Journal of Data Mining & Knowledge Management Process, 2014, 4(2): 29-38

[18] EsterM, KriegelH, SanderJ, XuX. A density-based algorithm for discovering clusters in large spatial databases with noise [C]//. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, 1996226231

[19] LuZ-m, LiuC, ZhangC-x, WangLei. Clustering method based on data division and partition [J]. Journal of Central South University, 2014, 21(1): 213-222

[20] DengZ, HuY-y, ZhuM, HuangX-h, DuBo. A scalable and fast OPTICS for clustering trajectory big data [J]. Cluster Computing, 2015, 18(2): 549-562