EDSUCh: A robust ensemble data summarization method for effective medical diagnosis

Abstract

Identifying rare patterns for medical diagnosis is a challenging task due to heterogeneity and the volume of data. Data summarization can create a concise version of the original data that can be used for effective diagnosis. In this paper, we propose an ensemble summarization method that combines clustering and sampling to create a summary of the original data to ensure the inclusion of rare patterns. To the best of our knowledge, there has been no such technique available to augment the performance of anomaly detection techniques and simultaneously increase the efficiency of medical diagnosis. The performance of popular anomaly detection algorithms increases significantly in terms of accuracy and computational complexity when the summaries are used. Therefore, the medical diagnosis becomes more effective, and our experimental results reflect that the combination of the proposed summarization scheme and all underlying algorithms used in this paper outperforms the most popular anomaly detection techniques.

Keywords

Data summarization / Ensemble / Medical diagnosis / Sampling

References

[1] A.N.M.B. Rashid, M. Ahmed, L.F. Sikos, P. Haskell-Dowland, A novel penalty-based wrapper objective function for feature selection in Big Data using cooperative co-evolution, IEEE Access 8 (2020) 150113-150129.

[2] A.N.M.B. Rashid, M. Ahmed, L.F. Sikos, P. Haskell-Dowland, Cooperative co-evolution for feature selection in big data with random feature grouping, J. Big Data 7 (1) (2020) 1-42.

[3] A.N.M.B. Rashid, M. Ahmed, L.F. Sikos, P. Haskell-Dowland, Anomaly detection in cybersecurity datasets via cooperative co-evolution-based feature selection, ACM Transact. Manag. Inform. Syst. 13 (3) (2022) 1-39.

[4] A.N.M. Rashid, M. Ahmed, S.R. Islam, A supervised rare anomaly detection technique via cooperative co-evolution-based feature selection using benchmark unsw_nb 15 dataset,in: International Conference on Ubiquitous Security, Springer, 2021, pp. 279-291, https://doi.org/10.1007/978-981-19-0468-4_21.

[5] A.N.M.B. Rashid, M. Ahmed, A.-S. K. Pathan, Infrequent pattern detection for reliable network traffic analysis using robust evolutionary computation, Sensors 21 (9) (2021) 3005.

[6] M. Ahmed, Intelligent big data summarization for rare anomaly detection, IEEE Access 7 (2019) 68669-68677.

[7] P. Lavanya, K. Kouser, M. Suresha, Effective feature representation using symbolic approach for classification and clustering of big data, Expert Syst. Appl. 173 (2021) 114658.

[8] A. Bharadwaj, A. Srinivasan, A. Kasi, B. Das, Extending the performance of extractive text summarization by ensemble techniques, in: 2019 11th International Conference on Advanced Computing, ICoAC, 2019, pp. 282-288, https://doi.org/10.1109/ICoAC48765.2019.246854.

[9] A. Onan, Two-stage topic extraction model for bibliometric data analysis based on word embeddings and clustering, IEEE Access 7 (2019) 145614-145633.

[10] T. Boongoen, N. Iam-On, Cluster ensembles: a survey of approaches with recent extensions and applications, Computer Sci. Rev. 28 (2018) 1-25.

[11] Z. Liao, L. Gao, T. Zhou, X. Fan, Y. Zhang, J. Wu, An oil painters recognition method based on cluster multiple kernel learning algorithm, IEEE Access 7 (2019) 26842-26854.

[12] J. Wu, S. Guo, H. Huang, W. Liu, Y. Xiang, Information and communications technologies for sustainable development goals: state-of-the-art, needs and perspectives, IEEE Commun. Surveys Tutorials 20 (3) (2018) 2389-2406.

[13] J. Wu, S. Guo, J. Li, D. Zeng, Big data meet green challenges: big data toward green applications, IEEE Syst. J. 10 (3) (2016) 888-900.

[14] R. Atat, L. Liu, J. Wu, G. Li, C. Ye, Y. Yang, Big data meet cyber-physical systems: a panoramic survey, IEEE Access 6 (2018) 73603-73636.

[15] S. Guha, R. Rastogi, K. Shim, Cure: an efficient clustering algorithm for large databases, Inf. Syst. 26 (1) (2001) 35-58.

[16] A.N. Mahmood, J. Hu, Z. Tari, C. Leckie, Critical infrastructure protection: resource efficient sampling to improve detection of less frequent patterns in network traffic, J. Netw. Comput. Appl. 33 (4) (2010) 491-502.

[17] M. Ahmed, A.N. Mahmood, M.J. Maher, An efficient technique for network traffic summarization using multiview clustering and statistical sampling, EAI Endorsed Transact. Scalable Informat. Syst. 2 (5) (2015) e4.

[18] M.S. Mahmud, J.Z. Huang, S. Salloum, T.Z. Emara, K. Sadatdiynov, A survey of data partitioning and sampling methods to support big data analysis, Big Data Min. Analy. 3 (2) (2020) 85-101.

[19] D. Gillman, A chernoff bound for random walks on expander graphs, SIAM J. Comput. 27 (4) (1998) 1203-1218.

[20] B. Efron, Bootstrap methods: another look at the jackknife,in: Breakthroughs in Statistics, Springer, 1992, pp. 569-593.

[21] K.M. Wolter, K.M. Wolter, Introduction to Variance Estimation, vol. 53, Springer, 2007, https://doi.org/10.1007/978-0-387-35099-8.

[22] E.L. Escobar, Y.G. Berger, A jackknife variance estimator for self-weighted two-stage samples, Stat. Sin. (2013) 595-613.

[23] Z. Mashreghi, D. Haziza, C. Léger, A survey of bootstrap methods in finite population sampling, Stat. Surv. 10 (2016) 1-52.

[24] M.Z. Islam, V. Estivill-Castro, M.A. Rahman, T. Bossomaier, Combining k-means and a genetic algorithm through a novel arrangement of genetic operators for high quality clustering, Expert Syst. Appl. 91 (2018) 402-417.

[25] M. Ahmed, A.N. Mahmood, J. Hu, A survey of network anomaly detection techniques, J. Netw. Comput. Appl. 60 (2016) 19-31.

[26] M. Ahmed, A.N. Mahmood, M.R. Islam, A survey of anomaly detection techniques in financial domain, Future Generat. Comput. Syst. 55 (2016) 278-288.

[27] M. Ahmed, A. Anwar, A.N. Mahmood, Z. Shah, M.J. Maher, An investigation of performance analysis of anomaly detection techniques for big data in scada systems, EAI Endorsed Transact. Industr. Networks Intelligent Syst. 2 (3) (2015) e5.

[28] M. Ahmed, Reservoir-based network traffic stream summarization for anomaly detection, Pattern Anal. Appl. 21 (2) (2018) 579-599.