Enhancing Data Lake Management Systems with LDA Approach

Abstract

In today’s fiercely competitive business landscape, data have emerged as a precious asset crucial for any company’s growth. It embodies a genuine catalyst for economic and strategic advantages, distinguishing industry leaders from the rest. Prominent organizations recognize the importance of not just amassing data from diverse sources but also harnessing the transformative power of data analytics for informed determination processes. Within this setting, the data lake solution stands as a robust framework handling vast data sources and enabling data investigations to support decision-making tasks. This paper delves into the realm of intelligent data lake management systems designed to overcome the limitations of traditional business intelligence, which struggles to meet the demands of data-driven decision-making. Data lakes excel in the analysis of data from myriad sources, particularly when data cleaning becomes a time-consuming endeavor. Still, managing diverse datasets devoid of a predefined data structure presents a significant challenge, potentially leading to a data lake devolving into a data swamp. Within this article, we adopt the Latent Dirichlet Allocation model to oversee the data lake environment’s handling, processing, analysis, and display of huge datasets. To evaluate the efficacy of our suggested approach, we conducted comprehensive assessments using the topic coherence metric. Our experiments yielded results indicating the superior accuracy of our approach when applied to the tested datasets.

Keywords

data lake management / Latent Dirichlet Allocation / data analytics / big data / topic modeling

Conflicts of Interest

The authors declare that they have no conflicts of interest to this work.

Data Availability Statement

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Author Contribution Statement

Mohamed Cherradi: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing - original draft, Writing - review & editing, Visualization, Project administration. Anass El Haddadi: Supervision.

References

[1] Alnoukari, M. (2021). From business intelligence to big data: The power of analytics. In A. Azevedo & M. Santos (Eds.), Integration challenges for analytics, business intelligence, and data mining (pp. 44-62). IGI Global. https://doi.org/10.4018/978-1-7998-5781-5.ch003

[2] Ruzgas T., & Dabulytė-Bagdonavičienė J. (2017). Business intelligence for big data analytics. International Journal of Computer Applications Technology and Research, 6(1), 1-8. https://doi.org/10.7753/IJCATR0601.1001

[3] Huang S. C., McIntosh S., Sobolevsky S., & Hung P. (2017). Big data analytics and business intelligence in industry. Information Systems Frontiers, 19, 1229-1232. https://doi.org/10.1007/s10796-017-9804-9

[4] Olawoyin A. M., Leung C. K., & Cuzzocrea A. (2021). Open data lake to support machine learning on arctic big data. In 2021 IEEE International Conference on Big Data, 5215-5224. https://doi.org/10.1109/BigData52589.2021.9671453

[5] Cherradi M., & El Haddadi A. (2023). DLDB-service:An extensible data lake system. In Emerging Trends in Intelligent Systems & Network Security: Conference Proceedings, 211-220. https://doi.org/10.1007/978-3-031-15191-0_20

[6] Rajapaksha M., & Silva T. (2019). Semantic information retrieval based on topic modeling and community interests mining. In 2019 Moratuwa Engineering Research Conference, 60-65. https://doi.org/10.1109/MERCon.2019.8818935

[7] Maini E., Venkateswarlu B., & Gupta A. (2018). Data lake - An optimum solution for storage and analytics of big data in cardiovascular disease prediction system. International Journal of Computational Engineering & Management, 21(6), 33-39.

[8] Khine P. P., & Wang Z. S. (2018). Data lake: A new ideology in big data era. In 4th Annual International Conference on Wireless Communication and Sensor Network, 17, 03025. https://doi.org/10.1051/itmconf/20181703025

[9] Terrizzano I., Schwarz P., Roth M., & Colino J. E. (2015). Data wrangling: The challenging journey from the wild to the lake. In Conference on Innovative Data Systems Research.

[10] Marty R. (2015). The security data lake: Leveraging big data technologies to build a common data repository for security. USA: O’Reilly.

[11] Sitarska-Buba, M., & Zygala R. (2020). Data lake: Strategic challenges for small and medium sized enterprises. In M. Herners, A. Rot & D. Jelonek (Eds.), Towards industry 4.0 Current—challenges in information systems (pp.183-200). Springer.

[12] Walker C., & Alrehamy H. (2015). Personal data lake with data gravity pull.In2015 IEEE Fifth International Conference on Big Data and Cloud Computing, 160-167. https://doi.org/10.1109/BDCloud.2015.62

[13] Fang H. (2015). Managing data lakes in big data era: What’sa data lake and why has it became popular in data management ecosystem. In 2015 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems, 820-824. https://doi.org/10.1109/CYBER.2015.7288049

[14] Ben S. (2018). Architecting data lakes. USA: O’Reilly.

[15] Cherradi M., Bouhafer F., & El Haddadi A. (2023). Data lake governance using IBM-Watson knowledge catalog. Scientific African, 21,e01854.https://doi.org/10.1016/j.sciaf.2023.e01854

[16] Inmon B. (2016). Data lake architecture: Designing the data lake andavoidingthe garbagedump. USA: Technics Publications.

[17] Madera C., & Laurent A. (2016). The next information architecture evolution: The data lake wave. In Proceedings of the 8th International Conference on Management of Digital EcoSystems, 174-180. https://doi.org/10.1145/3012071.3012077

[18] Ravat F., & Zhao Y. (2019). Data lakes: Trends and perspectives. Database and Expert Systems Applications: 30th International Conference, 304-313. https://doi.org/10.1007/978-3-030-27615-7_23

[19] Mathis C. (2017). Data lakes. Datenbank-Spektrum, 17(3), 289-293. https://doi.org/10.1007/s13222-017-0272-7

[20] Suriarachchi I., & Plale B. (2016). Crossing analytics systems: A case for integrated provenance in data lakes. In 2016 IEEE 12th International Conference on e-Science, 349-354. https://doi.org/10.1109/eScience.2016.7870919

[21] Cherradi M., & El Haddadi A. (2022). Data lakes: A survey paper.InProceedings of the 6th International Conference on Smart City Applications,823-835. https://doi.org/10.1007/978-3-030-94191-8_66

[22] Kim M., & Kim D. (2022). A suggestion on the LDA-based topic modeling technique based on ElasticSearch for indexing academic research results. Applied Sciences, 12(6), 3118. https://doi.org/10.3390/app12063118

[23] Jelodar H., Wang Y., Yuan C., Feng X., Jiang X., Li Y., & Zhao L. (2019). Latent Dirichlet Allocation (LDA) and topic modeling: Models, applications, a survey. Multimedia Tools and Applications, 78(11), 15169-15211. https://doi.org/10.1007/s11042-018-6894-4

[24] Costa Silva C., Galster M., & Gilson F. (2021). Topic modeling in software engineering research. Empirical Software Engineering, 26(6), 120. https://doi.org/10.1007/s10664-021-10026-0

[25] Amara A., Hadj Taieb M. A., & Ben Aouicha M. (2021). Multilingual topic modeling for tracking COVID-19 trends based on Facebook data analysis. Applied Intelligence, 51, 3052-3073. https://doi.org/10.1007/s10489-020-02033-3

[26] Kherwa P., & Bansal P. (2019). Topic modeling: A comprehensive review. EAI Endorsed Transactions on Scalable Information Systems, 7(24), e2. https://doi.org/10.4108/eai.13-7-2018.159623

[27] Sharif M., Maskat R., Baharum Z., & Maskat K. (2023). A scoping review of topic modelling on online data. Indonesian Journal of Electrical Engineering and Computer Science, 31(3), 1633-1641.

[28] Heintz I., Gabbard R., Srivastava M., Barner D., Black D., Friedman M., & Weischedel R. (2013). Automatic extraction of linguistic metaphors with LDA topic modeling. In Proceedings of the First Workshop on Metaphor in NLP, 58-66.

[29] Lui M., Lau J. H., & Baldwin T. (2014). Automatic detection and language identification of multilingual documents. Transactions of the Association for Computational Linguistics, 2,27-40. https://doi.org/10.1162/tacl_a_00163

[30] Levy K., & Franklin M. (2014). Driving regulation: Using topic models to examine political contention in the US trucking industry. Social Science Computer Review, 32(2), 182-194. https://doi.org/10.1177/0894439313506847

[31] Zirn C., & Stuckenschmidt H. (2014). Multidimensional topic analysis in political texts. Data & Knowledge Engineering, 90, 38-53. https://doi.org/10.1016/j.datak.2013.07.003

[32] Zhang Y., Chen M., Huang D., Wu D., & Li Y. (2017). iDoctor: Personalized and professionalized medical recommendations based on hybrid matrix factorization. Future Generation Computer Systems, 66,30-35. https://doi.org/10.1016/j.future.2015.12.001

[33] Zhang L., Sun X., & Zhuge H. (2015). Topic discovery of clusters from documents with geographical location. Concurrency and Computation: Practice and Experience, 27(15), 4015-4038. https://doi.org/10.1002/cpe.3474

[34] Yang X., Lo D., Li L., Xia X., Bissyandé T. F., & Klein J.(2017). Characterizing malicious Android apps by mining topic-specific data flow signatures. Information and Software Technology, 90,27-39. https://doi.org/10.1016/j.infsof.2017.04.007

[35] Li Y., Zhou X., Sun Y., & Zhang H. (2016). Design and implementation of Weibo sentiment analysis based on LDA and dependency parsing. China Communications, 13(11), 91-105. https://doi.org/10.1109/CC.2016.7781721

[36] Blei D. M., Ng A. Y., & Jordan M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3, 993-1022.