Decoding ChatGPT's impact on student satisfaction and performance: a multimodal machine learning and explainable AI approach

Abstract

The extensive use of artificial intelligence (AI) tools in education and their effectiveness and explicability have become an essential area of investigation because of their scope for ethical and fair use. Therefore, this innovative approach proposed a multimodal machine learning and explainable AI (XAI) approach to predict how ChatGPT’s usage impacts students’ academic outcomes, including satisfaction and performance. Three machine learning models, XGBoost, Random Forest, and Support Vector Machine, were used to predict students’ performance and level of satisfaction. The XAI techniques, SHapley Additive exPlanations and Local Interpretable Model-Agnostic Explanations, were used to investigate how models make decisions with ethical standards and ensure models are reliable and fair in their usage. A custom-created dataset, ChatGPT Survey Data, was utilized for the experiment. All three models gave the promised classification, with Support Vector Machine having the highest accuracy at 89% and the receiver operating characteristic curve and the area under this curve (AUC-ROC) at 92%. Random Forest has 87% accuracy and an AUC-ROC of 90%. XGBoost showed the best accuracy with 92% (R²). XAI analysis revealed ChatGPT usage and satisfaction as key predictors of academic performance, advancing AI’s role in education.

Keywords

ChatGPT / academic performance / student satisfaction / machine learning / explainable AI (XAI)

References

[1] Hadi Mogavi R,Juho Kim J.ChatGPT in education: a blessing or a curse? A qualitative study exploring early adopters’ utilization and perceptions.Comput Hum Behav Artif Hum2024;2:100027

[2] Bower M,Lai JWM,Alfano M.How should we change teaching and assessment in response to increasingly powerful generative Artificial Intelligence? Outcomes of the ChatGPT teacher survey.Educ Inf Technol2024;29:15403-39

[3] Fitria TN.Artificial intelligence (AI) technology in OpenAI ChatGPT application: a review of ChatGPT in writing English essay.ELTF2023;12:44-58

[4] Aithal PS.Application of ChatGPT in higher education and research - A futuristic analysis.Int J Appl Eng Manag Lett2023;7:168-94

[5] Dahri NA,Al-Rahmi WM.Extended TAM based acceptance of AI-Powered ChatGPT for supporting metacognitive self-regulated learning in education: a mixed-methods study.Heliyon2024;10:e29317 PMCID:PMC11016976

[6] Korinek A. The impact of language models on cognitive automation: a panel discussion. 2023. Available from: https://coilink.org/20.500.12592/rd9kg8 [Last accessed on 28 May 2025]

[7] Oravec JA.Artificial intelligence implications for academic cheating: expanding the dimensions of responsible human-AI collaboration with ChatGPT.J Interact Learn Res2023;34:213-37Available from: https://philarchive.org/archive/ORAAII [Last accessed on 28 May 2025]

[8] Lo CK.What is the impact of ChatGPT on education? A rapid review of the literature.Educ Sci2023;13:410

[9] Borger JG,Anderton H.Artificial intelligence takes center stage: exploring the capabilities and implications of ChatGPT and other AI-assisted technologies in scientific research and education.Immunol Cell Biol2023;101:923-35

[10] Jo H.Decoding the ChatGPT mystery: a comprehensive exploration of factors driving AI language model adoption.Inf Dev2023;02666669231202764

[11] Li B,Liu L.Users’ intention to adopt artificial intelligence-based chatbot: a meta-analysis.Serv Ind J2023;43:1117-39

[12] Zhou J,Chen W.How ChatGPT transformed teachers: the role of basic psychological needs in enhancing digital competence.Front Psychol2024;15:1458551 PMCID:PMC11483856

[13] Al-Kfairy M.Factors impacting the adoption and acceptance of ChatGPT in educational settings: a narrative review of empirical studies.Appl Syst Innov2024;7:110

[14] Wu C.Critical factors for why ChatGPT enhances learning engagement and outcomes.Educ Inf Technol2025;1-32

[15] Rehman AU,Islam MS,Imtiaz A.Assessing the usage of ChatGPT on life satisfaction among higher education students: the moderating role of subjective health.Technol Soc2024;78:102655

[16] Gupta A. Data-driven hiring: implementing AI and assessing the impact of AI on recruitment efficiency and candidate quality. 2024. Available from: https://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-531407 [Last accessed on 28 May 2025]

[17] Kotsiantis S,Pintelas P.Predicting students’ performance in distance learning using machine learning techniques.Appl Artif Intell2004;18:411-26

[18] Górriz J,Álvarez-Marquina A.Computational approaches to explainable artificial intelligence: advances in theory, applications and trends.Inf Fusion2023;100:101945

[19] Khan UA. Harnessing AI to boost metacognitive learning in education. 2023. Available from: http://urn.fi/URN:NBN:fi-fe20230825108259 [Last accessed on 28 May 2025]

[20] Dahri NA,Al-Rahmi WM.Investigating AI-based academic support acceptance and its impact on students’ performance in Malaysian and Pakistani higher education institutions.Educ Inf Technol2024;29:18695-744

[21] Annuš N.Educational software and artificial intelligence: students' experiences and innovative solutions.Inf Technol Learn Tools2024;101:200-26

[22] Kuddus K.Artificial intelligence in language learning: practices and prospects. In: Mire A, Malik S, Tyagi AK, editors. Advanced analytics and deep learning models. Wiley; 2022. pp. 1-17.

[23] Kumar A. AI-based language translation and interpretation services: improving accessibility for visually impaired students. 2024, pp.179-90. Available from: https://www.researchgate.net/profile/Ravindra-Kushwaha-2/publication/381408076_Transforming_Learning_The_Power_of_Educational_Technology/links/666c0a12b769e7691933ab54/Transforming-Learning-The-Power-of-Educational-Technology.pdf#page=192 [Last accessed on 28 May 2025]

[24] Essel HB,Essuman AB.ChatGPT effects on cognitive skills of undergraduate students: receiving instant responses from AI-based conversational large language models (LLMs).Comput Educ Artif Intell2024;6:100198

[25] Woo DJ,Guo K.Teaching EFL students to write with ChatGPT: students' motivation to learn, cognitive load, and satisfaction with the learning process.Educ Inf Technol2024;29:24963-90

[26] Kim JK,Rickard M.ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine.J Pediatr Urol2023;19:598-604

[27] Sandu N.Adoption of AI-chatbots to enhance student learning experience in higher education in India. In Proceedings of the 2019 18th International Conference on Information Technology Based Higher Education and Training (ITHET); IEEE, 2019; pp. 1-5.

[28] Javaid M,Singh RP,Khan IH.Unlocking the opportunities through ChatGPT Tool towards ameliorating the education system.BenchCouncil Trans Benchmarks Stand Eval2023;3:100115

[29] van den Berg G.ChatGPT and generative AI: possibilities for its contribution to lesson planning, critical thinking and openness in teacher education.Educ Sci2023;13:998

[30] Ng DTK,Leung JKL.Empowering student self-regulated learning and science education through ChatGPT: a pioneering pilot study.Br J Educ Technol2024;55:1328-53

[31] Elsayary A.An investigation of teachers' perceptions of using ChatGPT as a supporting tool for teaching and learning in the digital era.J Comput Assist Learn2024;40:931-45

[32] Stolpe K.Artificial intelligence literacy for technology education.Comput Educ Open2024;6:100159

[33] Yu H.The application and challenges of ChatGPT in educational transformation: new demands for teachers’ roles.Heliyon2024;10:e24289 PMCID:PMC10828640

[34] Pelima LR,Rosmansyah Y.Predicting university student graduation using academic performance and machine learning: a systematic literature review.IEEE Access2024;12:23451-65

[35] Fan J.A big data and neural networks driven approach to design students management system.Soft Comput2024;28:1255-76

[36] Bellaj M,Boudra S.Educational data mining: employing machine learning techniques and hyperparameter optimization to improve students’ academic performance.Int J Online Biomed Eng2024;20:55-74

[37] Khosravi A.Leveraging educational data mining: XGBoost and random forest for predicting student achievement.Int J Data Sci Adv Anal2024;6:387-93

[38] Kalra N,Verma S.Advancements in AI based healthcare techniques with focus on diagnostic techniques.Comput Biol Med2024;179:108917

[39] Abgrall G,Chelly Dagdia Z,Monnet X.Should AI models be explainable to clinicians?.Crit Care2024;28:301 PMCID:PMC11391805

[40] Patil D.Explainable artificial intelligence (XAI): enhancing transparency and trust in machine learning models. 2024.

[41] Mane D,Khode O,Bhat K.Unlocking machine learning model decisions: a comparative analysis of LIME and SHAP for Enhanced interpretability.J Electr Syst2024;20:1252-67.

[42] Ahmed S,Shahadat Hossain M.A comparative analysis of LIME and SHAP interpreters with explainable ML-based diabetes predictions.IEEE Access2025;13:37370-88

[43] Sanfo JM.Application of explainable artificial intelligence approach to predict student learning outcomes.J Comput Soc Sci2025;8:9

[44] Kala A,Yildiz TT.Early prediction of student performance in face-to-face education environments: a hybrid deep learning approach with XAI techniques.IEEE Access2024;12:191635-49

[45] Huang Y,Chen J.Applying machine learning and SHAP method to identify key influences on middle-school students’ mathematics literacy performance.J Intell2024;12:93 PMCID:PMC11508920

[46] Schneider J.Explainable generative AI (GenXAI): a survey, conceptualization, and research agenda.Artif Intell Rev2024;57:10916

[47] Villegas-Ch W,Sánchez-Viteri S.Application of artificial intelligence in online education: influence of student participation on academic retention in virtual courses.IEEE Access2024;12:73045-65

[48] Kim K,Shin S.Explainable prediction of problematic smartphone use among South Korea's children and adolescents using a machine learning approach.Int J Med Inform2024;186:105441

[49] Dahri NA,Vighio MS.Exploring the impact of ChatGPT on teaching performance: findings from SOR theory, SEM and IPMA analysis approach.Educ Inf Technol2025;1-36

[50] Bin Rofi I,Ahmed MS.Identifying influences: a machine learning and explainable AI approach to analyzing social media addiction resulting from academic frustration. In Proceedings of the 11th International Conference on Networking, Systems, and Security; 2024, pp. 128-36.

[51] Benavides D,Ruiz-Cortés A.Automated analysis of feature models 20 years later: a literature review.Inf Syst2010;35:615-36

[52] Kotsiantis SB,Pintelas PE.Machine learning: a review of classification and combining techniques.Artif Intell Rev2006;26:159-90

[53] Soibelman L.Data preparation process for construction knowledge generation through knowledge discovery in databases.J Comput Civ Eng2002;16:39-48

[54] Mahmud A,Sohel A,Bhuiyan T.Adoption of ChatGPT by university students for academic purposes: partial least square, artificial neural network, deep neural network and classification algorithms approach.Array2024;21:100339

[55] Sahin EK.Implementation of free and open-source semi-automatic feature engineering tool in landslide susceptibility mapping using the machine-learning algorithms RF, SVM, and XGBoost.Stoch Environ Res Risk Assess2023;37:1067-92

[56] Dehdarirad T.Evaluating explainability in language classification models: a unified framework incorporating feature attribution methods and key factors affecting faithfulness.Data Inf Manag2025;100101

[57] Lee S,Park J.Deep learning-based natural language processing for detecting medical symptoms and histories in emergency patient triage.Am J Emerg Med2024;77:29-38

[58] Wang S.Academic achievement prediction in higher education through interpretable modeling.PLoS One2024;19:e0309838 PMCID:PMC11376577

[59] Nnadi LC,Rahman MM.Prediction of students’ adaptability using explainable AI in educational machine learning models.Appl Sci2024;14:5141

[60] Goswamy A,Islam Z.Factors affecting injury severity at pedestrian crossing locations with Rectangular RAPID Flashing Beacons (RRFB) using XGBoost and random parameters discrete outcome models.Accid Anal Prev2023;181:106937

[61] Ali Y,Irfan M.An eXtreme gradient boosting model for predicting dynamic modulus of asphalt concrete mixtures.Constr Build Mater2021;295:123642

[62] Cappelli F,Grimaldi S.Random forest and feature importance measures for discriminating the most influential environmental factors in predicting cardiovascular and respiratory diseases.Int J Environ Res Public Health2024;21:867 PMCID:PMC11276884

[63] Antoniadis A,Poggi J.Random forests for global sensitivity analysis: a selective review.Reliab Eng Syst Saf2021;206:107312

[64] Pabreja K.Understanding college students’ satisfaction with ChatGPT: An exploratory and predictive machine learning approach using feature engineering.MIER J Educ Stud Trends Pract2024;14:37-63

[65] Gopal Y. Exploring academic perspectives: sentiments and discourse on ChatGPT adoption in higher education. 2024. Available from: https://d-nb.info/1343649315/34 [Last accessed on 28 May 2025]

[66] Ngo TTA,An GK.ChatGPT for educational purposes: investigating the impact of knowledge management factors on student satisfaction and continuous usage.IEEE Trans Learning Technol2024;17:1341-52

[67] Almulla MA.Investigating influencing factors of learning satisfaction in AI ChatGPT for research: university students perspective.Heliyon2024;10:e32220 PMCID:PMC11200296

[68] Wang S,Reeves TC.Professional development to enhance teachers' practices in using information and communication technologies (ICTs) as cognitive tools: lessons learned from a design-based research study.Comput Educ2014;79:101-15

[69] Panda S.Exploring the viability of ChatGPT as an alternative to traditional chatbot systems in library and information centers.LHTN2023;40:22-5

[70] Aydin Yildiz T.Yildiz T. The impact of ChatGPT on language learners’ motivation.J Teach Educ Lifelong Learn2023;5:582-97

[71] Aldughayfiq B,Jhanjhi NZ.Explainable AI for retinoblastoma diagnosis: interpreting deep learning models with LIME and SHAP.Diagnostics2023;13:1932 PMCID:PMC10253103

[72] Liu D,Webster J.Toward meaningful engagement.MIS Q2017;41:1011-34.

[73] Lin X.Exploring the role of ChatGPT as a facilitator for motivating self-directed learning among adult learners.Adult Learn2024;35:156-66

[74] Dempere J,Hesham A.The impact of ChatGPT on higher education.Front Educ2023;8:1206936

[75] Li Y,Chiu TK.Systematics review on artificial intelligence chatbots and ChatGPT for language learning and research from self-determination theory (SDT): what are the roles of teachers?.Interact Learn Environ2025;33:1850-64