Construction of AI Literacy Evaluation System for College Students and an Empirical Study at Wuhan University

Abstract

As artificial intelligence (AI) technology continues to evolve in the digital era, developing AI literacy among college students has become a crucial educational priority. This study aims to establish a scientific AI literacy evaluation system and to empirically assess the AI literacy levels of undergraduate students at Wuhan University, with the findings providing data support and theoretical reference for future AI education policy-making and curriculum design in higher education institutions. In response to the demands of AI education and university talent cultivation objectives, this study develops an AI literacy evaluation system for college students, based on the KSAVE (knowledge, skill, attitude, value, and ethics) model and the UNESCO AI competency framework. The system includes 4 level-1 indicators (AI attitude, AI knowledge, AI capability, and AI ethics), 10 level-2 indicators, and 25 level-3 indicators. The Delphi method was used to determine indicator content, while the analytic hierarchy process was employed to calculate the weights for each level of indicators. Through large-scale questionnaire surveys and statistical analysis, the study empirically measured the AI literacy levels of 1,651 undergraduate students at Wuhan University and analyzed variations in AI literacy across factors including gender, academic year, academic discipline, and technical background. The results demonstrate that the constructed AI literacy evaluation system is scientifically sound and highly applicable, providing a comprehensive and objective measure of students’ AI literacy levels. Furthermore, notable differences were observed in AI literacy levels across different dimensions among Wuhan University undergraduates, with variables such as academic discipline, technical background, and participation in digital intelligence education programs significantly influencing students’ AI literacy, particularly in knowledge and capability dimensions.

Graphical abstract

Keywords

artificial intelligence (AI) literacy / digital intelligence education / AI / evaluation system

References

[1] Andresen, S. L. (2002). John McCarthy: Father of AI.IEEE Intelligent Systems, 17(5): 84–85

[2] Agre, P. E. (1972). What to read: A biased guide to AI literacy for the beginner. Cambridge: MIT Artificial Intelligence Laboratory.

[3] Bewersdorff, A., Hornberger, M., Nerdel, C., & Schiff, D. (2024). AI advocates and cautious critics: How AI attitudes, AI interest, use of AI, and AI literacy build university students’ AI self-efficacy.Computers and Education: Artificial Intelligence, 8: 100340

[4] Casal-Otero, L., Catala, A., Fernández-Morante, C., Taboada, M., Cebreiro, B., & Barro, S. (2023). AI literacy in K-12: A systematic literature review.International Journal of STEM Education, 10(1): 29

[5] Celik, I. (2023). Towards intelligent-TPACK: An empirical study on teachers’ professional knowledge to ethically integrate artificial intelligence (AI)-based tools into education.Computers in Human Behavior, 138: 107468

[6] Černý, M. (2024). University students’ conceptualisation of AI literacy: Theory and empirical evidence.Social Sciences, 13(3): 129

[7] Chang, Y., Wang, X., Wang, J., Yuan, W., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P.S., Yang, Q., & Xie, X. (2024). A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology, 15(3): 39

[8] Chiu, T. (2023). The impact of generative AI (genAI) on practices, policies and research direction in education: A case of ChatGPT and midjourney.Interactive Learning Environments, 32(10): 1–17

[9] Druga, S., Williams, R., Breazeal, C., & Resnick, M. (2017). “Hey Google, is it ok if I eat you?” Initial explorations in child-agent interaction. In: Proceedings of the 2017 Conference on Interaction Design and Children, Stanford. New York: ACM, 595–600.

[10] Gill, K. S. (1991). Artificial intelligence for social citizenship: Toward an anthropocentric technology.Applied Artificial Intelligence, 5(1): 15–27

[11] Gratch, J., Lucas, G. M., King, A. A., & Morency, L. P. (2014). It’s only a computer: The impact of human-agent interaction in clinical interviews. In: Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, Paris. Richland: International Foundation for Autonomous Agents and Multiagent Systems, 85–92.

[12] Griffin, P., McGaw, B., & Care, E. (2012). Assessment and teaching of 21st century skills. Dordrecht: Springer.

[13] Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S. & Huber, P. (2016). Artificial intelligence and computer science in education: From kindergarten to university. In: Proceedings of the 2016 IEEE Frontiers in Education Conference. Erie: IEEE, 1–9.

[14] Karaca, O., Çalışkan, S. A., & Demir, K. (2021). Medical artificial intelligence readiness scale for medical students (MAIRS-MS)—Development, validity and reliability study.BMC Medical Education, 21: 1–9

[15] Latif, S., Usama, M., Malik, M. I., & Schuller, B. W. (2023). Can large language models aid in annotating speech emotional data? Uncovering new frontiers. arXiv Preprint, arXiv:2307.06090.

[16] Lintner, T. (2024). A systematic review of AI literacy scales.npj Science of Learning, 9(1): 50

[17] Liu, R., & Shah, N. B. (2023). ReviewerGPT? An exploratory study on using large language models for paper reviewing. arXiv Preprint, arXiv:2306.00622.

[18] Ministry of Education of the People’s Republic of China (MoE). (2024a). Notice from the Ministry of Education on Publishing “AI Innovation Action Plan for Institutions of Higher Education”. Available from Ministry of Education website.

[19] Ministry of Education of the People’s Republic of China (MoE). (2024b). Notice from the Department of Higher Education, Ministry of Education, on Publishing the First Batch of Typical Cases of “Artificial Intelligence + Higher Education” Application Scenarios. Available from Ministry of Education website. (in Chinese)

[20] Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Shen, M. Q. (2021). Conceptualizing AI literacy: An exploratory review.Computers and Education: Artificial Intelligence, 2(1): 100041

[21] Partridge, D. (1987). The scope and limitations of first generation expert systems.Future Generation Computer Systems, 3(1): 1–10

[22] Peng, L., Nuchged, B., & Gao, Y. (2023). Spoken language intelligence of large language models for language learning. arXiv Preprint, arXiv:2308.14536.

[23] Pereira, A., Prada, R., & Paiva, A. (2014). Improving social presence in human-agent interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto. New York: ACM, 1449–1458.

[24] Scammell, A. (2000). Visions of the information future.Aslib Proceedings, 52(7): 264–269

[25] Su, G. (2018). Unemployment in the AI age.AI Matters, 3(4): 35–43

[26] Su, W., Guo, H., Lu, Z., Pan Y., & Liu G (2024). Construction of artificial intelligence literacy evaluation index system and validation of its effectiveness in China’s college and university student population.Library Development, 1–25

[27] Tenório, K., Olari, V., Chikobava, M., & Romeike, R. (2023). Artificial intelligence literacy research field: A bibliometric analysis from 1989 to 2021. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education, Toronto. New York: ACM, 1083–1089.

[28] UNESCO. (2025a). AI competency framework for students. Available from UNESCO website.

[29] UNESCO. (2025b). K-12 AI curricula: A mapping of government-endorsed AI curricula. Available from UNESCO website.

[30] Wang, B., Rau, P.-L. P., & Yuan, T. (2022). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale.Behaviour & Information Technology, 42(9): 1–14

[31] Wang, Y. Y., & Chuang, Y. W. (2024). Artificial intelligence self-efficacy: Scale development and validation.Education and Information Technologies, 29(4): 4785–4808

[32] Wu, D. (2024). Wuhan University AI literacy evaluation guide. Wuhan: Wuhan University Press.

Acknowledgments

This study was funded by the Hubei Provincial Undergraduate Higher Education Provincial-Level Teaching Reform Research Project “Exploration and Practice of the ‘Four-in-One’ Digital Intelligence Education Transformation Path in Higher Education Institutions” and Wuhan University Undergraduate Education Quality Construction Comprehensive Reform Project Digital Intelligence Field Research Special Topic “Evaluation of Artificial Intelligence Literacy and Construction of Professional Map in Universities in the Digital Intelligence Era.”

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethics Statement

The authors declare that their Institutional Ethics Committee confirmed that no ethical review was required for this study. Written informed consent for participation was not required because all participants’ data was anonymized before the statistical analyses were done.

Data Availability Statements

The authors confirm that all data generated or analysed during this study are included in this published article.

RIGHTS & PERMISSIONS

2025 Higher Education Press