Evaluating a zero-shot GenAI assistant for clinical record writing in nursing education

Abstract

Objective: This study pilot-tested a zero-shot prompting approach for generating pedagogically appropriate AI feedback to support nursing students’ reflective record writing. A generative AI system was developed using GPT-4o, with prompts carefully designed to align with ethical, instructional, and contextual principles relevant to clinical practicum education. Following the ADDIE framework, this evaluation examined the feasibility and instructional applicability of the AI-generated feedback rather than aiming to establish generalizable effects.
Methods: Three nursing faculty members independently reviewed 126 AI-generated responses using a 5-point scale based on clarity, relevance, and educational appropriateness. Items scoring less than 5 were revised, and a second round of evaluation was conducted.
Results: The proportion of responses rated 5 increased from 69% to 96% after prompt refinement, confirming that even minor adjustments—such as clarifying vague instructions or reinforcing ethical boundaries—had a measurable impact. Inter-rater reliability was moderate, reflecting diverse faculty perspectives—a feature aligned with the contextual complexity of nursing education. The conservative scoring approach, in which the lowest score was adopted per item, ensured that potential pedagogical risks were not overlooked.
Conclusions: These findings suggest that zero-shot prompting offers a practical and scalable method for aligning generative AI systems with educational goals, even without training data or programming expertise. Rather than positioning AI as a replacement for instructors, this study frames GenAI as a formative support tool shaped by educator input. The study is limited by its use of simulated student inputs and a small, single-institution faculty sample; therefore, future work should assess implementation with real students to examine usability, personalization, and effectiveness in authentic practicum environments.

Keywords

Artificial Intelligence / Clinical competence / Computer-assisted instruction / Education / Nursing / Students

References

[1] Chen L, Kong D, Zhang S, et al. A quasi-experimental study of specialized training on the clinical decision-making skills and social problem-solving abilities of nursing students. Contemp Nurse. 2021; 57(1-2): 4-12. PMid:33820488 https://doi.org/10.1080/10376178.2021.1912616

[2] Zhang J, Shields L, Ma B, et al. The clinical learning environment, supervision and future intention to work as a nurse in nursing students: a cross-sectional and descriptive study. BMC Med Educ. 2022; 22: 548. PMid:35841091 https://doi.org/10.1186/s12909-022-03609-y

[3] Bjerkvik LK, Hilli Y. Reflective writing in undergraduate clinical nursing education: a literature review. Nurse Educ Pract 2019; 35: 32-41. PMid:30660960 https://doi.org/10.1016/j.nepr.2018.11.013

[4] Reierson IÅ, Mofossbakke RG, Solli H. Nursing students’ perspectives on learning electronic health record documentation in an academic setting and during clinical placement: a qualitative study. BMC Nurs. 2025; 24: 764. PMid:40597200 https://doi.org/10.1186/s12912-025-03320-5

[5] Shen M, Shen Y, Liu F, et al. Prompts, privacy, and personalized learning: integrating AI into nursing education—a qualitative study. BMC Nurs. 2025; 24: 470. PMid:40301862 https://doi.org/10.1186/s12912-025-03115-8

[6] Khlaif ZN, Salameh N, Ajouz M, et al. Using Generative AI in nursing education: students’ perceptions. BMC Med Educ. 2025; 25: 926. PMid:40598139 https://doi.org/10.1186/s12909-025-07416-z

[7] Saban M, Dubovi I. A comparative vignette study: evaluating the potential role of a generative AI model in enhancing clinical decision-making in nursing. J Adv Nurs. 2024; 81(11): 7489-99. PMid:38366690 https://doi.org/10.1111/jan.16101

[8] Rony MKK, Ahmad S, Tanha SM, et al. Nursing educators’ perspectives on the integration of artificial intelligence into academic settings. SAGE Open Nurs. 2025; 11. PMid:40386172 https://doi.org/10.1177/23779608251342931

[9] Yampolskiy RV. On the controllability of artificial intelligence: an analysis of limitations. J Cyber Secur Mob. 2022; 11(3): 321-403. https://doi.org/10.13052/jcsm2245-1439.1132

[10] Ogunleye B, Zakariyyah KI, Ajao O, et al. A systematic review of generative AI for teaching and learning practice. Educ Sci. 2024; 14(6): 636. https://doi.org/10.3390/educsci14060636

[11] Chan CKY, Tsi LHY. Will generative AI replace teachers in higher education? A study of teacher and student perceptions. Stud Educ Eval. 2024; 83: 101395. https://doi.org/10.1016/j.stueduc.2024.101395

[12] Hussein R, Yahya N, Chew HSJ, et al. Ethical and social considerations of applying artificial intelligence in healthcare: a two-pronged scoping review. BMC Med Ethics. 2025; 26: 68. PMid:40420080 https://doi.org/10.1186/s12910-025-01198-1

[13] Abdelwanis M, Simsekler MCE, Gabor AF, et al. Artificial intelligence adoption challenges from healthcare providers’ perspectives: a comprehensive review and future directions. Saf Sci. 2026; 193: 107028. https://doi.org/10.1016/j.ssci.2025.107028

[14] Ramesh G, Sahil M, Palan SA, et al. A review on NLP zero-shot and few-shot learning: methods and applications. Discov Appl Sci. 2025; 7(9): 2025. https://doi.org/10.1007/s42452-025-07225-5

[15] Pan R, García-Díaz JA, Valencia-García R. comparing fine-tuning, zero and few-shot strategies with large language models in hate speech detection in English. CMES 2024; 140(3): 2849-68. https://doi.org/10.32604/cmes.2024.049631

[16] Branch RM. Instructional design:the ADDIE approach. New York: Springer; 2009. https://doi.org/10.1007/978-0-387-09506-6

[17] Yusoff MSB. ABC of content validation and content validity index calculation. Educ Med J. 2019; 11(2): 49-54. https://doi.org/10.21315/eimj2019.11.2.6

[18] Holmes W, Bialik M, Fadel C. Artificial intelligence in education:promises and implications for teaching and learning. Boston, MA, USA: Center for Curriculum Redesign; 2019.

[19] Kojima T, Gu SS, Reid M, et al. Large language models are zero-shot reasoners. arXiv preprint arXiv: 2205.11916. 2022. Available from: https://arxiv.org/abs/2205.11916

[20] Okechukwu C. Exploring the concepts of diversity, equity and inclusion in nursing curricula. J Nurs Educ Pract. 2024; 14(3): 1-7.