Analyzing the precision and readability of a healthcare focused artificial intelligence platform on common questions regarding breast augmentation

Abstract

Aim: The purpose of this study was to determine the quality and accessibility of the outputs from a healthcare-specific artificial intelligence (AI) platform for common questions during the perioperative period for a common plastic surgery procedure.

Methods: Doximity GPT (Doximity, San Francisco, CA) and ChatGPT 3.5 (OpenAI, San Francisco, CA) were utilized to search 20 common perioperative patient inquiries regarding breast augmentation. The structure, content, and readability of responses were compared using t-tests and chi-square tests, with P < 0.05 used as the cutoff for significance.

Results: Out of 80 total AI-generated outputs, ChatGPT responses were significantly longer (331 vs. 218 words, P < 0.001). Doximity GPT outputs were structured as a letter from a medical provider to the patient, whereas ChatGPT outputs were a bulleted list. Doximity GPT outputs were significantly more readable by four validated scales: Flesch Kincaid Reading Ease (42.6 vs. 29.9, P < 0.001) and Flesch Kincaid Grade Level (11.4 vs. 14.1 grade, P < 0.001), Coleman-Liau Index (14.9 vs. 17 grade, P < 0.001), and Automated Readability Index (11.3 vs. 14.8 grade, P < 0.001). Regarding content, there was no difference between the two platforms regarding the appropriateness of the topic (99% overall). Medical advice from all outputs was deemed reasonable.

Conclusion: Doximity’s AI platform produces reasonable, accurate information in response to common patient queries. With continued reinforcement learning with human feedback (RLHF), Doximity GPT has the potential to be a useful tool to plastic surgeons and can assist with a range of tasks, such as providing basic information on procedures and writing appeal letters to insurance providers.

Keywords

Artificial intelligence / natural language processing / ChatGPT / AI / generative AI / plastic surgery / AI integration in surgery

References

[1] Bogdanovich B,Patel PA,Boyd CJ.Altmetric analysis of artificial intelligence articles in plastic surgery.Arch Plast Surg2024;51:262-4 PMCID:PMC11001444

[2] Bui T,Boyd CJ.Altmetric analysis of the online attention directed to artificial intelligence literature in ophthalmology.Asia Pac J Ophthalmol2023;12:625-6

[3] Kavian JA,Patel PA.Harvesting the power of artificial intelligence for surgery: uses, implications, and ethical considerations.Am Surg2023;89:5102-4

[4] Zhu C,Wirth PJ,Friedrich JB.Current applications of artificial intelligence in billing practices and clinical plastic surgery.Plast Reconstr Surg Glob Open2024;12:e5939 PMCID:PMC11216662

[5] Bogdanovich B,Kavian JA,Rodriguez ED.ChatGPT for the modern plastic surgeon.Plast Reconstr Surg2023;152:969e-70e

[6] Boyd CJ,Sorenson TJ.Artificial intelligence as a triage tool during the perioperative period: pilot study of accuracy and accessibility for clinical application.Plast Reconstr Surg Glob Open2024;12:e5580 PMCID:PMC10836902

[7] Blount T,Fakhre F.Readability of online materials in Spanish and English for breast reduction insurance coverage.Aesth Plast Surg2024;48:1436-43

[8] Bruce JC,Van Spronsen NR,Bharadia D.Analysis of online materials regarding DIEP and TRAM flap autologous breast reconstruction.J Plast Reconstr Aesthet Surg2023;82:81-91

[9] Soliman L,Gallo Marin B,Woo AS.Craniosynostosis: are online resources readable?.Cleft Palate Craniofac J2024;61:1228-32

[10] Tiourin E,Janis JE.Health literacy in plastic surgery: a scoping review.Plast Reconstr Surg Glob Open2022;10:e4247 PMCID:PMC9007188

[11] Vallurupalli M,Vyas RM.Optimizing readability of patient-facing hand surgery education materials using chat generative pretrained transformer (ChatGPT) 3.5.J Hand Surg Am2024;49:986-91

[12] Lim B,Cuomo R.Can AI answer my questions? Utilizing artificial intelligence in the perioperative assessment for abdominoplasty patients. Aesthetic Plast Surg 2024.

[13] Gomez-Cabello CA,Pressman SM.Artificial intelligence in postoperative care: assessing large language models for patient recommendations in plastic surgery.Healthcare2024;12:1083 PMCID:PMC11171524

[14] Bekisz JM,Salibian AA,Karp NS.Aesthetic characteristics of the ideal female breast.Plast Reconstr Surg Glob Open2023;11:e4770 PMCID:PMC9857454

[15] Khan S,Walker AM.The readability of online patient education materials on maxillomandibular advancement surgery.Sleep Breath2024;28:745-51

[16] Browne R,Carr S.Online resources for robin sequence; an analysis of readability.Cleft Palate Craniofac J2024;10556656241234587

[17] Wasserburg JR,Sanati-Mehrizy P,Taub PJ.Cleft care readability: can patients access helpful online resources?.Cleft Palate Craniofac J2021;58:1287-93

[18] Garg N,Yang A.Chatbots as patient education resources for aesthetic facial plastic surgery: evaluation of ChatGPT and Google Bard responses. Facial Plast Surg Aesthet Med 2024.

[19] Alharbi AA,Alsuhaibani KA,Alharbi MA.Perception of primary health care providers of plastic surgery and its influence on referral.J Family Med Prim Care2019;8:225-30 PMCID:PMC6396623

[20] McGoldrick C.Does plastic surgery have an image problem?: the perception of plastic surgery in an era of general practitioner commissioning.J Plast Reconstr Aesthet Surg2013;66:1635-6

[21] DiDonna N,Khan K.Unveiling the potential of AI in plastic surgery education: a comparative study of leading AI platforms’ performance on in-training examinations.Plast Reconstr Surg Glob Open2024;12:e5929 PMCID:PMC11191997

[22] Shah P,Patel PA.Assessing the plastic surgery knowledge of three natural language processor artificial intelligence programs.J Plast Reconstr Aesthet Surg2024;88:193-5

[23] Long C,Lowe K.ChatENT: augmented large language model for expert knowledge retrieval in otolaryngology-head and neck surgery.Otolaryngol Head Neck Surg2024;171:1042-51

[24] Awad SK,Patel J.Plastic surgeons are underrepresented when searching hospital websites for a hand surgeon.Plast Reconstr Surg2023;151:1055e-8e

[25] Singh NP,Aluri A.One in three chance of finding a plastic surgeon on major hospital websites.Plast Reconstr Surg Glob Open2023;11:e4781 PMCID:PMC9872967

[26] Boudreau H,Boyd CJ.Understanding the impact of social media information and misinformation producers on health information seeking. Comment on “Health information seeking behaviors on social media during the COVID-19 pandemic among american social networking site users: survey study”.J Med Internet Res2022;24:e31415 PMCID:PMC8857697

[27] Dhar S,Vasquez M.The utility and accuracy of ChatGPT in providing post-operative instructions following tonsillectomy: a pilot study.Int J Pediatr Otorhinolaryngol2024;179:111901

[28] Polat E,Senturk E.Evaluating the accuracy and readability of ChatGPT in providing parental guidance for adenoidectomy, tonsillectomy, and ventilation tube insertion surgery.Int J Pediatr Otorhinolaryngol2024;181:111998

[29] Kenig N,Rubi C.Ethics for AI in plastic surgery: guidelines and review.Aesthetic Plast Surg2024;48:2204-9

[30] Liu HY,Arellano JA.Can ChatGPT be the plastic surgeon’s new digital assistant? A bibliometric analysis and scoping review of ChatGPT in plastic surgery literature.Aesthetic Plast Surg2024;48:1644-52