Assessing the accuracy and utility of ChatGPT responses to patient questions regarding posterior lumbar decompression

Abstract

Aim: To examine the clinical accuracy and applicability of ChatGPT answers to commonly asked questions from patients considering posterior lumbar decompression (PLD).

Methods: A literature review was conducted to identify 10 questions that encompass some of the most common questions and concerns patients may have regarding lumbar decompression surgery. The selected questions were then posed to ChatGPT. Initial responses were then recorded, and no follow-up or clarifying questions were permitted. Two attending fellowship-trained spine surgeons then graded each response from the chatbot using a modified Global Quality Scale to evaluate ChatGPT’s accuracy and utility. The surgeons then analyzed each question, providing evidence-based justifications for the scores.

Results: Minimum scores across all ten questions would lead to a total score of 20, whereas a maximum score would be 100. ChatGPT’s responses in this analysis earned a score of 59, just under an average score of 3, when evaluated by two attending spine surgeons. A score of 3 denoted a somewhat useful response of moderate quality, with some important information adequately discussed but some poorly discussed.

Conclusion: ChatGPT has the ability to provide broadly useful responses to common preoperative questions that patients may have when considering undergoing PLD. ChatGPT has excellent utility in providing background information to patients and in helping them become more informed about their pathology in general. However, it often lacks the specific patient context necessary to provide patients with personalized, accurate insights into their prognosis and medical options.

Keywords

Artificial intelligence / ChatGPT / lumbar decompression / spine surgery

References

[1] Van Riel N, Auwerx K, Debbaut P, Van Hees S, Schoenmakers B. The effect of Dr Google on doctor-patient encounters in primary care: a quantitative, observational, cross-sectional study.BJGP Open2017;1:bjgpopen17X100833 PMCID:PMC6169945

[2] Cocco AM,Taylor DM.Dr Google in the ED: searching for online health information by adult emergency department patients.Med J Aust2018;209:342-7

[3] Fraval A,Holcdorf D,Tran P.Internet use by orthopaedic outpatients - current trends and practices.Australas Med J2012;5:633-8 PMCID:PMC3561591

[4] Kasthuri V,Alsoof D.Modern internet search analytics and spine: what are patients asking and reading online?.N Am Spine Soc J2023;14:100214 PMCID:PMC10192655

[5] Dave T,Singh S.ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations.Front Artif Intell2023;6:1169595 PMCID:PMC10192861

[6] Sallam M.ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns.Healthcare2023;11:887 PMCID:PMC10048148

[7] Cascella M,Bellini V.Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios.J Med Syst2023;47:33 PMCID:PMC9985086

[8] OpenAI. Introducing ChatGPT. 2022. Available from: https://openai.com/blog/chatgpt. [Last accessed on 27 Aug 2024]

[9] Lock S. What is AI chatbot phenomenon ChatGPT and could it replace humans? 2022. Available from: https://www.theguardian.com/technology/2022/dec/05/what-is-ai-chatbot-phenomenon-chatgpt-and-could-it-replace-humans. [Last accessed on 27 Aug 2024]

[10] Gilson A,Huang T.How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment.JMIR Med Educ2023;9:e45312 PMCID:PMC9947764

[11] Kaarre J,Keeling LE.Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information.Knee Surg Sports Traumatol Arthrosc2023;31:5190-8 PMCID:PMC10598178

[12] Dubin JA,Chen Z.Using a Google web search analysis to assess the utility of ChatGPT in total joint arthroplasty.J Arthroplasty2023;38:1195-202

[13] Duey AH,Zaidat B.Thromboembolic prophylaxis in spine surgery: an analysis of ChatGPT recommendations.Spine J2023;23:1684-91

[14] Fayed AM,de Carvalho KA,D’Hooghe P.Artificial intelligence and ChatGPT in orthopaedics and sports medicine.J Exp Orthop2023;10:74 PMCID:PMC10371934

[15] Hodakowski AJ,Damodar D.Rotator cuff repair: what questions are patients asking online and where are they getting their answers?.Clin Shoulder Elb2023;26:25-31 PMCID:PMC10030981

[16] Hurley ET,Lorentz SG.Evaluation high-quality of information from ChatGPT (artificial intelligence-large language model) artificial intelligence on shoulder stabilization surgery.Arthroscopy2024;40:726-31.e6

[17] Mika AP,Engstrom SM,Wilson JM.Assessing ChatGPT responses to common patient questions regarding total hip arthroplasty.J Bone Joint Surg Am2023;105:1519-26

[18] Subramanian T,Araghi K.Using artificial intelligence to answer common patient-focused questions in minimally invasive spine surgery.J Bone Joint Surg Am2023;105:1649-53

[19] Lattig F,OʼRiordan D.A comparison of patient and surgeon preoperative expectations of spinal surgery.Spine2013;38:1040-8

[20] Deyo RA,Martin BI,Goodman DC.Trends, major medical complications, and charges associated with surgery for lumbar spinal stenosis in older adults.JAMA2010;303:1259-65 PMCID:PMC2885954

[21] O’Lynnger TM,Morone PJ,Vasquez-Castellanos RA.Trends for spine surgery for the elderly: implications for access to healthcare in North America.Neurosurgery2015;77 Suppl 4:S136-41

[22] Bernard A,Hughes S,Leddin D.A systematic review of patient inflammatory bowel disease information resources on the World Wide Web.Am J Gastroenterol2007;102:2070-7

[23] Kreiner DS, Hwang SW, Easa JE, et al; North American Spine Society. An evidence-based clinical guideline for the diagnosis and treatment of lumbar disc herniation with radiculopathy. Spine J 2014;14:180-91.

[24] Zaina F,Carragee E.Surgical versus non-surgical treatment for lumbar spinal stenosis.Cochrane Database Syst Rev2016;2016:CD010264 PMCID:PMC6669253

[25] Katz JN,Mass H.Diagnosis and management of lumbar spinal stenosis: a review.JAMA2022;327:1688-99

[26] Kuris EO,Palumbo MA.Evaluation and management of cauda equina syndrome.Am J Med2021;134:1483-9

[27] Bulloch L,Spector L.Cauda equina syndrome.Orthop Clin North Am2022;53:247-54

[28] Issack PS,Pumberger M,Cammisa FP Jr.Degenerative lumbar spinal stenosis: evaluation and management.J Am Acad Orthop Surg2012;20:527-35

[29] Fritz JM,Welch WC.Lumbar spinal stenosis: a review of current concepts in evaluation, management, and outcome measurements.Arch Phys Med Rehabil1998;79:700-8

[30] Goacher E,Ivanov M.Safety and feasibility of same-day discharge following lumbar decompression surgery: a systematic review.Brain Spine2022;2:100888 PMCID:PMC9559968

[31] Degen T,Theiler R.Outcomes after spinal stenosis surgery by type of surgery in adults aged 60 years and older.Swiss Med Wkly2020;150:w20325

[32] Matsumoto K,Kelkar A.Biomechanical evaluation of a novel decompression surgery: transforaminal full-endoscopic lateral recess decompression (TE-LRD).N Am Spine Soc J2021;5:100045 PMCID:PMC8819954

[33] Hathi K,Richardson E.Minimally invasive vs. open surgery for lumbar spinal stenosis in patients with diabetes - a Canadian spine outcomes and research network study.Global Spine J2023;13:1602-11 PMCID:PMC10448101

[34] Goldstein CL,Sundararajan K.Perioperative outcomes and adverse events of minimally invasive versus open posterior lumbar fusion: meta-analysis and systematic review.J Neurosurg Spine2016;24:416-27

[35] Skovrlj B,Zarzour H.Perioperative outcomes in minimally invasive lumbar spine surgery: a systematic review.World J Orthop2015;6:996-1005 PMCID:PMC4686448

[36] Chalhoub R,Aoun M.Will ChatGPT be able to replace a spine surgeon in the clinical setting?.World Neurosurg2024;185:e648-52

[37] Mejia MR,Saturno M.Use of ChatGPT for determining clinical and surgical treatment of lumbar disc herniation with radiculopathy: a North American spine society guideline comparison.Neurospine2024;21:149-58 PMCID:PMC10992643

[38] Rajjoub R,Zaidat B.ChatGPT and its role in the decision-making for the diagnosis and treatment of lumbar spinal stenosis: a comparative analysis and narrative review.Global Spine J2024;14:998-1017 PMCID:PMC11192138