Assessing the utility of a natural language processing model in answering common urological questions

Abstract

Background: ChatGPT, an interactive natural language processing model, is becoming increasingly used for medical information gathering. The objective of this study is to investigate the appropriateness of ChatGPT's responses to common urological questions compared to the Canadian Urological Association (CUA) guideline recommendations.

Methods: A list of 10 urological questions were developed from the CUA guidelines and patient information materials. Each question was asked three times in ChatGPT (Version 4), totaling 30 ChatGPT‐generated responses. Responses were assessed by three reviewers using a 4‐point Likert scale (0–3) to score appropriateness with CUA guidelines as a reference. The median values and variance of answer scores were calculated to form a consensus score to determine model reliability.

Results: Forty percent (n = 12/30) of ChatGPT answers were deemed appropriate. The overall mean ± standard deviation score was 1.64 ± 0.85 (between “Some correct and some incorrect” and “Correct but inadequate”). When comparing question difficulty, “Easy” questions had an overall score of 1.87 ± 1.01, compared to “Medium” questions with a score of 1.31 ± 0.47. ChatGPT generated the most appropriate responses in the domains of prostate cancer (3.00 ± 0.25), erectile dysfunction (3.00 ± 0.28), andrology (3.00 ± 0.78), and kidney cancer (3.00 ± 1.00). The average variance between responses was 0.27, demonstrating strong model reliability.

Conclusion: This study examined the utility of ChatGPT in answering questions based on CUA clinical guidelines. Although promising, ChatGPT underperforms in answering common urological questions despite showing high levels of repeatability.

Graphical abstract

Keywords

ChatGPT / natural language model / patient education / patient information / urology

1 INTRODUCTION

The development of deep learning algorithms has led to the creation of chat generative pre‐trained transformer (ChatGPT), an interactive large language model (LLM) available for public use[1,2]. ChatGPT operates using neural network‐based deep learning to predict and generate text responses based on identified sequences and patterns from large collections of text data[1]. Due to its public access and ease of use, ChatGPT has extensive applications beyond traditional computational and research uses and is now a resource used by the public for education and information gathering[1,3].

ChatGPT has demonstrated accuracy in providing medical information in several medical and surgical areas, with a recent study demonstrating ChatGPT's accuracy in answering high‐level urological questions[3]. ChatGPT was able to answer multiple choice questions on two consecutive American Urological Association (AUA) Program exams with 30.0% and 42.3% accuracy, respectively[4,5]. Similarly, ChatGPT‐4.0 answered 44.4% of AUA multiple‐choice board questions correctly compared to ChatGPT‐3.5 scoring 30.9% demonstrating improvement in model knowledge over time[6].

Furthermore, ChatGPT has shown success in correctly answering 94.6% of questions about urolithiasis according to the European Association of Urology (EAU) guidelines[7]. Similarly, pediatric urology questions prepared according to the EAU guidelines were answered with 93.6% complete correctness by ChatGPT[8]. ChatGPT has also answered general urological guidelines‐based questions with 60% accuracy when comparing across Canadian, European, and American guidelines[9]. Despite this early success, most studies have focused on EAU and AUA guidelines as comparators, with limited data existing from the Canadian perspective[4,6–9]. The primary objective of this study is to investigate and quantify the appropriateness and reliability of ChatGPT's responses to common general urological questions when compared to the Canadian Urological Association (CUA) guidelines.

2 MATERIALS AND METHODS

2.1 Study design

A list of 10 common general urological questions in English were developed from a review of the CUA website and patient information materials[10]. Questions were included based on the areas of nephrolithiasis, prostate cancer, benign prostatic hyperplasia (BPH), erectile dysfunction (ED), overactive bladder (OAB), urinary tract infections (UTI), andrology, hypogonadism, pediatric urology, and kidney cancer (Table 1). Questions were formatted into plain language to emulate how a patient would interact with an online search engine. All questions were administered individually into ChatGPT version 4.0 (OpenAI, March 2, 2025 Version) using unique chat functions and responses were recorded[2]. ChatGPT (version 4, OpenAI) was used solely as the subject of analysis in this study. No AI‐assisted tools were used in the preparation or drafting of this manuscript. To account for potential variance in responses, each question was asked three times consecutively using the same text in new chat functions[21]. Evaluators included two practicing Canadian‐licensed urologists and one Canadian urology resident. Evaluators scored all 10 questions by perceived difficulty as either “Easy,” “Medium,” or “Hard.” Evaluators then scored all generated answers using a four‐point Likert scale (0 = Completely incorrect, 1 = Some correct and some incorrect, 2 = Correct but inadequate, and 3 = Comprehensive).

2.2 Statistical analysis

All data was imported into Statistical Package for Social Sciences (SPSS) Version 28 and analyzed using descriptive statistics and ANOVA[22]. Questions were assigned numerical values based on perceived difficulty by the reviewers (Easy = 1, Medium = 2, Hard = 3) and the average difficulty scores were used to assign question difficulty. Answer “appropriateness” was quantified by calculating the median Likert scale value for each topic. Scores greater than or equal to 2.00 on the Likert scale were used to define “appropriate” responses[8–10]. The variance between evaluator scale scores was calculated to measure ChatGPT's repeatability by question domain, in keeping with prior ChatGPT‐focused studies[8–10]. Model reliability was then assessed by calculating the average variance between all ChatGPT answers and the average variance amongst individual question domains[8–10]. Statistical significance was set at p < 0.05.

3 RESULTS

A total of 30 unique answers were generated by ChatGPT in response to the 10 urological questions (Figure 1). Questions were classified as “Easy” (60%, n = 6/10) and “Medium” (40%, n = 4/10) difficulty. When comparing the answer appropriateness based on question difficulty, ChatGPT performed better with “Easy” questions compared to “Medium” questions (1.87 ± 1.01 vs. 1.31 ± 0.47, p < 0.05).

Of the responses, 40% (n = 12/30) were found to be “appropriate,” with a score of ≥ 2.00 on the four‐point Likert scale. The overall mean ± standard deviation score was 1.64 ± 0.85 (between “Some correct and some incorrect” and “Correct but inadequate”). ChatGPT generated the most appropriate responses in the domains of prostate cancer (median [range]: 3.00 [0.00]), ED (3.00 [0.33]), andrology (3.00 [0.33]), and kidney cancer (3.00 [0.00]), out of a maximum score of 3.00 (Figure 2).

The average variance between ChatGPT answers was 0.27. The greatest variance in generated answers was noted in the domains of kidney cancer (σ² = 1.00) and andrology (σ² = 0.78). The lowest levels of variance were seen in the domains of pediatric urology (σ² = 0.001), hypogonadism (σ² = 0.001), OAB (σ² = 0.001), and nephrolithiasis (σ² = 0.001).

4 DISCUSSION

Since its release, ChatGPT has demonstrated extensive utility in academia and societal use[1]. With public access, ChatGPT has gained popularity as an alternative source of healthcare information, similar to how traditional search engines are used[23]. Therefore, assessing the appropriateness of the clinical information this program outputs is essential to optimizing patient‐centered education and care. Providing clinicians with a current understanding of the quality of information patients are receiving when querying artificial intelligence provides an important context for patient encounters[23].

In our study, ChatGPT was able to answer general urology answers with a 40% success rate in terms of providing an appropriate and comprehensive response. The average variance between ChatGPT answers was 0.27 demonstrating strong model reliability. This aligns with prior studies assessing the accuracy and completeness of ChatGPT, with a 52% correctness score when analyzing urology case‐based questions[24]. The urology‐question domains with the greatest appropriateness were prostate cancer, kidney cancer, ED, and andrology. Similarly, a study by Yudovich et al. found that both versions 3.5 and 4.0 of ChatGPT performed best on urologic oncology (33.9% and 43.8%, respectively) compared to other domains[6]. Despite these questions being assessed to be “Easy” difficulty, likely, the ChatGPT model performed best in these domains due to the more common nature of the topics and the objectivity in scoring these responses[25]. Prostate cancer testing and PSA surveillance, and options for small renal masses, represent common urologic problems that can be answered with online web searches[25]. Due to this, ChatGPT may be more experienced in generating the outputs to these questions. Similarly, these domains may have had higher scores due to the perceived difficulty of producing a comprehensive and complete answer to the question prompt. This is supported by “Easy” questions having higher scores compared to “Medium” questions. This observation is consistent with a study by Cocci et al., which demonstrated that ChatGPT was able to best provide appropriate answers to questions with definitive answers, with accuracy decreasing in questions involving summary or analysis[24,26].

Question domains in which ChatGPT underperformed included UTIs, OAB, nephrolithiasis, and hypogonadism, despite half of these questions being assessed to be “Easy” in difficulty. This may reflect the volume of inaccurate medical information on the internet on these topics or conflicting recommendations that exist in these areas when compared to Canadian recommendations. Additionally, despite ChatGPT demonstrating promise for answering urologic questions, the literature has focused primarily on European and American guidelines[4,6–9]. This may explain discrepancies in ChatGPT accuracy in this study when evaluating using Canadian guidelines as a reference.

While research suggests that ChatGPT offers promising clinical utility, it must be noted that ChatGPT information may include incomplete, outdated, and biased knowledge reproduced from its training data. Additionally, there are several additional known limitations of the program, including the inability to use real‐time information, answer ambiguous questions, and inconsistency in quality[1]. These limitations inherently would affect how a patient would use ChatGPT to seek medical information. This is aligned with the underlying theme of false information or omission of information in responses consistent across studies[23,27–30].

Although this study serves as one of the few studies to evaluate the use of ChatGPT using a Canadian context, limitations exist. First, ChatGPT responses were assessed subjectively using CUA guidelines as reference material. While this is the optimal way of assessing the clinical utility of these responses, review by additional trained specialists for appropriateness would further increase the statistical power of this study's findings. Furthermore, potential underperformance may be related to guideline discordance. Second, CUA guidelines used in this study may be outdated compared to the ChatGPT information used to respond to questions. Despite CUA guidelines being used as reference for clinical practice and are available for public use, the most recently updated guidelines used in this study were from 2022. Therefore, responses may have been marked as less appropriate compared to guidelines despite having more up‐to‐date information. Additionally, references were not provided or collected with ChatGPT responses. While this does not affect the content of the responses, tracking where the information ChatGPT uses to create responses would be beneficial in assessing whether CUA guidelines are considered in the ChatGPT model. This is a limitation intrinsic to ChatGPT in which references are not able to be provided. Furthermore, as the focus of this study was to examine the accuracy and comprehensiveness of ChatGPT in answering general urology questions, extensive questions were not developed and assessed in each urologic domain. It is possible that with further testing across multiple domains, ChatGPT may perform more optimally in certain content areas. Therefore, generalizability of the results must be made with caution. Additionally, model responses may improve if the complexity and specificity of the prompt was increased, although this may reduce study applicability to the general public. Only “Easy” and “Medium” difficulty questions were evaluated in this study based on evaluator review, therefore for more complex or difficult questions, the comprehensiveness and appropriateness of ChatGPT's responses may decline. Studies evaluating ChatGPT output should strive to classify question prompts by perceived difficulty and consider this factor when interpreting the results of LLM‐based research. Furthermore, only ChatGPT was assessed in this study, whereas future research may also compare the clinical utility of ChatGPT to other language processing models to determine superiority in clinical appropriateness. ChatGPT performance may improve in future studies with development of new models, which may improve appropriateness of results. Finally, this study posed questions exclusively in English. Future studies may investigate the appropriateness of ChatGPT responses in multiple languages to assess differences and better serve patients who do not speak/read English primarily.

Opinions from urologists worldwide were collected, revealing that while it has been largely accepted that programs like ChatGPT will inevitably play a role in future healthcare delivery, many urologists are concerned about the ethical utility of the program and patient acceptance[27–30]. This consideration, in conjunction with the information found in this study and others, suggests that urologists must play a key role in educating patients about the extent to which ChatGPT can be used and trusted to provide patient‐appropriate information. Whiles et al. recommended that ChatGPT implement a mandatory disclaimer to all medical‐related generated responses to inform users of potentially inappropriate medical advice[9]. Additionally, it is recommended that urologists advocate for accurate training of LLMs, such as ChatGPT, with appropriate clinical urology guidelines to ensure they inform patients appropriately and expand their effectiveness[27–30]. Based on our findings, urology clinicians should be aware that patients using ChatGPT for medical advice may be receiving incorrect and/or inadequate information.

5 CONCLUSIONS

ChatGPT produced appropriate responses for 40% of general urology questions, with higher levels of accuracy for prostate cancer, andrology, ED, and kidney cancer domains when using CUA guidelines as a reference. While this is promising for the future expanded clinical application of this LLM, further advancement is needed before widespread reliable use by providers and patients.

Urologists and other healthcare providers should continue to educate patients about the limitations of ChatGPT to avoid inappropriate use. With continued advancement, LLMs, like ChatGPT, will have increasing potential as an information resource for patient care.

References

[1] Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. IOTCPS. 2023;3:121–154.

[2] ChatGPT. 2023 [cited 2025 Mar 2]. Available from OpenAI website.

[3] Biswas SS. Role of Chat GPT in public health. Ann Biomed Eng. 2023;51:868–869.

[4] Deebel NA, Terlecki R. ChatGPT performance on the American Urological Association self‐assessment study program and the potential influence of artificial intelligence in urologic training. Urology. 2023;177:29–33.

[5] Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: potential for AI‐assisted medical education using large language models. PLOS Digital Health. 2023;2:e0000198.

[6] Yudovich MS, Makarova E, Hague CM, Raman JD. Performance of GPT‐3.5 and GPT‐4 on standardized urology knowledge assessment items in the United States: a descriptive study. J Educ Eval Health Prof. 2024;21:17.

[7] Cakir H, Caglar U, Yildiz O, Meric A, Ayranci A, Ozgor F. Evaluating the performance of ChatGPT in answering questions related to urolithiasis. Int Urol Nephrol. 2023;56:17–21.

[8] Caglar U, Yildiz O, Meric A, Ayranci A, Gelmis M, Sarilar O, et al. Evaluating the performance of ChatGPT in answering questions related to pediatric urology. J Pediatr Urol. 2024;20:1–5.

[9] Whiles BB, Bird VG, Canales BK, DiBianco JM, Terry RS. Caution! AI bot has entered the patient chat: ChatGPT has limitations in providing accurate urologic healthcare advice. Urology. 2023;180:278–284.

[10] Guidelines—Canadian Urological Association. 2025 [cited 2025 Mar 2]. Available from the website of The Canadian Urological Association (CUA).

[11] Bhojani N, Bjazevic J, Wallace B, Lee L, Kaler KS, Dion M, et al. Update—2022 Canadian Urological Association guideline: evaluation and medical management of the kidney stone patient. Can Urol Assoc J. 2022;16:175–188.

[12] Mason RJ, Marzouk K, Finelli A, Saad F, So AI, Violette PD, et al. UPDATE—2022 Canadian Urological Association recommendations on prostate cancer screening and early diagnosis: endorsement of the 2021 Cancer Care Ontario guidelines on prostate multiparametric magnetic resonance imaging. Can Urol Assoc J. 2022;16:E184–E196.

[13] Elterman D, Aubé‐Peterkin M, Evans H, Elmansy H, Meskawi M, Zorn KC, et al. UPDATE—2022 Canadian Urological Association guideline on male lower urinary tract symptoms/benign prostatic hyperplasia (MLUTS/BPH). Can Urol Assoc J. 2022;16:245–256.

[14] Domes T, Najafabadi BT, Roberts M, Campbell J, Flannigan R, Bach P, et al. Canadian Urological Association guideline: erectile dysfunction. Can Urol Assoc J. 2021;15:310–322.

[15] Corcos J, Przydacz M, Campeau L, Gray G, Hickling D, Honeine C, et al. CUA guideline on adult overactive bladder. Can Urol Assoc J. 2017;11:142–173.

[16] Anger J, Lee U, Ackerman AL, Chou R, Chughtai B, Clemens JQ, et al. Recurrent uncomplicated urinary tract infections in women: AUA/CUA/SUFU Guideline. J Urol. 2019;202:282–289.

[17] Zini A, Grantmyre J, Chow V, Chan P. UPDATE—2022 Canadian Urological Association best practice report: vasectomy. Can Urol Assoc J. 2022;16:E231–E236.

[18] Grober ED, Krakowsky Y, Khera M, Holmes DT, Lee JC, Grantmyre JE, et al. Canadian Urological Association guideline on testosterone deficiency in men: evidence‐based Q&A. Can Urol Assoc J. 2021;15:234–243.

[19] Braga LH, Lorenzo AJ, Romao RLP. Canadian Urological Association‐Pediatric Urologists of Canada (CUA‐PUC) guideline for the diagnosis, management, and followup of cryptorchidism. Can Urol Assoc J. 2017;11:E251–E260.

[20] Richard PO, Violette PD, Bhindi B, Breau RH, Kassouf W, Lavallée LT, et al. Canadian Urological Association guideline: management of small renal masses. Can Urol Assoc J. 2022;16:E61–E75.

[21] Sarraju A, Bruemmer D, Van Iterson E, Cho L, Rodriguez F, Laffin L. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat‐based artificial intelligence model. JAMA. 2023;329:842–844.

[22] IBM SPSS Statistics. 2021 [cited 2025 Mar 2]. Available from IBM website.

[23] Davis R, Eppler M, Ayo‐Ajibola O, Loh‐Doyle JC, Nabhani J, Samplaski M, et al. Evaluating the effectiveness of artificial intelligence‐powered large language models application in disseminating appropriate and readable health information in urology. J Urol. 2023;210:688–694.

[24] Cocci A, Pezzoli M, Lo Re M, Russo GI, Asmundo MG, Fode M, et al. Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate Cancer Prostatic Dis. 2023;27:103–108.

[25] Hellawell GO, Turner KJ, Le Monnier KJ, Brewster SF. Urology and the Internet: an evaluation of Internet use by urology patients and of information available on urological topics. BJU Int. 2000;86:191–194.

[26] Schoch J, Schmelz HU, Strauch A, Borgmann H, Nestler T. Performance of ChatGPT‐3.5 and ChatGPT‐4 on the European Board of Urology (EBU) exams: a comparative analysis. World J Urol. 2024;42:445.

[27] Szczesniewski JJ, Tellez Fouz C, Ramos Alba A, Diaz Goizueta FJ, García Tello A, Llanes González L. ChatGPT and most frequent urological diseases: analysing the quality of information and potential risks for patients. World J Urol. 2023;41:3149–3153.

[28] Zhu L, Mou W, Chen R. Can the ChatGPT and other large language models with internet‐connected database solve the questions and concerns of patient with prostate cancer and help democratize medical knowledge? J Transl Med. 2023;21:269.

[29] Nedbal C, Naik N, Castellani D, Gauhar V, Geraghty R, Somani BK. ChatGPT in urology practice: revolutionizing efficiency and patient care with generative artificial intelligence. Curr Opin Urol. 2024;34:98–104.

[30] Eppler M, Ganjavi C, Ramacciotti LS, Piazza P, Rodler S, Checcucci E, et al. Awareness and use of ChatGPT and large language models: A prospective cross‐sectional global survey in urology. Eur Urol. 2024;85:146–153.

Rights & permissions

2025 The Author(s). UroPrecision published by John Wiley & Sons Australia, Ltd on behalf of Higher Education Press.