Artificial intelligence versus humans: A comparative analysis of time, cost, and performance on a clinical code conversion task

Abstract

Healthcare services generate and store large quantities of data, requiring significant resources to manually analyze and gain meaningful insights. Recent advancements in automation tools—such as generative artificial intelligence (GenAI)—provide new opportunities to reduce human labor. This study explores the potential utilization of GenAI for a healthcare data analysis task—specifically, the conversion of clinical data from one diagnostic classification system to another (i.e., the Australian extension of the Systematized Nomenclature of Medicine Clinical Terms to the International Classification of Diseases, 10th Revision, Clinical Modification)—and examines the time and cost benefits of performing this using GenAI compared to a human rater. Conversions were completed using three methods: manual conversion using the National Library of Medicine’s I-MAGIC tool, ChatGPT-4o, and Claude 3.5 Sonnet. The accuracy of the GenAI tools was mapped against the manually extracted codes and examined in terms of a perfect, partial, or incorrect match. Task completion time was recorded and extrapolated to calculate and compare the cost associated with each method. When compared to the manually extracted codes, Claude 3.5 Sonnet yielded the highest level of agreement over ChatGPT-4o, whilst being the most time- and cost-effective. GenAI tools have greater utility than they have currently been given credit for. The automation of big data healthcare analytics, whilst still the domain of humans, is increasingly capable of being undertaken using automation tools with low barriers to entry. The further development of GenAI’s capabilities, alongside the capability of the healthcare system to use it appropriately, has the potential to result in significant resource savings.

Keywords

Data analytics / Diagnostic coding / Generative artificial intelligence / International Classification of Diseases 10^th revision / Systematized Nomenclature of Medicine Clinical Terms / SNOMED

Funding

This study was supported by an Australian Government Research Training Program Scholarship.

Conflict of interest

The authors declare that they have no competing interests.

References

[1] Murphy K. How data will improve healthcare without adding staff or beds. Cornell university, institut européen d’administration des affaires. In: Organisation WIP, editors. Global Innovation Index 2019. Ch. 8. Geneva: World Intellectual Property Organization; 2019.

[2] Dash S, Shakyawar SK, Sharma M, Kaushik S. Big data in healthcare: Management, analysis and future prospects. J Big Data. 2019; 6(1):54. doi: 10.1186/s40537-019-0217-0

[3] Australian Medical Association. 2024 Public Hopsital Report Card; 2024.

[4] Feuerriegel S, Hartmann J, Janiesch C, Zschech P. Generative AI. Bus Inform Syst Eng. 2024; 66(1):111-126. doi: 10.1007/s12599-023-00834-7

[5] Oluwagbenro MB. Generative AI: Definition, Concepts, Applications, and Future Prospects. Authorea Preprints; 2024.

[6] Banh L, Strobel G. Generative artificial intelligence. Electron Mark. 2023; 33(1):63. doi: 10.1007/s12525-023-00680-1

[7] Fui-Hoon Nah F, Zheng R, Cai J, Siau K, Chen L. Generative AI and chatGPT: Applications, challenges, and AI-human collaboration. J Inform Technol Case Appl Res. 2023; 25:277-304. doi: 10.1080/15228053.2023.2233814

[8] Chatterjee S, Fruhling A, Kotiadis K, Gartner D. Towards new frontiers of healthcare systems research using artificial intelligence and generative AI. Health Syst (Basingstoke). 2024; 13:263-273. doi: 10.1080/20476965.2024.2402128

[9] Yang Y, Ngai EW, Wang L. Resistance to artificial intelligence in health care: Literature review, conceptual framework, and research agenda. Inform Manage. 2024; 61:103961. doi: 10.1016/j.im.2024.103961

[10] Amin SM, El-Gazar HE, Zoromba MA, El-Sayed MM, Atta MHR. Sentiment of nurses towards artificial intelligence and resistance to change in healthcare organisations: A mixed-method study. J Adv Nurs. 2025; 81(4):2087-2098. doi: 10.1111/jan.16435

[11] Chaibi A, Zaiem I. Doctor resistance of artificial intelligence in healthcare. Int J Healthcare Inform Syst Inform (IJHISI). 2022; 17(1):1-13. doi: 10.4018/IJHISI.315618

[12] Sobaih AEE, Chaibi A, Brini R, Abdelghani Ibrahim TM. Unlocking patient resistance to AI in healthcare: A psychological exploration. Eur J Investig Health Psychol Educ. 2025; 15(1):6. doi: 10.3390/ejihpe15010006.

[13] Li J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: A taxonomy and systematic review. Comput Methods Programs Biomed. 2024; 245:108013. doi: 10.1016/j.cmpb.2024.108013

[14] Patel SB, Lam K. ChatGPT: The future of discharge summaries? Lancet Digit Health. 2023; 5(3):e107-e108. doi: 10.1016/S2589-7500(23)00021-3

[15] Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD. Evaluating ChatGPT as an adjunct for radiologic decision-making. MedRxiv. 2023. doi: 10.1101/2023.02.02.23285399

[16] Ahn C. Exploring ChatGPT for information of cardiopulmonary resuscitation. Resuscitation. 2023; 185:109729. doi: 10.1016/j.resuscitation.2023.109729

[17] Boßelmann CM, Leu C, Lal D. Are AI language models such as ChatGPT ready to improve the care of individuals with epilepsy? Epilepsia. 2023; 64(5):1195-1199. doi: 10.1111/epi.17570.

[18] Mogali SR. Initial impressions of ChatGPT for anatomy education. Anat Sci Educ. 2024; 17(2):444-447. doi: 10.1002/ase.2261

[19] Sng GGR, Tung JYM, Lim DYZ, Bee YM. Potential and pitfalls of ChatGPT and natural-language artificial intelligence models for diabetes education. Diabetes Care. 2023; 46(5):e103-e105. doi: 10.2337/dc23-0197

[20] Health-Europe TLR. Embracing generative AI in health care. Lancet Reg Health Eur. 2023; 30:100677. doi: 10.1016/j.lanepe.2023.100677.

[21] Open AI. ChatGPT. Available from: https://chat.openai.com [Last accessed on 2024 Sep 06].

[22] Thorne E. ChatGPT Hits 100M Weekly Users. LinkedIn News. Available from: https://www.linkedin.com/news/story/chatgpt-hits-100m-weekly-users-5808204 [Last accessed on 2024 Sep 09].

[23] Thakur K, Barker HG, Khan Pathan AS. Artificial Intelligence and Large Language Models: An Introduction to the Technological Future. United States: CRC Press, LLC; 2024.

[24] Google AI. Gemini. Available from: https://gemini.google.com [Last accessed on 2024 Sep 09].

[25] Microsoft. Copilot. Available from: https://copilot.microsoft.com [Last accessed on 2024 Sep 09].

[26] Anthropic. Claude.AI. Available from: https://claude.ai [Last accessed on 2024 Sep 06].

[27] Waldo J, Boussard S. GPTs and hallucination: Why do large language models hallucinate? Queue. 2024; 22(4):19-33. doi: 10.1145/3688007

[28] Snomed International. What is Snomed CT? Available from: https://www.snomed.org/what-is-snomed-ct [Last accessed on 2024 Sep 09].

[29] Snomed International. Members. Available from: https://www.snomed.org/members [Last accessed on 2024 Sep 16].

[30] World Health Organisation. International Statistical Classification of Diseases and Related Health Problems (ICD). Available from: https://www.who.int/standards/classifications/classification-of-diseases [Last accessed on 2024 Sep 09].

[31] National Library of Medicine. Snomed-CT Browser. Available from: https://www.nlm.nih.gov/research/umls/snomed/snomed-browsers.html [Last accessed on 2025 Jun 09].

[32] World Health Organisation. International Statistical Classification of Diseases and Related Health Problems 10^thRevision. Available from: https://icd.who.int/browse10/2019/en [Last accessed on 2025 Jun 09].

[33] Sallam M, Barakat M, Sallam M. A preliminary checklist (METRICS) to standardize the design and reporting of studies on generative artificial intelligence-based models in health care education and practice: Development study involving a literature review. Interact J Med Res. 2024; 13:e54704. doi: 10.2196/54704

[34] National Library of Medicine. Snomed CT to ICD-10-cm Map. Available from: https://www.nlm.nih.gov/research/umls/mapping-projects/snomedct-to-icd10cm.html [Last accessed on 2024 Sep 06].

[35] National Clinical Terminology Service. What is Snomed CT and the AMT? Available from: https://www.healthterminologies.gov.au/understanding-clinical-terminology-landing/what-is-snomed-ct-and-amt [Last accessed on 2024 Sep 23].

[36] Preibisch S. API Development. Germany: Springer; 2018.

[37] Hangzhou DeepSeek Artificial Intelligence Co. DeepSeek. Available from: https://www.deepseek.com [Last accessed on 2025 Jun 09].