A comparative performance analysis of live clinical triage using rules-based triage protocols versus artificial intelligence-based automated virtual triage

Abstract

Objective: Compare the triage care referral accuracy of artificial intelligence (AI)-based virtual triage (VT) to rules-based triage protocols (RBTP) live telephonic triage.
Methods: Clinical vignettes were selected for a comparison of care referral accuracy of RBTPs with a widely utilized AI-based VT solution. Vignettes (149) included patient complaints, expected triage and urgency assessment. Triage levels were mapped to three triage categories (urgent care, non-emergent care and self-care). Each vignette was evaluated/completed using AI-based VT and RBTP triage modalities by a total of four physicians in series, with independent assessment for errors and inconsistencies. Triage assessment precision was analyzed by matching the expected triage assessment, sensitivity and F1 scores (harmonic mean of precision and recall).
Results: Both modalities achieved > 70% triage accuracy, and safety performance was identical at 91%. AI-based VT was more accurate in care referral for emergency and non-emergency care and overtriaged to emergency care 50% less frequently than RBTP, but was less accurate than RBTP in self-care vignettes (neither statistically significant). Both modalities demonstrated decreased sensitivity as care urgency/acuity decreased, more pronounced in AI-based VT than RBTP. AI-based VT captured four times as much information and data as RBTP.
Conclusions: AI-based VT and RBTP were comparable in care referral accuracy and disposition safety. While AI-based VT provides accurate and safe triage recommendations at a lower total cost, care organizations should assess how AI-based VT compares to a live clinical triage capability with respect to organizational priorities, budgetary considerations, characteristics of the patient/member population served, and the existing technological environment.

Keywords

Virtual triage / Digital clinical triage / AI / Symptom checker / Rules-based triage protocols or pathways

ACKNOWLEDGEMENTS

The authors are grateful to Forson Chan, MD; Amanda Singh, MD; and Jocelyn Peters, MD at the Schulich School of Medicine & Dentistry at Western University in London, Ontario for their assistance.

AUTHORS CONTRIBUTIONS

KK, NJ, JJ, TP and GAG were involved in the study design, data analysis and presentation, and all participated in written manuscript preparation. PMO provided oversight and direction to the research and writing team.

ETHICAL STATEMENT

This analysis was not based on an experimental design utilizing human subjects and none were involved in completing this study. No formal ethical committee review was needed or pursued.

FUNDING

This work had no external financial support.

CONFLICTS OF INTEREST DISCLOSURE

All authors are either medical advisors to or employees of Infermedica.

ETHICS APPROVAL

The Publication Ethics Committee of the Sciedu Press. The journal’s policies adhere to the Core Practices established by the Committee on Publication Ethics (COPE).

PROVENANCE AND PEER REVIEW

Not commissioned; externally double-blind peer reviewed.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

DATA SHARING STATEMENT

No additional data are available.

OPEN ACCESS

This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).

COPYRIGHTS

Copyright for this article is retained by the author(s), with first publication rights granted to the journal.

References

[1] Schmaus A, Cooper IR, Whitten T, et al. Impact of health link utilization on emergency department visits. CJEM. 2023; 25(5): 429-433. PMid: 37087714. https://doi.org/10.1007/s43678-023-0 0504-3

[2] Kempe A, Luberti AA, Hertz AR, et al. Delivery of pediatric afterhours care by call centers: A multicenter study of parental perceptions and compliance. Pediatrics. 2001; 108(6): E111. PMid: 11731638. https://doi.org/10.1542/peds.108.6.e111

[3] Semigran HL, Linder JA, Gidengil C, et al. Evaluation of symptom checkers for self-diagnosis and triage: Audit study. BMJ. 2015; 351: h3480. PMid: 26157077. https://doi.org/10.1136/bmj.h3480

[4] Bartenschlager CC, Grieger M, Erber J, et al. COVID-19 triage in the emergency department 2.0: How analytics and AI transform a human-made algorithm for the prediction of clinical pathways. Health Care Manag Sci. 2023; 26(3): 412-429. PMid: 37428304. https://doi.org/10.1007/s10729-023-09647-2

[5] Entezarjou A, Bonamy AE, Benjaminsson S, et al. Human versus machine learning-based triage using digitalized patient histories in primary care: Comparative study. JMIR Med Inform. 2020; 8(9): e18930. PMid: 32880578. https://doi.org/10.2196/18930

[6] Dang A, Dang D, Vallish BN. Extent of use of artificial intelligence & machine learning protocols in cancer diagnosis: A scoping review. Indian J Med Res. 2023; 157(1): 11-22. PMid: 37040222. https://doi.org/10.4103/ijmr.IJMR_555_20

[7] Schmitt-Thompson Clinical Content, The Guidelines - Schmitt- Thompson Clinical Content. Available from: stcc-triage.com

[8] Gellert GA, Orzechowski PM, Price T, et al. A multinational survey of patient utilization of and value conveyed through virtual symptom triage and healthcare referral. Frontiers in Public Health. 2023. PMid: 36817183. https://doi.org/10.3389/fpubh.2022.1047291

[9] Case Files:Emergency Medicine, 3rd Edition. McGraw-Hill; September 15, 2012. ISBN: 0071768548.

[10] 100 Cases in Clinical Medicine, 2nd edition. Hodder Education Publishers; March 22, 2007. ISBN: 0340926597.

[11] Case Files: Family Medicine, 4th edition. McGraw Hill Medical; March 10, 2016. ISBN: 1259587703.

[12] Hill MG, Sim M and Mills B, The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. Medical Journal of Australia. 2020; 212: 514-19. PMid: 32391611. https://doi.org/10.5694/mja2.50600