How does artificial intelligence effect on the assessment of lung damage in COVID-19 on chest CT scan?

Abstract

BACKGROUND: During the pandemic, computed tomography (CT) was one of the most important tools for assessing COVID-19-related lung changes. In COVID-19 patients, radiologists in Moscow used the adapted CT0-4 scale to visually assess the dependence of the severity of the general condition on the nature and severity of radiological signs of changes in the lungs based on computed tomography. In a large stream of scans, the doctor may miss findings and make errors in assessing the volume of lung damage, so the use of AI services in outpatient healthcare during a pandemic can be beneficial.

AIM: The goal of this study is to compare the distribution of CT0-4 categories designed by radiologists with the results of AI services processing and categories formed without AI services.

METHODS: We used retrospective study design, full study protocol is registered on ClinicalTrials.gov (NCT04489992). The results of primary CT scans with the CT0-4 categories were analyzed in outpatient medical institutions of the Health Department from April 08, 2020, to December 01, 2020, and separately for November (from November 01, 2020, to December 01, 2020). CT was performed on 48 computed tomographs in accordance with standard protocols, and the data was processed by the single radiology information systems. CTs in the test group received AI services, while CTs in the control group did not. The analysis includes five AI services: RADLogics COVID-19 (RADLogics, USA), COVID-IRA (IRA labs, Russia), Care Mentor AI, COVID (Care Mentor AI, Russia), Third Opinion. CT-COVID-19 (Third Opinion, Russia), and COVID-MULTIVOX (Gammamed, Russia). Moreover, AI services are encoded at random.

RESULTS: The CT scan results of 260,594 patients were examined (m/f % = 44/56, mean age = 49.5). The test group consisted of 115,618 CT scans, while the control group consisted of 144,976 CT scans. Depending on the specific AI service, CT0 was established by 2.3–18.5% less than the control group for different subgroups of categories. The categories CT3-4 were established by 4.7–27.6% less than without AI, and the categories CT4 by 40–60% less than without AI (p < 0.0001). For November (from November 01, 2020, to December 01, 2020), the CT scan results of 41,386 patients were analyzed (m/f % = 44/56, average age = 53.2 years). The test group consisted of 28,881 CT scans, while the control group included 12,505 CT scans. Depending on the specific AI service, CT0 was established by 1–2.6% less than the control group for different subgroups of categories. Further, the categories CT3–CT4 were established by 0.2–15.7% less than without AI, and the categories CT4 were established by 25% less than without AI (p = 0.001).

CONCLUSION: The use of AI services for primary CT scans on an outpatient basis reduces the number of CT0 and CT3–CT4 results, which can influence the therapeutic approach for COVID-19 patients.

Keywords

COVID-19 / community-acquired pneumonia / computed tomography / artificial intelligence

References

[1] Experiment on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow [cited 2021 Feb 04]. (In Russ). Available from: https://mosmed.ai

[2] Эксперимент по использованию инновационных технологий в области компьютерного зрения для анализа медицинских изображений и дальнейшего применения в системе здравоохранения города Москвы [дата обращения: 04.02.2021]. Режим доступа: https://mosmed.ai. Дата обращения: 15.01.2021.

[3] Morozov SP, Ledikhova NV, Panina EV, et al. Re: Controversy in coronaViral Imaging and Diagnostics (COVID). Clin Radiol. 2020;75(11):871–872. doi: 10.1016/j.crad.2020.07.023

[4] Morozov S.P., Ledikhova N.V., Panina E.V., et al. Re: Controversy in coronaViral Imaging and Diagnostics (COVID) // Clin Radiol. 2020. Vol. 75, N 11. P. 871–872. doi: 10.1016/j.crad.2020.07.023

[5] Chang YC, Yu CJ, Chang SC, et al. Pulmonary sequelae in convalescent patients after severe acute respiratory syndrome: evaluation with thin-section CT. Radiology. 2005;236(3):1067–1075. doi: 10.1148/radiol.2363040958

[6] Chang Y.C., Yu C.J., Chang S.C., et al. Pulmonary sequelae in convalescent patients after severe acute respiratory syndrome: evaluation with thin-section CT // Radiology. 2005. Vol. 236, N 3. P. 1067–1075. doi: 10.1148/radiol.2363040958

[7] Haseli S, Khalili N, Bakhshayeshkaram M, et al. Lobar distribution of COVID-19 pneumonia based on chest computed tomography findings. A retrospective study. Arch Acad Emerg Med. 2020;8(1):e55.

[8] Haseli S., Khalili N., Bakhshayeshkaram M., et al. Lobar distribution of COVID-19 pneumonia based on chest computed tomography findings. A retrospective study // Arch Acad Emerg Med. 2020. Vol. 8, N 1. P. 55.

[9] Inui S, Fujikawa A, Jitsu M, et al. Chest CT findings in cases from the cruise ship “Diamond Princess” with Coronavirus Disease 2019 (COVID-19). Radiol Cardiothorac Imaging. 2020;2(2):e200110. doi: 10.1148/ryct.2020200110

[10] Inui S., Fujikawa A., Jitsu M., et al. Chest CT findings in cases from the cruise ship «Diamond Princess» with Coronavirus Disease 2019 (COVID-19) // Radiol Cardiothorac Imaging. 2020. Vol. 2, N 2. P. 200110. doi: 10.1148/ryct.2020200110

[11] Prokop M, van Everdingen W, van Rees Vellinga T, et al. CO-RADS: A Categorical CT assessment scheme for patients suspected of having COVID-19-definition and evaluation. Radiology. 2020;296(2):97–104. doi: 10.1148/radiol.2020201473

[12] Prokop M., van Everdingen W., van Rees Vellinga T., et al. CO-RADS: A Categorical CT assessment scheme for patients suspected of having COVID-19-definition and evaluation // Radiology. 2020. Vol. 296, N 2. P. 97–104. doi: 10.1148/radiol.2020201473

[13] Shen C, Yu N, Cai S, et al. Quantitative computed tomography analysis for stratifying the severity of Coronavirus Disease 2019. J Pharm Anal. 2020;10(2):123–129. doi: 10.1016/j.jpha.2020.03.004

[14] Shen C., Yu N., Cai S., et al. Quantitative computed tomography analysis for stratifying the severity of Coronavirus Disease 2019 // J Pharm Anal. 2020. Vol. 10, N 2. P. 123–129. doi: 10.1016/j.jpha.2020.03.004

[15] Pan F, Ye T, Sun P, et al. Time course of lung changes at chest CT during recovery from Coronavirus Disease 2019 (COVID-19). Radiology. 2020;295(3):715–721. doi: 10.1148/radiol.2020200370

[16] Pan F., Ye T., Sun P., et al. Time course of lung changes at chest CT during Recovery from Coronavirus Disease 2019 (COVID-19) // Radiology. 2020. Vol. 295, N 3. P. 715–721. doi: 10.1148/radiol.2020200370

[17] Revel MP, Parkar AP, Prosch H, et al. COVID-19 patients and the radiology department – advice from the European Society of Radiology (ESR) and the European Society of Thoracic Imaging (ESTI). Eur Radiol. 2020;30(9):4903–4909. doi: 10.1007/s00330-020-06865-y

[18] Revel M.P., Parkar A.P., Prosch H., et al. COVID-19 patients and the radiology department – advice from the European Society of Radiology (ESR) and the European Society of Thoracic Imaging (ESTI) // Eur Radiol. 2020. Vol. 30, N 9. P. 4903–4909. doi: 10.1007/s00330-020-06865-y

[19] Morozov SP, Protsenko DN, Smetanina SV, et al. Radiation diagnosis of coronavirus disease (COVID-19): organization, methodology, interpretation of results: preprint II. Version 2 of 17.04.2020. The series “Best practices of radiation and instrumental diagnostics”. Issue 65. Moscow: Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department; 2020. 78 p. (In Russ).

[20] Морозов С.П., Проценко Д.Н., Сметанина С.В., и др. Лучевая диагностика коронавирусной болезни (COVID-19): организация, методология, интерпретация результатов: препринт № ЦДТ-2020-II. Версия 2 от 17.04.2020. Серия «Лучшие практики лучевой и инструментальной диагностики». Вып. 65. Москва: ГБУЗ «НПКЦ ДиТ ДЗМ», 2020. 78 с.

[21] Sinitsyn VE, Tyurin IE, Mitkov VV. Consensus Guidelines of Russian Society of Radiology (RSR) and Russian Association of Specialists in Ultrasound Diagnostics in Medicine (RASUDM) “Role of Imaging (X-ray, CT and US) in Diagnosis of COVID-19 Pneumonia” (version 2). Journal of Radiology and Nuclear Medicine. 2020;101(2):72–89. (In Russ). doi: 10.20862/0042-4676-2020-101-2-72-89

[22] Синицын В.Е., Тюрин И.Е., Митьков В.В. Временные методические рекомендации Российского общества рентгенологов и радиологов (РОРР) и Российской ассоциации специалистов ультразвуковой диагностики в медицине (РАСУДМ) «Методы лучевой диагностики пневмонии при новой коронавирусной инфекции при COVID-19» (версия 2) // Вестник рентгенологии и радиологии. 2020. Т. 101, № 2. С. 72–89. doi: 10.20862/0042-4676-2020-101-2-72-89

[23] Morozov SP, Gombolevskiy VA, Cherninа VY, et al. Prediction of lethal outcomes in COVID-19 cases based on the results chest computed tomography. Tuberculosis and Lung Diseases. 2020;98(6):7–14. (In Russ). doi: 10.21292/2075-1230-2020-98-6-7-14

[24] Морозов С.П., Гомболевский В.А., Чернина В.Ю., и др. Прогнозирование летальных исходов при COVID-19 по данным компьютерной томографии органов грудной клетки // Туберкулез и болезни лёгких. 2020. Т. 98, № 6. С. 7–14. doi: 10.21292/2075-1230-2020-98-6-7-14

[25] Howard J. Cognitive errors and diagnostic mistakes. A case-based guide to critical thinking in medicine. New York: Springer; 2019.

[26] Morozov SP, Vladzimirsky AV, Klyashtorny VG, et al. Clinical trials of software based on intelligent technologies (radiation diagnostics). The series “Best practices of radiation and instrumental diagnostics”. Issue 57. Moscow: Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department; 2019. 51 p. (In Russ).

[27] Морозов С.П., Владзимирский А.В., Кляшторный В.Г., и др. Клинические испытания программного обеспечения на основе интеллектуальных технологий (лучевая диагностика). Серия «Лучшие практики лучевой и инструментальной диагностики». Вып. 57. Москва: ГБУЗ «НПКЦ ДиТ ДЗМ», 2019. 51 с.

[28] Sahiner B, Pezeshk A, Hadjiiski LM, et al. Deep learning in medical imaging and radiation therapy. Med Phys. 2019;46(1):1–36. doi: 10.1002/mp.13264

[29] Sahiner B., Pezeshk A., Hadjiiski L.M., et al. Deep learning in medical imaging and radiation therapy // Med Phys. 2019. Vol. 46, N 1. P. 1–36. doi: 10.1002/mp.13264

[30] Allen BJ, Seltzer SE, Langlotz CP, et al. A road map for translational research on artificial intelligence in medical imaging: from the 2018 national institutes of health/RSNA/ACR/The academy workshop. J Am Coll Radiol. 2019;16(9):1179–1189. doi: 10.1016/j.jacr.2019.04.014

[31] Allen B.J., Seltzer S.E., Langlotz C.P., et al. A road map for translational research on artificial intelligence in medical imaging: from the 2018 national institutes of health/RSNA/ACR/The academy workshop // J Am Coll Radiol. 2019. Vol. 16, N 9. P. 1179–1189. doi: 10.1016/j.jacr.2019.04.014

[32] Angus DC. Randomized clinical trials of artificial intelligence. Jama. 2020;323(11):1043–1045. doi: 10.1001/jama.2020.1039

[33] Angus D.C. Randomized clinical trials of artificial intelligence // Jama. 2020. Vol. 323, N 11. P. 1043–1045. doi: 10.1001/jama.2020.1039

[34] Wijnberge M, Geerts BF, Hol L, et al. Effect of a machine learning-derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial. Jama. 2020;323(11):1052–1060. doi: 10.1001/jama.2020.0592

[35] Wijnberge M., Geerts B.F., Hol L., et al. Effect of a machine learning-derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial // Jama. 2020. Vol. 323, N 11. P. 1052–1060. doi: 10.1001/jama.2020.0592

[36] Carlile M, Hurt B, Hsiao A, et al. Deployment of artificial intelligence for radiographic diagnosis of COVID-19 pneumonia in the emergency department. J Am Coll Emerg Physicians Open. 2020;1(6):1459–1464. doi: 10.1002/emp2.12297

[37] Carlile M., Hurt B., Hsiao A., et al. Deployment of artificial intelligence for radiographic diagnosis of COVID-19 pneumonia in the emergency department // J Am Coll Emerg Physicians Open. 2020. Vol. 1, N 6. P. 1459–1464. doi: 10.1002/emp2.12297

[38] Morozov SP, Chernina VYu, Blokhin IA, et al. Chest computed tomography for outcome prediction in laboratory-confirmed COVID-19: A retrospective analysis of 38,051 cases. Digital Diagnostics. 2020;1(1):27−36. (In Russ). doi: 10.17816/DD46791

[39] Морозов С.П., Чернина В.Ю., Блохин И.А., Гомболевский В.А. Прогнозирование исходов при лабораторно верифицированном COVID-19 по данным компьютерной томографии органов грудной клетки: ретроспективный анализ 38051 пациента // Digital Diagnostics. 2020. Т. 1, № 1. C. 27–36. doi: 10.17816/DD46791

RIGHTS & PERMISSIONS

Morozov S.P., Chernina V.Y., Andreychenko A.E., Vladzymyrskyy A.V., Mokienko O.А., Gombolevskiy V.A.