Evaluating the performance of artificial intelligence based software for digital mammography characterization

Abstract

BACKGROUND: Digital screening mammography is a key modality for early detection of breast cancer, reducing mortality by 20–40%. Many artificial intelligence (AI)-based services have been developed to automate the analysis of imaging data.

AIM: The aim of the study was to compare mammography assessments using three types of AI services in multiple versions with radiologists’ conclusions.

MATERIALS AND METHODS: Binary mammography scoring scales were compared with several types and versions of AI services regarding diagnostic accuracy, Matthews correlation coefficient, and maximum Youden’s index.

RESULTS: A comparative analysis showed that the use of a binary scale for evaluating digital mammography affects the number of detected abnormalities and accuracy of AI results. In addition, diagnostic accuracy was found to be threshold dependent. AI Service 1 in version 3 had the best performance, as confirmed by most diagnostic accuracy parameters.

CONCLUSIONS: Our results can be used to select AI services for interpreting mammography screening data. Using Youden’s index maximization to set up an AI service provides a balance of sensitivity and specificity that is not always clinically relevant.

Keywords

malignant tumors of breast / digital mammography / artificial intelligence services / diagnostic accuracy / Youden’s index

References

[1] Seely JM, Alhassan T. Screening for breast cancer in 2018-what should we be doing today? Curr Oncol. 2018;25(suppl 1):S115–S124. doi: 10.3747/co.25.3770

[2] Seely J.M., Alhassan T. Screening for breast cancer in 2018-what should we be doing today? // Curr Oncol. 2018. Vol. 25, Suppl. 1. P. S115–S124. doi: 10.3747/co.25.3770

[3] Artificial intelligence in mammography screening. Clinical applications, issues and directions for development [Internet; cited 20 August 2023]. Available from: https://www.itmportal.ru/upload/iblock/69e/7q981uhfaxjhcntal0exngxtq43xeth2/2.2.3. Kandoba ITM_AI-2022.pdf (in Russ.)

[4] Кандоба В.И. Искусственный интеллект в скрининговой маммографии. Клиническое использование, проблемы и направления развития [интернет]. Режим доступа: https://www.itmportal.ru/upload/iblock/69e/7q981uhfaxjhcntal0exngxtq43xeth2/2.2.3. Kandoba ITM_AI-2022.pdf Дата обращения: 20.08.2023

[5] Celsus — AI-software for analysis of X-ray and CT studies. Mammography [Internet; cited 20 Aug 2023]. Available from: https://celsus.ai/productsmammography/

[6] Цельс. Система поддержки принятия врачебных решений на базе технологий искусственного интеллекта для анализа цифровых медицинских изображений. Маммография [интернет]. Режим доступа: https://celsus.ai/products mammography/ Дата обращения: 20.08.2023

[7] Kim HE, Kim HH, Han BK, et al. Changes in cancer detection and false positive recall in mammography using artificial intelligence: a retrospective, multireader study. Lancet Digit Health. 2020;2(3):e138–e148. doi: 10.1016/S2589-7500(20)30003-0

[8] Kim H.E., Kim H.H., Han B.K., et al. Changes in cancer detection and false positive recall in mammography using artificial intelligence: a retrospective, multireader study // Lancet Digit Health. 2020. Vol. 2, N 3. P. e138–e148. doi: 10.1016/S2589-7500(20)30003-0

[9] Yoon JH, Strand F, Baltzer PAT, et al. Standalone AI for Breast Cancer Detection at Screening Digital Mammography and Digital Breast Tomosynthesis: A Systematic Review and Meta Analysis. Radiology. 2023;307(5):e222639. doi: 10.1148/radiol.222639

[10] Yoon J.H., Strand F., Baltzer P.A.T., et al. Standalone AI for Breast Cancer Detection at Screening Digital Mammography and Digital Breast Tomosynthesis: A Systematic Review and Meta Analysis // Radiology. 2023. Vol. 307, N 5. ID: e222639. doi: 10.1148/radiol.222639

[11] Zhou X H, Obuchowski NA, McClish DK. Statistical Methods in Diagnostic Medicine. NJ: John Wiley & Sons, Inc.; 2011. doi: 10.1002/9780470906514

[12] Zhou X. H., Obuchowski N.A., McClish D.K. Statistical Methods in Diagnostic Medicine. NJ: John Wiley & Sons, Inc.; 2011. doi: 10.1002/9780470906514

[13] Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cut off value: the case of tests with continuous results. Biochem Med (Zagreb). 2016;26(3):297–307. doi: 10.11613/BM.2016.034

[14] Habibzadeh F., Habibzadeh P, Yadollahie M. On determining the most appropriate test cut off value: the case of tests with continuous results // Biochem Med (Zagreb). 2016. Vol. 26, N 3. P. 297–307. doi: 10.11613/BM.2016.034

[15] Schaffter T, Buist DSM, Lee CI, et al. Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms. JAMA Netw Open. 2020;3(3):e200265. doi: 10.1001/jamanetworkopen.2020.0265

[16] Schaffter T., Buist D.S.M., Lee C.I., et al. Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms // JAMA Netw Open. 2020. Vol. 3, N 3. ID: e200265. doi: 10.1001/jamanetworkopen.2020.0265

[17] McKinney SM, Sieniek M, Godbole V, et al. International evaluation of an AI system for breast cancer screening. Nature. 2020;577(7788):89–94. doi: 10.1038/s41586-019-1799-6

[18] McKinney S.M., Sieniek M., Godbole V., et al. International evaluation of an AI system for breast cancer screening // Nature. 2020. Vol. 577, N 7788. P. 89–94. doi: 10.1038/s41586-019-1799-6

[19] Nam JG, Kim M, Park J, et al. Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. Eur Respir J. 2021;57(5):2003061. doi: 10.1183/13993003.03061-2020

[20] Nam J.G., Kim M., Park J., et al. Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs // Eur Respir J. 2021. Vol. 57, N 5. ID: 2003061. doi: 10.1183/13993003.03061-2020

[21] Sakhnov SN, Axenov KD, Axenova LE, et al. Development of a cataract screening model using an open dataset and deep machine learning algorithms. Fyodorov Journal of Ophthalmic Surgery. 2022;(S4):13–20. EDN: VEGPAW doi: 10.25276/0235-4160-2022-4S-13-20

[22] Сахнов С.Н., Аксенов К.Д., Аксенова Л.Е., и др. Разработка модели скрининга катаракты с использованием открытого набора данных и алгоритмов глубокого машинного обучения. Офтальмохирургия. 2022. № S4. С. 13–20. EDN: VEGPAW doi: 10.25276/0235-4160-2022-4S-13-20

[23] King G, Zeng L. Logistic Regression in Rare Events Data. Political Analysis. 2001;9(2):137–163. doi: 10.1093/oxfordjournals.pan.a004868

[24] King G., Zeng L. Logistic Regression in Rare Events Data // Political Analysis. 2001. Vol. 9, N 2. P. 137–163. doi: 10.1093/oxfordjournals.pan.a004868

[25] Chen F, Xue Y, Tan MT, Chen P. Efficient statistical tests to compare Youden index: accounting for contingency correlation. Stat Med. 2015;34(9):1560–1576. doi: 10.1002/sim.6432

[26] Chen F., Xue Y., Tan M.T., Chen P. Efficient statistical tests to compare Youden index: accounting for contingency correlation // Stat Med. 2015. Vol. 34, N 9. P. 1560–1576. doi: 10.1002/sim.6432

[27] Vasiliev YuA, Vladzimirsky AV, Sharova DE, et al. Clinical trials of artificial intelligence systems (radiation diagnostics). Moscow: State budgetary healthcare institution of the city of Moscow «Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Department». 2023. 40 p. (In Russ.) EDN: PUIJLD

[28] Васильев Ю.А., Владзимирский А.В., Шарова Д.Е., и др. Клинические испытания систем искусственного интеллекта (лучевая диагностика). Москва: Государственное бюджетное учреждение здравоохранения города Москвы «Научно-практический клинический центр диагностики и телемедицинских технологий Департамента здравоохранения города Москвы», 2023. 40 с. EDN: PUIJLD

[29] Arzamasov KM, Vasilev YuA, Vladzymyrskyy AV, et al. The use of computer vision for the mammography preventive research. The Russian Journal of Preventive Medicine. 2023;26(6):117–123. EDN: YBKHPS doi: 10.17116/profmed202326061117

[30] Арзамасов К.М., Васильев Ю.А., Владзимирский А.В., и др. Применение компьютерного зрения для профилактических исследований на примере маммографии // Профилактическая медицина. 2023. Т. 26, № 6. С. 117–123. EDN: YBKHPS doi: 10.17116/profmed202326061117

[31] Vasilev YuA, Tyrov IA, Vladzymyrskyy AV, et al. Double reading mammograms using artificial intelligence technologies: A new model of mass preventive examination organization. Digital Diagnostics. 2023;4(2):93–104. EDN: VRIEOH doi: 10.17816/DD321423

[32] Васильев Ю.А., Тыров И.А., Владзимирский А.В., и др. Двойной просмотр результатов маммографии с применением технологий искусственного интеллекта: новая модель организации массовых профилактических исследований // Digital Diagnostics. 2023. Т. 4, № 2. С. 93–104. EDN: VRIEOH doi: 10.17816/DD321423

[33] Vasilev YuA, Tyrov IA, Vladzymyrskyy AV, et al. A New Model of Organizing Mass Screening Based on Stand Alone Artificial Intelligence Used for Fluorography Image Triage. Public Health and Life Environment — PH&LE. 2023;31(11):23-32. EDN: SYIQBX doi: 10.35627/2219-5238/2023-31-11-23-32

[34] Васильев Ю.А., Тыров И.А., Владзимирский А.В., и др. Новая модель организации массовых профилактических исследований, основанная на автономном искусственном интеллекте для сортировки результатов флюорографии // Здоровье населения и среда обитания — ЗНиСО. 2023. Т. 31, № 11. С. 23–32. EDN: SYIQBX doi: 10.35627/2219-5238/2023-31-11-23-32

[35] Vladzimirskyy AV, Vasilev YuA, Arzamasov KM, et al. Computer vision in radiology: the first stage of the Moscow experiment. Moscow: Izdatel’skie resheniya; 2022. (In Russ.) EDN: FOYLXK

[36] Владзимирский А.В., Васильев Ю.А., Арзамасов К.М., и др. Компьютерное зрение в лучевой диагностике: первый этап Московского эксперимента. Москва: Издательские решения, 2022. EDN: FOYLXK

Funding

Moscow City Health DepartmentДепартамент здравоохранения города МосквыMoscow City Health Department(123031500004-5)

RIGHTS & PERMISSIONS

Eco-Vector