Deep vision transformers in neurodegenerative disease diagnosis using ¹⁸F-fluorodeoxyglucose positron emission tomography scans and anatomical brain atlas

Abstract

This research explores adapting vision transformers (ViTs) to classify neurodegenerative diseases while ensuring their decision-making process is interpretable. We developed a model to classify ¹⁸F-fluorodeoxyglucose (¹⁸F-FDG) positron emission tomography (PET) brain scans into three categories: cognitively normal (CN), mild cognitive impairment (MCI), and Alzheimer’s disease (AD). The dataset utilized in this research contains 580 samples of ¹⁸F-FDG PET scans obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). The proposed model obtained an F1 score of 81% (macro-average of all classes) on the test dataset, a significant performance improvement compared to the literature. Furthermore, we combined the model’s attention maps with the Automated Anatomical Atlas 3 (AAL3), which represents a digital brain map, to identify the most influential areas on the model’s predictions and to conduct a regions’ importance study as a step toward explainability. We demonstrated that ViTs can achieve competitive performance compared to convolutional neural networks while enabling the development of explainable models without extra computations due to the attention mechanism.

Keywords

Vision transformer / Neurodegenerative disease / ¹⁸F-FDG PET / Medical image analysis / Brain scan / Deep neural network

Funding

Data collection and sharing for this project was funded by the ADNI (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Dec 5, 2024 12:30 PM Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.;Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research provided funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (https://www.fnih.org/). The grantee organization is the Northern California Institute for Research and Education, and the study was coordinated by the Alzheimer’s Therapeutic Research Institute at the University of Southern California. ADNI data were disseminated by the Laboratory for Neuro Imaging at the University of Southern California. Farzaneh Etminani and Amira Soliman are supported by Center for Applied Intelligent Systems Research in Health (CAISR Health) funded by Knowledge Foundation (grant no.: 20200208 01H).

Conflict of interest

The authors declare no conflicts of interest.

References

[1] World Health Organization (WHO). Dementia Key Facts. Available from: https://www.who.int/news-room/fact-sheets/detail/dementia [Last accessed on 2025 Mar 30].

[2] Alzheimer Society of Canada. The 10 Benefits of Early Diagnosis. Available from: https://alzheimer.ca/en/about/dementia/do/i/have/dementia/how/get/tested/dementia/tips/individuals-families-friends/10 [Last accessed on 2025 Mar 30].

[3] Alzheimer’s Disease International. Over 41 Million Cases of Dementia go Undiagnosed Across the Globe - World Alzheimer Report Reveals. Available from: https://www.alzint.org/news/events/news/over/41/million/cases/of/dementia/go/undiagnosed-across-the-globe-world-alzheimer-report-reveals [Last accessed on 2025 Mar 30].

[4] Etminani K, Soliman A, Davidsson A, et al. A 3D deep learning model to predict the diagnosis of dementia with lewy bodies, Alzheimer’s disease, and mild cognitive impairment using brain 18F-FDG PET. Eur J Nucl Med Mol Imaging. 2022; 49(2):563-584. doi: 10.1007/s00259-021-05483-0

[5] Ding Y, Sohn JH, Kawczynski MG, et al. A deep learning model to predict a diagnosis of alzheimer disease by using 18F-FDG PET of the brain. Radiology. 2019; 290(2):456-464. doi: 10.1148/radiol.2018180958

[6] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. [ArXiv Preprint]; 2020.

[7] The Alzheimer’s Disease Neuroimaging Initiative. Alzheimer’s Disease Neuroimaging Initiative (ADNI). Available from: https://adni.loni.usc.edu [Last accessed on 2025 Mar 30].

[8] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in Neural Information Processing Systems. 2017. p. 30.

[9] Nobili F, Arbizu J, Bouwman F, et al. European association of nuclear medicine and European academy of neurology recommendations for the use of brain 18F-fluorodeoxyglucose positron emission tomography in neurodegenerative cognitive impairment and dementia: Delphi consensus. Eur J Neurol. 2018; 25(10):1201-1217. doi: 10.1111/ene.13728

[10] Mayo Foundation for Medical Education and Research (MFMER). Positron Emission Tomography Scan. Available from: https://www.mayoclinic.org/tests/procedures/pet/scan/about/pac-20385078 [Last accessed on 2025 Mar 30].

[11] Cleveland Clinic. PET Scan. Available from: https://my.clevelandclinic.org/health/diagnostics/10123-pet-scan [Last accessed on 2025 Mar 30].

[12] Rolls ET, Huang CC, Lin CP, Feng J, Joliot M. Automated anatomical labelling atlas 3. Neuroimage. 2020; 206:116189. doi: 10.1016/j.neuroimage.2019.116189

[13] Mayo Foundation for Medical Education and Research (MFMER). Mild Cognitive Impairment (MCI). Available from: https://www.mayoclinic.org/diseases/conditions/mild/cognitive/impairment/symptoms/causes/syc-20354578 [Last accessed on 2025 Mar 30].

[14] Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition. [ArXiv Preprint]; 2014.

[15] Zeiler MD, Fergus R. Visualizing and understanding convolutional networks. In: Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. Berlin: Springer; 2014. p. 818-833.

[16] Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. p. 2818-2826.

[17] Simonyan K, Vedaldi A, Zisserman A. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. [ArXiv Preprint]; 2013.

[18] Lozupone G, Bria A, Fontanella F, Meijer FJ, De Stefano C. AXIAL: Attention-Based Explainability for Interpretable Alzheimer’s Localized Diagnosis using 2D CNNs on 3D MRI Brain Scans. [ArXiv Preprint]; 2024.

[19] Khatri U, Kwon GR. Explainable vision transformer with self-supervised learning to predict Alzheimer’s disease progression using 18F-FDG PET. Bioengineering (Basel). 2023; 10(10):1225. doi: 10.3390/bioengineering10101225

[20] Shin H, Jeon S, Seol Y, Kim S, Kang D. Vision transformer approach for classification of Alzheimer’s disease using 18f-florbetaben brain images. Appl Sci. 2023; 13(6):3453. doi: 10.3390/app13063453

[21] The National Institutes of Health. How Biomarkers Help Diagnose Dementia. Available from: https://www.nia.nih.gov/health/alzheimers/symptoms/and/diagnosis/how/biomarkers-help-diagnose-dementia [Last accessed on 2025 Mar 30].

[22] Xing X, Liang G, Zhang Y, Khanal S, Lin AL, Jacobs N. Advit: vision transformer on multi-modality pet images for Alzheimer disease diagnosis. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). United States: IEEE; 2022. p. 1-4. doi: 10.1109/ISBI52829.2022.9761584

[23] Odusami M, Maskeliūnas R, Damaševičius R. Pixel-level fusion approach with vision transformer for early detection of Alzheimer’s disease. Electronics. 2023; 12(5):1218. doi: 10.3390/electronics12051218

[24] Lyu Y, Yu X, Zhu D, Zhang L. Classification of Alzheimer’s disease via vision transformer: Classification of Alzheimer’s disease via vision transformer. In: Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments. PETRA ’22. United States: Association for Computing Machinery; 2022. p. 463-468. doi: 10.1145/3529190.3534754

[25] Sarraf S, Sarraf A, DeSouza DD, Anderson JAE, Kabia M, The Alzheimer’s Disease Neuroimaging Initiative. OViTAD: Optimized vision transformer to predict various stages of Alzheimer’s disease using resting-State fMRI and structural MRI data. Brain Sci. 2023; 13(2):260. doi: 10.3390/brainsci13020260

[26] Hoang GM, Kim UH, Kim JG. Vision transformers for the prediction of mild cognitive impairment to Alzheimer’s disease progression using mid-sagittal sMRI. Front Aging Neurosci. 2023; 15:1102869. doi: 10.3389/fnagi.2023.1102869

[27] Aghdam MA, Bozdag S, Saeed F, Alzheimer’s Disease Neuroimaging Initiative. PVTAD: Alzheimer’s disease diagnosis using pyramid vision transformer applied to white matter of T1-weighted structural MRI data. Proc IEEE Int Symp Biomed Imaging. 2024; 2024:10. doi: 10.1109/isbi56570.2024.10635541

[28] Wang W, Xie E, Li X, et al. Pyramid vision transformer:A versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. United States: IEEE; 2021. p. 568-578.

[29] Kushol R, Masoumzadeh A, Huo D, Kalra S, Yang YH. Addformer: Alzheimer’s disease detection from structural MRI using fusion transformer. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). IEEE; 2022. p. 1-5. doi: 10.1109/ISBI52829.2022.9761421

[30] Shah SMAH, Khan MQ, Rizwan A, Jan SU, Samee NA, Jamjoom MM. Computer-aided diagnosis of Alzheimer’s disease and neurocognitive disorders with multimodal Bi-vision transformer (BiViT). Pattern Anal Appl. 2024; 27(3):76. doi: 10.1007/s10044-024-01297-6

[31] The Alzheimer’s Disease Neuroimaging Initiative. ADNI Documentation. Available from: https://adni.loni.usc.edu/help-faqs/adni-documentation [Last accessed on 2025 Mar 30].

[32] The MathWorks Inc. Matlab R; 2016a. Available from: https://www.mathworks.com [Last accessed on 2025 Mar 30].

[33] UCL Queen Square Institute of Neurology. Statistical Parametric Mapping. Available from: https://www.fil.ion.ucl.ac.uk/spm [Last accessed on 2025 Mar 30].

[34] Wolf T, Debut L, Sanh V, et al. HuggingFace’s Transformers: State-of-the-Art Natural Language Processing; 2020. Available from: https://arxiv.org/abs/1910.03771 [Last accessed on 2025 Mar 30].

[35] Ridnik T, Baruch EB, Noy A, Zelnik-Manor L. ImageNet- 21K Pretraining for the Masses. CoRR. 2021. Available from: https://arxiv.org/abs/2104.10972 [Last accessed on 2025 Mar 30].

[36] Russakovsky O, Deng J, Su H, et al. ImageNet large scale visual recognition challenge. Int J Comput Vis IJCV. 2015; 115(3):211-252. doi: 10.1007/s11263-015-0816-y

[37] Google. Vision Transformer (Base-Sized Model). Available from: https://huggingface.co/google/vit-base-patch32-384 [Last accessed on 2025 Mar 30].

[38] Rorden C, Brett M. Stereotaxic display of brain lesions. Behav Neurol. 2000; 12(4):191-200. doi: 10.1155/2000/421719

[39] Li Y, Wang X, Li Y, et al. Abnormal resting-state functional connectivity strength in mild cognitive impairment and its conversion to Alzheimer’s disease. Neural Plast. 2016; 2016:4680972. doi: 10.1155/2016/4680972

[40] Talwar P, Kushwaha S, Chaturvedi M, Mahajan V. Systematic review of different neuroimaging correlates in mild cognitive impairment and Alzheimer’s disease. Clin Neuroradiol. 2021; 31(4):953-967. doi: 10.1007/s00062-021-01057-7

[41] Salmon E, Collette F, Bastin C. Cerebral glucose metabolism in Alzheimer’s disease. Cortex. 2024; 179:50-61. doi: 10.1016/j.cortex.2024.07.004

[42] Arnold SE, Hyman BT, Van Hoesen GW. Neuropathologic changes of the temporal pole in Alzheimer’s disease and pick’s disease. Arch Neurol. 1994; 51(2):145-150. doi: 10.1001/archneur.1994.00540140051014

[43] Yang C, Liu G, Chen X, Le W. Cerebellum in Alzheimer’s disease and other neurodegenerative diseases: An emerging research frontier. MedComm (2020). 2024; 5(7):e638. doi: 10.1002/mco2.638

[44] Bruchhage MMK, Correia S, Malloy P, Salloway S, Deoni S. Machine learning classification identifies cerebellar contributions to early and moderate cognitive decline in Alzheimer’s disease. Front Aging Neurosci. 2020; 12:524024. doi: 10.3389/fnagi.2020.524024