Pixel-Level Fusion Approach with Vision Transformer for Early Detection of Alzheimer’s Disease
Abstract
:1. Introduction
1.1. Background
1.2. Deep Learning Methods for AD Recognition and Classification
1.3. Contribution, Novelty, and Research Questions
- 1.
- Discrete wavelet decomposition module to generate one approximate coefficient and three detail coefficients of MRI and PET image;
- 2.
- VGG16 module to generate approximate image and three detail coefficients, which represent low frequency sub-band and high frequency, respectively;
- 3.
- Inverse wavelet transform module to generate fused images on the four bands generated;
- 4.
- Pre-trained ViT module that is used to extract and classify features (structural and functional) from fused image.
- 1.
- How can MRI and PET data be effectively combined for the early detection of Alzheimer’s disease?
- 2.
- Can a ViT model be trained on the fused data for improved accuracy in the classification of Alzheimer’s disease stages (AD/EMCI and AD/LMCI)?
- 3.
- How does the proposed multimodal fusion approach compare with existing methods for the classification of Alzheimer’s disease stages?
- 4.
- Can the proposed ViT model generalise to new unseen data from the ADNI database for the classification of Alzheimer’s disease stages?
- 5.
- What are the limitations and potential improvements of the proposed multimodal fusion approach and the use of ViT for medical imaging data analysis?
- An image fusion technique has been proposed to fuse multimodal images for AD diagnosis, providing accurate diagnosis of AD to health professionals.
- Complementary information from MRI and PET images is incorporated using wavelet transform and transfer learning.
- Frequency and location information from MRI and PET images were captured.
- The proposed model is optimised using transfer learning, which improves the performance of the proposed model.
2. Methods
2.1. Dataset
2.2. Preprocessing
2.3. Image Registration
2.4. Noise Reduction
2.5. Multimodal Fusion
2.6. ViT Architecture
3. Experiments and Results
3.1. Experiments
3.2. Result
3.3. Visualization
3.4. Comparison with Existing Methods
4. Discussion
4.1. Answers to Research Questions
4.1.1. Answer to Research Question 1
4.1.2. Answer to Research Question 2
4.1.3. Answer to Research Question 3
4.1.4. Answer to Research Question 4
4.1.5. Answer to Research Question 5
4.2. Limitations
- Limited dataset: the study was carried out on a limited dataset from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database, and the results may not be generalised to larger or diverse datasets.
- Fusion parameters: the fusion parameters in the study were not optimised to their full potential, and further optimisation may be necessary to improve the performance of the fused image.
- Single-mode performance: the performance of the model using single modalities (MRI or PET) was not evaluated, so it is not clear how well the model would perform without the fusion of data.
- Limitations of ViT: the use of a ViT model for the analysis of medical imaging data is still a relatively new area of research, and its limitations have not been fully explored.
- Fusion technique: the fusion technique used in the study (DWT) may not be optimal for all types of medical imaging data, and other fusion techniques should be evaluated.
- Transfer learning: the study relied on transfer learning with a pre-trained VGG16 model, and the results may not generalise to other types of pre-training or architectures.
- Model selection: the selection of a ViT model for the study was based on its performance on a different task, and the suitability of ViT for the task of AD classification has not been fully established.
- These limitations highlight the need for further research and evaluation of the proposed multimodal fusion approach and the use of ViT for the analysis of medical imaging data.
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- DeTure, M.A.; Dickson, D.W. The neuropathological diagnosis of Alzheimer’s disease. Mol. Neurodegener. 2019, 14, 32. [Google Scholar] [CrossRef] [Green Version]
- Piovezan, R.D.; Oliveira, D.; Arias, N.; Acosta, D.; Prince, M.J.; Ferri, C.P. Mortality Rates and Mortality Risk Factors in Older Adults with Dementia from Low- and Middle-Income Countries: The 10/66 Dementia Research Group Population-Based Cohort Study. J. Alzheimer’s Dis. 2020, 75, 581–593. [Google Scholar] [CrossRef]
- Gaugler, J.E.; James, B.; Johnson, T.; Reimer, J.; Solis, M.; Weuve, J.; Buckley, R.F.; Hohman, T.J. 2022 Alzheimer’s disease facts and figures. Alzheimer’s Dement. 2022, 18, 700–789. [Google Scholar] [CrossRef]
- Odusami, M.; Maskeliunas, R.; Damaševičius, R.; Misra, S. Comparable Study of Pre-Trained Model on Alzheimer Disease Classification; Lecture Notes in Computer Science Volume 12953; Springer: Berlin/Heidelberg, Germany, 2021; pp. 63–74. [Google Scholar]
- Li, H.; Habes, M.; Wolk, D.A.; Fan, Y. A deep learning model for early prediction of Alzheimer’s disease dementia based on hippocampal magnetic resonance imaging data. Alzheimer’s Dement. 2019, 15, 1059–1070. [Google Scholar] [CrossRef]
- Bartos, A.; Gregus, D.; Ibrahim, I.; Tintěra, J. Brain volumes and their ratios in Alzheimer´s disease on magnetic resonance imaging segmented using Freesurfer 6.0. Psychiatry Res. Neuroimaging 2019, 287, 70–74. [Google Scholar] [CrossRef]
- Chandra, A.; Dervenoulas, G.; Politis, M. Magnetic resonance imaging in Alzheimer’s disease and mild cognitive impairment. J. Neurol. 2018, 266, 1293–1302. [Google Scholar] [CrossRef] [Green Version]
- Holiga, S.; Abdulkadir, A.; Klöppel, S.; Dukart, J. Functional Magnetic Resonance Imaging in Alzheimer’ Disease Drug Development. In Biomarkers for Alzheimer’s Disease Drug Development; Springer: Berlin/Heidelberg, Germany, 2018; pp. 159–163. [Google Scholar] [CrossRef]
- Forouzannezhad, P.; Abbaspour, A.; Fang, C.; Cabrerizo, M.; Loewenstein, D.; Duara, R.; Adjouadi, M. A survey on applications and analysis methods of functional magnetic resonance imaging for Alzheimer’s disease. J. Neurosci. Methods 2019, 317, 121–140. [Google Scholar] [CrossRef]
- Wang, X.H.; Zhao, J.; Du, Y.H.; Ding, X.T.; Men, G.Z. Alteration of functional connectivity in patients with Alzheimer’s disease revealed by resting-state functional magnetic resonance imaging. Neural Regen. Res. 2020, 15, 285. [Google Scholar] [CrossRef]
- Ossenkoppele, R.; Rabinovici, G.D.; Smith, R.; Cho, H.; Schöll, M.; Strandberg, O.; Palmqvist, S.; Mattsson, N.; Janelidze, S.; Santillo, A.; et al. Discriminative Accuracy of Positron Emission Tomography for Alzheimer Disease vs Other Neurodegenerative Disorders. JAMA 2018, 320, 1151. [Google Scholar] [CrossRef] [Green Version]
- Guo, J.; Qiu, W.; Li, X.; Zhao, X.; Guo, N.; Li, Q. Predicting Alzheimer’s Disease by Hierarchical Graph Convolution from Positron Emission Tomography Imaging. In Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 9–12 December 2019; IEEE: New York, NY, USA, 2019. [Google Scholar] [CrossRef] [Green Version]
- Velazquez, M.; Anantharaman, R.; Velazquez, S.; Lee, Y.; Alzheimer’s Disease Neuroimaging Initiative. RNN-Based Alzheimer’s Disease Prediction from Prodromal Stage using Diffusion Tensor Imaging. In Proceedings of the 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), San Diego, CA, USA, 18–21 November 2019; pp. 1665–1672. [Google Scholar]
- Amoroso, N.; Rocca, M.L.; Bellotti, R.; Fanizzi, A.; Monaco, A.; Tangaro, S. Alzheimer’s disease diagnosis based on the Hippocampal Unified Multi-Atlas Network (HUMAN) algorithm. BioMedical Eng. Online 2018, 17, 6. [Google Scholar] [CrossRef] [Green Version]
- Lian, C.; Liu, M.; Zhang, J.; Shen, D. Hierarchical Fully Convolutional Network for Joint Atrophy Localization and Alzheimer’s Disease Diagnosis Using Structural MRI. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 880–893. [Google Scholar] [CrossRef]
- Gupta, Y.; Lee, K.H.; Choi, K.Y.; Lee, J.J.; Kim, B.C.; Kwon, G.R. Early diagnosis of Alzheimer’s disease using combined features from voxel-based morphometry and cortical, subcortical, and hippocampus regions of MRI T1 brain images. PLoS ONE 2019, 14, e0222446. [Google Scholar] [CrossRef] [Green Version]
- Toshkhujaev, S.; Lee, K.H.; Choi, K.Y.; Lee, J.J.; Kwon, G.R.; Gupta, Y.; Lama, R.K. Classification of Alzheimer’s Disease and Mild Cognitive Impairment Based on Cortical and Subcortical Features from MRI T1 Brain Images Utilizing Four Different Types of Datasets. J. Healthc. Eng. 2020, 2020, 1–14. [Google Scholar] [CrossRef]
- Hossain, M.S.; Kimura, F.; Yagi, Y.; Yamaguchi, M.; Nakamura, T. Practical image quality evaluation for whole slide imaging scanner. In Proceedings of the Biomedical Imaging and Sensing Conference, Yokohama, Japan, 25–27 April 2018; Matoba, O., Awatsuji, Y., Yatagai, T., Aizu, Y., Eds.; SPIE: Bellingham, WA, USA, 2018. [Google Scholar] [CrossRef]
- Bi, X.; Jiang, Q.; Sun, Q.; Shu, Q.; Liu, Y. Analysis of Alzheimer’s Disease Based on the Random Neural Network Cluster in fMRI. Front. Neuroinformatics 2018, 12. [Google Scholar] [CrossRef] [Green Version]
- Hojjati, S.H.; Ebrahimzadeh, A.; Khazaee, A.; Babajani-Feremi, A. Predicting conversion from MCI to AD by integrating rs-fMRI and structural MRI. Comput. Biol. Med. 2018, 102, 30–39. [Google Scholar] [CrossRef] [Green Version]
- Wang, Y.; Li, C. Functional magnetic resonance imaging classification based on random forest algorithm in Alzheimer’s disease. In Proceedings of the 2019 International Conference on Image and Video Processing, and Artificial Intelligence, Shanghai, China, 23–25 August 2019; Su, R., Ed.; SPIE: Bellingham, WA, USA, 2019. [Google Scholar] [CrossRef]
- Marchitelli, R.; Aiello, M.; Cachia, A.; Quarantelli, M.; Cavaliere, C.; Postiglione, A.; Tedeschi, G.; Montella, P.; Milan, G.; Salvatore, M.; et al. Simultaneous resting-state FDG-PET/fMRI in Alzheimer Disease: Relationship between glucose metabolism and intrinsic activity. NeuroImage 2018, 176, 246–258. [Google Scholar] [CrossRef] [Green Version]
- Chen, D.; Jiang, J.; Lu, J.; Wu, P.; Zhang, H.; Zuo, C.; Shi, K. Brain Network and Abnormal Hemispheric Asymmetry Analyses to Explore the Marginal Differences in Glucose Metabolic Distributions Among Alzheimer’s Disease, Parkinson’s Disease Dementia, and Lewy Body Dementia. Front. Neurol. 2019, 10. [Google Scholar] [CrossRef]
- Kadry, S.; Damasevicius, R.; Taniar, D.; Rajinikanth, V.; Lawal, I.A. Extraction of Tumour in Breast MRI using Joint Thresholding and Segmentation—A Study. In Proceedings of the 2021 IEEE 7th International Conference on Bio Signals, Images and Instrumentation, Chennai, India, 25–27 March 2021. [Google Scholar]
- Rajinikanth, V.; Kadry, S.; Nam, Y. Convolutional-neural-network assisted segmentation and svm classification of brain tumor in clinical mri slices. Inf. Technol. Control 2021, 50, 342–356. [Google Scholar] [CrossRef]
- Khan, M.A.; Khan, A.; Alhaisoni, M.; Alqahtani, A.; Alsubai, S.; Alharbi, M.; Malik, N.A.; Damaševičius, R. Multimodal brain tumor detection and classification using deep saliency map and improved dragonfly optimization algorithm. Int. J. Imaging Syst. Technol. 2022; Early View. [Google Scholar] [CrossRef]
- Rajinikanth, V.; Kadry, S.; Damasevicius, R.; Sujitha, R.A.; Balaji, G.; Mohammed, M.A. Glioma/Glioblastoma Detection in Brain MRI using Pre-trained Deep-Learning Scheme. In Proceedings of the 2022 3rd International Conference on Intelligent Computing, Instrumentation and Control Technologies: Computational Intelligence for Smart Systems, Kannur, Kerala, 11–12 August 2022; pp. 987–990. [Google Scholar]
- Badjie, B.; Deniz Ülker, E. A Deep Transfer Learning Based Architecture for Brain Tumor Classification Using MR Images. Inf. Technol. Control 2022, 51, 332–344. [Google Scholar] [CrossRef]
- Odusami, M.; Maskeliūnas, R.; Damaševičius, R.; Misra, S. ResD Hybrid Model Based on Resnet18 and Densenet121 for Early Alzheimer Disease Classification; Lecture Notes in Networks and Systems Volume 418; Springer: Berlin/Heidelberg, Germany, 2022; pp. 296–305. [Google Scholar]
- Ramya, J.; Maheswari, B.U.; Rajakumar, M.P.; Sonia, R. Alzheimer’s Disease Segmentation and Classification on MRI Brain Images Using Enhanced Expectation Maximization Adaptive Histogram (EEM-AH) and Machine Learning. Inf. Technol. Control 2022, 51, 786–800. [Google Scholar] [CrossRef]
- Venugopalan, J.; Tong, L.; Hassanzadeh, H.R.; Wang, M.D. Multimodal deep learning models for early detection of Alzheimer’s disease stage. Sci. Rep. 2021, 11, 3254. [Google Scholar] [CrossRef] [PubMed]
- Sarraf, S.; Tofighi, G. Classification of Alzheimer’s Disease Structural MRI Data by Deep Learning Convolutional Neural Networks. arXiv 2016, arXiv:1607.06583. [Google Scholar] [CrossRef]
- Abdelaziz, M.; Wang, T.; Elazab, A. Alzheimer’s disease diagnosis framework from incomplete multimodal data using convolutional neural networks. J. Biomed. Inform. 2021, 121, 103863. [Google Scholar] [CrossRef]
- Jin, L.; Zhao, K.; Zhao, Y.; Che, T.; Li, S. A Hybrid Deep Learning Method for Early and Late Mild Cognitive Impairment Diagnosis With Incomplete Multimodal Data. Front. Neuroinformatics 2022, 16, 843566. [Google Scholar] [CrossRef]
- Khagi, B.; Kwon, G.R. 3D CNN Design for the Classification of Alzheimer’s Disease Using Brain MRI and PET. IEEE Access 2020, 8, 217830–217847. [Google Scholar] [CrossRef]
- Forouzannezhad, P.; Abbaspour, A.; Li, C.; Cabrerizo, M.; Adjouadi, M. A Deep Neural Network Approach for Early Diagnosis of Mild Cognitive Impairment Using Multiple Features. In Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA, 17–20 December 2018; IEEE: New York, NY, USA, 2018. [Google Scholar] [CrossRef]
- Forouzannezhad, P.; Abbaspour, A.; Li, C.; Fang, C.; Williams, U.; Cabrerizo, M.; Barreto, A.; Andrian, J.; Rishe, N.; Curiel, R.E.; et al. A Gaussian-based model for early detection of mild cognitive impairment using multimodal neuroimaging. J. Neurosci. Methods 2020, 333, 108544. [Google Scholar] [CrossRef] [PubMed]
- Aqeel, A.; Hassan, A.; Khan, M.A.; Rehman, S.; Tariq, U.; Kadry, S.; Majumdar, A.; Thinnukool, O. A Long Short-Term Memory Biomarker-Based Prediction Framework for Alzheimer’s Disease. Sensors 2022, 22, 1475. [Google Scholar] [CrossRef]
- Sarraf, S.; Sarraf, A.; DeSouza, D.D.; Anderson, J.A.E.; Kabia, M. OViTAD: Optimized Vision Transformer to Predict Various Stages of Alzheimer’s Disease Using Resting-State fMRI and Structural MRI Data. bioXiv 2021. [Google Scholar] [CrossRef]
- Xing, X.; Liang, G.; Zhang, Y.; Khanal, S.; Lin, A.L.; Jacobs, N. Advit: Vision Transformer On Multi-Modality Pet Images For Alzheimer Disease Diagnosis. In Proceedings of the 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), Kolkata, India, 28–31 March 2022. [Google Scholar] [CrossRef]
- Kushol, R.; Masoumzadeh, A.; Huo, D.; Kalra, S.; Yang, Y.H. Addformer: Alzheimer’s Disease Detection from Structural Mri Using Fusion Transformer. In Proceedings of the 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), Kolkata, India, 28–31 March 2022. [Google Scholar] [CrossRef]
- Pan, J.; Wang, S. Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer’s Disease. arXiv 2022, arXiv:2206.13393. [Google Scholar] [CrossRef]
- Muzammil, S.R.; Maqsood, S.; Haider, S.; Damaševičius, R. CSID: A novel multimodal image fusion algorithm for enhanced clinical diagnosis. Diagnostics 2020, 10, 904. [Google Scholar] [CrossRef]
- Andreella, A.; Finos, L. Procrustes Analysis for High-Dimensional Data. Psychometrika 2022, 87, 1422–1438. [Google Scholar] [CrossRef] [PubMed]
- Dwivedi, S.; Goel, T.; Tanveer, M.; Murugan, R.; Sharma, R. Multimodal Fusion-Based Deep Learning Network for Effective Diagnosis of Alzheimer’s Disease. IEEE MultiMedia 2022, 29, 45–55. [Google Scholar] [CrossRef]
- Grobbelaar, M.; Phadikar, S.; Ghaderpour, E.; Struck, A.F.; Sinha, N.; Ghosh, R.; Ahmed, M.Z.I. A Survey on Denoising Techniques of Electroencephalogram Signals Using Wavelet Transform. Signals 2022, 3, 577–586. [Google Scholar] [CrossRef]
- Khmag, A. Additive Gaussian noise removal based on generative adversarial network model and semi-soft thresholding approach. Multimed. Tools Appl. 2023, 82, 7757–7777. [Google Scholar] [CrossRef]
- Chen, Z.; Xie, L.; Niu, J.; Liu, X.; Wei, L.; Tian, Q. Visformer: The Vision-friendly Transformer. arXiv 2021, arXiv:2104.12533. [Google Scholar] [CrossRef]
- Khan, S.; Naseer, M.; Hayat, M.; Zamir, S.W.; Khan, F.S.; Shah, M. Transformers in Vision: A Survey. ACM Comput. Surv. 2022, 54, 1–41. [Google Scholar] [CrossRef]
- Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv 2020, arXiv:2010.11929. [Google Scholar] [CrossRef]
- Sandler, M.; Zhmoginov, A.; Vladymyrov, M.; Jackson, A. Fine-tuning Image Transformers using Learnable Memory. arXiv 2022, arXiv:2203.15243. [Google Scholar] [CrossRef]
- Kumar, M.; Ranjan, N.; Chourasia, B. Analysis of Medical Image Fusion Using Transform-Based Function and Neural Network. Ann. Rom. Soc. Cell Biol. 2021, 25, 6333–6347. [Google Scholar]
- Peng, X.; Kong, L.; Han, W.; Wang, S. Multi-Sensor Image Fusion Method for Defect Detection in Powder Bed Fusion. Sensors 2022, 22, 8023. [Google Scholar] [CrossRef] [PubMed]
- Meng, X.; Liu, J.; Fan, X.; Bian, C.; Wei, Q.; Wang, Z.; Liu, W.; Jiao, Z. Multi-Modal Neuroimaging Neural Network-Based Feature Detection for Diagnosis of Alzheimer’s Disease. Front. Aging Neurosci. 2022, 14, 911220. [Google Scholar] [CrossRef] [PubMed]
Modality | Similarity Score |
---|---|
MRI (AD) | 0.779 |
PET (AD) | 0.812 |
MRI (EMCI) | 0.720 |
PET (EMCI) | 0.840 |
MRI (LMCI) | 0.702 |
PET( LMCI) | 0.890 |
Group | MRI | PET | Fused (MRI + PET) |
---|---|---|---|
AD/EMCI (Proposed) | 98.1% | 97.09% | 98.50% |
AD/LMCI (Proposed) | 96.11% | 94.70% | 99.58% |
AD/EMCI (Pre-trained CNN) | 92.40% | 93.56% | 94.03% |
AD/LMCI (Pre-trained CNN) | 93.33% | 93.56% | 95.00% |
Group | MRI | PET |
---|---|---|
AD/EMCI | 81.25% | 93.75% |
AD/LMCI | 81.25% | 93.75% |
References | Method | Modality | Accuracy |
---|---|---|---|
[36] | Deep Neural Network | MRI + PET + | 83.20% |
[37] | Gaussian process | MRI + PET + DTI | 88.10% |
[54] | LassoNet + Neural network | DTI +fMRI | 85.00% |
Proposed Model | Pretrained ViT + pixel image fusion | MRI + PET (PET test data) | 93.75% |
Proposed Model | Pretrained ViT + pixel image fusion | MRI + PET (MRI test data) | 81.25% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Odusami, M.; Maskeliūnas, R.; Damaševičius, R. Pixel-Level Fusion Approach with Vision Transformer for Early Detection of Alzheimer’s Disease. Electronics 2023, 12, 1218. https://doi.org/10.3390/electronics12051218
Odusami M, Maskeliūnas R, Damaševičius R. Pixel-Level Fusion Approach with Vision Transformer for Early Detection of Alzheimer’s Disease. Electronics. 2023; 12(5):1218. https://doi.org/10.3390/electronics12051218
Chicago/Turabian StyleOdusami, Modupe, Rytis Maskeliūnas, and Robertas Damaševičius. 2023. "Pixel-Level Fusion Approach with Vision Transformer for Early Detection of Alzheimer’s Disease" Electronics 12, no. 5: 1218. https://doi.org/10.3390/electronics12051218
APA StyleOdusami, M., Maskeliūnas, R., & Damaševičius, R. (2023). Pixel-Level Fusion Approach with Vision Transformer for Early Detection of Alzheimer’s Disease. Electronics, 12(5), 1218. https://doi.org/10.3390/electronics12051218