Next Article in Journal
Immediate Effects of Distinct Intensities of Transcutaneous Spinal Direct Current Stimulation on Chronic Pain: A Randomized Controlled Trial
Next Article in Special Issue
Trends and Disparities in Parkinson’s Disease Mortality in the United States with Predictions Using Machine Learning
Previous Article in Journal
Exploring Music-Based Interventions for Executive Functioning and Emotional Well-Being in Stroke Rehabilitation: A Scoping Review
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
NeuroSci
Volume 5
Issue 4
10.3390/neurosci5040042
neurosci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Harmonization for Parkinson’s Disease Multi-Dataset T1 MRI Morphometry Classification

by
Mohammed Saqib
1,2 and
Silvina G. Horovitz
2,*
1
University of Pennsylvania, Philadelphia, PA 19104, USA
2
National Institute of Neurological Disorders and Strokes, National Institutes of Health, Bethesda, MD 20892, USA
*
Author to whom correspondence should be addressed.
NeuroSci 2024, 5(4), 600-613; https://doi.org/10.3390/neurosci5040042
Submission received: 30 August 2024 / Revised: 28 October 2024 / Accepted: 31 October 2024 / Published: 29 November 2024
(This article belongs to the Special Issue Parkinson's Disease Research: Current Insights and Future Directions)
Browse Figures
Review Reports Versions Notes

Abstract

Classification of disease and healthy volunteer cohorts provides a useful clinical alternative to traditional group statistics due to individualized, personalized predictions. Classifiers for neurodegenerative disease can be trained on structural MRI morphometry, but require large multi-scanner datasets, introducing confounding batch effects. We test ComBat, a common harmonization model, in an example application to classify subjects with Parkinson’s disease from healthy volunteers and identify common pitfalls, including data leakage. We used a multi-dataset cohort of 372 subjects (216 with Parkinson’s disease, 156 healthy volunteers) from 11 identified scanners. We extracted both FreeSurfer and the determinant of Jacobian morphometry to compare single-scanner and multi-scanner classification pipelines. We confirm the presence of batch effects by running single scanner classifiers which could achieve wildly divergent AUCs on scanner-specific datasets (mean:0.651 ± 0.144). Multi-scanner classifiers that considered neurobiological batch effects between sites could easily achieve a test AUC of 0.902, though pipelines that prevented data leakage could only achieve a test AUC of 0.550. We conclude that batch effects remain a major issue for classification problems, such that even impressive single-scanner classifiers are unlikely to generalize to multiple scanners, and that solving for batch effects in a classifier problem must avoid circularity and reporting overly optimistic results.

1. Introduction

Parkinson’s disease (PD) is the second most prevalent neurodegenerative disorder, afflicting up to 1% of all individuals over the age of 60 [1]. Current diagnosis relies heavily on the first appearance of motor symptoms which will is thought to occur when more than 50% of dopaminergic neurons in the substantia nigra have died [2]. Prodromal biomarkers for PD could allow for early evaluation of neuroprotective therapies with the potential of slowing neurodegeneration [2].
A T1 structural MRI (sMRI) scan is already commonly taken for PD diagnosis to rule out alternate causes for the symptoms, such as multiple system atrophy or progressive supranuclear palsy, which makes it a useful target modality for new PD biomarkers. Regional brain atrophy in PD has been reported even early on in the progression of the disease in the subcortical and basal ganglia regions of the brain [3]. Subcortical nuclei have been found to have altered shapes, with local atrophy in the left putamen and thalamus correlated with right UPDRS motor scores [4]. Different results can be driven by technical differences and even differences in the clinical population that is studied can introduce bias, thus adding to discrepancies across studies. The most robust analyses use large multicenter cohorts of thousands of patients and controls to find group differences in the structural morphometry of the brain [5]. T1 sMRI scans have the potential to find early brain structural changes in PD patients, but the small effect sizes and inconsistencies across studies hamper their use for easy identifications of early biomarkers.
To increase the sample size (and hence power) of statistical analyses involving smaller effect sizes, large datasets or multiple PD MRI datasets from different scanners can be used. Large sample sizes are necessary to sufficiently power neuroscience studies, especially when effect sizes are small [6]; however, while these statistical analyses have found reproducible group differences in sMRI scans, work on using classifiers for individuals has been limited to single-scanner or single datasets. Though statistical analyses have found group effects, adequate classifiers would be more useful for detecting an individual case of PD, as they may help with the identification of interesting biomarkers, especially for data-rich modalities such as MRI.
Prior studies have attempted machine learning approaches, such as classifiers in PD. A key source of open MRI data for many PD studies is the PPMI dataset, which consists of MRI data from multiple sites [7]. Peng 2017 used a combination of structural features as well as resting state functional MRI (rs-fMRI) to achieve an AUC of 0.84 on 69 PD and 103 HV subjects in the PPMI dataset [8]. The analysis used more than 3000 features in this cohort including correlative features, as well as a feature selection process, before running a support vector machine classifier. Though Peng used two nested cross-validations (train, valid, test folds), the analysis was not tested on another scanner.
Long et al. 2012 used both rsfMRI and voxel-based morphometry to achieve an accuracy of 0.87 in a cohort of 19 early PD, and 27 HV subjects [9]. They used a single scanner and sequence and used similar correlative feature approaches to extract more than 6000 features before using feature selection. The group used a leave-one-out cross-validation (LOOCV) approach, but did not test on another scanner, which would also hinder reproducibility on another scanner.
Salvatore et al. 2014 used only sMRI scans and achieved a sensitivity and specificity above 0.9 with 28 PD and 28 HV subjects when testing a single scanner classification task [10]. Similarly high results were achieved for a differential diagnosis task between 28 PD and 28 Pars Supranuclear Palsy (PSP) subjects, with sensitivity and specificity above 0.9. The group also used a LOOCV approach and a PCA approach on normalized brain MRI scans to test only 55 features in an SVM approach.
Huppertz et al. 2016 also considered PD classification in the context of differential diagnosis. An atlas-based volumetry study could identify PD with a sensitivity of 0.87 and specificity of 0.85 compared to PSP and MSA [11]. Huppertz indicated that PD vs. HV classification remains challenging, as brain atrophy between the two groups overlap, but found that PSP and MSA were easier due to relatively more distinct atrophy patterns.
The ENIGMA group organized the largest multi-site statistical analysis of sMRI morphometry using models where the site was a random effect [5]. While work found useful group effects only possible with such a huge sample size, individual overlap of subjects was high and effect sizes were small for early stages of PD. Of note, no classifier was tested.
MRI scanners and differing protocols can introduce significant variability into a study design. Researchers used multiple rs-fMRI datasets for determining biomarkers and finding statistically significant results and found that results found in single datasets often did not transfer to other datasets [12]. Different sMRI scanners are known to introduce additional variability to estimates of brain volume for even the same subject, due to effects such as gradient distortion [13], or different scanner or scanning protocols [14]. Of note, even differences in clinical populations between sites can introduce bias as neurobiological cofactors [15]. ComBat is the standard practice to harmonize MRI measurements across different scanners. ComBat (alternatively called NeuroComBat when applied to radiomics) handles variance as batch effects [16] and there is now significant work to extend the algorithm [17,18].
However, there is a lack of literature on the impact of NeuroComBat on classification and prediction problems, and the inherent assumptions on how any harmonization strategy is applied.
Tafuri 2022 used NeuroComBat in a LOOCV approach on a multisite PPMI dataset approach to achieve an AUC score of 0.77 using an SVM LASSO approach, which was an improvement over the 0.71 score without NeuroComBat [19]. NeuroComBat was applied before a nested CV approach, which allows for easier processing and keeps the same model between folds, but also means test set data must be known beforehand to create the NeuroComBat model.
In general, there remains insufficient study on the use of multiple sMRI datasets for PD classification. Though group effect studies are useful, we hope to study the feasibility of classification for PD. This is even though there are now various multi-center MRI protocols for the identification of biomarkers for PD [5,7]. Here, we look at the strengths and limitations of a PD sMRI classifier trained on multiple datasets at once.
We hypothesize that the scanner-specific classifier will perform well for its own scanner but will perform poorly for other scanners. We also hypothesize that scanner-specific classifiers will excel compared to a multi-scanner classifier’s scanner-specific metrics. However, after harmonization, we expect scanner-specific metrics to match or exceed single-scanner classifier metrics.

2. Materials and Methods

2.1. Datasets

We used datasets of T1 sMRI scans specifically targeted for PD. We collected three datasets at our institution [20,21,22], and also processed three public datasets. The three public datasets included the PPMI dataset [7] (See https://www.ppmi-info.org/access-data-specimens/download-data, accessed on 1 August 2021), the Neurocon dataset [12], and Tao Wu dataset [12]. Of note, the PPMI dataset itself consists of multiple scanners and sites, which we could identify with metadata. We used T1 PPMI data that was available as of August 2021. Parkinson’s disease diagnosis criterion varies according to dataset. For NIH-specific datasets, neurologists used the UK Parkinson’s Disease Society Brain Bank Clinical Diagnosis Criteria [23].
The device serial ID from DICOM file metadata uniquely identified a specific site/scanner for each scan. Multiple scans from different patients with the same scanner shared this serial number. The Neurocon and Tao Wu (See https://fcon_1000.projects.nitrc.org/indi/retro/parkinsons.html, accessed on 1 August 2021.) datasets did not include DICOM file metadata, so we assumed that the scans came from the same scanner for each dataset. We only included a device if it had at least 10 HV and 10 PD subjects. When more than one visit was available, we also used only the first scan for each subject, to avoid reusing the same subject and introducing additional bias. Our final combined cohort was 372 subjects, including 216 PDs and 156 HVs, from 11 sites, including 6 of the best-represented device serial IDs from PPMI (See Supplementary Figures S1 and S2). Images acquisition parameters are described in Supplementary Materials.

2.2. Terminology

In this work, we name classifiers trained on all scans from all scanners as multi-scanner classifiers. In contrast, we also look at classifiers trained specifically on a single scanner as a single-scanner classifier.
Single-scanner classifiers will not need to be harmonized during training, as they are trained on only scans from the single scanner.
For multi-scanner classifiers, we look at scanner-specific metrics to look at how a multi-scanner classifier performed on a specific scanner. We can compare a scanner-specific metric (from the performance of a multi-scanner classifier) against a single-scanner classifier’s metric to see if our approach is able to harmonize data between classifiers.

2.3. Data Processing

Morphometry is the study of a subject brain using physical measurements of the sMRI scan. There are morphometric approaches implemented in various software pipelines [24] to capture variability in brain shape and size. We tested FreeSurfer [25] (FS) and AFNI [26] to process sMRI and extract morphometric measurements of the brain. Details of processing are shown in Supplementary Materials and Figure 1. This processing creates quantitative measures characterizing the shape of the brain.

2.4. NeuroComBat Algorithm

The ComBat algorithm was originally designed to solve differences in batch effects in microarray datasets [28], but is effective for neuroimaging site-effects, such as differing scanners, sequences, or acquisition procedures [15] and is renamed NeuroComBat in such cases. To harmonize between scans from different scanners, we create a NeuroComBat model with age and sex covariates and fitted to the data using an empirical Bayes algorithm provided in previously published software [16]. The NeuroComBat model is within our full pipeline, and is trained independently of the classifier, but with necessary dependency on the train, valid, and test set used for the classifier.
This model fits with data from a subset of scans from our pipeline (Figure 2), but is used to transform all scans to be harmonized. We test both with and without the NeuroComBat model to see if it is useful in the multi-scanner classifier. We include both age and sex as covariates alongside the scanner identifier to generate the model. When training scanner-specific classifiers, NeuroComBat is not used. We include both PD and HV subjects when fitting the NeuroCombat model, as ComBat algorithms require ~20 scans per scanner and should attempt to model both site-effects from the different scanners and site-effects due to differences in the disease population (i.e., neurobiological cofactors) from differing populations [15].
We do note that while the group can be included as a possible covariate for some NeuroComBat approaches, including it for a machine learning approach leads to information leaking; we would need to know the group label ahead of time to harmonize data from an unseen set. We test a NeuroComBat approach with a group covariate, regardless of where the NeuroComBat model is fit using train set data only, to confirm information leakage in our model.
As shown in Figure 2, the NeuroComBat model may be fit with different splits of the data, based on assumptions of the eventual application of a multi-set scanner. We discuss these results and assumptions later in the results and discussion, to compare if NeuroComBat is successful for a classifier.

2.5. Data Augmentation/Class Balancing

Imbalanced datasets may hamper the performance of a classifier, so techniques around class balancing and data augmentation have been developed [29]. Most datasets and scanners that we investigated had far more PD subjects compared to HV (See Supplementary Figure S1). In addition, some scanners had nearly twice as many subjects as others. Classifiers that are fed unbalanced data may have poor performance in correctly avoiding a positive prediction for HV subjects (specificity). Scanners that are not well-represented in the training data may also suffer. We hypothesized that balancing the data between scanners could provide additional support for data harmonization, by preventing a multi-scanner classifier from becoming biased on a single scanner.
As per Figure 3, we only balance and augment the training data. It is crucial to maintain the same class distribution in the validation set, so we neither augment nor balance the valid split nor do we use validation data as part of the balance and augmentation step for the training set. We test SMOTE [30] and Random Undersampling (RUS) and compared to those without balancing or augmentation.
As a hyperparameter within our hyperparameter search, we looked at whether applying a balancing strategy with regard to the scanner and group versus balancing only with regard to the group. We also tested introducing artificial scans by flipping brains left to right to increase our sample size. We tested all these options as a hyperparameter within our hyperparameter search.

2.6. Classifiers

We implemented classifiers using the Sklearn Python library [31] and used logistic regression (LR), random forest (RF), support vector machines (SVM), and extreme gradient boosting (XGBoost). We first stratified our datasets and set aside 20% of our dataset (71 scans) ahead of time for a test split (stratified by scanner, age, sex, and group (PD or HV)). The remaining 80% (301 scans) were used to train a classifier and validate a hyperparameter combination.
During a training step, we used 5-fold cross-validation to make train and validation splits (Figure 2) to test for the best classifier hyperparameters for multi-scanner classifiers, and a LOOCV for single-scanner classifiers. As our data have many more PD scans compared to HV, we balanced our training data and tested different methods from a library used for data augmentation [32].
We used different classifier pipelines to investigate whether we could successfully separate PD and HV subjects from cohorts collected at multiple scanners. We tested multiple hyperparameters and implemented a hyperparameter search pipeline. For each combination of class balancing, data augmentation, classifier, and datasets, we ran a random search of classifier hyperparameters to find the best classifier (See Supplemental Materials). We captured the ROC-AUC metric and used that to determine the best hyperparameter combination. To investigate whether multi-scanner classifiers were biased by the scanner, we took the scanner-specific metrics and compared them against the single-scanner classifier metrics.

2.7. Single Scanner Classifiers vs. Full Dataset

To compare the performance of classification between scanners, we first tested the dataset as a whole and found the multi-scanner classifier that provided the best scores without NeuroComBat. We then included NeuroComBat in a multi-scanner classifier, to repeat the full classification, and hyperparameter search procedures to evaluate whether the use of the Neuro-ComBat algorithm. We checked and confirmed if the multi-scanner performance improved.
We then compared results by limiting the training-validation split of our dataset to scans from an individual scanner. Using this device subset of the train-valid split of the data, we applied the same hyperparameter search to find the best classifier for this single scanner for each specified device. When testing with a single-scanner classifier, we could not use NeuroComBat as there was only a single site. We compared the best multi-scanner performances and generated scanner-specific metrics to compare to the single-scanner classifier.

2.8. Evaluation

We chose the best set of hyperparameters out of 10,000 randomly chosen hyperparameter sets using the mean 5-fold validation AUC and determined a standard deviation of the validation AUC [33].
To evaluate the results, we used AUC-ROC to characterize the performance of classifiers. We could specify the entire train-validation set and use leave-one-out (LOO) with a classifier to predict each scan when using the single-scanner classifier, as each scanner did not have enough scans. For multi-scanner classifiers, we used a 5-fold CV approach and generated error bounds for validation metrics using the folds.
For test set metrics, we only picked the best classifier from the hyperparameter search for each classifier type (SVM, RF, LR, XGBoost), dataset, and augmentation strategy and ran the predictions for the test set after training on the train-valid set. We compared test set metrics based on whether we harmonized data only using the train-valid set or with both the train-valid and test set. If the classifier can predict successfully with data harmonized only by the train-valid set, it suggests that scanner variability can be easily calculated by modeling site effects in only train-valid data. However, if the metrics work best with scanner variability characterized by both the train-valid and test set, it suggests that scanner variability cannot be easily harmonized by NeuroComBat in an unseen set.

3. Results

3.1. Subject Demographics

Datasets showed various demographic information, as shown in Table 1. Of note, PPMI datasets showed earlier stage Parkinson’s disease subjects compared to other datasets. NIH datasets showed later stage Parkinson’s disease subjects compared to other datasets. Most datasets had more PD subjects compared to HV subjects.

3.2. Single Scanner Classifiers vs. Full Dataset Results

As shown in Table 2, classifiers trained with the NeuroComBat model where the group was used were more performant compared to classifiers trained without; the top three validation AUCs were achieved by algorithms that modeled group effect within NeuroComBat. Both validation and test AUC were significantly higher than a classifier without the group. Even though the NeuroComBat algorithm had parameters that fit with data only from the train set, it was likely that the group, which was necessary later to harmonize the valid and test set, inflated the metrics.
We compared against algorithms that either did not use NeuroComBat or did not include the group as a covariate, as shown in Table 3. The top algorithm, by validation AUC, without the group covariate was a logistic regression classifier with a marked drop in AUC.
We tested multiple algorithms and included results in the supplemental results. (See Supplemental Table S1 and Figures S4 and S5).
Single scanner classifier results were mostly performant, except for some PPMI scanners, with an average validation score across each scanner of 0.651 ± 0.144, as shown in Table 4. Some single scanner classifiers could achieve results that approached clinical usefulness, but likely would fail to generalize to other scanners or protocols. We compared the single-scanner performance with the scores from the multi-scanner classifiers within Table 4 and found that single-scanner classifiers showed both increased and decreased metrics across a wide variety of hyperparameter choices.
Analysis for many of the results seems to indicate that augmentation strategy or balancing strategy did not matter for either the most performant algorithm that used the group covariate or the most performant algorithm that did not (see Supplemental Figure S5).

4. Discussion

Machine learning requires significant data to power useful analyses, but is susceptible to batch effects in unharmonized data, regardless of algorithm. This batch effect/harmonization problem is also known as a domain shift, domain adaptation problem [34], inter-scanner variability [35], and site effects [15].
Towards this end, our work applies sMRI morphometric measurements to identify potential biomarkers in multiple scanners, and multiple datasets, to identify the sources of variability. We attempt methodologies to harmonize between scanners and cohorts but find that discrepancies in classifier performance persist. To our knowledge, we are the first to attempt to look at how statistical effects for sMRI may or may not generalize, and whether classifiers can be successful with multiple datasets and scanners. We also showcase some approaches which ostensibly worked well across all scanners but may have validity issues.
We compared our work on a single scanner classifier and found that the detection of Parkinson’s varied widely depending on the scanner. The Neurocon dataset provided the best single scanner classifier metrics, perfectly predicting all scans with LOO validation; we note that LOO in such a small sample size with so many features likely indicates overfitting, and a larger sample size would likely have worse metrics. Other single scanner classifiers failed to generalize at all (i.e., PPMI scanner 1). Moreover, scanner-specific metrics from the multi-scanner classifier do not always correspond to the single-scanner classifier performance.
Our work identifies that the best models used the group variable in a NeuroComBat model to remove batch effect variability while ostensibly keeping group differences intact, leading to extremely high AUC values over 0.9. The correction of the data was conducted using a NeuroComBat model only trained on training data, but it is still likely that this approach represents some data leakage regardless; we needed to input the test label to correct data to perform prediction on an otherwise unseen test set, which defeats the purpose of a classifier.
The ComBat algorithm (or NeuroComBat, when it is applied to neuroimaging data) models site effects and produces a set of harmonized data given both a site variable and other covariables. A key issue in the batch effect problem is issues of neurobiological differences between sites, which may be due to differences in inclusion/exclusion criteria between studies [15]. Modeling neurobiological differences between sites, especially with regard to disease between sites, remains hard when the neurobiological difference is the group effect we predict. NeuroComBat may be inappropriate for classifiers, as a result, especially when the group effect is suspected to covary between sites.
We note that ComBat variants have flourished, to extend the original algorithm [15]. Though many of these variants solve use cases we did not consider, such as longitudinal changes, one interesting algorithm we did not test was modified-ComBat, which picks a specific site as a base “average” that is modified at other sites [36].
We also attempted to see if controlling for the number of scans between scanners in our multi-scanner classifier could provide better results. For example, some scanners had nearly twice as many subjects compared to other scanners, and we hypothesized that not correcting for overrepresentation could also reduce performance. Supplemental Figures S4 and S5 indicate that balancing between classes or scanners or both did not lead to drastically different metrics during the hyperparameter search.
In a previous multicenter study, statistical analyses on structural morphometry between Parkinson’s disease datasets were able to use a model similar to NeuroComBat to remove site effects and identify the progression of the disease [5]. That study used a portion of the same datasets (i.e., PPMI, Tao Wu, Neurocon) as our study and modeled group differences using the site as a random effect in a linear mixed-effects model. Robust group effects were identified, suggesting NeuroComBat and similar statistical tools may be better suited for identifying group effects rather than classification tasks. It is encouraging, regardless, that analyses of our classifiers identified that the best features for classification seemed to match early features identified in this study (see Supplemental Table S2). Features involving the entorhinal cortex were among the top five features for the multi-scanner classifier model using the group covariate in NeuroComBat, and were similarly identified as being impacted early in Parkinson’s disease. Changes in the substantia nigra DOJ were also important, especially for the multi-scanner classifier model that did not include group as a covariate.
There is no clear correspondence between UPDRS3, H&Y scores, disease duration, and classifier performance. Single scanner classifiers for PPMI scans were generally worse performing compared to other scanners. As PPMI scans were mostly within the first year of diagnosis, it is expected that little time would have passed for further degeneration, which would have made classification difficult. Many of the scans from the NIH datasets, or the Tao Wu and Neurocon datasets can be considered early PD, though they have had some further time pass since diagnosis.
PPMI uses a harmonized sequence to control for some batch effects, so the failure of a multi-scanner classifier to match single-scanner classifier models suggests further work is necessary. We note that perhaps a PPMI-specific classifier could be useful to confirm sequence harmonization can control all batch effects. Other batch effects, such as differences in scanner models or differences in the device’s magnetic fields could all contribute, despite harmonizing for sequence.
Some previous Parkinson’s disease classifier works achieved high automatic classification metrics but used a single scanner site for all scans. One study, focusing on the task of differential diagnosis using classifiers, achieved a 65.7 accuracy for detecting PD against controls, suggesting our work is competitive [11]. A review of deep learning classifiers for PD in specific cohorts found classification accuracies ranging from 95 to 100%, though many were from studies tested in smaller cohort sizes [37]. As identified in our own work, data from a single scanner with sufficiently limited cohort size may produce a classifier that has perfect metrics, but which may not hold in larger cohorts (i.e., the Neurocon dataset).
Applications of machine learning techniques (and, in particular, classification tasks) to brain diseases are increasing exponentially and many methodologies use deep learning networks [38]. Some recent work has been conducted to model batch effects using deep learning through domain adaptation methods, though more work with regard to MRI and Parkinson’s disease is necessary [34]. A new method has recently integrated NeuroComBat into a training pipeline specifically to avoid data leakage [39]. Regardless, deep learning methods should remain cautious of the same circular reasoning problems that may occur with the usage of more traditional methods to remove site effects.
Batch effects are known to have multiple sources, including scanner platform, sequence and acquisition procedures, differing pipelines, and neurobiological cofactors [15]. Though all data in this work were run with the same postprocessing pipeline, controlling for other factors remains difficult. Of note, sources of batch effects that are neurobiological cofactors, including cofactors that vary directly with the disease, remain the most difficult to control within a classifier pipeline. Parkinson’s disease can have differing diagnosis criteria, and though confirmation with a DaTScan remains among the best ways to rule out other similar diseases with Parkinsonism symptoms, we note only the PPMI datasets used this criterion [40]. Other datasets, including our internal datasets, used clinical symptoms and associated differential diagnosis to rule out other diseases. The heterogeneity of Parkinson’s disease also could act as yet another neurobiological cofactor. We note that including Parkinson’s disease status in our NeuroComBat implementation pipeline during training leads to significant improvements. Though this could be possibly due to modeling the neurobiological source of a batch effect, it also represents a very clear data leakage in the model. Machine learning analyses must avoid such circular inference to remain valid in real-world usage.
A key limitation in our work is the need for scanners with enough data. We note that many of the scanners had significantly skewed datasets, with many more PD subjects compared to HV. We necessarily limited ourselves to scanners with at least 10 HV, to properly have enough subjects to fit NeuroComBat models. Though the NeuroComBat algorithm should be resilient to small sample sizes, we found that our algorithm fluctuated wildly when including a scanner with few scans, likely due to the inability to estimate model parameters. We also note that our method requires the estimation of scanner-specific parameters ahead of time to predict a scan; completely novel scanners outside of the original training dataset cannot be handled. A further extension of our work would be to train a model to predict scanner-specific NeuroComBat model variables, similar to other works in the field [41].
Deep learning remains an exciting pathway for further exploration as well, and new methods such as Cycle-Generative Adversarial Networks (c-GANS) and Variational Auto Encoders (VAEs) [34,42]. Such networks use clever loss functions to transfer or infer images in a new site (domain adaptation) but often require significant data. Though our work was limited, a larger study could potentially use enough data to make an exploration of a deep learning framework viable. We also note that a transfer learning approach (using a previously trained model from another group) could also be a potential foundation, obviating the need for a larger dataset.
Our work finds that tweaking individual hyperparameters (choices in how we design our classification methods) does not matter as much as the initial dataset. If the initial dataset is biased by site effects, the site effects will dramatically affect classifier performance.

5. Conclusions

Our work was able to achieve competitive classifier metrics in small single-scanner classifiers but suggests that such metrics do not translate to larger multi-scanner datasets. Application of NeuroComBat into a classifier is challenging and requires thought and consideration to avoid overly inflating metrics due to circularity.

Supplementary Materials

The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/neurosci5040042/s1. Figure S1. Dataset breakdown into final processed MRI scans. Figure S2: AFNI preprocessing pipeline. Figure S3: The proposed pipeline for determining the single scanner classifier. Figure S4: The analysis of the validation scores from the hyperparameter search. Figure S5: The analysis of the validation scores from the hyperparameter search. Table S1: Classifier results for training on the full dataset. Table S2: The most important features for the classifiers.

Author Contributions

Conceptualization, M.S. and S.G.H.; methodology, M.S. and S.G.H.; software, M.S.; resources, M.S. and S.G.H.; data curation, M.S. and S.G.H.; writing—original draft preparation, M.S.; writing—review and editing, M.S. and S.G.H.; visualization, M.S.; supervision, S.G.H.; project administration, S.G.H.; funding acquisition, S.G.H. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the NINDS-IRP program.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Board of NIH (protocol code NCT01496599 2011-12-17). Data from repositories had the Institutional Review approval of the respective institutions where datasets were collected.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author due to privacy and legal concerns.

Acknowledgments

This project was supported by the National Institute of Neurological Disorders and Stroke Intramural Research Program. Data used in the preparation of this article were obtained from the Parkinson’s Progression Markers Initiative (PPMI) database (www.ppmi-info.org/access-data-specimens/download-data), RRID:SCR 006431. For up-to-date information on the study, visit www.ppmi-info.org. Data was accessed 8 April 2021.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Tysnes, O.-B.; Storstein, A. Epidemiology of Parkinson’s Disease. J. Neural Transm. 2017, 124, 901–905. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  2. Miller, D.B.; O’Callaghan, J.P. Biomarkers of Parkinson’s Disease: Present and Future. Metabolism 2015, 64, S40–S46. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  3. Sarasso, E.; Agosta, F.; Piramide, N.; Filippi, M. Progression of Grey and White Matter Brain Damage in Parkinson’s Disease: A Critical Review of Structural MRI Literature. J. Neurol. 2021, 268, 3144–3179. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  4. Nemmi, F.; Sabatini, U.; Rascol, O.; Péran, P. Parkinson’s Disease and Local Atrophy in Subcortical Nuclei: Insight from Shape Analysis. Neurobiol. Aging 2015, 36, 424–433. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  5. Laansma, M.A.; Bright, J.K.; Al-Bachari, S.; Anderson, T.J.; Ard, T.; Assogna, F.; Baquero, K.A.; Berendse, H.W.; Blair, J.; Cendes, F.; et al. International Multicenter Analysis of Brain Structure Across Clinical Stages of Parkinson’s Disease. Mov. Disord. 2021, 36, 2583–2594. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  6. Button, K.S.; Ioannidis, J.P.A.; Mokrysz, C.; Nosek, B.A.; Flint, J.; Robinson, E.S.J.; Munafò, M.R. Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience. Nat. Rev. Neurosci. 2013, 14, 365–376. [Google Scholar] [CrossRef] [Scilit]
  7. Parkinson Progression Marker Initiative The Parkinson Progression Marker Initiative (PPMI). Prog. Neurobiol. 2011, 95, 629–635. [CrossRef] [Scilit]
  8. Peng, B.; Wang, S.; Zhou, Z.; Liu, Y.; Tong, B.; Zhang, T.; Dai, Y. A Multilevel-ROI-Features-Based Machine Learning Method for Detection of Morphometric Biomarkers in Parkinson’s Disease. Neurosci. Lett. 2017, 651, 88–94. [Google Scholar] [CrossRef] [Scilit]
  9. Long, D.; Wang, J.; Xuan, M.; Gu, Q.; Xu, X.; Kong, D.; Zhang, M. Automatic Classification of Early Parkinson’s Disease with Multi-Modal MR Imaging. PLoS ONE 2012, 7, e47714. [Google Scholar] [CrossRef] [Scilit]
  10. Salvatore, C.; Cerasa, A.; Castiglioni, I.; Gallivanone, F.; Augimeri, A.; Lopez, M.; Arabia, G.; Morelli, M.; Gilardi, M.C.; Quattrone, A. Machine Learning on Brain MRI Data for Differential Diagnosis of Parkinson’s Disease and Progressive Supranuclear Palsy. J. Neurosci. Methods 2014, 222, 230–237. [Google Scholar] [CrossRef] [Scilit]
  11. Huppertz, H.-J.; Möller, L.; Südmeyer, M.; Hilker, R.; Hattingen, E.; Egger, K.; Amtage, F.; Respondek, G.; Stamelou, M.; Schnitzler, A.; et al. Differentiation of Neurodegenerative Parkinsonian Syndromes by Volumetric Magnetic Resonance Imaging Analysis and Support Vector Machine Classification. Mov. Disord. 2016, 31, 1506–1517. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  12. Badea, L.; Onu, M.; Wu, T.; Roceanu, A.; Bajenaru, O. Exploring the Reproducibility of Functional Connectivity Alterations in Parkinson’s Disease. PLoS ONE 2017, 12, e0188196. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  13. Jovicich, J.; Czanner, S.; Greve, D.; Haley, E.; van der Kouwe, A.; Gollub, R.; Kennedy, D.; Schmitt, F.; Brown, G.; MacFall, J.; et al. Reliability in Multi-Site Structural MRI Studies: Effects of Gradient Non-Linearity Correction on Phantom and Human Data. NeuroImage 2006, 30, 436–443. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  14. Han, X.; Jovicich, J.; Salat, D.; van der Kouwe, A.; Quinn, B.; Czanner, S.; Busa, E.; Pacheco, J.; Albert, M.; Killiany, R.; et al. Reliability of MRI-Derived Measurements of Human Cerebral Cortical Thickness: The Effects of Field Strength, Scanner Upgrade and Manufacturer. NeuroImage 2006, 32, 180–194. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  15. Bayer, J.M.M.; Thompson, P.M.; Ching, C.R.K.; Liu, M.; Chen, A.; Panzenhagen, A.C.; Jahanshad, N.; Marquand, A.; Schmaal, L.; Sämann, P.G. Site Effects How-to and When: An Overview of Retrospective Techniques to Accommodate Site Effects in Multi-Site Neuroimaging Analyses. Front. Neurol. 2022, 13, 923988. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  16. Fortin, J.-P.; Cullen, N.; Sheline, Y.I.; Taylor, W.D.; Aselcioglu, I.; Cook, P.A.; Adams, P.; Cooper, C.; Fava, M.; McGrath, P.J.; et al. Harmonization of Cortical Thickness Measurements across Scanners and Sites. NeuroImage 2018, 167, 104–120. [Google Scholar] [CrossRef] [Scilit]
  17. Chen, A.A.; Beer, J.C.; Tustison, N.J.; Cook, P.A.; Shinohara, R.T.; Shou, H.; Initiative, T.A.D.N. Mitigating Site Effects in Covariance for Machine Learning in Neuroimaging Data. Hum. Brain Mapp. 2022, 43, 1179–1195. [Google Scholar] [CrossRef] [Scilit]
  18. Reynolds, M.; Chaudhary, T.; Torbati, M.E.; Tudorascu, D.L.; Batmanghelich, K. Alzheimer’s Disease Neuroimaging Initiative. ComBat Harmonization: Empirical Bayes versus Fully Bayes Approaches. bioRxiv 2022. [Google Scholar] [CrossRef] [Scilit]
  19. Tafuri, B.; Lombardi, A.; Nigro, S.; Urso, D.; Monaco, A.; Pantaleo, E.; Diacono, D.; De Blasi, R.; Bellotti, R.; Tangaro, S.; et al. The Impact of Harmonization on Radiomic Features in Parkinson’s Disease and Healthy Controls: A Multicenter Study. Front. Neurosci. 2022, 16, 1012287. [Google Scholar] [CrossRef] [Scilit]
  20. Tinaz, S.; Lauro, P.; Hallett, M.; Horovitz, S.G. Deficits in Task-Set Maintenance and Execution Networks in Parkinson’s Disease. Brain Struct. Funct. 2016, 221, 1413–1425. [Google Scholar] [CrossRef] [Scilit]
  21. Tinaz, S.; Lauro, P.M.; Ghosh, P.; Lungu, C.; Horovitz, S.G. Changes in Functional Organization and White Matter Integrity in the Connectome in Parkinson’s Disease. Neuroimage Clin. 2017, 13, 395–404. [Google Scholar] [CrossRef] [Scilit]
  22. Hattori, T.; Reynolds, R.; Wiggs, E.; Horovitz, S.G.; Lungu, C.; Chen, G.; Yasuda, E.; Hallett, M. Neural Correlates of Working Memory and Compensation at Different Stages of Cognitive Impairment in Parkinson’s Disease. Neuroimage Clin. 2022, 35, 103100. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  23. Hughes, A.J.; Ben-Shlomo, Y.; E Daniel, S.; Lees, A.J. What features improve the accuracy of clinical diagnosis in Parkinson’s disease: A clinicopathologic study. Neurology 1992, 57, S34–S38. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  24. Ashburner, J.; Friston, K. Chapter 6. Morphometry. In Human Brain Function; Elsevier: Amsterdam, The Netherlands, 2004. [Google Scholar]
  25. Fischl, B. FreeSurfer. Neuroimage 2012, 62, 774–781. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  26. Cox, R.W. AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance Neuroimages. Comput. Biomed. Res. 1996, 29, 162–173. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  27. Malone, I.B.; Leung, K.K.; Clegg, S.; Barnes, J.; Whitwell, J.L.; Ashburner, J.; Fox, N.C.; Ridgway, G.R. Accurate Automatic Estimation of Total Intracranial Volume: A Nuisance Variable with Less Nuisance. NeuroImage 2015, 104, 366–372. [Google Scholar] [CrossRef] [Scilit]
  28. Johnson, W.E.; Li, C.; Rabinovic, A. Adjusting Batch Effects in Microarray Expression Data Using Empirical Bayes Methods. Biostatistics 2007, 8, 118–127. [Google Scholar] [CrossRef] [Scilit]
  29. He, H.; Garcia, E.A. Learning from Imbalanced Data. IEEE Trans. Knowl. Data Eng. 2009, 21, 1263–1284. [Google Scholar] [CrossRef] [Scilit]
  30. Chawla, N.V.; Bowyer, K.W.; Hall, L.O.; Kegelmeyer, W.P. SMOTE: Synthetic Minority Over-Sampling Technique. J. Artif. Intell. Res. 2002, 16, 321–357. [Google Scholar] [CrossRef] [Scilit]
  31. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-Learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
  32. Lemaître, G.; Nogueira, F.; Aridas, C.K. Imbalanced-Learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning. J. Mach. Learn. Res. 2017, 18, 1–5. [Google Scholar]
  33. Bergstra, J.; Bengio, Y. Random Search for Hyper-Parameter Optimization. J. Mach. Learn. Res. 2012, 13, 281–305. [Google Scholar]
  34. Guan, H.; Liu, M. Domain Adaptation for Medical Image Analysis: A Survey. arXiv 2021, arXiv:2102.09508. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  35. Pomponio, R.; Erus, G.; Habes, M.; Doshi, J.; Srinivasan, D.; Mamourian, E.; Bashyam, V.; Nasrallah, I.M.; Satterthwaite, T.D.; Fan, Y.; et al. Harmonization of Large MRI Datasets for the Analysis of Brain Imaging Patterns throughout the Lifespan. NeuroImage 2020, 208, 116450. [Google Scholar] [CrossRef] [Scilit]
  36. Stein, C.K.; Qu, P.; Epstein, J.; Buros, A.; Rosenthal, A.; Crowley, J.; Morgan, G.; Barlogie, B. Removing Batch Effects from Purified Plasma Cell Gene Expression Microarrays with Modified ComBat. BMC Bioinform. 2015, 16, 63. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  37. Noor, M.B.T.; Zenia, N.Z.; Kaiser, M.S.; Mamun, S.A.; Mahmud, M. Application of Deep Learning in Detecting Neurological Disorders from Magnetic Resonance Images: A Survey on the Detection of Alzheimer’s Disease, Parkinson’s Disease and Schizophrenia. Brain Inform. 2020, 7, 1–21. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  38. Segato, A.; Marzullo, A.; Calimeri, F.; De Momi, E. Artificial Intelligence for Brain Diseases: A Systematic Review. APL Bioeng. 2020, 4, 041503. [Google Scholar] [CrossRef] [Scilit]
  39. Marzi, C.; Giannelli, M.; Barucci, A.; Tessa, C.; Mascalchi, M.; Diciotti, S. Efficacy of MRI Data Harmonization in the Age of Machine Learning: A Multicenter Study across 36 Datasets. Sci. Data 2024, 11, 115. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  40. Bega, D.; Kuo, P.H.; Chalkidou, A.; Grzeda, M.T.; Macmillan, T.; Brand, C.; Sheikh, Z.H.; Antonini, A. Clinical Utility of DaTscan in Patients with Suspected Parkinsonian Syndrome: A Systematic Review and Meta-Analysis. NPJ Park. Dis. 2021, 7, 1–8. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  41. Garcia-Dias, R.; Scarpazza, C.; Baecker, L.; Vieira, S.; Pinaya, W.H.L.; Corvin, A.; Redolfi, A.; Nelson, B.; Crespo-Facorro, B.; McDonald, C.; et al. Neuroharmony: A New Tool for Harmonizing Volumetric MRI Data from Unseen Scanners. Neuroimage 2020, 220, 117127. [Google Scholar] [CrossRef] [Scilit]
  42. Seoni, S.; Shahini, A.; Meiburger, K.M.; Marzola, F.; Rotunno, G.; Acharya, U.R.; Molinari, F.; Salvi, M. All You Need Is Data Preparation: A Systematic Review of Image Harmonization Techniques in Multi-Center/Device Studies for Medical Support Systems. Comput. Methods Programs Biomed. 2024, 250, 108200. [Google Scholar] [CrossRef] [Scilit] [PubMed]
Neurosci 05 00042 g001
Figure 1. sMRI scan morphometry was calculated with two separate methods, a determinant of the Jacobian method, and FreeSurfer-based morphometry. AAL3 Determinant of Jacobian (DOJ) measures expansion/contraction for each voxel to fit a template, while FreeSurfer (FS) uses physical measurements based on constructing surfaces of the gray and white matter. Of note, some FreeSurfer values were corrected for estimated Total Intracranial Volume (eTIV) by simple division. [27] The actual eTIV is passed as a feature for FS features as well. DOJ values were subtracted by 1 to center the DOJ values at 0 (i.e., no regional increase or decrease).
Figure 1. sMRI scan morphometry was calculated with two separate methods, a determinant of the Jacobian method, and FreeSurfer-based morphometry. AAL3 Determinant of Jacobian (DOJ) measures expansion/contraction for each voxel to fit a template, while FreeSurfer (FS) uses physical measurements based on constructing surfaces of the gray and white matter. Of note, some FreeSurfer values were corrected for estimated Total Intracranial Volume (eTIV) by simple division. [27] The actual eTIV is passed as a feature for FS features as well. DOJ values were subtracted by 1 to center the DOJ values at 0 (i.e., no regional increase or decrease).
Neurosci 05 00042 g001
Neurosci 05 00042 g002
Figure 2. (A) NeuroComBat is the application of the ComBat algorithm to neurological data. The algorithm takes site as a batch effect, along with other covariates that may help explain variance from an input dataset, and is used to harmonize the full train/valid/test set. The NeuroComBat algorithm can be applied with different splits, depending on the final assumptions of the multi-scanner classifier. We test learning the NeuroComBat model on just the train set (A), on the train-valid set (B), and all the data (C) as a comparison and to see if a NeuroComBat model fit on only classifier-trained data are sufficient to remove site-effects such that a multi-scanner classifier. We use the model to harmonize all data in our preprocessing pipeline. We compare against not using any NeuroComBat model (D). Our NeuroComBat model is trained on the scanner as the site-effect variable to be removed, with age and sex covariates. As a control, we compared NeuroCombat models against simply training data without any NeuroCombat model (E). We additionally test a NeuroComBat model that is fit with a group covariate on a train set only, though such a model still introduces data circularity when harmonizing validation and testing data (F).
Figure 2. (A) NeuroComBat is the application of the ComBat algorithm to neurological data. The algorithm takes site as a batch effect, along with other covariates that may help explain variance from an input dataset, and is used to harmonize the full train/valid/test set. The NeuroComBat algorithm can be applied with different splits, depending on the final assumptions of the multi-scanner classifier. We test learning the NeuroComBat model on just the train set (A), on the train-valid set (B), and all the data (C) as a comparison and to see if a NeuroComBat model fit on only classifier-trained data are sufficient to remove site-effects such that a multi-scanner classifier. We use the model to harmonize all data in our preprocessing pipeline. We compare against not using any NeuroComBat model (D). Our NeuroComBat model is trained on the scanner as the site-effect variable to be removed, with age and sex covariates. As a control, we compared NeuroCombat models against simply training data without any NeuroCombat model (E). We additionally test a NeuroComBat model that is fit with a group covariate on a train set only, though such a model still introduces data circularity when harmonizing validation and testing data (F).
Neurosci 05 00042 g002
Neurosci 05 00042 g003
Figure 3. Overview of a multi-scanner classifier pipeline with the proposed selection for removing site-effects which separate between scanners and select for PD. After training and validating to find the best hyperparameters, the final metric can be taken from the test set. We use the ROC-AUC score and the balanced accuracy score as metrics to measure performance. We use a similar pipeline for the single-scanner classifier, although multi-scanner steps are removed (see Supplemental Info, Supplemental Figure S3).
Figure 3. Overview of a multi-scanner classifier pipeline with the proposed selection for removing site-effects which separate between scanners and select for PD. After training and validating to find the best hyperparameters, the final metric can be taken from the test set. We use the ROC-AUC score and the balanced accuracy score as metrics to measure performance. We use a similar pipeline for the single-scanner classifier, although multi-scanner steps are removed (see Supplemental Info, Supplemental Figure S3).
Neurosci 05 00042 g003
Table 1. Demographic results show that there are similar age and sex distributions of subjects. There is a significantly lower disease duration in the PPMI cohort due to PPMI requiring an MRI scan within 6 months of diagnosis to be included in the study. N.A. means not available. UPDRS3 scores were obtained during the ON-medication state for patients in NIH and Neurocon datasets. PPMI dataset did not always specify the ON/OFF state for UPDRS3 scores.
Table 1. Demographic results show that there are similar age and sex distributions of subjects. There is a significantly lower disease duration in the PPMI cohort due to PPMI requiring an MRI scan within 6 months of diagnosis to be included in the study. N.A. means not available. UPDRS3 scores were obtained during the ON-medication state for patients in NIH and Neurocon datasets. PPMI dataset did not always specify the ON/OFF state for UPDRS3 scores.
Group (PD/HV)AgeSex
(M/F)		Disease DurationH&Y ScoreUPDRS3
PDHVPDHV
NIH
Scanner 124/1262.8 ± 9.258.1 ± 7.416/87/59.4 ± 6.22.5 ± 0.417.6 ± 9.0
Scanner 217/1962.2 ± 7.363.2 ± 6.712/515/46.7 ± 3.1N.A.28.6 ± 8.6
Scanner 323/2060.4 ± 8.059.9 ± 7.412/119/117.8 ± 4.92.2 ± 0.520.6 ± 7.2
PPMI
Scanner 116/1263.1 ± 8.361.9 ± 12.412/410/20.1 ± 0.11.0 ± 0.615.2 ± 8.7
Scanner 222/1262.0 ± 9.258.9 ± 6.913/99/30.1 ± 0.21.2 ± 0.815.5 ± 11.8
Scanner 321/1358.5 ± 10.962.6 ± 9.713/89/40.7 ± 1.11.0 ± 0.913.4 ± 11.8
Scanner 414/1061.7 ± 9.065.7 ± 3.16/87/30.3 ± 0.41.0 ± 0.815.6 ± 11.0
Scanner 514/1158.7 ± 8.260.5 ± 8.811/37/40.1 ± 0.030.8 ± 0.88.6 ± 9.2
Scanner 619/1063.5 ± 9.664.6 ± 6.812/74/60.3 ± 0.41.1 ± 0.812.0 ± 9.0
Other
Neurocon26/1668.8 ± 10.767.6 ± 11.917/912/4N.A.1.9 ± 0.39.2 ± 5.3
Tao Wu20/2065.2 ± 4.464.8 ± 5.611/912/85.4 ± 3.91.9 ± 0.6N.A.
Table 2. The top 3 combinations of classifier, dataset, and NC fit strategy, tested for 10 k other hyperparameters. All three of these classifiers used group to model out neurobiological batch effects.
Table 2. The top 3 combinations of classifier, dataset, and NC fit strategy, tested for 10 k other hyperparameters. All three of these classifiers used group to model out neurobiological batch effects.
ClassifierDataset UsedNC Fit StrategyValidation AUCTest AUC
XGBoostFS OnlyFit on Train, w/Group Covariate0.863 ± 0.0450.902
RFFS OnlyFit on Train, w/Group covariate0.841 ± 0.0360.837
RFBulk + FSFit on Train, w/Group covariate0.806 ± 0.0560.866
Table 3. The top 3 combinations of classifier, dataset, and NC fit strategy, were tested for 10 k other hyperparameters, where group covariate is not included. As seen, the group metrics significantly fall.
Table 3. The top 3 combinations of classifier, dataset, and NC fit strategy, were tested for 10 k other hyperparameters, where group covariate is not included. As seen, the group metrics significantly fall.
ClassifierDataset UsedNC Fit StrategyValidation AUCTest AUC
LRBulk OnlyFit on all splits, no group covariate0.650 ± 0.0520.550
SVCBulk + FSFit on Train, no group0.646 ± 0.0710.500
XGBoostBulk + FSNo NeuroComBat0.644 ± 0.0730.438
Table 4. Validation metrics, split up by scanner, for both single-scanner and multi-scanner classifiers. The single-scanner classifier results indicate a wide variance in performance for single-scanner performance and multi-scanner performance. Multi-scanner performance is colored green or red to indicate improvement or worsening compared to single-scanner metrics. We note NC on Train Only with Group overall has inflated AUC values, likely due to data leakage.
Table 4. Validation metrics, split up by scanner, for both single-scanner and multi-scanner classifiers. The single-scanner classifier results indicate a wide variance in performance for single-scanner performance and multi-scanner performance. Multi-scanner performance is colored green or red to indicate improvement or worsening compared to single-scanner metrics. We note NC on Train Only with Group overall has inflated AUC values, likely due to data leakage.
ScannerValidation AUC
Single-Scanner ClassifierMulti-Scanner Classifier (Best)
No NCNC On Train Only (No Group)NC On Train Only (Group)
NIH Dataset 10.7680.8220.7360.966
NIH Dataset 20.7000.7440.6940.914
NIH Dataset 30.6180.5830.5550.732
PPMI Scanner 10.4860.6140.6431.0
PPMI Scanner 20.5600.6820.5030.871
PPMI Scanner 30.5930.6530.6530.785
PPMI Scanner 40.5190.4440.4670.488
PPMI Scanner 50.5730.5180.5270.873
PPMI Scanner 60.6170.5090.6210.857
Neurocon1.0000.5640.5530.878
Tao Wu0.7270.6450.7190.930
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Saqib, M.; Horovitz, S.G. Harmonization for Parkinson’s Disease Multi-Dataset T1 MRI Morphometry Classification. NeuroSci 2024, 5, 600-613. https://doi.org/10.3390/neurosci5040042

AMA Style

Saqib M, Horovitz SG. Harmonization for Parkinson’s Disease Multi-Dataset T1 MRI Morphometry Classification. NeuroSci. 2024; 5(4):600-613. https://doi.org/10.3390/neurosci5040042

Chicago/Turabian Style

Saqib, Mohammed, and Silvina G. Horovitz. 2024. "Harmonization for Parkinson’s Disease Multi-Dataset T1 MRI Morphometry Classification" NeuroSci 5, no. 4: 600-613. https://doi.org/10.3390/neurosci5040042

APA Style

Saqib, M., & Horovitz, S. G. (2024). Harmonization for Parkinson’s Disease Multi-Dataset T1 MRI Morphometry Classification. NeuroSci, 5(4), 600-613. https://doi.org/10.3390/neurosci5040042

Article Metrics

Back to TopTop