Next Article in Journal
A Fragile Balance: Does Neutrophil Extracellular Trap Formation Drive Pulmonary Disease Progression?
Next Article in Special Issue
Beyond Activation: Characterizing Microglial Functional Phenotypes
Previous Article in Journal
Loop-Mediated Isothermal Amplification (LAMP): The Better Sibling of PCR?
Previous Article in Special Issue
The Influence of Virus Infection on Microglia and Accelerated Brain Aging
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Neuroinflammation and Alzheimer’s Disease: A Machine Learning Approach to CSF Proteomics

by
Lorenzo Gaetani
1,*,†,
Giovanni Bellomo
2,†,
Lucilla Parnetti
1,2,
Kaj Blennow
3,4,
Henrik Zetterberg
3,4,5,6 and
Massimiliano Di Filippo
1,*
1
Section of Neurology, Department of Medicine and Surgery, University of Perugia, 06100 Perugia, Italy
2
Laboratory of Clinical Neurochemistry, Department of Medicine and Surgery, University of Perugia, 06100 Perugia, Italy
3
Institute of Neuroscience and Physiology, Department of Psychiatry and Neurochemistry, The Sahlgrenska Academy at the University of Gothenburg, 431 41 Mölndal, Sweden
4
Clinical Neurochemistry Laboratory, Sahlgrenska University Hospital, 431 41 Mölndal, Sweden
5
Department of Neurodegenerative Disease, UCL Institute of Neurology, Queen Square, London WC1N 3BG, UK
6
UK Dementia Research Institute at UCL, London WC1E 6B, UK
*
Authors to whom correspondence should be addressed.
These authors contributed equally to this work.
Cells 2021, 10(8), 1930; https://doi.org/10.3390/cells10081930
Submission received: 18 May 2021 / Revised: 9 July 2021 / Accepted: 24 July 2021 / Published: 29 July 2021
(This article belongs to the Collection Microglia in Aging and Neurodegenerative Diseases)

Abstract

:
In Alzheimer’s disease (AD), the contribution of pathophysiological mechanisms other than amyloidosis and tauopathy is now widely recognized, although not clearly quantifiable by means of fluid biomarkers. We aimed to identify quantifiable protein biomarkers reflecting neuroinflammation in AD using multiplex proximity extension assay (PEA) testing. Cerebrospinal fluid (CSF) samples from patients with mild cognitive impairment due to AD (AD-MCI) and from controls, i.e., patients with other neurological diseases (OND), were analyzed with the Olink Inflammation PEA biomarker panel. A machine-learning approach was then used to identify biomarkers discriminating AD-MCI (n: 34) from OND (n: 25). On univariate analysis, SIRT2, HGF, MMP-10, and CXCL5 showed high discriminatory performance (AUC 0.809, p = 5.2 × 10−4, AUC 0.802, p = 6.4 × 10−4, AUC 0.793, p = 3.2 × 10−3, AUC 0.761, p = 2.3 × 10−3, respectively), with higher CSF levels in AD-MCI patients as compared to controls. These same proteins were the best contributors to the penalized logistic regression model discriminating AD-MCI from controls (AUC of the model 0.906, p = 2.97 × 10−7). The biological processes regulated by these proteins include astrocyte and microglia activation, amyloid, and tau misfolding modulation, and blood-brain barrier dysfunction. Using a high-throughput multiplex CSF analysis coupled with a machine-learning statistical approach, we identified novel biomarkers reflecting neuroinflammation in AD. Studies confirming these results by means of different assays are needed to validate PEA as a multiplex technique for CSF analysis and biomarker discovery in the field of neurological diseases.

1. Introduction

Alzheimer’s disease (AD) is the commonest cause of dementia in the elderly, and the most frequent human neurodegenerative disease worldwide [1]. In AD, amyloid-β (Aβ) pathology has long been considered as the central event in the pathophysiology of the disease, followed by the intraneuronal aggregation of misfolded and phosphorylated tau protein in form of neuritic plaques, neurofibrillary tangles, and neuropil threads, leading to neurodegeneration and ultimately neuronal death [2].
However, evidence suggests that amyloid and tau pathologies alone cannot explain AD pathophysiology. In this context, both the discovery of elevated levels of inflammatory markers in AD and of AD risk genes associated with innate immune function, suggest a potential contribution of neuroinflammation in AD pathogenesis [3]. Neuroinflammation refers to an abnormal inflammatory response within the central nervous system (CNS) that can be induced by a variety of triggers (e.g., infective, traumatic, vascular, or toxic insults) [4]. A pivotal role in neuroinflammation is played by CNS innate immune system cells, mainly microglia, together with astrocytes, endothelial cells responsible for blood-brain barrier (BBB) dysfunction, and infiltrating peripheral white blood cells [5].
Studies performed on animal models of neurodegeneration, as well as histological findings from AD brains, in vivo studies performed using brain positron emission tomography (PET) and, finally, evidence from human genetic diseases, have highlighted the possible contribution of neuroinflammation in AD [6]. However, whether neuroinflammation is a driving pathophysiological mechanism rather than a downstream effect of neurodegeneration is still a matter of debate [7].
The cerebrospinal fluid (CSF) has been extensively investigated in AD as a source of biomarkers, and a variety of studies has been performed on the levels of inflammatory and glial markers in AD patients compared with controls [6]. Multiplex analyses have also been performed on a more accessible matrix, such as plasma [8]. However, results have not been homogeneous between studies, measured biomarkers were restricted to a small number of inflammatory proteins, and the applied multiplex analyses were limited by antibody cross-reactivity and inter-assay variability. Recently, multiplex mass-spectrometry analyses have been applied to both CSF and brain tissues in AD, confirming the potential contribution of glial markers [9].
Proximity extension assay (PEA) technology is a 96-plex immunoassay for fluid protein detection, which relies on unique antibody–oligonucleotide protein binding for quantitative measurement by means of real-time polymerase chain reaction (PCR) [10]. PEA technology is less exposed to the risk of multiplex analyses limitations, and it has been previously applied to the CSF in neurological disorders, namely Parkinson’s disease and atypical parkinsonian syndromes [11], as well as in the blood in AD patients [12].
Since multiplex analyses provide many outcome variables, their interpretation may be non-trivial and there is a high risk of data overfitting. Machine learning approaches based on penalized regression analysis may therefore allow identifying the smallest possible panel of novel markers with high sensitivity and specificity.
Herein, we have used PEA technology to measure a broad panel of biomarkers reflecting immune activation in CSF samples from patients with early-stage AD and from a control group. To identify the proteins that best separated the AD and control groups, we applied machine-learning statistical models, the results of which may offer a significant contribution to biomarker discovery and novel insights into the pathophysiology of AD.

2. Patients and Methods

2.1. Patients Selection

A retrospective cohort of patients with mild cognitive impairment due to AD (AD-MCI) and other neurological diseases (OND) as the control group, with CSF samples stored in the Laboratory of Clinical Neurochemistry, Department of Medicine and Surgery, University of Perugia (Perugia, Italy), was selected for this study.
AD-MCI patients were enrolled among individuals referring to the Center for Memory Disturbances of the University Hospital of Perugia who underwent lumbar puncture as part of the routine diagnostic workup between 2008 and 2016. Beyond lumbar puncture, all patients underwent a clinical neurological examination, neuropsychological assessment including a Mini-Mental State Examination (MMSE) and Clinical Dementia Rating (CDR) scale, blood chemistry, brain computed tomography (CT), and/or magnetic resonance imaging (MRI) scan for ruling out differential diagnoses. All the patients underwent a clinical follow-up of at least 1 year. Patients with a CSF biomarker profile suggestive of amyloidosis (A+) and tauopathy (T+), were selected for the study [13].
Patients in the OND group were selected among individuals undergoing lumbar puncture for diagnostic purposes other than cognitive complaints and for whom the eventual diagnosis did not include inflammatory or degenerative neurological diseases. For this cohort of control subjects, CSF biomarker profile was not suggestive of amyloidosis (A-), tauopathy (T-), and neurodegeneration (N-).
All the patients were 55–85 years old and none of them had undergone steroid or other immunosuppressant/immunomodulatory treatments within 30 days before lumbar puncture. The local ethics committee approved the study (Protocol N°: 19369/08/AV, registry N°: 1287/08).

2.2. CSF Collection and Storage

CSF samples were collected over an 8-year period (2008–2016) in the Section of Neurology, Department of Medicine and Surgery, University of Perugia (Perugia, Italy). CSF samples were collected and stored with standardized procedures. Specifically, samples were obtained by means of lumbar puncture, performed between 8:00 a.m. and 11:00 a.m. CSF samples were collected in sterile polypropylene tubes, centrifuged for 10 min at 2000× g, divided into 0.5 mL aliquots, and immediately frozen at −80 °C, pending analysis. Collection and storage of CSF samples were carried out by following specific international guidelines [14].

2.3. CSF ATN Profile

In the CSF samples collected for the study, Aβ42, t-tau, and p-tau levels were previously measured by using INNOTEST ELISA (Fujirebio Europe, Gent, Belgium) at the Laboratory of Clinical Neurochemistry, Department of Medicine and Surgery, University of Perugia (Perugia, Italy). Depending on the lumbar puncture date, CSF samples were classified as A+ for Aβ42 ≤ 800 (2011–2016) or ≤ 1200 (2008–2010) pg/mL, T+ for p-tau ≥ 60 pg/mL and N+ for t-tau ≥ 400 (2011–2016) or ≥ 200 (2008–2010) pg/mL.

2.4. PEA Testing

Inflammatory proteins panel testing was performed in 2017 using the multiplex PEA technology as previously described by Olink (Uppsala, Sweden) [10]. All the samples were run on the inflammation panel, which consists of 92 biomarkers (Table S1), with up to 96 samples tested simultaneously on each run. The Olink panel validation data are freely available online (https://www.olink.com/data-you-can-trust/validation/, accessed on 23 July 2021). The resulting data for each biomarker were expressed as normalized protein expression (NPX) value. NPX is an arbitrary unit on a log2 scale that is obtained by normalizing the concentration values to minimize inter- and intra-assay variations. A high NPX value corresponds to a high protein concentration and can be linearized by using the formula 2NPX. NPX values were subsequently z-scored to allow for a better comparison in multivariate analysis.

2.5. Statistical Analysis

Data analysis was performed by using the R software v 3.6 and OriginPro v 9.0. Continuous variables were reported as the mean ± standard deviation. The Kolmogorov-Smirnov (KS) normality test was used to disproof data normality. The significance of age and gender differences between OND and AD-MCI groups was assessed by means of the Mann–Whitney U-test and Fisher’s Exact test for count data, respectively. Receiver Operating Characteristic (ROC) curve analysis was performed by using the R package pROC [15]. The area under the curve (AUC) and AUC sensitivity and specificity confidence intervals were calculated by using 2000 bootstrap replicates. Age-adjusted AUC were instead computed with the R package AROC, according to the semiparametric Bayesian approach described by F. Machado e Costa and A. C. Braga [16]. The significance of z-score differences between groups was assessed by two-way analysis of covariance (ANCOVA) by assuming age as a covariate. Multiple testing effects were taken into account by correcting the significance level according to Bonferroni [17]. Considering that 46 different proteins were taken into account for the analysis (proteins with a call-rate of > 95%) and a significance level of 0.05, the Bonferroni correction would require changing the significance level to 0.0011. The correlation among z-scored proteins percentages was represented in terms of Spearman’s correlation coefficients and displayed in a heatmap by using the pheatmap R package [18]. Proteins were grouped according to a hierarchical clustering [19,20]. Correlation coefficients and Ward’s linkage [21] were used as distance and linkage parameters for clustering. Least absolute shrinkage and selection operator (LASSO) [22,23], a feature selection technique used for machine learning predictive modeling, was used to identify the biomarker panel that offered the best discriminatory performance. The glmnet R package was used for this purpose [24]. Within the AD-MCI group, correlations between the z-scores of the most discriminatory proteins and classical AD biomarkers were evaluated in terms of Spearman’s correlation coefficients.

3. Results

3.1. Characteristics of the Patients

A total of 34 AD-MCI patients (M/F 14/20, mean age 72.2 ± 5.8 y) were recruited for the study. None of the patients were taking any CNS-active drugs, such as acetylcholinesterase inhibitors or NMDA receptor modulators, at the time of diagnostic work-up. All patients underwent a clinical follow-up at least 1 year (mean follow-up 2.4 ± 1.3 y) after the baseline assessment. Baseline MMSE was 22.4 ± 4.1, while follow-up MMSE was 19.2 ± 5.7.
As neurological controls, a total of 25 subjects (M/F 15/10, mean age 66.8 ± 7.5 y) who underwent lumbar puncture for diagnostic purposes were enrolled. This group included patients with headache (n = 13), psychiatric disorders (n = 10), or mononeuropathy (n = 2). Subjects did not show significant differences in terms of gender distribution among the groups (p = 0.19, Fisher’s exact test), while the age of AD-MCI patients was significantly higher than those of OND subjects (p = 0.005, Mann–Whitney U-test).

3.2. PEA Testing

Out of the 92 proteins determined through the PEA technique, 46 had a call rate <95% (<95% of the participants had a valid measurement of that protein) and were removed from the analysis (Table S1).

3.3. Univariate Diagnostic Performance of PEA-Tested Proteins

We performed a univariate ROC analysis of the z-scores relative to the measured proteins (for abbreviations see Table S1). The AUC and age-adjusted p-values (ANCOVA) of the proteins that were most able to differentiate between AD-MCI and OND are shown in Table 1. Only proteins whose AUC was different from 0.5 within its 99.9% CI (p < 0.001) were considered, in order to take into account possible multiple testing effects. All the z-scores relative to the proteins presented in Table 1 were approximately normally distributed by the KS test. Thus, considering the significant age difference between the OND and AD-MCI groups, we also applied a two-way ANCOVA to assess the significance of the differences in z-scores between OND and AD-MCI by controlling for age. Notably, SIRT2 and HGF z-scores significantly differed between AD-MCI and OND with a p < 0.001, even after adjusting for age differences.

3.4. Correlation Analysis of PEA-Tested Proteins

The majority of the z-scored NPX values in the OND and AD-MCI groups were approximately normally distributed by applying the KS test. However, since normality could be excluded for some analytes, we decided to quantify correlations among them in the entire cohort (AD-MCI and OND) with Spearman’s correlation coefficients. Only positive correlations were observed among the measured proteins in the OND and AD-MCI groups. By treating correlation coefficients as distances, we grouped together highly correlating proteins by hierarchical clustering. The result of this procedure is summarized in the heatmap shown in Figure 1.

3.5. Multivariate Diagnostic Performance of PEA-Tested Proteins

Considering that some of the z-scores relative to the proteins shown in Table 1 were highly correlated, we decided to apply the LASSO method to identify a panel of proteins able to differentiate AD-MCI from OND while keeping the lowest possible dimensionality by eliminating the highest possible number of parameters among highly correlated variables. Being λmin, the shrinkage parameter corresponding to minimum binomial deviance between the multinomial logistics regression model and groups categories (optimal model), we selected the one at 1 standard error (λmin + 1SE) from it, to select the simplest model with accuracy comparable to the optimal model, to avoid overfitting. The LASSO coefficients behavior from λ = 0 to λ = λmin + 1SE, together with the diagnostic performance of the LASSO logistic model (LLM) and of the three main contributing proteins are shown in Figure 2. The linear combination of the selected proteins with the coefficients displayed in Figure 2A was significantly different between AD-MCI and OND (ANCOVA age-adjusted p-value = 2.97 × 10−7), and LLM differentiated AD-MCI from OND with an AUC of 0.906. Considering the age bias between AD-MCI and OND groups, age-adjusted AUC for LLM was also calculated and displayed in Figure 2B.
Moreover, repeating the procedure on a smaller age-matched cohort (ANOVA p-value = 0.39, U-test p-value = 0.15), by removing the 6 youngest OND subjects (age from 55 y to 60 y), produced a 6-variable model (SIRT2, coef. 0.655; CXCL5, coef. 0.465; HGF, coef. 0.102; IL-12B, coef. −0.325; MMP-10 coef. 0.0237 and CDCP1 coef. 0.027). The model (λ = 0.0616) provided an AUC of 0.876 (0.757–0.966) in the cohort subset, similar to the age-adjusted AUC obtained for the LLM in the whole cohort (Figure 2B).
Among the tested proteins, SIRT2, HGF, MMP-10, and CXCL5 were present both in the LASSO selections and among the proteins showing the best performance in differentiating AD-MCI from OND (Table 1). The distributions of z-scored values relative to these proteins are shown in Figure 3.

3.6. Correlation Analysis between PEA-Tested Proteins and CSF AD Biomarkers

Within the AD-MCI group, correlation analysis between the z-scores of the most discriminatory PEA-tested proteins (listed in Table 1) and classical CSF AD biomarkers showed different significant associations (Table 2). A significant negative correlation was found between CSF Aβ42 and MMP-10 (ρ = −0.37, p < 0.05), while significant positive correlations were found between CSF p-tau and SIRT-2, HGF, uPA, LIF-R, and TWEAK, the most significant being those with SIRT2 (ρ = 0.48, p < 0.005), LIF-R (ρ = 0.51, p < 0.005), and TWEAK (ρ = 0.43, p < 0.005). Finally, a positive correlation was found between CSF t-tau and LIF-R (ρ = 0.35, p < 0.05).

4. Discussion

Our findings confirm the potential of the application of high-throughput multiplex assays in investigating novel pathophysiological pathways in neurodegenerative diseases such as AD. Although a study on an even broader panel of potential novel CSF AD biomarkers in a larger population has been recently performed [9], it was based on a multiplex mass spectrometry method that requires a large volume of CSF. An antibody-based method such as PEA may be more accessible and more easily translated into a clinical setting. Also, the population study included the entire AD continuum, while we focused only on AD-MCI patients, which makes our findings more specific to the prodromal AD stage. CSF multiplex PEA testing, coupled with a machine-learning statistical approach, led us to identify interesting biological pathways other than amyloidosis and tauopathy, potentially linked with AD pathophysiology in the early clinical phases of the disease (AD-MCI). This choice allowed us to better investigate the contribution of neuroinflammation as a potential early mechanism in AD, rather than as a consequence of advanced neurodegeneration. Moreover, the selection of patients with a CSF profile suggestive of AD (A+/T+) for a proteomics study ensured us a high level of biological homogeneity, thus reducing potential sources of variability.
In this specific cohort of early AD individuals, the subset of proteins that were present in the selected panel able to differentiate in single AD-MCI from controls with the highest accuracy were SIRT2, HGF, MMP-10, and CXCL5.
SIRT2 (sirtuin 2) is a protein deacetylase highly expressed in the mammalian CNS, mainly found in the cytoplasm of oligodendrocytes [25]. It is particularly expressed in the cortex, striatum, hippocampus, and spinal cord, but its functions are still largely unknown [25]. SIRT2 has demonstrated a possible role in mitosis regulation, genome integrity, cell differentiation, cell homeostasis, but also in microglial activation, and hampering autophagic activity [25]. Interestingly, SIRT2 mRNA expression has been found to be increased in the peripheral blood of AD patients, and an association between SIRT2 polymorphisms and AD risk has been identified [26,27]. Inhibition of SIRT2 has demonstrated effects in counteracting the amyloidogenic pathway in vitro, by reducing Aβ production as well as its toxic effects on neurons [28,29]. Evidence also suggests that SIRT2 inhibition may induce α-tubulin acetylation thus decreasing tau-phosphorylation [30]. Interestingly, from our analysis, CSF SIRT2 positively correlated with CSF p-tau levels (ρ = 0.48, p < 0.005) in AD-MCI patients, thus supporting the evidence of a possible association between SIRT2 activity and tau pathology. Moreover, since we found higher levels of SIRT2 in the CSF of AD-MCI patients compared to controls, it is possible to hypothesize that an overexpression of this protein may be associated with AD pathophysiology in the earliest phases of the disease.
HGF (hepatocyte growth factor), a potent mitogen for hepatocytes, is expressed, together with its high-affinity tyrosine kinase receptor (Met), in the CNS, and specifically in the hippocampal region [31]. It has been demonstrated that synaptic activity modulates HGF signaling in hippocampal neurons and, in turn, HGF modulates synaptic function and enhances dendritic arborization [32,33], suggesting a potential physiological role of HGF signaling in synaptic functions. Of interest, in AD brains, higher staining for HGF in astrocytes within the parietal and temporal lobe white matter compared to controls has been documented [34]. Moreover, in CSF, a higher concentration of HGF in AD patients vs. controls has already been found by means of ELISA measurement [35]. It has been hypothesized that the higher values of CSF HGF may depend upon the activation of astrocytes or an increased number of astrocytes in the white matter in AD [35], therefore serving as a biomarker of glial activation in neurodegeneration. According to our findings, such processes are therefore already evident and quantifiable in the CSF in the earliest prodromal phases of the disease, with a positive correlation with markers of tau pathology. The latter finding suggests a potential pathophysiological association between tau hyperphosphorylation and astrocytes activation as measured with CSF HGF.
MMP-10 (matrix metalloproteinase-10 or stromelysin-2) belongs to the family of matrix metalloproteinases, which are enzymes able to degrade components of the extracellular matrix. In the CNS, they are expressed in neurons but are also secreted by astrocytes and microglia [36,37,38]. Of interest, matrix metalloproteinases are able to degrade Aβ in vitro as well as in vivo in animal models of cerebral amyloidosis [39,40]. By means of multiplex bead-based immunoassays, MMP-10 has already been found to be increased in the CSF of AD patients compared to controls, as was its CSF/plasma ratio compared to other forms of dementia (namely, vascular dementia) [41]. Additionally, MMP-10 was found to be increased in the CSF of AD-dementia patients and patients with MCI and a CSF A+ profile compared to controls in a previous study based on the same multiplex testing and on a similar statistical approach [42]. The concordance of our findings, though obtained on a different population study, supports the possibility of MMP-10 as a biomarker of AD. Since MMP-10 is known to be highly expressed by activated microglia [43], it may be involved in the non-neuronal cells’ response to neuronal damage, thus playing an important pathophysiological role in neurodegeneration. Also, in AD patients, a positive correlation between CSF MMP-10 and both total tau and phosphorylated tau has been documented [41]. Of interest, in our work, we did not find significant correlations between CSF MMP-10 and p-tau/t-tau, while MMP-10 protein levels were found to be negatively correlated with CSF Aβ42 levels in AD-MCI patients (ρ = −0.37, p < 0.05), thus supporting a possible interaction between this specific metalloproteinase and amyloid pathology in AD patients.
CXCL5 (C-X-C motif chemokine 5) is a small cytokine that is produced by immune and vascular endothelial cells in response to proinflammatory cytokines through NF-kB activation [44]. Its CSF levels have been found to be increased in CNS inflammatory diseases, such as multiple sclerosis, being involved in blood-brain barrier (BBB) dysfunction [45]. Previously, a CSF increased concentration of CXCL5 in AD patients has not been described. Therefore, our findings support a possible role of immune mediators involved in BBB stability even in neurodegenerative diseases such as AD.
Of interest, the correlation and cluster analysis of all the CSF markers we measured with PEA showed at least two clusters of highly correlated protein levels in the entire study cohort (i.e., both AD-MCI and OND). The major cluster included SIRT2 and HGF, together with another 15 markers (Figure 1), suggesting that the neuroinflammatory mechanisms linked to AD involve multiple biological actors, among which SIRT2 and HGF were the most altered in the AD-MCI group with respect to OND.
Another independent secondary cluster included a small group of cytokines (MCP-2, CXCL11, and CXCL10) that are chemotactic for and activate many different immune cells that are involved in inflammatory response.
Taken together, our findings shed light on the potential contribution of immune-mediated mechanisms in the pathophysiology of AD. SIRT2 overexpression, together with astrocytic and microglial activation and BBB dysfunction seem to be the most promising mechanisms that can be quantified in the CSF of early-stage AD patients. As a limitation of our study, we must consider the limited sample size. Further replication data in larger cohorts are therefore needed.

5. Conclusions

  • We present promising findings using PEA biomarker technology coupled with a machine-learning statistical approach for the discovery of novel markers and pathophysiological mechanisms of AD.
  • Our findings suggest an association between AD pathology and the SIRT2 pathway, astrocyte and microglia activation, and BBB dysfunction, as reflected by concentration changes in the CSF of SIRT2, HGF, MMP10, and CXCL5.
  • Studies confirming these results in larger cohorts also by means of different assays, such as the single-molecule array and ELISA, might be needed to validate PEA as a promising tool for biomarker discovery in the field of neurological diseases.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/cells10081930/s1, Table S1: List of the biomarkers measured by means of PEA technique (inflammation panel) in the CSF of recruited patients.

Author Contributions

Conceptualization, L.G., M.D.F., L.P., H.Z. and K.B.; methodology, L.G., H.Z. and K.B.; statistical analysis, G.B.; investigation, L.G., G.B., H.Z. and K.B.; resources, L.P., H.Z. and K.B.; data curation, L.G. and G.B.; writing—original draft preparation, L.G., G.B. and M.D.F.; writing—review and editing, L.G., G.B., M.D.F., L.P., H.Z. and K.B.; supervision, M.D.F., L.P., H.Z. and K.B. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of Umbria Region (ID code: marcatori liquorali nelle malattie neurodegenerative, protocol N°: 19369/08/AV, registry N°: 1287/08, date: 9 October 2008).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author upon reasonable request. The data are not publicly available due to privacy and ethical reasons.

Acknowledgments

G.B. is supported by the JPND bPRIDE (blood Proteins for early Discrimination of dEmentias) project. The Project leading this result has received funding under the call “JPco-fuND-2: Multinational research projects on Personalised Medicine for Neurodegenerative Diseases” (CUP number J99C18000210005). H.Z. is a Wallenberg Scholar supported by grants from the Swedish Research Council (#2018-02532), the European Research Council (#681712), and Swedish State Support for Clinical Research (#ALFGBG-720931).

Conflicts of Interest

L.G. participated on advisory boards for and received writing honoraria and funding for traveling from Almirall, Biogen, Sanofi, Merck, Mylan, Novartis, Roche and Teva. M.D.F. participated on advisory boards for and received speaker or writing honoraria and funding for traveling from Bayer, Biogen Idec, Sanofi, Merck, Mylan, Novartis, Roche and Teva. K.B. has served as a consultant, at advisory boards, or at data monitoring committees for Abcam, Axon, Biogen, JOMDD/Shimadzu. Julius Clinical, Lilly, MagQu, Novartis, Prothena, Roche Diagnostics, and Siemens Healthineers, and is a co-founder of Brain Biomarker Solutions in Gothenburg AB (BBS), which is a part of the GU Ventures Incubator Program. HZ has served at scientific advisory boards for Alector, Eisai, Denali, Roche Diagnostics, Wave, Samumed, Siemens Healthineers, Pinteon Therapeutics, Nervgen, AZTherapies and CogRx, has given lectures in symposia sponsored by Cellectricon, Fujirebio, Alzecure and Biogen, and is a co-founder of Brain Biomarker Solutions in Gothenburg AB (BBS), which is a part of the GU Ventures Incubator Program.

References

  1. Scheltens, P.; Blennow, K.; Breteler, M.M.B.; de Strooper, B.; Frisoni, G.B.; Salloway, S.; Van der Flier, W.M. Alzheimer’s Disease. Lancet 2016, 388, 505–517. [Google Scholar] [CrossRef]
  2. Jellinger, K.A. Neuropathological Aspects of Alzheimer Disease, Parkinson Disease and Frontotemporal Dementia. Neuro-Degener. Dis. 2008, 5, 118–121. [Google Scholar] [CrossRef]
  3. Leng, F.; Edison, P. Neuroinflammation and Microglial Activation in Alzheimer Disease: Where Do We Go from Here? Nat. Rev. Neurol. 2020, 17, 157–172. [Google Scholar] [CrossRef] [PubMed]
  4. Ransohoff, R.M. How Neuroinflammation Contributes to Neurodegeneration. Science 2016, 353, 777–783. [Google Scholar] [CrossRef]
  5. Ransohoff, R.M.; Brown, M.A. Innate Immunity in the Central Nervous System. J. Clin. Investig. 2012, 122, 1164–1171. [Google Scholar] [CrossRef]
  6. Calsolaro, V.; Edison, P. Neuroinflammation in Alzheimer’s Disease: Current Evidence and Future Directions. Alzheimer’s Dement. 2016, 12, 719–732. [Google Scholar] [CrossRef]
  7. Jaturapatporn, D.; Isaac, M.G.E.K.N.; McCleery, J.; Tabet, N. Aspirin, Steroidal and Non-Steroidal Anti-Inflammatory Drugs for the Treatment of Alzheimer’s Disease. Cochrane Database Syst. Rev. 2012, 15, CD006378. [Google Scholar] [CrossRef]
  8. Elahi, F.M.; Casaletto, K.B.; Joie, R.L.; Walters, S.M.; Harvey, D.; Wolf, A.; Edwards, L.; Rivera-Contreras, W.; Karydas, A.; Cobigo, Y.; et al. Plasma Biomarkers of Astrocytic and Neuronal Dysfunction in Early- and Late-Onset Alzheimer’s Disease. Alzheimer’s Dement. 2020, 16, 681–695. [Google Scholar] [CrossRef]
  9. Higginbotham, L.; Ping, L.; Dammer, E.B.; Duong, D.M.; Zhou, M.; Gearing, M.; Hurst, C.; Glass, J.D.; Factor, S.A.; Johnson, E.C.B.; et al. Integrated Proteomics Reveals Brain-Based Cerebrospinal Fluid Biomarkers in Asymptomatic and Symptomatic Alzheimer’s Disease. Sci. Adv. 2020, 6, eaaz9360. [Google Scholar] [CrossRef]
  10. Assarsson, E.; Lundberg, M.; Holmquist, G.; Björkesten, J.; Thorsen, S.B.; Ekman, D.; Eriksson, A.; Rennel Dickens, E.; Ohlsson, S.; Edfeldt, G.; et al. Homogenous 96-Plex PEA Immunoassay Exhibiting High Sensitivity, Specificity, and Excellent Scalability. PLoS ONE 2014, 9, e95192. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. Jabbari, E.; Woodside, J.; Guo, T.; Magdalinou, N.K.; Chelban, V.; Athauda, D.; Lees, A.J.; Foltynie, T.; Houlden, H.; Church, A.; et al. Proximity Extension Assay Testing Reveals Novel Diagnostic Biomarkers of Atypical Parkinsonian Syndromes. J. Neurol. Neurosurg. Psychiatry 2019, 90, 768–773. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Ellegaard Nielsen, J.; Sofie Pedersen, K.; Vestergård, K.; Georgiana Maltesen, R.; Christiansen, G.; Lundbye-Christensen, S.; Moos, T.; Risom Kristensen, S.; Pedersen, S. Novel Blood-Derived Extracellular Vesicle-Based Biomarkers in Alzheimer’s Disease Identified by Proximity Extension Assay. Biomedicines 2020, 8, 199. [Google Scholar] [CrossRef]
  13. Jack, C.R.; Bennett, D.A.; Blennow, K.; Carrillo, M.C.; Dunn, B.; Haeberlein, S.B.; Holtzman, D.M.; Jagust, W.; Jessen, F.; Karlawish, J.; et al. NIA-AA Research Framework: Toward a Biological Definition of Alzheimer’s Disease. Alzheimer’s Dement. 2018, 14, 535–562. [Google Scholar] [CrossRef]
  14. Teunissen, C.E.; Petzold, A.; Bennett, J.L.; Berven, F.S.; Brundin, L.; Comabella, M.; Franciotta, D.; Frederiksen, J.L.; Fleming, J.O.; Furlan, R.; et al. A Consensus Protocol for the Standardization of Cerebrospinal Fluid Collection and Biobanking. Neurology 2009, 73, 1914–1922. [Google Scholar] [CrossRef] [Green Version]
  15. Robin, X.; Turck, N.; Hainard, A.; Tiberti, N.; Lisacek, F.; Sanchez, J.-C.; Müller, M. PROC: An Open-Source Package for R and S+ to Analyze and Compare ROC Curves. BMC Bioinform. 2011, 12, 77. [Google Scholar] [CrossRef]
  16. Machado e Costa, F.; Braga, A.C. Adjusting ROC Curve for Covariates with AROC R Package. In Computational Science and Its Applications, Proceedings of the ICCSA 2020, Cagliari, Italy, 1–4 July 2020; Springer: Cham, Switzerland, 2020; Volume 12251, pp. 185–198. [Google Scholar] [CrossRef]
  17. Bonferroni, C.E. Teoria Statistica delle Classi e Calcolo delle Probabilità; Libreria Internazionale Seeber: Florence, Italy, 1936. [Google Scholar]
  18. Kolde, R. Pheatmap: Pretty Heatmaps. 2019. Available online: https://CRAN.R-project.org/package=pheatmap (accessed on 23 July 2021).
  19. Rokach, L.; Maimon, O. Clustering Methods. In Data Mining and Knowledge Discovery Handbook; Maimon, O., Rokach, L., Eds.; Springer: Boston, MA, USA, 2005; pp. 321–352. ISBN 978-0-387-25465-4. [Google Scholar]
  20. Gu, Z.; Eils, R.; Schlesner, M. Complex Heatmaps Reveal Patterns and Correlations in Multidimensional Genomic Data. Bioinformatics 2016, 32, 2847–2849. [Google Scholar] [CrossRef] [Green Version]
  21. Ward, J.H., Jr. Hierarchical Grouping to Optimize an Objective Function. J. Am. Stat. Assoc. 1963, 58, 236–244. [Google Scholar] [CrossRef]
  22. Tibshirani, R. The Lasso Method for Variable Selection in the Cox Model. Stat. Med. 1997, 16, 385–395. [Google Scholar] [CrossRef] [Green Version]
  23. Muthukrishnan, R.; Rohini, R. LASSO: A Feature Selection Technique in Predictive Modeling for Machine Learning. In Proceedings of the 2016 IEEE International Conference on Advances in Computer Applications (ICACA), Coimbatore, India, 24 October 2016; pp. 18–20. [Google Scholar]
  24. Friedman, J.; Hastie, T.; Tibshirani, R. Regularization Paths for Generalized Linear Models via Coordinate Descent. J. Stat. Softw. 2010, 33, 1–22. [Google Scholar] [CrossRef] [Green Version]
  25. Wang, Y.; Yang, J.; Hong, T.; Chen, X.; Cui, L. SIRT2: Controversy and Multiple Roles in Disease and Physiology. Ageing Res. Rev. 2019, 55, 100961. [Google Scholar] [CrossRef]
  26. Wongchitrat, P.; Pakpian, N.; Kitidee, K.; Phopin, K.; Dharmasaroja, P.A.; Govitrapong, P. Alterations in the Expression of Amyloid Precursor Protein Cleaving Enzymes MRNA in Alzheimer Peripheral Blood. Curr. Alzheimer Res. 2019, 16, 29–38. [Google Scholar] [CrossRef] [PubMed]
  27. Polito, L.; Kehoe, P.G.; Davin, A.; Benussi, L.; Ghidoni, R.; Binetti, G.; Quadri, P.; Lucca, U.; Tettamanti, M.; Clerici, F.; et al. The SIRT2 Polymorphism Rs10410544 and Risk of Alzheimer’s Disease in Two Caucasian Case-Control Cohorts. Alzheimer’s Dement. 2013, 9, 392–399. [Google Scholar] [CrossRef] [PubMed]
  28. Biella, G.; Fusco, F.; Nardo, E.; Bernocchi, O.; Colombo, A.; Lichtenthaler, S.F.; Forloni, G.; Albani, D. Sirtuin 2 Inhibition Improves Cognitive Performance and Acts on Amyloid-β Protein Precursor Processing in Two Alzheimer’s Disease Mouse Models. J. Alzheimer’s Dis. 2016, 53, 1193–1207. [Google Scholar] [CrossRef]
  29. Gal, J.; Bang, Y.; Choi, H.J. SIRT2 Interferes with Autophagy-Mediated Degradation of Protein Aggregates in Neuronal Cells under Proteasome Inhibition. Neurochem. Int. 2012, 61, 992–1000. [Google Scholar] [CrossRef] [PubMed]
  30. Esteves, A.R.; Palma, A.M.; Gomes, R.; Santos, D.; Silva, D.F.; Cardoso, S.M. Acetylation as a Major Determinant to Microtubule-Dependent Autophagy: Relevance to Alzheimer’s and Parkinson Disease Pathology. Biochim. Biophys. Acta Mol. Basis Dis. 2019, 1865, 2008–2023. [Google Scholar] [CrossRef] [PubMed]
  31. Akimoto, M.; Baba, A.; Ikeda-Matsuo, Y.; Yamada, M.K.; Itamura, R.; Nishiyama, N.; Ikegaya, Y.; Matsuki, N. Hepatocyte Growth Factor as an Enhancer of Nmda Currents and Synaptic Plasticity in the Hippocampus. Neuroscience 2004, 128, 155–162. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  32. Nakano, M.; Takagi, N.; Takagi, K.; Funakoshi, H.; Matsumoto, K.; Nakamura, T.; Takeo, S. Hepatocyte Growth Factor Promotes the Number of PSD-95 Clusters in Young Hippocampal Neurons. Exp. Neurol. 2007, 207, 195–202. [Google Scholar] [CrossRef] [PubMed]
  33. Tyndall, S.J.; Patel, S.J.; Walikonis, R.S. Hepatocyte Growth Factor-Induced Enhancement of Dendritic Branching Is Blocked by Inhibitors of N-Methyl-D-Aspartate Receptors and Calcium/Calmodulin-Dependent Kinases. J. Neurosci. Res. 2007, 85, 2343–2351. [Google Scholar] [CrossRef]
  34. Yamada, T.; Tsubouchi, H.; Daikuhara, Y.; Prat, M.; Comoglio, P.M.; McGeer, P.L.; McGeer, E.G. Immunohistochemistry with Antibodies to Hepatocyte Growth Factor and Its Receptor Protein (c-MET) in Human Brain Tissues. Brain Res. 1994, 637, 308–312. [Google Scholar] [CrossRef]
  35. Tsuboi, Y.; Kakimoto, K.; Nakajima, M.; Akatsu, H.; Yamamoto, T.; Ogawa, K.; Ohnishi, T.; Daikuhara, Y.; Yamada, T. Increased Hepatocyte Growth Factor Level in Cerebrospinal Fluid in Alzheimer’s Disease. Acta Neurol. Scand. 2003, 107, 81–86. [Google Scholar] [CrossRef]
  36. Cuadrado, E.; Rosell, A.; Penalba, A.; Slevin, M.; Alvarez-Sabín, J.; Ortega-Aznar, A.; Montaner, J. Vascular MMP-9/TIMP-2 and Neuronal MMP-10 up-Regulation in Human Brain after Stroke: A Combined Laser Microdissection and Protein Array Study. J. Proteome Res. 2009, 8, 3191–3197. [Google Scholar] [CrossRef] [PubMed]
  37. Könnecke, H.; Bechmann, I. The Role of Microglia and Matrix Metalloproteinases Involvement in Neuroinflammation and Gliomas. Clin. Dev. Immunol. 2013, 2013, 914104. [Google Scholar] [CrossRef] [Green Version]
  38. Hernández-Guillamon, M.; Delgado, P.; Ortega, L.; Pares, M.; Rosell, A.; García-Bonilla, L.; Fernández-Cadenas, I.; Borrell-Pagès, M.; Boada, M.; Montaner, J. Neuronal TIMP-1 Release Accompanies Astrocytic MMP-9 Secretion and Enhances Astrocyte Proliferation Induced by Beta-Amyloid 25-35 Fragment. J. Neurosci. Res. 2009, 87, 2115–2125. [Google Scholar] [CrossRef] [PubMed]
  39. Backstrom, J.R.; Lim, G.P.; Cullen, M.J.; Tökés, Z.A. Matrix Metalloproteinase-9 (MMP-9) Is Synthesized in Neurons of the Human Hippocampus and Is Capable of Degrading the Amyloid-Beta Peptide (1-40). J. Neurosci. 1996, 16, 7910–7919. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  40. Lee, J.-M.; Yin, K.-J.; Hsin, I.; Chen, S.; Fryer, J.D.; Holtzman, D.M.; Hsu, C.Y.; Xu, J. Matrix Metalloproteinase-9 and Spontaneous Hemorrhage in an Animal Model of Cerebral Amyloid Angiopathy. Ann. Neurol. 2003, 54, 379–382. [Google Scholar] [CrossRef] [PubMed]
  41. Duits, F.H.; Hernandez-Guillamon, M.; Montaner, J.; Goos, J.D.C.; Montañola, A.; Wattjes, M.P.; Barkhof, F.; Scheltens, P.; Teunissen, C.E.; van der Flier, W.M. Matrix Metalloproteinases in Alzheimer’s Disease and Concurrent Cerebral Microbleeds. J. Alzheimer’s Dis. 2015, 48, 711–720. [Google Scholar] [CrossRef] [PubMed]
  42. Whelan, C.D.; Mattsson, N.; Nagle, M.W.; Vijayaraghavan, S.; Hyde, C.; Janelidze, S.; Stomrud, E.; Lee, J.; Fitz, L.; Samad, T.A.; et al. Multiplex Proteomics Identifies Novel CSF and Plasma Biomarkers of Early Alzheimer’s Disease. Acta Neuropathol. Commun. 2019, 7, 169. [Google Scholar] [CrossRef]
  43. Nuttall, R.K.; Silva, C.; Hader, W.; Bar-Or, A.; Patel, K.D.; Edwards, D.R.; Yong, V.W. Metalloproteinases Are Enriched in Microglia Compared with Leukocytes and They Regulate Cytokine Levels in Activated Microglia. Glia 2007, 55, 516–526. [Google Scholar] [CrossRef]
  44. Chandrasekar, B.; Melby, P.C.; Sarau, H.M.; Raveendran, M.; Perla, R.P.; Marelli-Berg, F.M.; Dulin, N.O.; Singh, I.S. Chemokine-Cytokine Cross-Talk. The ELR+ CXC Chemokine LIX (CXCL5) Amplifies a Proinflammatory Cytokine Response via a Phosphatidylinositol 3-Kinase-NF-Kappa B Pathway. J. Biol. Chem. 2003, 278, 4675–4686. [Google Scholar] [CrossRef] [Green Version]
  45. Haarmann, A.; Schuhmann, M.K.; Silwedel, C.; Monoranu, C.-M.; Stoll, G.; Buttmann, M. Human Brain Endothelial CXCR2 Is Inflammation-Inducible and Mediates CXCL5- and CXCL8-Triggered Paraendothelial Barrier Breakdown. Int. J. Mol. Sci. 2019, 20, 602. [Google Scholar] [CrossRef] [Green Version]
Figure 1. Correlation heatmap. Correlation coefficients were computed according to Spearman. Hierarchical clustering was used for ordering proteins by using correlation coefficients as distance and the Ward’s linkage criterion [21]. From the correlation and cluster analysis summarized in Figure 1, it emerges that, among the measured proteins, some of them strongly correlated with each other. The major cluster consisted in ADA, TWEAK, PD-L1, SIRT2, TRAIL, HGF, Flt3L, SCF, IL-10RB, uPA, CX3CL1, Beta-NGF, TGF-alpha, CSF-1, CD40, VEGF-A, and LIF-R. Another independent secondary cluster of highly correlated proteins consisted of MCP-2, CXCL11, and CXCL10. For abbreviations of PEA-tested proteins, see Table S1.
Figure 1. Correlation heatmap. Correlation coefficients were computed according to Spearman. Hierarchical clustering was used for ordering proteins by using correlation coefficients as distance and the Ward’s linkage criterion [21]. From the correlation and cluster analysis summarized in Figure 1, it emerges that, among the measured proteins, some of them strongly correlated with each other. The major cluster consisted in ADA, TWEAK, PD-L1, SIRT2, TRAIL, HGF, Flt3L, SCF, IL-10RB, uPA, CX3CL1, Beta-NGF, TGF-alpha, CSF-1, CD40, VEGF-A, and LIF-R. Another independent secondary cluster of highly correlated proteins consisted of MCP-2, CXCL11, and CXCL10. For abbreviations of PEA-tested proteins, see Table S1.
Cells 10 01930 g001
Figure 2. (A) LASSO coefficients in function of the shrinkage parameter λ from λ = 0.25924 (at least 1 coefficient ≠ 0) to λ = 0.0477 (λmin + 1SE). The coefficients found for λmin + 1SE were used to build a LASSO-based logistic model (LLM). (B) ROC curves relative to the diagnostic performance of LLM and the three proteins z-scores most contributing to the model, namely SIRT2, HGF, and MMP-10. For abbreviations of PEA-tested proteins, see Table S1.
Figure 2. (A) LASSO coefficients in function of the shrinkage parameter λ from λ = 0.25924 (at least 1 coefficient ≠ 0) to λ = 0.0477 (λmin + 1SE). The coefficients found for λmin + 1SE were used to build a LASSO-based logistic model (LLM). (B) ROC curves relative to the diagnostic performance of LLM and the three proteins z-scores most contributing to the model, namely SIRT2, HGF, and MMP-10. For abbreviations of PEA-tested proteins, see Table S1.
Cells 10 01930 g002
Figure 3. Distribution of SIRT2, HGF, MMP-10, and CXCL5 z-scores in OND and AD-MCI groups. Boxes representing data distributions are centered on the mean values, with the internal horizontal line representing the median. Box heights are equal to the standard error of mean values, whiskers represent the 10–90% data range. For abbreviations of PEA-tested proteins, see Table S1.
Figure 3. Distribution of SIRT2, HGF, MMP-10, and CXCL5 z-scores in OND and AD-MCI groups. Boxes representing data distributions are centered on the mean values, with the internal horizontal line representing the median. Box heights are equal to the standard error of mean values, whiskers represent the 10–90% data range. For abbreviations of PEA-tested proteins, see Table S1.
Cells 10 01930 g003
Table 1. List of most discriminatory proteins between AD-MCI and OND by univariate analysis.
Table 1. List of most discriminatory proteins between AD-MCI and OND by univariate analysis.
Protein NameAUC 1p-Value 2
SIRT20.8095.2 × 10−4
HGF0.8026.4 × 10−4
MMP-100.7933.2 × 10−3
pIL-10RB0.7861.3 × 10−3
uPA0.7714.0 × 10−3
CXCL50.7612.3 × 10−3
LIF-R0.7603.5 × 10−3
CX3CL10.7576.2 × 10−3
SCF0.7523.4 × 10−3
Flt3L0.7524.0 × 10−3
TWEAK0.7477.7 × 10−3
1 Greater than 0.5 within 99.9% CI. 2 From the ANCOVA. Legend. AUC: area under the curve. For abbreviations of PEA-tested proteins, see Table S1.
Table 2. Spearman’s correlation coefficients between the most discriminatory PEA-tested proteins and classical AD biomarkers within the AD-MCI group. For abbreviations of PEA-tested proteins, see Table S1.
Table 2. Spearman’s correlation coefficients between the most discriminatory PEA-tested proteins and classical AD biomarkers within the AD-MCI group. For abbreviations of PEA-tested proteins, see Table S1.
Protein NameAβ42p-taut-tau
SIRT20.070.48 **0.33
HGF−0.170.37 *0.35
MMP-10−0.37 *0.200.09
IL-10RB−0.110.290.17
uPA0.000.37 *0.29
CXCL5−0.17−0.020.07
LIF-R−0.030.51 **0.35 *
CX3CL10.010.42 *0.30
SCF−0.030.250.17
Flt3L−0.140.250.17
TWEAK−0.090.43 **0.29
* 0.01 < p < 0.05; ** 0.001 < p < 0.005.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Gaetani, L.; Bellomo, G.; Parnetti, L.; Blennow, K.; Zetterberg, H.; Di Filippo, M. Neuroinflammation and Alzheimer’s Disease: A Machine Learning Approach to CSF Proteomics. Cells 2021, 10, 1930. https://doi.org/10.3390/cells10081930

AMA Style

Gaetani L, Bellomo G, Parnetti L, Blennow K, Zetterberg H, Di Filippo M. Neuroinflammation and Alzheimer’s Disease: A Machine Learning Approach to CSF Proteomics. Cells. 2021; 10(8):1930. https://doi.org/10.3390/cells10081930

Chicago/Turabian Style

Gaetani, Lorenzo, Giovanni Bellomo, Lucilla Parnetti, Kaj Blennow, Henrik Zetterberg, and Massimiliano Di Filippo. 2021. "Neuroinflammation and Alzheimer’s Disease: A Machine Learning Approach to CSF Proteomics" Cells 10, no. 8: 1930. https://doi.org/10.3390/cells10081930

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop