Next Article in Journal
Schizophrenia-Like Behavioral Impairments in Mice with Suppressed Expression of Piccolo in the Medial Prefrontal Cortex
Next Article in Special Issue
The Signal Transducer IL6ST (gp130) as a Predictive and Prognostic Biomarker in Breast Cancer
Previous Article in Journal
Standardized Outcomes Measures in Physical Therapy Practice for Treatment and Rehabilitation of Cerebral PALSY: A Systematic Review
Previous Article in Special Issue
Circulating Tumor Cell Persistence Associates with Long-Term Clinical Outcome to a Therapeutic Cancer Vaccine in Prostate Cancer
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JPM
Volume 11
Issue 7
10.3390/jpm11070606
jpm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Salivary DNA Methylation as an Epigenetic Biomarker for Head and Neck Cancer. Part II: A Cancer Risk Meta-Analysis

by
Óscar Rapado-González
1,2,3,
Cristina Martínez-Reglero
4,
Ángel Salgado-Barreira
4,
María Arminda Santos
1,5,
Rafael López-López
3,6,
Ángel Díaz-Lagares
3,7 and
María Mercedes Suárez-Cunqueiro
1,3,6,*
1
Department of Surgery and Medical-Surgical Specialties, Medicine and Dentistry School, Universidade de Santiago de Compostela, 15782 Santiago de Compostela, Spain
2
Translational Medical Oncology Group (Oncomet), Liquid Biopsy Analysis Unit, Health Research Institute of Santiago (IDIS), 15706 Santiago de Compostela, Spain
3
Centro de Investigación Biomédica en Red de Cáncer (CIBERONC), Instituto de Salud Carlos III, 28029 Madrid, Spain
4
Methodology and Statistics Unit, Galicia Sur Health Research Institute (IISGS), 36312 Vigo, Spain
5
Department of Oral Rehabilitation, Instituto Universitario de Ciências da Saúde (IUCS), 1317|4585-116 Gandra, Portugal
6
Translational Medical Oncology Group (Oncomet), Health Research Institute of Santiago (IDIS), Complexo Hospitalario Universitario de Santiago de Compostela (SERGAS), 15706 Santiago de Compostela, Spain
7
Cancer Epigenomics, Translational Medical Oncology Group (Oncomet), Health Research Institute of Santiago (IDIS), University Clinical Hospital of Santiago (CHUS/SERGAS), 15706 Santiago de Compostela, Spain
*
Author to whom correspondence should be addressed.
J. Pers. Med. 2021, 11(7), 606; https://doi.org/10.3390/jpm11070606
Submission received: 3 May 2021 / Revised: 18 June 2021 / Accepted: 21 June 2021 / Published: 26 June 2021
(This article belongs to the Special Issue Cancer Biomarker Research and Personalized Medicine)
Browse Figures
Versions Notes

Abstract

:
Aberrant methylation of tumor suppressor genes has been reported as an important epigenetic silencer in head and neck cancer (HNC) pathogenesis. Here, we performed a comprehensive meta-analysis to evaluate the overall and specific impact of salivary gene promoter methylation on HNC risk. The methodological quality was assessed using the Newcastle–Ottawa scale (NOS). Odds ratios (ORs) and 95% confidence intervals (CIs) were calculated to evaluate the strength of the association and Egger’s and Begg’s tests were applied to detect publication bias. The frequency of salivary DNA promoter methylation was significantly higher in HNC patients than in healthy controls (OR: 8.34 (95% CI = 6.10–11.39; p < 0.01). The pooled ORs showed a significant association between specific tumor-related genes and HNC risk: p16 (3.75; 95% CI = 2.51–5.60), MGMT (5.72; 95% CI = 3.00–10.91), DAPK (5.34; 95% CI = 2.18–13.10), TIMP3 (3.42; 95% CI = 1.99–5.88), and RASSF1A (7.69; 95% CI = 3.88–15.23). Overall, our meta-analysis provides precise evidence on the association between salivary DNA promoter hypermethylation and HNC risk. Thus, detection of promoter DNA methylation in saliva is a potential biomarker for predicting HNC risk.

1. Introduction

The important role of epigenetic mechanisms in carcinogenesis has been widely reported. Identification of specific genes that are altered by aberrant epigenetic processes contributes to better understanding molecular pathogenesis in HNC [1]. As one of the most important epigenetic alterations, DNA hypermethylation may lead to transcriptional silencing of tumor suppressor genes and, thus, interfere in signaling pathways that control vital cell processes, such as DNA repair, apoptosis, cell proliferation, and cell-to-cell adhesion [2]. Gene promoter methylation is a common epigenetic event in early carcinogenesis, and therefore represents a promising biomarker for high-risk group stratification, early cancer detection, and prognosis prediction [3]. Numerous studies have evaluated DNA methylation as a biomarker in a wide variety of tumors [4,5,6,7]. Hypermethylation of tumor-related genes, such as cyclin-dependent kinase inhibitor 2A (CDKN2A), E-cadherin (CDH1), death-associated protein kinase (DAPK), phosphatase and tensin homolog (PTEN), and O6-methylguanine-DNA methyltransferase (MGMT), have been reported in HNC [8]. Likewise, various studies have focused on the detection of DNA methylation in liquid biopsies in HNC [9,10,11]. Although evidence suggests a potential association between aberrant salivary DNA methylation patterns and HNC risk, no prior research assessing overall impact is available. Therefore, we conducted a systematic review and meta-analysis to gain better insight into the magnitude of the association between salivary DNA hypermethylation and HNC risk.

2. Materials and Methods

2.1. Protocol and Registration

This study was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines [12], and the protocol was registered with the International Prospective Register of Systematic Reviews (reference No. CRD42020199123).

2.2. Search Strategy, Study Selection, and Data Extraction

The search strategy and data extraction were previously described in Part I [13].

2.3. Selection Criteria

The inclusion criteria were as follows: (1) case-control studies; (2) studies based on salivary DNA hypermethylation biomarkers for HNC; and (3) sufficient data to calculate odds ratios (ORs) and corresponding 95% confidential intervals (CIs). The exclusion criteria were as follows: (1) reviews, letters, personal opinions, book chapters, case reports, conference abstracts, and meetings; (2) duplicate publications; (3) incomplete data; and (4) in vitro or in vivo animal experiments.

2.4. Assessment of Study Quality

Independent investigators evaluated methodological quality by applying the Newcastle–Ottawa scale (NOS) [14] to each study selected. Discrepancies were resolved by consensus. For the interpretation of meta-analytic data, the NOS scale was used to score the quality of non-randomized studies based on their design, content, and ease of use. Items were scored according to a “star system” and fell under three broad categories: study group selection, group comparability, and ascertainment of exposure/outcome for case-control or cohort studies. The maximum quality score for each item was one star, except for the comparability item, which had a maximum of two stars. The NOS score ranged from 0 to 9 stars, with 8–9 stars being high quality; 6–7 stars being medium quality; and <5 stars being low quality.

2.5. Statistical Analysis

Statistical analysis was conducted using the meta package of free R software (v.3.4.4; https://www.r-project.org, accessed 30 November 2020). The pooled odds ratios (ORs) and their 95% confidence intervals (CIs) were calculated to assess the strength of the association between salivary promoter methylation and HNC. To evaluate the statistical model applied to the meta analytic database, heterogeneity was assessed on the basis of I-square (I2) value and Cochran’s Q statistic test-based Chi-squared test. Heterogeneity was considered significant when I2 > 50% and/or presence of a p < 0.10 for the Cochran’s Q test. If significant heterogeneity was detected, the DerSimonian and Laird random-effects model was applied to calculate the pooled OR with 95% CIs; otherwise, the Mantel–Haenszel fixed-effects model was used. Meta-regression and subgroup analyses were performed to explore the potential sources of heterogeneity among studies insofar as anatomic tumor location, sample type, sample size, DNA methylation method, and methylation gene profiling. Publication bias was assessed by Begg’s and Egger’s tests, and funnel plot inspection [15,16]. Begg’s rank test examines the correlation between the effect sizes and their corresponding sampling variances. Egger’s test regresses the standardized effect sizes on their precisions. In the presence of publication bias, both tests will be statistically significant. Moreover, publication bias was based on visual funnel-plot inspection, which shows the relationship between individual log ORs and their standard errors. The asymmetry of the funnel plot could indicate publication bias.
p < 0.05 was considered to be statistically significant.

3. Results

3.1. Study Selection and Characteristics of Included Studies

The main characteristics of the included studies have already been described in Part I [13].

3.2. Study Quality

Bias risk and quality were assessed according to NOS (Table S1). With respect to the selection category, each of the included studies was considered adequate. Regarding comparability, 5 out of the remaining 18 studies matched for age or gender, and 2 studies matched for at least one additional risk factor. Therefore, the median NOS score in our meta-analysis was 7.33 stars.

3.3. Association between Salivary DNA Promoter Hypermethylation and HNC Risk

A total of 7686 subjects, consisting of 4453 patients and 3233 controls, were included in this meta-analysis. As shown in Figure 1, the pooled analysis revealed a significant association between salivary DNA promoter hypermethylation and HNC with an OR of 8.34 (95% CI = 6.10–11.39; p < 0.01). A random-effects model was used because heterogeneity among the 18 studies (I2 = 72%) was identified. The shape of the Begg’s funnel plot did not reveal potential asymmetry (p = 0.271), although publication bias was detected by Egger’s test (p = 0.002) (Figure S1).

3.4. Meta-Regression and Subgroup Analysis

Due to the presence of significant heterogeneity in the overall analysis, meta-regression and subgroup analysis were performed in order to reveal potential sources. The outcomes of meta-regression analysis showed that sample type (p = 0.128), sample size (p = 0.349), and DNA methylation method (p = 0.275) were not significant sources of heterogeneity. However, anatomic tumor location (p = 0.002) and gene profiling (p < 0.001) were, in fact, potential sources of heterogeneity in this study (Table S1—see Part I) [13]. As shown in Table S2, significant heterogeneity was found in all subgroups. With respect to sample type-based subgroup analysis, a significant association between promoter hypermethylation and HNC was found in oral rinse samples (OR: 9.42; 95% CI = 6.30–14.08) and saliva samples (OR: 6.33; 95% CI = 3.90–10.27). In tumor-based subgroup analysis, methylation rates were higher in specific head and neck locations compared to studies that made no differentiation. The pooled OR for oropharyngeal cancer was 13.26 (95% CI = 3.17–5.42) and for oral cancer was 13.07 (95% CI = 8.19–20.88), while for HNC it was 5.78 (95% CI = 3.86–8.67). A significant association between salivary promoter methylation and HNC was found by both MSP (OR: 9.06; 95% CI = 6.30–13.03) and qMSP (OR: 6.81; 95% CI = 3.70–12.54) techniques. With respect to the subgroups categorized by sample size, a significant association was found between salivary promoter methylation and HNC in studies with N < 100 (OR: 9.58; 95% CI = 6.44–14.27) and N > 100 (OR: 8.34; 95% CI = 6.10–11.39). In subgroup analysis based on the gene-profiling approach, salivary promoter hypermethylated gene panels had a significantly higher association to HNC risk (OR: 36.79; 95% CI = 16.81–81.32) than hypermethylated single genes (OR: 6.02; 95% CI = 4.46–8.13).

3.5. Association between p16 Promoter Hypermethylation and HNC Risk

A total of 410 cases and 399 controls from 9 studies were included to estimate the effect of p16 promoter hypermethylation on HNC risk. As shown in Figure 2, a significant association was found between salivary p16 promoter hypermethylation and HNC risk (OR: 3.75; 95% CI = 2.51–5.60). The shape of the Begg’s funnel plot did not reveal potential asymmetry (p = 1), although publication bias was detected by Egger’s test (p = 0.040) (Figure S2).

3.6. Association between MGMT Promoter Hypermethylation and HNC Risk

A total of 328 cases and 231 controls from 5 studies were included to estimate the effect of MGMT promoter hypermethylation on HNC risk. As shown in Figure 3, salivary MGMT promoter hypermethylation was associated with an increased HNC risk (OR: 5.72; 95% CI = 3.00–10.91). Visual analysis of the funnel plot revealed a symmetrical distribution of the studies (Egger’s test, p = 0.767; Begg’s test, p = 0.624), indicating no evidence of publication bias (Figure S3).

3.7. Association between DAPK Promoter Hypermethylation and HNC Risk

A total of 270 cases and 123 controls from 4 studies were included to estimate the effect of DAPK promoter hypermethylation on HNC risk. As shown in Figure 4, the rate of salivary DAPK promoter hypermethylation was significantly higher in HNC patients compared to controls (OR: 5.34; 95% CI = 2.18–13.10). Visual examination of the funnel plot revealed a symmetrical distribution of the studies (Begg’s test, p = 0.041; Egger’s test, p = 0.187;), indicating no evidence of publication bias (Figure S4).

3.8. Association between TIMP3 Promoter Hypermethylation and HNC Risk

A total of 328 cases and 236 controls from 4 studies were included to estimate the effect of TIMP3 promoter hypermethylation on HNC risk. As shown in Figure 5, a significant association was found between salivary TIMP3 promoter hypermethylation and HNC risk (OR: 3.42; 95% CI = 1.99–5.88). Visual inspection of the funnel plot revealed a symmetrical distribution of the studies (Begg’s test, p = 0.174; Egger’s test, p = 0.419), indicating no evidence of publication bias (Figure S5).

3.9. Association between RASSF1A Promoter Hypermethylation and HNC Risk

A total of 191 cases and 192 controls from 3 studies were included to estimate the effect of RASSF1A promoter hypermethylation on HNC risk. As shown in Figure 6, salivary RASSF1A promoter hypermethylation was associated with an increased HNC risk (OR: 7.69; 95% CI = 3.88–15.23). Visual examination of the funnel plot revealed a symmetrical distribution of the studies (Begg’s test, p = 0.601; Egger’s test, p = 0.858), indicating no evidence of publication bias (Figure S6).

3.10. Association between APC Promoter Hypermethylation and HNC Risk

A total of 156 cases and 74 controls from 3 studies were included to estimate the effect of APC promoter hypermethylation on HNC risk. As shown in Figure 7, salivary APC promoter hypermethylation was not significantly associated with HNC (OR: 2.15; 95% CI = 0.84–5.51). Visual examination of the funnel plot revealed no potential asymmetry (Begg’s test, p = 0.601; Egger’s test, p = 0.609), indicating no evidence of publication bias (Figure S7).

4. Discussion

Aberrant DNA hypermethylation has been recognized as an important epigenetic mechanism involved in head and neck carcinogenesis [1], suggesting its potential as a biomarker for evaluating cancer risk. Although prior studies have focused on the detection of promoter DNA hypermethylation in saliva from HNC patients [10,17], the evidence of a direct relationship is unclear and findings have been inconsistent.
To the best of our knowledge, this is the first meta-analysis evaluating the contribution of salivary promoter hypermethylation to HNC risk. The present comprehensive analysis included 18 studies comprising 4453 patients and 3233 controls. Overall, our results indicate that salivary promoter hypermethylation was significantly associated with an 8.34-fold increase in HNC risk.
As significant heterogeneity was observed among studies, meta-regression and subgroup analyses were carried out based on anatomic tumor location, sample type, sample size, DNA methylation method, and methylation gene profiling. The stratified analysis revealed that salivary DNA hypermethylation was associated with HNC risk in all subgroups. The association between salivary DNA promoter hypermethylation and HNC risk was stronger in oral rinses compared to saliva. This could be explained by the higher methylation proportion of oral exfoliated cells in oral rinse compared to saliva samples. Subgroup analysis of anatomic tumor location showed that the OR was higher in oral cancer and oropharyngeal cancer than overall HNC. These findings could be explained by the direct contact of saliva samples with tumors located in the oral cavity and oropharynx, which could result in an increased number of exfoliated tumoral cells during sample collection. Based on the methylation detection method subgroup, the frequency of salivary DNA promoter methylation was higher in MSP than in qMSP. This may be because MSP was the most commonly used technique for detecting aberrant DNA methylation in saliva samples (11 studies). In addition, the qualitative nature and lower specificity of MSP could lead to an overestimation of methylation data compared to qMSP methods [18]. However, quantitative approaches, such as qMSP or pyrosequencing, have shown better sensitivity than MSP [19]. With respect to sample size, a similar significant association was found between n < 100 and n > 100 subgroups. On the other hand, the gene profiling subgroup revealed that HNC risk was clearly higher when aberrant gene-specific DNA methylation was analyzed using gene panels rather than single gene analysis. This suggests that multiple tumor suppressor genes are epigenetically silenced in HNC pathogenesis, and, therefore, gene methylation panels should be used to better identify HNC risk.
We also explored the association between gene-specific promoter DNA methylation and HNC risk by analyzing the methylation frequency of genes reported in at least three studies. Thus, promoter hypermethylation of p16, DAPK, TIMP3, MGMT, and RASSF1A was significantly higher in HNC patients compared to controls, suggesting that the methylation of these tumor suppressor genes may play an important role in head and neck carcinogenesis. The p16 gene acts as a negative cell cycle regulator that prevents the inactivation of retinoblastoma (Rb) protein by inhibiting the cyclin-dependent kinases and, therefore, cell cycle progression at G1/S phase [20]. Hypermethylation of p16 promoter has been reported as a frequent epigenetic event in oral carcinogenesis [21,22]. In the present meta-analysis, methylation of p16 promoter was significantly associated with a 3.75-fold increase in HNC risk, which is consistent with the study by Shi et al. (OR: 3.37) based on tissue and liquid biopsy methylation data [23]. In line with this, a more recent meta-analysis comprising 67 case-control studies reported an OR of 6.72. However, subgroup analysis in this study based on sample type revealed that OR was much higher in saliva (OR: 12.45) and blood (OR: 16.40) than in tissue (OR: 6.40) [24]. Overall, these findings indicate that hypermethylation of p16 gene promoter in saliva could be a predictive biomarker for HNC risk. The MGMT gene is involved in the repair of O6-methylguanine in DNA sequences originating from the carcinogenic effects of alkylating agents [25]. The inactivation of MGMT promoter by aberrant hypermethylation has been associated with an increased frequency of GC > AT transition mutations in TP53 and in KRAS oncogene, contributing to carcinogenesis and tumor progression [26,27]. In fact, our meta-analysis showed that methylation of MGMT promoter leads to a 5.72-fold increase in HNC risk. DAPK plays a critical role in the apoptotic process triggered by interferon-gamma (IFN-γ), tumor necrosis factor (TNF)-alpha, Fas ligand, and detachment from extracellular matrix [28]. Hypermethylation of DAPK gene promoter is a frequent alteration in HNC [29,30]. The results of the present meta-analysis show that individuals with salivary hypermethylation of DAPK gene promoter had a 5.34-fold higher HNC risk. A previous meta-analysis also showed that the frequency of DAPK promoter methylation was significantly higher in HNC vs. control groups (OR: 6.72) [31]. The TIMP3 gene is a tissue inhibitor of matrix metalloproteinases, which acts as a potential anticancer agent by inducing apoptosis and inhibiting proliferation, angiogenesis, and metastasis [32]. The methylation of TIMP3 promoter has been associated with HNC [33,34]. Interestingly, our meta-analysis revealed a significant association between salivary TIMP3 promoter methylation and HNC with an OR of 3.42. The RASSF1A gene prevents tumorigenesis through multiple cellular process, such as cell cycle arrest, migration, microtubular stabilization, and apoptosis promotion [35]. Epigenetic inactivation of RASSF1A by hypermethylation has been observed in various cancers, including HNC [36]. Our data showed that methylation of RASSF1A promoter led to a 7.69-fold increase in HNC risk compared to the control group. In a previous study, Meng et al. evaluated the methylation prevalence of RASSF1A between cancerous tissues and controls, finding a significant association (OR: 2.93) between aberrant methylation of RASSF1A and HNC [37]. The APC gene acts as a negative regulator in the Wnt/beta-catenin signaling pathway and its dysfunction leads to increased β-catenin transcriptional activity, promoting the activation of downstream targets involved in tumorigenesis, such as cyclin D1 and Myc [38]. Hypermethylation of the APC promoter has been reported as a mechanism for APC-gene inactivation in oral carcinogenesis [39]. Our study did not reveal a significant association between salivary APC promoter hypermethylation and HNC, which could be explained by the low APC-gene methylation rates detected in saliva from HNC patients. Until now, few studies have reported APC hypermethylation in saliva from HNC patients [40,41,42]; however, this epigenetic alteration has been frequently observed in head and neck tumors [29,39,43,44]. It is important to note that hypermethylation of p16, DAPK, TIMP3, MGMT, and RASSF1A plays an important role in the carcinogenesis of various tumors, such as lung, breast, colorectal, renal, or gastric [45,46,47,48,49,50]. In line with this, several studies have focused on the association of cancer risk with the hypermethylation of these tumor suppressor genes [51,52,53,54,55], which highlights its potential for early diagnosis of the disease.
The present study has several strengths. It is the first meta-analysis highlighting the association between salivary DNA promoter hypermethylation and HNC. It explores the magnitude of the association both overall and by specific hypermethylated gene. In addition, it involved a comprehensive literature review without language restrictions. However, our study is not exempt from limitations. Firstly, all included research involved case-control retrospective studies, which could lead to selection bias. Some bias could also stem from the fact that cases and controls were not matched for demographic variables, such as age, sex, and lifestyle habits. Secondly, significant heterogeneity was found among studies. Despite performing subgroup analysis by anatomic tumor location, sample type, sample size, DNA methylation method, and methylation gene profiling, we were unable to elucidate the potential sources of this heterogeneity. Further subgroup analysis was hindered by the lack of original data regarding lifestyle habits or ethnicity. Thirdly, the association of salivary DNA promoter hypermethylation and clinicopathological variables (i.e., TNM stage, histological grade) was not explored due to insufficient data. Therefore, well-designed prospective clinical studies with large sample sizes are necessary to validate the results of this meta-analysis.

5. Conclusions

Overall, the findings from this meta-analysis showed that salivary DNA promoter hypermethylation was associated with HNC risk. Salivary hypermethylation of p16, MGMT, DAPK, TIMP3, and RASSF1A showed an important role in HNC development. Thus, saliva could be used as a potential source of epigenetic biomarkers for predicting HNC. The development of HNC screening programs based on the combination of these 5-methylated genes in saliva could be useful for identifying high-risk patients and for detecting cancer before the occurrence of initial clinical symptoms. The clinical implementation of this salivary panel would represent the beginning of precision medicine for HNC. To attain this, prospective and multicenter studies should be carried out in order to validate the present results.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/jpm11070606/s1, Figure S1: Funnel plot for studies (of 18 studies) on the association between salivary DNA hypermethylation and HNC, Figure S2: Funnel plot for studies (of 9 studies) on the association between salivary hypermethylation of p16 gene promoter and HNC, Figure S3: Funnel plot for studies (of 5 studies) on the association between salivary hypermethylation of MGMT gene promoter and HNC, Figure S4: Funnel plot for studies (of 4 studies) on the association between salivary hypermethylation of DAPK gene promoter and HNC, Figure S5 Funnel plot for studies (of 4 studies) on the association between salivary hypermethylation of TIMP3 gene promoter and HNC, Figure S6: Funnel plot for studies (of 3 studies) on the association between salivary hypermethylation of RASSF1A gene promoter and HNC, Figure S7: Funnel plot for studies (of 3 studies) on the association between salivary hypermethylation of APC gene promoter and HNC, Table S1: The Newcastle-Ottawa Scale (NOS) for assessing the quality of included studies, Table S2: Subgroup analysis of salivary DNA methylation for HNC detection based on different covariates.

Author Contributions

Conceptualization, Ó.R.-G. and M.M.S.-C.; methodology, Ó.R.-G. and M.M.S.-C.; software, C.M.-R. and Á.S.-B.; formal analysis, C.M.-R. and Á.S.-B.; investigation, Ó.R.-G., Á.D.-L. and M.M.S.-C.; data curation, Ó.R.-G., Á.D.-L. and M.M.S.-C.; Writing—Original draft preparation, Ó.R.-G. and M.M.S.-C.; Writing—Review and Editing, Á.D.-L. and R.L.-L.; visualization, Ó.R.-G. and M.A.S.; supervision, M.M.S.-C.; project administration, M.M.S.-C.; funding acquisition, M.M.S.-C. All authors have read and agreed to the published version of the manuscript.

Funding

This work was co-funded by the Instituto de Salud Carlos III (ISCIII) (PI20/01449) and the European Regional Development Fund (FEDER). ADL is funded by a contract “Juan Rodés” from ISCIII (JR17/00016).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Conflicts of Interest

R.L.-L. reports other from Nasasbiotech, during the conduct of the study; grants and personal fees from Roche, grants and personal fees from Merck, personal fees from AstraZeneca, personal fees from Bayer, personal fees and non-financial support from BMS, personal fees from Pharmamar, personal fees from Leo, outside the submitted work. The rest of the authors have nothing to disclose. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Castilho, R.M.; Squarize, C.H.; Almeida, L.O. Epigenetic modifications and head and neck cancer: Implications for tumor progression and resistance to therapy. Int. J. Mol. Sci. 2017, 18, 1506. [Google Scholar] [CrossRef] [PubMed]
  2. Pfeifer, G.P. Defining driver DNA methylation changes in human cancer. Int. J. Mol. Sci. 2018, 19, 1166. [Google Scholar] [CrossRef] [Green Version]
  3. Laird, P.W. The power and the promise of DNA methylation markers. Nat. Rev. Cancer 2003, 3, 253–266. [Google Scholar] [CrossRef]
  4. Hulbert, A.; Jusue-Torres, I.; Stark, A.; Chen, C.; Rodgers, K.; Lee, B.; Griffin, C.; Yang, A.; Huang, P.; Wrangle, J.; et al. Early detection of lung cancer using DNA promoter hypermethylation in plasma and sputum. Clin. Cancer Res. 2017, 23, 1998–2005. [Google Scholar] [CrossRef] [Green Version]
  5. Patai, Á.V.; Valcz, G.; Hollósi, P.; Kalmár, A.; Peterfia, B.; Wichmann, B.; Spisák, S.; Barták, B.K.; Leiszter, K.; Tóth, K.; et al. Comprehensive DNA methylation analysis reveals a common ten-gene methylation signature in colorectal adenomas and carcinomas. PLoS ONE 2015, 10, e0133836. [Google Scholar] [CrossRef]
  6. Glodzik, D.; Bosch, A.; Hartman, J.; Aine, M.; Vallon-Christersson, J.; Reuterswärd, C.; Karlsson, A.; Mitra, S.; Niméus, E.; Holm, K.; et al. Comprehensive molecular comparison of BRCA1 hypermethylated and BRCA1 mutated triple negative breast cancers. Nat. Commun. 2020, 11, 1–15. [Google Scholar] [CrossRef]
  7. Chou, J.L.; Huang, R.L.; Shay, J.; Chen, L.Y.; Lin, S.J.; Yan, P.S.; Chao, W.T.; Lai, Y.H.; Lai, Y.L.; Chao, T.K.; et al. Hypermethylation of the TGF-β target, ABCA1 is associated with poor prognosis in ovarian cancer patients. Clin. Epigenet. 2015, 7, 1. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  8. Gaździcka, J.; Gołąbek, K.; Strzelczyk, J.K.; Ostrowska, Z. Epigenetic modifications in head and neck cancer. Biochem. Genet. 2020, 58, 213–244. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  9. Schröck, A.; Leisse, A.; De Vos, L.; Gevensleben, H.; Dröge, F.; Franzen, A.; Wachendörfer, M.; Schröck, F.; Ellinger, J.; Teschke, M.; et al. Free-circulating methylated DNA in blood for diagnosis, staging, prognosis, and monitoring of head and neck squamous cell carcinoma patients: An observational prospective cohort study. Clin. Chem. 2017, 63, 1288–1296. [Google Scholar] [CrossRef] [PubMed]
  10. Liyanage, C.; Wathupola, A.; Muraleetharan, S.; Perera, K.; Punyadeera, C.; Udagama, P. Promoter hypermethylation of tumor-suppressor genes p16INK4a, RASSF1A, TIMP3, and PCQAP/MED15 in salivary DNA as a quadruple biomarker panel for early detection of oral and oropharyngeal cancers. Biomolecules 2019, 9, 148. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. Sanchez-Cespedes, M.; Esteller, M.; Wu, L.; Nawroz-Danish, H.; Yoo, G.H.; Koch, W.M.; Jen, J.; Herman, J.G.; Sidransky, D. Gene promoter hypermethylation in tumors and serum of head and neck cancer patients. Cancer Res. 2000, 60, 892–895. [Google Scholar]
  12. McInnes, M.; Moher, D.; Thombs, B.D.; McGrath, T.A.; Bossuyt, P.M.; Clifford, T.; Cohen, J.F.; Deeks, J.J.; Gatsonis, C.; Hooft, L.; et al. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies. JAMA 2018, 319, 388–396. [Google Scholar] [CrossRef] [PubMed]
  13. Rapado-González, Ó.; Martínez-Reglero, C.; Salgado-Barreira, Á.; Muinelo-Romay, L.; Muinelo-Lorenzo, J.; López-López, R.; Díaz-Lagares, Á.; Suárez-Cunqueiro, M. Salivary DNA methylation as an epigenetic biomarker for head and neck cancer. Part I: A diagnostic accuracy meta-analysis. J. Pers. Med. 2021, 11, 568. [Google Scholar] [CrossRef]
  14. Wells, G.A.; Shea, B.; O’Connell, D.; Peterson, J.; Welch, V.; Losos, M.; Tugwell, P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomized studies in meta-analysis. Available online: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp (accessed on 16 December 2019).
  15. Egger, M.; Smith, G.D.; Schneider, M.; Minder, C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997, 315, 629–634. [Google Scholar] [CrossRef] [Green Version]
  16. Begg, C.B.; Mazumdar, M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994, 50, 1088. [Google Scholar] [CrossRef] [PubMed]
  17. Guerrero-Preston, R.; Soudry, E.; Acero, J.; Orera, M.; Moreno-López, L.A.; Macía-Colón, G.; Jaffe, A.; Berdasco, M.; Ili, C.; Brebi-Mieville, P.; et al. NID2 and HOXA9 promoter hypermethylation as biomarkers for prevention and early detection in oral cavity squamous cell carcinoma tissues and saliva. Cancer Prev. Res. 2011, 4, 1061–1072. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Claus, R.; Wilop, S.; Hielscher, T.; Sonnet, M.; Dahl, E.; Galm, O.; Jost, E.; Plass, C. A systematic comparison of quantitative high-resolution DNA methylation analysis and methylation-specific PCR. Epigenetics 2012, 7, 772–780. [Google Scholar] [CrossRef] [Green Version]
  19. Lee, E.-S.; Issa, J.-P.; Roberts, D.B.; Williams, M.D.; Weber, R.S.; Kies, M.S.; El-Naggar, A.K. Quantitative promoter hypermethylation analysis of cancer-related genes in salivary gland carcinomas: Comparison with methylation-specific PCR technique and clinical significance. Clin. Cancer Res. 2008, 14, 2664–2672. [Google Scholar] [CrossRef] [Green Version]
  20. Padhi, S.S.; Roy, S.; Kar, M.; Saha, A.; Roy, S.; Adhya, A.; Baisakh, M.; Banerjee, B. Role of CDKN2A/p16 expression in the prognostication of oral squamous cell carcinoma. Oral Oncol. 2017, 73, 27–35. [Google Scholar] [CrossRef]
  21. Kulkarni, V.; Saranath, D. Concurrent hypermethylation of multiple regulatory genes in chewing tobacco associated oral squamous cell carcinomas and adjacent normal tissues. Oral Oncol. 2004, 40, 145–153. [Google Scholar] [CrossRef]
  22. Su, P.F.; Huang, W.L.; Wu, H.T.; Wu, C.H.; Liu, T.Y.; Kao, S.Y. p16INK4A promoter hypermethylation is associated with invasiveness and prognosis of oral squamous cell carcinoma in an age-dependent manner. Oral Oncol. 2010, 46, 734–739. [Google Scholar] [CrossRef] [PubMed]
  23. Shi, H.; Chen, X.; Lu, C.; Gu, C.; Jiang, H.; Meng, R.; Niu, X.; Huang, Y.; Lu, M. Association between P16INK4a promoter methylation and HNSCC: A meta-analysis of 21 published studies. PLoS ONE 2015, 10, e0122302. [Google Scholar] [CrossRef] [Green Version]
  24. Zhou, C.; Shen, Z.; Ye, D.; Li, Q.; Deng, H.; Liu, H.; Li, J. The association and clinical significance of CDKN2A promoter methylation in head and neck squamous cell carcinoma: A meta-analysis. Cell. Physiol. Biochem. 2018, 50, 868–882. [Google Scholar] [CrossRef]
  25. Zhao, J.J.; Li, H.Y.; Wang, D.; Yao, H.; Sun, D.W. Abnormal MGMT promoter methylation may contribute to the risk of esophageal cancer: A meta-analysis of cohort studies. Tumor Biol. 2014, 35, 10085–10093. [Google Scholar] [CrossRef] [PubMed]
  26. Matsuda, S.; Mafune, A.; Kohda, N.; Hama, T.; Urashima, M. Associations among smoking, MGMT hypermethylation, TP53-mutations, and relapse in head and neck squamous cell carcinoma. PLoS ONE 2020, 15, e0231932. [Google Scholar] [CrossRef] [PubMed]
  27. Zuo, C.; Ai, L.; Ratliff, P.; Suen, J.Y.; Hanna, E.; Brent, T.P.; Fan, C.Y. O6-methylguanine-DNA methyltransferase gene: Epigenetic silencing and prognostic value in head and neck squamous cell carcinoma. Cancer Epidemiol. Biomark. Prev. 2004, 13, 967–975. [Google Scholar]
  28. Cohen, O.; Kimchi, A. DAP-kinase: From functional gene cloning to establishment of its role in apoptosis and cancer. Cell Death Differ. 2001, 8, 6–15. [Google Scholar] [CrossRef] [Green Version]
  29. Šupić, G.; Kozomara, R.; Branković-Magić, M.; Jović, N.; Magić, Z. Gene hypermethylation in tumor tissue of advanced oral squamous cell carcinoma patients. Oral Oncol. 2009, 45, 1051–1057. [Google Scholar] [CrossRef]
  30. Dammann, R.H.; Steinmann, K.; Sandner, A.; Schagdarsurengin, U. Frequent promoter hypermethylation of tumor-related genes in head and neck squamous cell carcinoma. Oncol. Rep. 2009, 22, 1519–1526. [Google Scholar] [CrossRef] [Green Version]
  31. Cai, F.; Xiao, X.; Niu, X.; Zhong, Y. Association between promoter methylation of DAPK gene and HNSCC: A meta-analysis. PLoS ONE 2017, 12, e0173194. [Google Scholar] [CrossRef]
  32. Su, C.W.; Lin, C.W.; Yang, W.E.; Yang, S.F. TIMP-3 as a therapeutic target for cancer. Ther. Adv. Med. Oncol. 2019, 11. [Google Scholar] [CrossRef] [Green Version]
  33. Rettori, M.M.; De Carvalho, A.C.; Longo, A.L.B.; De Oliveira, C.Z.; Kowalski, L.P.; Carvalho, A.L.; Vettore, A.L. TIMP3 and CCNA1 hypermethylation in HNSCC is associated with an increased incidence of second primary tumors. J. Transl. Med. 2013, 11, 316. [Google Scholar] [CrossRef] [Green Version]
  34. Sun, W.; Zaboli, D.; Wang, H.; Liu, Y.; Arnaoutakis, D.; Khan, T.; Khan, Z.; Koch, W.M.; Califano, J.A. Detection of TIMP3 promoter hypermethylation in salivary rinse as an independent predictor of local recurrence-free survival in head and neck cancer. Clin. Cancer Res. 2012, 18, 1082–1091. [Google Scholar] [CrossRef] [Green Version]
  35. Dubois, F.; Bergot, E.; Zalcman, G.; Levallet, G. RASSF1A, puppeteer of cellular homeostasis, fights tumorigenesis, and metastasis—An updated review. Cell Death Dis. 2019, 10, 1–13. [Google Scholar] [CrossRef]
  36. Raos, D.; Ulamec, M.; Bojanac, A.K.; Bulic-Jakus, F.; Jezek, D.; Sincic, N. Epigenetically inactivated RASSF1A as a tumor biomarker. Bosn. J. Basic Med. Sci. 2020. [Google Scholar] [CrossRef]
  37. Meng, R.W.; Li, Y.C.; Chen, X.; Huang, Y.X.; Shi, H.; Du, D.D.; Niu, X.; Lu, C.; Lu, M.X. Aberrant methylation of RASSF1A closely associated with HNSCC, a meta-Analysis. Sci. Rep. 2016, 6, 20756. [Google Scholar] [CrossRef] [Green Version]
  38. Zhang, L.; Shay, J.W. Multiple roles of APC and its therapeutic implications in colorectal cancer. J. Natl. Cancer Inst. 2017, 109, 109. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  39. Uesugi, H.; Uzawa, K.; Kawasaki, K.; Shimada, K.; Moriya, T.; Tada, A.; Shiiba, M.; Tanzawa, H. Status of reduced expression and hypermethylation of the APC tumor suppressor gene in human oral squamous cell carcinoma. Int. J. Mol. Med. 2005, 15, 597–602. [Google Scholar] [CrossRef] [PubMed]
  40. Rettori, M.M.; De Carvalho, A.C.; Longo, A.L.B.; De Oliveira, C.Z.; Kowalski, L.P.; Carvalho, A.; Vettore, A.L. Prognostic significance of TIMP3 hypermethylation in post-treatment salivary rinse from head and neck squamous cell carcinoma patients. Carcinogenesis 2012, 34, 20–27. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  41. Nagata, S.; Hamada, T.; Yamada, N.; Yokoyama, S.; Kitamoto, S.; Kanmura, Y.; Nomura, M.; Kamikawa, Y.; Yonezawa, S.; Sugihara, K. Aberrant DNA methylation of tumor-related genes in oral rinse. Cancer 2012, 118, 4298–4308. [Google Scholar] [CrossRef]
  42. Righini, C.A.; De Fraipont, F.; Timsit, J.-F.; Faure, C.; Brambilla, E.; Reyt, E.; Favrot, M.-C. Tumor-specific methylation in saliva: A promising biomarker for early detection of head and neck cancer recurrence. Clin. Cancer Res. 2007, 13, 1179–1185. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  43. López, F.; Sampedro, T.; Llorente, J.L.; Dominguez, F.; Hermsen, M.; Suárez, C.; Álvarez-Marcos, C. Utility of MS-MLPA in DNA methylation profiling in primary laryngeal squamous cell carcinoma. Oral Oncol. 2014, 50, 291–297. [Google Scholar] [CrossRef]
  44. Chen, K.; Sawhney, R.; Khan, M.; Benninger, M.S.; Hou, Z.; Sethi, S.; Stephen, J.K.; Worsham, M.J. Methylation of multiple genes as diagnostic and therapeutic markers in primary head and neck squamous cell carcinoma. Arch. Otolaryngol. Head Neck Surg. 2007, 133, 1131–1138. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  45. Seike, M.; Gemma, A.; Hosoya, Y.; Hemmi, S.; Taniguchi, Y.; Fukuda, Y.; Yamanaka, N.; Kudoh, S. Increase in the frequency of p16INK4 gene inactivation by hypermethylation in lung cancer during the process of metastasis and its relation to the status of p53. Clin. Cancer Res. 2000, 6, 4307–4313. [Google Scholar] [PubMed]
  46. Wahab, A.H.A.; El-Mezayen, H.A.; Sharad, H.; Rahman, S.A. Promoter hypermethylation of RASSF1A, MGMT, and HIC-1 genes in benign and malignant colorectal tumors. Tumor Biol. 2011, 32, 845–852. [Google Scholar] [CrossRef] [PubMed]
  47. Masson, D.; Rioux-Leclercq, N.; Fergelot, P.; Jouan, F.; Mottier, S.; Théoleyre, S.; Bach-Ngohou, K.; Patard, J.J.; Denis, M. Loss of expression of TIMP3 in clear cell renal cell carcinoma. Eur. J. Cancer 2010, 46, 1430–1437. [Google Scholar] [CrossRef]
  48. Van Der Auwera, I.; Bovie, C.; Svensson, C.; Trinh, X.B.; Limame, R.; Van Dam, P.; Van Laere, S.J.; Van Marck, E.A.; Dirix, L.Y.; Vermeulen, P.B. Quantitative methylation profiling in tumor and matched morphologically normal tissues from breast cancer patients. BMC Cancer 2010, 10, 97. [Google Scholar] [CrossRef] [Green Version]
  49. Krassenstein, R.; Sauter, E.; Dulaimi, E.; Battagli, C.; Ehya, H.; Klein-Szanto, A.; Cairns, P. Detection of breast cancer in nipple aspirate fluid by CpG island hypermethylation. Clin. Cancer Res. 2004, 10, 28–32. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  50. Mittag, F.; Kuester, D.; Vieth, M.; Peters, B.; Stolte, B.; Roessner, A.; Schneider-Stock, R. DAPK promotor methylation is an early event in colorectal carcinogenesis. Cancer Lett. 2006, 240, 69–75. [Google Scholar] [CrossRef]
  51. Belinsky, S.A.; Klinge, D.M.; Dekker, J.D.; Smith, M.W.; Bocklage, T.J.; Gilliland, F.D.; Crowell, R.E.; Karp, D.D.; Stidley, C.A.; Picchi, M.A. Gene promoter methylation in plasma and sputum increases with lung cancer risk. Clin. Cancer Res. 2005, 11, 6505–6511. [Google Scholar] [CrossRef] [Green Version]
  52. Shi, D.-T.; Han, M.; Gao, N.; Tian, W.; Chen, W. Association of RASSF1A promoter methylation with gastric cancer risk: A meta-analysis. Tumor Biol. 2013, 35, 943–948. [Google Scholar] [CrossRef] [PubMed]
  53. Chen, R.; Zheng, Y.; Zhuo, L.; Wang, S. Association between MGMT promoter methylation and risk of breast and gynecologic cancers: A systematic review and meta-Analysis. Sci. Rep. 2017, 7, 12783. [Google Scholar] [CrossRef] [Green Version]
  54. Cao, J.; Li, Z.; Yang, L.; Liu, C.; Luan, X. Association between tissue inhibitor of metalloproteinase-3 gene methylation and gastric cancer risk: A meta-Analysis. Genet. Test. Mol. Biomark. 2016, 20, 427–431. [Google Scholar] [CrossRef] [PubMed]
  55. Tang, B.; Li, Y.; Qi, G.; Yuan, S.; Wang, Z.; Yu, S.; Li, B.; He, S. Clinicopathological significance of cdkn2a promoter hypermethylation frequency with pancreatic cancer. Sci. Rep. 2015, 5, srep13563. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Jpm 11 00606 g001 550
Figure 1. Forest plot for the association between salivary DNA promoter hypermethylation and the HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Figure 1. Forest plot for the association between salivary DNA promoter hypermethylation and the HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Jpm 11 00606 g001
Jpm 11 00606 g002 550
Figure 2. Forest plot for the association between p16 promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Figure 2. Forest plot for the association between p16 promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Jpm 11 00606 g002
Jpm 11 00606 g003 550
Figure 3. Forest plot for the association between MGMT promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Figure 3. Forest plot for the association between MGMT promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Jpm 11 00606 g003
Jpm 11 00606 g004 550
Figure 4. Forest plot for the association between DAPK promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Figure 4. Forest plot for the association between DAPK promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Jpm 11 00606 g004
Jpm 11 00606 g005 550
Figure 5. Forest plot for the association between TIMP3 promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Figure 5. Forest plot for the association between TIMP3 promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Jpm 11 00606 g005
Jpm 11 00606 g006 550
Figure 6. Forest plot for the association between RASSF1A promoter hypermethylation and the HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Figure 6. Forest plot for the association between RASSF1A promoter hypermethylation and the HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Jpm 11 00606 g006
Jpm 11 00606 g007 550
Figure 7. Forest plot for the association between APC promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Figure 7. Forest plot for the association between APC promoter hypermethylation and HNC risk. The squares represent the ORs for individual studies. Bars represent the 95% CIs. The center of the diamond represents the summary effect size.
Jpm 11 00606 g007
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Rapado-González, Ó.; Martínez-Reglero, C.; Salgado-Barreira, Á.; Santos, M.A.; López-López, R.; Díaz-Lagares, Á.; Suárez-Cunqueiro, M.M. Salivary DNA Methylation as an Epigenetic Biomarker for Head and Neck Cancer. Part II: A Cancer Risk Meta-Analysis. J. Pers. Med. 2021, 11, 606. https://doi.org/10.3390/jpm11070606

AMA Style

Rapado-González Ó, Martínez-Reglero C, Salgado-Barreira Á, Santos MA, López-López R, Díaz-Lagares Á, Suárez-Cunqueiro MM. Salivary DNA Methylation as an Epigenetic Biomarker for Head and Neck Cancer. Part II: A Cancer Risk Meta-Analysis. Journal of Personalized Medicine. 2021; 11(7):606. https://doi.org/10.3390/jpm11070606

Chicago/Turabian Style

Rapado-González, Óscar, Cristina Martínez-Reglero, Ángel Salgado-Barreira, María Arminda Santos, Rafael López-López, Ángel Díaz-Lagares, and María Mercedes Suárez-Cunqueiro. 2021. "Salivary DNA Methylation as an Epigenetic Biomarker for Head and Neck Cancer. Part II: A Cancer Risk Meta-Analysis" Journal of Personalized Medicine 11, no. 7: 606. https://doi.org/10.3390/jpm11070606

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop