Next Article in Journal
Opportunities and Obstacles to the Development of Health Data Warehouses in Hospitals in France: The Recent Experience of Comprehensive Cancer Centers
Previous Article in Journal
Nutritional Analysis of the Spanish Population: A New Approach Using Public Data on Consumption
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 20
Issue 2
10.3390/ijerph20021644
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

What Are the Determinants of the Quality of Systematic Reviews in the International Journals of Occupational Medicine? A Methodological Study Review of Published Literature

by
Giuseppe La Torre
1,*,
Remigio Bova
1,
Rosario Andrea Cocchiara
1,
Cristina Sestili
1,
Anna Tagliaferri
1,
Simona Maggiacomo
1,
Camilla Foschi
1,
William Zomparelli
1,
Maria Vittoria Manai
1,
David Shaholli
1,
Vanessa India Barletta
1,
Luca Moretti
1,
Francesca Vezza
1 and
Alice Mannocci
2
1
Department of Public Health and Infectious Diseases, Sapienza University of Rome, 00185 Rome, Italy
2
Faculty of Economics, Universitas Mercatorum, 00185 Rome, Italy
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2023, 20(2), 1644; https://doi.org/10.3390/ijerph20021644
Submission received: 27 November 2022 / Revised: 12 January 2023 / Accepted: 14 January 2023 / Published: 16 January 2023
Browse Figures
Review Reports Versions Notes

Abstract

:
Objective: The aim of this study was to evaluate the methodological quality of systematic reviews published in occupational medicine journals from 2014 to 2021. Methods: Papers edited between 2014 and 2021 in the 14 open access journals with the highest impact were assessed for their quality. Studies were included if they were systematic reviews and meta-analyses, and if they were published in English. Results: The study included 335 studies. Among these, 149 were meta-analyses and 186 were systematic reviews. The values of the AMSTAR-2 score range between three and fourteen with a mean value of 9.85 (SD = 2.37). The factors that significantly and directly associate to a higher AMSTAR-2 score were impact factor (p = 0.003), number of consulted research databases (p = 0.011), declaration of PRISMA statement (p = 0.003), year of publication (p < 0.001) and performing a meta-analysis (p < 0.001).The R² values from the multivariate analysis showed that the AMSTAR-2 score could be predicted by the inclusion of these parameters by up to 23%. Conclusions: This study suggests a quality assessment methodology that could help readers in a fast identification of good systematic reviews or meta-analyses. Future studies should analyze more journals without applying language restrictions and consider a wider range of years of publication in order to give a more robust evidence for results.

1. Introduction

The use of reporting guidelines is uncommon in Public Health (PH) research, and no specific guideline application has been recommended at the European level [1,2,3].
The first formal assessment of Systematic Reviews (SRs) in medicine was performed by Cynthia Mulrow, who identified several poor reporting practices of 50 medical review articles published between June 1985 and June 1986 [4]. In the last 10 years, several reporting guidelines have been developed for reporting SRs. In 1999, an international group of 30 epidemiologists, clinicians, statisticians, editors and researchers developed a reporting guideline for meta-analyses of randomized trials—the QUOROM (QUality Of Reporting Of Meta-analyses) Statement [5]. In 2005, a meeting was convened to update QUOROM to address several conceptual and practical advances in the methodology of SRs and to help overcome several shortcomings identified in an audit of SRs [6]. The guideline was renamed the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) Statement, and it was published in 2009 [7]. It was accompanied by an explanation and elaboration document, which provided detailed guidance for each of the 27 items included and examples of exemplar reporting [8]. Since its publication, seven extensions to the PRISMA Statement have been developed to facilitate reporting of different types or aspects of SRs [9,10,11,12,13,14,15,16,17,18,19,20]. These guidelines present a sequence of indications that should enhance the understanding and interpretation of studies which may be difficult for the reader, such as relevant information which may not be adequately described or perhaps poorly presented [8,21].
In recent years, different assessment tools have been developed to assess the methodological quality of SRs, and among these tools, the ones used most commonly are AMSTAR, ROBIS and SIGN. The “Assessing the Methodological Quality of Systematic Reviews” (AMSTAR) was created in 2007 [22] and updated (AMSTAR-2) [23] in 2017 for facilitating the development of high-quality reviews by focusing on their methodological quality. AMSTAR 2 is one of the most widely used instruments for enabling rapid and reproducible assessments of the quality of systematic reviews in both randomised and non-randomised studies of healthcare interventions.
ROBIS is used to assess the risk of bias in SRs [24]. This tool is based on three phases, the first one being optional, and the other two covering 4 four domains each, i.e., identifying and appraising study eligibility criteria, identification and selection of studies, data collection and study appraisal, and synthesis and findings, for the evaluation of the overall risk of bias.
SIGN has been developed by the Scottish Intercollegiate Guidelines Network for systematic review checklist; it is a 12-item checklist developed on the basis of the AMSTAR checklist [25]. The objective of the tools developed by the SIGN is to improve the quality of health care for patients by reducing variation in practice and outcome, through the development and dissemination of national clinical guidelines containing recommendations for effective practice based on current evidence.
The aim of this study was to evaluate the methodological quality of the systematic reviews published in occupational medicine journals between 2014 and 2021 and to assess related factors of the scientific publication that are associated with the methodological quality assessed by the AMSTAR-2.

2. Materials and Methods

2.1. Eligibility Criteria

This research study included all the systematic reviews and meta-analyses published in the 14 occupational medicine journals with the highest impact from January 2014 to December 2021. All the papers written in a different period were excluded, as well as randomized control trials, narrative reviews, surveys, reports, protocols and pilot studies. All the articles not published in English language were excluded. The typology of the study (e.g., review or other) had to be mentioned in the title or abstract.

2.2. Information Sources

This study is a methodological study review carried out by checking for publications in the period 2014–2021 of open access occupational medicine journals with the highest impact.
Only English papers were selected.
The following occupational medicine journals were analyzed:
(1)
American Journal of Industrial Medicine;
(2)
Annals of Occupational Hygiene;
(3)
Archives of Environmental and Occupational Health;
(4)
Environmental Health Perspectives;
(5)
International Journal of Environmental Research and Public Health;
(6)
International Journal of Occupational and Environmental Health;
(7)
International Journal of Occupational Medicine and Environmental Health;
(8)
International Journal of Occupational Safety and Ergonomics;
(9)
Journal of Occupational and Environmental Hygiene;
(10)
Journal of Occupational and Environmental Medicine;
(11)
Journal of Occupational Medicine and Toxicology;
(12)
Occupational and Environmental Medicine;
(13)
Occupational Medicine;
(14)
Scandinavian Journal of Work, Environment & Health.
Each issue of the 14 journals, excluding supplements, was examined.

2.3. Selection of Sources of Evidence

The websites of the journals were consulted to evaluate the instructions for the authors. Literature search and data extraction were performed by two authors independently. Only systematic review and meta-analysis study designs were selected.

2.4. Data Items

The choice of items to be selected was based on the results of previous research in different health fields. Pauletto et al. made a critical appraisal of systematic reviews of intervention in dentistry, using as explanatory variables of methodological quality items such as publication year, continent and journal impact factor [26].
Cheung et al. assessed the methodological quality of systematic reviews on Chinese herbal medicine, finding higher levels in SRs conducted by more authors and published in higher impact factor journals [27].
McGregor et al. found that the quality of meta-analyses of non-opioid, pharmacological, perioperative interventions for chronic postsurgical pain is associated with time of publication and journal impact factor [28].
Yuan et al., assessing Breast Reconstruction Reviews, studied the association between AMSTAR score and impact factor, number of citations, number of studies and adherence to the PRISMA statement [29].
Chow et al., making the quality appraisal of systematic reviews on methods of labor induction, studied the association between study quality and number of citations, journal impact factor and publication year [30].
Yuan et al. made the assessment of the Quality of Systematic Reviews and Meta-Analyses About Breast Augmentation by studying the association between AMSTAR score and journal impact factor, number of citations, year of publication, number of included studies and adherence to PRISMA guidelines [31].
The following variables were collected:
  • Name of the first author;
  • Name of the journal;
  • Title of the article;
  • Year of publication;
  • Adherence to PRISMA Checklist made by the authors (yes/no);
  • Declaration by the authors of the use of PRISMA Checklist (yes/no);
  • Journal impact factor;
  • Numbers of authors;
  • Nationality of the first author;
  • Number of investigated databases in the review;
  • Multi-country research group;
  • Total number of articles included in the systematic review;
  • Implementation of meta-analysis;
  • Total AMSTAR-2 score: “yes” (all the criteria were met), “no” (none of the criteria were met), “partial yes” (not all the criteria were met).
  • AMSTAR-2 checklist.

2.5. AMSTAR-2 Checklist

All the authors were involved in the assessment, and two researchers scored the same paper. If differences in the assessment were present, a third reviewer was involved.
Even though AMSTAR2 is an instrument that explicitly states that it is not designed to generate an overall score, it can be useful for rating the overall confidence in the results of the review. Hence, we decided to calculate an overall score and use this as the dependent variable in the statistical analysis, as executed by several authors [32,33,34,35].
The AMSTAR-2 checklist answers to the following 16 questions:
-
Did the research questions and inclusion criteria for the review include the components of PICO?
-
Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?
-
Did the review authors explain their selection of the study design for inclusion in the review?
-
Did the review authors use a comprehensive literature search strategy?
-
Did the review authors perform study selection in duplicate?
-
Did the review authors perform data extraction in duplicate?
-
Did the review authors provide a list of excluded studies and justify the exclusions?
-
Did the review authors describe the included studies in adequate detail?
-
Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?
-
Did the review authors report on the sources of funding for the studies included in the review?
-
If a meta-analysis was performed, did the review authors use appropriate methods for the statistical combination of results?
-
If a meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?
-
Did the review authors account for RoB in primary studies when interpreting/discussing the results of the review?
-
Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?
-
If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?
-
Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?

2.6. Statistical Analysis

The description of results included qualitative variable description reporting the percentage of articles that declared and used the PRISMA statement, the AMSTAR-2 score frequency and distribution, a univariate and bivariate analysis of AMSTAR-2 score vs. qualitative and quantitative variables and a multivariate regression model that analyzed the influence of the variables on the AMSTAR-2 score.
A descriptive analysis was performed in order to summarize the characteristics of the articles. The AMSTAR-2 score was described with mean and standard deviation (SD), while the qualitative items (multi-country research team, PRISMA declared, PRISMA applied, meta-analysis) were described by frequencies and percentages. A univariate analysis was performed in order to assess the association between the AMSTAR-2 score and the collected variables from the form sheet. The correlation analysis was carried out to assess the relationship between two quantitative variables. Pearson’s r coefficient (also known as the linear correlation coefficient), used for variables measured by interval scales or relation scales, was reported. A multivariate analysis was conducted with a backward stepwise elimination procedure of non-significant variables generating a minimal model. The inclusion criterion for the covariates in the models was a significance level of <0.20 from the univariate analysis.
The goodness of fit of the models was assessed using the R2.
The level of significance was set at p < 0.05. The statistical analysis was performed using SPSS (Statistical Package for Social Sciences) software, version 27.

3. Results

Two different authors independently identified 1017 reviews (systematic reviews, narrative reviews, protocols, surveys and pilot studies) published in the selected journals between January 2014 and December 2021. From these 1017 potentially relevant articles, 553 were excluded since they were not systematic reviews and 123 were out of scope; 341 were selected. Among these, six articles were excluded because they were protocols. The number of articles included was 335, out of which 149 were meta-analyses and 186 were systematic reviews. The PRISMA flow diagram represented in Figure 1 summarizes this selection process.
Concerning the AMSTAR-2 score, Figure 2 shows the distribution of AMSTAR-2 score frequencies among the included studies. Values range between three and fourteen with a mean value of 9.85 (SD = 2.38) and a median value of 10. In Figure 3, the percentages of systematic reviews not reporting the 16 items of the AMSTAR-2 are shown, indicating the highest non-adherence for the items funding (74%), excluded study list (70.2%) and impact of risk of bias (52.5%).
The occupational health areas are described in Table 1. Half of the systematic reviews are represented by papers on toxicology (13.2%), health promotion and intervention (11%), environmental exposure and climate changes (10.7%), mental and neurological health (9.6%) and musculoskeletal disorders (9.6%).

3.1. Univariate and Bivariate Analysis

Table 2 shows the univariate analysis of AMSTAR-2 score regarding the qualitative variables taken in consideration for this study. The resulting AMSTAR-2 score was significantly associated with all qualitative variables such as multi-country research team, PRISMA statement application, PRISMA statement declared and meta-analysis. In fact, having a multi-country research team, having a PRISMA statement declared in the text, having applied PRISMA and having a meta-analysis increased the AMSTAR-2 score.
The correlation analysis is shown in Table 3. The correlation analysis studied the relationship between quantitative variables. A significant correlation with the AMSTAR-2 score was found for impact factor (p = 0.001), year of publication (p < 0.001), increasing the number of authors (p = 0.047) and the number of investigated databases (p = 0.008).
Conversely, the number of articles included resulted as not correlated with the AMSTAR-2 score achieved by the study.

3.2. Multivariate Analysis

The multivariate analysis assessed the influence of qualitative and quantitative variables on the AMSTAR-2 score (Table 4). The year of publication (p < 0.001), the impact factor (p = 0.003), the number of research databases (p = 0.011), PRISMA statement declared (p = 0.003), the inclusion of a meta-analysis (p < 0.001), and the number of authors showed to be statistically significantly associated to the AMSTAR-2 score. A multi-country research team and the journal impact factor did not show a statistical significance for an increased AMSTAR-2 score.
The R² from the multivariate analysis shows that the AMSTAR-2 score can be predicted by the inclusion of the parameters by up to 23%.
Table 5 shows in which areas the findings of the multivariate analysis are similar to what was found using all the SRs (Table 4). It is clear that the most frequent significant independent variables associated to the AMSTAR-2 score found for each area were year of publication, number of research databases and meta-analysis.

4. Discussion

The main objective of this study was to define the scientific articles’ quality and evaluate how qualitative or quantitative variables could influence the AMSTAR-2 score.
This study shows that number of authors, number of research databases, declared and applied PRISMA statement and meta-analysis inclusion increase the AMSTAR-2 score. Conversely, a multi-country research team and the journal impact factor did not significantly affect the AMSTAR-2 score.
An evident relationship between AMSTAR-2 score and the study impact factor score was found, and this is in agreement with the results found by Cheung et al. (Chinese Medicine) [27], McGregor et al. (pain management) [28], and in disagreement with Pauletto et al. (dentistry) [26] and Chow et al. (gynecology and obstetrics) [30]. In our study, an increase in the impact factor was associated with a higher score of the AMSTAR-2, suggesting that occupational medicine journals with a higher impact factor are more likely to publish systematic reviews with a higher methodological quality, as suggested by Saha et al., who found that usually researchers consider impact factor as a reasonable indicator of study quality [36].
Moreover, our study also found a significant and positive association between the AMSTAR-2 score and the number of authors in the study, and again, this is in agreement with the results found by Cheung et al. (Chinese Medicine) [27] and in disagreement with Chow et al. [30]. These findings could be linked to the so-called phenomenon of “knowledge diffusion”. As suggested by Tahamtan and coll. [37], by increasing the number of authors, a study is capable of increasing its faculty representation and the attention it receives, providing an increase in the credibility as well as the theoretical quality of evidence.
In this study, an improvement over time of the methodological quality was found, as well as the association between the PRISMA statement declaration in the systematic reviews, the number of databases used and the AMSTAR score. This is in agreement with McGregor et al. (pain management) [28], Yuan et al. (oncologic surgery) [29], Yuan et al. (cosmetic surgery) [31], Aran et al. (psychiatry) [34], Ross et al. (pain management) [35], and in disagreement with Chow et al. (gynecology and obstetrics) [30]. This fact could be related, as suggested by Shamseer et al. [38], to the adoption of reporting guidelines for systematic reviews, endorsement of complete and transparent reporting by higher impact journals and increased attention to methodological quality.
Additionally, in this study, the AMSTAR-2 score was better in SRs with a meta-analysis than in SRs without a meta-analysis, and this could be due to the fact that the meta-analyses are useful tools for summarizing research evidence, as they can give an overall panorama on a specific research question.
Finally, we found some differences in the independent variables associated with methodological quality of the SRs in different occupational health areas. However, this result must be considered with care due to the lower number of studies included in the subgroup analyses.
The first strength of this study is in the originality: similar studies could be found in the scientific literature, but they do not consider the AMSTAR-2 score compared with other qualitative and quantitative variables. A 2014 study about the impact of reporting guidelines on public health journals in Europe examined STROBE (Strengthening the Reporting of Observational Studies in Epidemiology), CONSORT (Consolidated Standards of Reporting Trials) and PRISMA and reported a strong heterogeneity in the application of guideline statements and suggested a common agreement among journals regarding research-reporting methodologies to improve the quality of PH (Public Health) research publishing [39]. A recent study analyzed the inter-rater reliability (IRR) of the AMSTAR-2 score and the ROBIS (Risk Of Bias In Systematic Reviews) scale considering individual domains and overall methodological quality/risk of bias of systematic reviews, the concurrent validity of the tools and the time required to apply them [40,41,42]. A second strength of this study is represented by the fact that the data extraction was performed independently by two authors, hence conferring robustness to the results. A multivariate regression model was then implemented in order to simultaneously analyze several variables.
The weaknesses of this study are the restriction to English language studies and the small number of investigated journals. In fact, only the 14 best occupational medicine journals were included, hence determining a quality bias for extracted data. Another possible limitation could be related to the fact that this search has not accounted for Cochrane reviews on this topic. However, we focused our attention on occupational health journals.

5. Conclusions

In conclusion, this study suggests that the rating of some variables such as number of authors, declared and applied PRISMA, number of research databases and meta-analysis inclusion, could be considered for an article quality analysis, with a predictivity in terms of R2 of 23% to define the AMSTAR-2 score. This screening could help for a fast identification of a good review or meta-analysis or, on the other hand, suggest some important elements that can have a positive association with the methodological quality of an SR. Future studies should analyze more journals without applying language restrictions and target a broader period of publication in order to furnish more robust evidence for results.

Author Contributions

Conceptualization, G.L.T. and A.M.; methodology, G.L.T. and A.M.; formal analysis, G.L.T.; investigation, R.B., R.A.C., C.S., A.T., S.M., C.F., W.Z., M.V.M., D.S., V.I.B., L.M. and F.V.; writing—original draft preparation, G.L.T. and D.S.; writing—review and editing, G.L.T., R.A.C., C.S., A.T., S.M., C.F., W.Z., M.V.M., D.S., V.I.B., L.M., F.V. and A.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data supporting reported results can be asked to the Authors.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Guo, J.W.; Sward, K.A.; Beck, S.L.; Staggers, N. Quality of Reporting Randomized Controlled Trials in Cancer Nursing Research. Nurs. Res. 2014, 63, 26–35. [Google Scholar] [CrossRef] [PubMed]
  2. Panic, N.; Leoncini, E.; De Belvis, G.; Ricciardi, W.; Boccia, S. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS ONE 2013, 26, e83138. [Google Scholar]
  3. Samaan, Z.; Mbuagbaw, L.; Kosa, D.; Debono, V.B.; Dillenburg, R.; Zhang, S.; Fruci, V.; Dennis, B.; Bawor, M.; Thabane, L. A systematic scoping review of adherence to reporting guidelines in health care literature. J. Multidiscip. Healthc. 2013, 6, 169–188. [Google Scholar]
  4. Mulrow, C.D. The Medical Review Article: State of the Science. Ann. Intern. Med. 1987, 106, 485–488. [Google Scholar] [CrossRef] [PubMed]
  5. Moher, D.; Cook, D.J.; Eastwood, S.; Olkin, I.; Rennie, D.; Stroup, D.F. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Qual. Report. Meta-Anal. Lancet 1999, 354, 1896–1900. [Google Scholar] [CrossRef]
  6. Moher, D.; Tetzlaff, J.; Tricco, A.C.; Sampson, M.; Altman, D.G. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007, 4, e78. [Google Scholar] [CrossRef] [Green Version]
  7. Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G.; Group, P. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 2009, 6, e1000097. [Google Scholar] [CrossRef] [Green Version]
  8. Liberati, A.; Altman, D.G.; Tetzlaff, J.; Mulrow, C.; Gøtzsche, P.C.; Ioannidis, J.P.; Clarke, M.; Devereaux, P.J.; Kleijnen, J.; Moher, D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. J. Clin. Epidemiol. 2009, 62, e1–e34. [Google Scholar] [CrossRef] [Green Version]
  9. Welch, V.; Petticrew, M.; Tugwell, P.; Moher, D.; O’Neill, J.; Waters, E.; White, H.; the PRISMA-Equity Bellagio group. PRISMA-Equity 2012 Extension: Reporting Guidelines for Systematic Reviews with a Focus on Health Equity. PLoS Med. 2012, 9, e1001333. [Google Scholar] [CrossRef] [Green Version]
  10. Welch, V.; the PRISMA-Equity Bellagio group; Petticrew, M.; Petkovic, J.; Moher, D.; Waters, E.; White, H.; Tugwell, P. Extending the PRISMA statement to equity-focused systematic reviews (PRISMA-E 2012): Explanation and elaboration. Int. J. Equity Health 2015, 14, 92. [Google Scholar] [CrossRef] [Green Version]
  11. Welch, V.; Petticrew, M.; Petkovic, J.; Moher, D.; Waters, E.; White, H.; Tugwell, P.; Atun, R.; Awasthi, S.; Barbour, V.; et al. Extending the PRISMA statement to equity-focused systematic reviews (PRISMA-E 2012): Explanation and elaboration. J. Clin. Epidemiol. 2016, 70, 68–89. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Beller, E.M.; Glasziou, P.P.; Altman, D.G.; Hopewell, S.; Bastian, H.; Chalmers, I.; Gøtzsche, P.C.; Lasserson, T.; Tovey, D.; For The PRISMA for Abstracts Group. PRISMA for Abstracts: Reporting Systematic Reviews in Journal and Conference Abstracts. PLoS Med. 2013, 10, e1001419. [Google Scholar] [CrossRef] [PubMed]
  13. Hutton, B.; Salanti, G.; Caldwell, D.M.; Chaimani, A.; Schmid, C.H.; Cameron, C.; Ioannidis, J.P.A.; Straus, S.; Thorlund, K.; Jansen, J.P.; et al. The PRISMA Extension Statement for Reporting of Systematic Reviews Incorporating Network Meta-analyses of Health Care Interventions: Checklist and Explanations. Ann. Intern. Med. 2015, 162, 777–784. [Google Scholar] [CrossRef] [Green Version]
  14. Stewart, L.A.; Clarke, M.; Rovers, M.; Riley, R.D.; Simmonds, M.; Stewart, G.; Tierney, J.F. Preferred reporting items for systematic review and meta-analyses of individual participant data: The PRISMA-IPD statement. JAMA 2015, 313, 1657–1665. [Google Scholar] [CrossRef] [PubMed]
  15. Moher, D.; Shamseer, L.; Clarke, M.; Ghersi, D.; Liberati, A.; Petticrew, M.; Shekelle, P.; Stewart, L.A. Preferred reporting items for systematic review and meta-analysis protocols (prisma-p) 2015 statement. Syst. Rev. 2015, 4, 1. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Shamseer, L.; Moher, D.; Clarke, M.; Ghersi, D.; Liberati, A.; Petticrew, M.; Shekelle, P.; Stewart, L.A. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ 2015, 349, g7647. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  17. Zorzela, L.; Loke, Y.K.; Ioannidis, J.P.; Golder, S.; Santaguida, P.; Altman, D.G.; Moher, B.; Vohra, S. PRISMA harms checklist: Improving harms reporting in systematic reviews. BMJ 2016, 352, i157. [Google Scholar] [CrossRef] [Green Version]
  18. Guise, J.M.; Butler, M.E.; Chang, C.; Viswanathan, M.; Pigott, T.; Tugwell, P. AHRQ series on complex intervention systematic reviews—Paper 6: PRISMA-CI extension statement and checklist. J. Clin. Epidemiol. 2017, 90, 43–50. [Google Scholar] [CrossRef]
  19. Guise, J.M.; Butler, M.; Chang, C.; Viswanathan, M.; Pigott, T.; Tugwell, P. AHRQ series on complex intervention systematic reviews—Paper 7: PRISMA-CI elaboration and explanation. J. Clin. Epidemiol. 2017, 90, 51–58. [Google Scholar] [CrossRef] [Green Version]
  20. Tricco, A.C.; Lillie, E.; Zarin, W.; O’Brien, K.K.; Colquhoun, H.; Levac, D.; Moher, D.; Peters, M.D.J.; Horsley, T.; Weeks, L.; et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann. Intern. Med. 2018, 169, 467–473. [Google Scholar] [CrossRef] [Green Version]
  21. Des Jarlais, D.C.; Lyles, C.; Crepaz, N.; Trend Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am. J. Public Health 2004, 94, 361–366. [Google Scholar] [CrossRef] [PubMed]
  22. Shea, B.J.; Reeves, B.C.; Wells, G.; Thuku, M.; Hamel, C.; Moran, J.; Moher, D.; Tugwell, P.; Welch, V.; Kristjansson, E.; et al. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017, 358, j4008. [Google Scholar] [CrossRef] [PubMed]
  23. Shea, B.; Grimshaw, J.M.; Wells, G.A.; Boers, M.; Andersson, N.; Hamel, C.; Porter, A.; Tugwell, P.; Moher, D. Development of AMSTAR-2: A Measurement Tool to Assess Systematic Reviews. BMC Med. Res. Methodol. 2007, 7, 10. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Whiting, P.; Savović, J.; Higgins, J.P.T.; Caldwell, D.M.; Reeves, B.C.; Shea, B.; Davies, P.; Kleijnen, J.; Churchill, R.; ROBIS group. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J. Clin. Epidemiol. 2016, 69, 225–234. [Google Scholar] [CrossRef] [Green Version]
  25. Scottish Intercollegiate Guidelines Network. Methodology Checklist 1: Systematic Reviews and Meta-Analyses. Available online: https://www.sign.ac.uk/what-we-do/methodology/checklists/ (accessed on 12 January 2023).
  26. Pauletto, P.; Polmann, H.; Réus, J.C.; de Oliveira, J.M.D.; Chaves, D.; Lehmkuhl, K.; Massignan, C.; Stefani, C.M.; Martins, C.C.; Flores-Mir, C.; et al. Critical appraisal of systematic reviews of intervention in dentistry published between 2019–2020 using the AMSTAR 2 tool. Evid.-Based Dent. 2022, 1–8. [Google Scholar] [CrossRef]
  27. Cheung, A.K.L.; Wong, C.H.L.; Ho, L.; Wu, I.X.Y.; Ke, F.Y.T.; Chung, V.C.H. Methodological quality of systematic reviews on Chinese herbal medicine: A methodological survey. BMC Complement. Med. Ther. 2022, 22, 48. [Google Scholar] [CrossRef]
  28. McGregor, R.H.; Warner, F.M.; Linde, L.D.; Cragg, J.J.; Osborn, J.A.; Varshney, V.P.; Schwarz, S.K.W.; Kramer, J.L.K. Quality of meta-analyses of non-opioid, pharmacological, perioperative interventions for chronic postsurgical pain: A systematic review. Reg. Anesth. Pain Med. 2022, 47, 263–269. [Google Scholar] [CrossRef]
  29. Yuan, M.; Wu, J.; Austin, R.E.; Hofer, S.O.P.; Lista, F.; Ahmad, J. Evaluating Breast Reconstruction Reviews Using A Measurement Tool to Assess Systematic Reviews (AMSTAR). Plast. Reconstr. Surg. Glob. Open 2021, 9, e3897. [Google Scholar] [CrossRef]
  30. Chow, R.; Li, A.; Wu, N.; Martin, M.; Wessels, J.M.; Foster, W.G. Quality appraisal of systematic reviews on methods of labour induction: A systematic review. Arch. Gynecol. Obstet. 2021, 304, 1417–1426. [Google Scholar] [CrossRef]
  31. Yuan, M.; Wu, J.; Austin, R.E.; Lista, F.; Ahmad, J. Evaluating the Quality of Systematic Reviews and Meta-Analyses About Breast Augmentation Using AMSTAR. Aesthetic Surg. J. Open Forum 2021, 3, ojab020. [Google Scholar] [CrossRef]
  32. Scott, J.; Howard, B.; Sinnett, P.; Schiesel, M.; Baker, J.; Henderson, P.; Vassar, M. Variable methodological quality and use found in systematic reviews referenced in STEMI clinical practice guidelines. Am. J. Emerg. Med. 2017, 35, 1828–1835. [Google Scholar] [CrossRef] [PubMed]
  33. Nissen, T.; Wayant, C.; Wahlstrom, A.; Sinnett, P.; Fugate, C.; Herrington, J.; Vassar, M. Methodological quality, completeness of reporting and use of systematic reviews as evidence in clinical practice guidelines for paediatric overweight and obesity. Clin. Obes. 2017, 7, 34–45. [Google Scholar] [CrossRef]
  34. Aran, G.; Hicks, C.; Demand, A.; Johnson, A.L.; Beaman, J.; Bailey, Y.; Haught, M.; Lane, A.; Sinnett, P.; Vassar, M. Treating schizophrenia: The quality of evidence behind treatment recommendations and how it can improve. BMJ Evid.-Based Med. 2020, 25, 138–142. [Google Scholar] [CrossRef] [PubMed]
  35. Ross, A.; Rankin, J.; Beaman, J.; Murray, K.; Sinnett, P.; Riddle, R.; Haskins, J.; Vassar, M. Methodological quality of systematic reviews referenced in clinical practice guidelines for the treatment of opioid use disorder. PLoS ONE 2017, 12, e0181927. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  36. Saha, S.; Saint, S.; Christakis, D.A. Impact factor: A valid measure of journal quality? J. Med. Libr. Assoc. 2003, 91, 42–46. [Google Scholar]
  37. Tahamtan, I.; Safipour Afshar, A.; Ahamdzadeh, K. Factors affecting number of citations: A comprehensive review of the literature. Scientometrics 2016, 107, 1195–1225. [Google Scholar] [CrossRef]
  38. Shamseer, L.; Stevens, A.; Skidmore, B.; Turner, L.; Altman, D.G.; Hirst, A.; Hoey, J.; Palepu, A.; Simera, I.; Schulz, K.; et al. Does Journal endorsement of reporting guidelines influence the completeness of reporting of health research? A systematic review protocol. Syst. Rev. 2012, 1, 24. [Google Scholar] [CrossRef] [Green Version]
  39. Mannocci, A.; Saulle, R.; Colamesta, V.; D’Aguanno, S.; Giraldi, G.; Maffongelli, E.; Meggiolaro, A.; Semyonov, L.; Unim, B.; La Torre, G. What is the impact of reporting guidelines on Public Health journals in Europe? The case of STROBE, CONSORT and PRISMA. J. Public Health 2015, 37, 737–740. [Google Scholar] [CrossRef] [Green Version]
  40. Banzi, R.; Cinquini, M.; Gonzalez-Lorenzo, M.; Pecoraro, V.; Capobussi, M.; Minozzi, S. Quality assessment versus risk of bias in systematic reviews: AMSTAR-2 and ROBIS had similar reliability but differed in their construct and applicability. J. Clin. Epidemiol. 2018, 99, 24–32. [Google Scholar] [CrossRef]
  41. Gates, A.; Gates, M.; Duarte, G.; Cary, M.; Becker, M.; Prediger, B.; Vandermeer, B.; Fernandes, R.M.; Pieper, D.; Hartling, L. Evaluation of the reliability, usability, and applicability of AMSTAR-2, AMSTAR-2 2, and ROBIS: Protocol for a descriptive analytic study. Syst. Rev. 2018, 7, 85. [Google Scholar] [CrossRef] [Green Version]
  42. Pieper, D.; Puljak, L.; González-Lorenzo, M.; Minozzi, S. Minor differences were found between AMSTAR-2 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. J. Clin. Epidemiol. 2019, 108, 26–33. [Google Scholar] [CrossRef] [PubMed]
Ijerph 20 01644 g001 550
Figure 1. PRISMA flow-chart diagram of study selection.
Figure 1. PRISMA flow-chart diagram of study selection.
Ijerph 20 01644 g001
Ijerph 20 01644 g002 550
Figure 2. Histogram of the AMSTAR-2 score Frequency Distribution.
Figure 2. Histogram of the AMSTAR-2 score Frequency Distribution.
Ijerph 20 01644 g002
Ijerph 20 01644 g003 550
Figure 3. Bar chart of the percentages of systematic reviews not reporting the 16 items of the AMSTAR-2.
Figure 3. Bar chart of the percentages of systematic reviews not reporting the 16 items of the AMSTAR-2.
Ijerph 20 01644 g003
Table
Table 1. Occupational health areas covered by the systematic reviews.
Table 1. Occupational health areas covered by the systematic reviews.
Occupational Health AreaN%
Toxicology4413.2
Health promotion and intervention3711.0
Environmental exposure and climate changes3610.7
Mental and neurological health329.6
Musculoskeletal disorders329.6
Stress and burnout236.9
Workplace injury and Violence216.3
Non-communicable diseases185.4
Shift work144.2
Return to work123.6
Vaccination and infectious diseases113.3
Healthcare workers92.7
Asbestos82.4
Ergonomy72.1
Respiratory disease61.8
Sickness absence61.8
Tobacco smoking51.5
Physical risk (ionizing and non-ionizing radiations, noise, vibration)51.5
Precarious work and Unemployment51.5
Other41.2
Table
Table 2. Univariate Analysis of the AMSTAR-2 score vs. Qualitative Variables.
Table 2. Univariate Analysis of the AMSTAR-2 score vs. Qualitative Variables.
Qualitative VariablesAMSTAR-2 Score Mean (SD)p Value
Multi-country
Research Team		NO9.88 (2.47)0.762
YES9.79 (1.99)
Prisma Statement DeclaredNO9.26 (2.53)<0.001
YES10.35 (2.11)
Prisma Statement AppliedNO9.01 (2.61)0.002
YES10.08 (2.25)
Meta-AnalysisNO9.38 (2.38)<0.001
YES10.44 (2.23)
Table
Table 3. Pearson’s correlation analysis of AMSTAR-2 score vs. quantitative variables.
Table 3. Pearson’s correlation analysis of AMSTAR-2 score vs. quantitative variables.
Variablesrp Value
Impact Factor0.1760.001
Publication year0.316<0.001
N. authors0.1090.047
N. of Research Database used0.1450.008
N. Articles Included0.0830.135
Table
Table 4. Multivariate regression model of AMSTAR-2 score as outcome.
Table 4. Multivariate regression model of AMSTAR-2 score as outcome.
Covariatesβp
Impact Factor0.1540.003
Year of publication0.274<0.001
N. of Research Databases0.1280.011
Prisma Statement Declared (YES/NO *)0.1530.003
Meta-Analysis (YES/NO *)0.211<0.001
N. of Authors0.1110.046
R2 = 0.226
* Reference group.
Table
Table 5. Independent variables present in the multivariate analysis for each occupational health area.
Table 5. Independent variables present in the multivariate analysis for each occupational health area.
Occupational Health Area
Independent VariableToxicologyHealth Promotion and InterventionEnvironmental
Exposure and Climate Changes		Mental and Neurological HealthMusculoskeletal DisordersStress and Burnout
Impact Factor x x
Year of publicationx xx
N. of Research Databasesxx x
Prisma Statement Declared
(YES/NO *)		 x x
Meta-Analysis (YES/NO *)x x x
N. of Authors
* Reference group.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

La Torre, G.; Bova, R.; Cocchiara, R.A.; Sestili, C.; Tagliaferri, A.; Maggiacomo, S.; Foschi, C.; Zomparelli, W.; Manai, M.V.; Shaholli, D.; et al. What Are the Determinants of the Quality of Systematic Reviews in the International Journals of Occupational Medicine? A Methodological Study Review of Published Literature. Int. J. Environ. Res. Public Health 2023, 20, 1644. https://doi.org/10.3390/ijerph20021644

AMA Style

La Torre G, Bova R, Cocchiara RA, Sestili C, Tagliaferri A, Maggiacomo S, Foschi C, Zomparelli W, Manai MV, Shaholli D, et al. What Are the Determinants of the Quality of Systematic Reviews in the International Journals of Occupational Medicine? A Methodological Study Review of Published Literature. International Journal of Environmental Research and Public Health. 2023; 20(2):1644. https://doi.org/10.3390/ijerph20021644

Chicago/Turabian Style

La Torre, Giuseppe, Remigio Bova, Rosario Andrea Cocchiara, Cristina Sestili, Anna Tagliaferri, Simona Maggiacomo, Camilla Foschi, William Zomparelli, Maria Vittoria Manai, David Shaholli, and et al. 2023. "What Are the Determinants of the Quality of Systematic Reviews in the International Journals of Occupational Medicine? A Methodological Study Review of Published Literature" International Journal of Environmental Research and Public Health 20, no. 2: 1644. https://doi.org/10.3390/ijerph20021644

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop