Epidemiology and Reporting Characteristics of Systematic Reviews in Orthopedic Journals: A Meta-Epidemiological Study

Systematic reviews (SRs) with complete reporting or rigorous methods can lead to less biased recommendations and decisions. A comprehensive analysis of the epidemiological and reporting characteristics of SRs in orthopedics is lacking. We evaluated 360 SRs, including 165 and 195 published in orthopedic journals in 2012 and 2022. According to the established reporting guidelines, we examined these SRs for key epidemiological characteristics, including focus areas, type of meta-analysis (MA), and reporting characteristics. Most SRs (71%) were therapy-related, with a significant proportion originating from authors in the USA, UK, and China. Pairwise MA was performed on half of the SRs. The proportion of protocol registrations improved by 2022 but remained low (33%). Despite a formal declaration of adherence to the reporting guidelines (68%), they were often not used and reported enough. Only 10% of the studies used full search strategies, including trial registries. Publication bias assessments, subgroup analyses, and sensitivity analyses were not even planned. The risk of bias assessment improved in 2022; however, the certainty of the evidence remained largely unassessed (8%). The use and reporting of standard methods in orthopedic SRs have remained suboptimal. Thus, authors, peer reviewers, journal editors, and readers should criticize the results more.


Introduction
Systematic reviews (SRs) collate and summarize available evidence on specific research questions [1].SRs can estimate the magnitude of benefits and harm, which can be reliable [2].Moreover, SRs with complete reporting or rigorous implementation methods can lead to less biased recommendations and decisions [3,4].
SRs have increased over the last 10 years, raising research waste concerns [2,5].Metaepidemiological studies have highlighted the mass production of SRs [2,6].The epidemiology and reporting characteristics of SRs were reported in biomedical journals in 2016 [7].The review revealed that many SRs were poorly performed.Thus, it reported and called for strategies to help reduce this avoidable waste in research.Following this review, the characteristics of SRs in specific fields, such as imaging and nutrition, were also underreported [8,9].
However, the epidemiology and reporting characteristics of SRs in orthopedics have not been sufficiently investigated.Most studies have been limited to reviews assessed for reporting or methodological quality (Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) or Assessment of Multiple Systematic Review) [10,11].Therefore, the epidemiology of SRs in orthopedic journals, such as the prevalence, types of reviews (therapy, epidemiology, diagnosis, or prognosis reviews), types of search databases, and characteristics of primary studies, remains unknown.
This study aimed to describe and compare the epidemiological and reporting characteristics of SRs published in orthopedic journals in 2012 and 2022.Through rigorous evaluation, we aimed to elucidate the current state of orthopedic SRs and identify areas of improvement.

Study Design and Protocol
This cross-sectional meta-epidemiological study was registered with osf.io (https://osf.io/z63ye (accessed on 8 October 2023)), and we followed the reporting guidelines [12].Publicly available data were used.Ethical approval and patient consent were not required for this study.

Study Search and Selection
We selected SRs published in orthopedic journals in 2012 and 2022.We followed the definition of SR by Moher et al. and PRISMA protocols (PRISMA-P) [13,14].We included SRs published in English, including those on therapy, epidemiology (prevalence, etiology), diagnosis, and prognosis [7].We excluded overviews of SRs (umbrella reviews), scoping reviews, and meta-epidemiological studies that provide data for methodological analysis following the original or secondary studies [7].
We defined the orthopedic journals based on the 2022 Thomas Institute of Science Information list [15], including 125 journals (Supplementary Materials S1).We included orthopedic SRs based on the following definitions: orthopedics is a branch of medicine that focuses on the care of the musculoskeletal system, consisting of the muscles, bones, joints, ligaments, and tendons.We defined publication dates in 2012 or 2022 as electronic pre-publications or traditional journal publication dates.
We searched for SRs published in orthopedic journals in 2012 and 2022 using MED-LINE (PubMed) with search strategies (Supplementary Materials S2) [16].

Screening
First, we randomly selected 500 studies (250 each in 2012 and 2022) using a random number generator in Microsoft Excel.Second, two of the six reviewers (NY, TM, TA, YTo, HO, and DS) independently screened the titles and abstracts and assessed their eligibility based on their full texts.Disagreements between both reviewers were resolved through discussion.Otherwise, a third reviewer acted as an arbiter (ST or YTs).

Data Extraction
Two of the six reviewers independently extracted data from the included studies using a standardized data collection form.We collected the epidemiological and reporting characteristics based on previous studies on the epidemiology and reporting of SRs [7] and the PRISMA 2020 statement [17] (Supplementary Materials S3).
The epidemiological characteristics in the SRs were as follows: journal name, journal impact factor (JIF) (2021) [15], number of authors, country of corresponding author, focus of review (therapy (treatment, prevention), epidemiology (prevalence, etiology), diagnosis, prognosis) [7], SR category (completely new (mentioning that it is a completely new SR), update of prior SR, newer scope than prior SR (e.g., new patients, intervention or outcomes reviewed), higher quality than prior SR, limitations of primary studies only (no mention of prior SR, and only mention of limitations of primary studies), others) [18], anatomical location, common International Classification of Diseases 11th Revision (ICD-11) codes, number of included studies, number of included participants, economics assessment (i.e., costs) considered, meta-analysis (MA) performed, and number of studies included in the largest Mas.
The conducting and reporting characteristics in the SRs were as follows: SR protocol registration (e.g., PROSPERO) mentioned, reporting guideline (e.g., PRISMA) mentioned, Cochrane handbook used, inclusion/exclusion criteria reported, number of databases searched (without trial registry), trial registry (e.g., ClinicalTrials.gov)searched, all identified studies screened by at least two authors, all data extracted by at least two authors, unpublished data acquired from original authors, study risk of bias/quality assessment by at least two authors, study risk of bias (RoB)/quality assessment tool used (e.g., RoB2), number of outcomes stated in the method, primary outcome stated, statistical significance of effect estimate for primary outcome, magnitude of heterogeneity (I 2 ) in the MAs for primary outcome, Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) assessment reported in a summary of findings table or text, proportion of certainty of evidence through GRADE assessment, risk of publication bias assessed (or intent to assess), subgroup analysis, sensitivity analysis, presence of conflict of interest (COI), and presence of funding.

Statistical Analysis
We summarized the data as frequency (number, %) for categorical data or median and interquartile range (IQR) for continuous data.We compared the epidemiological and reporting characteristics of SRs published in 2012 and 2022.Regarding exploratory analyses, we calculated odds ratios (95% confidence intervals (CI)) for the epidemiological and reporting characteristics of SRs published in 2022 (SRs with versus without protocol registration and those with versus without self-reported use of the PRISMA statement in therapeutic SRs).When the epidemiological or reporting characteristics were continuous variables, they were converted into binary data using the median values as the cutoff.All analyses were performed using Stata/SE 17.0.

Results
We identified 2491 SRs (Figure 1) and excluded 26 by screening for full texts (Supplementary Materials S4).Finally, we included 165 and 195 SRs published in 2012 and 2022, respectively, in the final analysis.

Epidemiological Characteristics
In total, 360 SRs were published in 80 journals, of which the highest number per journal was 15 (4%).SRs increased by 18% in 2022 compared to 2012.The JIF had a median of 3.2 (IQR: 2.6-4.4) in 2012, which declined to 2.8 (IQR: 2.5-3.5) in 2022 (Tables 1 and 2).The corresponding authors were most frequently based in the USA, UK, or China and were responsible for 185/360 (51%) of the included SRs.Most SRs (256/360 (71%)) were classified as therapy and 39/360 (11%) as prognosis.In 2012, the SR category was dominated by the "completely new" type.However, the proportion decreased to about half by 2022.In all four types of SRs, "Completely new" increased, while "Limitations of primary studies only" decreased.SRs for infections and injuries increased in all four types of SRs in 2022, whereas diseases of the musculoskeletal system or connective tissue decreased by 2022.Few SRs (14/360 (4%)) considered these costs.No "empty reviews" (i.e., identified no eligible studies) were found.A meta-analysis was performed on 180/360 (50%) SRs, with a median of eight (IQR: 5-14) studies included in the largest meta-analysis in each SR.The pairwise MA was common both in 2012 and 2022.A network meta-analysis was not performed in 2012; however, it was 3% in 2022.

Epidemiological Characteristics
In total, 360 SRs were published in 80 journals, of which the highest number per journal was 15 (4%).SRs increased by 18% in 2022 compared to 2012.The JIF had a median of 3.2 (IQR: 2.6-4.4) in 2012, which declined to 2.8 (IQR: 2.5-3.5) in 2022 (Tables 1 and 2).The corresponding authors were most frequently based in the USA, UK, or China and were responsible for 185/360 (51%) of the included SRs.Most SRs (256/360 (71%)) were classified as therapy and 39/360 (11%) as prognosis.In 2012, the SR category was dominated by the "completely new" type.However, the proportion decreased to about half by 2022.In all four types of SRs, "Completely new" increased, while "Limitations of primary studies only" decreased.SRs for infections and injuries increased in all four types of SRs in 2022, whereas diseases of the musculoskeletal system or connective tissue decreased by 2022.Few SRs (14/360 (4%)) considered these costs.No "empty reviews" (i.e., identified no eli-

Conducting and Reporting Characteristics
We summarized the reporting characteristics of 165 and 189 SRs in 2012 and 2022, respectively, and those of the four types of SRs in Table 3 and Supplementary Materials S5.In 2022, six SRs were excluded from the reporting evaluation because they did not meet the inclusion criteria.Protocol registration improved by 33% in 2022 compared to 2% in 2012, with the majority opting for registration with the International Prospective Register of Systematic Reviews (PROSPERO).Among the 2022 SRs, 7/56 (13%) were non-PROSPERO, of which two were International Platform of Registered Systematic Review and Meta-analysis Protocols, and one each was in the other registry.The authors used the reporting guidelines for 239/354 (68%) SRs.Among the SRs published in 2022 that used PRISMA-related reporting guidelines, PRISMA 2020 was used in only 23/153 (15%), whereas PRISMA 2009 was still used in 80/153 (52%).A few authors reported using Cochrane handbooks (56/354 (16%)).The number of databases was one for 27 SRs and two for 51 SRs in 2012 and 2022, respectively.A few authors searched the trial registry (26/354 (7%)).At least two authors screened 250/354 SRs (71%), whereas data extraction was performed by at least two authors in fewer SRs (40/354 (44%)).The percentage of unpublished data acquired from original authors in a few SRs (40/354 (11%)) was lower in 2022 than in 2012.
The risk of bias assessment by at least two authors improved slightly in 2022 compared to 2012.However, some authors of SRs published in 2022 did not report a risk of bias (32/189 (17%)).In SRs with MA, the risk of bias was assessed in 40/52 (77%) in 2012 and 74/76 (97%) in 2022.The primary outcome was more often specified in 2022 SRs 89/189 (47%).Among the SRs that reported an effect estimate for the primary outcome, the most common results were favorable and significant (56/128 (44%)).The SRs for therapy were more often significant (17% in 2012 and 22% in 2022).Most authors did not assess the magnitude of heterogeneity in MAs for the primary outcome (217/354 (61%)).A few authors reported the GRADE assessment (29/354 (8%)), and no improvement was observed in 2022.The certainty of evidence in MAs was very low (1-2%).Notably, most authors (approximately 80%) did not outline plans for assessing publication bias, subgroup analyses, or sensitivity analyses.

Discussion
We evaluated the characteristics of 165 and 195 SRs published in 2012 and 2022 in orthopedic journals.Most SRs focused on therapy and were conducted by authors from the USA, UK, and China.The SR category was mostly "Completely new" in 2012; however, the proportion decreased to about half by 2022.MA was performed in half of the SRs using pairwise MAs.The proportion of protocol registrations improved in 2022, but remained low.Only 10% of the studies used full search strategies, including trial registries.However, the heterogeneity in MAs has not yet been reported.Particularly, the assessment of publication bias, subgroup analysis, and sensitivity analysis was unplanned.The risk of bias assessment improved in 2022; nonetheless, the GRADE remained largely unassessed.
The epidemiological characteristics of SRs have varied over time.This tendency is consistent with the results of biomedical and imaging journals [7,8].This can be attributed to the changing clinical practice and research trends, evolving technologies, surgical techniques, and research methods.Particularly in therapy, network MA will increase in addition to the traditional pairwise MA as more RCTs on new interventions are conducted in the future.In addition, the individual participant data MA is expected to increase as the number of studies that share data increases.Once the treatment effectiveness is clarified, the next step should be further research on cost-effectiveness as a health-economic decision.
Orthopedic SRs remained suboptimal in using and reporting the standard method.Our findings align with earlier results of poor reporting in SRs of non-orthopedic fields [8,9], suggesting that these issues are not unique to orthopedics and warrant broader attention.Important items underreported in previous studies, such as protocol registration, PRISMA guidelines, and GRADE, were also poorly described in orthopedic SRs.Only 33% of orthopedic SRs in 2022 were protocol-registered, lower than the proportion in medical journals generally (55.8%) [19].In 2022, only 12% of the SRs used PRISMA 2020, indicating that the authors could not comply with the updated reporting guidelines.Only approximately 10% of SRs consider the certainty of evidence, similar to other fields [20,21].Interpreting the results based on the certainty of evidence rather than focusing only on the statistical significance of the effect estimates is essential.
The exploratory analysis revealed that the self-reported use of PRISMA was unassociated with sufficient reporting.Therefore, the formal description of the guidelines in the manuscript was not reflected in the use and description [22].Unfortunately, many journals that recommend reporting guidelines have incomplete reports [23].
We suggest some actions for improving the reporting of SRs.First, applying an electronic system powered by artificial intelligence during submission is an option to automatically check the minimum required reporting accuracy [24].Second, authors should design a method to avoid underreporting by using a format at the time of the protocol [25].Third, the journal should list the specific items that need to be described and the reporting guideline to be followed (e.g., PRISMA2020) in the submission guidelines.In addition, authors must submit the reporting guideline checklist.Finally, peer reviewers and editors should review and evaluate the SR according to the reporting checklist.
This review had several strengths.First, this was the first study to investigate the epidemiological and reporting characteristics of SRs in an orthopedic journal.The results are presented separately for each of the four SR focuses, owing to their diverse characteristics and analytical methods.Our results expand on existing knowledge by highlighting the specific shortcomings of orthopedic SRs and reaffirming the call for strategies to bolster the quality of orthopedic SRs.Second, two authors independently selected studies and extracted data to minimize the risk of measurement errors.Third, both groups of proportions were compared, ensuring a minimum of 100 SRs in each group, making 360 SRs sufficient for analysis.
This study had some limitations.First, there was a selection bias in the year of publication, studies published in the English language, and the single database search.Therefore, we may be unable to generalize our findings to SRs published in other years, languages, or databases.Second, these SRs (epidemiology, diagnosis, and prognosis) were based on a small sample size.Therefore, these proportions should be interpreted with caution.
In conclusion, the use and reporting of standard methods in orthopedic SRs have not improved significantly and remain suboptimal.Therefore, authors, peer reviewers, journal editors, and readers should be more critical of the results.Finally, strategies are needed to improve this modifiable reporting quality.

Table 1 .
Epidemiological characteristics of systematic reviews in 2012 and 2022.
SR, systematic review; MA, meta-analysis; ICD, International Classification of Diseases.Data are presented as number (percent) or median (IQR).

Table 2 .
Comparison of the epidemiological characteristics of systematic reviews in orthopedic journals published in 2012 and 2022, stratified by focus of systematic reviews.
SR, systematic review; MA, meta-analysis; ICD, International Classification of Diseases; NA, not applicable.Data are presented as number (percent) or median (IQR).Direction of change in 2012 versus 2022 means as follows; ↑, increase; ↓, decrease; =, no change.

Table 3 .
Comparison of the conducting and reporting characteristics of systematic reviews in orthopedic journals published in 2012 and 2022.
SR, systematic review; MA, meta-analysis; GRADE, Grading of Recommendations, Assessment, Development, and Evaluations; COI, conflicts of interest; MINORS, methodological index for non-randomized studies; PRISMA; Preferred Reporting Items for Systematic Reviews and Meta-Analysis; NA, not applicable.Data are presented as number (percent) or median (IQR).Direction of change in 2012 versus 2022 means as follows; ↑, increase; ↓, decrease; =, no change.