Change over Five Years in Important Measures of Methodological Quality and Reporting in Randomized Cardiovascular Clinical Trials

Objectives: The aim of our current study was to analyze whether the use of important measures of methodological quality and reporting of randomized clinical trials published in the field of cardiovascular disease research haschanged over time. A furtheraim was to investigate whether there was an improvement over time in the ability of these trials to provide a good estimate of the true intervention effect. Methods: We conducted two searches in the Cochrane Central Register of Controlled Trials (CENTAL) database to identify randomized cardiovascular clinical trials published in either 2012 or 2017. Randomized clinical trials (RCTs) trials in cardiovascular disease research with adult participants were eligible to be included. We randomly selected 250 RCTs for publication years 2012 and 2017. Trial characteristics, data on measures of methodological quality, and reporting were extracted and the risk of bias for each trial was assessed. Results: As compared to 2012, in 2017 there were significant improvements in the reporting of the presence of a data monitoring committee (42.0% in 2017 compared to 34.4% in 2012; p < 0.001), and a positive change in registering randomized cardiovascular disease research in clinical trial registries (78.4% in 2017 compared to 68.9% in 2012; p = 0.03). We also observed that significantly more RCTs reported sample size calculation (60.4% in 2017 compared to 49.6% in 2012; p < 0.01) in 2017 as compared to 2012. RCTs in 2017 were more likely to have a low overall risk of bias (RoB) than in 2012 (29.2% in 2017 compared to 21.2% in 2012; p < 0.01). However, fewer 2017 RCTs were rated low (50.8% compared to 65.6%; p < 0.001) risk for blinding of participants and personnel, for blinding of outcome assessors (82.4% compared to 90.8%; p < 0.001), and selective outcome reporting (62.8% compared to 80.0%; <0.001). Conclusions: As compared to 2012, in 2017 there were significant improvements in some, but not all, the important measures of methodological quality. Although more trials in the field of cardiovascular disease research had a lower overall RoB in 2017, the improvement over time was not consistently perceived in all RoB domains.


Introduction
Randomized clinical trials (RCTs) constitute the foundational background of modern medical practice [1].In the last three decades, the cardiovascular randomized clinical trial has emerged as the principal method by which new therapies are evaluated [2].Moreover, evidence generated from randomized clinical trials has greatly influenced the diagnosis and treatment of many heart diseases including arterial hypertension, arrhythmias, acute myocardial infarction, heart failure, and coronary revascularization [3][4][5].
The increasing prevalence of cardiovascular disease around the world requires highquality of clinical research and translation of its findings into new therapeutic and diagnostic strategies [6].
Unfortunately, although there was a significant increase in the quantity of scientific literature concerning cardiovascular disease published in recent years, it was indicated that this has not resulted in guideline recommendations with more certainty and supporting evidence.The American College of Cardiology and the American Heart Association (ACC/AHA) clinical practice guidelines are still based on a lower quality of evidence and expert opinions, indicating the lack of high-quality studies with relevant data [7].
Several tools exist that support researchers to plan and conduct high-quality research and make trial results completely and transparently available.Guidelines for clinical trial protocols (e.g., SPIRIT) facilitate trial planning in all important details.Reporting guidelines (e.g., CONSORT for RCTs) have the aim of decreasing the risk of non-reporting bias, i.e., facilitating that clinical trial methods are described as they were conducted and trial results are fully published [8].The requirement of clinical trial registration supports transparency in research.In the USA it is a requirement from the Food and Drug Administration (FDA) that all clinical trials are registered before the first patient is enrolled [9], and the European Medicines Agency and WHO also support clinical trial registration [10,11].In the field of cardiology, insufficient registration tendencies were reported [12].Cardiac and cardiovascular system journals infrequently require, recommend, and enforce the use of obligatory clinical trial registration [13].
Methodological flaws in the design, conduct, analysis, and reporting of randomized clinical trials can cause the true intervention effect to be underestimated or overestimated.This is why these systemic errors (defined as the risk of bias) are assessed when systematic reviews are conducted or evidence-based guidelines are developed [14].Concerns arising due to the high risk of bias in trials included in evidence syntheses lead to the downgrading of evidence level and consequently will decrease our certainty in the pooled results.
Our previous study compared the risk of bias in industry-funded and non-industryfunded cardiovascular disease research trials published in 2017 [15].The present study investigated tendencies over time to answer whether there was an improvement in measures of methodological quality and reporting in randomized cardiovascular clinical trials between 2012 and 2017.Further, assessed how well these trials were able to estimate the true intervention effect in 2017 as compared to 2012.

Methods
We conducted two searches to identify randomized cardiovascular clinical trials published in either 2012 or 2017.We searched the Cochrane Central Register of Controlled Trials (CENTAL) database, as this is the most comprehensive resource available containing randomized clinical trials.
We used the same search strategy for both years, containing subject headings and keywords related to adults (aged > 18 years) and cardiovascular diseases, restricted to the years 2012 or 2017.The first author (OB) searched CENTRAL and screening trials for eligibility.We included studies that were published in 2012 in English language journals, where the investigated intervention was related to cardiovascular practice, and where participants 18 years or older were included.Our search resulted in 2566 trials.All identified records were exported to Excel, where they were randomly ordered.In a subsequent step, we were reordering trials from the smallest to the highest number.
We included the first 250 (about 10%) eligible randomized clinical trials for both year 2012 and 2017.
We used a data extraction tool that was developed for assessing the methodological quality of RCTs in child health research [15].Two authors (OB, OF) independently extracted data for each study included.We discussed all unclear decisions until a consensus was established.For each cardiovascular disease study, we identified information about journal type (e.g., general or specialty medical journal and general or specialty cardiovascular journal), corresponding author's country, study type, study design, intervention type, type of control, number of study centers, study sample, primary diagnostic category in the study using the ICD-10 classification system, presence of data monitoring committee, type of primary outcome, outcome results, and trial registration.Trial registration numbers, author names, and keywords related to the specific cardiovascular intervention were used to find the published protocol via Google and Google Scholar.We completed data extraction after a precise analysis of full-text articles, trial registration, and published protocols.To retrieve information on the primary outcome of the trial, we defined primary outcome as (1) the outcome defined under the objective of the study, (2) the outcome used to calculate sample size, or (3) the first outcome reported in the randomized clinical trial.
We used the Cochrane RoB tool [16] to assess the methodological quality of randomized clinical trials.This tool evaluates 7 domains of bias and thereby determines the extent to which the RCT's design, conduct, analysis, and presentation were appropriate to answer the trial research question.These 7 domains are (1) sequence generation (whether the allocation sequence was adequately generated), (2) allocation concealment (whether the allocation of group assignment could not been foreseen prior to randomization), (3) blinding of participants and personnel (whether the knowledge of the allocated intervention was adequately prevented during study), (4) blinding of outcome assessors, (5) incomplete outcome data (whether the incomplete outcome data were adequately addressed), (6) selective outcome reporting (whether the study was free of apparent selective outcome reporting), and (7) other sources of bias (whether the study was free of other problems that could introduce bias).

Statistical Analysis
Statistical analyses were conducted by the statistical software R version 4.1.2(R Development Core Team 2021) [17].To analyze for 5-year changes in main study characteristics, we compared the 2017 sample with 250 RCTs published in 2012 [15].All collected binomial variables were analyzed using logistic regression analyses with generalized linear models.All collected categorical variables with more than two categories were analyzed with multinomial regression models.For these models, a single p-value for the entire model was presented to provide a concise overview of the overall significance.Variables of methodological quality and report were analyzed with separate univariable logistic regressions.
Model assumptions on residuals were checked using "model-checking plots".Statistical significance tests in the models were carried out with Chi-square tests.The value of p < 0.05 was considered as a significant result.

Descriptive Analysis
The main characteristics of included cardiovascular RCTs are shown in Table 1.Data from 2017 have been previously partly reported [15].Values from 2012, some data on additional measures of methodological quality and reporting for both years and statistical comparisons were novel.We observed significant differences in the country of origin defined based on the first author's affiliation between 2012 and 2017.In our 2017 sample, more publications were published in specialty medical journals (19.6% compared to 10.4%; the logistic regression result on the Type of Journal variable was: p < 0.001).In 2017 we included more RCTs with parallel design (92.4% compared to 80.4%; p < 0.01), and among the interventions there were more drug trials (55.6% compared to 46.8%) and surgical interventions (1.2% compared to 0.4%), (p < 0.001).In the 2017 sample, we had a larger number of multinational trials (27.6% compared to 18%), (p < 0.05) where developing (8.4% compared to 1.2%) and transitional economy countries (5.2% compared to 3.2%) were more often concerned (p < 0.001).In 2017 included trials were more often funded by pharmaceutical companiesor industry (p < 0.001).
Table 2 shows changes in important measures of methodological quality and reporting.The behavioral outcome included attitudes and specific (e.g., eating) behaviors; biomarkers were defined as markers measured as an indicator of biologicalor pathogenic processes or pharmacologic responses to an intervention;physiological outcomesr eflected how a patient feels, functions or survives; psychological and quality of life outcomes included different scales measuring these variables.We used 'no' when something hasn't been done when it could have been possible; and used 'N/A' when it doesn't apply to that particular trial.
As compared to 2012, we observed an improvement in 2017 in the reporting of the presence of a data monitoring committee (42.0%compared to 34.4%; p < 0.001).As compared to 2012, there was a positive change in registering trials in trial registries in 2017 and, among clinical trial registries, the clinicaltrials.govdatabase had increased popularity (registration rate in clinicaltrials.govwas: 78.4% compared to 68.9%; p = 0.03).Also, significantly more RCTs reported sample size calculation (60.4% compared to 49.6%; p < 0.01) in 2017 as compared to 2012.Although fewer RCTs specified plan to collect adverse effects in 2017 (48.4% compared to 74%; p < 0.001), they reported harms more often in 2017 (68%compared to 52%; p < 0.001).When we investigated the reporting of results, we observed that the number of RCTs with statistically significant results of the primary outcome was lower in the 2017 sample (69.2% compared to 78.8%; p < 0.01).Further, there were more publications with neutral conclusions in 2017 (18.4% compared to 7.2%; p < 0.01).There were no statistically significant differences between 2012 and 2017 in the number of intentions to treat analyses, in the type of outcomes (as most outcomes were objective), or specific types of primary outcomes.

Rısk of Bias Assessment
We provided a risk of bias assessment by each domain for trials published in 2012 and 2017 year (Table 3).Compared with 2012, more 2017 RCTs were rated low (70.4% compared to 38.8%) and fewer were rated unclear (20.4% compared to 50%; p < 0.001) risk for allocation concealment.Fewer 2017 RCTs were rated low (50.8% compared to 65.6%; p < 0.001) risk for blinding of participants and personnel, for blinding of outcome assessors (82.4% compared to 90.8%; p < 0.001), and selective outcome reporting (62.8% compared to 80.0%; p < 0.001).A similar proportion of 2017 RCTs were rated low risk for random sequence generation (59.6% compared to 56.0%), and for incomplete outcome data (74% compared to 73.6%;) compared to 2012.In 2017, more RCTs were rated low (42.8% compared to 33.6%) risk for other risk of bias (p < 0.01).More trials were rated low (29.2% compared to 21.2%) for overall risk of bias in 2017 compared to 2012 (p < 0.01).In 2017, multicenter trials (OR 0.39, 95% CI 0.18 to 0.80), drug trials (OR 0.53, 95% CI 0.29 to 0.97), and registered trials (OR 0.06, 95% CI 0.003 to 0.31) were also more likely to have a low overall RoB.In 2012, there was not yet a significant difference between multicenter or single-center trials (OR 0.52, 95% CI 0.24 to 1.22), drug trials, and non-drug trials (OR 0.82, 95% CI 0.44 to 1.56).Trial registration was not yet shown to have positive effects on RoB in 2012 either (OR 0.85, 95% CI 0.38 to 1.84).

Summary of Main Findings
We observedthat randomized cardiovascular clinical trials changed significantly from 2012 to 2017 years in several characteristics related to their study design and reporting.Respectively, RCTs in the 2017 sample were published mostly in specialty cardiovascular journalswith a higher number of authors from Asia and Europe compared to trials published in 2012.The 2017 sample includedmore parallel trialsevaluating drug interventions.The number of industry-funded clinical trials increased from 2012 to 2017, while the number of trials funded by the academy decreased.In the 2017 sample, more trials were from developing and transitional economy countries.Multinational trials had a significantly higher proportion in the 2017 sample compared with 2012.
As compared to 2012, in 2017 there were significant changes in important measures of methodological quality and reporting, including an improvement in the reporting of the presence of a data monitoring committee, and a positive tendency of registering trials in trial registries.Also, we observed that significantly more RCTs reported sample size calculations in 2017 as compared to 2012.
We also observed notable changes over five years in the ability of randomized cardiovascular research trials to properly estimate the true intervention effect.The 2017 trials were more likely to have a low RoB than 2012 for overallRoBs.However, the 5-year change was not clearly in the direction of improvement, as we observed a lower number of RCTs with alow RoB for blinding of participants and personnel and blinding of outcome assessors in 2017 as compared to 2012.In 2017, multicenter trials, drug trials, and registered trials were also more likely to have a low overall RoB than single-center, non-drug, non-registered trials.In 2012, these RoB differences were not yet present between RCTs with specific characteristics.

Strengths and Weaknesses of the Study
For both investigated publication years we selected our samplerandomlyfrom Cochrane CENTRAL as themost comprehensive resource of RCTs.The samples covered areas of the prevention, diagnosis, and treatment of cardiovascular diseases, including acute myocardial infarction, heart failure, arrhythmia, coronary revascularization, and chronic coronary artery disease.Most of our trials were registered in clinical trial registries, so essential trial details were double-checkedin both the full-text article andthe registry.We used the most accurate tool for risk of bias (methodologicalquality) assessment including RCTs.Two independent reviewers performed data extraction and RoB assessment, discrepancies were always resolved by discussion.
This study has also some limitations.Our sampleincludedabout 10% from all eligible cardiovascular disease trials published in the years 2012 and 2017 only in the English language.This study was not pre-registered witha detailed statisticalanalysis plan.We have chosen cardiovascular trials with participants aged 18 years or older, therefore our results are not applicable to pediatric trials in cardiovascular medicine.
As we started our research in the year 2018 and intended to evaluate changes over time, we decided to investigate publications from 2017, and from five years earlier, from 2012.However, since 2017, the publication characteristics may have changed further.It also has to be emphasized that the 2017 sample differed from the 2012 sample in many study design and reporting features, which may have impacted our RoB results.Although we have collected supplementary information from trial registries and published protocols, this was not detailed enough to compare information across these different information sources.

Discussion of Findings Considering Other Studies
The risk of bias in CVD RCTs generally decreased over the 5 years.This is consistent with the conclusions of Vinkers et al. [20], who reported significant improvement in the level of risk of bias of RCTs over the past years in connection with increased knowledge about mandatory trial registration and journal requirements.
Our study revealed that trial registration positively influenced RoB.This finding is in line with prior researchinvestigating clinical trial registration and the risk of bias.A study among clinical trials included in Cochrane systematic reviews of interventions published between 2014 and 2019 found that clinical trial registration was associated with a low risk of bias for all bias domains examined except for attrition bias, and for overall risk of bias [21].Registered trials were at lower risk of overall bias than non-registered trials in Latin America and the Caribbean [22].Prospectively registered trials had a significantly lower risk of bias compared to unregistered trials across all domains of health research [23].We found that multi-center trials were more likely at low risk of bias than single-center trials.This could be associated with that multi-center studies allow for better control of study quality than single-center studies [24].Our findings areconsistent with Tamborska et al., who found that RCTs at lower risk of bias were more likely to use multicenter recruitment in neurology trials [25].
Our investigation has shown that drug trials had a more favorableimpact on RoB than non-drug trials, which might be related tothe strict regulations these pharmaceuticaltrials must follow.Similarly, Cho Y et al. found that most drug trials were at low risk of bias for blinding participants and personnel, while almost two-thirds of non-drug trials were at high risk of bias for blinding participants and personnel in cardiopulmonary resuscitation and emergency cardiovascular care [26].
Existing differences and the positive beneficial impact of regulations can be observed in the planning phase of trials when regulated clinical trialprotocols were described to follow reporting guidelines to a greater extent than non-regulated trials [27].

Implication for Practice and Future Research
We observed some improvements with respect to some important study design features and some specific RoB domains over 5 years.However, there were also some methodological features and RoB domains thatchanged in an unfavorabledirection or remained unchanged.This points to the need to continue to pay close attention to the planning and conduct of RCTs in the field of cardiovascular clinical research.
This study identified several features of clinical trial planning and conducting that need further improvement in the field of cardiovascular research.Improvementsin study design, conduct, and reporting will decrease research waste and support the realization of evidence-based decisions in the field of cardiology.Journal adoption of existing reporting guidelines may lead to potential mechanisms to ensure improvements in overall clinical trialquality.We would emphasize that a paper that adheres to reporting guidelines better places a clinical decision-maker to assess the quality of the trial design and conduct and to interpret its findings accurately, improving the potential of the research to be impactful and meaningful to patients and clinical practice.

Table 1 .
Cont.Intervention categories were defined based onWood et al., 2008 [18]Multicenter trials were defined as trials with two or more administratively distinct study centers.Multinational applied to the countries from which patients were enrolled.The economic status of the country was defined based onPanagiotou et al., 2013 [19]. Footnote:

Table 2 .
Changes in important measures of methodological quality and reporting.