Adjusting for Publication Bias in Meta-Analysis with Continuous Outcomes: A Comparative Study

Abstract

Publication bias has been a problem facing meta-analysts. Methods adjusting for publication bias have been proposed in the literature. Comparative studies for methods adjusting for publication bias are found in the literature, but these studies are limited. We investigated and compared the performance of five methods adjusting for publication bias for the case of continuous outcomes. Three measures of continuous treatment effect, the mean difference, Cohen’s d and Hedges’ g, were considered. The methods studied were Copas, PET-PEESE, p-uniform, Trim and Fill and the limit meta-analysis. In addition, the performance of the random-effects meta-analysis using the DerSimonian estimator was also investigated. The analysis was conducted using a case study and an extensive simulation study including different scenarios. In general, the Copas and the PET-PEESE were found to be the least biased methods adjusting for publication bias. However, the Copas method, like other likelihood-based methods, can have convergence issues. In addition, the PET-PEESE method is robust in case of heteroscedasticity, making the PET-PEESE method a preferable technique to adjust for publication bias.

1. Introduction

Publication bias, also known as the file drawer problem or small-study problem, occurs when the published meta-analysis is not representative of the population of all studies, and can lead to incorrect conclusions. Publication bias is mainly attributed to the direction and the statistical significance of results, but other causes have been documented in the literature, e.g., language bias, availability bias and cost bias [1].

One popular method for adjustment for publication bias is the Trim and Fill method [2,3]. Assuming funnel plot symmetry, the method starts by estimating the number of unreported studies. Then, the treatment effect is re-estimated after removing the most extreme studies on the left side of the funnel plot. Using the new treatment effect estimate, a new number of unreported studies is estimated and removed. This process is repeated till the number of unreported studies is stable. Then, values are imputed for this number of studies on the right side of the funnel plot, and a treatment effect is estimated using the new set of data. The method has been evaluated in the literature for binary outcomes and found to be more biased in case of treatment effect heterogeneity [4,5].

Methods correcting for publication bias applying a selection model assume a mechanism for the publication process while modeling how the data are generated without publication bias. The first selection method developed by Hedges [1,6,7] assumed that only statistically significant results were published. Iyengar and Greenhouse [8] developed the Hedges method by using a weight function based on the relative likelihood that a non-significant-result study is published to that of a significant-result study. Later on the Hedges and Iyengar and Greenhouse methods have been extended to accommodate more complicated forms of publication bias [9]. Vevea and Hedges [9] proposed creating weights using a step function involving the one-tailed p-values. The Vevea and Hedges method is computationally complex since all weights need to be estimated along with the effect size and the between-study variance component. This can be especially problematic when the number of studies in a meta-analysis is small. For this reason, Vevea and Woods [10] proposed applying the Vevea and Hedges method with a priori specified weight functions. However, one disadvantage of the Vevea and Hedges method is assuming the study-specific covariates are available. Dear and Begg [11] introduced a similar method using a weighted likelihood function with the weights calculated using limits from the normal distribution based on each study’s p-value.

Another adjustment method based on a selection model which uses the effect size and its standard error instead of the p-values was developed as a sensitivity analysis by Copas and Shi [12,13]. The probability of selection was estimated using a linear function of a standard normally distributed variable, with the function parameters determined subjectively. Next, a conditional distribution density function was constructed based on the distribution of the treatment effects and of the selection probability, after which likelihood methods were applied to obtain estimates for the treatment effect. The sensitivity of the estimated treatment effect values was tested for different values of the parameters of the selection function. Reasonable estimates for the treatment effect were the ones associated with a p-value of a publication bias test of just above 0.1.

Another group of selection methods uses the distribution of the p-values [14,15,16]. The p-curve and p-uniform methods both estimate the effect size by minimizing the distance between the observed p-values distribution and the uniform distribution, each method using a different distance metrics. One disadvantage of the p-curve method is that it does not estimate a confidence interval for the bias-adjusted effect size [14]. More selection methods can be found in the literature; see [17,18,19,20] for an overview.

Regression methods have also been applied to adjust for publication bias. Moreno et al. [21] regresses the treatment effect on its variance, using the variance as a weight, and uses the predicted value for an infinitely large study as an adjusted estimate for the treatment effect. Moreno’s method is related to the PET-PEESE approach in which Stanley and Doucoullagos [22,23] suggest using the slope estimate from the Egger model as bias-adjusted overall treatment effect estimate, if the null hypothesis of zero treatment effect is not rejected. If this null hypothesis is rejected, the method uses the intercept estimate from Moreno’s regression equation as a bias-adjusted overall estimate. Although the PET-PEESE method has been shown to perform well with continuous outcomes, its estimate has been found to be less efficient when there is heterogeneity in treatment effects [24]. Rucker et al. [25] developed the limit meta-analysis method to adjust for publication bias. The authors adjusted the random-effects meta-analysis model by introducing a publication bias parameter and suggested the maximum likelihood method to estimate the model parameters. The authors noted that the estimates could be obtained using a linear regression model on the so-called generalized radial plot, with the standardized effect size dependent on the inverse of the standard error.

Comparison studies of methods adjusting for publication bias can be found in the literature. Terrin et al. [4] compared the performance of the Trim and Fill method with that of Hedges selection model in the case of odds ratio. The authors concluded that Hedges’ method performed better than the Trim and Fill method but noted the convergence problems of the iterative Hedges method. Schwarzer et al. [26] compared the performance of the Trim and Fill method with that of the Copas method in case of binary outcomes using the arcsine difference as an effect size. The authors concluded that the Trim and Fill method produced larger standard error, wider confidence intervals, i.e., it was more conservative and less efficient than the Copas method. Reed [27] compared six meta-analysis estimators in case of publication bias using a simulation study. Publication bias was created based on two scenarios: bias towards statistically significant estimates and bias against studies with wrong-signed estimated effects. The simulation set-up used was suitable for both binary and continuous outcomes. The six estimators considered were the fixed-effects estimate, the weighted least squares estimator, the random-effects estimator, the PET estimator [22,23], and the PEESE estimator [22,23], while using the arithmetic mean of estimated effects as the benchmark estimator. Reed concluded that the fixed-effects and the weighted least squares estimators were as efficient and sometimes more efficient than the PET and the PEESE estimators, while the random-effects estimator was often more biased than the other estimators [27].

Mcshane et al. [19] compared the performance of a number of publication bias adjustment methods using the mean difference as an effect size and applying one-sided selection. The authors compared the p-curve and the p-uniform approaches to the maximum likelihood estimation approaches of Hedges and Iyengar and Greenhouse [8]. The p-curve and p-uniform approaches were found to perform well under the setting they were designed for: namely, when only studies with results that were statistically significant and directionally consistent were published and effect sizes were homogeneous across studies. However, when one or both of these assumptions was violated, the p-curve and the p-uniform methods became less accurate than the Iyengar and Greenhouse method. Although McShane et al. [19] compared the performance of a few methods in case of continuous outcomes, the authors did not consider other methods not based on selection models and studied only one type of effect size.

Carter et al. [28] investigated the performance of eight methods correcting for publication bias. The authors carried out an extensive simulation study but only investigated the performance of Cohen’s d. Carter et al. concluded that no one single method clearly outperformed the other methods. Hong and Reed [29] compared 10 methods under different simulation setting, including publication selection mechanism. The authors concluded that the size of the meta-analysis sample and effect heterogeneity influenced the estimator performance and showed how these characteristics could help meta-analysts choose the most appropriate estimator for their research circumstances. However, the authors did not concentrate explicitly on continuous outcomes with its different effect size measures. Neither Carter et. al nor Hong and Reed included the popular Copas method.

This brief review shows the lack of a comprehensive comparative study for methods adjusting for publication bias in case of continuous outcomes. Therefore, our research question was as follows: Which publication bias adjusting method works best in the case of continuous outcomes? The objective was to investigate the performance of methods adjusting for publication bias in case of continuous outcomes, for all corresponding measures of the effect size. Investigating all adjustment methods in the literature was obviously beyond the scope of this article. We selected five publication bias adjustment methods to try to represent different techniques and applied these methods in the case of continuous outcomes using the mean difference, Cohen’s d and Hedges’ g as treatment effect estimates. One popular method based on selection models, the Copas method, was considered. Another method based on the p-values, the p-uniform method, was considered. The performance of the PET-PEESE method, a regression analysis-based method, was investigated. Finally, two likelihood-based methods, the Trim and Fill method and the limit meta-analysis method, were considered. In addition, the traditional random effects meta-analysis method based on the DerSimonian–Laird estimate was also included. The layout of this article is as follows: In Section 2.2, we describe the selected methods used to adjust for publication bias. Section 3 describes the case study, the simulation model and the selection model used to simulate publication bias. The results and the discussion are found in Section 4 and Section 5, respectively.

2. Adjustment Methods

A meta-analysis of continuous outcomes usually consists of m studies, i ($i=1,2,\dots ,m$). Each study typically has $n_i$ participants, with $n_{ij}$ subjects assigned to treatment arm $j,$ $jϵ\left\{0,1\right\}$. Here $j=1$ and $j=0$ indicate the treatment and the control arms, respectively. Define ${\widehat{y}}_{i1}$ and $S_{i1}$ as the average and the standard deviation of the treatment arm, respectively, and ${\widehat{y}}_{i0}$ and $S_{i0}$ the average and the standard deviation of the control arm, respectively. Traditionally, the mean difference ${\widehat{y}}_{i1}-{\widehat{y}}_{i0}$ is used as a treatment effect for study i. Another treatment effect measure is Cohen’s d [30], which can be given by $\left({\widehat{y}}_{i1}-{\widehat{y}}_{i0}\right)/s{e}_i$ where $s{e}_i=\sqrt{\left(\left(n_{i1}-1\right)\ast S_{i1}^2+\left(n_{i0}-1\right)\ast S_{i0}^2\right)/\left(n_{i1}+n_{i0}-2\right)}$. However, Cohen’s d has been found to be biased [6], and a corrected unbiased estimate is referred to as Hedges’ g. It multiplies Cohen’s d with the correction factor $J=1-3/\left(4\left(n_1+n_0-2\right)-1\right)$. From here on, we refer to the treatment effect for study i by ${\widehat{y}}_i$, which can refer to the mean difference, Cohen’s d, and Hedges’ g.

2.1. Copas Selection Model

Copas and Shi [12,13] assume that all studies follow the traditional random-effects meta-analysis model given by

$$y_i=y+{\theta }_i+{ϵ}_i$$

(1)

with y general treatment effect, ${\theta }_i\sim N\left(0,{\tau }^2\right)$ and ${ϵ}_i\sim N\left(0,V_i^2\right)$. However, the authors assume that only a subset of the studies has been published. The authors then introduce $z_i=A_1+A_2/\widehat{s{e}_i}+{\delta }_i$, with ${\delta }_i\sim N(0,1)$ correlated with ${ϵ}_i$ through $\rho =corr({ϵ}_i,{\delta }_i)$, and ${\widehat{y}}_i$ only reported if $z_i>0$. The conditional probability density function is given by $P\left({\widehat{y}}_i|z_i>0,\widehat{s{e}_i}\right)$ and the log likelihood function is constructed as [13]

$$L\left(y,{\tau }^2\right)=\sum _{i=1}^m\left[\log P\left({\widehat{y}}_i|z_i>0,\widehat{s{e}_i}\right)\right].$$

(2)

For fixed values of $A_1$ and $A_2$, $L\left(y,{\tau }^2\right)$ is maximized, and y and ${\tau }^2$ along with their confidence interval are estimated. We used the R-package “copas”, which is part of the R-package meta (R version 4.1.2) to carry out the Copas method [12,13].

2.2. The p-Uniform Method

The p-uniform is a selection model approach, with the selection model assuming that the probability of publishing a statistically significant treatment effect as well as a non-significant treatment effect are constant but may differ from each other [16]. Defining the $F_X\left(x_i\right)$, the distribution functions of the normal distributions $N\left(y^f,\widehat{s{e}_i}\right)$, with $y^f$ the fixed-effects meta-analysis effect size estimate and $\dot{y_i}$ the critical value for study i, the authors define the conditional p-value distribution [16]

$$p_i^{\dot{y}}={\displaystyle \frac{{\int }_{{\widehat{y}}_i}^{\infty }{F}_X\left(x_i\right)}{{\int }_{\dot{y_i}}^{\infty }{F}_X\left(x_i\right)}}$$

(3)

where $\dot{y}$ is the critical value for rejecting the null hypothesis of no treatment effect at significance level $\alpha$. Using the gamma-distributed test statistic $L^y=-{\sum }_{i=1}^m\log \left(p_i^y\right)$, the authors introduce the bias-adjusted estimate $\widehat{y}$ for y, with $\widehat{y}$ the solution to $L^{\widehat{y}}=m$. The p-uniform lower and upper limits of the point estimate ${\widehat{y}}_L$ and ${\widehat{y}}_U$ are then given by the solutions of $L^{{\widehat{y}}_L}={\Gamma }_{1-\alpha /2}\left(m,1\right)$ and $L^{{\widehat{y}}_U}={\Gamma }_{\alpha /2}\left(m,1\right)$, respectively, with $\Gamma$ the gamma function [16]. We used the package ‘puniform’ (R version 4.1.2) to perform the p-uniform method [31].

2.3. The PET-PEESE Method

Stanley and Doucouliagos [22,23] start with the linear regression model used in the Egger test given by

$${\displaystyle \frac{{\widehat{y}}_i}{\widehat{s{e}_i}}}={\alpha }_0+{\displaystyle \frac{{\alpha }_1}{\widehat{s{e}_i}}}+{\epsilon }_i,$$

(4)

with ${\alpha }_0$ and ${\alpha }_1$ an intercept and a regression coefficient, respectively, and ${\epsilon }_i\sim N\left(0,{\sigma }^2\right)$. The authors suggest using ${\widehat{\alpha }}_1$ in model (4) as a bias-adjusted estimate for the overall treatment effect, if the null hypothesis of ${\alpha }_1=0$ is not rejected by the Egger equation using a t-test (PET). If this null hypothesis is rejected, the model $y_i={\gamma }_0+{\gamma }_{1}s{e}_i^2+{ϵ}_i$, ${ϵ}_i\sim N\left(0,{\sigma }^2\right)$ is fitted using $1/s{e}_i^2$ as weight, and ${\widehat{\gamma }}_0$ is used as a bias-adjusted estimate of the overall treatment effect (PEESE). Note that the last regression model was the one suggested by Moreno et al. (2011) [21]. We used the GLM procedure in SAS version 9.4 to carry out the PET-PEESE method.

2.4. Trim and Fill Method

The Trim and Fill method can be briefly described as follows [2,3]. The studies are first ranked based on their distance from the pooled treatment effect estimated by the random-effects model ${\widehat{y}}_{RM}$ (1), i.e., ranking distances $|{\widehat{y}}_i-{\widehat{y}}_{RM}|$. Next, the number of unobserved studies is estimated using the estimator $L_0=[4T_m-m(m+1)]/[2m-1]$, where $T_m$ is the Wilcoxon rank-sum test statistic estimated from the ranks of studies with ${\widehat{y}}_i>{\widehat{y}}_{RM}$ (here, it is assumed that ${\widehat{y}}_{RM}$ is positive, and it is more likely that studies with effect sizes below ${\widehat{y}}_{RM}$ are potentially missing). Then, the $L_0$ most extreme studies (i.e., studies with positive effect sizes furthest away from zero) are trimmed off, and the pooled treatment effect ${\widehat{y}}_{RM}$ without these studies is re-estimated. Then, all studies are ranked again, based on their distance to the new pooled estimate, and $L_0$ is re-computed. This procedure is repeated until it stabilizes ($L_0$ does not change anymore), and we obtain a final estimate ${\widehat{y}}_{RM}$ and a final estimate $L_0$ of the number of studies missing. Then, $L_0$ studies are imputed by mirroring the $L_0$ studies with the highest effect sizes around the final estimate for ${\widehat{y}}_{RM}$, and the standard error $s{e}_i$ from the mirrored study is provided. After imputation, a final pooled estimate ${\widehat{y}}_{TF}$ with standard error is provided using the random-effects model on all $m+L_0$ studies. We used the function “trimfill” in the R package “metafor” (R version 4.1.2), with the average treatment effect estimated using the random-effects model [32].

2.5. The Limit Meta-Analysis Method

The limit meta-analysis can be described as follows [25]. The authors apply a publication-bias-adjusted version of the random-effects meta-analysis model as follows

$${\widehat{y}}_i=y+\left(s{e}_i+\tau \right)\left({ϵ}_i+\alpha \right),$$

(5)

with $\alpha$ a parameter representing publication bias. The authors propose using $\widehat{y}+\widehat{\tau }\widehat{\alpha }$ as a publication-bias-adjusted estimator for the effect size y and the maximum likelihood method to estimate the parameters y and $\tau$. However, the authors note that maximum likelihood estimates for y and $\tau$ correspond with the intercept and slope of the linear regression on a generalized radial plot. In such model, the response variable and the independent variable are given by ${\widehat{y}}_i/\left(\widehat{s{e}_i}+\widehat{\tau }\right)$ and $1/\left(\widehat{s{e}_i}+\widehat{\tau }\right)$, respectively [25]. We used the R package “metasens” (R version 4.1.2) to carry out the limit meta-analysis method [33].

3. Case Study, Simulation, and Selection Models

3.1. Case Study

As a case study, we used the results of 25 studies investigating the effect of the experience of weight on the concept of importance. Experience of weight, exemplified by heaviness and lightness, is metaphorically associated with concepts of seriousness and importance [34]. Robelo et al. [35] reported 25 studies that investigated the effect of weight on the concept of importance. This case study was selected for the following reasons. All published studies had continuous outcomes for the expression of importance and all applied Cohen’s d as an effect size. Moreover, there was evidence that these published studies were too good to be true, indicating the presence of publication bias [36]. In addition, the authors conducted their own research on this topic after applying the p-uniform method’s test for publication bias on these published studies [16] and finding evidence of significant publication bias [35]. The dataset containing the results of the 25 studies can found here [35].

3.2. Simulation Model

The simulation model used is described in Section 3.2.1. The simulation model explores different scenarios including different study sizes, number of units per treatment arm per study, treatment effect homogeneity, and different levels of treatment effect heterogeneity. In addition, two scenarios of equal and unequal study variance are presented for the treatment and control groups, respectively. A separate model for selecting a number of studies based on specific criteria to simulate publication bias is described in Section 3.2.2.

3.2.1. Aggregate Simulation Model

The simulation model applied can be described as follows [37,38]. In total, m studies were generated, and for the ith study, $n_{ij}$, $j=0,1$, were independently drawn from a Poisson distribution with parameter $\delta$. The estimated treatment effect for the jth treatment group in study i was given by ${\overline{y}}_{ij}=\left(\eta +{\theta }_i\right)·t_{ij}+{\epsilon }_{ij}$, with $\eta$ a fixed treatment effect, ${\theta }_i$ a random effect for study i, ${\theta }_i\sim N\left(0,{\tau }^2\right)$, $t_{ij}$ a treatment indicator variable for study i with a value of 1 when $j=1$ and 0 otherwise, and ${\epsilon }_{ij}$ an error term with ${\epsilon }_{ij}\sim N\left(0,S_{ij}^2/n_{ij}\right)$. For the equal study variance scenario, $S_{i0}^2=S_{i1}^2=S_i^2$, while for the unequal study variance scenario, the variance for the control and the treatment groups were created by $S_{i0}^2=0.8S_i^2$ and $S_{i1}^2=S_i^2$, respectively. The parameter values used to generate the data were $m\in \{10,30\}$, $\delta =\left\{15,30\right\}$, $\eta =5$, ${\tau }^2=\left\{0,1,5\right\}$, and $S_i^2$ was generated using $S_i^2\sim N\left(100,100\right)$. The mean difference, Cohen’s d, and Hedges’ g were then calculated as described in Section 2 earlier.

Note that some choices for ${\tau }^2$ made here are unrealistic for the corresponding treatment effect. Examples are the choice of ${\tau }^2=1$ for the mean difference and the choice of ${\tau }^2=5$ for Cohen’s d and Hedges’ g. However, as Morris et al. [39] point out, including unrealistically extreme data generation mechanisms can help show under which circumstances the considered methods fail.

3.2.2. Selection Model Based on Significant Effect Size

Selection models based on the p-value of the study effect have been proposed in the literature [19,22,23,31]. Let $\alpha$ be the significance level and $t_{q,{\nu }_i}$ be the qth quantile of a student t distribution with ${\nu }_i$ degrees of freedom for study i. When the effect size is significant (assuming more positive effect sizes), i.e., $d_i>t_{\left(1-\alpha /2\right),{\nu }_i}$, the study is included. To add randomness to the selected studies, a uniform distributed random variable $U(0,1)$ and a parameter ${\pi }_{\mathrm{pub}}$ are used. The drawings from the uniform distribution represent probabilities. The value of ${\pi }_{\mathrm{pub}}$ is chosen based on the desired percentage of published studies, see below. The drawn values of the uniform distribution and the ${\pi }_{\mathrm{pub}}$ determine which study will be selected as follows. If the uniform random variable is smaller than or equal to $1-{\pi }_{\mathrm{pub}}$, the non-significant study is included too; otherwise, it is excluded. Here, $\alpha =0.05$ and ${\pi }_{\mathrm{pub}}$ were chosen to obtain a desired publishing rate of around 80%. For the equal study variance scenario, we applied $\nu =n_{i0}+n_{i1}-2$. For the unequal variance scenario, we applied the Satterthwaite approximation for the degrees of freedom $\nu ={\left(S_{ic}^2/n_{ic}+S_{iT}^2/n_{iT}\right)}^2/\left({\left(S_{ic}^2/n_{ic}\right)}^2/\left(n_{ic}-1\right)+{\left(S_{iT}^2/n_{iT}\right)}^2/\left(n_{iT}-1\right)\right)$ [40].

4. Results

4.1. Results for the Case Study

Table 1. Results of treatment effect estimates for the case study.

Method	Mean Difference	Cohen’s d	Hedges’ g
Copas	0.05875 (−0.06967; 0.18716)	0.57825 (0.47577; 0.68074)	0.57052 (0.46803; 0.67301)
p-Uniform	0.33117 (0.15906; 0.47931)	0.4786477 (−2.560318; 0.7836985)	0.4597283 (−3.244159; 0.7352272 )
PET-PEESE	−0.64935 (−1.7651; 0.46637)	−0.001796206 (−0.034486; 0.030893)	−0.001712641 (−0.033384; 0.029959)
Trim and Fill	0.13392 (−0.05061; 0.31846)	0.54305 (0.36237; 0.72373)	0.54064 (0.35672; 0.72455)
Limit meta-analysis	0.03091 (−0.06347; 0.05768)	0.31356 (−0.56417; 0.66497)	0.31356 (−0.54872; 0.68042)
DL	0.53589 (0.40633; 0.66545)	0.57825 (0.47577; 0.68074)	0.57052 (0.46803; 0.67301)

shows the treatment effect estimates for the case study described in Section 3.1 using the five adjustment methods explained in Section 2. Also presented is the treatment effect estimate using the traditional random-effects meta-analysis method based on the DerSimonian–Laird estimator (DL).

For the mean difference, the Copas and limit meta-analysis methods have the lowest point estimates and their mean difference estimates do not significantly deviate from zero. The PET-PEESE method is the only negatively biased method, while the Trim and Fill method is the highest positively biased method. Both the PET-PEESE and the Trim and Fill methods have mean difference estimates that do not significantly deviate from zero. The p-uniform and the DL methods are both positively biased and have mean difference estimates that significantly deviate from zero.

In the case of Cohen’s d and Hedges’ g, all methods produce a treatment effect estimate of between 0.5 and 0.6, with the exception of the PET-PEESE and the limit meta-analysis methods,. The limit meta-analysis method produces a lower treatment effect estimate. The p-uniform, PET-PEESE, and the limit meta-analysis are the only methods indicating that there is no treatment effect, since their estimates do not significantly deviate from zero.

4.2. Results for the Simulation Study

4.2.1. Average Number of Studies After Selection

Table 2. Average number of remaining studies after applying the selection model for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Cohen’s d		Hedges’ g
			Equal Variance	Unequal Variance	Equal Variance	Unequal Variance
10	15	0	8.23	8.09	8.04	8.08
		1	8.22	8.08	8.02	8.07
		5	8.21	8.01	8.01	7.98
	30	0	8.11	8.09	8.20	8.00
		1	8.13	8.10	7.99	7.98
		5	7.87	8.14	8.12	8.12
30	15	0	23.0	24.1	24.1	24.0
		1	23.0	24.1	24.1	24.0
		5	24.5	24.0	24.1	23.9
	30	0	24.2	23.9	24.2	23.9
		1	24.2	24.1	23.7	23.8
		5	23.4	24.1	23.9	24.1

shows the average number of remaining studies per simulated meta-analysis after applying the selection model explained in Section 3.2.2 for all simulation settings. We tried to keep the average number of remaining studies around 80% of original studies for comparison purposes. This percentage is based on earlier research concluding that the reported proportion of published studies with positive outcomes was 82% in emergency or general medicine [41].

Table 3. An overview of the AMSE, bias, and coverage probabilities for the Copas method under equal variance assumption for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Mean Difference			Cohen’s d			Hedges’ g
			AMSE	Bias	COV	AMSE	Bias	COV	AMSE	Bias	COV
10	15	0	64.8	6.09	0.423	1.54	0.344	0.767	1.55	0.376	0.755
		1	63.4	6.07	0.425	1.60	0.351	0.784	1.65	0.371	0.779
		5	73.3	6.29	0.457	1.13	0.424	0.754	1.66	0.346	0.764
	30	0	35.9	5.37	0.065	0.927	0.232	0.587	0.944	0.227	0.597
		1	36.3	5.37	0.068	0.928	0.226	0.586	0.974	0.285	0.584
		5	52.0	6.09	0.180	1.37	0.390	0.666	1.37	0.325	0.703
30	15	0	51.0	6.39	0.101	1.11	0.420	0.763	0.895	0.321	0.801
		1	50.7	6.36	0.103	1.13	0.424	0.754	0.906	0.324	0.797
		5	47.5	6.10	0.188	1.04	0.372	0.842	1.021	0.384	0.833
	30	0	30.0	5.30	0.011	0.520	−0.146	0.651	0.503	−0.138	0.657
		1	30.5	5.35	0.008	0.540	−0.137	0.675	0.505	−0.076	0.677
		5	40.5	6.09	0.013	0.877	0.185	0.761	0.807	0.157	0.766

,

Table 4. An overview of the AMSE, bias, and coverage probabilities for the p-uniform method under equal variance assumption for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Mean Difference			Cohen’s d			Hedges’ g
			AMSE	Bias	COV	AMSE	Bias	COV	AMSE	Bias	COV
10	15	0	133	11.1	0.064	14.2	3.58	<0.05	13.6	3.51	<0.05
		1	134	11.1	0.067	14.3	3.60	<0.05	13.8	3.53	<0.05
		5	156	11.9	0.064	13.2	3.57	<0.05	16.2	3.84	<0.05
	30	0	48.0	6.72	<0.05	11.2	3.24	<0.05	10.8	3.18	<0.05
		1	48.1	6.74	<0.05	11.2	3.24	<0.05	11.0	3.22	<0.05
		5	71.8	8.18	<0.05	16.9	3.93	<0.05	16.0	3.78	<0.05
30	15	0	122	10.9	<0.05	13.1	3.56	<0.05	12.2	3.42	<0.05
		1	123	11.0	<0.05	13.2	3.57	<0.05	12.2	3.43	<0.05
		5	143	11.8	<0.05	15.3	3.84	<0.05	14.7	3.76	<0.05
	30	0	45.2	6.67	<0.05	10.4	3.19	<0.05	10.2	3.15	<0.05
		1	46.0	6.72	<0.05	10.6	3.21	<0.05	10.5	3.19	<0.05
		5	68.3	8.13	<0.05	15.9	3.92	<0.05	15.3	3.85	<0.05

,

Table 5. An overview of the AMSE, bias, and coverage probabilities for the PET-PEESE method under equal variance assumption for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Mean Difference			Cohen’s d			Hedges’ g
			AMSE	Bias	COV	AMSE	Bias	COV	AMSE	Bias	COV
10	15	0	9703	−5.78	0.965	6.51	−1.67	0.895	6.79	−1.68	0.892
		1	9522	−5.71	0.966	6.96	−1.67	0.879	7.12	−1.68	0.879
		5	10,793	−4.87	0.960	2.13	−1.08	0.829	7.57	−1.66	0.896
	30	0	1456	−3.48	0.966	2.33	−0.851	0.825	2.41	−0.926	0.819
		1	1515	−4.03	0.967	2.47	−0.877	0.820	2.45	− 0.832	0.828
		5	3072	−2.29	0.967	4.61	−1.16	0.850	4.43	−1.26	0.849
30	15	0	2399	−9.76	0.975	2.01	−1.04	0.827	2.02	−1.13	0.790
		1	2469	−9.78	0.976	2.13	−1.08	0.829	2.19	−1.17	0.782
		5	2834	−6.31	0.972	2.45	−1.24	0.804	2.39	−1.24	0.806
	30	0	360	−4.40	0.981	0.702	−0.617	0.718	0.728	−0.639	0.700
		1	360	−4.25	0.978	0.715	−0.624	0.725	0.662	−0.567	0.744
		5	740	−2.45	0.975	0.982	−0.700	0.812	1.045	−0.775	0.786

,

Table 6. Anoverview of the AMSE, bias, and coverage probabilities for the PET-PEESE method under unequal variance assumption for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Mean Difference			Cohen’s d			Hedges’ g
			AMSE	Bias	COV	AMSE	Bias	COV	AMSE	Bias	COV
10	15	0	8120	−1.67	0.960	5.59	−1.17	0.929	5.42	−1.18	0.929
		1	8501	−1.89	0.966	5.44	−1.14	0.938	5.33	−1.15	0.930
		5	9766	−1.78	0.956	6.89	−1.26	0.923	6.73	−1.28	0.925
	30	0	1239	−2.35	0.970	1.49	−0.121	0.785	1.52	−0.086	0.781
		1	1284	−2.47	0.970	1.50	−0.110	0.779	1.53	−0.080	0.775
		5	2911	0.026	0.958	2.60	−0.560	0.837	2.58	−0.573	0.837
30	15	0	1800	−5.36	0.966	1.09	−0.564	0.973	1.04	−0.577	0.969
		1	1829	−5.57	0.966	1.12	−0.568	0.967	1.10	−0.589	0.965
		5	2268	−4.96	0.966	1.41	−0.676	0.967	1.38	−0.691	0.964
	30	0	276	−3.80	0.978	0.346	0.091	0.809	0.340	0.069	0.816
		1	307	−3.34	0.979	0.344	0.081	0.809	0.339	0.059	0.813
		5	728	−1.55	0.977	0.436	−0.183	0.929	0.437	−0.200	0.929

,

Table 7. An overview of the AMSE, bias, and coverage probabilities for the Trim and Fill method under equal variance assumption for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Mean Difference			Cohen’s d			Hedges’ g
			AMSE	Bias	COV	AMSE	Bias	COV	AMSE	Bias	COV
10	15	0	50.8	6.19	0.492	5.00	1.85	0.530	4.98	1.86	0.521
		1	51.0	6.17	0.485	4.98	1.85	0.531	4.97	1.86	0.521
		5	55.8	6.33	0.533	3.34	1.65	0.271	5.61	1.93	0.545
	30	0	32.7	5.50	0.068	8.14	2.71	0.116	7.64	2.62	0.129
		1	32.9	5.51	0.081	8.08	2.69	0.125	8.14	2.71	0.123
		5	42.7	6.11	0.224	10.5	2.97	0.230	9.42	2.79	0.266
30	15	0	48.7	6.53	0.141	3.36	1.66	0.264	2.67	1.46	0.322
		1	48.4	6.52	0.147	3.34	1.65	0.271	2.64	1.45	0.326
		5	45.2	6.15	0.228	2.98	1.52	0.369	2.97	1.52	0.359
	30	0	30.9	5.46	<0.05	6.67	2.50	<0.05	6.47	2.46	<0.05
		1	31.3	5.49	<0.05	6.72	2.51	<0.05	6.98	2.56	<0.05
		5	39.3	6.05	<0.05	8.11	2.69	<0.05	7.42	2.57	0.057

,

Table 8. An overview of the AMSE, bias, and coverage probabilities for the limit meta-analysis method under equal variance assumption for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Mean Difference			Cohen’s d			Hedges’ g
			AMSE	Bias	COV	AMSE	Bias	COV	AMSE	Bias	COV
10	15	0	99.1	9.00	0.883	3.32	1.67	0.992	3.25	1.64	0.992
		1	99.8	9.05	0.882	3.37	1.68	0.995	3.28	1.65	0.994
		5	125	10.1	0.879	3.95	1.84	0.914	3.89	1.81	0.912
	30	0	37.4	5.51	0.803	1.70	1.18	0.503	1.62	1.16	0.503
		1	39.2	5.65	0.806	1.67	1.17	1.000	1.64	1.15	1.000
		5	86.9	8.31	0.845	3.62	1.66	0.999	3.40	1.61	0.999
30	15	0	20.7	4.35	0.735	0.464	0.646	0.965	0.392	0.593	0.972
		1	20.8	4.37	0.725	0.470	0.652	0.965	0.396	0.598	0.971
		5	25.7	4.87	0.774	0.575	0.729	0.977	0.539	0.703	0.973
	30	0	7.50	2.64	0.643	0.157	0.368	0.995	0.150	0.359	0.995
		1	7.96	2.71	0.646	0.165	0.379	0.992	0.160	0.371	0.991
		5	18.3	4.12	0.717	0.445	0.637	0.990	0.417	0.617	0.991

and

Table 9. An overview of the AMSE, bias, and coverage probabilities for the random-effects model under equal variance assumption for the different simulation settings.

m	${\mathit{n}}_{\mathit{i}}$	${\mathit{\tau }}^2$	Mean Difference			Cohen’s d			Hedges’ g
			AMSE	Bias	COV	AMSE	Bias	COV	AMSE	Bias	COV
10	15	0	50.9	6.27	0.499	6.10	2.11	0.574	6.00	2.11	0.559
		1	51.0	6.27	0.503	6.14	2.11	0.579	6.05	2.12	0.561
		5	55.7	6.43	0.540	6.81	2.18	0.617	6.74	2.18	0.598
	30	0	32.6	5.51	0.060	8.98	2.88	0.096	8.48	2.80	0.109
		1	33.0	5.53	0.065	9.02	2.88	0.110	9.08	2.90	0.099
		5	42.2	6.13	0.209	11.8	3.22	0.268	10.6	3.02	0.305
30	15	0	50.1	6.78	0.092	5.93	2.31	0.154	4.69	2.03	0.221
		1	50.1	6.78	0.094	5.94	2.32	0.161	4.70	2.04	0.225
		5	45.5	6.36	0.199	5.24	2.13	0.308	5.14	2.11	0.281
	30	0	30.8	5.48	<0.05	8.37	2.85	<0.05	8.14	2.81	<0.05
		1	31.2	5.51	<0.05	8.44	2.86	<0.05	8.62	2.89	<0.05
		5	39.3	6.13	<0.05	10.8	3.20	<0.05	9.85	3.05	<0.05

show the average mean squared error (AMSE), the bias and the coverage probabilities (COV) of all the methods considered. Note that the results of equal and unequal variance scenarios were comparable for all methods except the PET-PEESE method. For this reason, the results of unequal variance scenario are only shown for the PET-PEESE method. The results of this scenario for all other methods can be found in the Supplementary Materials.

4.2.2. Copas Selection Model

For the mean difference, the AMSE and bias increased as the treatment effect heterogeneity increased when $m=10$ for the equal study variance scenario. For the unequal study variance scenario, the mean difference, the AMSE, and bias increased as the treatment effect heterogeneity increased for both $m=10$ and $m=30$. Both the AMSE and bias were slightly higher for the unequal study variance scenario. Coverage probabilities were much lower than those of Cohen’s d and Hedges’ g. For Cohen’s d, when $m=10$ and $m=30$, the AMSE and bias of the Copas method were higher for the unequal study variance scenario when $n_i=15$ but not always when $n_i=30$. The coverage probability for Cohen’s d was comparable for the equal and unequal study variance scenarios. However, the coverage probability decreased when $m=30$ for both homogeneity and heterogeneity of the treatment effect. Hedges’ g showed comparable values of AMSE, bias, and coverage probability when $m=1$0 and $m=30$ and when $n_i=10$ for different values of ${\tau }^2$ and for both the equal and unequal study variance scenarios. However, the AMSE, bias, and coverage probability clearly decreased when $n_i=30$. Overall, Hedges’ g had a smaller AMSE and bias and higher coverage probabilities than Cohen’s d. The AMSE and bias of the method under the equal variance scenario were lower than under the unequal variance scenario. The statistical power of the Copas method was lower under the equal variance scenario.

4.2.3. The p-Uniform Method

For the mean difference, the AMSE and bias increased as the treatment effect heterogeneity increased for both the equal and unequal study variance scenarios. Both the AMSE and bias were slightly higher for the unequal study variance scenario. For Cohen’s d, the AMSE and bias of the p-uniform method were comparable for the unequal study variance scenario for all simulation settings. The coverage probability mostly increased for Cohen’s d as the treatment effect heterogeneity increased. The AMSE and bias of Cohen’s d decreased while the coverage probability increased as the treatment effect heterogeneity increased. Hedges’ g showed comparable values of AMSE, bias, and coverage probability when $n_i=15$ for different values of ${\tau }^2$ and for both the equal and unequal study variance scenarios. However, the AMSE, bias, and coverage probability mostly increased when $n_i=30$, except when ${\tau }^2$ = 5. The coverage probability mostly increased for Hedges’ g as the treatment effect heterogeneity increased. Cohen’s d and Hedges’ g had comparable bias, AMSE, and coverage probabilities. The AMSE and bias of the p-uniform method under the equal variance scenario were lower than under the unequal variance scenario. The statistical power of the method under the equal and the unequal variance scenario were comparable.

4.2.4. PET-PEESE

For the mean difference, the AMSE increased as the treatment effect heterogeneity increased for the equal and unequal study variance scenarios. The bias decreased as the treatment effect heterogeneity increased. The AMSE and bias were clearly higher for the equal study variance scenario than under the unequal study variance scenario. Coverage probabilities were conservative compared to those of Cohen’s d and Hedges’ g.

For Cohen’s d, the AMSE of the PET-PEESE method was higher for the equal study variance scenario than the unequal variance scenario for all simulation combinations. The bias was higher for the equal study variance scenario when $m=10$ and $n_i=15$. The AMSE increased and the bias decreased when the between-study variance increased. The coverage probability was, in general, nominal or almost nominal for all simulation settings. These results held for both homogeneity and heterogeneity of the treatment effect. The coverage probability, however, was comparable for different values of ${\tau }^2$.

Hedges’ g showed higher values for the AMSE for the equal study variance scenario than the unequal study variance scenario. The bias of Hedges’ g decreased in general as the treatment effect heterogeneity increased. The statistical power was nominal or almost nominal for all simulation settings.

The AMSE for Cohen’s d and Hedges’ g were comparable. When $n_i=15$, Hedges’ g showed a lower bias than Cohen’s d in the equal study variance scenario, but this difference reduced or disappeared when $n_i=30$. For the unequal study variance scenario, no clear pattern could be observed. Coverage probabilities were comparable for Cohen’s d and the Hedges g for all simulation scenarios.

The AMSE and bias of the PET-PEESE method under the equal variance scenario were higher than under the unequal variance scenario. The statistical power of the method was higher under the equal variance scenario.

4.2.5. Trim and Fill Method

For the mean difference, the AMSE and bias increased as the treatment effect heterogeneity increased when $m=10$ for the equal study variance scenario. For the unequal study variance scenario, the mean difference the AMSE, and bias increased as the treatment effect heterogeneity increased for both $m=10$ and $m=30$. Both the AMSE and bias were slightly higher for the unequal study variance scenario. Coverage probabilities were lower than those of Cohen’s d and Hedges’ g.

For Cohen’s d, the AMSE and bias of the Trim and Fill method were higher for the equal study variance scenario when $n_i=15$ but not when $n_i=30$. The AMSE and bias increased when ${\tau }^2=5$ for the equal and unequal study variance scenarios. The coverage probability for Cohen’s d was comparable for the equal and unequal study variance scenarios. Hedges’ g showed higher values of AMSE and bias and when $m=10$ and $n_i=15$ for the equal study variance scenario for different values of ${\tau }^2$ than when $m=30$. The AMSE and bias increased when ${\tau }^2=5$ for the equal and unequal study variance scenarios. The AMSE and bias were generally higher for the equal study variance scenario than the unequal study variance scenario. The coverage probabilities for Hedges’ g were comparable for the equal and unequal study variance scenarios. However, the coverage probability decreased when $m=30$ and $n_i=30$. The AMSE and bias of Cohen’s d were higher than those of Hedges’ g for the equal study variance scenario except when $m=30$ and $n_i=30$ and also when ${\tau }^2=5$. The coverage probabilities of Cohen’s d and Hedges’ g had comparable values except when ${\tau }^2=5$ for the equal study variance scenario, where Hedges’ g had a higher coverage probability.

The AMSE and bias of the Trim and Fill method under the equal study variance scenario were lower than under the unequal variance scenario. The coverage probability of the method was lower under the equal variance scenario.

4.2.6. The Limit Meta-Analysis Method

For the mean difference, the AMSE and bias increased as the treatment effect heterogeneity increased for the equal and unequal study variance scenarios. Coverage probabilities were much lower than those of Cohen’s d and Hedges’ g.

For Cohen’s d, the AMSE and bias of the limit meta-analysis method were higher for the equal study variance scenario except when $m=30$ and $n_i=30$. The bias increased as the treatment effect heterogeneity increased, especially when ${\tau }^2=5$. The coverage probability for Cohen’s d was comparable for the equal and unequal study variance scenarios. For the equal variance scenario, the coverage probability in general was above the nominal level, with the exception of the case $m=10$ and $n_i=30$ where it became very low. In the unequal variance scenario, when $n=10$, the coverage probability was above the nominal level except for the cases $n_i=15$ and ${\tau }^2=5$, and $n_i=30$ and ${\tau }^2=1$. When $m=30$, the coverage probability was either equal or slightly less than the nominal value. For the equal variance scenario, the AMSE and bias of Cohen’s d were higher than those of Hedges’ g when $n_i=15$. The values of the AMSE and bias of both methods were comparable for the unequal variance scenario. Hedges’ g showed higher values of the AMSE and bias when $n_i=15$ for both the equal and unequal study variance scenarios. For both methods, the AMSE and bias increased as the treatment effect heterogeneity increased for both the equal and unequal study variances scenarios. The coverage probability for Hedges’ g was comparable for the equal and unequal study variance scenarios, but it was slightly liberal when $m=10$. (A liberal coverage probability is lower than the nominal value, while a conservative one is higher than the nominal value.) Coverage probabilities were comparable for Cohen’s d and Hedges’ g methods for all simulation scenarios.

The AMSE and bias of the limit meta-analysis method under the equal variance scenario were lower than those under the unequal variance scenario. The statistical power of the method under the equal and unequal variance scenarios were comparable.

4.2.7. Random-Effects Model Using the DerSimonian–Laird Estimate

For the mean difference, the AMSE and bias increased when the ${\tau }^2=5$ for both the equal and unequal study variance scenarios. Coverage probabilities were slightly lower than those of Cohen’s d and Hedges’ g.

For Cohen’s d, the AMSE and bias of the random-effects model were higher for the equal study variance scenario when $n_i=15$ but not when $n_i=30$. The AMSE and bias increased when ${\tau }^2=5$ for the equal and unequal study variances scenarios. The coverage probability for Cohen’s d was comparable for the equal and unequal study variance scenarios. Hedges’ g showed lower values of the AMSE and bias and when $n_i=15$ for the equal study variance scenario for different values of ${\tau }^2$ than when $n_i=30$ for both the equal and unequal variance scenarios. The AMSE and bias of Hedges’ g increased when ${\tau }^2=5$ for both the equal and unequal study variances scenarios. The AMSE and bias were generally higher for the unequal study variance scenario than for the equal study variance scenario. The coverage probabilities for both Cohen’s d and Hedges’ g were generally low and decreased when $n_i=30$. The coverage probabilities of Cohen’s d and Hedges’ g had comparable values except when ${\tau }^2=5$ for the equal study variance scenario where Hedges’ g had a higher coverage probability.

The AMSE and bias of the random-effects model using the DerSimonian–Laird estimate method under the equal variance scenario were lower than those under the unequal variance scenario. The statistical power of the method was lower under the equal variance scenario.

4.3. Comparison of Results of the Methods Correcting for Publication Bias

4.3.1. AMSE

For the mean difference, the lowest AMSE was seen for the limit meta-analysis method when $m=30$. The highest AMSE was observed for the PET-PEESE method. As for Cohen’s d and Hedges’ g methods, the p-uniform had the lowest AMSE followed by the Copas and Trim and Fill methods. The highest AMSE was found for the PET-PEESE and the limit meta-analysis methods, respectively.

4.3.2. Bias

For the mean difference, the PET PEESE was almost always negatively biased but had the lowest bias when disregarding the sign. The p-uniform method had the highest bias among all methods. The Copas, Trim and Fill and random-effects model using the DerSimonian–Laird variance component had comparable biases. As for Cohen’s d and Hedges’ g methods, the limit meta-analysis method was generally positively biased, and its bias became smaller when $m=30$. In addition, the limit meta-analysis method had a lower absolute bias than the PET PEESE. The Copas method was positively biased but had a relatively small bias. The Trim and Fill and the random-effects meta-analysis methods were positively biased and had comparable biases. The p-uniform method was also positively biased but had the biggest bias among all the methods.

4.3.3. Coverage Probability

The PET PEESE usually had a coverage probability below the nominal value, except in the case of $m=10$ and $n_i=15$. As mentioned above, the limit meta-analysis had a liberal coverage probability when $m=10$ but a slightly conservative coverage probability when $m=30$. The Copas and the Trim and Fill methods had comparable coverage probability. Both methods clearly had a lower than nominal coverage probability, and their coverage probabilities decreased when $n_i$ = 30. The p-uniform method and the random-effects meta-analysis methods had the lowest coverage probability.

5. Discussion

The aim of this article was to compare the performance of five methods adjusting for publication bias. The five methods were Copas, p-uniform, PET-PEESE, Trim and Fill, and limit meta-analysis methods. In addition, the random-effects meta-analysis method based on the DerSimonian–Laird estimate was also included. This study focused on continuous outcomes and investigated the adjustment methods using the mean difference, Cohen’s d, and Hedges’ g. The performance of the adjustment methods were compared using a case study and a simulation study. The simulation settings included different scenarios with different number of studies, different number of individuals per study, treatment effect homogeneity, and different levels of treatment effect heterogeneity. In addition, the two scenarios of equal and unequal variances for the treatment and control groups were also investigated. The performance of the adjustment methods was investigated by calculating the average mean squared error, the bias, and the coverage probability.

The weak performance of the random-effects meta-analysis model had been reported earlier in the literature [27]. The PET-PEESE method performed slightly worse as the treatment effect heterogeneity increased, a result also stated earlier in the literature [24]. The p-uniform method overestimated the treatment effect, and its performance deteriorated as the treatment effect heterogeneity increased, a result also presented earlier in the literature [19]. The limit meta-analysis was too conservative compared to the other methods investigated. This is probably the result of the equivalence of the method to a regression analysis and the fact that the limits are basically prediction limits. These prediction limits not only take the precision of the mean value into account but also the precision of the between-study variance component and that of the parameter representing publication bias.

Copas’ method was compared to the Trim and Fill method using the odds ratio. The Copas method was preferred since it produced smaller standard errors [26]. Our analysis for Cohen’s d and Hedges’ g also showed that the Copas method had smaller bias and higher coverage probabilities than the Trim and Fill method. However, the Copas method performed worse as treatment effect heterogeneity increased.

Regarding the mean difference, all methods performed worse in the unequal variance scenario, with the exception of the PET-PEESE and the p-uniform methods. This can be explained by the fact that PET-PEESE applies weighted regression, taking heteroscedasticity into account. No direct explanation could be found for the robust performance of the p-uniform method under heteroscedasticity.

In general, no clear difference in performance was observed between Cohen’s d and Hedges’ g for all methods under the different scenarios. This can be explained by the similarity of both methods, as both assume equal variances and use a pooled variance. Both Cohen’s d and Hedges’ g performed better under the equal variance scenario for all methods, with the exception of the PET-PEESE method. The robustness of the PET-PEESE method in the case of heteroscedasticity could be explained by the fact that both estimates are based on weighted regression models, which take heteroscedasticity into account.

6. Conclusions

Disregarding the direction of the estimation bias, the Copas and PET-PEESE methods were the least biased methods. However, the Copas method, like all likelihood-based methods, can have convergence problems, rendering the method less applicable to meta-analysts. This disadvantage of the Copas method is an advantage to the PET-PEESE. In addition, the PET-PEESE method is robust in the case of unequal variances for the treatment and control arms. For these two reasons, meta-analysts are advised to apply the PET-PEESE method. Given the comparable performance of Cohen’s d and Hedges’ g, practitioners could be encouraged to use either measure for the meta analysis.

7. Study Limitations

There are many selection models in the literature introduced to correct for publication bias [1,9,10,29]. To include all these methods was beyond the scope of this study. In addition, no Bayesian methods have been included in this study, while these have been applied to correct for publication bias [42].

Highlights

Comparative studies for publication bias adjustment methods are available but limited. This study provides a broader comparative study focusing on continuous outcomes. Impact: recommendations for meta-analysts on which method to apply based on the least bias, among other factors.