Optimal Concordant Tests

Abstract

In meta-analyses, unlike model-based methods, such as fixed- or random-effect models, the p-value combining methods are distribution-free and robust. How to appropriately and powerfully combine p-values obtained from various sources remains an important but challenging topic in statistical inference. For cases where all or a majority of the individual alternative hypotheses have the same but unknown direction, concordant tests based on one-sided p-values can substantially improve the detecting power. However, there exists no test that is uniformly most powerful; therefore, figuring out how to choose a robust and powerful test to combine one-sided p-values for a given data set is desirable. In this paper, we propose and study a class of gamma distribution-based concordant tests. Those concordant tests are optimal under specific conditions. An asymptotically optimal concordant test is also studied. The excellent performances of the proposed tests were demonstrated through a numeric simulation study and real data example.

1. Introduction

Due to recent technical developments, larger volume data, including genome-wide genomic data, can be generated rapidly and at a low cost. As a consequence, powerful statistical approaches, including p-value combination tests, are highly desirable for analyzing these data sets. For instance, meta-analyses, when applied to genome-wide association studies (GWASs), have discovered many associated genetic variants that could not be identified from a single GWAS. With the development and application of advanced powerful p-value combination tests, we expect that many more casual genetic variants will be identified through existing data.

In meta-analyses, p-value combination methods are important alternatives to model-based techniques [1,2], especially when the fixed- or random-effect models have a lack of fit [3]. In the literature, many p-value combination methods, although first proposed a long time ago, are still widely used nowadays. These methods include popular ones: Fisher test [4], Pearson test [5], minimal p-test [6], z-test [7], and chi-square test [8,9]. Recently, we studied a class of p-value combination tests based on gamma distribution, which include those popular tests as special cases [10]. However, as noticed by Birnbaum, there exists no uniformly most powerful (UMP) p-value combination test for all conditions [11]. Therefore, robust methods that have good power under many conditions are desirable [12].

Under some conditions, we may know or reasonably assume that all or most of the true effects under the individual alternative hypotheses have the same but unknown direction. In this case, the concordant tests which use one-sided p-values have been shown to be more powerful than those methods based on two-sided p-values, such as the chi-square test [1]. For instance, in GWASs, studies with the same phenotype are usually conducted among several different subpopulations, from which the effect of a causal genetic variant may have the same direction but various sizes. Then, if the commonly used fixed- and random-effect meta-analyses are applied to combine the results from individual studies, they may lose the power to detect the associated genetic variants. To improve the detecting power, in this paper, we study a class of concordant tests based on gamma distribution, and then an optimal concordant test based on constrained likelihood ratio test (CLRT) is developed.

The rest of the manuscript is organized as follows. In Section 2, we first introduce the Pearson’s concordant test, which was studied by Owen [1] (Section 2.1). In Section 2.2, we describe our proposed generalized concordant tests based on gamma distribution and study their properties as UMP tests under some conditions. The asymptotically UMP concordant test based on CLRT is proposed and studied in Section 2.3. In Section 3, we compare the performances of the proposed concordant tests with some existing methods through a simulation study. In addition, an example of a real data application is illustrated to demonstrate the desired performance of the proposed tests. This paper concludes in Section 4 with a discussion and conclusions.

2. Methods

Suppose we have $n$ independent studies. For each study, $i \left(i=1,\cdots ,n\right)$, we can perform a two-sample test for testing the null hypothesis, $H_{i0}: {\mu }_i=0$, against the alternative hypothesis, $H_{i1}:{\mu }_i\ne 0$, where ${\mu }_i$ is the mean difference between two populations. The corresponding p-value is denoted as $P_i\left(i=1,\cdots ,n\right)$. Throughout this paper, we assume $P_i~\mathrm{U}\left(\mathrm{0,1}\right)$ under $H_{i0}$, where $\mathrm{U}\left(\mathrm{0,1}\right)$ stands for the uniform distribution between 0 and 1. In a meta-analysis, we consider testing the global null hypothesis, $H_0$ = $\cap H_{i0}$, vs. the global alternative hypothesis, $H_1=\cup H_{i1}$, using the $n$ observed p-values.

For a concordant test, we assume, under the global alternative, that the majority of the non-zero mean differences have the same but unknown direction (positive or negative): ${\mu }_i\ge 0$ and at least one strictly greater than holds, or ${\mu }_i\le 0$ and at least one strictly less than holds ($i=1,\cdots ,n$). Although, under this setting, a two-sided p-value can still be calculated from each data set and then used in p-value combination, in this paper, we focus on the concordant tests which use one-sided p-values, $P_i$, obtained from the directional alternatives ${\mu }_i>0$ or ${\mu }_i<0$ for all $i$ values $\left(i=1,\cdots ,n\right)$. In general, either the right- or left-sided p-values can be used, as they will produce the same result. Without loss of generality, we use the right-sided individual p-values hereafter, unless otherwise specified.

2.1. Pearson’s Concordant Test

Owen [1] revisited and studied the following Pearson’s concordant test, whose test statistic is defined as follows:

$$Q^C=\mathrm{m}\mathrm{a}\mathrm{x}(-2{\sum }_{i=1}^n\ln \left(P_i\right),-2{\sum }_{i=1}^n\ln \left({1-P}_i\right)),$$

(1)

where $P_i \left(i=1,\cdots ,n\right)$ represents the one-sided p-values obtained from the same directional alternatives. The two components, $-2\sum _{i=1}^n\ln \left(P_i\right)$ and $-2\sum _{i=1}^n\ln \left({1-P}_i\right)$, in $Q^C$ are obtained through the popular Fisher test for combining independent p-values [4]. Therefore, under the global null hypothesis, each component is the random variable, ${\chi }_{2n}^2$, which has a chi-square distribution with degrees of freedom equal to $2n$. Furthermore, although $-2\sum _{i=1}^n\ln \left(P_i\right)$ and $-2\sum _{i=1}^n\ln \left({1-P}_i\right)$ are not independent, the p-value of the Pearson’s concordant test defined in (1) can be easily approximated using the upper bound, $2P[{\chi }_{2n}^2>q^C]$, where $q^C$ is the observed test statistic. In addition, the approximation is very accurate when the true p-value is small [1].

2.2. Concordant Tests Based on Gamma Distribution

The above Pearson’s concordant test can be extended to concordant tests based on gamma distribution. For this purpose, we first study a class of combination tests based on gamma distribution [10].

For given shape parameter $\alpha$ and scale parameter $\beta$, a p-value combination test statistic based on the gamma distribution can be constructed as follows:

$$T_{G\left(\alpha ,\beta \right)}\left(P_1,P_2,\cdots ,P_n\right)=\sum _{i=1}^n{F}_{G\left(\alpha ,\beta \right)}^{-1}\left(1-P_i\right),$$

(2)

where $F_{G(\alpha ,\beta )}^{-1}\left(y\right)$ is the inverse function of the cumulative distribution function (CDF), $F_{G(\alpha ,\beta )}\left(x\right)$, of the random variable $Gamma\left(\alpha ,\beta \right)$, whose probability density function (PDF) is $f_{G(\alpha ,\beta )}\left(x\right)={\beta }^{\alpha }{x}^{\alpha -1}\mathrm{e}\mathrm{x}\mathrm{p}(-\beta x)/\mathsf{\Gamma }\left(\alpha \right)$ for $x>0$, where the gamma function $\mathsf{\Gamma }\left(z\right)={\int }_0^{\infty }{x}^{z-1}{e}^{-x}dx$. The right-sided p-value of the test defined in (2) can be defined as follows:

$$P=P\left(Gamma\left(n\alpha ,\beta \right)>t\right)=1-F_{G\left(n\alpha ,\beta \right)}\left(t\right)=S_{G\left(n\alpha ,\beta \right)}\left(t\right),$$

(3)

where $Gamma\left(n\alpha ,\beta \right)$ is the random variable having gamma distribution with the shape parameter, $n\alpha$, and the scale parameter, $\beta$; and $F_{G\left(n\alpha ,\beta \right)}$ and $S_{G\left(n\alpha ,\beta \right)}$ are the CDF and the survival function of $Gamma\left(n\alpha ,\beta \right)$.

It can be shown that, due to the property of gamma distribution, parameter $\beta$ has no effect on the test defined in (2) and (3) [10]. Therefore, for simplicity, we use $\beta =1$ in the above test and denote $T_{G\left(\alpha ,1\right)}$ as $T_{G\left(\alpha \right)}$ hereafter. Furthermore, we can prove the following properties for the above class of gamma distribution-based tests [10]: (i) $T_{G\left(0\right)}\triangleq \underset{\alpha \to 0+}{\lim }{T}_{G\left(\alpha \right)}$ is equivalent to the Tippett’s minimal-p test; (ii) $T_{G\left(0.5\right)}$ is the same as the chi-square test with degrees of freedom equal to $n$; (iii) $T_{G\left(1\right)}$ is equavalent to Fisher test; and (iv) $T_{G(\infty )}\triangleq \underset{\alpha \to \infty }{\lim }{T}_{G\left(\alpha \right)}$ is the same as the z-test. Therefore, the class of gamma distribution-based tests, $T_{G\left(\alpha \right)}$, includes many popular tests as special cases. This observation motivates us to generalize the Pearson’s concordant test.

We propose the gamma distribution-based concordant test with the following test statistic:

$$T_{G\left(\alpha \right)}^C=\mathrm{m}\mathrm{a}\mathrm{x}(T_{G\left(\alpha \right)}^L,T_{G\left(\alpha \right)}^R),$$

(4)

where $T_{G\left(\alpha \right)}^L=T_{G\left(\alpha \right)}\left(P_1,P_2,\cdots ,P_n\right)=\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}(1-P_i)$ and $T_{G\left(\alpha \right)}^R=T_{G\left(\alpha \right)}\left({1-P}_1,{1-P}_2,\cdots ,{1-P}_n\right)=\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}(P_i)$ are the gamma distribution-based tests using the left- and right-sided p-values, respectively.

It is easy to see that when $\alpha =1$, $T_{G\left(\alpha \right)}^C$ is equivalent to the Pearson’s concordant test $Q^C$. Therefore, $T_{G\left(\alpha \right)}^C$ is a generalization of the Pearson’s concordant test. We show that the p-value for the proposed concordant test, $T_{G\left(\alpha \right)}^C$, can also be estimated in a similar way as $Q^C$: $P\left[T_{G\left(\alpha \right)}^C>A\right]\approx 2P[T_{G\left(\alpha \right)}^L>A]$. Now, we study the properties of $T_{G\left(\alpha \right)}^C$. First we need the following definitions and lemmas [13].

Definition 1. 

A function $f$ on ${\mathbb{R}}^n$ is nondecreasing if it is nondecreasing in each of its $n$ arguments when the other $n-1$ values are held fixed.

Definition 2. 

Let $X_1, \cdots ,X_n$ be random variables with a joint distribution. These random variables are associated if the covariance $Cov(f\left(X\right),g(X\left)\right)\ge 0$ holds for all nondecreasing functions, $f$ and $g$, for which the covariance is defined.

Lemma 1. 

(Theorem 2.1 of Esary, Proschan, and Walkup 1967 [13]) Independent random variables are associated.

Lemma 2. 

(Theorem 5.1 of Esary, Proschan, and Walkup 1967 [13]) Let $X_1, \cdots ,X_n$ be associated random variables, $X=\left(X_1,\cdots ,X_n\right),S_i=f_i\left(X\right),i=1,\cdots ,k$, then $P[S_1\le s_1,\cdots ,S_k\le s_k]\ge \prod _{i=1}^k[S_i\le s_i]$ and $P[S_1>s_1,\cdots ,S_k>s_k]\ge \prod _{i=1}^k[S_i>s_i]$ for all $s_1,s_2,\cdots ,s_k$.

Lemma 3. 

For integer $n\ge 1$, let $X=(X_1,\cdots ,X_n)\in {\left(\mathrm{0,1}\right)}^n$ have independent components. Set $T_{G\left(\alpha \right)}^L=T_{G\left(\alpha \right)}(X_1,\cdots ,X_n)$, and $T_{G\left(\alpha \right)}^R=T_{G\left(\alpha \right)}({1-X}_1,\cdots ,{1-X}_n)$. Then, for any $A_L>0$, and $A_R>0$, $P[T_{G\left(\alpha \right)}^L>A_L,T_{G\left(\alpha \right)}^R>A_R]\le P[T_{G\left(\alpha \right)}^L>A_L]P[T_{G\left(\alpha \right)}^R>A_R]$.

Proof. 

Denote $f\left(X\right)=T_{G\left(\alpha \right)}^L=-\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}(1-P_i)$ and $g\left(X\right)=T_{G\left(\alpha \right)}^R=\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}\left(P_i\right)$. It is easy to see that both $f$ and $g$ are nondecreasing functions of $X$. And the components of $X$ are independent and hence associated. Therefore, we have

$$\begin{array}{c}P\left[T_{G\left(\alpha \right)}^L>A_L,T_{G\left(\alpha \right)}^R>A_R \right]=P\left[-f\left(X\right)>A_L, g\left(X\right)>A_R\right]=P[f\left(X\right)<\\ -A_L, g\left(X\right)>A_R]=P\left[g\left(X\right)>A_R\right]-P\left[f\left(X\right)>-A_L, g\left(X\right)>A_R\right]\le \\ P\left[g\left(X\right)>A_R\right]-P\left[f\left(X\right)>-A_L\right]P\left[g\left(X\right)>A_R\right]=P\left[g\left(X\right)>A_R\right]P[f\left(X\right)<\\ -A_L]=P\left[T_{G\left(\alpha \right)}^R>A_R\right]P[T_{G\left(\alpha \right)}^L>A_L].\end{array}$$

□

The concordant test, $T_{G\left(\alpha \right)}^C$, defined in (4) has the following property.

Theorem 1. 

Let $X=(X_1, \cdots ,X_n)\in {\left(\mathrm{0,1}\right)}^n$ be a random vector with independent components, and denote $T_{G\left(\alpha \right)}^L=T_{G\left(\alpha \right)}(X_1,\cdots ,X_n)$, $T_{G\left(\alpha \right)}^R=T_{G\left(\alpha \right)}({1-X}_1,\cdots ,{1-X}_n)$, and $T_{G\left(\alpha \right)}^C=max(T_{G\left(\alpha \right)}^L,T_{G\left(\alpha \right)}^R)$. For any $A\in \mathbb{R}$, let $P_L=P[T_{G\left(\alpha \right)}^L>A]$, $P_R=P[ T_{G\left(\alpha \right)}^R>A]$, and $P_C=P[T_{G\left(\alpha \right)}^C>A]$. Then $P_L+P_R-P_L{P}_R\le P_C\le P_L+P_R$

Proof. 

From Lemmas 2 and 3, we have $P_C=P\left[T_{G\left(\alpha \right)}^C>A\right]=P\left[T_{G\left(\alpha \right)}^L>A or T_{G\left({\alpha }_0\right)}^R>A\right]=P\left[T_{G\left(\alpha \right)}^L>A]+P[T_{G\left(\alpha \right)}^R>A\right]-P\left[T_{G\left(\alpha \right)}^L>A,T_{G\left(\alpha \right)}^R>A\right]\ge P\left[T_{G\left(\alpha \right)}^L>A]+P[T_{G\left(\alpha \right)}^R>A\right]-P\left[T_{G\left(\alpha \right)}^L>A\left]P\right[T_{G\left(\alpha \right)}^R>A\right]=P_L+P_R-P_L{P}_R$. On the other hand, from Bonferroni inequality, we have $P_C\le P\left[T_{G\left(\alpha \right)}^L>A]+P[T_{G\left(\alpha \right)}^R>A\right]=P_L+P_R$. □

Under the condition specified in Theorem 1 (i.e., the global null hypothesis), it is easy to see that $T_{G\left(\alpha \right)}^L$ and $T_{G\left(\alpha \right)}^R$ have the same distribution; hence, $P\left[T_{G\left(\alpha \right)}^L>A\right]=P[T_{G\left(\alpha \right)}^R>A]$.

From Theorem 1, we have the following result.

Corollary 1. 

Suppose $X~U{\left(\mathrm{0,1}\right)}^n$, for $A>0$, let $p_A=P[T_{G\left(\alpha \right)}^L>A]$, then ${2p}_A-{p_A}^2\le P[T_{G\left(\alpha \right)}^C>A]\le {2p}_A$.

From Corollary 1, the p-value of the concordant test, $T_{G\left(\alpha \right)}^C$, defined in (4),$P\left[T_{G\left(\alpha \right)}^C>A\right],$ can be easily approximated by twice of the p-value from $T_{G\left(\alpha \right)}^L$, i.e., $P\left[T_{G\left(\alpha \right)}^C>A\right]\approx 2P[T_{G\left(\alpha \right)}^L>A]$. The approximation is very accurate if the true p-value (e.g., the p-value from the sampling distribution under the alternative) is small, as ${p_A}^2$ is even much smaller under this situation.

In the following subsection, we study an optimal concordant test when the optimal value for $\alpha$ is unknown and needs to be estimated from the data.

2.3. Optimal Concordant Test

For a given set of one-sided p-values, the class of concordant tests, $T_{G\left(\alpha \right)}^C$, proposed in the previous section, Section 2.2, with different $\alpha$ values can be applied. However, in order to control type I error, the $\alpha$ value must be chosen before we see the data. But usually, the optimal value is unknown, and therefore how to choose an optimal value for $\alpha$ is critical in practice. We may choose a set of $\alpha$ values to apply the tests $T_{G\left(\alpha \right)}^C$ to the data and then use the smallest p-value among them. There are two limitations associated with this procedure. First, we need to adjust for multiple comparisons, which may result in power loss. Second, the final result will depend on the chosen $\alpha$ values; it might be difficult to determine those values. To circumvent these difficulties, we propose to estimate the optimal $\alpha$ value from the data, and therefore an asymptotically optimal concordant test can be constructed accordingly.

In this subsection, we make the following assumption: $P_1,P_2,\cdots ,P_n$ are independently and identically distributed $\left(iid\right)$ with the following density function for parameters $\alpha >0$ and $c<1$:

$$f_{\alpha ,c}\left(p\right)={\left(1-c\right)}^{\alpha }\mathrm{e}\mathrm{x}\mathrm{p}\left[c{F}_{G\left(\alpha \right)}^{-1}\left(1-p\right)\right] \mathrm{for} p\in \left(\mathrm{0,1}\right).$$

(5)

For the above densify function, $f_{\alpha ,c}\left(p\right)$, it is not difficult to show that when $c=0$, $f_{\alpha ,0}\left(p\right)=1 (0<p<1)$. Therefore, $f_{\alpha ,0}\left(p\right)$ corresponds to the global null hypothesis. In addition, we have the following property [10].

Proposition 1. 

Suppose that $P_1,P_2,\cdots ,P_n$ are $iid$ with the common density function defined in (5) with parameters $\alpha >0$ and $c<1$. If both $\alpha$ and $c$ are known, then the gamma distribution-based test, $T_{G\left(\alpha \right)}$, is uniformly most powerful (UMP).

When both parameters $\alpha$ and $c$ in (5) are unknown, they can be estimated via the constrained maximum likelihood estimation (CMLE), from which the following constrained likelihood ratio test (CLRT) can be defined:

$$T_{CLRT}(P_1,\cdots ,P_n)=2{\hat{\alpha }}_{CLRT}n\mathrm{l}\mathrm{n}\left({1-\hat{c}}_{CLRT}\right)+2\left({\hat{c}}_{CLRT}\right){\sum }_{i=1}^n{F}_{G\left({\hat{\alpha }}_{CLRT}\right)}^{-1}\left({1-P}_i\right),$$

(6)

where ${\hat{\alpha }}_{CLRT}$ and ${\hat{c}}_{CLRT}$ are the CMLEs for parameters $\alpha$ and $c$, respectively, through maximizing the log-likelihood function $l\left(\alpha ,c\right)=n\alpha \mathrm{l}\mathrm{n}\left(1-c\right)+c\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}\left({1-P}_i\right)$ with the constrains $c<1$ and $\alpha >0$.

For the above CLRT-based test, $T_{CLRT}$, we have the following result [10,14].

Proposition 2. 

The asymptotic distribution of the test $T_{CLRT}$ is a mixture of chi-square distributions, $\sum _{i=0}^2{w}_i{\chi }_i^2$, where ${\chi }_i^2$ is the chi-square distribution with $df=i$; ${\chi }_0^2$ is the random variable with probability 1 of being 0; and the weights $w_0$, $w_1$, and $w_2$ are determined by the null and the alternative hypothesis.

The above asymptotic result may not be directly applicable to estimate the p-value for this test. First, the number of p-values, $n$, is usually small, and the asymptotic result may not provide approximate solutions. Second and more seriously, the above weights, ${w_i}^{\prime }s$, are difficult to obtain. Hence, here we use a simple resampling method to approximate the null distribution and to estimate the p-value of $T_{CLRT}$. More specifically, for a given sample size $n$, we randomly sample $n$ independent null p-values from the uniform distribution U(0,1) and then calculate the test statistic using (6). Repeat this process for B times (e.g., ${B=10 }^5$) to get the empirical null distribution of the test statistic, which will be used to approximate the p-value of $T_{CLRT}$:$p=\#\left[{T_i\ge T}_{CLRT}\left(p_1,\cdots ,p_n\right)\right]/B$, where $T_i (i=1,\cdots ,B)$ is the test statistic from the $i^{th}$ resampling, and $T_{CLRT}\left(p_1,\cdots ,p_n\right)$ is the observed test statistic from the data.

For $T_{CLRT}$, we have the following result [10]:

Proposition 3. 

Under the conditions specified in (5), the CLRT-based test, $T_{CLRT}$, is asymptotically UMP.

We now propose an optimal concordant test. Denote $T_{CLRT}^L=T_{CLRT}(P_1,P_2,\cdots ,P_n)$, and $T_{CLRT}^R=T_{CLRT}({1-P}_1,{1-P}_2,\cdots ,{1-P}_n)$, and then define the following concordant test statistic:

$$T_{CLRT}^C=\mathrm{m}\mathrm{a}\mathrm{x}(T_{LRT}^L,T_{LRT}^R).$$

(7)

Note that the null distributions of $T_{LRT}^L \mathrm{a}\mathrm{n}\mathrm{d} T_{LRT}^R$ are identical. For the concordant test, $T_{CLRT}^C$, we have the following property.

Theorem 2. 

Let $X=(X_1, \cdots ,X_n)\in {\left(\mathrm{0,1}\right)}^n$ be a random vector with independent components, and let $T_{CLRT}^L=\underset{\alpha >0,c<1}{\mathit{max}}[n\alpha ln\left(1-c\right)+c\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}\left({1-X}_i\right)]$, $T_{CLRT}^R=\underset{\alpha >0,c<1}{\mathit{max}}[n\alpha ln\left(1-c\right)+c\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}\left(X_i\right)]$, and $T_{CLRT}^C=max(T_{CLRT}^L,T_{CLRT}^R)$. For any $A\in \mathbb{R}$, let $P_L=P[T_{CLRT}^L>A]$, $P_R=P[T_{CLRT}^R>A]$, and $P_C=P[T_{CLRT}^C>A]$. Then, $P_L+P_R-P_L{P}_R\le P_C\le P_L+P_R$.

Proof. 

First notice that $f\left(X\right)=-\underset{\alpha >0,c<1)}{\max }[n\alpha \mathrm{l}\mathrm{n}\left(1-c\right)+c\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}\left({1-X}_i\right)]$ and $g(X)=\underset{\alpha >0,c<1)}{\max }[n\alpha \mathrm{l}\mathrm{n}\left(1-c\right)+c\sum _{i=1}^n{F}_{G\left(\alpha \right)}^{-1}\left(X_i\right)]$ are both nondecreasing functions of associated random variables, $X$. The rest of the proof is similar to that for Theorem 1. □

In addition, we have the following result for $T_{CLRT}^C$.

Corollary 2. 

Suppose $X~U{\left(\mathrm{0,1}\right)}^n$; for $A>0$, let $p_A=P[T_{CLRT}^L>A]$, and then ${2p}_A-{p_A}^2\le P[T_{CLRT}^C>A]\le {2p}_A$.

From Corollary 2, the p-value of the concordant test, $T_{CLRT}^C$, defined in (7) can be estimated by the upper bound, ${2p}_A$, where $p_A$ is the p-value from the CLRT-based test, $T_{CLRT}^L$, and can be estimated using the resampling method described previously. Again, when the true p-value is small, this approximation is very accurate.

3. Results

3.1. Numeric Results from Simulation Study

In this section, we assess the performance of the proposed tests through a simulation study. In the simulation, we compare the proposed optimal concordant test, $T_{CLRT}^C$, with gamma distribution-based concordant tests, $T_{G\left(0\right)}^C$, $T_{G\left(1\right)}^C$ (i.e., the Owen’s Pearson concordant test), and $T_{G\left(\infty \right)}^C$, using one-sided p-values.

In the simulation study, we want to combine fifty ($n=50$) randomly simulated independent p-values using different methods. We assume that a number, $m (m=10, 20, 40, 50)$, out of these n p-values are from the true individual alternative hypotheses, and the rest ($n-m$) are from the true individual null hypotheses. The n-m p-values from the true null hypotheses are randomly sampled from the uniform distribution with range 0 and 1. For a true individual alternative hypothesis, $H_{i1} (i=1,\cdots ,m)$, the p-value $p_i$ is obtained via a normal variable, $z_i~N({\mu }_i,1)$. To consider the situations where the true effects from different studies may have opposite directions (positive and negative), we randomly set $k$ of the $m$ ${\mu }_i s$ to have the same direction (positive or negative), and the rest of $m-k$ has the other direction. A one-sided (right-sided) p-value for each true individual alternative hypothesis is obtained via the standard z-test, i.e., $p=P[Z>z]$, where Z is the standard normal distribution, $N\left(\mathrm{0,1}\right)$, and $z$ is the simulated number as described above.

For the hypotheses under true alternative, we consider three different scenarios for the effects of ${{\mu }_i}^{\prime }s$. Scenario 1: We use $\left|{\mu }_i\right|=\mu v_i/\sum _{i=1}^m{v}_i$, where $v_i={10}^{r_i}$, $r_i~N\left(\mathrm{0.3,1}\right)$, and $\mu =0.8, 0.6, 0.4, 0.3$ when there are 10, 20, 40, and 50 true individual alternatives, respectively. Scenario 2: We set $\left|{\mu }_i\right|=\mu v_i/\sum _{i=1}^m{v}_i$, where $v_i~U\left(\mathrm{1,100}\right)$ and $\mu =1.2, 1.0, 0.6, 0.5$ when the number of true individual alternatives is $m=10, 20, 40, \mathrm{and} 50$, respectively. Scenario 3: We consider $\left|{\mu }_i\right|=\mu /m$, $\mu =1.5, 1.2, 0.8, \mathrm{and} 0.6$ when $m=10, 20, 40, \mathrm{and} 50,$ respectively. Note that, in the simulation, we consider the following: (i) The constants (e.g., $\mu$, the parameters in the normal distribution for $r_i$ and the uniform distribution for $v_i$) are chosen in a way that the empirical powers are appreciable for comparison. (ii) The sum of the absolute effect sizes is equal to $\mu$ for all the three scenarios. (iii) The degree of heterogeneity of the effect sizes among the true individual alternatives decreases from Scenario 1 to Scenario 3 for each given $m$.

The empirical power values of the tests are estimated as the rejection proportions based on $1000$ replicates, with 0.05 as the significance level. For the CLRT-based concordant test, $T_{CLRT}^C$, we use ${B=10}^5$ samples to estimate the p-values from the resampling method described in Section 2.3.

When all of the 50 p-values are from null hypotheses, the empirical power is the type 1 error rate. All methods considered here were able to control the type I error rate: their empirical powers under this situation were close to 0.05 when the significance level 0.05 was used.

Figure 1, Figure 2 and Figure 3 plot the empirical power values for the concordant tests based on minimal p, $T_{G\left(0\right)}^C$ (denoted as Min_CC), Fisher $T_{G\left(1\right)}^C$ (denoted as Fisher_CC, i.e., Pearson’s concordant test), z-test $T_{G\left(\infty \right)}^C$ (denoted as Z_CC), and the proposed concordant test, $T_{CLRT}^C$ (denoted as LRT_CC), when one-sided p-values under Scenarios 1 to 3, respectively, are used.

From the simulation study, we have the following observations. First, under Scenario 1 (Figure 1), where the effect sizes under alternative hypotheses are extremely heterogeneous, usually the minimal p-based concordant test (i.e., $T_{G\left(0\right)}^C$) performs better than the Pearson’s concordant test ($T_{G\left(1\right)}^C$), which in turn outperforms the z-test-based concordant test ($T_{G\left(\infty \right)}^C$). Second, under Scenarios 2 and 3, when the degrees of heterogeneity of the effect sizes among individual alternative hypotheses are less extreme than in Scenario 1, $T_{G\left(1\right)}^C$ and the $T_{G\left(\infty \right)}^C$ usually perform better or much better than $T_{G\left(0\right)}^C$ (Figure 2 and Figure 3). Third, for the three concordant tests with fixed $\alpha$ values, namely $T_{G\left(0\right)}^C$, $T_{G\left(1\right)}^C$, and $T_{G\left(\infty \right)}^C$, one may perform well under some conditions but do poorly under others. Fourth, under all conditions considered, the CLRT-based concordant test, $T_{CLRT}^C$, always has the best or the second-best power. This demonstrates that, as expected, $T_{CLRT}^C$ is a robust test, meaning that, under many conditions, it has reasonable detection power compared with other tests.

3.2. Real Data Application

In this section, we demonstrate the usefulness of the proposed tests by applying them to a real-world problem. A data set from a meta-analysis is used. In that meta-analysis, results from 12 randomized trials—each examined the effect of patient rehabilitation designed for geriatric patients versus usual care on improving functional outcome at 3–12-month follow-ups—were collected [15,16]. The 12 estimated odds ratios (ORs) from the individual studies are listed in

Table 1. Estimated odds ratio and its 95% CI from each study in a meta-analysis with 12 trials. Data were taken from Bachmann et al. [15] and Riley et al. [16].

Study	OR	95% CI	Study	OR	95% CI	Study	OR	95% CI
1	1.11	0.51, 2.39	5	0.88	0.39, 1.95	9	1.06	0.63, 1.79
2	0.97	0.78, 1.21	6	1.28	0.71, 2.30	10	2.95	1.54, 5.63
3	1.13	0.73, 1.72	7	1.19	0.69, 2.08	11	2.36	1.18, 4.72
4	1.08	0.42, 2.75	8	3.82	1.37, 10.60	12	1.68	1.05, 2.70

.

The commonly used fixed-effect model for meta-analyses was inadequate for this data set, as the p-value from the Cochran’s test for homogeneity was 0.021. Hence, the random-effect model of the meta-analysis was applied, which estimated the overall OR as 1.36 with a 95% CI (1.08, 1.71) [16]. However, when we applied the goodness-of-fit test for the random-effect model, we obtained a p-value of 0.025 [3], which indicates a lack of fit of the random-effect model for this data set. We then applied the p-value combination methods to test whether there is an overall effect among these 12 independent trials.

To use the p-value combination methods, we need the individual p-values. For each trial, the corresponding p-value is calculated based on the reported 95% CI of the OR. If we use U and L to denote the upper and lower limits of the 95% CI, then the test statistic can be approximated as $z=\ln \left(U\times L\right)/\sqrt{4\ln \left(U/L\right)/3.92},$ which is asymptotically distributed as $N\left(\mathrm{0,1}\right)$ under the null hypothesis that there is no difference between the new treatment and the control. Since the sample sizes of these 12 trials were relatively large, ranging from 108 to 1388, their p-values can be reasonably estimated using the asymptotic null distribution by comparing the test statistic with $N\left(\mathrm{0,1}\right)$. For example, the right-sided p-value can be calculated as $P[Z>z]$, where Z has a standard normal distribution, $N\left(\mathrm{0,1}\right)$, and $z$ is the test statistic calculated as described above.

We first calculate the one-sided p-value for each trial and then use the values for the concordant tests. The p-values from the concordant tests $T_{G\left(0\right)}^C$, $T_{G\left(1\right)}^C$, $T_{G\left(\infty \right)}^C$, and $T_{CLRT}^C$ are 0.013, 0.00017, 0.00031, and 0.00030, respectively. For comparison, we also applied the gamma distribution-based tests to this data set, using two-sided p-values calculated from each individual trial. The resulting p-values are 0.013, 0.0068, 0.075, and 0.0047 from $T_{G\left(0\right)}$, $T_{G\left(1\right)}$, $T_{G(\infty )}$, and $T_{CLRT}$, respectively. As expected, under the situation that a majority of the trials have the same direction (10 out of the 12 estimated ORs were greater than 1), concordant tests are usually more powerful than the corresponding gamma distribution-based tests using two-sided p-values. Noticeably, the z-test-based concordant test, $T_{G\left(\infty \right)}^C$, obtained a much smaller p-value (0.00031) compared with the z-test using two-sided p-values (0.075). Among the concordant tests, the proposed test has the second smallest p-value (0.00030).

4. Discussion and Conclusions

In this paper, we studied a class of gamma distribution-based concordant tests, which include the Pearson’s concordant test as a special case. We also proposed and studied a CLRT-based concordant test, which is asymptotically optimal, robust, and powerful under many conditions. The advantages of the proposed tests were demonstrated based on numeric simulation study and real data application.

In addition to their use for meta-analyses, the proposed concordant tests have many other applications. In a two-way contingency table, if at least one of the two variables is ordinal, a trend test may be more powerful than the popular unidirectional Pearson’s chi-square test [17]. One challenge in trend tests is how to appropriately assign scores for each level of the ordinal variable; results can be quite different with different score assignments. Our proposed concordant test, $T_{CLRT}^C$, can circumvent this difficulty and provide another solution. Specifically, by using the asymptotically independent statistics with the same direction obtained by the Lancaster’s chi-square partition [18,19], we can obtain asymptotically independent one-sided p-values, to which $T_{CLRT}^C$ can be applied.

The proposed tests will also be very useful for detecting casual genetic variants associated with diseases when applied to GWASs. It offers an alternative but powerful tool to combine information from different (subpopulations) but related studies (similar phenotype). We would expect that many genetic variants will be identified in the near future due to the new approach. In the future, we will also study how to combine dependent or correlated p-values.