Global Hypothesis Test to Compare the Predictive Values of Diagnostic Tests Subject to a Case-Control Design

Abstract

Use of a case-control design to compare the accuracy of two binary diagnostic tests is frequent in clinical practice. This design consists of applying the two diagnostic tests to all of the individuals in a sample of those who have the disease and in another sample of those who do not have the disease. This manuscript studies the comparison of the predictive values of two diagnostic tests subject to a case-control design. A global hypothesis test, based on the chi-square distribution, is proposed to compare the predictive values simultaneously, as well as other alternative methods. The hypothesis tests studied require knowing the prevalence of the disease. Simulation experiments were carried out to study the type I errors and the powers of the hypothesis tests proposed, as well as to study the effect of a misspecification of the prevalence on the asymptotic behavior of the hypothesis tests and on the estimators of the predictive values. The proposed global hypothesis test was extended to the situation in which there are more than two diagnostic tests. The results have been applied to the diagnosis of coronary disease.

1. Introduction

The main parameters to assess and compare the accuracy of binary diagnostic tests (BDTs) are sensitivity and specificity. The sensitivity (Se) is the probability of the result of the BDT being positive when the individual has the disease, and the specificity (Sp) is the probability of the result of the BDT being negative when the individual does not have the disease. Other parameters that are used to assess and compare two BDTs are the predictive values (PVs). The positive predictive value (PPV) is the probability of an individual having the disease when the result of the BDT is positive, and the negative predictive value (NPV) is the probability of an individual not having the disease when the result of the BDT is negative. The PVs represent the accuracy of the diagnostic test when it is applied to a cohort of individuals, and they are measures of the clinical accuracy of the BDT. The PVs depend on Se, Sp and on the disease prevalence (p), and are easily calculated applying Bayes Theorem, i.e.,

$$PPV=\frac{p\times Se}{p\times Se+\left(1-p\right)\times \left(1-Sp\right)}\mathrm{and}NPV=\frac{\left(1-p\right)\times Sp}{p\times \left(1-Se\right)+\left(1-p\right)\times Sp}$$

(1)

whereas the Se and the Sp quantify how well the BDT reflects the true disease status (present or absent), the PVs quantify the clinical value of the BDT, since both the individual and the clinician are more interested in knowing how probable it is to have the disease given a BDT result.

The comparison of the performance of two binary diagnostic tests is a topic of special importance in the study of statistical methods for the diagnosis of diseases. This comparison is made through a paired-design or through a case-control design. The paired design consists of applying the two BDTs and the gold standard to all of the individuals in a single sample. The case-control design consists of applying the two BDTs to all of the individuals in two samples, one made up of individuals who have the disease (case sample) and another made up of individuals who do not have the disease (control sample). The advantages and disadvantages of the case-control design over the paired design can be seen in the book by Pepe [1]. Summarizing, the case-control design has some advantages over the paired design: (a) the case-control design is more efficient in terms of sample size requirements, (b) case-control studies allow for the exploration of subject-related characteristics of the test. Nevertheless, the case-control design has the disadvantage is that by using it we cannot estimate the prevalence of the disease.

In paired designs, the comparison of PVs has been the subject of several studies. Bennett [2,3], Leisenring et al. [4], Wang et al. [5] and Kosinski [6] studied hypothesis tests to independently compare the PPVs and the NPVs of two BDTs. Moskowitz and Pepe [7] studied the estimation of the PVs through a confidence region. Roldán-Nofuentes et al. [8] studied the joint comparison of the PPVs and NPVs of two BDTs, and proposed a global hypothesis test based on the chi-square distribution to simultaneously compare the PVs of two BDTs.

In a case-control design, Mercaldo et al. [9] have studied the estimation of the PVs of a BDT, assuming that the prevalence of the disease (p) is known. The prevalence can be known from other studies, such as population studies of health services, cohort studies, etc. Mercado et al. have verified through simulation experiments that the confidence interval with the best asymptotic behavior is the logit interval, whose equations are:

$$\begin{array}{r}PPV\in \left(\frac{\exp \left\{\mathrm{logit}\left(\widehat{PPV}\right)-z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{PPV}\right)\right)}\right\}}{1+\exp \left\{\mathrm{logit}\left(\widehat{PPV}\right)-z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{PPV}\right)\right)}\right\}}\right.;\\ \left.\frac{\exp \left\{\mathrm{logit}\left(\widehat{PPV}\right)+z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{PPV}\right)\right)}\right\}}{1+\exp \left\{\mathrm{logit}\left(\widehat{PPV}\right)+z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{PPV}\right)\right)}\right\}}\right)\end{array}$$

and

$$\begin{array}{r}NPV\in \left(\frac{\exp \left\{\mathrm{logit}\left(\widehat{NPV}\right)-z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{NPV}\right)\right)}\right\}}{1+\exp \left\{\mathrm{logit}\left(\widehat{NPV}\right)-z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{NPV}\right)\right)}\right\}}\right.;\\ \left.\frac{\exp \left\{\mathrm{logit}\left(\widehat{NPV}\right)+z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{NPV}\right)\right)}\right\}}{1+\exp \left\{\mathrm{logit}\left(\widehat{NPV}\right)+z_{1-\alpha /2}\sqrt{\widehat{V}ar\left(\mathrm{logit}\left(\widehat{NPV}\right)\right)}\right\}}\right)\end{array}$$

where $\widehat{PPV}$ and $\widehat{NPV}$ are the estimators of the PVs calculated from Equation (1), $z_{1-\alpha /2}$ is the $100\left(1-\alpha /2\right)\mathrm{th}$ percentile of the normal standard distribution, and the variances are:

$$\widehat{V}ar\left(\mathrm{logit}\left(\widehat{PPV}\right)\right)=\frac{1-\widehat{S}e}{n_1\widehat{S}e}+\frac{\widehat{S}p}{n_2\left(1-\widehat{S}p\right)}\mathrm{and}\widehat{V}ar\left(\mathrm{logit}\left(\widehat{NPV}\right)\right)=\frac{\widehat{S}e}{n_1\left(1-\widehat{S}e\right)}+\frac{1-\widehat{S}p}{n_2\widehat{S}p},$$

where $\widehat{S}e$ and $\widehat{S}p$ are the estimators of sensitivity and specificity, $n_1$ is the size of the case sample and $n_2$ is the size of the control sample.

In this article, we extended the study of Mercaldo et al. [9] to the case of two BDTs, studying different hypothesis tests to compare the PVs of the two BDTs subject to a case-control design. Subject to a case-control design, the two BDTs are applied to all of the individual in two samples, one of $n_1$ individuals who have the disease (case sample) and another with $n_2$ individuals who do not have the disease (control sample). In this design, the sample sizes $n_1$ and $n_2$ are set by the researcher. The sample of individuals that have the disease is extracted from a population of individuals that have the disease (e.g., registers of diseases), and the control sample is extracted from a population of individuals who are known not to have the disease. As the PVs depend on the disease prevalence and subject to a case-control design the quotient $n_1/\left(n_1+n_2\right)$ is not an estimator of the prevalence, in order to estimate and compare the PVs subject to this design it is necessary to know the value of the prevalence of the disease. This value can be obtained from health surveys or from previous studies. Consequently, the methods of comparison of the PVs subject to a paired design cannot be applied when there is a case-control design. In Section 2, we study hypothesis tests to simultaneously compare the PVs of two BDTs subject to a case-control design. A global hypothesis test is studied to simultaneously compare the PVs of the two BDTs, i.e.,

$$H_0:\left(PP{V}_1=PP{V}_2\mathrm{and}NP{V}_1=NP{V}_2\right),$$

and simultaneous comparison is also studied from individual hypothesis tests, i.e.,

$$H_0:PP{V}_1=PP{V}_2\mathrm{and}{H}_0:NP{V}_1=NP{V}_2$$

each of them to the $\alpha$ error and also applying multiple comparison methods. In Section 3, simulation experiments are carried out to study the type I errors and the powers of the hypothesis tests proposed in Section 2, and we study the effect of the misspecification of the prevalence on the asymptotic behavior of the hypothesis tests proposed in Section 2 and on the estimators of the PVs. In Section 4, the results are applied to a real example on the diagnosis of coronary heart disease. In Section 5, the model proposed in Section 2 was extended to the situation in which we compare the PVs of more than two BDTs, and in Section 6 the results are discussed.

2. Global Hypothesis Test

Let us consider two BDTs, Test 1 and Test 2, which are applied to all of the individuals in two samples, one of $n_1$ individuals who have the disease (case sample) and another of $n_2$ individuals who do not have it (control sample). Let $T_1$ and $T_2$ be two binary variables that model the results of each BDT, in such a way that $T_i=1$ when the result of the corresponding BDT is positive and $T_i=0$ when it is negative. In

Table 1. Probabilities and observed frequencies subject to case-control design.

Probabilities
Case				Control
	$T_2=1$	$T_2=0$	Total		$T_2=1$	$T_2=0$	Total
$T_1=1$	${\xi }_{111}$	${\xi }_{110}$	$S{e}_1$	$T_1=1$	${\xi }_{211}$	${\xi }_{210}$	$1-S{p}_1$
$T_1=0$	${\xi }_{101}$	${\xi }_{100}$	$1-S{e}_1$	$T_1=0$	${\xi }_{201}$	${\xi }_{200}$	$S{p}_1$
Total	$S{e}_2$	$1-S{e}_2$	1	Total	$1-S{p}_2$	$S{p}_2$	1
Observed Frequencies
Case				Control
	$T_2=1$	$T_2=0$	Total		$T_2=1$	$T_2=0$	Total
$T_1=1$	$n_{111}$	$n_{110}$	$n_{11·}$	$T_1=1$	$n_{211}$	$n_{210}$	$n_{21·}$
$T_1=0$	$n_{101}$	$n_{100}$	$n_{10·}$	$T_1=0$	$n_{201}$	$n_{200}$	$n_{20·}$
Total	$n_{1·1}$	$n_{1·0}$	$n_1$	Total	$n_{2·1}$	$n_{2·0}$	$n_2$

, we can see the probabilities associated to the application of both BDTs to both types of individuals (cases and controls), as well as the frequencies observed.

Using the conditional dependence model of Vacek [10], the probabilities given in the table are written as:

$${\xi }_{1jk}=S{e}_1^j{\left(1-S{e}_1\right)}^{1-j}S{e}_2^k{\left(1-S{e}_2\right)}^{1-k}+{\delta }_{jk}{\epsilon }_1,$$

and

$${\xi }_{2jk}=S{p}_1^{1-j}{\left(1-S{p}_1\right)}^{j}S{p}_2^{1-k}{\left(1-S{p}_2\right)}^k+{\delta }_{jk}{\epsilon }_2,$$

with $j,k=0,1$. The parameter ${\epsilon }_1$ (${\epsilon }_2$) is the covariance between the two BDTs in cases (controls), where ${\delta }_{jk}=1$ if $j=k$ and ${\delta }_{jk}=-1$ if $j\ne k$, and it is verified that $0\le {\epsilon }_1\le \mathrm{Min}\left\{S{e}_1\left(1-S{e}_2\right),S{e}_2\left(1-S{e}_1\right)\right\}$ and $0\le {\epsilon }_2\le \mathrm{Min}\left\{S{p}_1\left(1-S{p}_2\right),S{p}_2\left(1-S{p}_1\right)\right\}$. If ${\epsilon }_i=0$ then the two BDTs are conditionally independent on the disease status. In practice, the assumption of the conditional independence is not realistic, and therefore ${\epsilon }_1>0$ and/or ${\epsilon }_2>0$. In terms of the probabilities ${\xi }_{ijk}$, the sensitivities are written as:

$$S{e}_1={\xi }_{111}+{\xi }_{110}\mathrm{and}S{e}_2={\xi }_{111}+{\xi }_{101},$$

and the specificities are written as:

$$S{p}_1={\xi }_{201}+{\xi }_{200}\mathrm{and}S{p}_2={\xi }_{210}+{\xi }_{200}.$$

The estimators of sensitivities are $\widehat{S}{e}_1=n_{11·}/n_1$ and $\widehat{S}{e}_2=n_{1·1}/n_1$, and the estimators of specificities are $\widehat{S}{p}_1=n_{20·}/n_2$ and $\widehat{S}{p}_2=n_{2·0}/n_1$. The estimators of their variances are $\widehat{V}ar\left(\widehat{S}{e}_1\right)=\widehat{S}{e}_1\left(1-\widehat{S}{e}_1\right)/n_1$, $\widehat{V}ar\left(\widehat{S}{e}_2\right)=\widehat{S}{e}_2\left(1-\widehat{S}{e}_2\right)/n_1$, $\widehat{V}ar\left(\widehat{S}{p}_1\right)=\widehat{S}{p}_1\left(1-\widehat{S}{p}_1\right)/n_2$ and $\widehat{V}ar\left(\widehat{S}{p}_2\right)=\widehat{S}{p}_2\left(1-\widehat{S}{p}_2\right)/n_2$. Therefore, the sensitivities and the specificities are estimated as proportions of marginal totals. In this way, in the case sample we are interested in the marginal frequencies $n_{11·}$ and $n_{1·1}$, and therefore these frequencies are the product of a type I bivariate binomial distribution [11]. In an analogous way, from the control sample, the marginal frequencies $n_{20·}$ and $n_{2·0}$ are the product of a type I bivariate binomial distribution. In the individuals with the disease, the type I bivariate binomial distribution is characterized [11] by $S{e}_1$, $S{e}_2$ and the correlation coefficient (${\rho }_1$) between $T_1$ and $T_2$. In an analogous way, in the individuals who do not have the disease, the type I bivariate binomial distribution is characterized by $S{p}_1$, $S{p}_2$ and the correlation coefficient (${\rho }_2$) between $T_1$ and $T_2$. Therefore, the proposed model is a parametric model based on the distribution of the marginal frequencies in each $2\times 2$ table. In the individuals with the disease (cases), the correlation coefficient between the two BDTs is:

$${\rho }_1=\frac{{\xi }_{111}-S{e}_{1}S{e}_2}{\sqrt{S{e}_1\left(1-S{e}_1\right)S{e}_2\left(1-S{e}_2\right)}}=\frac{{\epsilon }_1}{\sqrt{S{e}_1\left(1-S{e}_1\right)S{e}_2\left(1-S{e}_2\right)}},$$

and in the individuals who do not have the disease (controls), the correlation coefficient between the two BDTs is:

$${\rho }_2=\frac{{\xi }_{200}-S{p}_{1}S{p}_2}{\sqrt{S{p}_1\left(1-S{p}_1\right)S{p}_2\left(1-S{p}_2\right)}}=\frac{{\epsilon }_2}{\sqrt{S{p}_1\left(1-S{p}_1\right)S{p}_2\left(1-S{p}_2\right)}}.$$

It is easy to show that ${\widehat{\epsilon }}_1=\left(n_1{n}_{111}-n_{11·}{n}_{1·1}\right)/n_1^2$, ${\widehat{\epsilon }}_2=\left(n_2{n}_{200}-n_{20·}{n}_{2·0}\right)/n_2^2$, $\widehat{C}ov\left(\widehat{S}{e}_1,\widehat{S}{e}_2\right)={\widehat{\epsilon }}_1/n_1$ and $\widehat{C}ov\left(\widehat{S}{p}_1,\widehat{S}{p}_2\right)={\widehat{\epsilon }}_2/n_2$. All of the other covariances are zero, since the two samples are independent. The estimators of ${\rho }_1$ and ${\rho }_2$ are ${\widehat{\rho }}_1=\frac{n_1{n}_{111}-n_{11·}{n}_{1·1}}{\sqrt{n_{11·}\left(n_1-n_{11·}\right)n_{1·1}\left(n_1-n_{1·1}\right)}}$ and ${\widehat{\rho }}_2=\frac{n_2{n}_{200}-n_{20·}{n}_{2·0}}{\sqrt{n_{20·}\left(n_2-n_{20·}\right)n_{2·0}\left(n_2-n_{2·0}\right)}}$. Assuming that the disease prevalence p is known, the estimators of the predictive values are:

$$\widehat{PP{V}_1}=\frac{p{n}_2{n}_{11·}}{p{n}_2{n}_{11·}+q{n}_1\left(n_2-n_{20·}\right)}\mathrm{and}\widehat{NP{V}_1}=\frac{q{n}_1{n}_{20·}}{p{n}_2\left(n_1-n_{11·}\right)+q{n}_1{n}_{20·}},$$

for Test 1, and

$$\widehat{PP{V}_2}=\frac{p{n}_2{n}_{1·1}}{p{n}_2{n}_{1·1}+q{n}_1\left(n_2-n_{2·0}\right)}\mathrm{and}\widehat{NP{V}_2}=\frac{q{n}_1{n}_{2·0}}{p{n}_2\left(n_1-n_{1·1}\right)+q{n}_1{n}_{2·0}}$$

for Test 2, where $q=1-p$. Let the variance-covariance matrixes be defined as:

$${\sum }_{\widehat{S}e}=\left(\begin{array}{cc}Var\left(\widehat{S}{e}_1\right)& Cov\left(\widehat{S}{e}_1,\widehat{S}{e}_2\right)\\ Cov\left(\widehat{S}{e}_1,\widehat{S}{e}_2\right)& Var\left(\widehat{S}{e}_2\right)\end{array}\right)$$

and

$${\sum }_{\widehat{S}p}=\left(\begin{array}{cc}Var\left(\widehat{S}{p}_1\right)& Cov\left(\widehat{S}{p}_1,\widehat{S}{p}_2\right)\\ Cov\left(\widehat{S}{p}_1,\widehat{S}{p}_2\right)& Var\left(\widehat{S}{p}_2\right)\end{array}\right).$$

Let $\mathsf{\theta }={\left(S{e}_1,S{e}_2,S{p}_1,S{p}_2\right)}^T$ be a vector whose components are the sensitivities and the specificities, and let $\mathsf{\omega }={\left(PP{V}_1,PP{V}_2,NP{V}_1,NP{V}_2\right)}^T$ be a vector whose components are the PVs. The variance-covariance matrix of $\widehat{\mathsf{\theta }}$ is:

$${\sum }_{\widehat{\mathsf{\theta }}}=\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right)\otimes {\sum }_{\widehat{S}e}+\left(\begin{array}{cc}0& 0\\ 0& 1\end{array}\right)\otimes {\sum }_{\widehat{S}p},$$

(2)

where $\otimes$ is the Kronecker product. Applying the delta method, the matrix of variances- covariances of $\widehat{\mathsf{\omega }}$ is:

$${\sum }_{\widehat{\mathsf{\omega }}}=\left(\frac{\partial \mathsf{\omega }}{\partial \mathsf{\theta }}\right){\sum }_{\widehat{\mathsf{\theta }}}{\left(\frac{\partial \mathsf{\omega }}{\partial \mathsf{\theta }}\right)}^T.$$

(3)

Expressions of the variances-covariances of the PVs can be seen in Appendix A. The PVs of each BDT depend on the same parameters, the sensitivity and the specificity of the test and disease prevalence, and therefore they are related parameters. Consequently, the PVs of the two BDTs can be compared simultaneously. The global hypothesis test to simultaneously compare the PVs of the two BDTs is:

$$\begin{array}{c}{H}_0:PP{V}_1=PP{V}_2\mathrm{and}NP{V}_1=NP{V}_2\\ H_1:\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{equality}\mathrm{is}\mathrm{not}\mathrm{true},\end{array}$$

which is equivalent to the hypothesis test:

$$H_0:A\mathsf{\omega }=0\mathrm{vs}{H}_1:A\mathsf{\omega }\ne 0$$

(4)

where $A$ is a complete range matrix sized $2\times 4$ whose elements are known constants, i.e.:

$$A=\left(\begin{array}{cccc}1& -1& 0& 0\\ 0& 0& 1& -1\end{array}\right).$$

As the vector $\widehat{\mathsf{\omega }}$ is distributed asymptotically according to a multivariate normal distribution, i.e., $\sqrt{n_1+n_2}\left(\widehat{\mathsf{\omega }}-\mathsf{\omega }\right)\underset{n_1+n_2\to \infty }{\to }N\left(0,{\Sigma }_{\mathsf{\omega }}\right)$, then the test statistic for the global hypothesis test (4) is:

$$Q^2={\widehat{\mathsf{\omega }}}^T{A}^T{\left(A{\widehat{\sum }}_{\widehat{\mathsf{\omega }}}{A}^T\right)}^{-1}A\widehat{\mathsf{\omega }},$$

(5)

which is distributed asymptotically according to Hotelling’s T-squared distribution with a dimension 2 and $n_1+n_2$ degrees of freedom, where 2 is the dimension of the vector $A\widehat{\mathsf{\omega }}.$ When $n_1+n_2$ is large, the statistic $Q^2$ is approximately distributed according to a central chi-square distribution with 2 degrees of freedom when the null hypothesis is true.

On the other hand, the individual comparison of the positive (negative) predictive values is solved with the hypothesis test:

$$H_0:P{V}_1=P{V}_2\mathrm{vs}{H}_0:P{V}_1\ne P{V}_2,$$

where PV is PPV or NPV. Based on the asymptotic normality of the estimators, the test statistic for this hypothesis test is:

$$z=\frac{\widehat{P{V}_1}-\widehat{P{V}_2}}{\sqrt{\widehat{V}ar\left(\widehat{P{V}_1}\right)+\widehat{V}ar\left(\widehat{P{V}_2}\right)-2\widehat{C}ov\left(\widehat{P{V}_1},\widehat{P{V}_2}\right)}},$$

(6)

which is distributed asymptotically according to a normal standard distribution, and where the variances-covariances are obtained from the Equation (3) (see Appendix A).

The global hypothesis test $H_0:A\mathsf{\omega }=0$ simultaneously compares the PPVs and the NPVs of the two BDTs. Some alternative methods to this global hypothesis test, based on the individual hypothesis tests, are: (1) testing the hypotheses $H_0:PP{V}_1=PP{V}_2$ and $H_0:NP{V}_1=NP{V}_2$ (Equation (6)) each one to an $\alpha$ error; (2) testing the hypotheses $H_0:PP{V}_1=PP{V}_2$ and $H_0:NP{V}_1=NP{V}_2$ (Equation (6)) and applying a multiple comparison method such as the Bonferroni method [12] or the Holm method [13], which are methods that are very easy to apply based on the p-values. Bonferroni method [12] consists of solving each individual hypothesis test to an error equal to $\alpha /2$. The Holm method is a step-down method which is based on Bonferroni method but is more conservative. In Appendix B, the Holm method [13] is summarized.

3. Simulation Experiments

Simulation experiments were carried out to study the type I errors and the powers of the four methods proposed to simultaneously compare the predictive values: the global hypothesis test based on the chi-square distribution (Equation (5)), the individual hypothesis tests each one to an $\alpha$ error (Equation (6)), the individual hypothesis tests (Equation (6)) applying the Bonferroni method and the individual hypothesis tests (Equation (6)) applying the Holm method. We have also studied the effect of a misspecification of the prevalence on the asymptotic behavior of these methods and on the estimators of the PVs.

The experiments were designed setting the values of the PVs. For each BDT, we took as PVs the values 0.60, 0.65, …, 0.90, 0.95, and as disease prevalence we took the values 10%, 25% and 50%. Based on the PVs and the prevalence, Se and Sp of each BDT were calculated from the Equation (1), only considering those cases in which the solutions are between 0 and 1. As values of the correlation coefficients ${\rho }_1$ and ${\rho }_2$ we took low values (25% of the maximum value), intermediate (50% of the maximum value) and high (75% of the maximum value), where the maximum value of each correlation coefficient is: $\max \left({\rho }_1\right)=\frac{\min \left\{S{e}_1\left(1-S{e}_2\right),\left(1-S{e}_1\right)S{e}_2\right\}}{\sqrt{S{e}_1\left(1-S{e}_1\right)S{e}_2\left(1-S{e}_2\right)}}$ and $\max \left({\rho }_2\right)=\frac{\min \left\{S{p}_1\left(1-S{p}_2\right),\left(1-S{p}_1\right)S{p}_2\right\}}{\sqrt{S{p}_1\left(1-S{p}_1\right)S{p}_2\left(1-S{p}_2\right)}}$ respectively. As sample sizes, we took the values $n_i=\left(50,75,100,200,500\right)$. The simulation experiments were carried out with R [14], using the “bindata” package [15] to generate the samples of each type I bivariate binomial distribution.

Regarding the random samples, these were generated in the following way. Firstly, once the values of the PVs and of the prevalence were set, we calculated the sensitivities, the specificities and the maximum values of the coefficients ${\rho }_1$ and ${\rho }_2$. We then generated 10,000 random samples from a type I bivariate binomial distribution with a sample size $n_1$, probabilities $S{e}_1$ and $S{e}_2$, and correlation coefficient ${\rho }_1$. Similarly, we generated another 10,000 random samples from a type I bivariate binomial distribution with a sample size $n_2$, probabilities $S{p}_1$ and $S{p}_2$, and correlation coefficient ${\rho }_2$. In this way, we obtained the marginal frequencies $n_{11·}$ and $n_{1·1}$ ($n_{20·}$ and $n_{2·0}$) of each one of the 10,000 case (control) samples. The rest of the marginal frequencies were easily calculated: $n_{10·}=n_1-n_{11·}$, $n_{1·0}=n_1-n_{1·1}$, $n_{21·}=n_2-n_{20·}$ and $n_{2·1}=n_2-n_{2·0}$. In order to construct the $2\times 2$ table of each case sample, we generated a random value $n_{111}$ from a doubly truncated binomial distribution of parameters $n_1$ and ${\xi }_{111}=S{e}_{1}S{e}_2+{\epsilon }_1$, with $n_{11·}+n_{1·1}-n_1\le n_{111}\le \min \left(n_{11·},n_{11·}\right)$. This is necessary so that the sum of the frequencies leads to the marginal totals randomly generated through the type I bivariate binomial distribution. In the same way, in order to construct the $2\times 2$ table of each control sample, we generated a random value $n_{200}$ from a doubly truncated binomial distribution of parameters $n_2$ and ${\xi }_{200}=S{p}_{1}S{p}_2+{\epsilon }_2$, with $n_{20·}+n_{2·0}-n_2\le n_{200}\le \min \left(n_{20·},n_{2·0}\right)$. For each one of the 10,000 case (control) samples, once we have generated the values $n_{11·}$, $n_{1·1}$ and $n_{111}$ ($n_{20·}$, $n_{2·0}$ and $n_{200}$) it is easy to construct the complete $2\times 2$ table. Thus, $n_{110}=n_1-n_{11·}$, $n_{101}=n_{1·1}-n_{111}$ and $n_{100}=n_{10·}-n_{101}$ for the case samples, and $n_{201}=n_{20·}-n_{200}$, $n_{210}=n_{2·0}-n_{200}$ and $n_{211}=n_{21·}-n_{210}$ for the control samples. For the experiments $\alpha =5\%$ was set. Moreover, all of the samples were generated in such a way that in all of them the parameters and the variances-covariances can be estimated. If in a random sample it is obtained that $n_{i10}=n_{i01}=0$, with $i=1,2$, then $\widehat{S}{e}_i=\widehat{S}{p}_i=1$ and $\widehat{V}ar\left(\widehat{S}{e}_i\right)=\widehat{V}ar\left(\widehat{S}{p}_i\right)=0$, and therefore the test statistic $Q^2={\widehat{\mathsf{\omega }}}^T{A}^T{\left(A{\widehat{\sum }}_{\widehat{\mathsf{\omega }}}{A}^T\right)}^{-1}A\widehat{\mathsf{\omega }}$ cannot be calculated since $A{\widehat{\sum }}_{\widehat{\mathsf{\omega }}}{A}^T$ is a non-singular matrix. This problem occurs mainly when the sample size is small or moderate. In this situation, the sample has been discarded and another is generated in its place until the 10,000 samples are obtained.

3.1. Type I Errors and Powers

In

Table 2. Type I errors for $PP{V}_1=PP{V}_2=0.70$ and $NP{V}_1=NP{V}_2=0.95$.

$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.5385}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9744}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.5385}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9744}\mathbf{,}\mathit{p}=\mathbf{10}\mathbf{\%}$
		${\rho }_1=0.25 {\rho }_2=0.25$			${\rho }_1=0.50 {\rho }_2=0.50$			${\rho }_1=0.75 {\rho }_2=0.75$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.031	0.051	0.029	0.027	0.048	0.027	0.004	0.013	0.004
50	75	0.029	0.059	0.029	0.025	0.051	0.026	0.004	0.017	0.005
50	100	0.028	0.063	0.030	0.029	0.061	0.028	0.008	0.018	0.007
75	75	0.023	0.061	0.026	0.031	0.056	0.028	0.015	0.034	0.017
100	100	0.027	0.063	0.029	0.023	0.052	0.024	0.020	0.043	0.019
200	200	0.044	0.086	0.045	0.032	0.063	0.031	0.025	0.050	0.026
500	500	0.055	0.107	0.056	0.058	0.102	0.057	0.040	0.077	0.039
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.8615}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.8769}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.8615}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.8769}\mathbf{,}\mathit{p}=\mathbf{25}\mathbf{\%}$
		${\rho }_1=0.25 {\rho }_2=0.25$			${\rho }_1=0.50 {\rho }_2=0.50$			${\rho }_1=0.75 {\rho }_2=0.75$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.048	0.094	0.046	0.018	0.047	0.018	0.001	0.007	0.002
50	75	0.053	0.100	0.051	0.025	0.063	0.026	0.002	0.012	0.003
50	100	0.053	0.106	0.057	0.034	0.076	0.032	0.008	0.023	0.008
75	75	0.059	0.105	0.055	0.039	0.087	0.037	0.007	0.016	0.006
100	100	0.059	0.117	0.059	0.056	0.102	0.054	0.011	0.040	0.010
200	200	0.058	0.099	0.057	0.048	0.094	0.049	0.044	0.090	0.042
500	500	0.052	0.098	0.053	0.051	0.101	0.052	0.049	0.090	0.048
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.9692}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.5846}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.9692}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.5846}\mathbf{,}\mathit{p}=\mathbf{50}\mathbf{\%}$
		${\rho }_1=0.25 {\rho }_2=0.25$			${\rho }_1=0.50 {\rho }_2=0.50$			${\rho }_1=0.75 {\rho }_2=0.75$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.026	0.049	0.026	0.025	0.061	0.026	0.006	0.017	0.006
50	75	0.020	0.049	0.023	0.019	0.052	0.024	0.007	0.028	0.010
50	100	0.019	0.043	0.023	0.016	0.045	0.019	0.010	0.034	0.014
75	75	0.024	0.065	0.027	0.020	0.051	0.027	0.012	0.038	0.017
100	100	0.028	0.066	0.029	0.021	0.052	0.025	0.012	0.042	0.019
200	200	0.047	0.088	0.044	0.034	0.074	0.032	0.021	0.058	0.026
500	500	0.052	0.099	0.052	0.050	0.097	0.049	0.037	0.077	0.034

Global: global hypothesis test based on the chi-square distribution. $\alpha =5\%$: individual hypothesis tests each one to an error of 5%. Bonf.: Bonferroni method.

and

Table 3. Type I errors for $PP{V}_1=PP{V}_2=0.85$ and $NP{V}_1=NP{V}_2=0.95$.

$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.5312}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9896}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.5312}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9896}\mathbf{,}\mathit{p}=\mathbf{10}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{1}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{1}$
		${\rho }_1=0.25 {\rho }_2=0.25$			${\rho }_1=0.50 {\rho }_2=0.50$			${\rho }_1=0.75 {\rho }_2=0.75$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.033	0.056	0.034	0.020	0.051	0.024	0.004	0.014	0.004
50	75	0.024	0.049	0.024	0.026	0.050	0.025	0.005	0.019	0.006
50	100	0.032	0.057	0.033	0.030	0.056	0.030	0.004	0.016	0.004
75	75	0.034	0.054	0.033	0.025	0.052	0.026	0.014	0.036	0.015
100	100	0.027	0.055	0.026	0.027	0.055	0.026	0.017	0.041	0.017
200	200	0.033	0.059	0.031	0.025	0.050	0.024	0.022	0.055	0.021
500	500	0.046	0.087	0.049	0.031	0.068	0.033	0.018	0.050	0.024
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.85}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.95}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.85}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.95}\mathbf{,}\mathit{p}=\mathbf{25}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{1}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{1}$
		${\rho }_1=0.25 {\rho }_2=0.25$			${\rho }_1=0.50 {\rho }_2=0.50$			${\rho }_1=0.75 {\rho }_2=0.75$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.023	0.058	0.022	0.005	0.030	0.007	0.001	0.005	0.001
50	75	0.037	0.077	0.036	0.014	0.039	0.015	0.001	0.008	0.001
50	100	0.049	0.092	0.048	0.022	0.056	0.022	0.001	0.007	0.002
75	75	0.042	0.087	0.041	0.025	0.055	0.025	0.004	0.014	0.004
100	100	0.048	0.095	0.043	0.028	0.066	0.027	0.005	0.025	0.005
200	200	0.033	0.059	0.031	0.025	0.050	0.024	0.022	0.055	0.021
500	500	0.048	0.097	0.046	0.056	0.101	0.051	0.050	0.099	0.049
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.9562}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.8312}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.9562}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.8312}\mathbf{,}\mathit{p}=\mathbf{50}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{1}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{1}$
		${\rho }_1=0.25 {\rho }_2=0.25$			${\rho }_1=0.50 {\rho }_2=0.50$			${\rho }_1=0.75 {\rho }_2=0.75$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.031	0.072	0.031	0.014	0.041	0.015	0.001	0.007	0.001
50	75	0.032	0.069	0.033	0.022	0.049	0.022	0.005	0.015	0.005
50	100	0.025	0.057	0.026	0.025	0.064	0.026	0.008	0.025	0.008
75	75	0.038	0.081	0.037	0.027	0.054	0.025	0.006	0.017	0.006
100	100	0.039	0.084	0.038	0.031	0.073	0.030	0.008	0.030	0.009
200	200	0.033	0.059	0.031	0.025	0.050	0.024	0.022	0.055	0.021
500	500	0.051	0.099	0.049	0.050	0.097	0.047	0.043	0.087	0.042

Global: global hypothesis test based on the chi-square distribution. $\alpha =5\%$: individual hypothesis tests, each one to an error of 5%. Bonf.: Bonferroni method.

, we can see some results obtained for the type I errors of the global test and of the alternative methods proposed in Section 2. In these tables, we can only see the results for the global test, the individual comparisons with $\alpha =5\%$ and with the Bonferroni method. The results obtained with the Holm method are not shown as they are practically the same as those obtained with the Bonferroni method. From the results obtained we can draw the following conclusions. In general terms, the type I error of the global hypothesis test fluctuates around the nominal error, especially in the case of samples sized $n_i\ge 100$, depending on the prevalence and the correlations between the two BDTs. For samples with smaller sizes $\left(n_i\le 75\right)$, the type I error of the global test is lower than $\alpha =5\%$. The correlations between the two BDTs have an important effect on the type I error of the global test, with a decrease in the type I error when there is an increase in the correlation coefficients.

Regarding the method based on the individual hypothesis tests $H_0:PP{V}_1=PP{V}_2$ and $H_0:NP{V}_1=NP{V}_2$ to an error $\alpha =5\%$ each one of them, the type I error may clearly overwhelm the nominal error (a situation that we have considered when the type I error is greater than 7%), especially when the correlations are not high. Consequently, this method may lead to erroneous results (false significances) and, therefore, should not be used. As for solving the global test from the individual tests applying the Bonferroni (Holm) method, the type I error has a very similar behavior to that of the global hypothesis test.

Regarding the powers of the hypothesis tests, in

Table 4. Powers for $PP{V}_1=0.75$, $NP{V}_1=0.95$, $PP{V}_2=0.60$ and $NP{V}_2=0.95$.

$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.5357}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9802}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.5455}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9596}\mathbf{,}\mathit{p}=\mathbf{10}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9805}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.6933}$
		${\rho }_1=0.25 {\rho }_2=0.17$			${\rho }_1=0.49 {\rho }_2=0.35$			${\rho }_1=0.74 {\rho }_2=0.52$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.025	0.056	0.031	0.023	0.049	0.024	0.005	0.019	0.007
50	75	0.037	0.077	0.036	0.029	0.063	0.030	0.010	0.030	0.011
50	100	0.054	0.103	0.052	0.042	0.084	0.038	0.019	0.046	0.016
75	75	0.038	0.078	0.038	0.032	0.066	0.033	0.018	0.042	0.018
100	100	0.053	0.098	0.047	0.044	0.081	0.037	0.031	0.063	0.026
200	200	0.199	0.276	0.180	0.208	0.286	0.181	0.168	0.252	0.138
500	500	0.495	0.575	0.462	0.591	0.668	0.556	0.720	0.785	0.678
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.8571}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9048}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.8727}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.8061}\mathbf{,}\mathit{p}=\mathbf{25}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9354}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.6614}$
		${\rho }_1=0.23 {\rho }_2=0.17$			${\rho }_1=0.47 {\rho }_2=0.33$			${\rho }_1=0.70 {\rho }_2=0.50$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.259	0.335	0.230	0.254	0.345	0.230	0.195	0.334	0.210
50	75	0.409	0.496	0.378	0.454	0.543	0.424	0.467	0.606	0.470
50	100	0.505	0.584	0.462	0.598	0.677	0.556	0.683	0.776	0.675
75	75	0.416	0.498	0.382	0.469	0.557	0.436	0.501	0.608	0.476
100	100	0.528	0.606	0.488	0.625	0.699	0.579	0.718	0.793	0.685
200	200	0.822	0.862	0.790	0.891	0.923	0.873	0.974	0.983	0.964
500	500	0.996	0.999	0.996	1	1	1	1	1	1
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.9643}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.6786}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.9818}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.3455}\mathbf{,}\mathit{p}=\mathbf{50}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.7071}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.50}$
		${\rho }_1=0.18 {\rho }_2=0.13$			${\rho }_1=0.35 {\rho }_2=0.25$			${\rho }_1=0.53 {\rho }_2=0.38$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.890	0.939	0.893	0.935	0.969	0.941	0.977	0.989	0.978
50	75	0.978	0.990	0.977	0.995	0.997	0.993	0.999	0.999	0.999
50	100	0.995	0.998	0.995	0.999	0.998	0.999	1	1	1
75	75	0.984	0.992	0.983	0.995	0.999	0.994	0.999	1	0.999
100	100	0.998	0.999	0.998	1	1	0.999	1	1	1
200	200	1	1	1	1	1	1	1	1	1
500	500	1	1	1	1	1	1	1	1	1

Global: global hypothesis test based on the chi-square distribution. $\alpha =5\%$: individual hypothesis tests, each one to an error of 5%. Bonf.: Bonferroni method.

and

Table 5. Powers for $PP{V}_1=0.95$ $NP{V}_1=0.95$, $PP{V}_2=0.75$ and $NP{V}_2=0.95$.

$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.5278}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9969}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.5357}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9802}\mathbf{,}\mathit{p}=\mathbf{10}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9841}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.3910}$
		${\rho }_1=0.25 {\rho }_2=0.10$			${\rho }_1=0.49 {\rho }_2=0.19$			${\rho }_1=0.74 {\rho }_2=0.29$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.030	0.059	0.030	0.019	0.048	0.020	0.007	0.019	0.008
50	75	0.031	0.063	0.032	0.023	0.054	0.023	0.009	0.024	0.009
50	100	0.033	0.064	0.033	0.030	0.063	0.030	0.010	0.031	0.009
75	75	0.033	0.057	0.032	0.025	0.055	0.025	0.015	0.036	0.015
100	100	0.034	0.067	0.033	0.026	0.059	0.026	0.026	0.054	0.026
200	200	0.123	0.182	0.094	0.122	0.177	0.095	0.108	0.168	0.078
500	500	0.666	0.770	0.662	0.669	0.781	0.667	0.699	0.811	0.696
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.8444}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9852}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.8571}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9048}\mathbf{,}\mathit{p}=\mathbf{25}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9511}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.3779}$
		${\rho }_1=0.24 {\rho }_2=0.09$			${\rho }_1=0.48 {\rho }_2=0.19$			${\rho }_1=0.71 {\rho }_2=0.28$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.172	0.247	0.142	0.154	0.234	0.127	0.107	0.198	0.097
50	75	0.422	0.537	0.396	0.398	0.521	0.378	0.365	0.541	0.367
50	100	0.627	0.734	0.615	0.641	0.755	0.638	0.674	0.779	0.653
75	75	0.434	0.549	0.400	0.432	0.555	0.410	0.402	0.552	0.391
100	100	0.635	0.753	0.634	0.655	0.774	0.656	0.666	0.796	0.683
200	200	0.965	0.981	0.964	0.977	0.987	0.974	0.989	0.994	0.988
500	500	1	1	1	1	1	1	1	1	1
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.95}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.95}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.9643}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.6786}\mathbf{,}\mathit{p}=\mathbf{50}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}^{+}\le \mathbf{0.8388}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}^{-}\le \mathbf{0.3333}$
		${\rho }_1=0.21 {\rho }_2=0.08$			${\rho }_1=0.42 {\rho }_2=0.17$			${\rho }_1=0.63 {\rho }_2=0.25$
$n_1$	$n_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	0.929	0.969	0.942	0.954	0.983	0.966	0.965	0.992	0.978
50	75	0.994	0.998	0.995	0.999	0.999	0.999	0.999	1	0.999
50	100	1	1	1	1	1	1	1	1	1
75	75	0.995	0.998	0.995	0.997	0.999	0.998	1	1	1
100	100	1	1	1	1	1	1	1	1	1
200	200	1	1	1	1	1	1	1	1	1
500	500	1	1	1	1	1	1	1	1	1

Global: global hypothesis test based on the chi-square distribution. $\alpha =5\%$: individual hypothesis tests, each one to an error of 5%. Bonf.: Bonferroni method.

we can see some of the results obtained for the global test and other alternative methods. The results obtained with the Holm method are not shown as they are practically the same as those obtained with the Bonferroni method. The power of the global hypothesis test is calculated as the proportion of samples in which it is accepted that $PP{V}_1\ne PP{V}_2$ or $NP{V}_1\ne NP{V}_2$ (being true that $PP{V}_1\ne PP{V}_2$ or $NP{V}_1\ne NP{V}_2$). From the results, the following conclusions are obtained. The disease prevalence has an important effect on the power of each one of the methods to solve the global test, and the power increases with an increase in the prevalence. Regarding the correlations ${\rho }_1$ and ${\rho }_2$, these do not have a clear effect on the power, and the power increases sometimes and decreases other times when the correlations increase. In general terms, when the prevalence is small $\left(p=10\%\right)$ we need large samples $\left(n_i>500\right)$ so that the power of the global hypothesis test is greater than 80%; for a prevalence of 25% with sample sizes $n_i\ge 200$ we obtain a power greater than 80%; and for a very large prevalence $\left(p=50\%\right)$ with sample sizes $n_i\ge 50$ we obtain a very higher power, greater than 80%–90%, depending on the difference between the PVs.

The power of the method based on the individual hypothesis tests to an error $\alpha =5\%$ is greater than that of the global test based on the chi-square distribution due to the fact that its type I error is also greater. Regarding the hypothesis test based on the individual tests with the Bonferroni method, in general terms, its power is very similar to that of the global test when the sample sizes are large. When the sample sizes are small or moderate, in general terms and depending on prevalence and correlations, the power of the global test is slightly greater than that of the individual tests with the Bonferroni method. The same conclusions are obtained when the Holm method is applied (whose results are almost identical to those of the Bonferroni method).

The graphs in Figure 1 show the powers of the three methods when $PP{V}_1=0.90$, $NP{V}_1=\left(0.80,0.85,0.90,0.95\right)$, $PP{V}_2=0.85$ and $NP{V}_2=0.90$, for different sample sizes $n_1=n_2=\left(50,100,200\right)$, $p=\left(25\%,50\%\right)$ and values intermediate of the correlation coefficients. These graphs show that when $NP{V}_1$ varies and the rest of the PVs are constant, the powers decrease when the prevalence increases. Similarly, the graphs in Figure 2 show the powers of the three methods when $PP{V}_1=\left(0.80,0.85,0.90,0.95\right)$, $NP{V}_1=0.95$, $PP{V}_2=0.60$ and $NP{V}_2=0.95$, for different sample sizes $n_1=n_2=\left(50,100,200\right)$, $p=\left(10\%,25\%\right)$ and values intermediate of the correlation coefficients. These graphs show that when the $PP{V}_1$ varies and the rest of the PVs are constant, the power of each method increases when the prevalence increases.

As conclusions of the results obtained in the simulation experiments, the global hypothesis test based on the chi-square distribution behaves well in terms of the type I error (it does not overwhelm the nominal error of 5%), the same as the individual tests along with the Bonferroni (Holm) method. The method based on the individual tests to a global error $\alpha =5\%$ should not be used as it may clearly overwhelm the nominal error.

In the simulation experiments, the proportion of times that $PP{V}_1\ne PP{V}_2$ and that $NP{V}_1\ne NP{V}_2$ are correctly concluded has also been studied. This issue is of special interest when the alternative hypothesis of the global test is true, as it can be a valid method to investigate the causes of significance. The study was carried out by applying the individual hypothesis tests together with the Bonferroni (Holm) method. Individual tests to an $\alpha$ error have not been considered as they have a type I error that can exceed the nominal error. If it is verified that $P{V}_1\ne P{V}_2$, then this study is equivalent to studying the power of the individual test $H_0:P{V}_1=P{V}_2$ to an $\alpha /2$ error (since the Bonferroni method has been applied), where $P{V}_i$ is $PP{V}_i$ or $NP{V}_i$. If it is verified that $P{V}_1=P{V}_2$ then this study is equivalent to studying the type I error of the individual test $H_0:P{V}_1=P{V}_2$ to an $\alpha /2$ error. In the scenarios considered in Table 4 and Table 5 it is verified that $PP{V}_1\ne PP{V}_2$ and that $NP{V}_1=NP{V}_2$. Therefore, for these two scenarios, the power of the test $H_0:PP{V}_1=PP{V}_2$ and the type I error of the test $H_0:NP{V}_1=NP{V}_2$ have been studied, each with an error equal to $\alpha /2=2.5\%$.

Table 6. Power of the test $H_0:PP{V}_1=PP{V}_2$ and type I error of the test $H_0:NP{V}_1=NP{V}_2$ when $PP{V}_1=0.75$, $NP{V}_1=0.95$, $PP{V}_2=0.60$ and $NP{V}_2=0.95$.

$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.5357}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9802}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.5455}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9596}\mathbf{,}\mathit{p}=\mathbf{10}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9805}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.6933}$
		${\rho }_1=0.25 {\rho }_2=0.17$		${\rho }_1=0.49 {\rho }_2=0.35$		${\rho }_1=0.74 {\rho }_2=0.52$
$n_1$	$n_2$	Power	Type I error	Power	Type I error	Power	Type I error
50	50	0.001	0.030	0.001	0.024	0.001	0.007
50	75	0.007	0.029	0.004	0.027	0.002	0.001
50	100	0.021	0.029	0.013	0.027	0.008	0.009
75	75	0.008	0.031	0.005	0.030	0.003	0.018
100	100	0.026	0.026	0.017	0.026	0.010	0.021
200	200	0.160	0.024	0.159	0.028	0.119	0.024
500	500	0.449	0.026	0.544	0.023	0.672	0.020
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.8571}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9048}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.8727}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.8061}\mathbf{,}\mathit{p}=\mathbf{25}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9354}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.6614}$
		${\rho }_1=0.23 {\rho }_2=0.17$		${\rho }_1=0.47 {\rho }_2=0.33$		${\rho }_1=0.70 {\rho }_2=0.50$
$n_1$	$n_2$	Power	Type I error	Power	Type I error	Power	Type I error
50	50	0.211	0.025	0.223	0.011	0.210	0.002
50	75	0.346	0.021	0.420	0.008	0.470	0.002
50	100	0.448	0.026	0.551	0.010	0.675	0.001
75	75	0.352	0.030	0.435	0.020	0.479	0.005
100	100	0.472	0.027	0.559	0.025	0.683	0.010
200	200	0.785	0.023	0.870	0.027	0.961	0.019
500	500	0.996	0.024	0.999	0.026	1	0.027
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.9643}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.6786}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.9818}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.3455}\mathbf{,}\mathit{p}=\mathbf{50}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.7071}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.50}$
		${\rho }_1=0.18 {\rho }_2=0.13$		${\rho }_1=0.35 {\rho }_2=0.25$		${\rho }_1=0.53 {\rho }_2=0.38$
$n_1$	$n_2$	Power	Type I error	Power	Type I error	Power	Type I error
50	50	0.893	0.003	0.941	0.002	0.980	0.001
50	75	0.977	0.003	0.991	0.002	0.999	0.001
50	100	0.990	0.002	0.999	0.002	1	0.001
75	75	0.983	0.005	0.993	0.002	1	0.003
100	100	0.997	0.005	1	0.004	1	0.002
200	200	1	0.012	1	0.006	1	0.007
500	500	1	0.024	1	0.024	1	0.019

and

Table 7. Power of the test $H_0:PP{V}_1=PP{V}_2$ and type I error of the test $H_0:NP{V}_1=NP{V}_2$ $PP{V}_1=0.95$, $NP{V}_1=0.95$, $PP{V}_2=0.75$ and $NP{V}_2=0.95$.

$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.5278}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9969}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.5357}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9802}\mathbf{,}\mathit{p}=\mathbf{10}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9841}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.3910}$
		${\rho }_1=0.25 {\rho }_2=0.10$		${\rho }_1=0.49 {\rho }_2=0.35$		${\rho }_1=0.74 {\rho }_2=0.29$
$n_1$	$n_2$	Power	Type I error	Power	Type I error	Power	Type I error
50	50	0.001	0.030	0.001	0.024	0	0.006
50	75	0.002	0.032	0.002	0.027	0.001	0.008
50	100	0.005	0.031	0.003	0.027	0.002	0.008
75	75	0.002	0.032	0.001	0.026	0.001	0.017
100	100	0.010	0.030	0.006	0.022	0.005	0.023
200	200	0.071	0.027	0.068	0.028	0.056	0.022
500	500	0.654	0.025	0.678	0.028	0.693	0.025
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.8444}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.9852}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.8571}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.9048}\mathbf{,}\mathit{p}=25\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.9511}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.3779}$
		${\rho }_1=0.24 {\rho }_2=0.09$		${\rho }_1=0.48 {\rho }_2=0.19$		${\rho }_1=0.71 {\rho }_2=0.28$
$n_1$	$n_2$	Power	Type I error	Power	Type I error	Power	Type I error
50	50	0.120	0.025	0.118	0.012	0.097	0.001
50	75	0.378	0.028	0.371	0.012	0.376	0.001
50	100	0.604	0.026	0.634	0.012	0.652	0.001
75	75	0.382	0.027	0.396	0.022	0.388	0.005
100	100	0.622	0.031	0.644	0.026	0.679	0.012
200	200	0.963	0.025	0.974	0.024	0.987	0.026
500	500	1	0.028	1	0.024	1	0.024
$\mathit{S}{\mathit{e}}_{\mathbf{1}}=\mathbf{0.95}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{1}}=\mathbf{0.95}\mathbf{,}\mathit{S}{\mathit{e}}_{\mathbf{2}}=\mathbf{0.9643}\mathbf{,}\mathit{S}{\mathit{p}}_{\mathbf{2}}=\mathbf{0.6786}\mathbf{,}\mathit{p}=\mathbf{50}\mathbf{\%}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{1}}\le \mathbf{0.8388}\mathbf{,}\mathbf{0}\le {\mathit{\rho }}_{\mathbf{2}}\le \mathbf{0.3333}$
		${\rho }_1=0.21 {\rho }_2=0.08$		${\rho }_1=0.42 {\rho }_2=0.17$		${\rho }_1=0.63 {\rho }_2=0.25$
$n_1$	$n_2$	Power	Type I error	Power	Type I error	Power	Type I error
50	50	0.942	0.002	0.965	0.001	0.978	0
50	75	0.995	0.003	0.999	0	0.997	0
50	100	1	0.002	1	0.002	1	0
75	75	0.996	0.006	0.997	0.003	0.999	0
100	100	1	0.010	1	0.006	1	0.002
200	200	1	0.029	1	0.015	1	0.010
500	500	1	0.026	1	0.024	1	0.024

show the results obtained applying the Bonferroni method. The results obtained with the Holm method are not shown as they are practically the same as those obtained with the Bonferroni method.

In general terms, the hypothesis test $H_0:PP{V}_1=PP{V}_2$ has a high power when the sample sizes are moderate or high, depending on the prevalence and the correlation coefficients. Its behavior is very similar to that of the global hypothesis test. With respect to the test $H_0:NP{V}_1=NP{V}_2$, its type I error fluctuates around the nominal error (2.5%) when the sample sizes are moderate or large, depending on the prevalence of the correlation coefficients. In general terms, the hypothesis tests $H_0:PP{V}_1=PP{V}_2$ and $H_0:NP{V}_1=NP{V}_2$ have a good asymptotic behavior, both in terms of power and type I error.

From the results obtained in the simulation experiments, we propose the following method to compare the PVs of two BDTs subject to a case-control design: (1) Applying the global hypothesis test based on the chi-square distribution (Equation (5)) to an $\alpha$ error; (2) If the global hypothesis test is not significant, the equality hypothesis of the PVs is not rejected; if the global hypothesis test is significant to an $\alpha$ error, the investigation of the causes of the significance is made by testing the individual tests (Equation (6)) and applying the Bonferroni method or the Holm method to an $\alpha$ error. Therefore, if the global test is significant, the investigation of the significance consists in solving the individual hypothesis tests $H_0:PP{V}_1=PP{V}_2$ and $H_0:NP{V}_1=NP{V}_2$, each of them to an $\alpha /2$ error (Bonferroni method) or applying Holm method.

This method to simultaneously compare the PVs is very similar to other methods used in other statistical models, such as the analysis of variance: first the global test is resolved to an $\alpha$ error, and if it is significant then the causes of significance are investigated from pairwise comparisons and the application of a multiple comparison method.

3.2. Effect of the Prevalence

The estimation and comparison of the PVs of two BDTs subject to a case-control design requires knowledge of the disease prevalence. To study the effect of a misspecification of the prevalence on the comparison of the PVs and on the estimators of the PVs, we carried out simulation experiments similar to those made to study the type I errors and the powers. For this purpose, we took as the prevalence for the inference a misspecification equal to 10% and to 20% of the value of the prevalence set, and we have studied the type I errors and the powers of the global test and of the Bonferroni and Holm methods, and the relative root mean square error (RRMSE) of the estimator of each PVs. Thus, for each estimator we calculated the relative root mean square error (RRMSE), i.e.,

$$RRMSE\left({\widehat{PV}}_i\right)=\frac{\sqrt{\frac{1}{N}{\displaystyle \sum _{k=1}^N{\left({\widehat{PV}}_{ik}-P{V}_i\right)}^2}}}{P{V}_i},$$

where $P{V}_i$ is the PPV or the NPV of the ith BDT $\left(i=1,2\right)$ and ${\widehat{PV}}_{ik}$ is its estimator calculated from the kth sample $\left(k=1,\dots ,N\right)$, and $N=10,000$. For the values of the parameters we took as prevalence $p=\left(10\%,25\%,50\%\right)$ respectively, and to estimate the PVs we took as prevalence $p^{\prime }=p\pm d\times p$ with $d=\left(10\%,20\%\right)$. A value $d=10\%$ $\left(d=20\%\right)$ can be considered as a small (moderate) value of the relative deviation.

Table 8. Effect of a misspecification of the prevalence.

Type I errors
$\begin{array}{c}PP{V}_1=PP{V}_2=0.90,NP{V}_1=NP{V}_2=0.80\\ S{e}_1=0.2571,S{p}_1=0.9905,S{e}_2=0.2571,S{p}_2=0.9905,p=25\%,{\rho }_1=0.75,{\rho }_2=0.75\end{array}$
		$p{}^{\prime }=p=25\%$		$p{}^{\prime }=20\%$		$p{}^{\prime }=22.50\%$		$p{}^{\prime }=27.50\%$		$p{}^{\prime }=30\%$
$n_1$	$n_2$	Global	Bonf.	Global	Bonf.	Global	Bonf.	Global	Bonf.	Global	Bonf.
50	50	0.001	0.001	0.001	0.002	0.001	0.002	0.001	0.001	0.001	0.001
50	75	0.001	0.002	0.001	0.002	0.001	0.002	0.001	0.002	0.001	0.002
50	100	0.002	0.003	0.002	0.003	0.002	0.003	0.002	0.003	0.002	0.003
75	75	0.003	0.007	0.003	0.008	0.003	0.008	0.003	0.007	0.003	0.007
100	100	0.006	0.010	0.006	0.011	0.006	0.011	0.006	0.009	0.006	0.009
200	200	0.010	0.020	0.010	0.020	0.010	0.020	0.010	0.019	0.010	0.019
500	500	0.020	0.024	0.020	0.024	0.020	0.024	0.020	0.023	0.020	0.023
Powers
$\begin{array}{c}PP{V}_1=0.95,PP{V}_2=0.75,NP{V}_1=0.95,NP{V}_2=0.95\\ S{e}_1=0.8444,S{p}_1=0.9852,S{e}_2=0.8571,S{p}_2=0.9048,p=25\%,{\rho }_1=0.71,{\rho }_2=0.28\end{array}$
		$p{}^{\prime }=p=25\%$		$p{}^{\prime }=20\%$		$p{}^{\prime }=22.50\%$		$p{}^{\prime }=27.50\%$		$p{}^{\prime }=30\%$
$n_1$	$n_2$	Global	Bonf.	Global	Bonf.	Global	Bonf.	Global	Bonf.	Global	Bonf.
50	50	0.172	0.142	0.145	0.139	0.143	0.135	0.142	0.121	0.139	0.117
50	75	0.422	0.396	0.381	0.382	0.385	0.382	0.374	0.380	0.375	0.379
50	100	0.647	0.615	0.600	0.587	0.627	0.604	0.616	0.614	0.598	0.596
75	75	0.434	0.400	0.414	0.396	0.415	0.396	0.397	0.394	0.390	0.390
100	100	0.635	0.634	0.618	0.604	0.632	0.612	0.639	0.620	0.623	0.615
200	200	0.965	0.964	0.941	0.931	0.951	0.952	0.958	0.956	0.942	0.933
500	500	1	1	1	1	1	1	1	1	1	1
RRMSEs of the estimators of PVs of Test 1
$\begin{array}{c}PP{V}_1=PP{V}_2=0.90,NP{V}_1=NP{V}_2=0.80\\ S{e}_1=0.2571,S{p}_1=0.9905,S{e}_2=0.2571,S{p}_2=0.9905,p=25\%,{\rho }_1=0.75,{\rho }_2=0.75\end{array}$
		$p{}^{\prime }=p=25\%$		$p{}^{\prime }=20\%$		$p{}^{\prime }=22.50\%$		$p{}^{\prime }=27.50\%$		$p{}^{\prime }=30\%$
$n_1$	$n_2$	${\widehat{PPV}}_i$	${\widehat{NPV}}_i$	${\widehat{PPV}}_i$	${\widehat{NPV}}_i$	${\widehat{PPV}}_i$	${\widehat{NPV}}_i$	${\widehat{PPV}}_i$	${\widehat{NPV}}_i$	${\widehat{PPV}}_i$	${\widehat{NPV}}_i$
50	50	27.4	1.8	36.6	5.5	31.8	2.6	29.3	3.8	34.5	6.4
50	75	20.5	1.7	27.0	5.1	23.5	2.8	21.9	3.6	25.5	6.1
50	100	16.1	1.7	21.9	5.2	18.8	2.9	17.8	3.5	21.1	6.0
75	75	20.1	1.4	26.5	5.1	23.1	2.6	21.4	3.5	24.1	6.1
100	100	15.4	1.2	19.2	4.9	18.1	2.5	16.1	3.3	18.6	5.9
200	200	8.0	0.9	11.7	4.7	9.9	2.4	8.7	3.0	10.2	5.6
500	500	4.4	0.5	7.1	4.1	5.4	2.2	4.9	2.8	5.7	5.5

Global: global hypothesis test based on the chi-square distribution. Bonf.: Bonferroni method.

shows some of the results obtained for the type I errors and the powers of the global test and the Bonferroni method (the results of the Holms method are not shown as they are practically identically to those obtained with the Bonferroni method). In this Table we show the results when there is no misspecification of the prevalence $\left(p^{\prime }=p\right)$ and when there is a misspecification of the prevalence ($p^{\prime }<p$ or $p^{\prime }>p$). From the results of these experiments, it is verified that the type I errors of the methods studied do not overwhelm the nominal error ($\alpha =5\%$). In general terms there are no important differences between the type I errors when there is a misspecification of the prevalence and when there is not. Regarding the powers, the conclusions are also very similar, there are no important differences between the powers when there is a misspecification of the prevalence and when there is not.

Regarding the estimators, Table 8 shows some of the results obtained for the RRMSEs (in %) of the estimators of the PVs of Test 1 (the results for Test 2 are identical). In general terms, the difference between the RRMSEs is small (around 5% or less, in absolute value) when the two sample sizes are moderate $\left(n_i=100\right)$ or large $\left(n_i\ge 200\right)$ and the relative deviation is small (10%) or moderate (20%). Therefore, a small or moderate misspecification of the prevalence ($p^{\prime }<p$ or $p^{\prime }>p$) does not have an important effect on the estimators of the PVs when the samples are moderate or large. Additionally, there is not an important difference between the RRMSEs when the sample sizes are small $\left(n_i\le 75\right)$ and the relative deviation is small. However, the difference between the RRMSEs is larger when the sample sizes are small and the relative deviation is moderate. In this situation, a misspecification of the prevalence has an important effect on the estimators of the PVs.

4. Example

The results obtained have been applied to the diagnosis of coronary heart disease, using an electrocardiogram and an echocardiography as diagnostic tests. Both tests have been applied to a sample of 105 older men with coronary heart disease (case sample) and to another sample of 120 older men without this disease (control sample). In

Table 9. Diagnosis of coronary heart disease.

Observed Frequencies
Case				Control
	$T_2=1$	$T_2=0$	Total		$T_2=1$	$T_2=0$	Total
$T_1=1$	77	10	87	$T_1=1$	4	2	6
$T_1=0$	6	12	18	$T_1=0$	13	101	114
Total	83	22	105	Total	17	103	120

we can see the frequencies obtained, where the random variable $T_1$ models the result of the electrocardiogram and the variable $T_2$ models the result of the echocardiography. In order to illustrate the method proposed, we are going to consider that the prevalence of the disease in older men is 5%. The objective is to compare the clinical accuracy (PVs) of both BDTs in the population whose prevalence of coronary heart disease is 5%. The comparison of the PVs will be made with $\alpha =5\%$.

From the case sample, the estimates of the two sensitivities (and their standard errors, SE) and of the correlation coefficient between them are $\widehat{S}{e}_1\pm SE=0.829\pm 0.037$, $\widehat{S}{e}_1\pm SE=0.790\pm 0.040$ and ${\widehat{\rho }}_1=0.511$. From the control sample, the estimates of the two specificities and of the correlation coefficient between them are $\widehat{S}{p}_1\pm SE=0.950\pm 0.020$, $\widehat{S}{e}_1\pm SE=0.858\pm 0.032$ and ${\widehat{\rho }}_2=0.345$. Assuming that the prevalence of coronary heart disease is 5%, the estimates of the PVs are:

$$\begin{array}{l}{\widehat{PPV}}_1=\frac{0.05\times 0.829}{0.05\times 0.829+\left(1-0.05\right)\times \left(1-0.950\right)}=0.466,\\ {\widehat{PPV}}_2=\frac{0.05\times 0.790}{0.05\times 0.790+\left(1-0.05\right)\times \left(1-0.858\right)}=0.227,\\ {\widehat{NPV}}_1=\frac{\left(1-0.05\right)\times 0.950}{0.05\times \left(1-0.829\right)+\left(1-0.05\right)\times 0.950}=0.991\end{array}$$

and

$${\widehat{NPV}}_2=\frac{\left(1-0.05\right)\times 0.858}{0.05\times \left(1-0.790\right)+\left(1-0.05\right)\times 0.858}=0.987.$$

Applying the delta method (Equation (3)), the estimated variance-covariance matrix of the estimators of the PVs is:

$${\widehat{\sum }}_{\widehat{\mathsf{\omega }}}=\left(\begin{array}{cccc}0.009926& 0.001398& 0.000041& 0.000029\\ 0.001398& 0.001632& 0.000012& 0.000039\\ 0.000041& 0.000012& 0.000004& 0.000002\\ 0.000029& 0.000039& 0.000002& 0.000006\end{array}\right).$$

Applying Equation (5), the value of the test statistic for the global test:

$$\begin{array}{c}{H}_0:PP{V}_1=PP{V}_2\mathrm{and}NP{V}_1=NP{V}_2\\ H_1:\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{equality}\mathrm{is}\mathrm{not}\mathrm{true},\end{array}$$

is $Q^2=7.516$ and the p-value is 0.023, and therefore the null hypothesis of the global test is rejected. Testing the individual hypothesis tests it is found that the value of the test statistic for $H_0:PP{V}_1=PP{V}_2$ is equal to 2.552 (two sided p-value is 0.011) and that the value of the test statistic for $H_0:NP{V}_1=NP{V}_2$ is equal to 1.469 (two sided p-value is 0.142). Applying the Bonferroni (or Holm) method, the hypothesis of equality of the positive predictive values is rejected and the hypothesis of equality of the negative predictive values is not rejected. Therefore, in a population in which the prevalence of coronary heart disease is 5%, the positive predictive value of electrocardiogram is significantly greater than that of the echocardiography (95% confidence interval for the difference: 0.056 to 0.422), while there are no significant differences between the two negative predictive values.

5. More Than Two BDTs

Let us consider that J BDTs $\left(J\ge 3\right)$ are applied to all of the individuals in the case sample and the control sample. For each BDT we define the random variable $T_j$ in a similar way to how this was done in Section 2. Let $S{e}_j$ and $S{p}_j$ be the sensitivity and the specificity of the jth BDT, with $j=1,\dots ,J$. Let $n_{1i_1\dots i_J}$ be the number of individuals with the disease for whom $T_1=i_1$,…, $T_J=i_J$, with $i_j=1$ when the result of the jth BDT is positive and $i_j=0$ when it is negative. In a similar way, $n_{2i_1\dots i_J}$ is the number of without the disease for whom $T_1=i_1$,…, $T_J=i_J$. Let us consider the probabilities ${\xi }_{h{i}_1,\dots ,i_J}=P\left(T_1=i_1,T_2=i_2,\dots ,T_J=i_J\right)$, with $h=1,2$. Thus, for example for three BDTs, using the dependence model of Torrance–Rynard and Walter [16], these probabilities are:

$${\xi }_{1i_1{i}_2{i}_3}={\displaystyle \prod _{j=1}^{3}S{e}_j^{i_j}{\left(1-S{e}_j\right)}^{1-i_j}}+{\displaystyle \sum _{j,k,j<k}^3{\left(-1\right)}^{\left|i_j-i_k\right|}{\epsilon }_{1jk}}$$

and

$${\xi }_{2i_1{i}_2{i}_3}={\displaystyle \prod _{j=1}^{3}S{p}_j^{1-i_j}{\left(1-S{p}_j\right)}^{i_j}}+{\displaystyle \sum _{j,k,j<k}^3{\left(-1\right)}^{\left|i_j-i_k\right|}{\epsilon }_{2jk}},$$

with $i_j=0,1$, $i_k=0,1$ and $j,k=1,2,3$, and where ${\epsilon }_{1jk}$ (${\epsilon }_{2jk}$) is the covariance between the jth BDT and the kth BDT for individuals with the disease (without the disease). The estimators of these probabilities are ${\widehat{\xi }}_{h{i}_1\dots i_J}=n_{h{i}_1\dots i_J}/n_h$, with $h=1,2$. The sensitivity and the specificity of the jth BDT are:

$$S{e}_j={\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=1\end{array}}^1{\xi }_{1i_1,\dots ,i_J}}\mathrm{and}S{p}_j={\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=0\end{array}}^1{\xi }_{2i_1,\dots ,i_J}},$$

and its estimators are:

$$\widehat{S}{e}_j=\frac{{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=1\end{array}}^1{n}_{1i_1,\dots ,i_J}}}{n_1}\mathrm{and}\widehat{S}{p}_j=\frac{{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=0\end{array}}^1{n}_{2i_1,\dots ,i_J}}}{n_2}.$$

The estimators of the variances-covariances of these estimators are $\widehat{V}ar\left(\widehat{S}{e}_j\right)=\widehat{S}{e}_j\left(1-\widehat{S}{e}_j\right)/n_1$, $\widehat{V}ar\left(\widehat{S}{p}_j\right)=\widehat{S}{p}_j\left(1-\widehat{S}{p}_j\right)/n_2$, $\widehat{C}ov\left(\widehat{S}{e}_j,\widehat{S}{e}_k\right)={\widehat{\epsilon }}_{1jk}/n_1$ and $\widehat{C}ov\left(\widehat{S}{p}_j,\widehat{S}{p}_k\right)={\widehat{\epsilon }}_{2jk}/n_2$, and the rest of the covariances are equal to zero. The estimators of the PVs of the jth BDT are:

$$\widehat{PP{V}_j}=\frac{p{n}_2{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=1\end{array}}^1{n}_{1i_1,\dots ,i_J}}}{p{n}_2{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=1\end{array}}^1{n}_{1i_1,\dots ,i_J}}+q{n}_1{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=1\end{array}}^1{n}_{2i_1,\dots ,i_J}}}$$

and

$$\widehat{NP{V}_j}=\frac{q{n}_1{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=0\end{array}}^1{n}_{2i_1,\dots ,i_J}}}{q{n}_1{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=0\end{array}}^1{n}_{2i_1,\dots ,i_J}}+p{n}_2{\displaystyle \sum _{\begin{array}{c}{i}_1,\dots ,i_J=0\\ i_j=0\end{array}}^1{n}_{1i_1,\dots ,i_J}}},$$

where p is the disease prevalence and $q=1-p$.

Let $\mathsf{\theta }={\left(S{e}_1,\dots ,S{e}_J,S{p}_1,\dots ,S{p}_J\right)}^T$ be the vector whose components are the sensitivities and the specificities, and let $\mathsf{\omega }={\left(PP{V}_1,\dots ,PP{V}_J,NP{V}_1,\dots ,NP{V}_J\right)}^T$ be the vector whose components are the PVs. The variance-covariance matrix of the vector $\widehat{\mathsf{\theta }}$, with a dimension $2J\times 2J$, is similar to that given in Equation (2), where ${\sum }_{\widehat{S}e}$ and ${\sum }_{\widehat{S}p}$ are matrixes with a dimension $J\times J$. Applying the delta method, the variance-covariance matrix of $\widehat{\mathsf{\omega }}$, with a dimension $2J\times 2J$, has an expression similar to that given in Equation (3). The PVs of each one of the J BDTs depend on the same parameters (the sensitivity and the specificity of the jth diagnostic test) and, therefore, these parameters can be compared simultaneously. The global hypothesis test to simultaneously compare the PVs of the J BDTs is:

$$\begin{array}{c}{H}_0:PP{V}_1=PP{V}_2=\dots =PP{V}_J\mathrm{and}NP{V}_1=NP{V}_2=\dots =NP{V}_J\\ H_1:\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{equality}\mathrm{is}\mathrm{not}\mathrm{true},\end{array}$$

which is equivalent to the hypothesis test:

$$H_0:A\mathsf{\omega }=0\mathrm{vs}{H}_1:A\mathsf{\omega }\ne 0,$$

where $A$ is a matrix with a dimension $2\left(J-1\right)\times 2J$, i.e.,

$$A=\left(\begin{array}{cc}{A}_1& A_0\\ A_0& A_1\end{array}\right).$$

$A_0$ is a matrix with a dimension $\left(J-1\right)\times J$ whose elements are all equal to 0, and $A_1$ is a matrix with a dimension $\left(J-1\right)\times J$ where each component $\left(i,i\right)$ is equal to 1, each element $\left(i,i+1\right)$ is equal to $-1$ for $i=1,\dots ,J-1$, and the rest of the elements in this matrix are equal to 0. Applying the multivariate central limit theorem it is verified that $\sqrt{n_1+n_2}\left(\widehat{\mathsf{\omega }}-\mathsf{\omega }\right)\underset{n_1+n_2\to \infty }{\to }{N}_{2J}\left(0,{\Sigma }_{\mathsf{\omega }}\right)$. Then, the statistic $Q^2={\left(\widehat{\mathsf{\omega }}A\right)}^T{\left(A{\displaystyle {\widehat{\sum }}_{\widehat{\mathsf{\omega }}}}{A}^T\right)}^{-1}A\widehat{\mathsf{\omega }}$ is distributed according to Hotelling’s T-squared distribution with a dimension $2\left(J-1\right)$ and $n_1+n_2$ degrees of freedom, where $2\left(J-1\right)$ is the dimension of the vector $A\widehat{\mathsf{\omega }}$. When $n_1+n_2$ is large, the statistic $Q^2$ is distributed according to a central chi-squared distribution with $2\left(J-1\right)$ degrees of freedom when the null hypothesis is true, i.e.,

$$Q^2={\left(\widehat{\mathsf{\omega }}A\right)}^T{\left(A{\widehat{\sum }}_{\widehat{\mathsf{\omega }}}{A}^T\right)}^{-1}A\widehat{\mathsf{\omega }}\underset{n\to \infty }{\to }{\chi }_{2\left(J-1\right)}^2.$$

Finally, the method to compare the PVs of the J BDTs would consist of the following steps: (1) Solve the global hypothesis test to an $\alpha$ error calculating the statistic $Q^2={\left(\widehat{\mathsf{\omega }}A\right)}^T{\left(A{\displaystyle {\widehat{\sum }}_{\widehat{\mathsf{\omega }}}}{A}^T\right)}^{-1}A\widehat{\mathsf{\omega }}$ based on the chi-squared distribution; (2) if the global test is not significant to an $\alpha$ error then we do not reject the homogeneity of the J PVs, but if the hypothesis test is significant then the causes of significance are investigated comparing the PPVs (NPVs) in pairs (Equation (6)) and applying a multiple comparison method (e.g., Bonferroni or Holm).

6. Discussion

The comparison of the positive and negative predictive values of two binary diagnostic tests is an important topic in the study of statistical methods in diagnostic medicine. Subject to a paired design, this topic has been subject to different studies. In this article we studied the simultaneous comparison of the predictive values of two diagnostic tests subject to a case-control design, analyzing and comparing several methods. These methods consisted of a global test based on the chi-square distribution, a method based on the individual comparisons each one to an $\alpha$ error, and other two methods based on individual comparisons along with a multiple comparison method. The multiple comparison methods that were used were the Bonferroni method and the Holm method, which are methods based on the p-values of the individual hypothesis tests and are very easy to apply. The methods studied to compare the predictive values require knowing the prevalence of the disease. The prevalence can be known from other studies, such as population studies of health services, cohort studies, etc. If the researcher has a great uncertainty about the value of the prevalence, the problem can be solved by using several values for the prevalence and then analyzing and comparing the results obtained.

Simulation experiments were carried out to study the type I errors and the powers of the four methods proposed. These experiments were based on the generation of random samples with type I bivariate binomial distributions, which are the distributions that are inherent to case-control design, since proportions of marginal totals are estimated from these samples. The results have shown that the global hypothesis test based on the chi-square distribution behaves well in terms of type I error, and does not overwhelm the nominal error. Regarding its power, in general this strongly depends on the disease prevalence, and it is necessary to have very large samples when the prevalence is small and relatively small sample sizes when the prevalence is high, so that the power will be high.

Based on the results of the simulation experiments, a method has been proposed to compare the predictive values of two diagnostic tests subject to a case-control design. This method, which is similar to that proposed by Roldán-Nofuentes et al. [8], consists of the following steps: (1) Simultaneously comparing the predictive values applying the global hypothesis test based on the chi-square distribution to an $\alpha$ error; (2) if the global hypothesis test is not significant, then the equality hypothesis of the PVs is not rejected. If the global hypothesis test is significant to an $\alpha$ error, then the causes of the significance are studied solving the individual hypothesis tests and applying the Bonferroni method or the Holm method to an $\alpha$ error. This procedure that we propose is similar to the analysis of variance: firstly, the global test is solved and, if this is significant, then the causes of the significance are studied starting with paired comparisons along with some multiple comparison method.

Simulation experiments were carried out to study the effect of a misspecification of the prevalence in the asymptotic behavior of the global hypothesis test based on the chi-square distribution and on the methods based on multiple comparisons. In general terms, we can conclude that a small or moderate misspecification of the prevalence do not have an important effect on the behavior of these hypothesis tests, especially when the sample sizes are moderate or large.

The global hypothesis test was extended to the situation in which we simultaneously compare the PVs of more than two BDTs, and for this we propose a method which is similar to that proposed for two BDTs. To be able to calculate the global test statistic $Q^2={\left(\widehat{\mathsf{\omega }}A\right)}^T{\left(A{\displaystyle {\widehat{\sum }}_{\widehat{\mathsf{\omega }}}}{A}^T\right)}^{-1}A\widehat{\mathsf{\omega }}$ it is necessary that $A{\displaystyle {\widehat{\sum }}_{\widehat{\mathsf{\omega }}}}{A}^T$ to be non-singular. For two BDTs, this matrix is non-singular when it is verified that $n_{i10}+n_{i01}>0$, with $i=1,2$. If $n_{i10}=n_{i01}=0$ then the method proposed to compare the PVs cannot be applied A solution to this problem consists in adding the value 0.5 to all the observed frequencies (the sample size increases by two units), a very frequent solution in the analysis of $2\times 2$ tables. Simulation experiments have been carried out to study the type I errors and the powers of the hypothesis tests proposed in Section 2, using this solution when in a sample it is verified that $n_{i10}=n_{i01}=0$. These experiments have been designed in a similar way to those performed in Section 3.

Table 10. Type I errors and powers when 0.5 is added to the samples in which $n_{i10}=n_{i01}=0$.

Type I Errors
$\begin{array}{c}PP{V}_1=PP{V}_2=0.70,NP{V}_1=NP{V}_2=0.95\\ S{e}_1=0.5385,S{p}_1=0.9744,S{e}_2=0.5385,S{p}_2=0.9744,p=10\%\end{array}$
				${\rho }_1=0.25{\rho }_2=0.25$			${\rho }_1=0.50{\rho }_2=0.50$			${\rho }_1=0.75{\rho }_2=0.75$
$n_1$	$n_2$	${\overline{P}}_1$	${\overline{P}}_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	3.3	78.1	0.028	0.065	0.030	0.023	0.056	0.025	0.012	0.040	0.015
50	75	3.1	63.7	0.038	0.075	0.039	0.026	0.060	0.026	0.012	0.041	0.014
50	100	2.7	51.9	0.047	0.096	0.046	0.035	0.074	0.037	0.011	0.041	0.015
75	75	0.4	64.3	0.037	0.074	0.037	0.026	0.062	0.027	0.015	0.045	0.019
100	100	0.1	51.8	0.040	0.085	0.038	0.032	0.090	0.034	0.021	0.048	0.023
200	200	0	22.7	0.061	0.104	0.060	0.045	0.088	0.042	0.024	0.055	0.025
500	500	0	2.8	0.052	0.099	0.0524	0.056	0.101	0.048	0.044	0.094	0.045
Powers
$\begin{array}{c}PP{V}_1=0.75,PP{V}_2=0.60,NP{V}_1=0.95,NP{V}_2=0.95\\ S{e}_1=0.8571,S{p}_1=0.9048,S{e}_2=0.8727,S{p}_2=0.8061,p=25\%,0\le {\rho }_1\le 0.9354,0\le {\rho }_2\le 0.6614\end{array}$
				${\rho }_1=0.25{\rho }_2=0.25$			${\rho }_1=0.50{\rho }_2=0.50$			${\rho }_1=0.75{\rho }_2=0.75$
$n_1$	$n_2$	${\overline{P}}_1$	${\overline{P}}_2$	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.	Global	$\alpha =5\%$	Bonf.
50	50	14.8	18.1	0.297	0.369	0.269	0.342	0.427	0.306	0.369	0.484	0.340
50	75	14.5	9.6	0.417	0.501	0.379	0.492	0.576	0.449	0.604	0.688	0.568
50	100	14.2	5.5	0.515	0.590	0.476	0.629	0.693	0.576	0.739	0.798	0.704
75	75	5.8	9.7	0.424	0.506	0.392	0.502	0.584	0.456	0.622	0.705	0.579
100	100	2.5	5.5	0.525	0.603	0.486	0.624	0.693	0.583	0.761	0.819	0.728
200	200	0.1	0.7	0.817	0.863	0.786	0.910	0.930	0.876	0.975	0.983	0.965
500	500	0	0	0.996	0.999	0.997	1	1	1	1	1	1

${\overline{P}}_1$: average proportion (in %) of case samples in which 0.5 has been added. ${\overline{P}}_2$: average proportion (in %) of control samples in which 0.5 has been added. Global: global hypothesis test based on the chi-square distribution. $\alpha =5\%$: individual hypothesis tests, each one to an error of 5%. Bonf.: Bonferroni method.

shows some results (type I errors and powers) for some of the scenarios considered in Section 3, as well as the average proportions (of the three correlation scenarios) of case (control) samples in which the value 0.5 has been added. Obviously, the proportion of samples in which the value 0.5 has been added is greater when the sample size is small. In general terms, the conclusions are the same as those obtained in the simulation experiments presented in Section 3, although the powers of all hypothesis tests are slightly lower when the sample sizes are small. Therefore, adding 0.5 to all the observed frequencies of a sample in which $n_{i10}=n_{i01}=0$ is an adequate solution to be able to apply the PVs comparison method.