Predictive Power Analysis of Multiple Test Procedures Under Arbitrary Dependence

Abstract

Many statistical problems can be addressed by applying a multiple testing procedure (MTP) that controls either the Family-Wise Error Rate (FWER) or False Discovery Rate (FDR) under unknown arbitrarily interdependent p-values, without explicitly modeling these inter-correlations. They include the FWER-controlling Bonferroni MTP and Holm MTP; the FDR-controlling Benjamini-Yekutieli MTP; and the DP-MTP, based on a Dirichlet process (DP) prior distribution supporting the entire space of MTPs that control either the FWER or the FDR. For such an MTP, this study introduces a new and congenial method for Bayesian predictive power analysis, for power calculation and sample size determination for any given planned future (e.g., replication or interim) study. This novel MTP predictive power analysis method is based on a joint prior distribution defining a scale matrix mixture of asymmetric multivariate normal mean-variance mixture distributions, factorized as a general prior distribution for effect sizes (e.g., obtained from expert judgment or results of prior studies), and a uniform prior distribution for correlation matrices representing arbitrary dependencies between p-values of test statistics of given multiple hypothesis tests under their alternative hypotheses. The new MTP power analysis method also results in p-value weights which can be used to minimize the relative impacts of and assess for significance-chasing biases (e.g., publication bias, p-hacking, etc.) in multiple testing, without needing to assume that p-values (effect sizes) are independent. Previous MTP power analysis methods are conditional in that they assume fixed effect sizes and a fixed correlation matrix for test statistics (p-values) which may be difficult to specify; while, incongenially, not fully accounting for their uncertainty, especially for MTPs that control either the FWER or FDR for arbitrarily correlated p-values. The new simulation-based MTP predictive power analysis method is illustrated through the analysis of p-values obtained by a famous study of lead exposure and re-analyzed by the previous MTP literature, using R package bnpMTP (v1.0).

1. Introduction

Multiple hypothesis testing problems continue to arise in many fields of the natural (life, physical), social (e.g., psychology, education), and the formal sciences, especially the bio- and statistical sciences where various hypothesis testing and multiple testing procedures (MTPs) have been developed over decades.

A hypothesis testing procedure is based on a test statistic, typically defined as a ratio of an effect size to its standard error (for example, refs. [1,2]). Examples of effect size include a raw or standardized mean difference; correlation or regression coefficient; and (log-)odds ratio; etc.; each of which can be converted from one metric to another [3]. Under the given null hypothesis ($H_0$) tested (against the corresponding alternative hypothesis $H_1$, respectively), such a test statistic follows a central-zero-mean (non-central, resp.) t-distribution or normal distribution (e.g., asymptotically for large samples) under the null hypothesis tested, perhaps after transforming another test statistic that follows a central (non-central, resp.) F, ${\chi }_{\nu }^2$, or other standard distribution under $H_0$ ($H_1$, resp.) [4]. A p-value is a standardized deterministic transformation of the given test statistic onto the $[0,1]$ interval ([5], p. 17), such that the smaller the p, the more decisively the given null hypothesis $H_0$ is rejected ([6], p. 31). The p-value ($p\left(x\right)$), obtained from the test statistic computed from the given sample data set $x\equiv x_n$ of size n, is declared as “significant”, i.e., rejecting the given null $H_0$, if $p\le \alpha$, based on a small specified fixed Type I error probability ($\alpha$) of incorrectly rejecting $H_0$ over imaginary datasets $x_n^{*}$ randomly sampled from a population data-generating process where $H_0$ is true. Meanwhile, p-values provide a common “bottom line” language for the communication of statistical results that are still very often communicated by scientific journals, despite concerns [7]. The p-value is basic to statistical inference in that it can be used to compute the upper bound of the Bayes factor (for example, refs. [8,9]) and hence can provide approximate Bayesian computation.

The conditional statistical power ($1-\overline{\beta }$) of a null hypothesis testing procedure is the probability that it will detect an effect size ($\theta$) to reject the given null $H_0$, given specified $\theta$, n, $\alpha$, and the Type II error probability ($\overline{\beta }$) of incorrectly not rejecting $H_0$ over imaginary datasets $x_n^{*}$ randomly sampled from a population data-generating process where $H_1$ is true. The conditional power of virtually any classical ${\chi }_{\nu }^2$-, F, z-, or t-test procedure can be calculated by a closed-form formula (for example, ref. [1]), which can be easily rearranged to solve for any of the four quantities (power, n, $\theta$, and $\alpha$) given the other three. Power analysis is used to calculate the power ($1-\overline{\beta }$) of a test procedure, or the minimum data sample size needed to achieve some minimum desired power (e.g., $0.80$) for a planned future (e.g., replication or interim) study, given anticipated or estimated $\theta$ and $\alpha$; and less often, to evaluate the sensitivity of studies, or to make decisions about criteria of statistical significance [10]. However, conditional power analysis assumes a specified fixed effect size ($\theta$) without accounting for its uncertainty. This issue can be addressed theoretically by predictive power analysis ([11,12,13], Section 6.5), which averages the conditional power over a specified prior distribution representing available knowledge for the effect size (for example, refs. [14,15,16,17]).

Widely available software packages now easily enable common statistical analyses of data to routinely output results of typically correlated p-values from multiple hypothesis tests (often, on many variables), either as the main data analysis goal, and/or as automatic by-products of other main statistical modeling objectives. Also, an MTP for marginal p-values can reduce the results of different (e.g., t, F, ${\chi }^2$, z, Wilcoxon, and/or log-rank, etc.) test statistics to a common interpretable p-value scale, while conveniently, not requiring the statistician to assume nor to specify and estimate an explicit model for the potentially complex joint distributions of the test statistics, having typically unknown correlations. All things considered, it is no wonder why MTPs for marginal p-values are popular and relevant in applied statistics ([18], and references therein), which are easily usable and computable, while not requiring direct access to the original data, so that such an MTP can be readily used for meta-analysis.

Given that there are very many MTPs, and that p-values in applied statistics are typically dependent (correlated), we henceforth focus on the subset of MTPs for marginal p-values, where each MTP provides either Family-Wise Error Rate (FWER) or False Discovery Rate (FDR) control under unknown arbitrary dependencies between p-values, even when the given observed p-values arise from a highly heterogeneous mix of different hypothesis testing procedures, and without requiring direct access to the original data. They include the un/weighted: Bonferroni [19] MTP (B-MTP) and Holm [20] MTP (H-MTP), each of which control the FWER; the Benjamini & Yekutieli [21] MTP (BY-MTP), which controls the FDR; and the Dirichlet process MTP (DP-MTP) [22], defined by a DP prior distribution [23] which supports the entire space of MTPs controlling either the FWER or the FDR under arbitrary dependencies between p-values. A weighted version of the B-MTP, H-MTP, BY-MTP, or DP-MTP is defined by importance weights assigned to respective p-values and null hypotheses [18,22,24,25]. All these MTPs are reviewed in Section 2.

MTP power analysis methods have a shorter history and focus on conditional instead of predictive power. For any MTP for testing a given set of $m\ge 1$ hypothesis with FWER or FDR control, power can be defined by any one of the following four ways, with respect to the m-variate distribution of their test statistics under the alternative hypotheses (null hypothesis, resp.) for all m tests (resp.) with $m\times m$ correlation matrix. The marginal (individual) power of each test of hypotheses $H_{0,j}$ (for $j=1,\dots ,m$) is the marginal probability that the corresponding p-value leads to a rejection under level $\alpha$ by the MTP, with respect to the marginal distribution of the corresponding jth test statistic under the alternative hypotheses [26,27]. The average power is the un/weighted average over these m marginal probabilities (resp.) under the alternative hypotheses (weights sum to m over all m hypotheses tested) [24,28,29,30]; which relates to the criterion of expected number of rejected hypotheses [31]. The disjunctive ($d(=r)=1$/minimal/any-pair) power is the probability that at least one false null hypotheses is rejected at level $\alpha$ by the given MTP [28,32,33,34,35]. Finally, the conjunctive (complete/all-pairs) power is the probability that all false null hypotheses are rejected at level $\alpha$ by the MTP [26,28,34].

For modern applied statistics which routinely outputs multiple p-values, MTP power analysis is more relevant than power analysis of a single test procedure. A power analysis can be used to calculate the given MTP’s power, or the data sample size required to achieve some minimum desired power for the MTP for a planned future (e.g., replication or interim) study, and for the other mentioned uses of power analysis for a single test. For some FWER-controlling MTPs applied to up to three hypotheses, explicit formulas are available to compute (conditional) conjunctive power and disjunctive power [32]. Simulation methods can be used to calculate the power(s) for any FWER- or FDR-controlling MTP (for example, ref. [26]).

For any MTP that controls either the FWER or FDR for m given hypothesis tests, optimal weights for respective p-values (hypothesis tests) can be determined through maximization of a defined power function(s) of interest (e.g., disjunctive, conjunctive, or marginal power, etc.) based on the specified marginal power for each test ([36,37], and references therein). In principle, an MTP that accounts for arbitrary dependence between p-values, while incorporating optimized weights for the respective p-values, provides an MTP analysis that relatively down-weights the impacts of any present significance-chasing biases (due to publication bias and/or p-hacking, etc.) while accounting for any heterogeneous effects, and without needing to assume independence among p-values (effect sizes) as done in traditional meta-analysis [38]. This is because the significance-chasing bias level of any null hypothesis test’s binary-valued result (1 = significant, $p\le \alpha$; or 0 = non-significant, $p>\alpha$) can be measured by the binary result’s distance from the test’s marginal power probability, given the estimated or anticipated effect size of the test, as done by the ${\chi }^2$ test of excess significance [39], while other tests and models of publication bias can have limited power and be difficult to use [39,40,41]. As an aside, while the test of excess significance is based on a ${\chi }^2$ type distance, in principle, any other distance measure from the general class of power divergence statistics [42] can be used, such as the Hellinger distance. While the ${\chi }^2$ test of excess significance aims to detect the presence of significance-chasing biases instead of correcting for them, one key insight of this established bias testing procedure is that the marginal power of each of the m hypothesis tests of a given multiple hypothesis testing application provide standards by which to address significance-chasing biases. To elaborate, a significant result of a null hypothesis test with low marginal power is more likely to be a result of significance-chasing biases, compared to a significant result of a null hypothesis test with high marginal power which is more likely to be a result of a true effect or signal and less likely to be a result of bias. It is less likely because such a high powered test already makes it easier for the data analyst to detect a true effect or signal and thus lowers the need for the analyst to engage in significance-chasing bias behavior. Therefore, assigning weights to p-values as increasing functions of their respective marginal powers would produce results of multiple hypothesis testing that would place greater relative weight to p-values that are less susceptible to significance-chasing biases.

For a given weighted multiple testing problem, optimal p-value weights can be found using any one of the available optimization algorithms that maximize any of the mentioned criteria of MTP power. See the Appendix A for a more detailed review (based on [37]). However, while each of the available optimization methods are based on a clear objective, they are not fully satisfactory, as they are limited to Bonferroni, graphical or chain MTPs; while numerically optimizing disjunctive or conjunctive power can be computationally costly when the number of hypothesis tests (m) is large, it can yield non-unique p-value weights under equal (specified) marginal powers and correlated test statistics; it can be difficult to determine or interpret (e.g., numerically optimizing for weighted or average power is easier, but involves introducing another set of weights for power that may be difficult to determine or interpret); and it employs only conditional power, in that each method assumes that both the effect sizes and the correlation matrix of the m-variate distribution of the test statistics are fixed and known.

More ideally, a predictive MTP power analysis can be used, which would assign a prior distribution on both of these multivariate quantities in order to account for their uncertainty. Indeed, for specialized multiple testing problems employing m hypothesis tests, the $m\times m$ correlation matrix of the test statistics can be directly derived or calculated, either from uncorrelated test statistics (p-values) or orthogonal designs or contrasts; permutation- or bootstrap-based estimates of test statistics; or for pairwise mean comparisons done via a Dunnett or Tukey–Kramer type MTP (see the Appendix A for a more detailed review). However, more broadly speaking, in the modern data analysis era where MTP inferences need to be made from multiple p-values which often result from applications of highly heterogeneous combinations of different hypothesis testing procedures (e.g., not only for group mean comparisons), it can be very challenging to derive explicit equations or a sharp prior distribution about the correlations among corresponding test statistics, especially when the number (m) of hypothesis tests is large. Moreover, as mentioned, this paper focuses on MTPs for marginal p-values, which each accounts for arbitrary dependencies between p-values while not needing to model or estimate these correlations directly from data.

Further, while each of the available MTP power analysis methods provides only conditional power analysis, i.e., based on fixed effect sizes and a fixed specified correlation matrix, such a method is uncongenial with any MTP (e.g., B-MTP, H-MTP, or DP-MTP) that controls either the FWER or FDR under arbitrary dependence (correlations) between the p-values (test statistics). This is because such an MTP accounts for all possible correlation matrices of the test statistics, not only one. While it is tempting to view equation- and optimization-based methods for computing MTP power as more expediently convenient and precise compared to simulation methods, such an optimization method typically needs to assume a fixed correlation matrix in order to be computationally tractable. A simulation-based MTP power analysis can flexibly model more complex random correlation structures beyond a fixed correlation matrix.

Based on all the above considerations and the current state of the related MTP power analysis literature, we propose the first MTP predictive power analysis method, a straightforward simulation-based method of predictive power analysis for any MTP that provides FWER or FDR control under arbitrary-dependent p-values. This new MTP power analysis method, for any m given hypothesis tests, is based on a joint prior distribution for the parameters of a newly developed m-variate (upper) non-central t distribution, defined by an $m\times 1$ location vector, $m\times m$ scale matrix (mainly determined by an underlying correlation matrix), and a vector of m degrees of freedom parameters, for the distribution of the $m\ge 1$ test statistics under their alternative hypotheses. The first part of this joint prior is defined by a prior distribution for the location vector representing the respective ratios of m effect sizes to their respective standard errors, which can represent either: prior subjective clinical judgment information; be centered on the estimated effect sizes from a previous study; or more generally by a power prior, defined as the posterior distribution of the effect sizes from prior historical data [43] for, e.g., a future replication (or interim) study. The second part of the joint prior is defined by a uniform prior distribution for the underlying $m\times m$ correlation matrix for the m test statistics under their respective alternative hypotheses, which can fully account for uncertainty and typical lack of prior information about this matrix, and ensure a congenial power analysis of the given MTP that controls FWER or FDR under arbitrarily dependent p-values (test statistics). Therefore, the new MTP power analysis method is based on a scale matrix mixture of asymmetric multivariate normal mean-variance mixtures of the test statistics under the alternative hypotheses of all the given multiple hypothesis tests (resp.). Indeed, this is the first paper that provides a method of MTP power analysis for arbitrary-correlated p-values, which can rapidly compute power results using an efficient algorithm [44] for sampling the uniform distribution of correlation matrices.

In Bayesian statistical inference, the prior distribution must reflect the current state of knowledge and prior information about model parameters available to the data analyst. When there is a lack of prior information about model parameters, then Bayesian theory implies that a coherent choice of prior distribution is an “objective” non-informative prior distribution for the correlation matrix. Meanwhile, most priors used in statistical applications are objective instead of subjective, in part, because subjective elicitation is too difficult to be done even in a limited way for more than a few unknown parameters in the given statistical problem, so that such unknown parameters must be handled via objective Bayesian methods ([45,46], p. 216). Correspondingly, for the current context of predictive MTP power analysis, the “objective” uniform prior distribution on the correlation matrices often honestly reflects the statistician’s available prior knowledge in many MTP statistical situations. This is because of the fact that in such common situations, the MTP statistician only has access to the p-values of the m hypothesis tests, without any access to the raw data used to compute these p-values, as in traditional meta-analysis. This thus leaves the statistician without the ability to estimate this dependence structure and to elicit a more informative prior distribution, and renders the ”objective” uniform prior over all valid correlation matrices as a reasonable choice prior in such a situation because it would honestly represent the available prior information. Indeed, such common MTP data analysis scenarios are in large part what makes MTPs that control either the FWER or FDR under arbitrarily dependent p-values quite relevant and useful ([21], for example), including the B-MTP, H-MTP, BY-MTP, and DP-MTP. While it is arguable that the uniform prior may not meaningfully capture the more likely dependence structures among test statistics, such as equi-correlated or AR(1) correlation structures, more informative priors can be difficult to determine in more common MTP scenarios. The uniform prior is not “wrong” because it assigns non-zero probability to all arbitrary valid correlations structures for test statistics supported by the B-MTP, H-MTP, BY-MTP, and DP-MTP.

This new MTP predictive power analysis method: (1) determines estimates of the marginal powers of the m given hypothesis tests (resp.), which then can either be used for: (2) sample size determination for each of these tests given desired minimum marginal powers; (3)–(4) the other two mentioned purposes of power analysis of a single testing procedure; (5) constructing normalized weights of the corresponding m p-values (resp.) while optimizing the power of the MTP, and down-weighting the impact of significance-chasing biases in the given MTP analysis; and/or (6) comparing with observed p-values to provide an assessment of such biases for each individual p-value using Hellinger distance, in a similar spirit of the original seminal ${\chi }^2$ test of excess significance (but here, not using a measure of bias over all m hypothesis tests or ”studies” based on ${\chi }^2$ type distance). It is well-known in the MTP field that a weighted MTP, which gives relatively higher weights to the subset of tested null hypotheses that are likely to be false, tends to have higher statistical power compared to unweighted multiple testing procedures, for various correlation structures according to extensive simulation studies (e.g., ref. [47]). Further, the MTP predictive power analysis method achieves all these six objectives without needing to assume independence of the m p-values, if the power-analyzed MTP controls for the FWER or FDR under arbitrarily dependent p-values.

Next, Section 2 reviews MTPs that each strongly control FWER and/or FDR under arbitrarily dependent p-values, in order to contextualize the description of the new predictive MTP power analysis method for such an MTP in Section 3. Then, Section 4 illustrates the new predictive MTP power analysis method to evaluate the power of the DP-MTP, and of the B-MTP and H-MTP for comparison purposes, through the MTP analysis of p-values arising from 41 hypothesis tests reported by a famous study on the neuropsychologic effects of unidentified childhood exposure to lead. Section 5 proves a theorem on the convergence and convergence speed of the algorithm. Section 6 concludes by summarizing the main ideas of the paper.

Central to the new predictive MTP power analysis method is Algorithm 1, presented in Section 3, which can be used to simulate draws of samples from the prior predictive distribution of the joint prior mentioned above, in order to yield estimates of the various measures of powers of the m test statistics, respectively, and the associated statistics mentioned above. This includes the corresponding m marginal powers, being well-defined concepts in the MTP literature [26,27] that can be used for estimating the Hellinger distance measures of significant-chasing bias, and for constructing p-value weights that can be used in a subsequent weighted MTP analysis. Meanwhile, the univariate marginal distributions of a prior predictive distribution are unique given a specifically defined prior and sampling distribution (likelihood), implying that the corresponding marginal powers and associated functions of them are uniquely defined, including the associated Hellinger distance bias measures and p-value weights, all while accounting for the uncertainty for the parameters underlying MTP power analysis by assigning them a joint prior distribution which averages conditional power over this prior. As mentioned earlier, this is unlike some of the previous approaches to MTP power analysis, all based on conditional power, which can yield non-unique p-value weights. While Algorithm 1 by default is based on the uniform prior on all valid correlation matrices for the m given test statistics, the algorithm can be easily modified to implement more informative prior, such as priors supporting equi-correlated or AR(1) correlation structures.

2. Review of MTPs Valid Under Arbitrary Dependence

Here, a formal multiple hypothesis testing framework [48] is reviewed, useful for MTP inference [22]. Consider the probability space, $(\mathcal{X},\mathfrak{X},P)$, with probability function P a member of a set $\mathcal{P}$ of distributions representing a parametric or non-parametric model. A null hypothesis, $H_0$, is a subset (submodel), $H_0\subset \mathcal{P}$, of distributions on $(\mathcal{X},\mathfrak{X})$, where $P\in H_0$ denotes that P satisfies $H_0$, and $\mathcal{X}$ is the sample space of any observable dataset sampled from the distribution P, with $\mathfrak{X}$ the corresponding sigma algebra of events. Any application of multiple testing, performed on a sample dataset $x\sim P\in \mathcal{P}$, aims to determine whether P satisfies distinct null hypotheses, belonging to a certain set (family) $\mathcal{H}$ of candidate null hypotheses, with this set being typically countable, or even uncountable.

For any given sampled dataset, $x\sim P\in \mathcal{P}$, a multiple testing procedure (MTP) is a decision function $\mathcal{R}:x\in \mathcal{X}\mapsto \mathcal{R}\left(x\right)\subset \mathcal{H}$ that returns the subset of rejected null hypotheses; ${\mathcal{R}}^c\left(x\right)=\mathcal{H}-\mathcal{R}\left(x\right)$ is the subset of non-rejected hypotheses; and the MTP $\mathcal{R}\left(x\right)$ commits a Type I error if $H_0\in \mathcal{R}\left(x\right)\cap {\mathcal{H}}_0\left(P\right)$; and commits a Type II error if $H_0\notin \mathcal{R}\left(x\right)\cap {\mathcal{H}}_1\left(P\right)$, where ${\mathcal{H}}_1\left(P\right)$ is the subset of the alternative hypotheses that are true under the given true data-generating distribution P. In a typical multiple testing application, $\mathcal{H}$ is a finite or countable set of m null hypotheses, $\mathcal{H}=\{H_{0,1},\dots ,H_{0,m}\}$, tested respectively against given alternative hypotheses ${\left\{H_{1,j}\right\}}_{j=1}^m$, with $m=\left|\mathcal{H}\right|\in {\mathbb{Z}}^{+}$ the number of candidate null hypotheses, $m_0\left(P\right)=\left|{\mathcal{H}}_0\left(P\right)\right|\le m$ true null hypotheses, $m_1\left(P\right)=\left|{\mathcal{H}}_1\left(P\right)\right|=m-m_0\le m$ (truly) false null hypotheses, and ${\pi }_0=m_0/m$, under any $P\in \mathcal{P}$. For simplicity and with no loss of generality, it is henceforth assumed that $\mathcal{H}$ is countable, while the subsequently described hypothesis testing-related ideas can easily be extended to continuous hypothesis testing after adopting some more complex notation.

A typical MTP is a function $\mathcal{R}\left(\mathbf{p}\right)$ of a family of p-values, $\mathbf{p}=(p_j,H_{0,j}\in \mathcal{H})$, where each $p_j$-value measures how probable are the observed data x, given that the null hypothesis $H_{0,j}$ is true ([5], p. 19, Definition 2.1). Assume that for each null hypothesis $H_{0,j}\in \mathcal{H}$ there exists a p-value function, $p_j:\mathcal{X}\to [0,1]$, having a marginally super-uniform probability distribution ($\mathbb{P}$) when $H_{0,j}$ is true, i.e.,

$${\mathbb{P}}_{X\sim P}[p_j\left(X\right)\le t]\le t,\mathrm{for}\forall P\in \mathcal{P},\forall H_{0,j}\in {\mathcal{H}}_0\left(P\right),\mathrm{and}\forall t\in [0,1],$$

(1)

with strict equality (${\mathbb{P}}_{X\sim P}[p_H\left(X\right)\le t]=t$ for all $t\in [0,1]$) when the $p_j$-value is calibrated, i.e., uniform $\mathcal{U}[0,1]$ distributed under the null hypothesis $H_{0,j}$, which can be achieved (or improved on) using any of the available p-value calibration methods, e.g., for a test of a discrete model or a composite null hypothesis; or for model checking ([5,49,50,51,52], and references therein).

Thus, the assumption (1) for each p-value does not require a calibrated p-value for MTPs considered in the paper, since the $\mathcal{U}[0,1]$ distribution is a special case of the super-uniform distribution, even though a calibrated p-value would allow for a universal interpretation of the p-value, e.g., based on Fisher’s scale of evidence for p-values ([6], p. 31). The p-values analyzed in the case study illustration in Section 4 are calibrated asymptotically. Indeed, it is possible to define p-values only via the super-uniformity property (1) ([53], Definition 8.3.26), which does not make specific requirements on the distribution of the underlying test statistic under the null hypothesis. Though, the predictive MTP power analysis method introduced in Section 3 requires the specification of an explicit functional relationship between each p-value and its test statistic, between the joint distribution of test statistics and corresponding p-values for any multiple testing problem applying m hypothesis tests. Then, a p-value function can be written as $p\left(x\right)\equiv p\left\{t\right(x\left)\right\}$ from a test statistic $t\equiv t\left(x\right)$ of any random dataset x.

When using any MTP $\mathcal{R}$ to test a set of hypotheses, $\mathcal{H}$, a traditional criterion for Type I error control is the FWER, the probability ($\mathbb{P}$) of making at least one false discovery [54]:

$${\mathrm{FWER}}_P\left(\mathcal{R}\right)={\mathbb{P}}_{X\sim P}[\mathrm{Reject}\mathrm{any}\mathrm{true}\mathrm{hypothesis},H_{0,j}\in {\mathcal{H}}_0\left(P\right)\subseteq \mathcal{H}];$$

(2)

while an alternative criterion is the FDR [55], the expected ($\mathbb{E}$) proportion of false rejections of null hypotheses out of the total number of rejected hypotheses:

$${\mathrm{FDR}}_P\left(\mathcal{R}\right)={\mathbb{E}}_{X\sim P}\left[{\mathrm{FDP}}_P\left(\mathcal{R}\left(X\right)\right)\right]={\mathbb{E}}_{X\sim P}\left[{\displaystyle \frac{|\mathcal{R}\left(X\right)\cap {\mathcal{H}}_0\left(P\right)|}{\max \left\{\right|\mathcal{R}\left(X\right)|,1\}}}\right].$$

(3)

The practice of multiple hypothesis testing aims to maximize the expected number of rejections while controlling the FWER or FDR at a preset level $\alpha \in (0,1)$, typically $\alpha =0.05$ or $0.01$, etc. An MTP strongly controls the FWER (FDR, resp.) if ${\mathrm{FWER}}_P\left(\mathcal{R}\right)\le \alpha$ (if ${\mathrm{FDR}}_P\left(\mathcal{R}\right)\le {\pi }_0\alpha \le \alpha$, resp.) for any $\alpha \in (0,1)$ and all $P\in \mathcal{P}$ (all ${\pi }_0\in [0,1]$, resp.), while $\mathrm{FDR}\le \mathrm{FWER}$, with $\mathrm{FDR}=\mathrm{FWER}$ if all null hypotheses are true, and thus $\mathrm{FWER}\le \alpha$ implies $\mathrm{FDR}\le \alpha$, i.e., FDR control is more liberal [18].

Many MTPs can each be characterized as a step-up MTP, ${\mathcal{R}}_{SU}^{{\Delta }_{\alpha }}$, which for the order statistics $p_{\left(1\right)}\le \cdots \le p_{\left(m\right)}$ of m p-values of the given multiple testing problem, specifies a non-decreasing sequence of thresholds $0\le {\Delta }_{\alpha }\left(H_{0,\left(1\right)}\right)\le \cdots \le {\Delta }_{\alpha }\left(H_{0,\left(m\right)}\right)\le 1$ for the respective m ordered tested null hypotheses, and then rejects the null hypotheses having the $R_{\alpha }\left(x\right)$ smallest p-values, with

$${\mathit{R}}_{\alpha }\left(x\right)=\underset{r\in \{0,1,\dots ,m\}}{\max }\{r:p_{\left(r\right)}\left(x\right)\le {\Delta }_{\alpha }\left(H_{0,\left(r\right)}\right)\}.$$

(4)

where $p_{\left(0\right)}\equiv 0$, and given a specified threshold function:

$${\Delta }_{\alpha ,\nu }\left(H_{0.\left(r\right)}\right)\equiv \alpha \pi \left(H_{0,\left(r\right)}\right){\beta }_{\nu }\left(r\right),$$

(5)

with weight function $\pi :\mathcal{H}\to [0,1]$ and ${\beta }_{\nu }:{\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ being a shape (reshaping) function. MTPs that strongly control FWER under arbitrary dependencies between p-values, include: the (unweighted) Bonferroni [19] MTP (B-MTP), defined by thresholds ${\Delta }_{\alpha ,\nu }\left(H_{0,\left(r\right)}\right)\equiv \alpha {\beta }_{\nu }\left(r\right)/m=\alpha /m$ with shapes ${\beta }_{\nu }\left(r\right)\equiv 1$ and weights $\pi \left(H_{0,\left(r\right)}\right)\equiv w_{\left(r\right)}=1/m$; the Holm [20] step-down MTP (H-MTP), defined by thresholds ${\Delta }_{\alpha ,\nu }\left(H_{0,\left(r\right)}\right)\equiv {\displaystyle \frac{\alpha }{m-r+1}}$. The weighted B-MTP is defined by thresholds ${\Delta }_{\alpha ,\nu }\left(H_{0,\left(r\right)}\right)=\alpha \pi \left(H_{0,\left(r\right)}\right){\beta }_{\nu }\left(r\right)=\alpha w_{\left(r\right)}$ using weights $\pi \left(H_j\right)\equiv w_j\in [0,1]$ assigned to respective null hypotheses $H_{0,j}\in \mathcal{H}$ such that ${\sum }_{i=1}^m{w}_j=1$, where $\pi \left(H_j\right)$ is an arbitrary probability distribution on $\mathcal{H}$ defining the relative importances of the m null hypotheses tested [18]. The weighted H-MTP is defined by thresholds ${\Delta }_{\alpha ,\nu }\left(H_{0,\left(r\right)}\right)\equiv {\displaystyle \frac{w_{\left(r\right)}}{{\sum }_{k=r}^m{w}_{\left(k\right)}}}\alpha$ with weights $w_j\ge 0$, $j=1,\dots ,m$.

For any probability measure $\nu$ on $(0,\infty )$, the step-up procedure (4) and (5) based on the shape function

$${\beta }_{\nu }\left(r\right)\equiv {\int }_0^{r}x\mathrm{d}\nu \left(x\right)$$

(6)

strongly controls $\mathrm{FDR}\le \alpha {\pi }_0\le \alpha$ under arbitrary dependencies between p-values, where $\pi :\mathcal{H}\to [0,1]$ is a probability mass function with respect to a counting measure $\mathsf{\Lambda }$ on $\mathcal{H}$ and ${\pi }_0={\sum }_{H_{0,j}\in {\mathcal{H}}_0}\mathsf{\Lambda }\left(\left\{H_{0,j}\right\}\right)\pi \left(H_{0,j}\right)$. Using different weights $\pi \left(H_{0,j}\right)$ (weights $\mathsf{\Lambda }\left(\left\{H_{0,j}\right\}\right)$, resp.) over countable $\mathcal{H}$ gives rise to weighted p-values (weighted FDR; Benjamini & Hochberg [24], resp.). For any finite set of m null hypotheses $\mathcal{H}$ and p-values, the (BY; Benjamini & Yekutieli [21]) distribution-free step-up MTP (i.e., BY-MTP) is defined by probability measure $\nu \left(\left\{k\right\}\right)={\left(k{\sum }_{j=1}^m\frac{1}{j}\right)}^{-1}$ with support in $\{1,\dots ,m\}$ and threshold function ${\Delta }_{\alpha ,\nu }(H_{0,\left(r\right)},r)=\alpha {\beta }_{\nu }\left(r\right)/m$, based on linear shape function ${\beta }_{\nu }\left(r\right)={\sum }_{k=1}^{r}k\nu \left(\left\{k\right\}\right)=r/\left({\sum }_{j=1}^m\frac{1}{j}\right)$ and p-value weights $\pi \left(H_{0,j}\right)\equiv w_j=1/m$ for $j=1,\dots ,m$ ([48], Lemma 3.2, p. 976). The weighted BY-MTP is instead defined by an arbitrary probability distribution $\pi \left(H_{0,j}\right)$ on $\mathcal{H}$.

Each of the above MTPs provides conservatively control the FWER or FDR, especially when the number of hypothesis tests (m) is large, while other choices of $\nu$ can sometimes improve the power of the corresponding MTP ([48], Section 4.2). With this in mind, the DP-MTP method was developed [22]: this MTP is defined by a Bayesian nonparametric, Dirichlet process (DP) [23] prior distribution that supports the entire space of random probability measures (r.p.m.s), $\nu$, which in turn, for an observed set of p values ${\left\{p_j\right\}}_{j=1}^m$, induces a DP prior distribution of random MTP thresholds ${\Delta }_{\alpha ,\nu }\left(H_{0,\left(r\right)}\right)\equiv \alpha \pi \left(H_{0,\left(r\right)}\right){\beta }_{\nu }\left(r\right)$ via (4)–(6), where each random threshold ${\Delta }_{\alpha ,\nu }$ (and DP random $\nu$), and thus the DP-MTP, strongly controls either the FWER or FDR under arbitrary dependence among p-values. Thus, the DP-MTP procedure naturally accounts for the uncertainty in the selection of MTPs and their respective decisions regarding which numbers of the smallest p-values are significant discoveries, from any set of hypotheses tested. Further, DP-MTP also measures each p-value’s probability of significance relative to the DP prior predictive distribution of this space of all MTPs.

Specifically, for any probability space $(\mathcal{X},\mathcal{A},G)$, an r.p.m. $\nu$ follows a Dirichlet process (DP) prior with baseline probability measure ${\nu }_0$ and mass parameter M, denoted $\nu \sim \mathrm{DP}\left(M{\nu }_0\right)$, if 

$$(\nu \left(B_1\right),\dots ,\nu \left(B_m\right))\sim {\mathrm{Dirichlet}}_m(M{\nu }_0\left(B_1\right),\dots ,M{\nu }_0\left(B_m\right))$$

(7)

for any (pairwise-disjoint) partition $B_1,\dots ,B_m$ of the sample space $\mathcal{X}$, with expectation $\mathbb{E}\left[\nu (·)\right]={\nu }_0(·)$, variance $\mathbb{V}\left[\nu (·)\right]={\displaystyle \frac{{\nu }_0(·)[1-{\nu }_0(·)]}{M+1}}$, and almost-sure support of the space of discrete r.p.m.s $\nu$ [23]. The DP-MTP is based on the DP prior (7) centered on baseline measure ${\nu }_0(r-1,r]\equiv {\left(r{\sum }_{j=1}^m\frac{1}{j}\right)}^{-1}$ (for $r=0,1,\dots ,m$) chosen to match in prior expectation (given M) the probability measure $\nu$ defining the BY MTP; and based on an exponential hyper-prior distribution of the DP mass parameter M in (7), given by

$$M\sim \mathrm{Exponential}\left(\mu \right),$$

(8)

and a default hyper-prior is chosen by $\mu \equiv 1$. Karabatsos [22] provides further discussions of the choices and characterizations of the DP prior for DP-MTP.

With respect to the joint hierarchical DP prior distribution (7) and (8) with BY-MTP baseline ${\nu }_0$, the DP-MTP, for each p-value from a given set of m ordered p-values, ${\left\{p_{\left(r\right)}\right\}}_{r=1}^m$, counts the proportion of times the p-value is significant, represented by the following vector (denoted $[·]$) of prior predictive probabilities:

$$\begin{array}{c}[Pr\{\mathbf{1}(r\le {\mathit{R}}_{\alpha ,\nu }\left(x\right);p_{\left(r\right)})=1\}:r=0,1,\dots ,m]\hfill \\ =\int \dots \int [\mathbf{1}(r\le {\mathit{R}}_{\alpha ,\nu }\left(x\right);p_{\left(r\right)}):r=0,1,\dots ,m]{\mathcal{D}}_m(\mathrm{d}{\nu }_1,\dots ,\mathrm{d}{\nu }_m\mid M{\nu }_0),\hfill \end{array}$$

(9)

where $\mathbf{1}(·)$ is the (1 or 0 valued) indicator function, ${\mathcal{D}}_m({\nu }_1,\dots ,{\nu }_m\mid M{\nu }_0)$ (with ${\nu }_r\equiv B_j=\nu (r-1,r]$ for $r=1,\dots ,m$) denotes the cumulative distribution function (CDF) of the Dirichlet distribution (7), and

$$\begin{array}{cc}\hfill R_{\alpha ,\nu }\left(x\right)& =\underset{r\in \{0,1,\dots ,m\}}{\max }\{r:p_{\left(r\right)}\left(x\right)\le {\Delta }_{\alpha ,\nu }\left(H_{0,\left(r\right)}\right)\}\hfill \\ \hfill & =\underset{r\in \{0,1,\dots ,m\}}{\max }\left\{r:p_{\left(r\right)}\left(x\right)\le \alpha \pi \left(H_{0,\left(r\right)}\right){\beta }_{\nu }\left(r\right)\right\}\hfill \\ \hfill & =\underset{r\in \{0,1,\dots ,m\}}{\max }\left\{r:p_{\left(r\right)}\left(x\right)\le \alpha \pi \left(H_{0,\left(r\right)}\right){\displaystyle \sum _{j=1}^r}j\nu (j-1,j]\right\},\hfill \end{array}$$

(10)

with $p_{\left(0\right)}\equiv 0$, while inducing a prior predictive distribution of the number $R_{\alpha ,\nu }$ in (10) of the smallest p-values from ${\left\{p_{\left(r\right)}\right\}}_{r=1}^m$ that represent significant discoveries. Also, (10) is defined by weights $\pi \left(H_{0,\left(r\right)}\right)$ and corresponding test levels ${\alpha }_{\left(r\right)}=\alpha \pi \left(H_{0,\left(r\right)}\right)$ given the overall prespecified level $\alpha$ (e.g., $\alpha =0.05$ or $0.01$, etc.) for the respective m ordered p-values ${\left\{p_{\left(r\right)}\right\}}_{r=1}^m$ satisfying ${\sum }_{r=1}^m\pi \left(H_{0,\left(r\right)}\right)\equiv 1$, such that for $r=1,\dots ,m$, equal weights $\pi \left(H_{0,\left(r\right)}\right)=1/m$ defines the unweighted DP-MTP [22], while a weighted DP-MTP is defined by an arbitrary probability distribution on $\mathcal{H}$ (as in the weighted B-MTP).

The joint hierarchical DP prior distribution (7) and (8) of the DP-MTP induces a prior predictive distribution for the shape parameter ${\beta }_{\nu }$ and corresponding threshold parameter ${\Delta }_{\alpha ,\nu }$ and number of discoveries $R_{\alpha }$ from (4), thereby treating these functions as random instead of fixed as done by the standard MTPs, while accounting for uncertainty in the selection of MTPs and their respective cut-off points and decisions regarding which of the smallest p-values are significant discoveries from a given set $\mathcal{H}$ of null hypotheses tested. The DP-MTP method thus emphasizes prior predictive hypothesis testing [56] and multiple bias modeling [57] while extending Vibrations of Effects analysis [58] to provide multiple hypothesis testing with uncertainty quantification. See Karabatsos [22] for further discussion.

The DP-MTP method can be run by R package bnpMTP [59] using the code line bnpMTP(·), which, given m input p-values (and perhaps optional inputs of $\alpha$, p-value weights, N, and $\mu$), uses standard methods to generate N Monte Carlo samples of r.p.m.s ${\left\{{\nu }_g\right\}}_{g=1}^m$ from the joint prior distribution (7) and (8) (with BY-MTP ${\nu }_0$) and of corresponding N samples from the prior predictive distribution of (9) and (10). In principle, DP-MTP can be extended to online multiple hypothesis testing ([60], and references therein) of a potentially infinite number of hypothesis tests, by relaxing the constraints of the weights $\pi \left(H_{0,\left(r\right)}\right)$ of the respective m ordered p values ${\left\{p_{\left(r\right)}\right\}}_{r=1}^{m\left(t\right)}$ obtained at any given time point t to satisfy ${\sum }_{r=1}^m\pi \left(H_{0,\left(r\right)}\right)<1$ (with total level ${\sum }_{r=1}^m\alpha \pi \left(H_{0,\left(r\right)}\right)<\alpha$) before all tests are performed, and to satisfy ${\sum }_{r=1}^m\pi \left(H_{0,\left(r\right)}\right)\equiv 1$ (with intended total level ${\sum }_{r=1}^m\alpha \pi \left(H_{0,\left(r\right)}\right)=\alpha$) after all tests are done (the same method can be used to define an online version of any other MTP mentioned in this section). This defines the only online MTP that controls FWER and/or FDR under arbitrarily dependent p-values with uncertainty quantification in multiple testing. Further, it is straightforward to extend the DP-MTP method to handle tests of continuous hypotheses, either by modeling $\nu$ as a discrete r.p.m. (e.g., assigned a DP prior), or by assigning $\nu$ a more general Bayesian nonparametric prior distribution which supports the space of continuous random probability measures, such as a DP mixture of continuous densities [61].

3. Predictive MTP Power Analysis Under Arbitrary Dependence

Consider any (off/online, un/weighted, and discrete or continuous) multiple hypothesis testing scenario employing $m\ge 1$ hypothesis tests using a preset total Type I error rate control $\alpha$, and corresponding random vector of $m\ge 1$ test statistics $\mathit{T}={(T_1,\dots ,T_m)}^{\top }$ with possible realizations $\mathit{T}=\mathit{t}={(t_1,\dots ,t_m)}^{\top }\in {\mathbb{R}}^m$. Each test statistic $T_j\in \mathit{T}$ typically has (or can be specified to have) the general form $T_j\equiv {\theta }_j/{\sigma }_{{\theta }_j}$, with $\theta$ an effect size parameter of interest (e.g., Section 1). Any realized value $t_j\equiv t_{n_j}={\widehat{\theta }}_j/{\sigma }_{{\widehat{\theta }}_j}$ of $T_j$ is determined by some estimate ${\widehat{\theta }}_j$ of $\theta$ with (estimated) standard error ${\widehat{\sigma }}_{{\widehat{\theta }}_j}$ based on a dataset of size $n_j$.

Further, assume that each of the m test procedures tests a null hypothesis $H_{0,j}\in \mathcal{H}$ against an alternative hypothesis $H_{1,j}$ using a test statistic $T_j\in \mathit{T}$ which, under $H_{0,j}$, follows a standard central location($\mu$)-scale($\sigma$) Student t-distribution, $T_j\sim \mathcal{T}(\mu \equiv 0,\sigma \equiv 1,{\nu }_j)$ with degrees of freedom ${\nu }_j>0$ an increasing function of $n_j$; and which, under $H_{1,j}$, follows a non-central t-distribution, $T_j\sim \mathcal{NT}({\mu }_j,\sigma \equiv 1,{\nu }_j)$, with non-zero non-centrality parameter ${\mu }_j\in \mathbb{R}/\left\{0\right\}$; perhaps after transforming the original test statistic to have a normal-like distribution under $H_{0,j}$ and under $H_{1j}$. The non-central $\mathcal{N}\mathcal{T}({\mu }_j,1,\nu )$ distribution (and asymptotic normal distribution $\mathcal{N}({\mu }_j,1)$ distribution, resp.) is thus the basis for power analysis for a t-test (z-test, resp.) of a mean difference, correlation or regression coefficient, (log-)odds ratio; etc., as mentioned in Section 1 and by statistics textbooks. Further examples using z-tests and t-tests are illustrated in Section 4.

Recall that if $\mu \ne 0$, then the $\mathcal{N}\mathcal{T}(\mu ,1,\nu )$ distribution is unimodal and asymmetric; and if $\mu =0$, this distribution is symmetric and coincides with the $\mathcal{T}(0,1,\nu )$ distribution. For each test procedure $j=1,\dots ,m$, we have that $n_j\to \infty$ implies ${\nu }_j\stackrel{d}{\to }\infty$ and asymptotic convergences to normal distributions, $\mathcal{T}(0,1,\nu )\stackrel{d}{\to }\mathcal{N}(0,1)$ under $H_{0,j}$, and $\mathcal{N}\mathcal{T}({\mu }_j,1,\nu )\stackrel{d}{\to }\mathcal{N}({\mu }_j,1)$ under $H_{1,j}$, while the test procedure may alternatively employ these asymptotic normal distributions; the two-tailed p-value is given by $p_j=2F_{{\nu }_j}(-|t_j\left|\right)$ with asymptotic p-value $p_j=2\mathsf{\Phi }(-|t_j|)$; and alternatively, a one-sided lower-tailed (upper-tailed, resp.) p-value is given by $p_j=F_{{\nu }_j}\left(t\right)$ with asymptotic p-value $p_j=\mathsf{\Phi }\left(t_j\right)$ (or by $p_j=1-F_{{\nu }_j}\left(t_j\right)$ with asymptotic p-value $p_j=1-\mathsf{\Phi }\left(t_j\right)$, resp.), where $F_{\nu }$ is the cumulative distribution function (CDF) of the $\mathcal{T}(0,1,\nu )$ distribution and $\mathsf{\Phi }$ is the normal $\mathcal{N}(0,1)$ distribution CDF.

Consider $m(\ge 1)$ variate generalizations of the normal, t, and non-central t distributions. Let ${\mathcal{N}}_m(\mathsf{\mu },\mathsf{\Sigma })$ be the m-variate normal distribution of a random vector $\mathit{Y}={(Y_1,\dots ,Y_m)}^{\top }$, defined by parameters of mean vector $\mathsf{\mu }={({\mu }_1,\dots ,{\mu }_m)}^{\top }$ and $m\times m$ symmetric positive-definite (s.p.d.) covariance matrix, $\mathsf{\Sigma }={\left({\sigma }_{j,k}\right)}_{m\times m}$, with diagonal elements ${({\sigma }_{1,1}^2\equiv {\sigma }_1^2,\dots ,{\sigma }_{m,m}^2\equiv {\sigma }_m^2)}^{\top }$ the respective variances of ${(T_1,\dots ,T_m)}^{\top }$, and each off-diagonal element ${\sigma }_{j,k}$ is the covariance $\mathbb{C}(T_j,T_k)$ for each distinct $j,k\in \{1,\dots ,m\}$. It can be convenient to model the (co)variance matrix $\mathsf{\Sigma }$ through either of the three following separation strategies: $\mathsf{\Sigma }\equiv \mathrm{diag}\left(\mathsf{\sigma }\right)\mathit{P}\mathrm{diag}\left(\mathsf{\sigma }\right)$, $\mathsf{\Sigma }\equiv {\sigma }^2\mathit{P}$, or $\mathsf{\Sigma }\equiv \mathit{P}$; where $\mathrm{diag}\left(\mathsf{\sigma }\right)$ is the $m\times m$ diagonal matrix of standard deviations $\mathsf{\sigma }={({\sigma }_{1,1},\dots ,{\sigma }_{m,m})}^{\top }$, ${\sigma }^2$ is a common variance, and $\mathit{P}={\left({\rho }_{j,k}\right)}_{m\times m}$ is the $m\times m$ correlation matrix (i.e., a s.p.d. matrix with ${\rho }_{j,j}=1$ and ${\rho }_{j,k}\in [-1,1]$) for all distinct $j,k\in \{1,\dots ,m\}$). A sample random vector $\mathit{Y}\sim {\mathcal{N}}_m(\mathsf{\mu },\mathsf{\Sigma })$ can be generated by $\mathit{Y}=\mathsf{\mu }+A\mathit{Z}$, with $\mathit{Z}={(Z_1,\dots ,Z_m)}^{\top }$ and ${\left\{Z_j\right\}}_{j=1}^m\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,1)$, using the Cholesky decomposition $\mathsf{\Sigma }=A{A}^{\top }$ of the s.p.d. matrix $\mathsf{\Sigma }$, where $A={\left(A_{j,k}\right)}_{m\times m}$ is the lower-triangular matrix with $A_{jj}>0$ for $j=1,\dots ,m$.

Let ${\mathcal{T}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$ be the m-variate Student’s t-distribution of random $\mathit{T}={(T_1,\dots ,T_m)}^{\top }$, a symmetric distribution defined by parameters of location (mean) vector $\mathsf{\mu }={({\mu }_1,\dots ,{\mu }_m)}^{\top }$, scale matrix $\mathsf{\Sigma }={\left({\sigma }_{j,k}\right)}_{m\times m}$, and degrees of freedoms $\mathsf{\nu }={({\nu }_1,\dots ,{\nu }_m)}^{\top }$; with corresponding pairwise covariances $\mathbb{C}(T_j,T_k)=\frac{{\sigma }_{j,k}\sqrt{{\nu }_j{\nu }_k}}{2}\frac{\mathsf{\Gamma }\{({\nu }_j-1)/2\}}{\mathsf{\Gamma }({\nu }_j/2)}\frac{\mathsf{\Gamma }\{({\nu }_k-1)/2\}}{\mathsf{\Gamma }({\nu }_k/2)}$ (if ${\nu }_j>1,{\nu }_k>1$) and correlations $\mathbb{P}(T_j,T_k)=\frac{{\rho }_{j,k}}{2}\frac{\mathsf{\Gamma }\{({\nu }_j-1)/2\}\sqrt{{\nu }_j-2}}{\mathsf{\Gamma }({\nu }_j/2)}\frac{\mathsf{\Gamma }\{({\nu }_k-1)/2\}\sqrt{{\nu }_k-2}}{\mathsf{\Gamma }({\nu }_k/2)}$ (if ${\nu }_j>2,{\nu }_k>2$; and ${\rho }_{j,k}\equiv \frac{{\sigma }_{j,k}}{\sqrt{{\sigma }_{j.j}{\sigma }_{k,k}}}$) for all distinct $j,k\in \{1,\dots ,m\}$, where $\mathsf{\Gamma }(·)$ is the gamma function. If $\mathsf{\Sigma }$ is a diagonal matrix, then the univariate marginal distributions of ${\mathcal{T}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$ are independent symmetric Student t-distributions, $\mathcal{T}({\mu }_j,{\sigma }_{j,j},{\nu }_j)$ for $j=1,\dots ,m$ ([62], Corollary 3). A sample random vector $\mathit{T}\sim {\mathcal{T}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$ can be generated by

$$\mathit{T}=\mathsf{\mu }+A\mathit{Z}\oslash {(\sqrt{V_1/{\nu }_1},\dots ,\sqrt{V_m/{\nu }_m})}^{\top },$$

(11)

with $\mathit{Z}={(Z_1,\dots ,Z_m)}^{\top }$, $Z_j\sim \mathcal{N}(0,1)$ and $V_j\sim {\chi }_{{\nu }_j}^2$ for $j=1,\dots ,m$ ([62], Definition 1), where ⊘ denotes the Hadamard (element-wise) division of vectors. A special case of ${\mathcal{T}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$ is the ${\mathcal{T}}_m(\mathsf{\mu },\mathsf{\Sigma },\nu )$ distribution ([63,64], Equation (1.1)), defined by stochastic representation $\mathit{T}=\mathsf{\mu }+A\mathit{Z}/\sqrt{V/\nu }$, $\mathit{Z}={(Z_1,\dots ,Z_m)}^{\top }$, ${\left\{Z_j\right\}}_{j=1}^m\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,1)$, $V\sim {\chi }_{\nu }^2$, $\nu >0$, and non-independent, m univariate marginal $\mathcal{T}({\mu }_j,{\sigma }_j,\nu )$ distributions, $j=1,\dots ,m$.

Now, introduce the m-variate upper non-central t-distribution, $\mathit{T}\sim {\mathcal{NT}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$, defined by parameters of non-centrality vector $\mathsf{\mu }\in {\mathbb{R}}^m$, scale matrix $\mathsf{\Sigma }$, and degrees of freedoms, $\mathsf{\nu }={({\nu }_1,\dots ,{\nu }_m)}^{\top }$. This m-variate distribution is a member of the class of normal mean-variance mixtures ([65], Section 3.2.2), and defined by the stochastic representation

$$\mathit{T}=(\mathsf{\mu }+A\mathit{Z})\oslash {(\sqrt{V_1/{\nu }_1},\dots ,\sqrt{V_m/{\nu }_m})}^{\top },$$

(12)

with $\mathit{Z}={(Z_1,\dots ,Z_m)}^{\top }$, $Z_j\sim \mathcal{N}(0,1)$ and $V_j\sim {\chi }_{{\nu }_j}^2$ for $j=1,\dots ,m$. The ${\mathcal{NT}}_m(\mathsf{\mu }$,$\mathsf{\Sigma },\mathsf{\nu })$ distribution is asymmetric when $\mathsf{\mu }\ne \mathbf{0}$, and coincides with the symmetric ${\mathcal{T}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$ distribution when $\mathsf{\mu }=\mathbf{0}$. A special case of ${\mathcal{NT}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$ is the m-variate upper non-central t-distribution, ${\mathcal{NT}}_m(\mathsf{\mu },\mathsf{\Sigma },\nu )$, defined by a scalar degrees of freedom $\mathsf{\nu }\equiv \nu$ and univariate marginal non-central t-distributions $\mathcal{NT}({\mu }_j,\nu )$ for $j=1,\dots ,m$ ([64,66,67], p. 81; Section 3.1; Equation (5.1); resp.), and by a complicated m-variate probability density function (pdf) even if $\mathsf{\Sigma }\equiv {\sigma }^2\mathit{P}$. Therefore, most applications of this distribution $\mathit{T}\sim {\mathcal{NT}}_m(\mathsf{\mu },\mathsf{\Sigma },\nu )$ employ its stochastic representation, given by $\mathit{T}={\left(\frac{V}{\nu }\right)}^{-1/2}(\mathsf{\mu }+A\mathit{Z})$, where $\mathit{Z}={(Z_1,\dots ,Z_m)}^{\top }$ with ${\left\{Z_j\right\}}_{j=1}^m\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,1)$ and $V\sim {\chi }_{\nu }^2$ ([66,68], Section 3.1; p.133, Equation (8); resp.), e.g., this distribution can be sampled using the mvtnorm R package code line rmvt(..., type = “Kshirsagar”) [69]. A straightforward modification of this code to divide a m-variate normal random $(\mathsf{\mu }+A\mathit{Z})$ by a random ${(\sqrt{V_1/{\nu }_1},\dots ,\sqrt{V_m/{\nu }_m})}^{\top }$ instead of $V/\nu$, enables sampling from the ${\mathcal{NT}}_m(\mathsf{\mu },\mathsf{\Sigma },\mathsf{\nu })$ distribution with vector $\mathsf{\nu }$ via its stochastic representation (12).

Let $\widehat{\mathsf{\theta }}={({\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_m)}^{\top }$ be the vector of anticipated or estimated effect sizes for a planned future study that will conduct tests of a specific set of m null hypotheses. In particular, $\widehat{\mathsf{\theta }}$ may represent the data analyst’s anticipated effect sizes, or may be the estimates of effect sizes obtained from a prior study (used for a future replication or interim study) of the same m hypothesis tests. Also, let ${\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}}={({\sigma }_{{\widehat{\theta }}_1},\dots ,{\sigma }_{{\widehat{\theta }}_m})}^{\top }$ be the standard errors of these m effect sizes $\widehat{\mathsf{\theta }}$, which are respectively decreasing functions of the (anticipated or actual) sample sizes ${(n_1,\dots ,n_m)}^{\top }$, and are the square roots of the diagonal elements of the (e.g., inverse Fisher information) variance–covariance matrix ${\mathsf{\Sigma }}_{\widehat{\theta }}$ of $\widehat{\mathsf{\theta }}$. In more common applications of power analysis, ${\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}}$ is known instead of the full matrix ${\mathsf{\Sigma }}_{\widehat{\theta }}$, though, covariances can sometimes be derived for certain effect size measures [70]. Specifying such covariances may not be crucial as this paper emphasizes power analysis of MTPs controlling FWER or FDR under arbitrarily dependent p-values (test statistics and effect sizes).

A conditional MTP power analysis m null hypotheses tests is based on test statistics distributed as $\mathit{T}\sim {\mathcal{T}}_m(\mathbf{0},\mathit{P},\mathsf{\nu })$ under all null hypotheses ${\left\{H_{0,j}\right\}}_{j=1}^m$ jointly, and distributed as $\mathit{T}\sim {\mathcal{NT}}_m(\mathsf{\mu },\mathit{P},\mathsf{\nu })$ under all alternative hypotheses ${\left\{H_{1,j}\right\}}_{j=1}^m$ jointly, given a specified fixed location vector $\mathsf{\mu }=|\widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}}|$; $m\times m$ correlation matrix $\mathit{P}$ of the underlying m-variate normal ${\mathcal{N}}_m(\mathsf{\mu },\mathit{P})$ distribution; and degrees of parameters $\mathsf{\nu }={({\nu }_1,\dots ,{\nu }_m)}^{\top }$ (where ${\nu }_j\to \infty$ based on sample size $n_j\to \infty$ for any test based on a normal distribution approximation of its test statistic $T_j$ under the null $H_{0,j}$). For $m=1$ hypothesis test, ${\mathcal{NT}}_m(\mathsf{\mu },\mathit{P},\mathsf{\nu })$ becomes the univariate non-central $\mathcal{NT}(\mu ,1,\nu )$ distribution ($\mathcal{N}(\mu ,1)$ distribution, resp.) used by classical conditional power analysis of a t-test (z-test if $\nu \to \infty$, resp.) [1]. However, conditional MTP power analysis relies on a prespecified fixed parameters $(\mathsf{\mu },\mathit{P},\mathsf{\nu })$ which can be difficult to specify when the number of tests m is sufficiently large, and does not account for their uncertainty; and cannot provide a congenial power analysis of an MTP that controls the FWER or FDR under arbitrary dependence between p-values, because such an MTP accounts for all possible $m\times m$ s.p.d. correlation matrices of test statistics underlying these m p-values, instead of only one fixed correlation matrix.

These issues can be addressed by an predictive power analysis of such an MTP, based on a joint prior distribution for the parameters $(\mathsf{\mu },\mathit{P})$ driving the m-variate distribution of test statistics, $\mathit{T}\sim {\mathcal{NT}}_m(\mathsf{\mu },\mathit{P},\mathsf{\nu })$, and corresponding p values under all alternative hypotheses ${\left\{H_{1,j}\right\}}_{j=1}^m$. Such a prior accounts for uncertainty in these parameters, and leads to the calculation of marginal powers of m hypothesis tests, thereby addressing all six aims of MTP power analysis (mentioned at the end of Section 1) without needing to assume independent p-values. The joint prior distribution (probability density function) for $(\mathsf{\mu },\mathit{P})$ is specified by

$$\pi (\mathsf{\mu },\mathit{P})={\mathcal{N}}_m(\mathsf{\mu }\mid \widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}},{\mathit{I}}_m)\mathrm{LKJ}(\mathit{P}\mid \eta \equiv 1),$$

(13)

based on an m-variate normal distribution for the mean parameters $\mathsf{\mu }$, with prior mean vector defined by anticipated or estimated effect sizes and their respective standard errors, and with (co)variances given by the $m\times m$ identity matrix (${\mathit{I}}_m$); and a uniform prior distribution for the $m\times m$ (s.p.d.) correlation matrix parameters $\mathit{P}$ (i.e., the LKJ distribution [71] with parameter $\eta \equiv 1$), which ensures a congenial (predictive) power analysis for such an MTP that controls the FWER or FDR under arbitrary dependent p-values, because this uniform prior supports all possible $m\times m$ correlation matrices of the test statistics underlying these p-values. In contrast, a conditional MTP power analysis assigns a point-mass prior at a chosen point $(\widehat{\mathsf{\mu }}=\widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}},\mathit{P})$ with a fixed correlation matrix $\mathit{P}$.

The default, m-variate normal prior ${\mathcal{N}}_m(\mathsf{\mu }\mid \widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}},{\mathit{I}}_m)$ in (13) is equivalent to the prior for $\mathsf{\mu }\equiv \mathsf{\theta }\oslash {\widehat{\mathsf{\sigma }}}_{\mathsf{\theta }}$ based on a prior $\mathsf{\theta }\sim {\mathcal{N}}_m(\widehat{\mathsf{\theta }},\mathrm{diag}\left({\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}}^2\right))$ on the m effect sizes $\mathsf{\theta }$, previously considered for $m=1$ effect size [15]. The ${\mathcal{N}}_m(\mathsf{\mu }\mid \widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}},{\mathit{I}}_m)$ prior is convenient because it allows for prior mean specification without needing explicit knowledge of standard errors possibly unavailable from a given published study, as illustrated in the case study in Section 4.

That said, publication bias and the winner’s curse often lead to overestimated original effect estimates (for example, refs. [72,73,74]), implying that for a power analysis of a replication study, the ${\mathcal{N}}_m(\mathsf{\mu }\mid \widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}},{\mathit{I}}_m)$ prior may be over-optimistic and lead to underpowered replication studies. A simple correction for this over-optimism is to multiply the effect sizes $\widehat{\mathsf{\theta }}$ in this prior by a vector factor $\mathit{d}={(d_1,\dots ,d_m)}^{\top }$ that takes on values between the vector of zeros ${\mathbf{0}}_m$ and a vector of ones ${\mathbf{1}}_m$, with shrinkage factor $\mathit{s}={\mathbf{1}}_m-\mathit{d}$ that can be chosen based on previous replication studies in the same field (as in [15] for the $m=1$ case). Further, a prior for $\mathsf{\mu }$ can be based on a (non-diagonal) s.p.d. covariance matrix ${\mathsf{\Sigma }}_{\widehat{\theta }}$ for $\mathsf{\theta }$. More generally, a power prior distribution [43] can be specified for $\left(\mathsf{\mu }\right)$ (or even for $(\mathsf{\mu },\mathit{P},\mathsf{\nu })$), defined as a posterior distribution constructed from previous historical data (likelihood) and a prior density for the parameters [16].

Simulation Algorithm 1 calculates the predictive powers for either the B-MTP, H-MTP, BY-MTP, and/or DP-MTP, for m given hypothesis tests, applicable to either an offline or online multiple testing scenario. This algorithm efficiently samples from the specified prior distribution for $\mathsf{\mu }$, and rapidly samples from the uniform prior distribution on $m\times m$ correlation matrices using the Pourahmadi & Wang [44] algorithm, run by the randcorr package (v1.0) [75] of the R software (v4.5.1) [76].

Algorithm 1 employs a default “objective” uniform prior that support all valid correlation matrices, for reasons explained in Section 1. However, when necessary, the algorithm can be easily modified in obvious ways to multiple testing situations where the prior on the correlation matrix is informative, such as a prior supporting correlation matrices that have either an equi-correlated structure, or alternatively, an AR(1) structure (e.g., for time-series or repeated measures designs).

Algorithm 1 Simulation algorithm to compute the predictive marginal powers for each MTP.

Inputs: The $m\ge 1$ test procedures used to test a given set of null hypotheses ${\left\{H_{0,j}\right\}}_{j=1}^m$, and their:

Test statistics $\mathit{T}={(T_1,\dots ,T_m)}^{\top }$; Type(s) of test(s) (lower-, upper, or two-tailed test);

Ratios, $(\widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}})={({\widehat{\theta }}_1/{\sigma }_{{\widehat{\theta }}_1},\dots ,{\widehat{\theta }}_m/{\sigma }_{{\widehat{\theta }}_m})}^{\top },$ of anticipated or estimated

effect sizes $\widehat{\mathsf{\theta }}={({\widehat{\theta }}_1,\dots ,{\widehat{\theta }}_m)}^{\top }$ divided by their corresponding standard errors;

Degrees of freedoms, $\mathsf{\nu }={({\nu }_1,\dots ,{\nu }_m)}^{\top }$; Total Type I error, $\alpha$; indicator function $\mathbf{1}(·)$;

Choice of MTP to use: B-MTP, H-MTP, BY-MTP, and/or DP-MTP;

Number of samples of DP r.p.m.s for DP-MTP if used: N (e.g., $N\equiv 1000$);

Number of sampling iterations for power analysis, S (e.g., $S\equiv 5000$).

for $s=1,\dots ,S$ do:

(a) Draw ${\mathsf{\mu }}^{\left(s\right)}\sim {\mathcal{N}}_m(\widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}},{\mathit{I}}_m)$   (or draw from another prior for $\mathsf{\mu }$ mentioned in paper).

(b) Draw ${\mathit{P}}^{\left(s\right)}\sim \mathrm{LKJ}(\eta \equiv 1)$       (using Pourahmadi & Wang [44] algorithm).

(c) Draw ${\mathit{t}}^{\left(s\right)}\sim {\mathcal{NT}}_m({\mathsf{\mu }}^{\left(s\right)},{\mathit{P}}^{\left(s\right)},\mathsf{\nu })$     (using stochastic representation Equation (12)).

(d) From each $t_j^{\left(s\right)}\in {\mathit{t}}_m^{\left(s\right)}={(t_1^{\left(s\right)},\dots ,t_m^{\left(s\right)})}^{\top }$, for $j=1,\dots ,m$, calculate (as intended) the

lower-, upper-, and/or 2-tailed p-values $p_1^{\left(s\right)}\equiv p\left(t_1^{\left(s\right)}\right),\dots ,p_j^{\left(s\right)}\equiv p\left(t_j^{\left(s\right)}\right),\dots p_m^{\left(s\right)}\equiv p\left(t_m^{\left(s\right)}\right)$.

(e) Apply each used B-MTP, H-MTP, BY-MTP, and/or DP-MTP, on ${\mathit{p}}^{\left(s\right)}$, at level $\alpha$, and

for sorted p-values $p_{\left(1\right)}^{\left(s\right)}\le \dots \le p_{\left(r\right)}^{\left(s\right)}\le \dots \le p_{\left(m\right)}^{\left(s\right)}$, calculate for $r=1,\dots ,m$,

$d_{\alpha ,\mathrm{M},\left(r\right)}=\mathbf{1}\{p_{\left(r\right)}\le {\Delta }_{\alpha ,\nu ,\mathrm{M}}\left(H_{0,\left(r\right)}\right)\}$, for MTP $\mathrm{M}\in \{\mathrm{B},\mathrm{H},\mathrm{BY}\}$,

$d_{\alpha ,\mathrm{DP},\left(r\right)}^{\left(s\right)}\equiv {\displaystyle \frac{1}{N}}{\sum }_{g=1}^N{d}_{\alpha ,\mathrm{DP},\left(r\right),g}^{\left(s\right)}$, where $d_{\alpha ,\mathrm{DP},\left(r\right),g}^{\left(s\right)}\equiv \mathbf{1}\{p_{\left(r\right)}^{\left(s\right)}\le {\Delta }_{\alpha ,{\nu }_g^{\left(s\right)},\mathrm{M}}\left(H_{0,\left(r\right)}\right)\}$,

to yield: ${\left(d_{\alpha ,\mathrm{B},\left(r\right)}^{\left(s\right)}\right)}_{r=1}^m$, ${\left(d_{\alpha ,\mathrm{H},\left(r\right)}^{\left(s\right)}\right)}_{r=1}^m$, ${\left(d_{\alpha ,\mathrm{BY},\left(r\right)}^{\left(s\right)}\right)}_{r=1}^m$, and/or ${\left(d_{\alpha ,\mathrm{DP},\left(r\right)}^{\left(s\right)}\right)}_{r=1}^m$, as intended.

end for

Output: For each MTP used, $\mathrm{M}\in \{\mathrm{B},\mathrm{H},\mathrm{BY},\mathrm{DP}\}$, the calculated estimates of:

Predictive marginal powers:   ${\overline{\mathbf{pmp}}}_{\alpha ,\mathrm{M}}\equiv \frac{1}{S}{\displaystyle \sum _{s=1}^S}{\left(d_{\alpha ,\mathrm{M},\left(r\right)}^{\left(s\right)}\right)}_{r=1}^m={\left({\overline{d}}_{\alpha ,\mathrm{M},\left(r\right)}\right)}_{r=1}^m$;

Predictive average power:      ${\overline{\mathrm{pap}}}_{\alpha ,\mathrm{M}}\equiv \frac{1}{S}\frac{1}{m}{\displaystyle \sum _{s=1}^S}{\displaystyle \sum _{r=1}^m}{d}_{\alpha ,\mathrm{M},\left(r\right)}^{\left(s\right)}$;

Predictive disjunctive power: ${\overline{\mathrm{pdp}}}_{\alpha ,\mathrm{M}}\equiv \frac{1}{S}{\displaystyle \sum _{s=1}^S}\mathbf{1}({\sum }_{r=1}^m{d}_{\alpha ,\mathrm{M},\left(r\right)}^{\left(s\right)}\ge 1)$;

Predictive conjunctive power: ${\overline{\mathrm{pcp}}}_{\alpha ,\mathrm{M}}\equiv \frac{1}{S}{\displaystyle \sum _{s=1}^S}\mathbf{1}({\sum }_{r=1}^m{d}_{\alpha ,\mathrm{M},\left(r\right)}^{\left(s\right)}=m)$.

p-value weights:                                ${\widehat{\pi }}_{\mathrm{M}}\left(H_{0,\left(r\right)}\right)\equiv {\displaystyle \frac{\exp [\log \left({\overline{d}}_{\alpha ,\mathrm{M},\left(r\right)}\right)-\widehat{a}]}{{\sum }_{k=1}^m\exp [\log \left({\overline{d}}_{\alpha ,\mathrm{M},\left(k\right)}\right)-\widehat{a}]}}$, for $r=1,\dots ,m$,

where $\widehat{a}\equiv {\max }_{k=1,\dots ,m}\left[\log \left({\overline{d}}_{\alpha ,\mathrm{M},\left(k\right)}\right)\right]$.

Significance Chasing Bias (Hellinger Distance):

${\mathrm{SigChase}}_{\left(r\right)}\equiv {\displaystyle \frac{1}{\sqrt{2}}}[{\{d_{\alpha ,\mathrm{M},\left(r\right)}^{1/2}\left(x\right)-{\overline{d}}_{\alpha ,\mathrm{M},\left(r\right)}^{1/2}\}}^2+{\{{(1-d_{\alpha ,\mathrm{M},j}\left(x\right))}^{1/2}-{(1-{\overline{d}}_{\alpha ,\mathrm{M},\left(r\right)})}^{1/2}\}}^2]$;

for sorted p-values $p_{\left(1\right)}\le \dots \le p_{\left(r\right)}\le \dots \le p_{\left(m\right)}$ and nulls ${\left\{H_{0,\left(r\right)}\right\}}_{r=1}^m$ tested on data x,

with $d_{\alpha ,\mathrm{M},\left(r\right)}\left(x\right)=\mathbf{1}\{p_{\left(r\right)}\le {\Delta }_{\alpha ,\nu ,\mathrm{M}}\left(H_{0,\left(r\right)}\right)\}$, for $r=1,\dots ,m$, for MTP $\mathrm{M}\in \{\mathrm{B},\mathrm{H},\mathrm{BY}\}$,

and/or $d_{\alpha ,\mathrm{DP},\left(r\right)}\equiv {\displaystyle \frac{1}{N}}{\sum }_{g=1}^N{d}_{\alpha ,\mathrm{DP},\left(r\right),g}={\displaystyle \frac{1}{N}}{\sum }_{g=1}^N\mathbf{1}\{p_{\left(r\right)}\le {\Delta }_{\alpha ,{\nu }_g,\mathrm{M}}\left(H_{0,\left(r\right)}\right)\}$, as intended.

Speed of Monte Carlo convergence of ${\overline{h}}_{S,\mathrm{M}}=\frac{1}{S}{\sum }_{s=1}^{s}h\left({\mathit{T}}_s\right)$:

Assessed by sample variance $v_{g,S}=\frac{1}{S^2}{\sum }_{s=1}^S{\{h\left({\mathit{t}}_m^{\left(s\right)}\right)-{\overline{h}}_S\}}^2$,

for each $g\in {\overline{\mathbf{pmp}}}_{\alpha ,\mathrm{M}}$ and each $g\in \{{\overline{\mathrm{pap}}}_{\alpha ,\mathrm{M}},{\overline{\mathrm{pdp}}}_{\alpha ,\mathrm{M}},{\overline{\mathrm{pcp}}}_{\alpha ,\mathrm{M}}\}$.

Also, a typical hypothesis test is most often defined by a test statistic that has a known distribution under the null hypothesis, most often a normal or Student t distribution (at least approximately). This is why Section 3 and Section 4 and the MTP power analysis Algorithm 1 emphasize the use of these distributions by default. In other situations, the hypothesis test statistic may have a non-normal, discrete, and/or unknown distribution under the null hypothesis (the latter which can be estimated through permutation- or bootstrap-based simulation), for one or more hypothesis test(s) among the m total tests performed. In these other hypothesis testing situations, it is straightforward to use the non-normal and/or estimated distribution of the hypothesis test statistic under the null hypothesis, instead of the normal or Student t- null hypothesis distribution(s) within Algorithm 1 in order to conduct an MTP power analysis that would be more accurate than a power analysis assuming a null or Student t null hypothesis distribution. Recall from Section 2 that the super-uniformity condition does not make specific requirements on the distribution of the test statistic under the null hypothesis.

4. Case Study Illustration

The applicability of the new MTP predictive power analysis method is showcased through the analysis of two-tailed p-values from forty-one hypothesis tests, presented in the third column of

Table 1. Two-tailed p-values from (Needleman et al. [77], Tables 3, 7 and 8); test (t) statistics; and DP-MTP results of marginal predictive power (MargPwr); p-value weights; probability of discovery for each p-value, PrSig (each weighted p-value PrSig.w); and significance-chasing bias index in Hellinger distance. Superscripts indicate significant discoveries according to either the un/weighted B-MTP, H-MTP, or BY-MTP.

Test	Scale	p-Value	t Stat.	MargPwr	p-Weights	PrSig	PrSig.w	sigChase
1	Behavior 1	0.003	2.97	0.46	0.05	0.28	0.53 BY	0.13
2	Behavior 2	0.05	1.96	0.23	0.02	0.00	0.00	0.35
3	Behavior 3	0.05	1.96	0.23	0.02	0.00	0.00	0.35
4	Behavior 4	0.14	1.48	0.15	0.01	0.00	0.00	0.28
5	Behavior 5	0.08	1.75	0.18	0.02	0.00	0.00	0.31
6	Behavior 6	0.01	2.58	0.37	0.04	0.02	0.08 BY	0.35
7	Behavior 7	0.04	2.05	0.25	0.03	0.00	0.00	0.37
8	Behavior 8	0.01	2.58	0.38	0.04	0.02	0.08 BY	0.36
9	Behavior 9	0.05	1.96	0.23	0.02	0.00	0.00	0.35
10	Behavior 10	0.003	2.97	0.47	0.05	0.28	0.53 BY	0.14
11	Behavior 11	0.003	2.97	0.47	0.05	0.28	0.53 BY	0.14
12	Sum Behavior	0.02	2.35	0.30	0.03	0.00	0.00	0.40
13	Verbal IQ 1	0.04	2.07	0.25	0.03	0.00	0.00	0.37
14	Verbal IQ 2	0.05	1.98	0.23	0.02	0.00	0.00	0.35
15	Verbal IQ 3	0.02	2.35	0.31	0.03	0.00	0.00	0.41
16	Verbal IQ 4	0.49	0.69	0.06	0.01	0.00	0.00	0.18
17	Verbal IQ 5	0.08	1.76	0.18	0.02	0.00	0.00	0.30
18	Verbal IQ 6	0.36	0.92	0.08	0.01	0.00	0.00	0.20
19	Performance IQ 1	0.03	2.19	0.28	0.03	0.00	0.00	0.39
20	Performance IQ 2	0.38	0.88	0.07	0.01	0.00	0.00	0.19
21	Performance IQ 3	0.15	1.45	0.14	0.01	0.00	0.00	0.27
22	Performance IQ 4	0.54	0.61	0.05	0.01	0.00	0.00	0.17
23	Performance IQ 5	0.90	0.13	0.03	0.00	0.00	0.00	0.13
24	Performance IQ 6	0.37	0.90	0.08	0.01	0.00	0.00	0.20
25	Full Verbal IQ	0.03	2.19	0.27	0.03	0.00	0.00	0.38
26	Full Perf. IQ	0.03	2.19	0.28	0.03	0.00	0.00	0.39
27	Full VerbalPerf.IQ	0.08	1.76	0.20	0.02	0.00	0.00	0.32
28	Seashore 1	0.002	3.15	0.50	0.05	0.38	0.70 B,BY	0.09
29	Seashore 2	0.03	2.19	0.26	0.03	0.00	0.00	0.37
30	Seashore 3	0.07	1.82	0.19	0.02	0.00	0.00	0.32
31	Total Seashore	0.002	3.15	0.51	0.05	0.38	0.70 B,BY	0.09
32	Token 1	0.37	0.90	0.07	0.01	0.00	0.00	0.19
33	Token 2	0.90	0.13	0.04	0.00	0.00	0.00	0.14
34	Token 3	0.42	0.81	0.07	0.01	0.00	0.00	0.19
35	Token 4	0.05	1.98	0.21	0.02	0.00	0.00	0.34
36	Total Token	0.09	1.71	0.18	0.02	0.00	0.00	0.30
37	Sentence	0.04	2.07	0.24	0.02	0.00	0.00	0.36
38	Reaction Time 1	0.32	1.00	0.08	0.01	0.00	0.00	0.20
39	Reaction Time 2	0.001	3.36	0.56	0.06	0.57 B,H	0.76 B,H,BY	0.01
40	Reaction Time 3	0.001	3.36	0.55	0.05	0.57 B,H	0.76 B,H,BY	0.02
41	Reaction Time 4	0.01	2.61	0.37	0.04	0.02	0.08 BY	0.35

.

These results were reported by a study [77] of the neuropsychologic effects of childhood lead exposure measured from the shed baby teeth that teachers collected from first- and second-grade Massachusetts schoolchildren during 1975–1978. This landmark study applied an innovative non-invasive way to collect data on accumulated lead content in children (e.g., lead from ingested wall paint), and yielded statistical results that motivated much further research on lead poisoning and related areas, and inspired actions by the Center for Disease Control to lower the blood lead standard for children, and by the U.S. Congress and Environmental Protection Agency (EPA) to eliminate lead from gasoline, paint, plumbing, and other uses. Still, this study was criticized for methodological flaws, including that it reported p-values that did not jointly control for multiple comparisons for all reported p-values (for further discussion, see [78], pp. 279–283). Therefore, these p-values (subsets) have been reanalyzed several times by the MTP literature ([21], Table 1).

The tooth measurements identified a group of 100 children with low lead exposure (<6 ppm; lowest 10th percentile) and a group of 58 children with high lead exposure (>24 ppm; highest 10th percentile). Both groups were compared on each of 11 items of a teachers’ behavior ratings scale, using respective (${\chi }^2$) z-tests of equal group proportions of negative behavior responses. These groups were also compared using analysis of covariance (ANCOVA) t-tests, comparing group mean: total sum behavioral rating score; sub-scale and full-scale (or sum) scores on the Wechsler Intelligence Scales for Children measuring verbal IQ and performance IQ; and scores from Seashore and Token tests of verbal processing; and reaction times varied by four time intervals. Each ANCOVA controlled for five covariates: mother’s age at subject’s birth; mother’s educational level; father’s socioeconomic status; number of pregnancies; and parental IQ.

In total, $m=41$ hypothesis tests were performed, which Table 1 summarizes, including the number of tests performed for each kind of testing procedure and variable. The fourth column of Table 1 shows the test statistic (t) derived from each two-tailed p-value, obtained by $t_j={\mathsf{\Phi }}^{-1}(1-p_j/2)$ for each of the z-tests (i.e., tests $j=1,\dots ,11$ in Table 1), and by $t_j={\mathcal{T}}^{-1}(1-p_j/2\mid 0,1,{\nu }_j=151)$ for each of the ANCOVA t-tests (tests $j=12,\dots ,41=m$), where $\mathsf{\Phi }$ denotes the standard normal $\mathcal{N}(0,1)$ CDF, and $\mathcal{T}$ is the standard t CDF on ${\nu }_j=n_j-(k+2)=158-(5+2)=151$ degrees of freedom, based on $k=5$ covariates and 2 treatment groups. Each test statistic t is an absolute ratio of an effect size measured by a raw group mean difference, divided by the standard error of this difference, where as appropriate, the effect size is either a between-group difference in proportions for a z-test, or a difference in dependent outcome means for an ANCOVA t-test.

The predictive MTP power analysis Algorithm 1 using $\alpha =0.05$ was applied for $S=5000$ sampling iterations, to evaluate the powers of the DP-MTP (for $N\equiv 1000$ iterations), the Bonferroni MTP, Holm MTP, and the Benjamini–Yekutieli MTP (BY-MTP), based on 41-variate normal ${\mathcal{N}}_{41}(\widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}},{\mathit{I}}_{41})$ prior for $\mathsf{\mu }$, with mean vector $\widehat{\mathsf{\theta }}\oslash {\mathsf{\sigma }}_{\widehat{\mathsf{\theta }}}$ specified by the fourth column of Table 1, the vector of test statistics. Further, while a uniform LKJ prior is assigned to the correlation matrix of the 41 test statistics, the degrees of freedom for these test statistics were set by ${\nu }_j\to \infty$ for the z-tests of proportions indexed by $j=1,\dots ,11$ for each of the 11 behavior items, and set by ${\nu }_j\equiv 151$ for each of the ANCOVA t-tests ($j=12,\dots ,41$). (For each ANCOVA t-test, Needleman et al. [77] did not provide $r^2$ of the combined 5 adjusting covariates nor the residual variance of the unadjusted dependent responses for both treatments, which could permit a more exact ANCOVA power analysis [79].) Overall, for each of the four MTPs, this MTP predictive power analysis is for a future replication study applying the same 41 null hypothesis tests and total sample sizes $n_j=158$ used for each test, based on an ignorable shrinkage factor vector, $\mathit{d}\equiv {\mathbf{1}}_m$ (a vector of m ones) implying no over-optimism in the vector of m effect sizes $\widehat{\mathsf{\theta }}$. The code used to run this MTP power analysis is available at: https://github.com/GeorgeKarabatsos/Predictive-Power-Analysis-of-Multiple-Test-Procedures-Under-Arbitrary-Dependence- provide the software code used to run this MTP power analysis, including R code from the bnpMTP package [59] and from the other cited packages.

The following results of the predictive MTP power analysis provided by Algorithm 1 are as follows, for the DP-MTP, B-MTP, H-MTP, and BH-MTP. The DP-MTP can be considered as the baseline MTP method, because as mentioned earlier, the DP prior supports the B-MTP, H-MTP, and BH-MTP, and all other MTPs which control the FWER or FDR under arbitrarily correlated p-values. For the 41 hypothesis tests, the DP-MTP, B-MTP, H-MTP, and BH-MTP obtained average power 0.25, 0.21, 0.22, and 0.26, respectively; all with disjunctive power 1.00 and conjunctive power 0.

Table 1 shows the marginal power for each of the 41 hypothesis tests, estimated by Algorithm 1. These are respectively the marginal powers of the 41 test statistics, with respect to the prior predictive distribution of the test statistics under their alternative hypotheses, under the prior centered on the observed effect sizes, and the uniform prior for the correlation matrices. The table shows that the marginal power tends to increase with the absolute t-statistic, a function of the absolute effect size, as intuitively expected. In other words, these are the marginal powers of the 41 hypothesis tests after accounting for arbitrarily correlated p-values (test statistics), powers which are different across these tests. The marginal powers are not very large for the tests, which reflects the relatively small sample size of the study and confirms the known fact that there is a decrease in power for individual tests in multiple testing scenarios [27]. Further, the DP-MTP marginal powers (Table 1) are similar to those of the three other MTPs (Figure 1, left side).

For DP-MTP, Table 1 also shows the p-value weights, and probability of significance discovery (PrSig and PrSig.w) for each unweighted and weighted p-value among the 41 total tests. These results are compared with the significance discovery indicators for each of the unweighted and weighted versions of B-MTP, H-MTP, and BY-MTP, obtained from the p.adjust() and p.adjust.w() codes of R package $\mathtt{someMTP}$ (v1.4.1.1) [80]. DP-MTP indicates that any test results in a discovery by PrSig > 0 (or by PrSig.w > 0 for weighted testing), while providing uncertainty quantification in multiple testing. The table shows that zero values of PrSig (and PrSig.w) correspond to non-discoveries found by the other MTPs.

For DP-MTP, Table 1 shows the significance-chasing bias index for each of the 41 p-values (tests), where non-bias is indicated by a small index value. The 41 bias indices of DP-MTP tended to be lower than the corresponding indices of B-MTP, H-MTP, and BY-MTP (Figure 1, right side), perhaps because of the DP prior supports all MTPs controlling the FWER or FDR instead of one significance cutoff for p-values.

We now consider a prior sensitivity analysis for the DP-MTP, with respect to four different levels of the shrinkage factor (Section 3). The shrinkage (and corresponding sensitivity analysis) can address the fact that, for a future replication setting, the previously observed test statistics defining the prior mean can possibly be affected by overestimation and the winner’s curse, and so the shrinkage can help rule out any systematic optimism in the reported power estimates.

Specifically, we consider shrinkage factors $s=0$ (already considered above), $1/4$, $1/2$, and $3/4$, such that each shrinkage factor s is defined as the same for all 41 hypothesis tests, and they are respectively referred to as zero, low, medium and large shrinkage factors. For these shrinkage factors, the DP-MTP respectively obtained: average powers of 0.25, 0.13, 0.05, and 0.02; disjunctive powers of 1.00, 0.98, 0.69, 0.29; and conjunctive powers of zero.

Figure 2 compares the marginal powers, p-value weights, and significance-chasing Hellinger distance measures across these four shrinkage levels. Overall, as expected, the measures of power decreases with increasing shrinkage level. Interestingly, the estimated p-value weights are rather insensitive to varying shrinkage. The significance-chasing measures mostly increased with decreasing shrinking level, but not always, and not necessarily in a strictly ordered fashion. As reasonably expected, these measures can vary with respect to the specified prior distribution on the effect sizes.

5. Theorem

Algorithm 1 addresses MTP predictive power analysis problem by evaluating the expectation

$${\mathbb{E}}_f\left\{h\left(\mathit{T}\right)\right\}=\int \cdots {\int }_{\mathcal{T}}h\left({\mathit{t}}_m\right)f\left({\mathit{t}}_m\right)\mathrm{d}{t}_1,\dots ,\mathrm{d}{t}_m,$$

(14)

where f is the marginal prior predictive probability density function of test statistics ${\mathit{t}}_m={(t_1,\dots ,t_m)}^{\top }$:

$$f\left({\mathit{t}}_m\right)\equiv \int \cdots \int {\mathcal{NT}}_m({\mathit{t}}_m\mid \mathsf{\mu },\mathit{P},\mathsf{\nu })\mathrm{LKJ}(\mathit{P}\mid \eta \equiv 1)\mathrm{d}{\mu }_1\cdots \mathrm{d}{\mu }_m\mathrm{d}{\rho }_{1,2}\cdots \mathrm{d}{\rho }_{m-1,m},$$

(15)

and where $h:{\mathbb{R}}^m\to {\mathbb{R}}^{+}$ is a bounded function $h(·)$ of ${\mathit{t}}_m$ corresponding to m ordered p-values $\underline{\mathit{p}}\left({\mathit{t}}_m\right)\equiv {(p_{\left(r\right)}\equiv p\left(t_{\left(r\right)}\right))}_{r=1}^m$, and defined either for a vector of predictive marginal powers: ${\mathbf{pmp}}_{\alpha ,\mathrm{M}}={(h\left\{p\left(t_{\left(r\right)}\right)\right\}\equiv d_{\alpha ,\mathrm{M},\left(r\right)})}_{r=1}^m\equiv {\left(\mathbf{1}\{p_{\left(r\right)}\le {\Delta }_{\alpha ,\nu ,\mathrm{M}}\left(H_{0,\left(r\right)}\right)\}\right)}_{r=1}^m$; for predictive average power: $h\left\{\underline{\mathit{p}}\left({\mathit{t}}_m\right)\right\}\equiv \mathrm{pap}={\displaystyle \frac{1}{m}}{\sum }_{r=1}^m{d}_{\alpha ,\mathrm{M},\left(r\right)}\left(x\right)$; for disjunctive power: $h\left\{\underline{\mathit{p}}\left({\mathit{t}}_m\right)\right\}\equiv \mathrm{pdp}=\mathbf{1}({\sum }_{r=1}^m{d}_{\alpha ,\mathrm{M},\left(r\right)}\ge 1)$; or for conjunctive power: $h\left\{\underline{\mathit{p}}\left({\mathit{t}}_m\right)\right\}\equiv \mathrm{pcp}=\mathbf{1}({\sum }_{r=1}^m{d}_{\alpha ,\mathrm{M},\left(r\right)}=m)$.

For any choice of these functions h, the power analysis algorithm estimates ${\mathbb{E}}_f\left\{h\left(\mathit{T}\right)\right\}$ by ${\overline{h}}_S=\frac{1}{S}{\sum }_{s=1}^{S}h\left({{\mathit{t}}_{\mathit{m}}}^{\left(s\right)}\right)$, and assesses the speed of convergence of ${\overline{h}}_S$ by the sample variance:

$$v_{h,S}=\frac{1}{S^2}{\displaystyle \sum _{s=1}^S}{\{h\left({{\mathit{t}}_{\mathit{m}}}^{\left(s\right)}\right)-{\overline{h}}_S\}}^2.$$

(16)

Theorem 1. 

The estimator ${\overline{h}}_S$ almost surely (a.s.) converges ${\overline{h}}_S\stackrel{a.s.}{\to }{\mathbb{E}}_f\left\{h\left(\mathit{T}\right)\right\}$ by the Strong Law of Large Numbers (SLLN), and $v_S$ is an estimate of the variance

$$\mathbb{V}\left\{{\overline{h}}_S\right\}={\displaystyle \frac{1}{S}}\int \dots {\int }_{\mathcal{T}}{[h\left({\mathit{t}}_m\right)-{\mathbb{E}}_f\left\{h\left(\mathit{T}\right)\right\}]}^{2}f\left({\mathit{t}}_m\right)\mathrm{d}{t}_1,\dots ,\mathrm{d}{t}_m.$$

(17)

Proof. 

For the iid $h\left({{\mathit{t}}_{\mathit{m}}}^{\left(s\right)}\right)$ obtained from ${{\mathit{t}}_{\mathit{m}}}^{\left(s\right)}\stackrel{iid}{\sim }f$, $h<\infty$ implies ${\mathbb{E}}_f\left\{h\left(\mathit{T}\right)\right\}<\infty$ with finite second moment. Then, a.s. convergence of ${\overline{h}}_S$ under SLLN holds ([81], Section 22), and $h^2$ has a finite expectation under f, which ensures that $v_S$ estimates (17). □

Further, given that each of the above choices of function h is binary (0 or 1) valued, its variance attains maximum $1/4$ when ${\mathbb{E}}_f\left\{h\left(\mathit{T}\right)\right\}=1/2$, which implies the upper bound $1/4S$ for its Monte Carlo variance (16). For example, the case study in Section 4 was based on running $S\equiv 5000$ sampling iterations of the algorithm, yielding an upper bound of $1/4S=1/\mathrm{20,000}=0.00005$ for the Monte Carlo variance of each of the presented estimates of power, implying fast convergence of the Monte Carlo power analysis algorithm.

6. Conclusions

This study proposed and illustrated a practical automatic method for congenially evaluating the power of any MTP which controls FWER or FDR under unknown and unanalyzed arbitrary dependencies (inter-correlations) between p-values. The predictive power analysis method is defined by a general prior distribution for the effect sizes and a joint uniform prior for the correlation matrix for the test statistics to fully account for uncertainty in these parameters.

The new method not only can be used to marginal powers of tests (resp.) or corresponding sample size determinations (given desired marginal powers) for a future planned (e.g., replication or interim) study, but also, the calculated marginal powers can either be used to weight p-values to increase MTP power while minimizing the relative impacts of any significance-chasing biases, or compared with raw p-values to evaluate for the presence of such biases.

As mentioned, Algorithm 1 outputs consistent estimates of the marginal predictive powers of the given m hypothesis tests, which can then be used as a basis for assigning weights to p-values, respectively, in a subsequent application of a weighted MTP based on these p-value weights. Recall that a weighted MTP, which assigns relatively higher weights to the subset of tested null hypotheses that are likely to be false, tends to have higher statistical power compared to unweighted multiple testing procedure. An interesting open question is how accurately such an MTP can detect the subset of truly false null hypotheses in any given multiple hypothesis testing problem, for a range of plausible correlation structures for the test statistics (p-values). This question deserves to be addressed extensively in future research.