Transfer Learning in Multiple Hypothesis Testing

Abstract

In this investigation, a synthesis of Convolutional Neural Networks (CNNs) and Bayesian inference is presented, leading to a novel approach to the problem of Multiple Hypothesis Testing (MHT). Diverging from traditional paradigms, this study introduces a sequence-based uncalibrated Bayes factor approach to test many hypotheses using the same family of sampling parametric models. A two-step methodology is employed: initially, a learning phase is conducted utilizing simulated datasets encompassing a wide spectrum of null and alternative hypotheses, followed by a transfer phase applying this fitted model to real-world experimental sequences. The outcome is a CNN model capable of navigating the complex domain of MHT with improved precision over traditional methods, also demonstrating robustness under varying conditions, including the number of true nulls and dependencies between tests. Although indications of empirical evaluations are presented and show that the methodology will prove useful, more work is required to provide a full evaluation from a theoretical perspective. The potential of this innovative approach is further illustrated within the critical domain of genomics. Although formal proof of the consistency of the model remains elusive due to the inherent complexity of the algorithms, this paper also provides some theoretical insights and advocates for continued exploration of this methodology.

1. Introduction

Multiple hypothesis testing (MHT) is a collection of statistical methods that are used to perform more than one statistical test simultaneously. Specifically, if there are m statistical tests to be performed, these methods aim to categorize these m tests into two groups: $m_0$ tests where the null hypothesis is true, indicated as ${\mathcal{H}}_0$, and $m_1=m-m_0$ tests where the alternative hypothesis is true, indicated as ${\mathcal{H}}_1$. This categorization is conducted in a way that controls the maximum allowable value of some error. MHT methods are particularly useful in applied statistics because they can handle large datasets, such as Big Data, where the real problem at hand often involves testing multiple scientific hypotheses simultaneously. One of the pioneering applications for MHT methods is in Genome-Wide Association Studies (GWASs) using RNA-seq technology. In this context, m represents the number of gene abundance tests for an Expression Sequence Tag (EST) in a given biological sample. These abundance counts are then compared across two different biological populations to test m hypotheses about comparisons and identify genes associated with a specific characteristic or phenotype related to the two biological populations.

Because it is not feasible to model all tests together using a single comprehensive sampling model, each test (each gene) is modeled independently. These individual models are combined in an MHT procedure. The MHT procedure takes the marginal evidence from each test as input and outputs the joint evidence in all tests for a given threshold t in all tests which leads to a separation between tests supposed to be under ${\mathcal{H}}_0$ from those under ${\mathcal{H}}_1$. The primary goal of using an MHT procedure is to control certain types of errors. On the one side those related to the number of errors in declaring tests as coming from ${\mathcal{H}}_1$ when they are actually from ${\mathcal{H}}_0$: these are the False Positive Rate ($FPR\left(t\right)$) the proportion of false rejections among all rejections and the False Discovery Rate (FDR), which is the expectation over the random sample of $FPR\left(t\right)$, it is the proportion of those rejections that should not have been rejected; while on the other side, we wish to control the proportion of missed rejections (declaring tests as coming from the ${\mathcal{H}}_0$ when they are from ${\mathcal{H}}_1$) overall rejection, the False Nonrejection Proportion ($FNP\left(t\right)$) and its expectation False Non-rejection Rate (FNR). An MHT procedure aims to generate a Receiver Operating Characteristic (ROC) curve: $1-FPR\left(t\right)$ versus $FPR\left(t\right)$ so that the Area Under the Curve (AUC) approaches 1. This happens when $FPR\left(t\right)\to 0$ for all thresholds t and considering the expectation referred to above, this is equivalent to having a $FDR\to 0$. In this sense, we consider equivalent controlling FPR and FDR, although they are not exactly the same. This occurs, for example, for the well-known Benjamini–Hochberg (BH) procedure [1] for $m\to \infty$ under assumptions untestable on a given sample [2]. These assumptions generally relate to the dependencies between the tests and the marginal sampling distributions of the test statistics [3], which are represented by the p-values $p{v}_1,\dots ,p{v}_m$.

The most common MHT methods usually rely on p-values generated by statistical tests. Because p-values can be derived from a wide range of statistical models, from simple to complex models such as Bayesian models with intractable likelihoods [4], they are often the go-to choice in MHT. In ideal conditions where the null hypothesis is simple or the test statistic is ancillary to the nuisance parameters, the p-values are uniformly distributed between 0 and 1. Such p-values are referred to as calibrated, making constructing reliable MHT procedures straightforward.

However, these ideal conditions are rare. In many practical cases, the p-values deviate from the uniform distribution, making them unreliable for controlling the FDR [3].

This problem has led researchers to create adaptive MHT methods that estimate the true distribution of p-values [5,6,7]. There is also growing interest in using additional samples to recalibrate existing MHT methods [7,8]. We propose a contemporary approach that utilizes deep learning techniques for this recalibration. The way we use this technique is also known as transfer learning in the machine learning language. Transfer learning is meant as a statistical technique in machine learning where a model developed for a particular task (i.e., analyzing an MHT problem where the truth is known) is reused as the starting point for a model on a second task (i.e., analyzing an MHT problem where the true is unknown). It is an optimization that allows for rapid progress or improved performance when modeling the second task. In the context of neural networks, this involves transferring the weights and learned features from a pre-trained neural network to a new neural network being trained for a different, but related, task.

Here, to work around the difficulties of using p-values that are often uncalibrated, we call for a Bayesian approach using Bayes factors (BF) for testing. BFs are the ratio between the marginal probability of the sample under the alternative hypothesis and the null. Individual test evidence from a BF strongly depends on the prior evidence for the unknown parameters in the composite null hypotheses. These are parameters that are not common to all hypotheses, and the prior distribution affects the marginal of the data and hence the BFs. Then the BFs could be arbitrarily driven by the prior rather than by the data. In contrast, for MHT, it has been shown in [9] that it is possible to use the so-called uncalibrated BF, $B_i$, from $c{B}_1,\dots ,c{B}_m$, where cBi is the ith full BF of the alternative hypothesis against the null in test i, and c > 0, is the ratio between the two prior normalization constants for the null and alternative hypotheses. $B_i$ is not a BF as it misses the prior normalizing constants c which is why it is referred to as uncalibrated, i.e., its interpretation is not that of the relative evidence between two hypotheses. Suppose that many tests involve models with one or more nuisance parameters (composite null hypotheses). In this case, an expert would need to elicit a prior distribution of unknowns in each test, which is unfeasible given the large value of m. Therefore, substituting the presence of an expert by employing formal rules in the prior definition leads to two usually vague or improper priors that must be employed for the alternative and null models. Therefore, the BF for the single hypothesis test is not determined due to the ratio between the prior pseudo-constants $c=c_1/c_0$, where $c_0$ and $c_1$ are the unbounded prior normalization constants for the parameters of the null hypothesis, $H_0$ and the alternative $H_1$, respectively (see [9,10]). Furthermore, in [9,10], it is shown that the use of proper well-calibrated priors, leading to fully defined and interpretable BFs $c{B}_i$, is not necessary in MHT. Uncalibrated BFs, $B_i$ in MHT avoids employing nonscalable computational techniques to obtain a properly defined BF $c{B}_i$ for each test.

Although a comprehensive overview of the literature on Multiple Hypotheses Testing (MHT) is beyond the scope of this article, readers are encouraged to consult seminal review articles such as [11,12,13] for insights specifically relevant to Genome-wide Association Studies (GWAS).

The crucial insight to highlight is that all MHT methodologies essentially rely on an ordered sequence of test statistics, and the order is sample-dependent, and thus per se random. This is typically presented as $p{v}_1\ge p{v}_2\ge \cdots \ge p{v}_i\cdots \ge p{v}_m$ in conventional methods or as $c{B}_1\le c{B}_2\le \cdots \le c{B}_i\cdots \le c{B}_m$ in the BF approach [9,10], in which it is clear that $c>0$ does not alter the order. These ordered sequences serve as the output of various statistical tests and the input of MHT and the methodology proposed in this article. The sequences are valuable for segregating the tests that fall under the null hypothesis from those that fall under the alternative hypothesis.

The core concept is that the same sampling models used for obtaining p-values or BFs can also produce a training set of either p-values or BFs. The labels for the null and alternative hypotheses under which p-values or BFs are generated are already known in this training set. This training set can then be used to fit a classification model during a learning phase. In this light, the problem of MHT becomes one of classifying subsequences of p-values or BFs originating from either the null or alternative hypotheses.

Today, the problem of classifying subsequences is very well handled using complicated functions known as Neural Networks (NN), for instance, the one-channel convolutional NN (CNN). CNN is a specific class of NN suitable for analyzing structured (e.g., dependent) samples as images [14] with the general purpose of classifying them to obtain, for example, a medical diagnosis [15]. Furthermore, for our purposes, it is worth stressing that there is evidence that CNNs are very useful for analyzing Time Series [16] better than other architectures such as NN recurrent and long-short-term memory [17]. The architecture of a CNN is based on a connecting set of neurons (mathematical operations on some input) that are supposed to be trained to recognize relevant features in the observed sequence to achieve the minimum classification error, and thus the minimum FPR and FNP.

In the following sections, we limit our discussion to those aspects of CNNs directly relevant to their application in MHT. We refer the reader to existing reviews or books on the subject for a more comprehensive understanding, such as [14]. The rest of this paper provides the actual code (based on Keras and Tensorflow) used to construct and train CNN. Detailed technical information is available there, and we conclude by emphasizing that all computations were performed on a standard laptop, negating the need for specialized hardware. The accompanying R code is accessible at: 14 December 2023 https://colab.research.google.com/drive/1TdM1FSVKm1GI55riUXoLbzcEcM3FeoNg?usp=sharing.

The remainder of this paper is structured as follows. Initially, Section 2 outlines the MHT approach that we propose, with particular emphasis on estimating the probability, denoted $\widehat{p}$, that a hypothesis is part of the alternative set. Subsequently, Section 2.1 demonstrates this framework through a simulation study based on a parametric example. Subsequently, Section 4 delves into more intricate models by revisiting two RNA-seq experiments, where we contrast the evidence derived from BF and conventional p-values. Concluding thoughts with an exposition of the limitations of this approach and additional comments are reserved for Section 5.

2. Convolutional Neural Network for Multiple Testing Arising from Uncalibrated Bayes Factors

The MHT methodology elaborated in this study is based on a sequence of ordered BFs, $c{B}_i$, $i=1,\dots ,m$, ranging from the weakest to the strongest evidence of some common alternative hypothesis (to m) against a common null hypothesis. This empirically determined sequence serves as the classification subject for CNN.

To be precise, consider $c{B}_0=c{B}_1\le c{B}_2\le \dots \le c{B}_m$ as an ordered set of $m+1$ BFs, where $c{B}_0$ is only introduced for notational convenience. We define the relative weights of the evidence among the m alternative hypotheses as

$$\mathbf{W}=\left\{W_i=log(c{B}_i/c{B}_{i-1}), \mathrm{for} i=2,\dots ,m\right\},$$

with $W_1=1$. This set $\mathbf{W}$ represents the differential evidence that favors each alternative hypothesis relative to its predecessor.

The ordered sequence of interest is $\mathbf{w}=(w_1,\dots ,w_m)$, where each $w_i$ corresponds to a BF $c{B}_i$. Tests appearing earlier in this sequence are supposed to originate from the null hypothesis, while those appearing later are more likely from the alternative. In particular, although the individual $c{B}_i$ may not be interpretable due to the unknown scaling factor c, the relative evidence $W_i$ remains meaningful [9].

Let the original set of hypotheses, denoted as $\mathcal{H}$, be partitioned into two supposed nonempty sets ${\mathcal{H}}_1$ and ${\mathcal{H}}_0$, representing true alternative and true null hypotheses, respectively. To estimate ${\mathcal{H}}_1$ with an ordered sequence of W, it would suffice to estimate its size $m_1$, subject to $1\le m_1\le m$. As discussed in Section 3, this can often, but not necessarily, be accomplished by estimating the position $\widehat{i}\in (1,\dots m)$ of a change point in the sequence $\mathbf{w}$.

To elucidate, Figure 1 shows a simulated example with tests $m=10,000$ and $m_1=1000$, detailing how this methodology works in practice.

Figure 1 also reports the already mentioned training set for CNN that will be applied later in real-world scenarios where ${\mathcal{H}}_1$ and ${\mathcal{H}}_0$ are unknown.

The efficacy of using this particular CNN formulates the crux of our argument, affirming that the method’s robustness is not heavily dependent on various unknowns such as the number of tests under the null, $m_0$, data signal strength (i.e., sharp or no sharp evidence from tests), test dependencies, and sampling distributions, among others.

2.1. Sequence of the Relative Weights of Evidence

This section introduces a generalized definition of $\mathbf{W}$ customized later for specific parametric model environments. Consider a vector of experimental outcomes, $\mathbf{x}=({\mathbf{x}}_1,\dots ,{\mathbf{x}}_m)$, each featuring m distinct attributes such as gene abundance levels measured by RNA-seq counts. The vector ${\mathbf{x}}_i$ includes $n_i$ replications for the ith feature, with $i\in (1,\dots m)$.

The MHT issue can be framed as a Bayesian multiple model selection problem, where each test i compares the evidence supporting the alternative hypothesis $H_{i1}$ against the null hypothesis $H_{i0}$, which have the same prior probabilities (i.e., $P\left(H_{i1}\right)=P\left(H_{i0}\right)=1/2$) as follows:

$$\begin{array}{cc}\hfill H_{i0}& :f_{i0}({\mathbf{x}}_i\mid {\theta }_{i0}),{\pi }_{i0}\left({\theta }_{i0}\right),{\theta }_{i0}\in {\Theta }_{i0},\hfill \\ \hfill H_{i1}& :f_{i1}({\mathbf{x}}_i\mid {\theta }_{i1}),{\pi }_{i1}\left({\theta }_{i1}\right),{\theta }_{i1}\in {\Theta }_{i1},\hfill \end{array}i=1,\dots ,m.$$

(1)

Here, ${\pi }_{i0}\left({\theta }_{i0}\right)$ and ${\pi }_{i1}\left({\theta }_{i1}\right)$ are typically default and often improper prior distributions on the unknown model parameters. The sets $\{{\Theta }_{i0},{\Theta }_{i1}\}$ are a of partition ${\Theta }_i\subset {\Re }^K$, where $K\ge 1$.

Prior distributions are assumed to be the same across all tests (as the sampling model in each test) and are derived from a standard formal rule applied to each $f_{ik}(·\mid ·)$ for $k=0,1$. Such rules include but are not limited to, Jeffreys, Intrinsic, Reference, Matching, Nonlocal priors, or Conventional priors [18,19,20,21,22,23].

Consequently, for all i:

$$\begin{array}{cc}\hfill {\pi }_{i0}\left({\theta }_{i0}\right)& ={\pi }_0\left({\theta }_0\right)\propto c_0{g}_0\left({\theta }_0\right),\hfill \\ \hfill {\pi }_{i1}\left({\theta }_{i1}\right)& ={\pi }_1\left({\theta }_1\right)\propto c_1{g}_1\left({\theta }_1\right),\hfill \end{array}$$

(2)

where $g_0$ and $g_1$ are two positive functions (not necessarily measurable) and $c_0$ and $c_1$ act as normalizing pseudoconstants.

We assume the existence of prior predictive distributions for both the null and alternative hypotheses.

$$m_{ik}\left({\mathbf{x}}_i\right)={\int }_{{\theta }_k\in {\Theta }_k}{f}_k({\mathbf{x}}_i\mid {\theta }_k){\pi }_k\left({\theta }_k\right)d{\theta }_k,\mathrm{for}k=0,1,i=1,\dots ,m.$$

(3)

The BF of $H_{i1}$ against $H_{i0}$ can then be formulated as:

$$c{B}_i=\frac{m_{i1}\left({\mathbf{x}}_i\right)}{m_{i0}\left({\mathbf{x}}_i\right)}=\frac{c_1}{c_0}·\frac{{\int }_{{\theta }_1\in {\Theta }_1}{g}_1\left({\theta }_1\right)f_1({\mathbf{x}}_i\mid {\theta }_1)d{\theta }_1}{{\int }_{{\theta }_0\in {\Theta }_0}{g}_0\left({\theta }_0\right)f_0({\mathbf{x}}_i\mid {\theta }_0)d{\theta }_0},$$

(4)

This calibrated BF, in the sense that it reports the posterior relative evidence of $H_{i1}$ against $H_{i0}$ according to the Jeffreys interpretation, is practically unscaled due to the arbitrarily low ratio $c=c_1/c_0>0$. We then define the uncalibrated or unscaled BF as follows:

$$B_i=\frac{{\int }_{{\theta }_1\in {\Theta }_1}{g}_1\left({\theta }_1\right)f_1({\mathbf{x}}_i\mid {\theta }_1)d{\theta }_1}{{\int }_{{\theta }_0\in {\Theta }_0}{g}_0\left({\theta }_0\right)f_0({\mathbf{x}}_i\mid {\theta }_0)d{\theta }_0}.$$

(5)

Although $B_i$ lacks standalone interpretability, it serves as a comparative measure [9,10]. For example, if $exp\left(W_i\right)=\frac{B_i}{B_{i^{\prime }}}>1$ for all $i,i^{\prime }$, then the evidence supporting $H_{i1}$ over $H_{i0}$ is stronger than that for $H_{i^{\prime }1}$ over $H_{i^{\prime }0}$, regardless of c.

In summary, even if the priors are specified as vague or improper, their normalizing constants are effectively simplified in the sequence of Ws. This does not imply that priors are irrelevant in MHT, but their impact, specifically that of the constants $c_0$ and $c_1$, is mitigated in the collective evidence derived from the tests.

2.2. Convolutional Neural Network on the Sequence of Relative Weights of Evidence

A one-dimensional CNN is fitted to the observed sequence $\mathbf{w}$. The loss function used is the binary cross-entropy, which is the logarithmic representation of the Bernoulli density: ${\sum }_{i=1}^{m}log({\widehat{p}}_i^{H_i}{(1-{\widehat{p}}_i)}^{1-H_i}$, where $H_i=1$ ($H_i=0$) if the test i has been observed under the alternative (null) in the training set. This function minimizes the classification error, and thus the FPR and FNP. For each test i, the fitted CNN produces a point estimate of the probability, ${\widehat{p}}_i$, that it belongs to ${\mathcal{H}}_1$. The decision about the set of observed tests from the alternative is formulated as

$${\widehat{\mathcal{H}}}_1=\left\{i:{\widehat{p}}_i>1-q\right\},$$

where q is the FDR level (or the averaged FPR) that we want to control when testing the hypotheses. It can be argued that ${\widehat{p}}_i$ is the maximum a posteriori probability that the test i is observed under the alternative given the set of m tests and the priors of the underlying Gaussian process prior to the NN weights [24].

Using CNN, we establish a complicated function $\mathrm{CNN}:\mathbf{w}\mapsto \widehat{\mathbf{p}}=({\widehat{p}}_1,\dots {\widehat{p}}_m)$ that accounts for the dependencies among the W’s. This is how the m tests are jointly considered to control the FDR at level q, as will be discussed in the next Section 3.

The assumption of exchangeability between the training (simulated) sample and the observed sample of W’s is understood within the context of the sampling model induced by the fitted CNN. The existence of untestable assumptions needed when using the usual MHT has to be compared with the possibility of assessing the goodness-of-fit of a trained CNN. This can be conducted using routine analyses typical of the machine learning literature by assessing performance on test sets, which can also be simulated. This is exactly what is conducted in Section 4.2. In general, the fact that in Section 4.2 the sampling model for individual tests differs from the one used in the training set lends reliability to the estimates $\widehat{\mathbf{p}}$ even if CNN was fitted only to a simulated data set of $m=10,000$, ${\mathbf{w}}^{*}$, as shown at the top of Figure 1. For this purpose, it is crucial to note that we trained only one CNN in ${\mathbf{w}}^{*}$ in Figure 1 for all subsequent analyses in this paper. In the training sample, we have $m=10,000$ values of ${\mathbf{w}}^{*}$ simulated from independent tests on the mean of two independent normal populations with equal but unknown unit variances, as detailed in Section 4.2. In ${\mathbf{w}}^{*}$, we have $m_1=1000$ tests simulated from ${\mathcal{H}}_1$ with a mean 3 in one population and the rest from ${\mathcal{H}}_0$ with both populations with zero mean.

Understanding CNN from the MHT perspective is important for confidence in the proposed method. A CNN is an NN where (deterministic) nodes are functions of inputs and are connected according to a specific structure. Nodes are typically operations with weights that are set to minimize the global error in classifying the results of tests in ${\mathcal{H}}_1$ (or ${\mathcal{H}}_0$) when they come from ${\mathcal{H}}_0$ (or ${\mathcal{H}}_1$). In a CNN, we have two types of nodes:

1.

Feature detection nodes. They have as input the subsequences of $\mathbf{w}$,

$${\tilde{\mathbf{w}}}_{i,k}=(w_i,w_{i+1},\dots ,w_{i+k-1}),$$

where $k\ge 2$ is known as the kernel size. These subsequences of a minimum length of $\mathbf{w}$ bear local information about the random sequence of W’s. Such nodes return subsequences of $\tilde{\mathbf{w}}$’s all of the same size k, in which relevant features are detected through the so-called filters. Filter functions are defined on sets of weights and are devoted to detecting locally features on each ${\tilde{\mathbf{w}}}_{i,k}$. The systematic application of the same filter across the $\tilde{\mathbf{w}}$ sequences is useful for our problem. Each filter, that is, each set of weights, detects a specific behavior in the series $\mathbf{w}$, especially near the separation point $W_{\widehat{i}}$, mentioned above. The problem is that we do not know the relevant behavior to be detected and where it should be expected in the W’ s sequence. Therefore, all filters are applied to all sequences $\tilde{\mathbf{w}}$, along the entire observed sequence of W’s. This filtering process allows CNN to discover what and where the behavior of W is expected to estimate $\widehat{\mathbf{p}}$ correctly. We know, for example, that it should be important to analyze the behavior around $W_{\widehat{i}}$ where $\widehat{i}$ can be anywhere between 1 and m. This capability is commonly referred to as the translation invariance of a CNN. Fortunately, and contrary to the usual interpretation of NN as black boxes, it is possible to show the features detected by each filter, as shown in Figure 2, which reports the CNN weight values for the first convolution layer, which has 64 filters and thus weights $64\times m$. From Figure 2, it is possible to appreciate a change of activation around position 9001, which is the actual $\widehat{i}$ point mentioned above.

2.

Pooling nodes. These nodes connect all filters through the pooling function. The filter is just a dot product of the input, ${\tilde{\mathbf{w}}}_{i,k}$, using a set of weights. The output of such a product is the input of the pooling function that leads to a result of a dimension less than k, for example, considering the maximum output resulting from a filtered ${\tilde{\mathbf{w}}}_{k,i}$. The set of pooling nodes is also called the feature map because it gives a map of the relevant filters along the sequence of W’s for classifying tests.

The architecture of the CNN captures three vital characteristics when modeling $\mathbf{w}$:

• Localized Feature Detection: Kernels of small sizes are employed to focus on local features in the series of $\mathbf{w}$’s. This contributes to sparse modeling of the sequence and enables the capture of intricate dependencies between tests.
• Parameter Efficiency: To achieve parsimonious modeling, the same set of weights (i.e., model parameters) are reused throughout the sequence $\mathbf{w}$. This design leverages the power of shared evidence for NN parameters, offering a more accurate weight estimation based on multiple samples.
• Robust Feature Recognition: The CNN is equipped to identify critical features in the data sequence, invariant to factors such as location, scale, position of the separation point $\widehat{i}$, and test dependencies. This robustness potentially uncovers features to be described that are instrumental in estimating the sets ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$. For example, the translation invariance property mentioned above is not shared by common change point detection techniques [25], such as the cumulative sum control chart (CUSUM) [26].

Further details about the specific CNN architecture used in this study are provided in Appendix B. Although we do not insist that this architecture is universally optimal for MHT, it has been proven effective for the illustrative purposes of this paper.

To reiterate, we trained a single CNN model using the simulated set of m tests displayed in Figure 1. Subsequent results validate CNN’s capability for MHT, demonstrating its proficiency in classifying tests. For other practical scenarios, CNN could be trained using the results of a calibration experiment—assuming that such data are available and the ground truth is known—instead of relying on simulated data as in Figure 1.

3. Sketch of the Theory

The objective of this section is not to provide rigorous proof of the method’s consistency but to offer theoretical insights supporting its asymptotic behavior, as observed in the simulation studies. Specifically, our objective is to theoretically substantiate that the proposed method demonstrates asymptotic consistency with respect to both the sample size n and the number of tests m, as evidenced by negligible FDR and FNR for sufficiently large values of n and m.

First, at a specific separation point $\widehat{i}$ in the sequence of ordered Bayes factors $B_i$, the corresponding $W_{\widehat{i}}$ is defined as:

$$W_{\widehat{i}}=log\left(\frac{Pr(\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{a}\mathrm{t} \widehat{i}\in {\mathcal{H}}_1|\mathcal{W})}{1-Pr(\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{a}\mathrm{t} \widehat{i}\in {\mathcal{H}}_1|\mathcal{W})}\right)-log\left(\frac{Pr(\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{a}\mathrm{t} \widehat{i}-1\in {\mathcal{H}}_1|\mathcal{W})}{1-Pr(\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{a}\mathrm{t} \widehat{i}-1\in {\mathcal{H}}_1|\mathcal{W})}\right),$$

where $Pr(\mathrm{T}\mathrm{e}\mathrm{s}\mathrm{t} \mathrm{a}\mathrm{t} \widehat{i}\in {\mathcal{H}}_1|\mathcal{W})$ indicates the marginal probability of observing the evidence of test $\widehat{i}$ in the alternative set, which is the numerator of BF $c{B}_{\widehat{i}}$.

The proposition guarantees the asymptotic existence of this separation point $\widehat{i}$ in the sequence of W’s.

Proposition 1. 

For $n\to \infty$ then $W_{\widehat{i}}\to \infty$.

Proof. 

The proof relies on the well-known consistency of BF (see, e.g., [27]) for every $0<c<\infty$, that is, for $n\to \infty$ and $i\ne \widehat{i}$, we have $c{B}_i\to 0$ (for $i<\widehat{i}$) or $c{B}_i\to \infty$ (for $i>\widehat{i}$) and thus $W_i\to 0$ for $i\ne \widehat{i}$. At the separation point $\widehat{i}$ we have $c{B}_{\widehat{i}}\to \infty$ as $\widehat{i}\in {\mathcal{H}}_1$ and $c{B}_{\widehat{i}-1}\to 0$ as the test $\widehat{i}-1\in {\mathcal{H}}_0$, therefore, $W_{\widehat{i}}=c{B}_{\widehat{i}}/c{B}_{\widehat{i}-1}\to \infty$.    □

Second, the objective of the paper is to present evidence supporting the asymptotic consistency of the CNN estimator $\widehat{\mathbf{p}}$ as $m\to \infty$. Previous work [28,29], has established the consistency of feedforward NNs in distance $L_2$, which can, in principle, be applied to CNNs, although there is no specific literature on CNN consistency [30,31]. Define $h_0\left(\mathcal{W}\right)$ as a CNN oracle such that $h_0\left(w_i\right)=1$ if you test $i\in {\mathcal{H}}_1$ and 0 otherwise. The CNN adjusted to the tests of m is denoted by $\widehat{h}\left(\mathcal{W}\right)$. According to the aforementioned literature [28,29,32], the distance $L_2$ between these CNNs vanishes asymptotically as $m\to \infty$:

$$\int {\left(h_0\left(\mathcal{W}\right)-\widehat{h}\left(\mathcal{W}\right)\right)}^{2}d\mathcal{W}\to 0.$$

Furthermore, we argue that $h_0\left(w_i\right)\to j$ is as $n\to \infty$ for each $i\in {\mathcal{H}}_j$, where $j\in \{0,1\}$. This is a less restrictive condition than assuming the same stochastic process generating the observable variables, and thus the BFs. It is well known that the asymptotic consistency of the BFs can be achieved in the true model, but as n increases, the BF favors the model (although not the true one), minimizing the Kullback–Leibler (KL) divergence, making the model closest (to the true one) increasingly probable. CNN learns this characteristic of BF as illustrated in Figure 2, where CNN successfully identifies the characteristics around the discrimination point $\widehat{i}$.

In this framework, the CNN can accurately classify the tests $i<\widehat{i}$ as belonging to ${\mathcal{H}}_0$ and the rest to ${\mathcal{H}}_1$. The method offers bounded FDR and FNR as n and m grow. As mentioned above, the joint control of FDR and FNR suggests that the area under the ROC curve is $AUC\to 1$.

Empirical validation supports $AUC\to 1$, which confirms the robustness and effectiveness of the model.

We also applied the transfer learning approach to the p-values by repeating the same analysis on BFs, but we do not have evidence of high $AUC$ as in the case of BFs.

4. Simulations and Real Examples

4.1. Training Dataset

The data set shown in Figure 1 is generated to address the classical statistical problem of testing the equivalence of means between two independent normal populations subject to heteroskedasticity. Specifically, we consider two independent populations, $X\sim \mathrm{Normal}({\mu }_X,{\sigma }_X^2)$ and $Y\sim \mathrm{Normal}({\mu }_Y,{\sigma }_Y^2)$, each with sample sizes $n_x$ and $n_y$, respectively. This example has been extensively detailed in [9,10], and additional information on computing the unscaled Bayes factor, $B_i$, is provided in Appendix A.

Figure 1 illustrates a sequence of Ws generated with parameters $n_x=n_y=10$, $m=10,000$, $m_0/m=0.9$, $m_1=1000$, and distribution parameters set as specified. CNN underwent 30 optimization epochs, with the training result presented in Figure 3.

The trained CNN achieves an accuracy slightly exceeding 96%, indicating that fewer than 4% of the tests are misclassified relative to $H_0$ and $H_1$ in this large sample.

Furthermore, to assess the advantage of using Ws over p-values for better hypothesis testing representation, as discussed in [9,10], we also trained the CNN on an ordered sequence of p-values, achieving comparable accuracy levels as shown in Figure 3.

In subsequent sections, we juxtapose our CNN-based approach with traditional methods commonly used in medical research. We compare the evidence derived from ordered p-values, obtained using Student’s t-test with Welch’s correction, to that obtained through our CNN model. These p-values are further adjusted using the BH FDR procedure, serving as our benchmark in the actual practice of MHT. Other procedures could have been considered [13,33,34], but keep in mind that, for instance, the BH procedure is also considered the limiting procedure of other approaches to MHT as the q-values when $m\to \infty$ [35]. Therefore, other approaches would not have added much to the exposed results.

We then evaluate these methods using the ROC curve to account for various experimental conditions in which different types of error may be of differing importance. The AUC serves as a summary metric to evaluate the precision in classifying the null and alternative hypothesis sets, ${\mathcal{H}}_0$ and ${\mathcal{H}}_1$, thus controlling the corresponding FDR and FNR.

4.2. Simulation Study

Employing the pre-trained CNN with varying input features—specifically Ws, p-values, and p-values adjusted according to the Benjamini–Hochberg (BH) scale—we systematically evaluate the average AUC through 1000 replications of the AUC. These replications are obtained under a composite set of scenarios designed to mirror various real-world conditions encountered in GWAS. The scenarios are delineated as follows.

• Signal Variation for Alternatives: The means for the alternative hypotheses, ${\mu }_{Y_i}$, are set to values in the set $\{1,2,3,4\}$ for $i\in {\mathcal{H}}_1$.
• Sample Size and Asymptotic Behavior: Both $n_x$ and $n_y$ are set to the value n, which ranges from 3 to 10.
• Heteroscedasticity: The standard deviations ${\sigma }_{X_i}$ and ${\sigma }_{Y_i}$ for $i=1,\dots ,m$ and $i\in {\mathcal{H}}_1$ vary in the set $\{1,2,3\}$.
• Proportion of True Alternatives: The ratio $m_0/m$ is adjusted to one of the following: 0.9, 0.95, or 0.99.
• Asymptotic Number of Tests: The total number of tests, m, is set to 1000 or 10,000.
• Test Dependence: Two schemes are considered, one with all independent tests and another with block-dependent tests. In block-dependent tests, there are tests belonging to a set that are dependent on them and independent of the others, and there are also different sets. These tests induced a block-diagonal correlation matrix among the test statistics (see [36]). In the block-dependent case, variable blocks $X_i$, $Y_i$ of sizes 2, 5, and 10 are formed, and their correlations are drawn randomly from a uniform distribution between −1 and 1, subject to the constraint of a positive semidefinite correlation matrix.

Figure 4 and Figure 5 present the statistical evaluation of the average AUC along with its 99.9% confidence intervals. These metrics are computed across 1000 replications and under various simulation scenarios explicitly enumerated in each figure’s axis labels and captions.

Consider the top-left panel of Figure 4 for illustrative purposes. It shows the average AUC and associated confidence intervals, conditioned on independent tests with parameters $m=1000$, $m_0/m=0.9$, ${\mu }_{Y_i}=1$ for $i\in {\mathcal{H}}_1$ and n ranging from 3 to 10. Color-coded markers represent the BH procedure (in red), CNN applied to Ws (in blue), and CNN also applied to p-values (in green). The results in this panel are marginal outcomes aggregated over 1000 replications and for all combinations of ${\sigma }_{X_i}$ and ${\sigma }_{Y_i}$. Figure 5 extends this analysis to include block-dependent tests, which are also conditioned on the sizes of these blocks.

The results of our simulation study provide compelling evidence that the proposed CNN demonstrates robust control over FDR and FNR in a diverse range of conditions. Remarkably, these performance metrics are better than those achieved using BH procedures. Furthermore, CNN’s efficacy is noticeably higher when trained on test statistics W than when trained on classical p-values.

The distinctive advantages of the proposed methodology become more pronounced in scenarios with weaker signals, characterized by lower means (${\mu }_Y$), greater variances, or smaller sample sizes (n). These disparities can be attributed to the fact that the p-values are calibrated asymptotically in n, and the number of samples is often limited, especially given the high costs associated with replications of RNA sequences. Interestingly, the selected range of m has a negligible impact on CNN performance. On the contrary, traditional MHT procedures, such as the BH method, control FDR asymptotically in m, which is evident through slight improvements in the ROC curve at higher values of m. In summary, the increased dependencies among tests further accentuate the benefits of employing the CNN-based approach over traditional methods.

4.3. Example RNA-Seq 1: Squamous Cell Carcinomas versus Matched Normal Tissue

We examine RNA-seq count data from a paired design study focusing on oral squamous cell carcinomas and the corresponding normal tissue samples from six patients [37].

The primary objective of the analysis is to identify genes that exhibit differential expression between tumor and normal tissue samples. To account for patient-specific variations, we employ a mixed-effects Bayesian Poisson regression model with flat improper priors for the Poisson regression coefficients. Additionally, an exceedingly vague prior is utilized for the logarithm of the random effects variance (refer to the Appendix C for complete details). It should be noted that the analytical framework employed here is substantially more flexible than the conventional negative binomial regression commonly used in RNA-seq count data analysis [37]. Furthermore, it diverges from the t test-based analysis that generated the training sample depicted in Figure 1.

In this experiment, we consider $m=10,512$ genes and only $n_x=n_y=3$ patients under each condition, making a total of six patients. For every gene i, the unscaled Bayes factor, $B_i$, is computed separately to evaluate the model incorporating tissue effects against the null model devoid of such effects. Both models incorporate patient-specific random effects. Using the trained Convolutional Neural Network (CNN) based on the data shown in Figure 1, we report in

Table 1. Probability of genes associated with Squamous cell carcinomas according to the CNN trained with W in Figure 1 and the BH procedure.

Gene:	TTN	KRT13	SPRR3	NEB	KRT4	ITGB4	PLEC
CNN, $\widehat{p}$:	0.999	0.890	0.813	0.788	0.762	0.700	0.582
Gene:	PTHLH	PTHLH	PTHLH	PTHLH	COL4A6	PTGFR	PTGFR
BH:	$8·{10}^{-17}$	$1·{10}^{-16}$	$1·{10}^{-16}$	$2·{10}^{-16}$	$8·{10}^{-15}$	$5·{10}^{-15}$	$4·{10}^{-15}$

the probabilities and genes that could be linked to carcinoma tissue. These results are presented in conjunction with the results obtained by the BH procedure, which tests the significance of the tissue effect coefficient.

Most of the remaining genes exhibit a probability of less than 50% of relevance to the condition under study. In particular, 658 genes produced adjusted p-values less than 0.001 according to the BH procedure, which is a very noisy result.

Interestingly, according to the proposed procedure, only the TTN and KRT genes had previously been identified in the study by [37]. Other genes such as SPRR [38], NEB [39], ITGB [40], and PLEC [41] have been subsequently associated with tumor conditions in the cited literature. This observation underscores the valuable insights that could be gleaned from the data if analyzed using our proposed approach. In contrast, genes highlighted by the BH method, such as PTHLH, COL4A6, and PTGFR, have only tangential associations with tumors. For example, PTHLH has been discussed in the context of cow tumors [42], again suggesting that the BH method may produce noisy results compared to the proposed one.

Unlike the situation described in Section 4.2, the ground truth in this case is unknown. However, our objective is to demonstrate that the proposed CNN methodology outperforms the BH approach. This is particularly noteworthy given that the Bayesian Poisson sampling model (see Appendix C) diverges substantially from the model used to generate Figure 1. This argument is further substantiated by an additional simulation study detailed in Appendix D. The suboptimal performance of the BH method is attributable to the misspecification of the negative binomial model [37] due to the inclusion of patient-specific random effects. Furthermore, the presence of a nuisance dispersion parameter adversely affects the reliability of p-values, since these are no longer calibrated with respect to the $U(0,1)$ distribution, thus compromising the efficacy of multiple hypothesis testing procedures such as BH [3].

4.4. Example RNA-Seq 2: Normal vs. Tumor Tissue

We reviewed the RNA-Seq data of Arabidopsis thaliana as discussed in [43]. The data set focuses on the plant’s response to the bacterium Pseudomonas syringae, a model organism for studying plant-pathogen interactions. The purpose of the analysis is to identify differentially expressed genes that illuminate how plants defend themselves against such pathogens.

Three Arabidopsis plants, each six weeks old, were treated with Pseudomonas syringae, while control plants were given a mock pathogen. Subsequently, total RNA was extracted from the leaves, resulting in three independent biological replicates. Each set, comprising $n_x=n_y=3$ RNA samples, was sequenced and RNA-Seq counts were collected for $m=13,930$ genes. Samples are not independent, and this requires an adjustment for time-dependent effects.

The Bayesian regression model used is analogous to the one described in Section 4.3. However, an additional layer of complexity is introduced by incorporating a first-order autoregressive process to account for temporal random effects (see Appendix E). As before, the BFs are unscaled due to improper priors on the model coefficients.

Our results, detailed in

Table A1. Probability of genes associated with defending Arabidopsis plants to Pseudomonas syringae according to the CNN trained with W in Figure 1. Genes with ${}^{*}$ have not been reported in [43] but in other studies.

Gene	$\widehat{\mathit{p}}$	Gene	$\widehat{\mathit{p}}$
AT1G76930	1.00	AT5G13220	0.75
AT4G22470	0.99	AT5G37600	0.74
AT4G12500	0.94	AT3G22120	0.74
AT2G45180	0.86	AT4G30190	0.73
AT5G64120	0.85	AT3G63160	0.72
AT2G43620	0.82	AT1G02930	0.70
AT4G38770	0.80	AT1G65845	0.69
AT5G54160	0.80	AT1G02920	0.69
AT2G10940	0.80	AT4G12470	0.68
AT1G67090	0.80	AT4G12480	0.67
*AT1G54410	0.80	AT3G46280	0.65
AT2G39200	0.79	AT4G10340	0.63
AT1G29930	0.77	*AT4G12800	0.63
AT4G12490	0.76	AT3G26740	0.58
AT5G54770	0.76	AT1G09560	0.56

of the Appendix F, highlight only 31 genes with a probability greater than 0.5 of having an interaction effect with Pseudomonas syringae. This contrasts starkly with the 387 genes identified in the original study by [43], which employed the q-values [33] to control FDR at 5%. The BH method identified as many as 1805 genes, reinforcing the notion of an inflated Type I error because of maybe-dependent tests. In particular, the FDR control using q-values has been shown to converge to the BH control [33] asymptotically, validating our use of the BH method as a benchmark. Interestingly, all but two of the 31 genes were previously reported in [43]. The two exceptions, AT4G12800 and AT1G54410, have been implicated in the response of the pathogen in subsequent studies [44,45].

5. Remarks

We alert the reader to three critical limitations associated with the use of CNNs, which also constitute the main theoretical drawbacks of this study: (i) The output of the neural network does not come with associated uncertainty measures; ($ii$) Due to its complex architecture and predictive focus, an exact interpretation of the trained CNN is elusive, although some insights can still be gleaned; ($iii$) Like any statistical model, the efficacy of an NN rests on the (untestable) assumption on the sampling model for tests that makes the training and testing samples exchangeable.

Looking at the usage of BFs in this work, we may think that there always exist proper priors that make $c{B}_i$, a proper BF for test i interpretable in the sense of providing the evidence for test i as suggested by Jeffreys, which is true but at the cost of introducing arbitrary c in the term. Then $c{B}_i$ is interpretable, but it is also arbitrarily interpretable as a measure of evidence due to the presence of c. In contrast, in the definition of W, based on the proper BFs $c{B}_i$, c simplifies and the same occurs when ordering tests according to the BFs $c{B}_i$. The problem of finding a cutoff on $c{B}_i$ is exactly that of fitting a CNN that allows one to fix a cutoff on the scale of $\widehat{\mathbf{p}}$ that considers the multiplicity of tests, as do some MHT procedures with adjusted p-values. We claim that the proposed scale on which $\widehat{\mathbf{p}}$ lies is much more interpretable as the direct probability that the test is observed under the null or alternative given the evidence from all tests.

The current methodology can also be implemented without explicit Bayesian computations. In particular, BFs, denoted $B_1,\dots ,B_m$, can be substitute calibrated p values according to the lower bound of the BF as elaborated in [46,47]. Specifically, for all $p{v}_i<exp(-1)$, the infimum of the BF for the ith test can be expressed as

$$c{B}_i\ge B_i=\left\{\begin{array}{cc}{[-ep{v}_{i}ln\left(p{v}_i\right)]}^{-1},\hfill & \mathrm{for}p{v}_i<exp(-1)\hfill \\ 1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(6)

where c symbolizes the calibration constant for the unknown true BF $c{B}_i$. This uncalibrated BF can then be utilized in our method after scaling the computed p-values by Equation (5). In the end, this allows us to generally extend the use of the proposed method to statistical analyses that are not per se Bayesian.

Our strategy relies on a singularly trained CNN. Future work could explore alternative architectures, such as bidirectional CNNs [48]. However, our existing CNN demonstrates remarkable performance in various settings, almost achieving an AUC close to 1, irrespective of the underlying sampling model specifically considered.

This innovative application of transfer learning [49] to MHT serves as the cornerstone of this study. The approach draws parallels with classical statistical techniques like the use of the Central Limit Theorem but within a computational context. The primary advantage lies in reusing CNN weights trained on one dataset (simulated and observed from calibration studies), as seen in Figure 1, to analyze different MHT problems. This approach essentially mirrors the untestable assumptions used in the BH procedure, such as positive regression dependence [1,50].

In summary, our results are promising for broader adoption of CNN-based strategies in MHT, especially given that the network performs consistently across divergent testing frameworks (e.g., t-tests, mixed-effect models, dependent/independent tests).