Data-Dependent Weighted E-Value Aggregation for Fusion Learning

Abstract

We propose a data-dependent weighted e-value aggregation framework for synthesizing discoveries across partially overlapping studies. The key idea is to convert within study p-value-based multiple testing results into e-values and aggregate them using data-dependent leave-one-out weights, thereby mitigating the power loss associated with naive averaging. We show that applying the e-Benjamini–Hochberg procedure to the aggregated e-values yields finite-sample control of the global false discovery rate under standard conditions. Simulation studies and real-data analyses demonstrate the effectiveness and practical advantages of the proposed methods.

1. Introduction

In modern applications with high dimensional data, investigators routinely test many hypotheses simultaneously to detect signals. This multiplicity requires rigorous control of error rates, especially the false discovery rate (FDR) [1], to ensure valid statistical inference.

The Benjamini–Hochberg (BH) procedure [1] is the most widely used method for FDR control. Recently, Barber and Candes [2] introduced the Barber–Candès (BC) procedure, which estimates the number of false discoveries by exploiting the symmetry of null p-values and achieves FDR control under suitable independence conditions. In the recent literature, a growing line of work advocates e-values as convenient tools for multiple testing. An e-value is a non-negative random variable $e\ge 0$ with expectation at most one under the null hypothesis. There is increasing interest in using e-values to control the FDR in both offline and online settings [3,4,5]. In particular, Wang and Ramdas [6] proposed the e–Benjamini–Hochberg (e-BH) procedure, which applies the BH rule to the inverses of e-values, and proved that it controls the FDR under arbitrary dependence among e-values. Recent work by Li and Zhang [7] establishes an equivalence between several p-value-based methods (including BH and BC) and e-BH when appropriate e-values are constructed; equivalence here means the procedures yield identical rejection sets.

In this paper, we consider J studies that test overlapping subsets of a total of m hypotheses. Study j tests $m_j$ hypotheses, where $m_j\le m$. This partial-overlap regime arises naturally in fusion learning from multiple data sources and has been studied in recent work [8,9,10]. Examples include multicenter clinical trials [11], multicohort genomics [12,13], and federated A/B testing [14]. The goal is to synthesize evidence across studies into a unified set of discoveries while controlling the global FDR.

Principled aggregation is motivated by practical constraints. Studies often examine different subsets of hypotheses, vary in sample size and signal strength, and face data sharing restrictions that provide only summary statistics such as vectors of p-values. Naive aggregation of study specific findings performs poorly in practice. For example, taking the intersection of rejection sets is typically too conservative, whereas taking their union can inflate the global FDR because error control within each study does not automatically imply control of the overall FDR.

Since p-values are the most widely used quantities in multiple testing, we assume that each study applies a p-value-based procedure. Direct aggregation of p-values is difficult, so motivated by Li and Zhang [7], we map each study’s results to e-values and then aggregate across studies using e-values. However, how to combine e-values across studies is a central design choice. A common approach is simple averaging [15], which underlies several recent aggregation methods [7,16]. Ren and Barber [16] propose derandomized knockoffs that aggregate via averaged e-values, while Banerjee et al. [9] introduce an Integrative Ranking and Thresholding (IRT) framework that aggregates only binary decisions. However, our simulations show that naive averaging can substantially reduce power. Motivated by Li and Zhang [17], we adapt leave-one-out (LOO) ideas to construct data-dependent weights for combining e-values across studies. These weights enhance power while preserving global FDR control.

In this work, we propose two e-value-based fusion methods tailored to overlapping studies. The first, BH-based fusion, runs BH within each study, maps its outputs to e-values, combines these e-values across studies using data-dependent weights, and then applies e-BH globally. The second, BC-based fusion, runs BC within each study and proceeds analogously. Our main theoretical results establish finite-sample FDR control for e-BH applied to the aggregated e-values produced by our weighting schemes.

The remainder of the paper is organized as follows. Section 2 reviews preliminaries on FDR, multiple testing procedures, and e-values. Section 3 presents our data-dependent weighted e-value aggregation method for fusion learning. Section 4 reports simulation results, and Section 5 illustrates the method on real data. Section 6 concludes. All proofs and additional numerical results are deferred to the Appendixes Appendix A and  Appendix B.

2. Preliminaries

2.1. Basic Setup and False Discovery Rate

Consider m hypotheses $\mathcal{M}:=\{H_1,\dots ,H_m\}$, indexed by $\left[m\right]:=\{1,\dots ,m\}$. Let $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_m)\in {\{0,1\}}^m$ denote the ground truth, where ${\theta }_i=0$ if $H_i$ is a true null and ${\theta }_i=1$ otherwise. Define the index sets of true nulls and non-nulls as ${\mathcal{H}}_0:=\{i\in \left[m\right]:{\theta }_i=0\}$ and ${\mathcal{H}}_1:=\{i\in \left[m\right]:{\theta }_i=1\}$, respectively. A decision rule is a vector $\delta =({\delta }_1,\dots ,{\delta }_m)\in {\{0,1\}}^m$, where ${\delta }_i=1$ indicates rejection of $H_i$ and ${\delta }_i=0$ indicates non-rejection. Throughout, “non-null” and “under the alternative” are used interchangeably.

The false discovery rate (FDR) associated with a decision rule $\delta$ is defined as the expectation of the false discovery proportion (FDP):

$$\mathrm{FDR}\left(\delta \right)=\mathbb{E}\left[\mathrm{FDP}\left(\delta \right)\right],\mathrm{where}\mathrm{FDP}\left(\delta \right)={\displaystyle \frac{{\sum }_{i=1}^m(1-{\theta }_i){\delta }_i}{1\vee {\sum }_{i=1}^m{\delta }_i}},$$

with $a\vee bmax\{a,b\}$. The convention $1\vee {\sum }_{i=1}^m{\delta }_i$ prevents division by zero when no hypotheses are rejected. An FDR-controlling procedure guarantees that $\mathrm{FDR}\left(\delta \right)\le \alpha$ for a pre-specified level $\alpha \in (0,1)$.

We consider a multi-study setting in which the m hypotheses in $\mathcal{M}$ are tested across J studies. Study j tests a subset ${\mathcal{M}}_j\subseteq \mathcal{M}$ with index set ${\mathcal{I}}_j\subseteq \left[m\right]$, so that ${\mathcal{M}}_j=\{H_i:i\in {\mathcal{I}}_j\}$. Let $m_j\left|{\mathcal{I}}_j\right|$; studies may overlap, hence ${\bigcup }_{j=1}^J{\mathcal{M}}_j=\mathcal{M}$ and ${\sum }_{j=1}^J{m}_j\ge m$. For each hypothesis i, let ${\mathcal{N}}_i\subseteq \left[J\right]$ denote the set of study indices that test $H_i$ : ${\mathcal{N}}_i=\{j\in \left[J\right]:i\in {\mathcal{I}}_j\}$, and $n_i\left|{\mathcal{N}}_i\right|$.

For $i\in {\mathcal{I}}_j$, let ${\delta }_{j,i}\in \{0,1\}$ denote study j’s decision for $H_i$, with ${\delta }_{j,i}=1$ indicating rejection. The results of study j are collected in the decision vector ${\delta }_j{\left({\delta }_{j,i}\right)}_{i\in {\mathcal{I}}_j}\in {\{0,1\}}^{m_j}.$ For study j, the false discovery rate is defined as

$$\mathrm{FDR}\left({\delta }_j\right)=\mathbb{E}\left[\mathrm{FDP}\left({\delta }_j\right)\right],\mathrm{where}\mathrm{FDP}\left({\delta }_j\right)={\displaystyle \frac{{\sum }_{i\in {\mathcal{I}}_j}(1-{\theta }_i){\delta }_{j,i}}{1\vee {\sum }_{i\in {\mathcal{I}}_j}{\delta }_{j,i}}}.$$

For each study j, let ${\alpha }_j\in (0,1)$ be the pre-specified FDR level for which the guarantee $\mathrm{FDR}\left({\delta }_j\right)\le {\alpha }_j$ holds. Our objective is to synthesize the evidence from all J studies to produce a combined set of rejections for the hypotheses in $\mathcal{M}$, while controlling the overall FDR at a user-specified level $\alpha$.

2.2. Multiple Testing Procedures

Suppose we observe m p-values $p_1,p_2,\dots ,p_m$ corresponding to the hypotheses $H_1,H_2,\dots ,H_m$. Li and Zhang [7] showed that several widely used multiple-testing procedures can be expressed in the following unified form. Define

$$T=sup\left\{t\in \mathcal{T}:{\displaystyle \frac{S\left(t\right)}{1\vee {\sum }_{i=1}^m{R}_i\left(t\right)}}\le \alpha \right\},$$

(1)

where $\mathcal{T}$ is the admissible threshold domain, $S\left(t\right)$ is a conservative estimator of the number of false rejections at threshold t, and $R_i\left(t\right)$ indicates whether hypothesis $H_i$ should be rejected at threshold t. The rule selects the largest threshold T such that the estimated FDP doesn’t exceed target FDR level $\alpha$.

To instantiate an FDR-controlling procedure within this framework, one specifies $S\left(t\right)$ and $R_i\left(t\right)$. The classical Benjamini–Hochberg (BH) procedure [1] uses

$$S\left(t\right)=mt,R_i\left(t\right)=\mathbb{1}\{p_i\le t\},$$

which yields $\mathrm{FDR}\le \alpha$ under independence or certain positive dependence conditions of p-values. The Barber–Candès (BC) procedure [2], a model-free approach that exploits the symmetry of null p-values, sets

$$S\left(t\right)=1+\sum _{i=1}^m\mathbb{1}\{p_i\ge 1-t\},R_i\left(t\right)=\mathbb{1}\{p_i\le t\}.$$

Both procedures fit into (1) by pairing a data-driven, conservative estimate $S\left(t\right)$ of false discoveries with the candidate rejection set $\{i:R_i\left(t\right)=1\}$ and then selecting the largest threshold t for which the estimated FDP does not exceed $\alpha$.

2.3. E-Values, E-BH and Equivalence to Multiple Testing Procedures

A non-negative random variable e is called an e-value if, under the null hypothesis, it satisfies $\mathbb{E}\left[e\right]\le 1$. Suppose we observe m e-values $e_1,e_2,\dots ,e_m$ corresponding to hypotheses $H_1,H_2,\dots ,H_m$. The $\alpha$-level e–Benjamini–Hochberg (e-BH) procedure [6] proceeds by sorting the e-values in decreasing order, $e_{\left(1\right)}\ge e_{\left(2\right)}\ge \dots \ge e_{\left(m\right)}$, and rejecting the hypotheses associated with the $\widehat{k}$ largest e-values, where

$$\widehat{k}:=max\left\{i\in \{1,\dots ,m\}:e_{\left(i\right)}\ge {\displaystyle \frac{m}{i\alpha }}\right\}.$$

A key advantage of e-BH, established in Theorem 2 of Wang and Ramdas [6], is that it controls the FDR at level $\alpha$ under arbitrary dependence among the e-values, provided the following condition holds.

Proposition 1

(Wang and Ramdas [6], Theorem 2). Suppose the e-values ${\left\{e_i\right\}}_{i=1}^m$ satisfy

$$\sum _{i\in {\mathcal{H}}_0}\mathbb{E}\left[e_i\right]\le m.$$

(2)

Then the e-BH procedure controls the FDR at level α.

In multiple testing applications, the requirement $\mathbb{E}\left[e\right]\le 1$ under the null in the definition of e-values can be relaxed. Throughout this paper, we will refer to ${\left\{e_i\right\}}_{i=1}^m$ as a set of e-values whenever they satisfy Condition (2).

Given the threshold T in (1), define e-values by $e_i=\left(m{R}_i\left(T\right)\right)/S\left(T\right)$ for $1\le i\le m$. Then, applying the e-BH procedure to ${\left\{e_i\right\}}_{i=1}^m$ is equivalent to the original multiple-testing procedure based on the same functions $S(·)$ and $R_i(·)$ [7]. Here, equivalence means that both methods produce exactly the same set of rejections.

3. Data-Dependent Weighted E-Value Aggregation for Fusion Learning

In this paper, leveraging the equivalence between e-BH and other multiple-testing procedures, we use e-values as a bridge to synthesize evidence across all J studies and to construct a combined rejection set while controlling the overall FDR at a user-specified level $\alpha$.

Let $e_{j,i}$ denote the e-value for the ith hypothesis in the jth study. We define the combined e-value for the ith hypothesis as

$$e_i=\sum _{j\in {\mathcal{N}}_i}{w}_{j,i}{e}_{j,i},$$

(3)

where $w_{j,i}\ge 0$ are combination weights. A simple choice is $w_{j,i}=1/n_i$, in which case $e_i$ is the average of the individual e-values across studies that test hypothesis i. It is well known that the average of e-values is again an e-value [15], and averaging is widely used to combine e-values across studies. For example, Ren and Barber [16] use this idea proposing the derandomized knockoff method, Li and Zhang [7] propose averaging e-values to combine results, and Banerjee et al. [9] adopt averaging in fusion learning. However, our simulations indicate that such a naive aggregation can reduce power. Motivated by Li and Zhang [17], we therefore propose to use data-dependent weights to aggregate e-values in the fusion-learning setting, with the goal of improving power while maintaining FDR control. The data-dependent weights are constructed using a leave-one-out analysis, a key technical tool used to establish FDR control for the BH and BC procedures [1,18].

3.1. Fusion Learning Based on the BH Procedure

Let ${\mathbf{p}}_j=(p_{j,1},\dots ,p_{j,m_j})$ denote the vector of p-values from the jth study, and let $T_j$ be the BH rejection threshold in that study. Following Li and Zhang [7], define the e-value for the ith hypothesis in the jth study by

$$e_{j,i}={\displaystyle \frac{1}{T_j}}\mathbb{1}\{p_{j,i}\le T_j\},i\in {\mathcal{M}}_j.$$

With these e-values, the e-BH procedure at the same target level reproduces the BH rejection set in the jth study.

To construct data-dependent weights, let ${\mathbf{p}}_{j,-i}$ be the vector obtained from ${\mathbf{p}}_j$ by setting its ith entry to 0. Viewing the BH threshold as a function of ${\mathbf{p}}_j$, let $T_{j,i}$ denote the threshold obtained by applying BH to ${\mathbf{p}}_{j,-i}$. Define the data-dependent weights by

$$w_{j,i}=\left\{\begin{array}{cc}0,\hfill & j\notin {\mathcal{N}}_i,\hfill \\ {\displaystyle \frac{T_{j,i}}{T_{j,i}+{\sum }_{l\ne j,l\in {\mathcal{N}}_i}{T}_{l,i}}},\hfill & j\in {\mathcal{N}}_i.\hfill \end{array}\right.$$

(4)

Following the discussion in Li and Zhang [17], to make more rejections, we should assign larger weights to e-values associated with relatively large leave-one-out thresholds. Consequently, the weight $w_{j,i}$ is designed to increase when the leave-one-out threshold $T_{j,i}$ is large relative to the corresponding thresholds from other studies.

The following theorem establishes finite-sample FDR control for the e-BH procedure applied to the combined e-values in (3) with data-dependent weights in (4).

Theorem 1.

In each study, suppose the null p-values are mutually independent, satisfy super-uniformity, and are independent of the nonnull p-values. Then the e-values defined in (3) with weights given by (4) satisfy Condition (2). Consequently, the corresponding e-BH procedure controls the overall FDR in finite samples.

Proof. 

See in the Appendix A.    □

Although our data-dependent weighting scheme is inspired by Li and Zhang [17], our setting differs from theirs. Whereas Li and Zhang [17] focus on combining outputs from different testing methods, we aggregate e-values across multiple studies. Specifically, Li and Zhang [17] study multi-procedure aggregation within a single study: multiple procedures (or tuning choices) are applied to the same vector of p-values and their e-values are combined. In contrast, we consider multi-study aggregation: a fixed procedure is applied across distinct studies, each providing its own study specific vector of p-values, with heterogeneity in sample size and signal strength. Consequently, unlike the equal length, perfectly aligned e-value vectors in the multi-procedure setting, the e-value vectors we aggregate across studies can differ in length and index sets because different studies test different subsets of hypotheses. This lack of alignment induces missingness and indexing issues that complicate aggregation and motivate our study level weighting and combination rules designed to remain valid under such heterogeneity.

3.2. Fusion Learning Based on the BC Procedure

Besides the BH procedure, another well-known method is the BC procedure [2], which exploits symmetry to estimate the FDR. Let ${\mathbf{p}}_j=(p_{j,1},\dots ,p_{j,m_j})$ denote the vector of p-values from the jth study, and, with a slight abuse of notation, let $T_j$ be the BC rejection threshold in that study. In practice, we assume that all studies use the same multiple-testing procedure, so this notation is unambiguous regardless of the method employed. Following Li and Zhang [7], Ren and Barber [16], define the e-value for the ith hypothesis in the jth study by

$$e_{j,i}={\displaystyle \frac{m_j\mathbb{1}\{p_{j,i}\le T_j\}}{1+{\sum }_{l=1}^{m_j}\mathbb{1}\{p_{j,l}\ge 1-T_j\}}}.$$

With these e-values, the e-BH procedure at the same target level reproduces the BC rejection set in the jth study.

To define data-dependent weights under the BC framework, we adopt the following convention (with a slight abuse of notation) for this section: let ${\mathbf{p}}_{j,-i}$ be the same vector as ${\mathbf{p}}_j$, except that its ith element is replaced by $min\{p_{j,i},1-p_{j,i}\}$. Viewing the BC threshold $T_j$ as a function of ${\mathbf{p}}_j$, let $T_{j,i}$ denote the threshold obtained by applying the BC procedure to ${\mathbf{p}}_{j,-i}$. The data-dependent weights are then defined as

$$w_{j,i}=\left\{\begin{array}{cc}0,\hfill & j\notin {\mathcal{N}}_i,\hfill \\ {\displaystyle \frac{\frac{m}{m_j}\left(1+{\sum }_{l\in {\mathcal{M}}_j,l\ne i}\mathbb{1}\{p_{j,l}\ge 1-T_j\}\right)}{1+{\sum }_{l\in {\mathcal{M}}_j,l\ne i}\mathbb{1}\{p_{j,l}\ge 1-T_j\}+{\sum }_{s\ne j}{\sum }_{l\in {\mathcal{M}}_s}\mathbb{1}\{p_{s,l}\ge 1-T_{s,l}\}}},\hfill & j\in {\mathcal{N}}_i.\hfill \end{array}\right.$$

(5)

For $J=2$ and all studies test m hypotheses, the weights simplify to

$$\begin{array}{cc}\hfill w_{1,i}& ={\displaystyle \frac{1+{\sum }_{l\ne i}\mathbb{1}\{p_{1,l}\ge 1-T_1\}}{1+{\sum }_{l\ne i}\mathbb{1}\{p_{1,l}\ge 1-T_1\}+{\sum }_{l=1}^m\mathbb{1}\{p_{2,l}\ge 1-T_{2,l}\}}},\hfill \\ \hfill w_{2,i}& ={\displaystyle \frac{1+{\sum }_{l\ne i}\mathbb{1}\{p_{2,l}\ge 1-T_2\}}{{\sum }_{l=1}^m\mathbb{1}\{p_{1,l}\ge 1-T_{1,l}\}+1+{\sum }_{l\ne i}\mathbb{1}\{p_{2,l}\ge 1-T_2\}}}.\hfill \end{array}$$

The following theorem establishes finite-sample FDR control for the e-BH procedure applied to the combined e-values in (3) with data-dependent weights in (5), provided that

$$P(p_{j,i}\le a)\le P(p_{j,i}\ge 1-a)=P(1-p_{j,i}\le a),\mathrm{for}\mathrm{all}0\le a\le 0.5,i\in {\mathcal{H}}_0,j\in {\mathcal{N}}_i.$$

(6)

is satisfied. Condition (6) is the usual mirror-conservativeness assumption: under the null, large p-values are at least as likely as small p-values in a symmetric fashion. This ensures that the BC-type estimator does not underestimate the number of false discoveries.

Theorem 2.

Suppose the null p-values are mutually independent, satisfy (6), and are independent of the alternative p-values. Additionally, assume that, for each hypothesis, the p-values are independent across studies. Then the e-values defined in (3) with weights given by (5) satisfy Condition (2). Consequently, the corresponding e-BH procedure controls the overall FDR in finite samples.

Proof. 

See in Appendix A.    □

In our aggregation framework, each hypothesis i may be tested in multiple studies. For the same hypothesis $H_i$, under the BH procedure, our analysis provides FDR control without requiring independence among $p_{j,i}$ and $p_{s,i}$ for $j\ne s$ with $j,s\in {\mathcal{N}}_i$, which enhances its practical applicability. In contrast, the BC procedure does require independence between $p_{j,i}$ and $p_{s,i}$. This difference arises from distinct leave-one-out arguments used in the two proofs. When this assumption fails, the weights must be revised to

$$w_{j,i}=\left\{\begin{array}{cc}0,\hfill & j\notin {\mathcal{N}}_i,\hfill \\ {\displaystyle \frac{\frac{m}{m_j}\left(1+{\sum }_{l\in {\mathcal{M}}_j,l\ne i}\mathbb{1}\{p_{j,l}\ge 1-T_j\}\right)}{1+{\sum }_{l\in {\mathcal{M}}_j,l\ne i}\mathbb{1}\{p_{j,l}\ge 1-T_j\}+{sup}_{q\in [0,1]}{\sum }_{s\ne j}{\sum }_{l\in {\mathcal{M}}_s}\mathbb{1}\{p_{s,l}\ge 1-T_{s,l,i}^q\}},}\hfill & j\in {\mathcal{N}}_i.\hfill \end{array}\right.$$

(7)

where $T_{s,l,i}^q$ denotes the BC threshold obtained from the p-value vector ${\mathbf{p}}_s$ after replacing $p_{s,l}$ with $min\{p_{s,l},1-p_{s,l}\}$ and setting $p_{s,i}=q$. The following theorem establishes finite-sample FDR control for the e-BH procedure applied to the combined e-values in (3) with the data-dependent weights in (7).

Theorem 3.

Suppose the null p-values are mutually independent, satisfy (6), and are independent of the alternative p-values. Then the e-values defined in (3) with weights given by (7) satisfy Condition (2). Consequently, the corresponding e-BH procedure controls the overall FDR in finite samples.

Proof. 

See in Appendix A.    □

The construction of $w_{j,i}$ involves taking a supremum over $q\in [0,1]$, which is crucial for the proof. This step ensures that $w_{j,i}$ is independent of all $p_{l,i}$ with $l\in {\mathcal{N}}_i$, a property used in the analysis. The idea of defining weights via supremum has appeared elsewhere—for instance, in the $\tau$-censored weighted BH procedure of Zhao and Zhou [19]. However, our construction differs in goal. The $\tau$-censored weighted BH procedure is a variant of the weighted BH method that uses a leave-one-out technique to define the weights, whereas our method is built upon the BC procedure and employs weights solely to combine e-values across studies.

In our simulation studies, we observed that the weights defined in (5) empirically control the FDR even if there is dependence structure between studies. Furthermore, these weights are computationally more efficient than those defined in (7), which require solving an optimization problem without a closed-form solution. Accordingly, we recommend non-supremum weights as the default in routine applications, reserving supremum-based weights for settings where the independence assumption is untenable and formal guarantees under potential dependence are essential.

4. Numerical Studies

In this section, we conduct simulation studies to evaluate the performance of the proposed method. We adopt the simulation framework of Banerjee et al. [9] to test the hypotheses $H_{0i}:{\mu }_i=0$ versus $H_{1i}:{\mu }_i\ne 0$ or ${\mu }_i>0$. For any $a\in \mathbb{R}$, let $\delta \left(a\right)$ denote a point mass at a. In each simulation setting, study j conducts $m_j$ hypothesis tests based on data $X_{ij}$. The distributions of ${\mu }_i$ and $X_{ij}$ will be specified below.

Throughout all settings, we fix the total number of hypotheses at $m=1000$, consistent with the simulation design of Banerjee et al. [9]. We also consider $m=500$ and $m=1500$; the results are qualitatively similar to those for $m=1000$, so we defer the details to Appendix B.1 For the target FDR level, we set the nominal significance level to $\alpha =0.05$, and let ${\alpha }_j=0.025$ for all j. The choice of ${\alpha }_j$ influences the power of the combined tests but, by construction, does not affect the overall FDR controlled at level $\alpha$. The choice of conservative FDR across studies follows earlier e-value aggregation work [16]. A detailed discussion of the choice of ${\alpha }_j$ is provided in Appendix B.2. In brief, to ensure adequate power one should choose ${\alpha }_j<\alpha$; if ${\alpha }_j$ is too large, the proposed method can have essentially zero power. In practice, we recommend ${\alpha }_j=\alpha /2$.

In each simulation setting, we conduct 500 replicates and report the average FDP and the expected number of true positives (ETP). The average FDP serves as an estimator of the FDR; in all our settings, the FDR-controlling method should control the FDR at or below $\alpha =0.05$. For a fixed target FDR level, a higher ETP indicates better performance.

We denote by eBH-BH the proposed method that (i) applies the BH procedure within each study, (ii) converts the resulting per-hypothesis evidence to e-values, and (iii) aggregates these e-values across studies using the weights in (4). The variant that uses the BC procedure within each study with aggregated weights (5) is denoted by eBH-BC. In practice, neither BH nor BC uniformly dominates each other; their relative performance depends on distributional characteristics among the p-values. For a detailed comparison of BH and BC, see Arias-Castro and Chen [20].

We compare our methods against state-of-the-art alternatives from the Integrative Ranking and Thresholding (IRT) framework of Banerjee et al. [9], namely IRT, IRT^∗, and IRT-H. A key advantage of these IRT-based methods is that they operate using only the binary decision vectors ${\delta }_j$ from each study and do not require access to the exact per-hypothesis p-values; see Banerjee et al. [9] for methodological and implementation details. As an additional baseline, we consider a Fisher-combination approach that pools p-values from all studies via Fisher’s method [21], followed by BH correction. We refer to this method as Fisher hereafter.

• Setting 1.

In Setting 1, we consider J independent studies. We would like to test the hypotheses $H_{0i}:{\mu }_i=0$ versus $H_{1i}:{\mu }_i\ne 0$, where the effect sizes are generated from the mixture distribution ${\mu }_i\sim 0.8\delta \left(0\right)+0.1\mathcal{N}(3,1)+0.1\mathcal{N}(-3,1)$. For study $j\in \{1,\dots ,J\}$ and hypothesis $i\in \{1,\dots ,m_j\}$, we generate

$$X_{j,i}\mid {\mu }_i,{\sigma }_j\stackrel{\mathrm{ind}.}{\sim }\mathcal{N}({\mu }_i,{\sigma }_j^2),{\sigma }_j\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }\mathrm{Unif}(0.75,2).$$

We set $m_j=m$ for all j and vary the number of studies $J\in \{5,6,\dots ,10\}$. When the null distribution of the test statistic is correctly specified and the resulting p-values are mutually independent, Fisher’s combination test is expected to have higher power than the competing methods. Additionally, in this setting, all methods have theoretical FDR control.

Figure 1 displays the average FDP and ETP across methods. As expected, the Fisher method outperforms the alternatives. Relative to the IRT-based method, the proposed eBH-BC and eBH-BH achieve better performance, with eBH-BH outperforming eBH-BC in this setting. Among the IRT-based procedures, IRT^∗ performs best, whereas the standard IRT procedure exhibits nearly zero power.

• Setting 2.

In Setting 2, we allow dependence across $J=2$ studies and evaluate methods using one-sided p-values. Specifically, we test $H_{0i}:{\mu }_i=0$ versus $H_{1i}:{\mu }_i>0$, where ${\mu }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }0.8\delta \left(0\right)+0.2\mathcal{N}(3,1)$. Conditional on ${\mu }_i$, the study-level statistics are jointly Gaussian: for each i,

$${(X_{1,i},\dots ,X_{d,i})}^{\top }\sim \mathcal{N}({\mu }_i{\mathbf{1}}_d,\Sigma ),{\Sigma }_{jj}={\sigma }_j^2=1,{\Sigma }_{jk}=\rho \mathrm{for}j\ne k,$$

i.e., $\mathrm{Corr}(X_{j,i},X_{k,i})=\rho$ for all $j\ne k$. We vary $\rho$ over a grid of equally spaced values in $[-0.9,0.9]$.

Figure 2 reports the average FDP and ETP across methods. In this setting, Fisher, eBH-BC, and IRT^∗ do not have theoretical FDR guarantees. Empirically, eBH-BC and IRT^∗ control the FDR, whereas Fisher exhibits FDR inflation, especially for large $\rho$; accordingly, we omit some points of Fisher from the FDP plot because its average FDP often exceeds the y-axis limit of 0.1. In terms of power, eBH-BH and IRT perform similarly, with eBH-BC slightly less powerful. Notably, in this setting IRT outperforms IRT^∗, in contrast to Setting 1.

• Setting 3.

In Setting 3, the data are generated as in Setting 1 with ${\sigma }_j=1$ for all j, and $J=5$, but we allow the study-specific numbers of tested hypotheses $m_j$ to vary across the J studies. We fix the maximum across studies at $m_{max}=max\{m_1,\dots ,m_d\}=900$ and define the imbalance ratio $\zeta =min\{m_1,\dots ,m_d\}/m_{max}\in (0,1]$. For a given $\zeta$, we draw $m_1,\dots ,m_d$ independently and uniformly from the integers in $[\lceil \zeta m_{max}\rceil ,m_{max}]$. Given these $m_j$’s, for each study j we then select $m_j$ hypotheses uniformly at random without replacement from the m total hypotheses.

Figure 3 reports the average FDP and ETP across methods. In this independent setting, all methods have theoretical FDR control, and Fisher again attains the highest power. Among the IRT-based methods, only IRT^∗ achieves nontrivial power, whereas IRT and IRT_H exhibit negligible power. For the proposed methods, eBH-BH and eBH-BC perform similarly and both outperform IRT^∗.

• Setting 4.

In Setting 4, we consider a scenario in which the study-specific p-values are conservative, under which eBH-BC is more stable and achieves higher power than eBH-BH. Specifically, we set $m_j=m$ for all j and $J=5$. For study j and hypothesis i, the summary statistics satisfy $X_{j,i}\sim \mathcal{N}({\mu }_i,1)$, where the effects ${\mu }_i$ are i.i.d. from the mixture $0.8\delta \left(0\right)+0.2\mathcal{N}(3,1)$. We test $H_{0i}:{\mu }_i=0$ versus $H_{1i}:{\mu }_i>0$, and compute p-values as $p_{ij}=\Phi (X_{j,i}/\sigma )$ for $\sigma \in \{1.1,1.2,1.3,1.4,1.5\}$. Thus, as $\sigma$ increases, the p-values become more conservative. To induce dependence across studies, we set $Cov(X_{j,i},X_{k,i})=0.5$ for all $j\ne k$ (for each fixed i).

Figure 4 reports the average FDP and ETP across methods. In this dependent setting, Fisher and eBH-BC lack theoretical FDR guarantees. While eBH-BC empirically control the FDR, Fisher exhibits inflation. Therefore, Fisher results exceeding the 0.1 y-axis limit are excluded from the FDP plot. From Setting 2, the IRT method performed best among IRT-based procedures; hence, we display only IRT here. In this setting, eBH-BC performs best, and its power does not decrease as the p-values become more conservative. By contrast, the power of the other methods decreases with increasing conservativeness. Additionally, eBH-BH outperforms IRT in all cases.

5. Real Data Analysis

We illustrate the proposed methods using $J=4$ publicly available microarray studies of human prostate cancer. These include datasets from Welsh et al. [22] ($m_1=8799$), Singh et al. [23] ($m_2=8798$), Nanni et al. [24] ($m_3=$ 12,688), and Wallace et al. [25] ($m_4=9703$). These datasets were also utilized in Banerjee et al. [9]. For each study $j\in \{1,\dots ,J\}$, the observed data consist of an expression matrix $X_j$ with dimension $m_j\times d_j$, where $m_j$ denotes the number of genes profiled and $d_j$ denotes the sample size. Each sample is labeled as either case or control. The hypothesis for each gene i within a study is a two-group test for differential expression. After merging features across platforms and mapping probe sets to genes, we obtained a total of $m=$ 16,779 unique genes.

Our objective is to apply the proposed methods to produce a unified list of rejected genes, thereby aggregating evidence across studies to improve the reliability of the results. Before analysis, we merged features across platforms by gene symbol, removed probes with missing or duplicated mappings, and applied standard normalization as implemented in the limma pipeline. In each study, we first employed the limma package [26] to fit gene-wise linear models and compute vectors of p-values. We set the target FDR level for each individual study at $0.05$ and the overall target FDR level at $0.1$. We implemented the proposed methods, eBH-BH and eBH-BC, and compared them against IRT and P2E. The P2E algorithm is an e-value-based method implemented in Banerjee et al. [9]; we refer readers to that work for implementation details.

The results of the analysis are presented in

Table 1. Number of rejections for different methods across the real data sets.

Method	IRT	P2E	eBH-BH	eBH-BC
Number of Rejections	0	1494	3123	6314

. In this setting, IRT yielded no discoveries. Both proposed eBH-based methods significantly outperformed the P2E method by identifying a larger number of differentially expressed genes. We also observed that eBH-BC generated more rejections than eBH-BH. This is likely because the signal in these datasets is dense (with over $1/3$ of hypotheses rejected by eBH-BC). In such dense signal settings, the BC procedure is often outperformed than the BH procedure, a phenomenon also observed in Arias-Castro and Chen [20] and Li and Zhang [17].

6. Conclusions

Motivated by Li and Zhang [17], we propose a data-dependent weighting framework for aggregating e-values to synthesize discoveries across partially overlapping studies. By combining within-study e-values, the method produces study-integrated e-values that leverage information across studies. Extensive simulations indicate that data-dependent weighting substantially increases power relative to naive averaging, while maintaining finite-sample control of the FDR across a range of realistic scenarios.

Several avenues merit further investigation. First, beyond the BH and BC procedures, extending the framework to additional multiple-testing paradigms, including structure-adaptive methods, would broaden its applicability. Second, developing aggregation strategies that accommodate settings in which different studies implement heterogeneous within-study procedures is a natural next step. Third, building on the equivalence established in Li and Zhang [7], our current construction assigns a fixed e-value to each rejected hypothesis within a study; designing p-to-e calibrations that preserve more information from the original p-values while maintaining e-value validity remains an important direction for future work.