Aggregating Knockoffs for False Discovery Rate Control with an Application to Gut Microbiome Data

Abstract

Recent discoveries suggest that our gut microbiome plays an important role in our health and wellbeing. However, the gut microbiome data are intricate; for example, the microbial diversity in the gut makes the data high-dimensional. While there are dedicated high-dimensional methods, such as the lasso estimator, they always come with the risk of false discoveries. Knockoffs are a recent approach to control the number of false discoveries. In this paper, we show that knockoffs can be aggregated to increase power while retaining sharp control over the false discoveries. We support our method both in theory and simulations, and we show that it can lead to new discoveries on microbiome data from the American Gut Project. In particular, our results indicate that several phyla that have been overlooked so far are associated with obesity.

1. Introduction

Research on associations between the microbiome in the human gut and health and disease has surged in recent years [1,2,3,4]. Data on the microbiome are abundant in view of citizen science endeavors such as the American Gut Project [5], but these non-standard ways of data collection limit data quality.

Another difficulty in the analysis of microbiome data is the high dimensionality, which means that the number of parameters is large. The high-dimensionality is due to the diversity of the microbiome: at the phylum level, there are typically several dozen types of microbes; at the genus level, there are even several hundred types of microbes. Finding the microbes that are connected to a trait, therefore, requires the use of variable selection techniques. Consequently, in view of the low data quality and the high dimensionality, research on microbiome data is in particular need for controlling false discovery rates.

As a specific example where false discovery control can be important, we are interested in finding phyla and genera of microbes that are related to obesity. Successful detection of such groups of microbes could eventually lead to new means for weight control. We model the task as a variable selection problem in logistic regression with the obesity as the dependent variable and the (log-transformed) counts of the microbial relative abundancies as the variables. The number of parameters, that is, the number of potential phyla/genera, is large, while the number of actually relevant phyla/genera is assumed to be small; hence, we speak of sparse, high-dimensional logistic regression. A number of corresponding variable selection methods have been established, including especially ${\ell }_1$-penalized logistic regression [6], which can be equipped with knockoffs to do FDR control [7]. However, in our application, these standard pipelines perform insufficiently: for example, phyla that are known to be associated with obesity are missed, and few phyla are selected in the first place.

To increase the variable selection performance, we propose a simple aggregation scheme. It consists of two steps: the knockoff method is run k times with a decreasing FDR target level, and the selections are then combined. We show that this aggregation scheme keeps FDR control intact while improving power in practice.

Our three key contributions are:

• We introduce an aggregation scheme that provably retains the original methods’ guarantees—see Theorem 1.
• We show numerically that the aggregation can increase the original methods’ power—see Section 3.1 and Section 3.2.
• We show that the resulting pipelines for FDR control can be readily applied to empirical data and lead to new discoveries—see Section 3.3.

The remainder of the paper is organized as follows. In Section 2, we introduce our pipeline and establish its theory. In Section 3.1 and Section 3.2, we verify the accuracy of the pipeline in simulations. In Section 3.3, we then show the usefulness of the pipeline in selecting phyla and genera connected with obesity. In Section 4, we finally conclude with a brief discussion.

2. Methods and Theory

In this section, we introduce and study our aggregation scheme for knockoffs. We presume n independent data pairs $({\mathit{x}}_1,y_1),\cdots ,({\mathit{x}}_n,y_n)$, where each ${\mathit{x}}_i\in {\mathbb{R}}^p$ is a vector of variables and $y_i\in \mathbb{R}$ an outcome. We keep the relationship between the outcomes and variables completely general at this point—the relationship could be linear, logistic, or anything else—but we assume that this relationship is captured by a parameter $\mathit{\beta }\in {\mathbb{R}}^p$. Our target for inference is then the active set ${\mathcal{S}}^{\ast }:=\{j:{\beta }_j\ne 0\}$.

Two important quality measures for an estimate ${\widehat{\mathcal{S}}}_q[\mathit{X},\mathit{y}]$ of ${\mathcal{S}}^{\ast }$ are the FDR level

$$\mathrm{FDR}:=\mathbb{E}\left[\frac{|{\widehat{\mathcal{S}}}_q[\mathit{X},\mathit{y}]\backslash {\mathcal{S}}^{\ast }|}{|{\widehat{\mathcal{S}}}_q[\mathit{X},y]|\vee 1}\right]$$

and the power

$$\mathrm{power}:=\mathbb{E}\left[\frac{|{\widehat{\mathcal{S}}}_q[\mathit{X},\mathit{y}]\cap {\mathcal{S}}^{\ast }|}{|{\widehat{\mathcal{S}}}_q[\mathit{X},\mathit{y}]|\vee 1}\right],$$

where $|·|$ denotes the cardinality of a set. Our focus is on estimators that provide FDR control at level $q\in [0,1]$, that is,

$$\mathrm{FDR}\le q,$$

(1)

while having large power. In the following, we recall that the knockoff filter provides FDR control. We then equip the knockoff filter with an aggregation step to improve its power.

2.1. A Brief Introduction to the Knockoff Filter

The main ingredient of the knockoff method [7] is a "knockoff version" $\tilde{\mathit{X}}\in {\mathbb{R}}^{n\times p}$ of the design matrix $\mathit{X}$. This new matrix $\tilde{\mathit{X}}$ is essentially $\mathit{X}$ with additional noise such that (1) the estimator can distinguish predictors from $\mathit{X}$ and $\tilde{\mathit{X}}$ but (2) both design matrices still have a similar correlation structure. The idea is then to compare the selections of predictors in $\tilde{\mathit{X}}$ and in $\mathit{X}$ when estimating on the extended design $\left[\tilde{\mathit{X}}\mathit{X}\right]$.

Denoting $\tilde{\mathit{X}}={({\tilde{\mathit{x}}}_1,\dots ,{\tilde{\mathit{x}}}_n)}^{\top }$ and $\mathit{X}={({\mathit{x}}_1,\dots ,{\mathit{x}}_n)}^{\top }$, the underlying assumption is that ${\mathit{x}}_i\sim \mathcal{N}({\mathbf{0}}_p,\Sigma )$ for a positive definite matrix $\Sigma \in {\mathbb{R}}^{p\times p}$. The knockoffs are then generated according to [8]

$${\tilde{\mathit{x}}}_i|{\mathit{x}}_i\sim \mathcal{N}({\mathit{\mu }}_i,\mathit{V})\mathrm{for}\mathrm{each}i\in \{1,\dots ,n\}$$

(2)

with

$$\begin{array}{cc}\hfill {\mathit{\mu }}_i& :={\mathit{x}}_i-{\mathit{x}}_i{\Sigma }^{-1}\mathrm{diag}\left\{\mathit{a}\right\},\hfill \\ \hfill \mathit{V}& :=2\mathrm{diag}\left\{\mathit{a}\right\}-\mathrm{diag}\left\{\mathit{a}\right\}{\Sigma }^{-1}\mathrm{diag}\left\{\mathit{a}\right\},\hfill \end{array}$$

where $\mathit{a}\in {\mathbb{R}}^p$ is such that $\mathit{V}$ is positive definite.

The variable selection is then based on the estimator

$$\widehat{\mathit{\beta }}[\tau ,\mathit{X},\tilde{\mathit{X}},y]\in arg\underset{\mathit{\alpha }\in {\mathbb{R}}^{2p}}{min}\left\{l\left[\mathit{\alpha }\left|\right[\mathit{X}\tilde{\mathit{X}}],y\right]+\sum _{j=1}^{2p}{h}_{\tau }\left[{\alpha }_j\right]\right\},$$

(3)

where $l:{\mathbb{R}}^p\mapsto \mathbb{R}$ is a loss function, $\left[\mathit{X}\tilde{\mathit{X}}\right]\in {\mathbb{R}}^{n\times 2p}$ the extended design matrix, and $h_{\tau }:\mathbb{R}\mapsto [0,\infty )$ a penalty function with tuning parameter $\tau >0$. Examples include the lasso (where $l\left[\mathit{\alpha }\left|\right[\mathit{X}\tilde{\mathit{X}}],y\right]:=\left|\right|y-\left[\mathit{X}\tilde{\mathit{X}}\right]\mathit{\alpha }{\left|\right|}_2^2$ and $h_{\tau }\left[{\alpha }_j\right]:=\tau \left|{\alpha }_j\right|$) and the penalized logistic regression (where $l\left[\mathit{\alpha }\left|\right[\mathit{X}\tilde{\mathit{X}}],y\right]=-{\sum }_{i=1}^n(y_i{x}_i^{\top }\mathit{\alpha }-ln(1+exp\left(x_i^{\top }\mathit{\alpha }\right)))$ and $h_{\tau }\left[{\alpha }_j\right]:=\tau \left|{\alpha }_j\right|$).

The basic test statistics of the knockoff approach then capture the maximal tuning parameter of each variable entering the model:

$$\begin{array}{cc}\hfill Z_j[\mathit{X},y]& :=sup\left\{\tau :{\widehat{\mathit{\beta }}}_j[\tau ,\mathit{X},\tilde{\mathit{X}},y]\ne 0\right\}\hfill \\ \hfill {\tilde{Z}}_j[\mathit{X},y]& :=sup\left\{\tau :{\widehat{\mathit{\beta }}}_{p+j}[\tau ,\mathit{X},\tilde{\mathit{X}},y]\ne 0\right\}\hfill \end{array}$$

for $j\in \{1,\dots ,p\}$. We surpress the dependence on $[\mathit{X},y]$ in the following for notational ease. The final test statistic $W:={(W_1,\dots ,W_p)}^{\top }$ then combines the basic test statistics into

$$W_j:=max\{Z_j,{\tilde{Z}}_j\}·\mathrm{sign}(Z_j-{\tilde{Z}}_j)\mathrm{for}j\in \{1,\dots ,p\}.$$

This statistic compares how much earlier the original predictor enters the model as compared to the fake predictor—or the other way around. The threshold of the standard knockoff procedure for a given FDR level $q\in [0,1]$ is then

$$T_q:=min\left\{t\in \mathcal{W}:\frac{\#\{j:W_j\le -t\}}{\#\{j:W_j\ge t\}\vee 1}\le q\right\},$$

where $\mathcal{W}:=\left\{\right|W_j|:j\in \{1,\dots ,p\}\}$.

The knockoff procedure has another variant, called knockoff+, which differs in the threshold:

$$T_q^{+}:=min\left\{t\in \mathcal{W}:\frac{1+\#\{j:W_j\le -t\}}{\#\{j:W_j\ge t\}\vee 1}\le q\right\}.$$

These definitions finally yield the active sets

$$\begin{array}{cc}\hfill {\widehat{\mathcal{S}}}_q& :=\left\{j:W_j\ge T_q\right\},\hfill \\ \hfill {\widehat{\mathcal{S}}}_q^{+}& :=\left\{j:W_j\ge T_q^{+}\right\}.\hfill \end{array}$$

The active sets ${\widehat{\mathcal{S}}}_q^{+}$ indeed provide FDR control at level q, that is, satisfy inequality (1)—see [7], [Theorem 2]; the active sets ${\widehat{\mathcal{S}}}_q$ provide an approximate version of it—see [7], [Theorem 1].

2.2. Aggregating Knockoffs

We now introduce the aggregation scheme and its theory. The aggregation scheme applies the knockoff method k times and combines the results:

Step 1: Given a target FDR $q\in [0,1]$, choose a sequence $q_1,\dots ,q_k\in [0,1]$ such that $q={\sum }_{i=1}^k{q}_i$. Apply the standard knockoff (or knockoff+) procedure k times, once for each target FDR $q_i$, and denote the corresponding k-estimated active sets by ${\widehat{\mathcal{S}}}_{q_1},\dots ,{\widehat{\mathcal{S}}}_{q_k}$ (or ${\widehat{\mathcal{S}}}_{q_1}^{+},\dots ,{\widehat{\mathcal{S}}}_{q_k}^{+}$).

Step 2: Combine these k-estimated active sets by taking the union:

$${\widehat{\mathcal{S}}}_{q,AKO}\left[k\right]:={\cup }_{i=1}^k{\widehat{\mathcal{S}}}_{q_i}(\mathrm{or}{\widehat{\mathcal{S}}}_{q,AKO}^{+}\left[k\right]:={\cup }_{i=1}^k{\widehat{\mathcal{S}}}_{q_i}^{+}).$$

The intuition behind this scheme is that increasing the number of knockoffs stabilizes the outcome and improves the power. While applied here to the knockoff method, we emphasize that the aggregation scheme can be applied to every model and estimator as long as the there is a method for FDR control to start with. Hence, rather than the standard knockoffs as used below, we could equally well use model-X knockoffs [9], deep knockoffs [10], or KnockoffGAN [11].

On the other hand, the aggregation scheme retains the knockoffs’ theoretical guarantees:

Theorem 1.

Given a target FDR level $q\in [0,1]$, the set ${\widehat{\mathcal{S}}}_{q,AKO}^{+}\left[k\right]$ of the aggregation scheme for knockoff+ provides FDR control at level q:

$$\mathbb{E}\left[\frac{|{\widehat{\mathcal{S}}}_{q,AKO}^{+}\left[k\right]\backslash {\mathcal{S}}^{\ast }|}{|{\widehat{\mathcal{S}}}_{q,AKO}^{+}\left[k\right]|\vee 1}\right]\le q.$$

Similarly, the scheme retains the approximate FDR control of standard knockoffs—we will demonstrate this in our simulations.

Proof of Theorem 1.

The proof is—maybe suprisingly—simple. By Theorem 2 in [7], we have for all $q_i$, $i\in \{1,\dots ,k\}$,

$$\mathbb{E}\left[\frac{|{\widehat{\mathcal{S}}}_{q_i}^{+}\backslash {\mathcal{S}}^{\ast }|}{|{\widehat{\mathcal{S}}}_{q_i}^{+}|\vee 1}\right]\le q_i.$$

Hence,

$$\begin{array}{cc}\hfill \mathbb{E}\left[\frac{|{\widehat{\mathcal{S}}}_{q,AKO}^{+}\left[k\right]\backslash {\mathcal{S}}^{\ast }|}{|{\widehat{\mathcal{S}}}_{q,AKO}^{+}\left[k\right]|\vee 1}\right]& =\mathbb{E}\left[\frac{|\left({\cup }_{i=1}^k{\widehat{\mathcal{S}}}_{q_i}^{+}\right)\backslash {\mathcal{S}}^{\ast }|}{|{\cup }_{i=1}^k{\widehat{\mathcal{S}}}_{q_i}^{+}|\vee 1}\right]\hfill \\ \hfill & =\mathbb{E}\left[\frac{|{\cup }_{i=1}^k({\widehat{\mathcal{S}}}_{q_i}^{+}\backslash {\mathcal{S}}^{\ast })|}{|{\cup }_{i=1}^k{\widehat{\mathcal{S}}}_{q_i}^{+}|\vee 1}\right]\hfill \\ \hfill & \le \mathbb{E}\left[\sum _{i=1}^k\frac{|{\widehat{\mathcal{S}}}_{q_i}^{+}\backslash {\mathcal{S}}^{\ast }|}{|{\cup }_{i=1}^k{\widehat{\mathcal{S}}}_{q_i}^{+}|\vee 1}\right]\hfill \\ \hfill & \le \mathbb{E}\left[\sum _{i=1}^k\frac{|{\widehat{\mathcal{S}}}_{q_i}^{+}\backslash {\mathcal{S}}^{\ast }|}{|{\widehat{\mathcal{S}}}_{q_i}^{+}|\vee 1}\right]\hfill \\ \hfill & =\sum _{i=1}^k\mathbb{E}\left[\frac{|{\widehat{\mathcal{S}}}_{q_i}^{+}\backslash {\mathcal{S}}^{\ast }|}{|{\widehat{\mathcal{S}}}_{q_i}^{+}|\vee 1}\right]\hfill \\ \hfill & \le \sum _{i=1}^k{q}_i=q,\hfill \end{array}$$

as desired. □

For $k=1$, our method equals the standard knockoff (or knockoff+) procedure; in practice, we recommend $k\approx 5$ as a trade-off between computational effort and statistical effect. We also recommend $q_i=q/2^{i-1}$, which, strictly speaking, does not meet our assumptions on $q_1,\dots ,q_k$ exactly, but it works well empirically—see the simulations.

2.3. Other Approaches

While preparing this manuscript, two other ways of aggregating knockoffs were proposed [12,13]. While we use k knockoffs in k processes individually and aggregate at the end, they use multiple knockoffs simultaneously in one process. They can also show that their schemes provide valid FDR control, but we can argue that our approach is considerably simpler. We illustrate in the Appendix A.1 that we typically get more power than the multiple knockoffs method proposed by [12]; we have not gotten the scheme of [13] to run yet.

3. Simulations and a Real Data Analysis

In this section, we test our method empirically both on synthetic and on real data. We incorporate the specifics of microbiome data: First, in line with recent proposals for generalized linear models in this context [14], we study linear regression as well as logistic regression, and we transform count data with the standard log-transformation [15]. Second, since gut microbiome data tends to be correlated [16], we ensure that the synthetic data are also correlated. Third, since gut microbime data also tend to be zero-inflated, we replace zero values by 0.5 times the observed minimum abundance, which is standard approach in microbiome analysis [17,18]. We compare our modified aggregating knockoff pipeline applied to ${\ell }_1$-penalized regression (called AKO henceforth) with the standard knockoff pipeline (KO).

Throughout, we set $k=5$ and $q_i=q/2^{i-1}$. We also show the results for other choice of $q_1,\dots ,q_k$ in the Appendix A.2.

3.1. Simulation 1: Linear Regression

We first consider linear regression. The dimensions of the synthetic data are $(n,p)\in \left\{\right(200,100),(400,200\left)\right\}$. The rows ${\mathit{x}}_i$ of the design matrix are sampled i.i.d. from $\mathcal{N}({\mathbf{0}}_p,\Sigma )$ with the elements in $\Sigma$ satisfying ${\Sigma }_{ij}={\rho }^{|i-j|}$ with correlation factor $\rho =0.5$. The noise is drawn from $\mathbf{u}\sim \mathcal{N}(\mathbf{0},{\sigma }^2{\mathbf{I}}_n)$. The true parameter $\mathit{\beta }$ has 20 nonzero coefficients, which are selected uniformly at random from $\{1,\dots ,p\}$ and set to 1 before the entire vector is rescaled to have signal-to-noise ratio ${\left|\right|\mathit{X}\mathit{\beta }\left|\right|}_2^2/n{\sigma }^2=5$ with ${\sigma }^2=1$. The outcome $y$ is then generated by the linear model

$$y=\mathit{X}\mathit{\beta }+\mathit{u}.$$

The test statistic W is based on the lasso method as described in the preceeding section.

We compare the empirical FDR and power averaged over $r=100$ repetitions of the simulations for each method. The ideal result would be an average FDR of at most q and a power equal to 1.

The results in Figure 1a,b show that our pipeline retains the KO’s FDR control while increasing the power.

3.2. Simulation 2: Logistic Regression

We now consider logistic regression (the paper [19] was the first one to apply our scheme to logistic regression). The above settings remain the same except for the outcomes $y_i$ being generated by

$$Pr(y_i=1)=\frac{exp\left(x_i^{\top }\mathit{\beta }\right)}{1+exp\left(x_i^{\top }\mathit{\beta }\right)}\mathrm{for}i\in \{1,\dots ,n\}.$$

The results in Figure 2a,b show again that our pipeline retains the KO’s FDR control while increasing the power. For more simulations in various settings, please refer to Appendix A.3.

3.3. Influence of the Gut Microbiome on Obesity

A well-functioning gut microbiome is essential for health [2]; for example, there is strong evidence that the composition of the microbiome is connected to obesity [20,21,22]. Existing research about this connection has focused only on phyla of bacteria that are abundant in most guts, such as Actinobacteria, Bacteroidetes, Firmicutes, and Verrucomicrobia [3,23,24,25,26]. In the following analysis, in contrast, we include the microbiome in its entirety. Our findings suggest that also phyla beyond those ones mentioned in the literature are connected to obesity.

The data for our analysis are from the American Gut Project [5]. The scope is American adults with age between 20 and 69 and BMI between $15\mathrm{kg}/{\mathrm{m}}^2$ and $60\mathrm{kg}/{\mathrm{m}}^2$ in the collection up to January, 2018. This includes $n=8404$ subjects, of which 278 are underweight (uw; BMI below $18.5\mathrm{kg}/{\mathrm{m}}^2$), 4972 have normal weight (nor; 18.5–25$\mathrm{kg}/{\mathrm{m}}^2$), 2253 are overweight (ow; 25–30$\mathrm{kg}/{\mathrm{m}}^2$), and 901 are obese (ob; above $30\mathrm{kg}/{\mathrm{m}}^2$). We transform the data and deal with the zero-counts as decribed earlier. The total number of phyla in our scope is $p=55$.

The underlying model for these data is ${\ell }_1$-penalized logistic regression model with outcomes $y_i=1$ (ob) when BMI $\ge 30$ and $y_i=0$ (non-ob) otherwise. The target FDR level is $q=0.1$. Seven different groupings are considered to get the most out of the data: (i) all four groups (uw+nor+ow+ob), (ii) uw+ob, (iii) nor+ob, (iv) ow+ob, (v) uw+nor+ob, (vi) uw+ow+ob and (vii) nor+ow+ob. The AKO is applied with $\{q_i=q/2^{i-1}:i\in \{1,\dots ,k\}\}$.

The results in

Table 1. Selected bacterial phyla by our pipeline (AKO) and the original pipeline (KO) at FDR level $q=0.1$ for seven groupings. AKO consistently selects more phyla than KO.

(i) all		(ii) uw + ob
KO	AKO	KO	AKO
Actinobacteria	Actinobacteria	Actinobacteria	Actinobacteria
	Bacteroidetes
Cyanobacteria	Cyanobacteria		Cyanobacteria
			Firmicutes
Proteobacteria	Proteobacteria
	Spirochaetes
Synergistetes	Synergistetes		Synergistetes
Tenericutes	Tenericutes	Tenericutes	Tenericutes
	Verrucomicrobia
(iii) nor + ob		(iv) ow + ob
KO	AKO	KO	AKO
Actinobacteria	Actinobacteria		Actinobacteria
Bacteroidetes	Bacteroidetes
Cyanobacteria	Cyanobacteria	Cyanobacteria	Cyanobacteria
			Firmicutes
	Lentisphaerae
Proteobacteria	Proteobacteria		Proteobacteria
	Spirochaetes		Spirochaetes
Synergistetes	Synergistetes		Synergistetes
			TM7
Tenericutes	Tenericutes	Tenericutes	Tenericutes
	Verrucomicrobia
	Thermi
(v) uw + nor + ob		(vi) uw + ow + ob
KO	AKO	KO	AKO
Actinobacteria	Actinobacteria		Actinobacteria
Bacteroidetes	Bacteroidetes
Cyanobacteria	Cyanobacteria	Cyanobacteria	Cyanobacteria
			Firmicutes
	Lentisphaerae
Proteobacteria	Proteobacteria		Proteobacteria
	Spirochaetes		Spirochaetes
Synergistetes	Synergistetes		Synergistetes
			TM7
Tenericutes	Tenericutes	Tenericutes	Tenericutes
(vii) nor+ow+ob
KO	AKO
	Actinobacteria
	Bacteroidetes
Cyanobacteria	Cyanobacteria
Proteobacteria	Proteobacteria
	Spirochaetes
Synergistetes	Synergistetes
Tenericutes	Tenericutes
	Verrucomicrobia

show that our pipeline selects more phyla in general. Since the theory and the above similations suggest that both methods have similar FDR, the results indicate that our pipeline has more power. In particilar, phyla that are known to be connected with obesity, such as Actinobacteria, Bacteroidetes, Firmicutes, and Verrucomicrobia [3,23,24,25,26], are selected by AKO more often across the seven groupings.

The results also highlight Spirochaetes, which have not been associated with obesity in the literature, yet. The standard pipeline does not seem to have the power to select it, and similarly, two other known FDR control methods for microbiome data—the standard BH procedure [27] and the TreeFDR [28]—select Spirochaetes only for large FDR levels ($q\gtrsim 0.8$). (Futher comparisons with the BH procedure, TreeFDR, and the plain KO can be found in the Appendix A.4.) In contrast, our method selects Spirochaetes even at very small FDR levels (such as $q=0.01$), which strongly suggests a connection between Spirochaetes and obesity.

Our pipeline can, of course, also be applied to more detailed taxonomic ranks. As an illustration, we report the results of an application to genera for ALL—cf. (i) in Table 1—in

Table 2. Selected bacterial genera by our pipeline (AKO) and the original pipeline (KO) at FDR level $q=0.1$ for ALL—cf. (i) in Table 1. AKO selects more genera than the original KO.

Phylum	KO	AKO
Actinobacteria	Actinomyces	Actinomyces
	Collinsella	Collinsella
	Eggerthella	Eggerthella
Cyanobacteria	YS2	YS2
		Streptophyta
Firmicutes	Bacillus	Bacillus
		Lactobacillus
	Lactococcus	Lactococcus
		Clostridium
	Lachnospira	Lachnospira
	Ruminococcus	Ruminococcus
		Peptostreptococcaceae
	Acidaminococcus	Acidaminococcus
	Megasphaera	Megasphaera
		Mogibacteriaceae
	Erysipelotrichaceae	Erysipelotrichaceae
	Catenibacterium	Catenibacterium
Proteobacteria	RF32	RF32
	Haemophilus	Haemophilus
Tenericutes		RF39

. The data sampling is same with the phyla analysis. The only difference is that the total number of genera is $p=969$. The rest of results for other six different groupings are given in the Appendix B. We find correspondingly that AKO selects more genera than the standard KO.

4. Discussion

Our aggregation scheme for knockoffs is supported by theory (Section 2) and simulations (Section 3.1 and Section 3.2) and may lead to new discoveries in microbiomics (Section 3.3).

While we focus on a specific pipeline, our concept applies very generally. For example, it does not depend on the underlying statistical model or estimator but only on the availability of FDR control. In particular, the FDR control can be established via standard knockoffs or any other scheme. This flexibility is particularly interesting in practice: while the standard knockoffs rely on normality, other knockoff procedures, such as model-X knockoffs [9], deep knockoffs [10], and KnockoffGAN [11], allow for much more general designs. Hence, the standard knockoffs can be readily swapped for those procedures in our pipeline (without any changes to the methodology or theory) when indicated in an application.

Considerably later than our paper has been put online, two other papers on the topic have also been put online [29,30]—apparently without being aware of our manuscript. A comparison to those results would also be of interest.