On Gap-Based Lower Bounding Techniques for Best-Arm Identification

Abstract

In this paper, we consider techniques for establishing lower bounds on the number of arm pulls for best-arm identification in the multi-armed bandit problem. While a recent divergence-based approach was shown to provide improvements over an older gap-based approach, we show that the latter can be refined to match the former (up to constant factors) in many cases of interest under Bernoulli rewards, including the case that the rewards are bounded away from zero and one. Together with existing upper bounds, this indicates that the divergence-based and gap-based approaches are both effective for establishing sample complexity lower bounds for best-arm identification.

1. Introduction

The multi-armed bandit (MAB) problem [1] provides a versatile framework for sequentially searching for high-reward actions, with applications including clinical trials [2], online advertising [3], adaptive routing [4], and portfolio design [5]. The best-arm identification problem seeks to find the arm with the highest mean using as few arm pulls as possible, and dates back to the works of Bechhofer [6] and Paulson [7]. More recently, several algorithms have been proposed for best-arm identification, including successive elimination [8], lower-upper confidence bound algorithms [9,10], PRISM [11], and gap-based elimination [12]. The latter establishes a sample complexity that is known to be optimal in the two-arm case [13], and more generally near-optimal.

Complementary to these upper bounds is information-theoretic lower bounds on the performance of any algorithm. Such bounds serve as a means to assess the degree of optimality of practical algorithms, and identify where further improvements are possible, thus focusing research towards directions that can have the greatest practical impact. Lower bounds were given by Mannor and Tsitsiklis [14] for Bernoulli bandits, and by Kaufmann et al. [15] for more general reward distributions. Both of these works were based on the difficulty of distinguishing bandit instances that differ in only a single arm distribution, but the subsequent analysis techniques differed significantly, with [14] using a direct change-of-measure analysis and introducing gap-based quantities equaling the difference between two arm means, and [15] using a form of the data processing inequality for KL divergence. We refer to these as the gap-based and divergence-based approaches, respectively. Further works on best-arm identification lower bounds include [16,17,18].

The divergence-based approach was shown in [15] to attain a stronger result than that of [14] with a simpler proof, as we outline in Section 2.2. In this paper, we address the question of whether the gap-based approach is fundamentally limited, or can be refined to attain a similar results to [15]. We show that the correct answer is the latter in many cases of interest, by suitable refinements of the analysis of [14]. The existing results and our results are presented in Section 2, and our analysis is presented in Section 3.

2. Overview of Results

2.1. Problem Setup

We consider the following setup:

• There are M arms with Bernoulli rewards; the means are $\mathbf{p}=(p_1,p_2,\cdots ,p_M)$, and this set of means is said to define the bandit instance. Our analysis will consider instances with arms sorted such that $p_1\ge p_2\cdots \ge p_M$, without loss of generality.
• The agent would like to find an arm whose arm mean is within $ϵ$ of the highest arm mean for some $0<\epsilon <1$, i.e., $p_l>p_1-\epsilon$. Even if there are multiple such arms, just identifying one of them is good enough.
• In each round, the agent can pull any arm $l\in \left[M\right]$ and observe an reward $X_l^{\left(s\right)}\sim Bernoulli\left(p_l\right)$, where s is the number of times the l-th arm has been pulled so far. We assume that the rewards are independent, both across arms and across times.
• In each round, the agent can alternatively choose to terminate and output an arm index $\widehat{l}$ believed to be $ϵ$-optimal. The index at which this occurs is denoted by T, and is a random variable because it is allowed to depend on the rewards observed. We are interested in the expected number of arm pulls (also called the sample complexity) ${\mathbb{E}}_{\mathbf{p}}\left[T\right]$ for a given instance $\mathbf{p}$, which should ideally be as low as possible.
• An algorithm is said to be $(\epsilon ,\delta )$-PAC (Probably Approximately Correct) if, for all bandit instances, it outputs an $\epsilon$-optimal arm with probability at least $1-\delta$ when it terminates at the stopping time T.

We will frequently make use of some fundamental quantities. First, the best arm mean and the gap to the best arm are denoted by

$$\begin{array}{cc}\hfill p_{*}& :=p_1,\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\Delta }_l& :=p_{*}-p_l.\hfill \end{array}$$

(2)

The set of $ϵ$-optimal arms and the set of $ϵ$-suboptimal arms are respectively given by

$$\mathcal{M}(\mathbf{p},\epsilon ):=\{l\in \left[M\right]:p_l>p_{*}-\epsilon \},$$

(3)

$$\mathcal{N}(\mathbf{p},\epsilon ):=\{l\in \left[M\right]:p_l\le p_{*}-\epsilon \},$$

(4)

and we make use of the binary KL divergence function

$$\mathrm{KL}(p,q):=plog\frac{p}{q}+(1-p)log\frac{1-p}{1-q},$$

(5)

where here and subsequently, $log(·)$ denotes the natural logarithm.

2.2. Existing Lower Bounds

For any fixed $\underline{p}\in (0,1/2)$, Mannor and Tsitsiklis [14] showed that if an algorithm is $(ϵ,\delta )$-PAC with respect to all instances with ${min}_l{p}_l\ge \underline{p}>0$, and if $ϵ\le \frac{1-p^{*}}{4}$ and $\delta \le e^{-8}/8$, then for any constant $\alpha \in (0,2)$, there exists $c_1=O\left({\underline{p}}^2\right)$ (depending on $\alpha$) such that

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathbf{p}}\left[T\right]\ge c_1\left[\frac{{\left(|\tilde{\mathcal{M}}(\mathbf{p},\epsilon )|-1\right)}^{+}}{{\epsilon }^2}+\sum _{l\in \tilde{\mathcal{N}}(\mathbf{p},\epsilon )}\frac{1}{{\Delta }_l^2}\right]log\frac{1}{8\delta }\end{array}$$

(6)

where

$$\begin{array}{c}\hfill \tilde{\mathcal{M}}(\mathbf{p},\epsilon )=\mathcal{M}(\mathbf{p},\epsilon )\cap \left\{l\in \left[M\right]:p_l\ge \frac{\epsilon +p_{*}}{2-\alpha }\right\},\end{array}$$

(7)

$$\begin{array}{c}\hfill \tilde{\mathcal{N}}(\mathbf{p},\epsilon )=\mathcal{N}(\mathbf{p},\epsilon )\cap \left\{l\in \left[M\right]:p_l\ge \frac{\epsilon +p_{*}}{2-\alpha }\right\}.\end{array}$$

(8)

Note that the subsets $\tilde{\mathcal{M}}(\mathbf{p},\epsilon )$ and $\tilde{\mathcal{N}}(\mathbf{p},\epsilon )$ do not always form a partition of the arms, i.e., it may hold that $\tilde{\mathcal{M}}(\mathbf{p},\epsilon )\cup \tilde{\mathcal{N}}(\mathbf{p},\epsilon )⊊\left[M\right]$. The sets increase in size as $\alpha$ decreases, but implicitly this leads to a lower value of $c_1$. In addition, as we will see below, the ${\underline{p}}^2$ dependence entering via $c_1$ is not necessary.

We also note that the lower bound in (6) depends on the instance-specific quantities $\tilde{\mathcal{M}}(\mathbf{p},\epsilon )$, $\tilde{\mathcal{N}}(\mathbf{p},\epsilon )$, and ${\Delta }_l$, and is thus an instance-dependent bound. On the other hand, the lower bound is only stated for $(\epsilon ,\delta )$-PAC algorithms, and the PAC guarantee requires the algorithm to eventually succeed on any instance (subject to the assumptions given on $p_l$, $ϵ$, and $\delta$).

Kaufmann et al. [15] improved Mannor and Tsitsiklis’s lower bound by using a form of data processing inequality for KL divergence, leading to the following whenever $\delta \le 0.15$ and $0<\epsilon <min\{p_{*},1-p_{*}\}$ [15] (Remark 5):

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathbf{p}}\left[T\right]\ge \left[\frac{\left|\mathcal{M}\right(\mathbf{p},\epsilon \left)\right|-1}{\mathrm{KL}(p_{*}-\epsilon ,p_{*}+\epsilon )}+\sum _{l\in \mathcal{N}(\mathbf{p},\epsilon )}\frac{1}{\mathrm{KL}(p_l,p_{*}+\epsilon )}\right]log\frac{1}{2.4\delta }.\end{array}$$

(9)

To directly compare this result with (6), it is useful to apply the following inequality [19] (Equation (2.8)):

$$\begin{array}{c}\hfill 2{(p-q)}^2\le \mathrm{KL}(p,q)\le \frac{{(p-q)}^2}{q(1-q)},\end{array}$$

(10)

which yields

$$\begin{array}{c}\hfill {\mathbb{E}}_{\mathbf{p}}\left[T\right]\ge (p^{*}+ϵ)(1-p^{*}-ϵ)\left[\frac{\left|\mathcal{M}\right(\mathbf{p},\epsilon \left)\right|-1}{4{ϵ}^2}+\sum _{l\in \mathcal{N}(\mathbf{p},\epsilon )}\frac{1}{{(\epsilon +{\Delta }_l)}^2}\right]log\frac{1}{2.4\delta }.\end{array}$$

(11)

Even this weakened bound can significantly improve on (6), since (i) $\mathcal{M}(\mathbf{p},\epsilon )\supset \tilde{\mathcal{M}}(\mathbf{p},\epsilon )$ and $\mathcal{N}(\mathbf{p},\epsilon )\supset \tilde{\mathcal{N}}(\mathbf{p},\epsilon )$, (ii) the ${\underline{p}}^2$ dependence is replaced by $(p^{*}+ϵ)(1-p^{*}-ϵ)$, so the dependence on the smallest arm mean is avoided (The $1-p^{*}-ϵ$ term is potentially small when $ϵ$ is close to $1-p^{*}$, but since (6) assumes $ϵ\le \frac{1-p^{*}}{4}$, we can still say that (11) is at least as good as (6)), and (iii) the assumption $ϵ\le \frac{1-p^{*}}{4}$ is avoided.

2.3. Our Result and Discussion

Our lower bound, stated in the following theorem, is developed based on Mannor and Tsitsiklis’s analysis for best-arm identification [14] (Theorem 1), but uses novel refinements of the techniques therein to further optimize the bound (see Appendix C for an overview of these refinements).

Theorem 1.

For any bandit instance $\mathbf{p}\in {(0,p_{*}]}^M$ with $p_{*}\in (0,1)$, and any $(\epsilon ,\delta )$-PAC algorithm with $0<\epsilon <1-p_{*}$ and $0<\delta <{\delta }_0$ for some ${\delta }_0<1/4$, we have

$$\begin{array}{cc}\hfill {\mathbb{E}}_{\mathbf{p}}\left[T\right]& \ge \frac{2{\gamma }_0(p_{*}+\epsilon )(1-p_{*}-\epsilon )}{7(\xi +1)}\left[\frac{\left|\mathcal{M}\right(\mathbf{p},\epsilon \left)\right|-1}{4{\epsilon }^2}+\sum _{l\in \mathcal{N}(\mathbf{p},\epsilon )}\frac{1}{{(\epsilon +{\Delta }_l)}^2}\right]log\frac{1+4{\delta }_0}{4\delta },\hfill \end{array}$$

(12)

where

$$\begin{array}{cc}\hfill {\gamma }_0& =\frac{1-4{\delta }_0}{8},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill \theta & =\frac{2\delta }{1-4{\gamma }_0}=\frac{4\delta }{1+4{\delta }_0},\hfill \end{array}$$

(14)

and $\xi >0$ is the unique positive solution of the following quadratic equation:

$$\begin{array}{c}\hfill 7{\gamma }_0{\xi }^{2}log\frac{1}{\theta }=3(\xi +1).\end{array}$$

(15)

Observe that this result matches (11) (with modified constants), and therefore exhibits the above benefit of depending on the full sets $\mathcal{M}$ and $\mathcal{N}$ without the condition $p_l\ge \frac{\epsilon +p_{*}}{2-\alpha }$ (see (7)–(8)), as well as avoiding the dependence on $\underline{p}$, and permitting the broadest range of $ϵ$ and $\delta$ among the above results.

The result (11) in turn matches (9) whenever the right-hand inequality in (10) is tight (i.e., whenever $\mathrm{KL}(p,q)=\mathsf{\Theta }\left(\frac{{(p-q)}^2}{q(1-q)}\right)$). This is clearly true when p and q (representing the arm means) are bounded away from zero and one, and also in certain limiting cases approaching these endpoints (e.g., when p and q both tend to one, but $\frac{1-p}{1-q}=\mathsf{\Theta }\left(1\right)$). However, there are also limiting cases where the upper bound in (10) is not tight (e.g., $p=1-\sqrt{\eta }$ and $q=1-\eta$ as $\eta \to 0$), and in such cases, the bound (9) remains tighter than that of Theorem 1.

3. Proof of Theorem 1

We follow the general steps of (Theorem 5 [14]), but with several refinements to improve the final bound. The main differences are outlined in Appendix C.

Step 1: Defining a Hypothesis Test

Let us denote the true (unknown) expected reward of each arm by $Q_l$ for all $l\in \left[M\right]$. Similarly to [14,15], we consider M hypotheses as follows:

$$\begin{array}{c}\hfill H_1:Q_l=p_l,\end{array}\begin{array}{c}\forall l\in \left[M\right],\end{array}$$

(16)

and for each $l\ne 1$,

$$\begin{array}{c}\hfill H_l:Q_l=p_{*}+\epsilon ,\end{array}\begin{array}{c}{Q}_{l^{\prime }}=p_{l^{\prime }}\end{array}\begin{array}{c}\forall l^{\prime }\in \left[M\right]\backslash \left\{l\right\}.\end{array}$$

(17)

If hypothesis $H_l$ is true, the $(ϵ,\delta )$-PAC algorithm must return arm l with probability at least $1-\delta$. We will bound sample complexity when the hypothesis $H_l$ is true. We denote by ${\mathbb{E}}_l$ and ${\mathbb{P}}_l$ the expectation and probability, respectively, under hypothesis $H_l$.

Let $B_l$ be the event that the algorithm returns arm l. Since ${\sum }_{l\in \mathcal{M}(\mathbf{p},\epsilon )}{\mathbb{P}}_1\left(B_l\right)\le 1$ and $\left|\mathcal{M}\right(\mathbf{p},\epsilon \left)\right|\ge 1$, there is at most one arm $l_0\in \mathcal{M}(\mathbf{p},\epsilon )$ that satisfies ${\mathbb{P}}_1\left(B_{l_0}\right)>\frac{1}{2}$. Defining

$$\begin{array}{c}\hfill {\mathcal{M}}_0(\mathbf{p},\epsilon ):=\left\{l\in \mathcal{M}(\mathbf{p},\epsilon ):{\mathbb{P}}_1\left[B_l\right]\le \frac{1}{2}\right\}=\left\{l\in \mathcal{M}(\mathbf{p},\epsilon ):l\ne l_0\right\},\end{array}$$

(18)

it follows that

$$\begin{array}{c}\hfill |{\mathcal{M}}_0{(\mathbf{p},\epsilon )|\ge (\left|\mathcal{M}(\mathbf{p},\epsilon )\right|-1)}^{+}.\end{array}$$

(19)

Define

$$\begin{array}{c}\hfill \mathcal{T}(\mathbf{p},\epsilon ):={\mathcal{M}}_0(\mathbf{p},\epsilon )\cup \mathcal{N}(\mathbf{p},\epsilon ),\end{array}$$

(20)

as well as

$$\begin{array}{c}\hfill B_{\mathcal{M}(\mathbf{p},\epsilon )}:=\bigcup _{l\in \mathcal{M}(\mathbf{p},\epsilon )}{B}_l,\end{array}$$

(21)

which is the event that the policy eventually select an arm in the $\epsilon$-neighborhood of the best arm in $\left[M\right]$. Since the policy is $(\epsilon ,\delta )$-correct with $\delta <{\delta }_0$, we must have

$$\begin{array}{c}\hfill {\mathbb{P}}_1\left[B_{\mathcal{M}(\mathbf{p},\epsilon )}\right]\ge 1-\delta >1-{\delta }_0,\end{array}$$

(22)

and it follows from (18) and (22) that

$$\begin{array}{cc}\hfill {\mathbb{P}}_1\left[B_l\right]& \le max\left\{{\delta }_0,1/2\right\}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & =\frac{1}{2}\hfill \end{array}$$

(24)

for all $l\in \mathcal{T}(\mathbf{p},\epsilon )$.

Step 2: Bounding the Number of Pulls of Each Arm

Before proceeding, we make some additional definitions:

$$\begin{array}{cc}{\alpha }_l\hfill & :=\frac{\epsilon +{\Delta }_l}{(1-p_l)p_l},\hfill \end{array}$$

(25)

$$\begin{array}{cc}{\beta }_l\hfill & :={\alpha }_l\sqrt{\frac{1-p_l}{1-(p_{*}+\epsilon )}},\hfill \end{array}$$

(26)

$$\begin{array}{cc}{\tilde{\alpha }}_l\hfill & :={\alpha }_l-\frac{4}{3}{\left({\alpha }_l(1-p_l)\right)}^2,\hfill \end{array}$$

(27)

$$\begin{array}{cc}{\tilde{\beta }}_l\hfill & :={\beta }_l-\frac{4}{3}{\left({\alpha }_l(1-p_l)\right)}^2,\hfill \end{array}$$

(28)

The definitions (27) and (28) will only be used for arms with $\frac{\epsilon +{\Delta }_l}{p_l}\le \frac{1}{2}$, and for such arms, we will establish in the analysis that ${\tilde{\alpha }}_l\ge 0$ and ${\tilde{\beta }}_l\ge 0$.

We prove the following lemma, characterizing the probability of a certain event in which (i) the number of pulls of some arm $l\in \mathcal{T}(\mathbf{p},\epsilon )$ falls below a suitable threshold (event $A_l$ below), (ii) a deviation bound holds regarding the number of observed 1’s from pulling arm l (event $C_l$ below), and (iii) arm l is not returned (event $B_l^c$).

Lemma 1.

For each $l\in \left[M\right]$, let $T_l$ be the total number of times that arm l is pulled under the $(\epsilon ,\delta )$-correct policy. Let $K_l=X_l^{\left(1\right)}+X_l^{\left(2\right)}+\cdots +X_l^{\left(T_l\right)}$ be the total number of unit rewards obtained from pulling the arm l up to the $T_l$-th time. Let

$$\begin{array}{c}{G}_{1,l}:=\frac{7p_l^2{\alpha }_l^2{(1-p_l)}^2}{2(p_{*}+\epsilon )\sqrt{(1-p_l)(1-(p_{*}+\epsilon ))}}{T}_l,\hfill \end{array}$$

(29)

$$\begin{array}{c}{G}_{2,l}:=\left[{\tilde{\beta }}_l(p_l{T}_l-K_l)\mathbf{1}\{p_l{T}_l>K_l\}+{\tilde{\alpha }}_l(p_l{T}_l-K_l)\mathbf{1}\{p_l{T}_l\le K_l\}\right]\mathbf{1}\left\{\frac{\epsilon +{\Delta }_l}{p_l}\le \frac{1}{2}\right\},\hfill \end{array}$$

(30)

where ${\alpha }_l$, ${\tilde{\alpha }}_l$, and ${\tilde{\beta }}_l$ are defined in (25), (27), and (28), respectively. Let

$$\begin{array}{c}\hfill {\nu }_l:=(\xi +1)\left(\sqrt{\frac{1-p_l}{1-p_{*}-\epsilon }}\right),\end{array}$$

(31)

where ξ is defined in (15). Define the following events:

$$\begin{array}{c}{A}_l:=\left\{G_{1,l}\le \frac{1}{{\nu }_l}log\frac{1}{\theta }\right\},\hfill \end{array}$$

(32)

$$\begin{array}{c}{C}_l=\left\{G_{2,l}\le \frac{\xi }{{\nu }_l}\left(\sqrt{\frac{1-p_l}{1-p_{*}-\epsilon }}\right)log\frac{1}{\theta }\right\},\hfill \end{array}$$

(33)

$$\begin{array}{c}{S}_l:=A_l\cap B_l^c\cap C_l.\hfill \end{array}$$

(34)

If $l\in \mathcal{T}(\mathbf{p},\epsilon )$ (see (20)), then under the condition that

$$\begin{array}{c}\hfill {\mathbb{E}}_1\left[G_{1,l}\right]<\frac{{\gamma }_0}{{\nu }_l}log\frac{1}{\theta },\end{array}$$

(35)

we have

$$\begin{array}{c}\hfill {\mathbb{P}}_1\left[S_l\right]>\frac{1-4{\gamma }_0}{2}.\end{array}$$

(36)

Proof. 

See Appendix A. □

Intuitively, $A_l$ is the event that the total number of times that arm l is pulled is small, and $C_l$ is the event that $|p_l{T}_l-K_l|$ is not too large (since pulling an arm $T_l$ times should produce roughly $p_l{T}_l$ ones). The lemma indicates that if $\mathbb{E}\left[T_l\right]$ is not too large, then $\mathbb{P}[A_l\cap B_l^c\cap C_l]$ is lower bounded, and this will ultimately lead to a lower bound on $\mathbb{P}\left[B_l^c\right]$, the event of primary interest.

In Lemma 2 below, we will use Lemma 1 to deduce a lower bound on ${\mathbb{E}}_1\left[G_{1,l}\right]$, which amounts to a lower bound on the average number of arm pulls by the definition of $G_{1,l}$. Before doing so, we introduce a likelihood ratio that will be used in a change-of-measure argument [14].

For any given time $t\ge 1$ and $l\in \left[M\right]$, let $T_l\left(t\right)$ be the total number of times that arm l is pulled by time t. Define

$$\begin{array}{c}\hfill X_l^{T_l\left(t\right)}:=\{X_l^{\left(1\right)},X_l^{\left(2\right)},\cdots ,X_l^{\left(T_l\left(t\right)\right)}\},\end{array}$$

(37)

and let

$$\begin{array}{c}\hfill {\mathcal{F}}_t:=\sigma (X_1^{T_1\left(t\right)},X_2^{T_2\left(t\right)},\cdots ,X_M^{T_M\left(t\right)})\end{array}$$

(38)

be the $\sigma$-algebra generated by $X_1^{T_1\left(t\right)},X_2^{T_2\left(t\right)},\dots ,X_M^{T_M\left(t\right)}$ for all $t=1,2,\dots$.

Recall that T is the stopping time of the algorithm, and that $T_l:=T_l\left(T\right)$ for all $l\in \left[M\right]$. Moreover, let $W={\mathcal{F}}_T$ be the entire history up to the stopping time T. We define the following likelihood ratio:

$$\begin{array}{c}\hfill L_l\left(w\right)=\frac{{\mathbb{P}}_l(W=w)}{{\mathbb{P}}_1(W=w)}\end{array}$$

(39)

for every possible history w. Moreover, we let $L_l\left(W\right)$ denote the corresponding random variable. Given the history up to time $T-1$ (i.e., ${\mathcal{F}}_{T-1}$), the arm reward at time T has the same probability distribution under $H_1$ and $H_l$ unless the chosen arm is arm l. Therefore, we have

$$\begin{array}{c}\hfill L_l\left(W\right)=\frac{{(p_{*}+\epsilon )}^{K_l}{(1-p_{*}-\epsilon )}^{T_l-K_l}}{p_l^{K_l}{(1-p_l)}^{T_l-K_l}},\end{array}$$

(40)

where $K_l:=X_l^{\left(1\right)}+X_l^{\left(2\right)}+\cdots +X_l^{\left(T_l\right)}$ (or the total number of 1’s in the $T_l$ pulls of the arm l).

The following proposition presents one of our key technical results towards establishing the lower bound. We use the definitions in (1)–(5), along with (25)–(28).

Proposition 1.

Fix the bandit instance $\mathbf{p}$, the parameter $0<\epsilon <1-p_{*}$, and the history W with corresponding values $K_l$ and $T_l$. Recalling the definitions of ${\alpha }_l$, ${\tilde{\alpha }}_l$, and ${\tilde{\beta }}_l$ in (25), (27), and (28), respectively, we have

$$\begin{array}{c}\hfill L_l\left(W\right)\ge exp\left(-G_{1,l}-G_{2,l}\right),\end{array}$$

(41)

where $G_{1,l}$ and $G_{2,l}$ are defined in (29)–(30).

Proof. 

See Appendix B. □

Based on Lemma 1 and Proposition 1, we obtain the following extension of [14] (Lemma 6) lower bounding the average of each $G_{1,l}$; this lower bound will later be translated to a lower bound on the number of arm pulls $T_l$.

Lemma 2.

For any arm $l\in \mathcal{T}(\mathbf{p},\epsilon )$, the following holds:

$$\begin{array}{c}\hfill {\mathbb{E}}_1\left[G_{1,l}\right]\ge \frac{{\gamma }_0}{{\nu }_l}log\frac{1}{\theta },\end{array}$$

(42)

where θ and ${\nu }_l$ are defined in (14) and (31), respectively.

Proof. 

We use a proof by contradiction. Assume that

$$\begin{array}{c}\hfill {\mathbb{E}}_1\left[G_{1,l}\right]<\frac{{\gamma }_0}{{\nu }_l}log\frac{1}{\theta },\end{array}$$

(43)

then by Lemma 1, Equation (36) holds. Moreover, by Proposition 1, we have

$$\begin{array}{c}\hfill L_l\left(W\right)\ge exp\left(-G_{1,l}-G_{2,l}\right),\end{array}$$

(44)

and recalling the definition of $S_l$ in (34), it follows from (44) that

$$\begin{array}{}\mathrm{(45)}& \hfill L_l\left(W\right){\mathbf{1}}_{S_l}& \ge exp\left(-G_{1,l}-G_{2,l}\right){\mathbf{1}}_{S_l}\hfill \mathrm{(46)}& & \ge exp\left(-\frac{1}{{\nu }_l}\left[\xi \left(\sqrt{\frac{1-p_l}{1-p_{*}-\epsilon }}\right)+1\right]log\frac{1}{\theta }\right){\mathbf{1}}_{S_l}\hfill \mathrm{(47)}& & \ge exp\left(-\frac{1}{{\nu }_l}\left[(\xi +1)\left(\sqrt{\frac{1-p_l}{1-p_{*}-\epsilon }}\right)\right]log\frac{1}{\theta }\right){\mathbf{1}}_{S_l}\hfill \end{array}$$

where (46) follows from the definitions in (32)–(33), and (47) follows from the fact that $1-p_l\ge 1-p_{*}\ge 1-p_{*}-\epsilon$ for all $l\in \left[M\right]$.

By the choice of ${\nu }_l>0$ given in (31), it holds that

$$\begin{array}{c}\hfill (\xi +1)\left(\sqrt{\frac{1-p_l}{1-p_{*}-\epsilon }}\right)\frac{1}{{\nu }_l}=1.\end{array}$$

(48)

Hence, from (47) and (48), we have

$$\begin{array}{c}\hfill L_l\left(W\right){\mathbf{1}}_{S_l}\ge \theta {\mathbf{1}}_{S_l}=\frac{2\delta }{1-4{\gamma }_0}{\mathbf{1}}_{S_l},\end{array}$$

(49)

for all $l\in \mathcal{T}(\mathbf{p},\epsilon )$, by the definition of $\theta$ in (14).

We are now ready to complete the proof:

$$\begin{array}{}\mathrm{(50)}& \hfill {\mathbb{P}}_l\left[B_l^c\right]& \ge {\mathbb{P}}_l\left[S_l\right]\hfill \mathrm{(51)}& & ={\mathbb{E}}_l\left[{\mathbf{1}}_{S_l}\right]\hfill \mathrm{(52)}& & ={\mathbb{E}}_1\left[L_l\left(W\right){\mathbf{1}}_{S_l}\right]\hfill \mathrm{(53)}& & \ge {\mathbb{E}}_1\left[\frac{2\delta }{1-4{\gamma }_0}{\mathbf{1}}_{S_l}\right]\hfill \mathrm{(54)}& & =\frac{2\delta }{1-4{\gamma }_0}{\mathbb{P}}_1\left[S_l\right]\hfill \mathrm{(55)}& & >\frac{2\delta }{1-4{\gamma }_0}\left(\frac{1-4{\gamma }_0}{2}\right)\hfill \mathrm{(56)}& & =\delta ,\hfill \end{array}$$

where (50) follows from the definition of set $S_l$ in (34), (52) follows by a standard change of measure [20], (53) follows from (49), and (55) follows from (36) of Lemma 1 (recall that we assumed (43)).

The inequality (56) shows a contradiction to the fact that under $H_l$, the $(\epsilon ,\delta )$-correct bandit policy must return the arm l with probability at least $1-\delta$, i.e., ${\mathbb{P}}_l\left(B_l^c\right)\le \delta$. This concludes the proof. □

From Lemma 2 and the definition of $G_{1,l}$ in (29), it holds that

$$\begin{array}{c}\hfill {\mathbb{E}}_1\left[\frac{7{\alpha }_l^2{p}_l^2{(1-p_l)}^2}{2(p_{*}+\epsilon )\sqrt{(1-p_l)(1-(p_{*}+\epsilon ))}}{T}_l\right]\ge \frac{{\gamma }_0}{{\nu }_l}log\frac{1}{\theta }\end{array}$$

(57)

for all $l\in \mathcal{T}(\mathbf{p},\epsilon )$. Hence, and using the definition of ${\nu }_l$ in (31), we have

$$\begin{array}{}\mathrm{(58)}& \hfill {\mathbb{E}}_1\left[{\alpha }_l^2{p}_l^2{(1-p_l)}^2{T}_l\right]& \ge \frac{2{\gamma }_0(p_{*}+\epsilon )\sqrt{(1-p_l)(1-p_{*}-\epsilon )}}{7(1+\xi )}\left(\sqrt{\frac{1-p_{*}-\epsilon }{1-p_l}}\right)log\frac{1}{\theta }\hfill \mathrm{(59)}& & =\frac{2{\gamma }_0(p_{*}+\epsilon )(1-p_{*}-\epsilon )}{7(1+\xi )}log\frac{1}{\theta }.\hfill \end{array}$$

Step 3: Deducing a Lower Bound on the Sample Complexity

For any arm $l\in \mathcal{T}(\mathbf{p},\epsilon )$, by the definition of ${\alpha }_l$ in (25), we have

$$\begin{array}{c}\hfill {\alpha }_l^2{p}_l^2{(1-p_l)}^2={(\epsilon +{\Delta }_l)}^2.\end{array}$$

(60)

Note that $0\le {\Delta }_l<\epsilon$ for all $l\in {\mathcal{M}}_0(\mathbf{p},\epsilon )$, since ${\mathcal{M}}_0\subseteq \mathcal{M}$, the set of $ϵ$-optimal arms. Therefore, we can further simplify (60) to

$$\begin{array}{c}\hfill {\alpha }_l^2{p}_l^2{(1-p_l)}^2\le 4{ϵ}^2\end{array}$$

(61)

for $l\in {\mathcal{M}}_0(\mathbf{p},\epsilon )$.

Substituting (60)–(61) into (59), we obtain

$$\begin{array}{}\mathrm{(62)}& \hfill {\mathbb{E}}_{\mathbf{p}}\left[T\right]& ={\mathbb{E}}_{\mathbf{p}}\left[\sum _{l=1}^M{T}_l\right]\hfill \mathrm{(63)}& & \ge {\mathbb{E}}_{\mathbf{p}}\left[\sum _{l\in {\mathcal{M}}_0(\mathbf{p},\epsilon )}{T}_l\right]+{\mathbb{E}}_{\mathbf{p}}\left[\sum _{l\in \mathcal{N}(\mathbf{p},\epsilon )}{T}_l\right]\hfill \mathrm{(64)}& & \ge \frac{2{\gamma }_0(p_{*}+\epsilon )(1-p_{*}-\epsilon )}{7(\xi +1)}\left[\frac{|{\mathcal{M}}_0(\mathbf{p},\epsilon )|}{4{\epsilon }^2}+\sum _{l\in \mathcal{N}(\mathbf{p},\epsilon )}\frac{1}{{(\epsilon +{\Delta }_l)}^2}\right]log\frac{1}{\theta }\hfill \mathrm{(65)}& & =\frac{2{\gamma }_0(p_{*}+\epsilon )(1-p_{*}-\epsilon )}{7(\xi +1)}\left[\frac{|{\mathcal{M}}_0(\mathbf{p},\epsilon )|}{4{\epsilon }^2}+\sum _{l\in \mathcal{N}(\mathbf{p},\epsilon )}\frac{1}{{(\epsilon +{\Delta }_l)}^2}\right]log\frac{1+4{\delta }_0}{4\delta },\hfill \end{array}$$

where (65) uses the definition of $\theta$ in (14). Finally, we obtain (12) from (65) and (19).

4. Conclusion

We have presented a refined analysis of best-arm identification following the gap-based approach of [14], but incorporating refinements that circumvent some weaknesses, leading to a bound matching the divergence-based approach [15] in many cases. It would be of interest to determine whether further refinements could allow this approach to match [15] in all cases, or the extent to which the gap-based approach extends beyond Bernoulli rewards and/or beyond the standard best-arm identification problem (e.g., to ranking problems [21]).