Thompson Sampling for Non-Stationary Bandit Problems

Abstract

Non-stationary multi-armed bandit (MAB) problems have recently attracted extensive attention. We focus on the abruptly changing scenario where reward distributions remain constant for a certain period and change at unknown time steps. Although Thompson sampling (TS) has shown success in non-stationary settings, there is currently no regret bound analysis for TS with uninformative priors. To address this, we propose two algorithms, discounted TS and sliding-window TS, designed for sub-Gaussian reward distributions. For these algorithms, we establish an upper bound for the expected regret by bounding the expected number of times a suboptimal arm is played. We show that the regret upper bounds of both algorithms are $\tilde{O}(\sqrt{T{B}_T})$, where T is the time horizon and $B_T$ is the number of breakpoints. This upper bound matches the lower bound for abruptly changing problems up to a logarithmic factor. Empirical comparisons with other non-stationary bandit algorithms highlight the competitive performance of our proposed methods.

1. Introduction

MAB is a classic sequential decision problem. At each time step, the learner selects an arm from a finite set of arms (also known as actions) based on its past observations, and it only observes the reward of the chosen action. The learner’s goal is to maximize its expected cumulative reward or minimize the regret incurred during the learning process. The regret is defined as the difference between the expected reward of the optimal arm and the expected reward achieved by the MAB algorithm.

MAB has found practical use in various scenarios, with one of the earliest applications being the diagnosis and treatment experiments proposed by Robbins [1]. In this experiment, each patient’s treatment plan corresponds to an arm in the MAB problem, and the goal is to minimize the patient’s health loss by making optimal treatment decisions. Recently, MAB has gained wide-ranging applicability. For example, MAB algorithms have been used in online recommendation systems to improve user experiences and increase engagement [2,3,4]. Similarly, MAB has been employed in online advertising campaigns to optimize the allocation of resources and maximize the effectiveness of ad placements [5]. While the standard MAB model assumes fixed reward distributions, real-world scenarios often involve changing distributions over time. For instance, in online recommendation systems, the collected data gradually become outdated, and user preferences are likely to evolve [6]. This dynamic nature necessitates the development of algorithms that can adapt to these changes, leading to the exploration of non-stationary MAB problems.

In recent years, there has been much research on non-stationary multi-armed bandit problems. These methods can be roughly divided into two categories: they either actively detect changes in the reward distribution using change-point detection algorithms [7,8,9,10,11], or they passively reduce the effect of past observations [12,13,14,15]. Ghatak [16], Alami and Azizi [17] use the active algorithm for non-stationary settings, which combines change-detection and TS. Viappiani [18], Gupta et al. [19], Cavenaghi et al. [20] also address the non-stationary problem with TS algorithm. However, they are experimental paper without theoretical analysis. Liu et al. [21] propose a novel sampling method-predictive sampling. They use information theory tools to analyze the Bayesian regret of their method.

The active methods need to make some assumptions about the change in arms distribution to ensure the effectiveness of the change-point detection algorithm. For instance, refs. [7,8] require a lower bound on the amplitude of change in each arm’s expected rewards. The passive method requires fewer assumptions about the characteristics of the change. They often use a sliding window or discount factor to forget past information to adapt to the change in arms distribution.

However, TS with a passive method has received little theoretical analysis of regret in non-stationary MAB problems. Raj and Kalyani [13] have studied the discounted Thompson sampling with Beta priors. While they only derive the probability of picking a suboptimal arm for the simple case of a two-armed bandit. To the best of our knowledge, only sliding-window Thompson sampling with Beta priors [14] provides the regret upper bounds. However, their proof is not correct. Recently, Fiandri et al. [22] have corrected their proof errors by using the techniques proposed in [23].

Our contributions are as follows: we propose discounted TS (DS-TS) and sliding-window TS (SW-TS) with uninformative priors for abruptly changing settings. We adopt a unified method to analyze the regret upper bound for both algorithms. The theoretical analysis results show that their regret upper bounds are of order $\tilde{O}(\sqrt{T{B}_T})$, where T is the number of time steps, $B_T$ is the number of breakpoints. This regret bound matches the $\Omega (\sqrt{T})$ lower bound proven by Garivier and Moulines [12] in an order sense. We also verify the algorithms in various environmental settings with Gaussian and Bernoulli rewards, and both DS-TS and SW-TS achieve competitive performance.

2. Related Works

Many works are based on the idea of forgetting past observations. Discounted UCB (DS-UCB) [12,24] uses a discounted factor to average the past rewards. In order to achieve the purpose of forgetting information, the weight of the early reward is smaller. Garivier and Moulines [12] also propose the sliding-window UCB (SW-UCB) by only using a few recent rewards to compute the UCB index. They calculate the regret upper bound for DS-UCB and SW-UCB as $\tilde{O}(\sqrt{T{B}_T})$. EXP3.S, as proposed in [25], has been shown to achieve the regret upper bound by $\tilde{O}(\sqrt{T{B}_T})$. Under the assumption that the total variation of the expected rewards over the time horizon is bounded by a budget $V_T$, Besbes et al. [26] introduce REXP3 with regret $\tilde{O}(T^{2/3})$. Combes and Proutiere [27] propose the SW-OSUB algorithm, specifically for the case of smoothly changing with an upper bound of $\tilde{O}({\sigma }^{1/4}T)$, where $\sigma$ is the Lipschitz constant of the evolve process. Raj and Kalyani [13] propose the discounted Thompson sampling for Bernoulli priors without providing the regret upper bound. They only calculate the probability of picking a sub-optimal arm for the simple case of a two-armed bandit. Trovo et al. [14] propose the sliding-window Thompson sampling algorithm with regret $\tilde{O}(T^{{\displaystyle \frac{1+\alpha }{2}}})$ for abruptly changing settings and $\tilde{O}(T^{\beta })$ for smoothly changing settings. Baudry et al. [15] propose a novel algorithm named Sliding-Window Last Block Subsampling Duelling Algorithm (SW-LB-SDA) with regret $\tilde{O}(\sqrt{T{B}_T})$. They assume that the reward distributions belong to the same one-parameter exponential family for all arms during each stationary phase. This means that SW-LB-SDA is not applicable to Gaussian reward distributions with unknown variance.

There are also many works that exploit techniques from the field of change detection to deal with reward distributions varying over time. Mellor and Shapiro [28] combine a Bayesian change point mechanism and Thompson sampling strategy to tackle the non-stationary problem. Their algorithm can detect global switching and per-arm switching. Liu et al. [7] propose a change-detection framework that combines UCB and a change-detection algorithm named CUSUM. They obtain an upper bound for the average detection delay and a lower bound for the average time between false alarms. Cao et al. [8] propose M-UCB, which is similar to CUSUM but uses another simpler change-detection algorithm. M-UCB and CUSUM are nearly optimal, their regret bounds are $\tilde{O}(\sqrt{T{B}_T})$.

The above works assume that the rewards distribution are bounded except for SW-LB-SDA. We assume the rewards distribution is a subGaussian distribution, which is a more general setting that includes both bounded distributions and Gaussian distributions.

Recently, there are some works that derive regret bounds without knowing the number of changes. For example, Auer et al. [9] propose an algorithm called ADSWITCH with optimal regret bound $\tilde{O}(\sqrt{B_{T}T})$. Suk and Kpotufe [29] improve the work [9] so that the obtained regret bound is smaller than $\tilde{O}(\sqrt{ST})$, where S only counts the best arms switches. There are also some studies investigating non-stationary representation learning in bandit problems [30,31]. Their focus is mainly on sequential representation learning and introducing an online algorithm that is able to detect task switches and learn and transfer a non-stationary representation in an adaptive fashion.

3. Problem Formulation

Assume that the non-stationary MAB problem has K arms $\mathcal{A}:=\{1,2,\dots ,K\}$ with finite time horizon T. At each round t, the learner must select an arm $i_t\in \mathcal{A}$ and obtain the corresponding reward $X_t(i_t)$. The rewards are generated from $\sigma$-subGaussian distributions. The expectation of $X_t(i)$ is denoted as ${\mu }_t(i)=\mathbb{E}\left[X_t(i)\right]$. A policy $\pi$ is a function that selects arm $i_t$ to play at round t. Let ${\mu }_t(\ast ):={max}_{i\in \{1,\dots ,K\}}{\mu }_t(i)$ denote the expected reward of the optimal arm $i_t^{\ast }$ at round t. Unlike the stationary MAB settings, where an arm is optimal all of the time (i.e., $\forall t\in \{1,\dots ,T\},i_t^{\ast }=i^{\ast }$), while in the non-stationary settings, the optimal arms might change over time. The performance of a policy $\pi$ is measured in terms of cumulative expected regret:

$$R_T^{\pi }=\mathbb{E}\left[\sum _{t=1}^T({\mu }_t(\ast )-{\mu }_t(i_t))\right],$$

(1)

where $\mathbb{E}[·]$ is the expectation with respect to randomness of $\pi$. Let ${\Delta }_t(i)={\mu }_t(\ast )-{\mu }_t(i)$ and let

$$k_T(i)=\sum _{t=1}^T\mathbb{1}\{i_t=i,i\ne i_t^{\ast }\}$$

denote the number of plays of arm i when it is not the best arm until time T. When we analyze the upper bound of $R_T^{\pi }$, we can directly analyze $\mathbb{E}\left[k_T(i)\right]$ to obtain the regret upper bound of each arm.

Abruptly Changing Setting

The abruptly changing setting is introduced by Garivier and Moulines [12] for the first time. The number of breakpoints is denoted as $B_T={\sum }_{t=1}^{T-1}\mathbb{1}\{\exists i\in \mathcal{A}:{\mu }_t(i)\ne {\mu }_{t+1}(i)\}$. Suppose the set of breakpoints is $\mathcal{B}=\{b_1,\dots ,b_{B_T}\}$ (we define $b_1=1$). At each breakpoint, the reward distribution changes for at least one arm. The rounds between two adjacent breakpoints are called stationary phase. Abruptly changing bandits pose a more challenging problem, as the learner needs to balance exploration and exploitation within each stationary phase and during the changes between different phases. Trovo et al. [14] makes an assumption about the number of breakpoints to facilitate more generalized analysis, while we explicitly use $B_T$ to represent the number of breakpoints for analysis. An implicit assumption we use is that the number of breakpoints $B_T$ is much smaller than T, i.e., $B_T\ll T$. In the community of piecewise stationary bandit problems, it is commonly assumed that $B_T$ is much smaller than T. When $B_T$ and T are comparable, researchers typically consider scenarios with smooth changes [27].

4. Algorithms

In this section, we propose the DS-TS and SW-TS with uninformative priors for the non-stationary stochastic MAB problems. Different from [32], we assume that the reward distribution follows a $\sigma$-subGaussian distribution rather than a bounded distribution. An uninformative prior can be obtained by letting the variance of a Gaussian prior approach infinity. First, assume that $X_1,\dots ,X_n$ are independently and identically distributed, following a $\sigma$-subGaussian distribution with mean $\mu$ and the prior distribution is a Gaussian distribution $\mathcal{N}(0,{\sigma }_0^2)$. The posterior distribution is also a Gaussian distribution $\mathcal{N}({\mu }_1,{\sigma }_1^2)$ where

$${\mu }_1={\sigma }_1^2({\displaystyle \frac{0}{{\sigma }_0^2}}+{\displaystyle \frac{{\sum }_{i=1}^n{X}_i}{{\sigma }^2}}),{\sigma }_1^2={\displaystyle \frac{1}{\frac{1}{{\sigma }_0^2}+\frac{n}{{\sigma }^2}}}.$$

Let ${\sigma }_0=+\infty$, we obtain the posterior distribution as $\mathcal{N}({\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{X}_i,{\displaystyle \frac{{\sigma }^2}{n}})$. In fact, when ${\sigma }_0$ is infinite, the prior distribution turns to be an uninformative prior.

4.1. DS-TS

DS-TS uses a discount factor $\gamma$ ($0<\gamma <1$) to dynamically adjust the estimate of each arm’s distribution. The key to our algorithm is to decrease the sampling variance of the selected arm while increasing the sampling variance of the unselected arms.

Specifically, let

$$N_t(\gamma ,i)=\sum _{j=1}^t{\gamma }^{t-j}\mathbb{1}\{i_j=i\}$$

denote the discounted number of plays of arm i until time t. We use

$${\widehat{\mu }}_t(\gamma ,i)={\displaystyle \frac{1}{N_t(\gamma ,i)}}\sum _{j=1}^t{\gamma }^{t-j}{X}_j(i)\mathbb{1}\{i_j=i\}$$

called discounted empirical average to estimate the expected rewards of arm i. In non-stationary settings, we use the discounted average and discounted number of plays instead of the true average and number of plays, respectively. Therefore, the posterior distribution is $\mathcal{N}({\widehat{\mu }}_t(\gamma ,i),{\displaystyle \frac{{\sigma }^2}{N_t(\gamma ,i)}})$.

Algorithm 1 shows the pseudocode of DS-TS. Step 3 is the Thompson sampling. For each arm, we draw a random sample ${\theta }_t(i)$ from $\mathcal{N}({\widehat{\mu }}_t(\gamma ,i),{\displaystyle \frac{4{\sigma }^2}{N_t(\gamma ,i)}})$. We use ${\displaystyle \frac{4{\sigma }^2}{N_t(\gamma ,i)}}$ as the posterior variance instead of ${\displaystyle \frac{{\sigma }^2}{N_t(\gamma ,i)}}$, which helps the subsequent analysis. Then, we select arm $i_t$ with the maximum sample value and obtain the reward $X_t(i_t)$ (Step 5). To avoid the time complexity going to $O(T^2)$, we introduce ${\tilde{\mu }}_t(\gamma ,i)={\sum }_{j=1}^t{\gamma }^{t-j}{X}_j(i)\mathbb{1}\{i_j=i\}$ to calculate ${\widehat{\mu }}_t(\gamma ,i)$ using an iterative method (Steps 7–9).

If arm i is selected at round t, the posterior distribution is updated as follows:

$${\widehat{\mu }}_{t+1}(\gamma ,i)={\displaystyle \frac{\gamma {\widehat{\mu }}_t(\gamma ,i)N_t(\gamma ,i)+X_t(i)}{\gamma N_t(\gamma ,i)+1}}={\displaystyle \frac{{\tilde{\mu }}_{t+1}(\gamma ,i)}{N_{t+1}(\gamma ,i)}}$$

If arm i is not selected at round t, the posterior distribution is updated as

$${\widehat{\mu }}_{t+1}(\gamma ,i)={\displaystyle \frac{{\tilde{\mu }}_{t+1}(\gamma ,i)}{N_{t+1}(\gamma ,i)}}={\displaystyle \frac{\gamma {\tilde{\mu }}_t(\gamma ,i)}{\gamma N_t(\gamma ,i)}}={\widehat{\mu }}_t(\gamma ,i)$$

i.e., the expectation of posterior distribution remains unchanged.

Algorithm 1 DS-TS

1: discounted factor $\gamma \in (0,1)$, ${\widehat{\mu }}_1(i)=0$, ${\tilde{\mu }}_1(i)=0$, $N_t(\gamma ,i)=0$  2: for $t=1,\dots ,T$ do  3:    for $i=1,\dots ,K$ do  4:      sample ${\theta }_t(i)\sim \mathcal{N}({\widehat{\mu }}_t(\gamma ,i),{\displaystyle \frac{4{\sigma }^2}{N_t(\gamma ,i)}})$  5:    end for  6:    Pull arm $i_t=arg{max}_i{\theta }_t(i)$, observe reward $X_t(i_t)$  7:    for $i=1,\dots ,K$ do  8:      ${\tilde{\mu }}_{t+1}(\gamma ,i)=\gamma {\tilde{\mu }}_t(\gamma ,i)+\mathbb{1}\{i_t=i\}{X}_t(i)$  9:      $N_{t+1}(\gamma ,i)=\gamma N_t(\gamma ,i)+\mathbb{1}\{i_t=i\}$ 10:      ${\widehat{\mu }}_{t+1}(\gamma ,i)={\displaystyle \frac{{\tilde{\mu }}_{t+1}(\gamma ,i)}{N_{t+1}(\gamma ,i)}}$ 11:    end for 12: end for

4.2. SW-TS

SW-TS uses a sliding window $\tau$ to adapt to changes in the reward distribution. Let

$$N_t(\tau ,i)=\sum _{j=t-\tau +1}^t\mathbb{1}\{i_j=i\},{\widehat{\mu }}_t(\tau ,i)={\displaystyle \frac{1}{N_t(\tau ,i)}}\sum _{j=t-\tau +1}^t{X}_j(i)\mathbb{1}\{i_j=i\}.$$

If $t<\tau$, the range of summation is from 1 to t. Similar to DS-TS, the posterior distribution is $\mathcal{N}({\widehat{\mu }}_t(\tau ,i),{\displaystyle \frac{4{\sigma }^2}{N_t(\tau ,i)}})$. Algorithm 2 shows the pseudocode of SW-TS. To avoid the time complexity going to $O(T^2)$, we introduce ${\tilde{\mu }}_t(\tau ,i)={\sum }_{j=t-\tau +1}^t{X}_j(i)\mathbb{1}\{i_j=i\}$ to update ${\widehat{\mu }}_t(\tau ,i)$.

Algorithm 2 SW-TS

1: sliding window $\tau$, ${\widehat{\mu }}_1(i)=0$, ${\tilde{\mu }}_1(i)=0$, $N_t(\tau ,i)=0$  2: for $t=1,\dots ,T$ do  3:    for $i=1,\dots ,K$ do  4:      sample ${\theta }_t(i)\sim \mathcal{N}({\widehat{\mu }}_t(\tau ,i),{\displaystyle \frac{4{\sigma }^2}{N_t(\tau ,i)}})$  5:    end for  6:    Pull arm $i_t=arg{max}_i{\theta }_t(i)$, observe reward $X_t(i_t)$  7:    for $i=1,\dots ,K$ do  8:      $N_{t+1}(\tau ,i)=N_t(\tau ,i)+\mathbb{1}\{i_t=i\}-\mathbb{1}\{i_{t-\tau }=i\}$  9:      ${\tilde{\mu }}_{t+1}(\tau ,i)={\tilde{\mu }}_t(\tau ,i)+\mathbb{1}\{i_t=i\}{X}_t(i)-\mathbb{1}\{i_{t-\tau }=i\}{X}_{t-\tau }(i)$ 10:      ${\widehat{\mu }}_{t+1}(\tau ,i)={\displaystyle \frac{{\tilde{\mu }}_{t+1}(\tau ,i)}{N_{t+1}(\tau ,i)}}$ 11:    end for 12: end for

4.3. Results

In this section, we give the regret upper bounds of DS-TS and SW-TS. Then, we discuss how to take the values of the parameters so that these algorithms reach the optimal upper bound.

Recall that ${\Delta }_t(i)={\mu }_t(\ast )-{\mu }_t(i)$. Let ${\Delta }_T(i)=min\{{\Delta }_t(i):t\le T,i\ne i_t^{\ast }\}$, be the minimum difference between the expected reward of the best arm $i_t^{\ast }$ and the expected reward of arm i in all time T when the arm i is not the best arm. Let ${\Delta }_{max}^T=max\{{\mu }_{t_1}(i)-{\mu }_{t_2}(i):t_1\ne t_2,i\in \left[K\right]\}$ denote the maximum expected variation of arms.

Theorem 1

(DS-TS). Let $\gamma \in (0,1)$ satisfying ${({\displaystyle \frac{\sigma }{{\Delta }_{max}^T}})}^2<{\displaystyle \frac{e}{1-\gamma }}$. For any suboptimal arm i,

$$\mathbb{E}\left[k_T(i)\right]\le B_{T}D(\gamma )+C_1(\gamma )L_1(\gamma ){\gamma }^{-{\displaystyle \frac{1}{1-\gamma }}}T(1-\gamma ),$$

where

$$D(\gamma )={\displaystyle \frac{log({(\frac{\sigma }{{\Delta }_{max}^T})}^2{(1-\gamma )}^{2}log\frac{1}{1-\gamma })}{log\gamma }},C_1(\gamma )=e^{17}+15log{\displaystyle \frac{1}{1-\gamma }},$$

$$L_1(\gamma )={\displaystyle \frac{1152log(\frac{1}{1-\gamma }+e^{17}){\sigma }^2}{{\gamma }^{1/(1-\gamma )}{({\Delta }_T(i))}^2}}.$$

Remark 1.

The condition ${({\displaystyle \frac{\sigma }{{\Delta }_{max}^T}})}^2<{\displaystyle \frac{e}{1-\gamma }}$ can ensure that $D(\gamma )$ is well defined. In general, we do not need to know ${\Delta }_{max}^T$ in advance when setting the value of γ. If we choose a γ close to 1, then the condition ${({\displaystyle \frac{\sigma }{{\Delta }_{max}^T}})}^2<{\displaystyle \frac{e}{1-\gamma }}$ in Theorem 1 is easily satisfied, as shown in the corollary below.

Corollary 1.

If the time horizon T and number of breakpoints $B_T$ are known in advance, the discounted factor can be chosen as $\gamma =1-{\displaystyle \frac{1}{\sigma }}\sqrt{{\displaystyle \frac{B_T}{TlogT}}}$. If $B_T\ll T$,

$${({\displaystyle \frac{\sigma }{{\Delta }_{max}^T}})}^2(1-\gamma )={\displaystyle \frac{\sigma }{{({\Delta }_{max}^T)}^2}}\sqrt{{\displaystyle \frac{B_T}{TlogT}}}<e.$$

we have

$$\mathbb{E}\left[k_T(i)\right]=O(\sqrt{T{B}_T}{(logT)}^{{\displaystyle \frac{3}{2}}}).$$

Theorem 2

(SW-TS). Let $\tau >0$, for any suboptimal arm i,

$$\mathbb{E}\left[k_T(i)\right]\le B_T\tau +C_2(\tau )L_2(\tau ){\displaystyle \frac{T}{\tau }},$$

where

$$C_2(\tau )=e^{11}+15log\tau ,L_2(\tau )={\displaystyle \frac{1152log(\tau +e^{11}){\sigma }^2}{{({\Delta }_T(i))}^2}}.$$

Corollary 2.

If the time horizon T and number of breakpoints $B_T$ are known in advance, the sliding window can be chosen as $\tau =\sigma \sqrt{T/B_T}logT$, then

$$\mathbb{E}\left[k_T(i)\right]=O(\sqrt{T{B}_T}logT).$$

5. Proofs of Upper Bounds

Before giving the detailed proof, we discuss the main challenges in regret analysis of Thompson sampling in a non-stationary setting. These challenges are addressed by Lemmas 1–3.

5.1. Challenges in Regret Analysis

Existing analyses of regret bounds for Thompson sampling [32,33,34] decompose the regret into two parts. The first part of regret comes from the over-estimation of the suboptimal arm, which can be dealt with by the concentration properties of the sampling distribution and rewards distribution. The second part is the under-estimation of the optimal arm, which mainly relies on bounding the following equation.

$$\sum _{t=1}^T\mathbb{E}[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },{\theta }_t(\ast )\le {\mu }_t(\ast )-{ϵ}_i\}],$$

(2)

where $p_{i,t}=\mathbb{P}({\theta }_t(\ast )>{\mu }_t(\ast )-{ϵ}_i)$ is the probability that the best arm will not be under-estimated from the mean reward by a margin ${ϵ}_i$.

The first challenge is specific to the DS-TS algorithm. Unlike SW-TS, which completely forgets previous information after $\tau$ rounds following a breakpoint, DS-TS cannot fully forget past information.

This makes it challenging to utilize the concentration properties of the reward distribution to bound regret comes from the over-estimate of the suboptimal arm. And this will further affect the analysis of Equation (2).

The second challenge is the under-estimation of the optimal arm. In stationary settings, $p_{i,t}$ changes only when the optimal arm is selected, Equation (2) can be bounded by the method proposed by Agrawal and Goyal [32]. However, the distribution of ${\theta }_t(\ast )$ may vary over time in non-stationary settings. It is challenging and nontrivial to obtain a tight bound of Equation (2).

To overcome the first challenges, we adjust the posterior variance to be ${\displaystyle \frac{4{\sigma }^2}{N_t(\gamma ,i)}}$. This slightly larger variance is specifically designed for the ${\sigma }^2$-subGaussian distribution, which helps to bound $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]$ (In Appendix B.4, we have shown that our analysis method requires the variance to be greater than ${\displaystyle \frac{2{\sigma }^2}{N}}$. And we set the variance to ${\displaystyle \frac{4{\sigma }^2}{N}}$ for a more convenient presentation of the paper’s results.). Then, we define $U_t(\gamma ,i)$, which serves a role similar to the upper confidence bound in the UCB algorithm. We solve this problem through Lemmas 1 and 2.

For the second challenge, we use the new defined $U_t(\gamma ,i)$ and employ a new regret decomposition for Equation (2) based on whether the event $\{N_t(\gamma ,\ast )>L_1(\gamma )\}$ occurs. Intuitively, if $N_t(\gamma ,\ast )>L_1(\gamma )$, $p_{i,t}$ is close to 1, which will lead to a sharp bound. If $N_t(\gamma ,\ast )\le L_1(\gamma )$, using Lemma A3 we can also obtain the upper bound of Equation (2). We derive the upper bound of $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]$ for non-stationary settings, with an extra logarithmic term compared with the stationary settings. The proof of Lemma 3 in Appendix B.3 demonstrates these details.

5.2. Proofs of Theorem 1

For arm $i\ne i_t^{\ast }$, we choose two threshold $x_t(i),y_t(i)$ such that $x_t(i)={\mu }_t(i)+{\displaystyle \frac{{\Delta }_t(i)}{3}},y_t(i)={\mu }_t(\ast )-{\displaystyle \frac{{\Delta }_t(i)}{3}}$. Then ${\mu }_t(i)<x_t(i)<y_t(i)<{\mu }_t(\ast )$ and $y_t(i)-x_t(i)={\displaystyle \frac{{\Delta }_t(i)}{3}}$. The history ${\mathcal{F}}_t$ is defined as the plays and rewards of the previous t plays. ${\widehat{\mu }}_t(\gamma ,i),i_t$ and the distribution of ${\theta }_t(i)$ are determined by the history ${\mathcal{F}}_{t-1}$.

The abruptly changing setting is in fact piecewise-stationary. The rounds between two adjacent breakpoints are stationary. Based on this observation, we define the pseudo-stationary phase as

$$\mathcal{T}(\gamma )=\{t\le T:\forall s\in (t-D(\gamma ),t],{\mu }_s(·)={\mu }_t(·)\}.$$

The rounds in $\mathcal{T}(\gamma )$ can to some extent ensure that the rewards are “stationary”. For any $t\in \mathcal{T}(\gamma )$, the rewards distribution remain unchanged between $(t-D(\gamma ),t)$. Therefore, we can obtain a good estimate of the rewards distribution in $\mathcal{T}(\gamma )$ (Lemma 1). Let $\mathcal{S}(\gamma )$ denote the complement of $\mathcal{T}(\gamma )$, i.e., $\mathcal{S}(\gamma )=\{t\le T:t\notin \mathcal{T}(\gamma )\}$. Note that, there is at most $D(\gamma )$ rounds belonging to $\mathcal{S}(\gamma )$ after each breakpoint. This is because the rounds between two adjacent breakpoints are stationary. The time steps after the breakpoint $D(\gamma )$ rounds, the rewards distribution do not change and therefore belongs to $\mathcal{T}(\gamma )$. Therefore, the number of elements in the set $\mathcal{S}(\gamma )$ has an upper bound $B_{T}D(\gamma )$, i.e.,

$$\left|\mathcal{S}(\gamma )\right|\le B_{T}D(\gamma )$$

(3)

Figure 1 shows $\mathcal{T}(\gamma )$ and $\mathcal{S}(\gamma )$ in two different situations. Since during the rounds in $\mathcal{S}(\gamma )$, i.e., the rounds following a breakpoint, the estimate of the expected rewards may be poor, we directly bound the regret during $\mathcal{S}(\gamma )$ by $B_{T}D(\gamma )$ and only focus on the regret in $\mathcal{T}(\gamma )$.

Figure 1. Illustration of $\mathcal{T}(\gamma )$ and $\mathcal{S}(\gamma )$ in two different situations. $b_i,b_{i+1},b_{i+2}$ are the breakpoints. The situation that $b_{i+1}-b_i>D(\gamma )$ is shown in the top figure, and $b_{i+1}-b_i\le D(\gamma )$ is in the bottom.

To facilitate the analysis, we define the following quantities and events.

$$n=6\sqrt{2}+3\sqrt{1-\gamma },A(\gamma )={\displaystyle \frac{n^{2}log(\frac{1}{1-\gamma }){\sigma }^2}{{({\Delta }_T(i))}^2}},U_t(\gamma ,i)=\sigma \sqrt{{\displaystyle \frac{(1-\gamma )log\frac{1}{1-\gamma }}{N_t(\gamma ,i)}}}.$$

(4)

Definition 1.

Define $E_t(\gamma ,i)$ as the event $\{i_t=i,N_t(\gamma ,i)>A(\gamma )\}$. Define $E_t^{\theta }(i)$ as the event $\{{\theta }_t(i)<y_t(i)\}$.

Intuitively, event $E_t(\gamma ,i)$ represents selecting a sufficiently explored suboptimal arm. Event $E_t^{\theta }(i)$ denotes ${\theta }_t(i)$ is not too far from the mean ${\mu }_t(i)$.

Now we list some useful lemmas. The detailed proofs are provided in the appendix. The following lemma depicts that after finite rounds at the breakpoint, i.e., in the pseudo-stationary phase, the distance between ${\mu }_t(i)$ and discounted average of expectation for arm i can be bounded by $U_t(\gamma ,i)$. $U_t(\gamma ,i)$ is analogous to the upper confidence bound in the UCB algorithm.

Lemma 1.

Let ${\ddot{\mu }}_t(\gamma ,i)={\displaystyle \frac{1}{N_t(\gamma ,i)}}{\sum }_{j=1}^t{\gamma }^{t-j}\mathbb{1}\{i_j=i\}{\mu }_j(i)$ denote the discounted average of expectation for arm i at time step t. $\forall t\in \mathcal{T}(\gamma )$, the distance between ${\mu }_t(i)$ and ${\ddot{\mu }}_t(\gamma ,i)$ is less than $U_t(\gamma ,i)$.

$$|{\mu }_t(i)-{\ddot{\mu }}_t(\gamma ,i)|\le U_t(\gamma ,i),$$

(5)

Using Lemma 1 and the self-normalized Hoeffding-type inequality for subGaussian distributions (Lemma A1), we have the following lemma. This lemma helps to bound regret comes from the over-estimation of suboptimal arm.

Lemma 2.

$\forall t\in \mathcal{T}(\gamma ),i\ne i_t^{\ast }$,

$$\mathbb{P}({\widehat{\mu }}_t(\gamma ,i)>x_t(i),N_t(\gamma ,i)>A(\gamma ))\le {(1-\gamma )}^2$$

The following key lemma helps bound the regret comes from the under-estimation of the optimal arm. This is the most tricky part of analyzing TS. Note that, the proof in [14] does not prove the result of the following lemma.

Lemma 3.

Let $p_{i,t}=\mathbb{P}({\theta }_t(\ast )>y_t(i)\mid {\mathcal{F}}_{t-1})$. For any $t\in \mathcal{T}(\gamma )$ and $i\ne i_t^{\ast }$,

$$\begin{array}{c}\hfill \sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },{\theta }_t(i)<y_t(i)\}\right]\le (e^{17}+12log{\displaystyle \frac{1}{1-\gamma }})T(1-\gamma )L_1(\gamma ){\gamma }^{-{\displaystyle \frac{1}{1-\gamma }}}.\end{array}$$

Before we give the detailed proof, we give a outline of our proof.

Proof Outlines. First, since the regret incurred in $\{t\notin \mathcal{T}(\gamma )\}$ can be bounded by $B_{T}D(\gamma )$, we only consider the regret in rounds $\{t\in \mathcal{T}(\gamma )\}$. Then, we consider the event $\{N_t(\gamma ,i)>A(\gamma )\}$. If this event is not true, we can use Lemma A3 to bound the regret by $T(1-\gamma )A(\gamma ){\gamma }^{-1/(1-\gamma )}$. If this event holds true, we additionally consider whether the suboptimal arm is over-estimation (${\widehat{\mu }}_t(\gamma ,i)>x_t(i)$) and whether event $\{{\theta }_t(i)<y_t(i)\}$ is true to decompose the regret into three parts as Equation (10). The first part comes from the over-estimation of the suboptimal arm, which can be bounded by Lemma 2. The second part comes from the bias in sampling the suboptimal arm, which can be bounded by the properties of Gaussian distribution Equation (A1). The third part denotes that the regret comes from the under-estimation of the optimal arm and can be bounded by Lemma 3.

The proof is in 5 steps:

Step 1: We can divide the rounds $t\in \{1,\dots ,T\}$ into two parts: $\{t\in \mathcal{T}(\gamma )\}$ and $\{t\notin \mathcal{T}(\gamma )\}$. Equation (3) shows that the number of elements in the second part is smaller than $B_{T}D(\gamma )$, we have

$$\mathbb{E}\left[k_T(i)\right]\le B_{T}D(\gamma )+\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i).$$

(6)

Step 2: Then, we consider the event $\{N_t(\gamma ,i)>A(\gamma )\}$.

$$\begin{array}{c}\hfill \sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i)=\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i,N_t(\gamma ,i)<A(\gamma ))+\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i,N_t(\gamma ,i)>A(\gamma )).\end{array}$$

We first bound ${\sum }_{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i,N_t(\gamma ,i)<A(\gamma ))$.

$$\begin{array}{cc}\hfill \sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i,N_t(\gamma ,i)<A(\gamma ))& =\sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[\mathbb{P}(i_t=i,N_t(\gamma ,i)<A(\gamma )\mid {\mathcal{F}}_{t-1})\right]\hfill \\ \hfill & =\sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[\mathbb{E}\left[\mathbb{1}\{i_t=i,N_t(\gamma ,i)<A(\gamma )\mid {\mathcal{F}}_{t-1}\}\right]\right]\hfill \\ \hfill & =\sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[\mathbb{1}\{i_t=i,N_t(\gamma ,i)<A(\gamma )\}\right],\hfill \end{array}$$

(7)

where the last equation uses the tower rule of expectation.

Using Lemma A3, we have

$$\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i,N_t(\gamma ,i)<A(\gamma ))\le T(1-\gamma )A(\gamma ){\gamma }^{-1/(1-\gamma )}$$

(8)

Therefore,

$$\begin{array}{c}\hfill \mathbb{E}\left[k_T(i)\right]\le T(1-\gamma )A(\gamma ){\gamma }^{-{\displaystyle \frac{1}{1-\gamma }}}+B_{T}D(\gamma )+\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(i_t=i,N_t(\gamma ,i)>A(\gamma ))\end{array}$$

(9)

Step 3: Recall that we use $E_t(\gamma ,i)$ to denote the event $\{i_t=i,N_t(\gamma ,i)>A(\gamma )\}$ and $E_t^{\theta }(i)$ denote the event ${\theta }_t(i)<y_t(i)$. Equation (9) may be decomposed as follows:

$$\begin{array}{cc}\hfill \sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(E_t(\gamma ,i))& =\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(E_t(\gamma ,i),{\widehat{\mu }}_t(\gamma ,i)>x_t(i))\hfill \\ \hfill & +\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(E_t(\gamma ,i),{\widehat{\mu }}_t(\gamma ,i)<x_t(i),\overline{E_t^{\theta }(i)})\hfill \\ \hfill & +\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(E_t(\gamma ,i),{\widehat{\mu }}_t(\gamma ,i)<x_t(i),E_t^{\theta }(i))\hfill \end{array}$$

(10)

Using Lemma 2, the first part in Equation (10) can be bounded by $T{(1-\gamma )}^2$.

Step 4: Then, we bound the second part in Equation (10). Use the fact that $N_t(\gamma ,i)$ and ${\widehat{\mu }}_t(i)$ are determined by the history ${\mathcal{F}}_{t-1}$, we have

$$\begin{array}{cc}\hfill & \sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(E_t(\gamma ,i),{\widehat{\mu }}_t(\gamma ,i)<x_t(i),\overline{E_t^{\theta }(i)})\hfill \\ \hfill & =\mathbb{E}\left[\sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[\mathbb{1}\{i_t=i,N_t(\gamma ,i)>A(\gamma ),{\widehat{\mu }}_t(\gamma ,i)<x_t(i),\overline{E_t^{\theta }(i)}\}\mid {\mathcal{F}}_{t-1}\right]\right]\hfill \\ \hfill & =\mathbb{E}\left[\sum _{t\in \mathcal{T}(\gamma )}\mathbb{1}\{N_t(\gamma ,i)>A(\gamma ),{\widehat{\mu }}_t(\gamma ,i)<x_t(i)\}\mathbb{P}(i_t=i,\overline{E_t^{\theta }(i)}\mid {\mathcal{F}}_{t-1})\right]\hfill \\ \hfill & \le \mathbb{E}\left[\sum _{t\in \mathcal{T}(\gamma )}\mathbb{1}\{N_t(\gamma ,i)>A(\gamma ),{\widehat{\mu }}_t(\gamma ,i)<x_t(i)\}\mathbb{P}({\theta }_t(i)>y_t(i)\mid {\mathcal{F}}_{t-1})\right].\hfill \end{array}$$

(11)

Given the history ${\mathcal{F}}_{t-1}$ such that $N_t(\gamma ,i)>A(\gamma )$ and ${\widehat{\mu }}_t(\gamma ,i)<x_t(i)$, we have

$$y_t(i)-{\widehat{\mu }}_t(\gamma ,i)>y_t(i)-x_t(i)={\displaystyle \frac{{\Delta }_t(i)}{3}}\ge {\displaystyle \frac{{\Delta }_T(i)}{3}}.$$

Therefore,

$$\begin{array}{cc}\hfill & \mathbb{P}({\theta }_t(i)>y_t(i)\mid {\mathcal{F}}_{t-1}))\le \mathbb{P}({\theta }_t(i)-{\widehat{\mu }}_t(\gamma ,i)>{\displaystyle \frac{{\Delta }_T(i)}{3}}\mid {\mathcal{F}}_{t-1})\hfill \\ \hfill & \le {\displaystyle \frac{1}{2}}exp(-{\displaystyle \frac{{({\Delta }_T(i))}^{2}A(\gamma )}{72{\sigma }^2}})\hfill \\ \hfill & \le {\displaystyle \frac{1}{2}}(1-\gamma ),\hfill \end{array}$$

(12)

where the second inequality follows ${\theta }_t(i)\sim \mathcal{N}\left({\widehat{\mu }}_t(\gamma ,i),{\displaystyle \frac{4{\sigma }^2}{N_t(\gamma ,i)}}\right)$ and Equation (A1).

For other ${\mathcal{F}}_{t-1}$, the indicator term $\mathbb{1}\{N_t(\gamma ,i)>A(\gamma ),{\widehat{\mu }}_t(\gamma ,i)<x_t(i)\}$ will be 0. Hence, we can bound the second part by ${\displaystyle \frac{T}{2}}(1-\gamma )$

Step 5: Finally, we focus the third term in Equation (10). Using Lemma A2 and the fact that $p_{i,t}$ is fixed given ${\mathcal{F}}_{t-1}$,

$$\begin{array}{cc}\hfill \sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(E_t(\gamma ,i),{\widehat{\mu }}_t(\gamma ,i)<x_t(i),E_t^{\theta }(i))& \le \sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{P}(i_t=i_t^{\ast },E_t^{\theta }(i)\mid {\mathcal{F}}_{t-1})\right]\hfill \\ & =\sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },E_t^{\theta }(i)\mid {\mathcal{F}}_{t-1}\}\right]\right]\hfill \\ & =\sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },E_t^{\theta }(i)\}\right]\hfill \end{array}$$

Then, by Lemma 3, we have

$$\sum _{t\in \mathcal{T}(\gamma )}\mathbb{P}(E_t(\gamma ,i),{\widehat{\mu }}_t(\gamma ,i)<x_t(i),E_t^{\theta }(i))\le (e^{17}+12log{\displaystyle \frac{1}{1-\gamma }})T(1-\gamma )L_1(\gamma ){\gamma }^{-{\displaystyle \frac{1}{1-\gamma }}}.$$

(13)

Substituting the results in Step 3–5 to Equation (10) and Equation (9),

$$\begin{array}{cc}\hfill \mathbb{E}\left[k_T(i)\right]& \le T(1-\gamma )A(\gamma ){\gamma }^{-1/(1-\gamma )}+B_{T}D(\gamma )+2T(1-\gamma )\hfill \\ & +(e^{17}+12log{\displaystyle \frac{1}{1-\gamma }})T(1-\gamma )L_1(\gamma ){\gamma }^{-1/(1-\gamma )}\hfill \\ & \le B_{T}D(\gamma )+(e^{17}+15log{\displaystyle \frac{1}{1-\gamma }})L_1(\gamma ){\gamma }^{-{\displaystyle \frac{1}{1-\gamma }}}T(1-\gamma ).\hfill \end{array}$$

5.3. Proofs of Theorem 2

The proof of Theorem 2 is similar to Theorem 1. The main difference is that the pseudo-stationary phase is now defined as $\mathcal{T}(\tau )=\{t\le T:\forall s\in (t-\tau ,t],{\mu }_s(·)={\mu }_t(·)\}$. Let

$${\ddot{\mu }}_t(\tau ,i)={\displaystyle \frac{1}{N_t(\tau ,i)}}\sum _{j=t-\tau +1}^t\mathbb{1}\{i_j=i\}{\mu }_j(i).$$

If $t\in \mathcal{T}(\tau )$,

$${\ddot{\mu }}_t(\tau ,i)={\displaystyle \frac{1}{N_t(\tau ,i)}}\sum _{j=t-\tau +1}^t\mathbb{1}\{i_j=i\}{\mu }_t(i)={\mu }_t(i).$$

This means the bias ($U_t(\gamma ,i)$) vanishes. We no longer need an n related to $\tau$ to deal with the bias issue. We only need to define $A(\tau )$ as

$$A(\tau )={\displaystyle \frac{72log(\tau ){\sigma }^2}{{({\Delta }_T(i))}^2}}.$$

We directly list the following two lemmas, corresponding to Lemma 2 and Lemma 3, respectively.

Lemma 4.

$\forall t\in \mathcal{T}(\tau ),t\ne i_t^{\ast }$,

$$\mathbb{P}({\widehat{\mu }}_t(\tau ,i)>x_t(i),N_t(\tau ,i)>A(\tau ))\le {\displaystyle \frac{1}{{\tau }^2}}.$$

This lemma is similar to Lemma 2. It can be used to bound the regret that comes from over-estimation of the suboptimal arms. This lemma can be proved by Hoeffding-type inequality for subGaussian distributions (Lemma A1). The detailed proofs can be found in Appendix B.5.

Lemma 5.

Let $p_{i,t}=\mathbb{P}({\theta }_t(\ast )>y_t(i)|{\mathcal{F}}_{t-1})$. For any $t\in \mathcal{T}(\tau )$ and $i\ne i_t^{\ast }$,

$$\sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },{\theta }_t(i)<y_t(i)\}\right]\le (e^{11}+9+3log\tau ){\displaystyle \frac{T}{\tau }}{L}_2(\tau ).$$

This key lemma helps bound the regret that comes from the under-estimation of the optimal arm (Step 5 in the proof of DS-TS) which is similar to Lemma 3. It can be proved by Lemma A2, which transforms the probability of selecting the ith arm into the probability of selecting the optimal arm $i_t^{\ast }$. The detailed proofs can be found in Appendix B.6.

The rest of the proof is nearly identical to the proof of Theorem 1.

6. Experiments

In this section, we empirically compare the performance of our method with state-of-the-art algorithms on Bernoulli and Gaussian reward distributions (our code is available at https://github.com/qh1874/TS_NonStationary (accessed on 14 August 2024)). Specifically, we compare DS-TS and SW-TS with Thompson sampling to evaluate the improvement obtained thanks to the employment of the discounted factor $\gamma$ and sliding window $\tau$. We also compare our method with the UCB method, DS-UCB and SW-UCB [12] to evaluate the effect of Thompson sampling and UCB. Furthermore, we compare our method with some novel and efficient algorithms such as CUSUM [7], M-UCB [8] and SW-LB-SDA [15]. Note that SW-LB-SDA is not applicable to Gaussian reward distributions with unknown variance. We measure the performance of each algorithm with the cumulative expected regret defined in Equation (1). The expected regret is averaged over 100 independently runs. The 95% confidence interval is obtained by performing 100 independent runs and is depicted as a semi-transparent region in the figure.

6.1. Gaussian Arms

6.1.1. Experimental Setting for Gaussian Arms

We fix the time horizon as T = 100,000. The mean and standard deviation are drawn from distributions $\mathcal{N}(0,5^2)$ and $U(1,5)$. For Gaussian rewards, we conduct two experiments. In the first experiment, we split the time horizon into five phases and use a number of arms $K=5$. While in the second experiment, we split the time horizon into 10 phases and use a number of arms $K=10$. Figure 2 depicts the expected rewards for Gaussian arms and Bernoulli arms with $K=5$ and $B_T=5$.

Figure 2. $K=5,B_T=5$. Gaussian arms (a), Bernoulli arms (b).

The analysis of SW-UCB and DS-UCB is conducted under the bounded reward assumption, but the algorithms can adapt to Gaussian scenarios. To achieve reasonable performance, it is necessary to adjust the discounted factor and the sliding-window appropriately. We use the settings recommended in [15], where $\tau =2(1+2\sigma )\sqrt{Tlog(T)/B_T}$ for SW-UCB and $\gamma =1-1/(4(1+2\sigma ))\sqrt{B_T/T}$ for DS-UCB.

6.1.2. Results

Figure 3 illustrates the performance of these algorithms for Gaussian rewards under two different settings. Notably, CUSUM and M-UCB are not applicable to Gaussian rewards: CUSUM is designed for Bernoulli distributions, while M-UCB assumes bounded distributions. The discounted methods tend to perform better than sliding-window methods in Gaussian rewards.

Figure 3. Gaussian arms. (a) $K=5,B_T=5$. (b) $K=10,B_T=10$.

Among these algorithms, only our algorithms and SW-LB-SDA provide regret analysis for unbounded rewards. Our algorithm (DS-TS) and SW-LB-SDA have demonstrated highly competitive experimental performance.

6.2. Bernoulli Arms

6.2.1. Experimental Setting for Bernoulli Arms

The time horizon is set as T = 100,000. We split the time horizon into $5,10$ phases of equal length and use a number of arms $K=\{5,10\}$, respectively.

For Bernoulli rewards, the expected value ${\mu }_t(i)$ of each arm i is drawn from a uniform distribution over $[0,1]$. In the stationary phase, the rewards distributions remain unchanged. The Bernoulli arms for each phase are generated as ${\mu }_t(i)\sim U(0,1).$

For a Bernoulli distribution, we modify the Thompson sampling (step 3) in our algorithm as ${\theta }_t(i)\sim \mathcal{N}({\widehat{\mu }}_t(\gamma ,i),{\displaystyle \frac{1}{N_t(\gamma ,i)}})$ and ${\theta }_t(i)\sim \mathcal{N}({\widehat{\mu }}_t(\tau ,i),{\displaystyle \frac{1}{N_t(\tau ,i)}})$. Based on Corollaries 1 and 2, we set $\gamma =1-\sqrt{{\displaystyle \frac{B_T}{TlogT}}}$ and $\tau =\sqrt{T/B_T}logT$. To allow for fair comparison, DS-UCB uses the discount factor $\gamma =1-\sqrt{B_T/T}/4$, SW-UCB uses the sliding window $\tau =2\sqrt{TlogT/B_T}$ suggested by [12]. Based on [15], we set $\tau =2\sqrt{Tlog(T)/B_T}$ for LB-SDA. For changepoint detection algorithm M-UCB, we set $w=800,b=\sqrt{w/2log(2K{T}^2)}$ as suggested by [8]. But we set the amount of exploration as $\gamma =\sqrt{K{B}_{T}log(T)/T}$. In practice, it has been found that using this value instead of the one guaranteed in [8] will improve empirical performance [15]. For CUSUM, following from [7], we set $\alpha =\sqrt{B_T/Tlog(T/B_T)}$ and $h=log(T/B_T)$. For our experiment settings, we choose $M=50,ϵ=0.05$.

6.2.2. Results

Figure 4 presents the results for Bernoulli arms in abruptly changing settings. It can be observed that our method (SW-TS) and SW-LB-SDA exhibit almost identical performance. Thompson sampling, designed for stationary MAB problems, shows significant oscillations at the breakpoints. The changepoint detection algorithm CUSUM [7] also shows competitive performance. Note that our experiment does not satisfy the detectability assumption of CUSUM. As the number of arms and breakpoints increase, the performance of UCB-class algorithms (DS-UCB, SW-UCB) declines, while two TS-based algorithms (DS-TS, SW-TS) still work well.

Figure 4. Bernoulli arms. Settings with $K=5,B_T=5$ (a), $K=10,B_T=10$ (b).

6.2.3. Storage and Compute Cost

These algorithms can be divided into three class: UCB, TS and SW-LB-SDA. At each round, UCB-class and TS-class algorithms require $O(K)$ storage and spend $O(K)$ time complexity for computational cost. However, for round T, SW-LB-SDA require $O(K{(logT)}^2)$ storage and spend $O(KlogT)$ time cost. Although the experimental performance of SW-LB-SDA is similar to our algorithms, our algorithm has less storage space and lower computational complexity.

6.3. Different Variance

The non-stationary setting has greater noise for estimation as compared to the stationary setting. Intuitively, TS with standard variance for the non-stationary setting should have worse regret as compared to the one with a larger variance. In this subsection, we conduct some experiments to verify this point. Table 1 shows the experimental results. For TS and the SW-TS algorithm, larger variance does indeed lead to smaller regret. This conclusion does not hold for DS-TS. We believe this does not contradict the above conclusion, because for DS-TS, the discount factor plays a more important role. If an arm has not been selected for some rounds, then $N_t(\gamma ,i)$ will be small ($N_t(\gamma ,i)$ can be close to 0, while if $N_t(\tau ,i)$ for SW-TS is greater than 1 or equal to 0), then ${\displaystyle \frac{\sigma }{N_t(\gamma ,i)}}$ has already become large, ensuring exploration performance. Therefore, DS-TS achieves the minimum regret. However, ${\displaystyle \frac{2\sigma }{N_t(\gamma ,i)}}$ may be too large, leading to excessive exploration and thus reducing regret compared to ${\displaystyle \frac{\sigma }{N_t(\gamma ,i)}}$.

Table 1. Settings with T = 100,000, $B_T$ = 5, K = 5 for Gaussian arms. The mean and standard deviation are drawn from distributions $\mathcal{N}(0,5^2)$ and $U(1,5)$. We set $\sigma =5$.

Algorithms	TS		DS-TS		SW-TS
std	${\displaystyle \frac{\mathit{\sigma }}{\mathit{N}}}$	${\displaystyle \frac{\mathbf{2}\mathit{\sigma }}{\mathit{N}}}$	${\displaystyle \frac{\mathit{\sigma }}{\mathit{N}}}$	${\displaystyle \frac{\mathbf{2}\mathit{\sigma }}{\mathit{N}}}$	${\displaystyle \frac{\mathit{\sigma }}{\mathit{N}}}$	${\displaystyle \frac{\mathbf{2}\mathit{\sigma }}{\mathit{N}}}$
Regret	333,835	305,064	41,790	52,909	83,731	83,150

7. Conclusions

In this paper, we analyze the regret upper bound of the TS algorithm with an uninformative prior in non-stationary settings, filling a research gap in this field. Our approach builds upon previous works while tackling two key challenges specific to non-stationary environments: under-estimation of the optimal arm and the inability of DS-TS algorithm to fully forget previous information. Finally, we conduct some experiments to verify the theory results. Below we discuss the results and propose directions for future research.

(1) The standard posterior update rule for Thompson sampling has a sampling variance as ${\displaystyle \frac{{\sigma }^2}{N}}$. We use ${\displaystyle \frac{4{\sigma }^2}{N}}$ only for ease of analysis. While this discrepancy is significant only for relatively small values of N, it would be valuable to develop proof techniques that leverage the variance of standard Bayesian updates.

(2) Our regret upper bound includes an additional logarithmic term compared to DS-UCB and SW-UCB, along with coefficients of $e^{17}$ and $e^{11}$. It would be interesting to explore whether the additional logarithm and large coefficients are intrinsic to DS-TS and SW-TS algorithms or are a limitation of our analysis.

Appendix B.1. Proof of Lemma 1

Recall that ${\ddot{\mu }}_t(\gamma ,i)={\displaystyle \frac{1}{N_t(\gamma ,i)}}{\sum }_{j=1}^t{\gamma }^{t-j}\mathbb{1}\{i_j=i\}{\mu }_j(i)$. Since ${\ddot{\mu }}_t(\gamma ,i)$ is a convex combination of elements ${\mu }_j(i),j=1,\dots ,t$, we have

$$|{\mu }_t(i)-{\ddot{\mu }}_t(\gamma ,i)|\le {\Delta }_{max}^T$$

(A2)

We can write ${\mu }_t(i)$ as ${\mu }_t(i)={\displaystyle \frac{1}{N_t(\gamma ,i)}}{\sum }_{j=1}^t{\gamma }^{t-j}\mathbb{1}\{i_j=i\}{\mu }_t(i)$. Thus, we have

$$|{\mu }_t(i)-{\ddot{\mu }}_t(\gamma ,i)|={\displaystyle \frac{1}{N_t(\gamma ,i)}}\left|{\displaystyle \sum _{j=1}^t}{\gamma }^{t-j}({\mu }_j(i)-{\mu }_t(i))\mathbb{1}\{i_j=i\}\right|.$$

Recall that $\mathcal{T}(\gamma )=\{t\le T:\forall s\in (t-D(\gamma ),t],{\mu }_s(·)={\mu }_t(·)\}$. If $t\in \mathcal{T}(\gamma )$, we have ${\mu }_j(i)={\mu }_t(i),\forall j\in (t-D(\gamma ),t).$

Therefore, $\forall t\in \mathcal{T}(\gamma )$, we have

$$\begin{array}{cc}\hfill |{\mu }_t(i)-{\ddot{\mu }}_t(\gamma ,i)|& ={\displaystyle \frac{1}{N_t(\gamma ,i)}}\left|\sum _{j=1}^{t-D(\gamma )}{\gamma }^{t-j}({\mu }_j(i)-{\mu }_t(i))\mathbb{1}\{i_j=i\}\right|\hfill \\ & \le {\displaystyle \frac{{\Delta }_{max}^T}{N_t(\gamma ,i)}}\sum _{j=1}^{t-D(\gamma )}{\gamma }^{t-j}\mathbb{1}\{i_j=i\}\hfill \\ & ={\displaystyle \frac{{\Delta }_{max}^T}{N_t(\gamma ,i)}}{\gamma }^{D(\gamma )}{N}_{t-D(\gamma )}(\gamma ,i)\hfill \\ & \le {\displaystyle \frac{{\Delta }_{max}^T{\gamma }^{D(\gamma )}}{N_t(\gamma ,i)(1-\gamma )}}\hfill \end{array}$$

where the last inequality follows from $N_{t-D(\gamma )}(\gamma ,i)\le {\displaystyle \frac{1}{1-\gamma }}$.

If ${\displaystyle \frac{{\gamma }^{D(\gamma )}}{N_t(\gamma ,i)(1-\gamma )}}<1$, ${\displaystyle \frac{{\gamma }^{D(\gamma )}}{N_t(\gamma ,i)(1-\gamma )}}<\sqrt{{\displaystyle \frac{{\gamma }^{D(\gamma )}}{N_t(\gamma ,i)(1-\gamma )}}}$, we have

$$|{\mu }_t(i)-{\ddot{\mu }}_t(\gamma ,i)|\le {\Delta }_{max}^T\sqrt{{\displaystyle \frac{{\gamma }^{D(\gamma )}}{N_t(\gamma ,i)(1-\gamma )}}}.$$

If ${\displaystyle \frac{{\gamma }^{D(\gamma )}}{N_t(\gamma ,i)(1-\gamma )}}\ge 1$, from Equation (A2), we also have

$$|{\mu }_t(i)-{\ddot{\mu }}_t(\gamma ,i)|\le {\Delta }_{max}^T\le {\Delta }_{max}^T\sqrt{{\displaystyle \frac{{\gamma }^{D(\gamma )}}{N_t(\gamma ,i)(1-\gamma )}}}.$$

By the definition of $D(\gamma )={\displaystyle \frac{log({(\frac{\sigma }{{\Delta }_{max}^T})}^2{(1-\gamma )}^{2}log\frac{1}{1-\gamma })}{log\gamma }}$,

$$|{\mu }_t(i)-{\ddot{\mu }}_t(\gamma ,i)|\le \sigma \sqrt{{\displaystyle \frac{(1-\gamma )log\frac{1}{1-\gamma }}{N_t(\gamma ,i)}}}$$

Appendix B.2. Proof of Lemma 2

From the definition of $n,A(\gamma ),U_t(\gamma ,i)$ in Equation (4), we can obtain

$$U_t(\gamma ,i)={\displaystyle \frac{\sqrt{1-\gamma }{\Delta }_T(i)}{n}}\sqrt{{\displaystyle \frac{A(\gamma )}{N_t(\gamma ,i)}}}.$$

(A3)

If $N_t(\gamma ,i)>A(\gamma )$, $U_t(\gamma ,i)<{\displaystyle \frac{\sqrt{1-\gamma }}{n}}{\Delta }_T(i)$. Thus, we have

$${\displaystyle \frac{{\Delta }_t(i)}{3}}-U_t(\gamma ,i)>{\displaystyle \frac{{\Delta }_T(i)}{3}}-{\displaystyle \frac{\sqrt{1-\gamma }}{n}}{\Delta }_T(i)={\displaystyle \frac{2\sqrt{2}}{n}}{\Delta }_T(i).$$

(A4)

Therefore,

$$\begin{array}{cc}\hfill & \mathbb{P}({\widehat{\mu }}_t(\gamma ,i)>{\mu }_t(i)+{\displaystyle \frac{{\Delta }_t(i)}{3}},N_t(\gamma ,i)>A(\gamma ))\hfill \\ \hfill & \stackrel{(a)}{\le }\mathbb{P}({\widehat{\mu }}_t(\gamma ,i)-{\ddot{\mu }}_t(\gamma ,i)>{\displaystyle \frac{{\Delta }_t(i)}{3}}-U_t(\gamma ,i),N_t(\gamma ,i)>A(\gamma ))\hfill \\ \hfill & \stackrel{(b)}{\le }\mathbb{P}({\widehat{\mu }}_t(\gamma ,i)-{\ddot{\mu }}_t(\gamma ,i)>{\displaystyle \frac{2\sqrt{2}}{n}}{\Delta }_T(i),N_t(\gamma ,i)>A(\gamma ))\hfill \\ \hfill & \stackrel{(c)}{\le }\mathbb{P}({\displaystyle \frac{N_t(\gamma ,i)({\widehat{\mu }}_t(\gamma ,i)-{\ddot{\mu }}_t(\gamma ,i))}{\sqrt{N_t({\gamma }^2,i)}}}>{\displaystyle \frac{2\sqrt{2}}{n}}{\Delta }_T(i)\sqrt{A(\gamma )})\hfill \\ \hfill & \stackrel{(d)}{\le }log{\displaystyle \frac{1}{1-\gamma }}exp(-{\displaystyle \frac{3{({\Delta }_T(i))}^2}{n^2{\sigma }^2}}A(\gamma ))\hfill \\ \hfill & \le {(1-\gamma )}^{3}log{\displaystyle \frac{1}{1-\gamma }}\hfill \end{array}$$

(A5)

where (a) uses Lemma 1, (b) uses Equation (A4), (c) follows from $N_t(\gamma ,i)>N_t({\gamma }^2,i)$, (d) uses Lemma A1.

Since $(1-\gamma )log{\displaystyle \frac{1}{1-\gamma }}\le {\displaystyle \frac{1}{e}}<1$, this ends the proof.

Appendix B.3. Proof of Lemma 3

This proof is adapted from [32] for the stationary settings. However, there are some technical problems that are difficult to overcome in non-stationary settings. The tricky problem is to lower bound the probability of the mean’s estimation of optimal arm Equation (A9). By designing the function $U_t(\gamma ,i)$ and decomposing the regret to use Lemma A3 again, we solve this challenge. We use blue font to emphasize the techniques used in the proof.

The proof is in three steps.

Step 1: We first prove that $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]$ has an upper bound independent of t.

Define a Bernoulli experiment as sampling from $\mathcal{N}({\widehat{\mu }}_t(\ast ),{\displaystyle \frac{4{\sigma }^2}{N_t(\gamma ,\ast )}})$, where success implies that ${\theta }_t(\ast )>y_t(i)$. Let $G_t$ denote the number of experiments performed when the event $\{{\theta }_t(\ast )>y_t(i)\}$ first occurs. Then,

$$\mathbb{E}[{\displaystyle \frac{1}{p_{i,t}}}]=\mathbb{E}\left[\mathbb{E}[G_t\mid {\mathcal{F}}_{t-1}]\right]=\mathbb{E}\left[G_t\right]$$

Let $z=\sqrt{logr}+{\displaystyle \frac{1}{2}}$ ($r\ge 1$ is an integer ) and let ${\mathrm{MAX}}_r$ denote the maximum of r independent Bernoulli experiments. Then,

$$\begin{array}{cc}\hfill \mathbb{P}(G_t\le r)& \ge \mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\ge y_t(i))\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{E}\left[\mathbb{1}\{{\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\ge y_t(i)\}\mid {\mathcal{F}}_{t-1}\right]\right]\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{1}\{{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\ge y_t(i)\}·\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\mid {\mathcal{F}}_{t-1})\right]\hfill \end{array}$$

(A6)

Using Fact A1,

$$\begin{array}{cc}\hfill \mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\mid {\mathcal{F}}_{t-1})& \ge 1-{\left(1-{\displaystyle \frac{1}{\sqrt{2\pi }}}{\displaystyle \frac{z}{z^2+1}}{e}^{-z^2/2}\right)}^r\hfill \\ \hfill & =1-{\left(1-{\displaystyle \frac{1}{\sqrt{2\pi }}}{\displaystyle \frac{\sqrt{logr}+\frac{1}{2}}{{(\sqrt{logr}+\frac{1}{2})}^2+1}}{\displaystyle \frac{e^{-1/4-\sqrt{logr}/2}}{\sqrt{r}}}\right)}^r\hfill \\ \hfill & \ge 1-e^{-{\displaystyle \frac{\sqrt{r}{e}^{-\sqrt{logr}/2}}{e^{0.25}\sqrt{2\pi }(\sqrt{logr}+1)}}}\hfill \end{array}$$

(A7)

For any $r\ge e^{17}$, $e^{-{\displaystyle \frac{\sqrt{r}{e}^{-\sqrt{logr}/2}}{e^{0.25}\sqrt{2\pi }(\sqrt{logr}+1)}}}\le {\displaystyle \frac{1}{r^2}}$. Hence, for any $r\ge e^{17}$,

$$\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\mid {\mathcal{F}}_{t-1})\ge 1-{\displaystyle \frac{1}{r^2}}.$$

Therefore, for any $r\ge e^{17}$,

$$\mathbb{P}(G_t\le r)\ge (1-{\displaystyle \frac{1}{r^2}})\mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z}{\sqrt{N_t(\gamma ,\ast )}}}\ge y_t(i))$$

Next, we apply Lemma A1 to lower bound $\mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\ge y_t(i))$.

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\ge y_t(i))& \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\le {\mu }_t(\ast ))\hfill \\ \hfill & \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )-{\ddot{\mu }}_t(\ast )\le U_t(\gamma ,\ast )-{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}})\hfill \end{array}$$

(A8)

Since $U_t(\gamma ,\ast )={\displaystyle \frac{\sigma \sqrt{(1-\gamma )log\frac{1}{1-\gamma }}}{\sqrt{N_t(\gamma ,\ast )}}},z=logr+{\displaystyle \frac{1}{2}}$,

$$U_t(\gamma ,\ast )-{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}={\displaystyle \frac{\sigma \sqrt{(1-\gamma )log\frac{1}{1-\gamma }}-\sigma -2\sigma \sqrt{logr}}{\sqrt{N_t(\gamma ,\ast )}}}<-{\displaystyle \frac{2\sigma \sqrt{logr}}{\sqrt{N_t(\gamma ,\ast )}}}.$$

Then, we have

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}\ge y_t(i))& \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )-{\ddot{\mu }}_t(\ast )<-{\displaystyle \frac{2\sigma \sqrt{logr}}{\sqrt{N_t(\gamma ,\ast )}}})\hfill \\ \hfill & \ge 1-log({\displaystyle \frac{1}{1-\gamma }})e^{-{\displaystyle \frac{3}{2}}logr}\hfill \\ \hfill & \ge 1-log{\displaystyle \frac{1}{1-\gamma }}{\displaystyle \frac{1}{r^{1.5}}}.\hfill \end{array}$$

(A9)

Substituting, for any $r>e^{17}$,

$$\mathbb{P}(G_t\le r)\ge 1-log{\displaystyle \frac{1}{1-\gamma }}{\displaystyle \frac{1}{r^{1.5}}}-{\displaystyle \frac{1}{r^2}}$$

(A10)

Therefore,

$$\begin{array}{cc}\hfill \mathbb{E}\left[G_t\right]& =\sum _{r=0}^{\infty }\mathbb{P}(G_t\ge r)\hfill \\ \hfill & \le 1+e^{17}+\sum _{r>e^{17}}(log{\displaystyle \frac{1}{1-\gamma }}{\displaystyle \frac{1}{r^{1.5}}}+{\displaystyle \frac{1}{r^2}})\hfill \\ \hfill & \le e^{17}+3+3log{\displaystyle \frac{1}{1-\gamma }}\hfill \end{array}$$

(A11)

This proves a bound of $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]\le e^{17}+3+3log{\displaystyle \frac{1}{1-\gamma }}$ independent of t.

Step 2: Define $L(\gamma )={\displaystyle \frac{1152log(\frac{1}{1-\gamma }+e^{17}){\sigma }^2}{{({\Delta }_T(i))}^2}}$. We consider the upper bound of $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]$ when $N_t(\gamma ,\ast )>L(\gamma )$.

$$\begin{array}{cc}\hfill \mathbb{P}(G_t\le r)& \ge \mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\ge y_t(i))\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{1}\right\{{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\ge y_t(i)\}\hfill \\ \hfill & ·\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\mid {\mathcal{F}}_{t-1})]\hfill \end{array}$$

(A12)

Now, since $N_t(\gamma ,\ast )>L(\gamma )$,${\displaystyle \frac{1}{\sqrt{N_t(\gamma ,\ast )}}}<{\displaystyle \frac{{\Delta }_t(i)}{48\sqrt{log(\frac{1}{1-\gamma }+e^{17})}\sigma }}$. Therefore, for any $r\le {({\displaystyle \frac{1}{1-\gamma }}+e^{17})}^2$,

$${\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}={\displaystyle \frac{2\sigma \sqrt{logr}+\sigma }{\sqrt{N_t(\gamma ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\le -{\displaystyle \frac{{\Delta }_t(i)}{12}}.$$

Using Fact A1,

$$\mathbb{P}({\theta }_t(i)>{\widehat{\mu }}_t(i)-{\displaystyle \frac{{\Delta }_t(i)}{12}}|{\mathcal{F}}_{t-1})\le 1-{\displaystyle \frac{1}{2}}{e}^{-{\displaystyle \frac{N_t(\gamma ,\ast )}{4{\sigma }^2}}{\displaystyle \frac{{\Delta }_t{(i)}^2}{288}}}\ge 1-{\displaystyle \frac{1}{2(1/(1-\gamma )+e^{17})}}.$$

This implies

$$\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z}{\sqrt{N_t(\gamma ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}|{\mathcal{F}}_{t-1})\ge 1-{\displaystyle \frac{1}{2^r{(1/(1-\gamma )+e^{17})}^r}}.$$

Also, apply the self-normalized Hoeffding-type inequality,

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\gamma ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\ge y_t(i))& \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )\le {\mu }_t(\ast )-{\displaystyle \frac{{\Delta }_t(i)}{6}})\hfill \\ & \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )-{\ddot{\mu }}_t(\ast )\ge -U_t(\gamma ,\ast )+{\displaystyle \frac{{\Delta }_t(i)}{6}})\hfill \\ & >1-\mathbb{P}({\widehat{\mu }}_t(\ast )-{\ddot{\mu }}_t(\ast )\ge {\displaystyle \frac{{\Delta }_T(i)}{8}}\sqrt{{\displaystyle \frac{L(\gamma )}{N_t(\gamma ,\ast )}}})\hfill \\ & \ge 1-log({\displaystyle \frac{1}{1-\gamma }}+e^{17}){\displaystyle \frac{1}{{(1/(1-\gamma )+e^{17})}^3}}.\hfill \end{array}$$

Let ${\gamma }^{\prime }={({\displaystyle \frac{1}{1-\gamma }}+e^{17})}^2$. Therefore, for any $1\le r\le {\gamma }^{\prime }$,

$$\mathbb{P}(G_t\le r)\ge 1-{\displaystyle \frac{1}{2^r{{\gamma }^{\prime }}^{r/2}}}-log({\displaystyle \frac{1}{1-\gamma }}+e^{17}){\displaystyle \frac{1}{{{\gamma }^{\prime }}^{1.5}}}.$$

When $r\ge {\gamma }^{\prime }>e^{17}$, we can use Equation (A10) to obtain

$$\mathbb{P}(G_t\le r)\ge 1-log{\displaystyle \frac{1}{1-\gamma }}{\displaystyle \frac{1}{r^{1.5}}}-{\displaystyle \frac{1}{r^2}}$$

Combining these results

$$\begin{array}{cc}\hfill \mathbb{E}\left[G_t\right]& \le \sum _{r=0}^{\infty }\mathbb{P}(G_t\ge r)\hfill \\ & \le 1+\sum _{r=1}^{{\gamma }^{\prime }}\mathbb{P}(G_t\ge r)+\sum _{r={\gamma }^{\prime }}^{\infty }\mathbb{P}(G_t\ge r)\hfill \\ & \le 1+\sum _{r=1}^{{\gamma }^{\prime }}({\displaystyle \frac{1}{2^r{{\gamma }^{\prime }}^{r/2}}}+log({\displaystyle \frac{1}{1-\gamma }}+e^{17}){\displaystyle \frac{1}{{{\gamma }^{\prime }}^{1.5}}})+\sum _{r={\gamma }^{\prime }}^{\infty }(log{\displaystyle \frac{1}{1-\gamma }}{\displaystyle \frac{1}{r^{1.5}}}+{\displaystyle \frac{1}{r^2}})\hfill \\ & \le 1+{\displaystyle \frac{1}{2\sqrt{{\gamma }^{\prime }}}}+log({\displaystyle \frac{1}{1-\gamma }}+e^{17}){\displaystyle \frac{1}{\sqrt{{\gamma }^{\prime }}}}+{\displaystyle \frac{2}{{\gamma }^{\prime }}}+log({\displaystyle \frac{1}{1-\gamma }}){\displaystyle \frac{3}{\sqrt{{\gamma }^{\prime }}}}\hfill \\ & \le 1+6(1-\gamma )log({\displaystyle \frac{1}{1-\gamma }}+e^{17}).\hfill \end{array}$$

Therefore, when $N_t(\gamma ,\ast )>L(\gamma )$, it holds that

$$\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]-1=\mathbb{E}\left[G_t\right]-1\le 6(1-\gamma )log({\displaystyle \frac{1}{1-\gamma }}+e^{17}).$$

Step 3: Let $\mathcal{A}(\gamma ,\ast )=\{t\in \{1,\dots ,T\}:N_t(\gamma ,\ast )\le L(\gamma )\}$.

$$\begin{array}{cc}\hfill & \sum _{t\in \mathcal{T}(\gamma )}\mathbb{E}[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },{\theta }_t(i)<y_t(i)\}]\hfill \\ \hfill & \le \left(\sum _{t\in \mathcal{T}(\gamma )\cap \mathcal{A}(\gamma ,\ast )}+\sum _{t\in \mathcal{T}(\gamma )\setminus \mathcal{A}(\gamma ,\ast )}\right)\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },{\theta }_t(i)<y_t(i)\}\right]\hfill \\ \hfill & \le |\{t:i_t=i_t^{\ast },N_t(\gamma ,\ast )\le L(\gamma )\}|(e^{17}+3+3log{\displaystyle \frac{1}{1-\gamma }})+\sum _{t\in \mathcal{T}(\gamma )\setminus \mathcal{A}(\gamma ,\ast )}\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\right]\hfill \\ \hfill & \le T(1-\gamma )L(\gamma ){\gamma }^{-1/(1-\gamma )}(e^{17}+3+3log{\displaystyle \frac{1}{1-\gamma }})+6T(1-\gamma )log({\displaystyle \frac{1}{1-\gamma }}+e^{17})\hfill \\ \hfill & \le (e^{17}+9+3log{\displaystyle \frac{1}{1-\gamma }})T(1-\gamma )L(\gamma ){\gamma }^{-1/(1-\gamma )}.\hfill \end{array}$$

(A13)

Appendix B.4. Larger Variance

In fact, Lemma A1 has a stricter upper bound as ${\displaystyle \frac{log\frac{1}{1-\gamma }}{log(1+\eta )}}exp(-{\displaystyle \frac{1}{2{\sigma }^2}}(1-{\displaystyle \frac{{\eta }^2}{16}}))$, where $\eta \in (-1,4)$. Let $\eta =2$, then we obtain Lemma A1. Suppose the variance of Thompson sampling is ${\displaystyle \frac{\xi {\sigma }^2}{N_t(\gamma ,i)}}$. The lower bound of Equation (A9) becomes

$${\displaystyle \frac{log\frac{1}{1-\gamma }}{log(1+\eta )}}exp(-{\displaystyle \frac{\xi logr}{2}}(1-{\displaystyle \frac{{\eta }^2}{16}})).$$

To ensure that $\mathbb{E}\left[G_t\right]$ in Equation (A11) has a finite upper bound, our analysis method requires the infinite series

$$\sum _{r=1}^{\infty }exp(-{\displaystyle \frac{\xi logr}{2}}(1-{\displaystyle \frac{{\eta }^2}{16}}))=\sum _{r=1}^{\infty }{\displaystyle \frac{1}{r^{\frac{\xi }{2}(1-\frac{{\eta }^2}{16})}}}$$

is convergent. Thus, we have

$$\xi >{\displaystyle \frac{2}{1-\frac{{\eta }^2}{16}}}>2,$$

i.e., the sampling variance needs to be strictly greater than ${\displaystyle \frac{2{\sigma }^2}{N_t(\gamma ,i)}}$.

Appendix B.5. Proof of Lemma 4

Recall that $A(\tau )={\displaystyle \frac{72log(\tau ){\sigma }^2}{{({\Delta }_T(i))}^2}}$. Using Lemma A1, we have

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\mu }}_t(\tau ,i)>x_t(i),N_t(\tau ,i)>A(\tau ))& =\mathbb{P}({\widehat{\mu }}_t(\tau ,i)-{\mu }_t(i)>{\displaystyle \frac{{\Delta }_t(i)}{3}},N_t(\tau ,i)>A(\gamma ))\hfill \\ \hfill & \le \mathbb{P}(\sqrt{N_t(\tau ,i)}({\widehat{\mu }}_t(\tau ,i)-{\mu }_t(i))>{\displaystyle \frac{{\Delta }_T(i)}{3}}\sqrt{A(\gamma )})\hfill \\ \hfill & \le log\tau exp(-{\displaystyle \frac{3{({\Delta }_T(i))}^2}{72{\sigma }^2}}A(\gamma ))\hfill \\ \hfill & \le {\displaystyle \frac{1}{{\tau }^2}}\hfill \end{array}$$

(A14)

Appendix B.6. Proof of Lemma 5

The proof is similar to the proof of Lemma 3.

Step 1: We first prove that $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]$ has an upper bound independent of t.

Let $z=\sqrt{logr}$ ($r\ge 1$ is an integer). Then,

$$\begin{array}{cc}\hfill \mathbb{P}(G_t\le r)& \ge \mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\ge y_t(i))\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{1}\{{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\ge y_t(i)\}\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\mid {\mathcal{F}}_{t-1})\right]\hfill \end{array}$$

(A15)

Using Fact A1,

$$\begin{array}{cc}\hfill \mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\mid {\mathcal{F}}_{t-1})& \ge 1-{\left(1-{\displaystyle \frac{1}{\sqrt{2\pi }}}{\displaystyle \frac{z}{z^2+1}}{e}^{-z^2/2}\right)}^r\hfill \\ \hfill & =1-{\left(1-{\displaystyle \frac{1}{\sqrt{2\pi }}}{\displaystyle \frac{\sqrt{logr}}{{(\sqrt{logr})}^2+1}}{\displaystyle \frac{1}{\sqrt{r}}}\right)}^r\hfill \\ \hfill & \ge 1-e^{-{\displaystyle \frac{\sqrt{r}}{\sqrt{2\pi }(\sqrt{logr}+1)}}}\hfill \end{array}$$

(A16)

For any $r\ge e^{11}$, $e^{-{\displaystyle \frac{\sqrt{r}}{\sqrt{2\pi }(\sqrt{logr}+1)}}}\le {\displaystyle \frac{1}{r^2}}$. Hence, for any $r\ge e^{11}$,

$$\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\mid {\mathcal{F}}_{t-1})\ge 1-{\displaystyle \frac{1}{r^2}}.$$

Therefore, for any $r\ge e^{11}$,

$$\mathbb{P}(G_t\le r)\ge (1-{\displaystyle \frac{1}{r^2}})\mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z}{\sqrt{N_t(\tau ,\ast )}}}\ge y_t(i))$$

Next, we apply Lemma A1 to lower bound $\mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\ge y_t(i))$.

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\ge y_t(i))& \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}\le {\mu }_t(\ast ))\hfill \\ & \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )-{\mu }_t(\ast )<-{\displaystyle \frac{2\sigma \sqrt{logr}}{\sqrt{N_t(\tau ,\ast )}}})\hfill \\ & \ge 1-log\tau e^{-{\displaystyle \frac{3}{2}}logr}\hfill \\ & =1-log\tau {\displaystyle \frac{1}{r^{1.5}}}.\hfill \end{array}$$

Substituting, for any $r>e^{11}$,

$$\mathbb{P}(G_t\le r)\ge 1-log\tau {\displaystyle \frac{1}{r^{1.5}}}-{\displaystyle \frac{1}{r^2}}$$

(A17)

Therefore,

$$\begin{array}{cc}\hfill \mathbb{E}\left[G_t\right]& =\sum _{r=0}^{\infty }\mathbb{P}(G_t\ge r)\hfill \\ & \le 1+e^{11}+\sum _{r>e^{11}}(log\tau {\displaystyle \frac{1}{r^{1.5}}}+{\displaystyle \frac{1}{r^2}})\hfill \\ & \le e^{11}+3+3log\tau \hfill \end{array}$$

This proves a bound of $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]\le e^{11}+3+3log\tau$ independent of t.

Step 2: Define $L(\tau )={\displaystyle \frac{1152log(\tau +e^{11}){\sigma }^2}{{({\Delta }_T(i))}^2}}$. We consider the upper bound of $\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]$ when $N_t(\tau ,\ast )>L(\tau )$.

$$\begin{array}{cc}\hfill \mathbb{P}(G_t\le r)& \ge \mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\ge y_t(i))\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{1}\right\{{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\ge y_t(i)\}\hfill \\ \hfill & ·\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\mid {\mathcal{F}}_{t-1})]\hfill \end{array}$$

(A18)

Now, since $N_t(\tau ,\ast )>L(\tau )$,${\displaystyle \frac{1}{\sqrt{N_t(\tau ,\ast )}}}<{\displaystyle \frac{{\Delta }_t(i)}{48\sqrt{log(\tau +e^{11})}\sigma }}$. Therefore, for any $r\le {(\tau +e^{11})}^2$,

$${\displaystyle \frac{z·2\sigma }{\sqrt{N_t(\tau ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}={\displaystyle \frac{2\sigma \sqrt{logr}+\sigma }{\sqrt{N_t(\tau ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\le -{\displaystyle \frac{{\Delta }_t(i)}{12}}.$$

Using Fact A1,

$$\mathbb{P}({\theta }_t(i)>{\widehat{\mu }}_t(i)-{\displaystyle \frac{{\Delta }_t(i)}{12}}|{\mathcal{F}}_{t-1})\le 1-{\displaystyle \frac{1}{2}}{e}^{-{\displaystyle \frac{N_t(\tau ,\ast )}{4{\sigma }^2}}{\displaystyle \frac{{\Delta }_t{(i)}^2}{288}}}\ge 1-{\displaystyle \frac{1}{2(\tau +e^{11})}}.$$

This implies

$$\mathbb{P}({\mathrm{MAX}}_r>{\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z}{\sqrt{N_t(\tau ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}|{\mathcal{F}}_{t-1})\ge 1-{\displaystyle \frac{1}{2^r{(\tau +e^{11})}^r}}.$$

Also, apply Lemma A1,

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\mu }}_t(\ast )+{\displaystyle \frac{z}{\sqrt{N_t(\tau ,\ast )}}}-{\displaystyle \frac{{\Delta }_t(i)}{6}}\ge y_t(i))& \ge 1-\mathbb{P}({\widehat{\mu }}_t(\ast )-{\mu }_t(\ast )\ge {\displaystyle \frac{{\Delta }_t(i)}{6}})\hfill \\ & \ge 1-log(\tau +e^{11}){\displaystyle \frac{1}{{(\tau +e^{11})}^3}}.\hfill \end{array}$$

Let ${\tau }^{\prime }={(\tau +e^{11})}^2$. Therefore, for any $1\le r\le {\tau }^{\prime }$,

$$\mathbb{P}(G_t\le r)\ge 1-{\displaystyle \frac{1}{2^r{{\tau }^{\prime }}^{r/2}}}-log(\tau +e^{11}){\displaystyle \frac{1}{{{\tau }^{\prime }}^{1.5}}}.$$

When $r\ge {\tau }^{\prime }>e^{11}$, we can use Equation (A10) to obtain

$$\mathbb{P}(G_t\le r)\ge 1-log\tau {\displaystyle \frac{1}{r^{1.5}}}-{\displaystyle \frac{1}{r^2}}$$

Combining these results,

$$\begin{array}{cc}\hfill \mathbb{E}\left[G_t\right]& \le \sum _{r=0}^{\infty }\mathbb{P}(G_t\ge r)\hfill \\ & \le 1+\sum _{r=1}^{{\tau }^{\prime }}\mathbb{P}(G_t\ge r)+\sum _{r={\tau }^{\prime }}^{\infty }\mathbb{P}(G_t\ge r)\hfill \\ & \le 1+{\displaystyle \frac{6}{\tau }}log(\tau +e^{11}).\hfill \end{array}$$

Therefore, when $N_t(\tau ,\ast )>L(\tau )$, it holds that

$$\mathbb{E}\left[{\displaystyle \frac{1}{p_{i,t}}}\right]-1=\mathbb{E}\left[G_t\right]-1\le {\displaystyle \frac{6}{\tau }}log(\tau +e^{11}).$$

Step 3: Let $\mathcal{A}(\tau ,\ast )=\{t\in \{1,\dots ,T\}:N_t(\tau ,\ast )\le L(\tau )\}$ and $C=e^{11}+9$.

$$\begin{array}{cc}\hfill & \sum _{t\in \mathcal{T}(\tau )}\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },{\theta }_t(i)<y_t(i)\}\right]\hfill \\ \hfill & \le \left(\sum _{t\in \mathcal{T}(\tau )\cap \mathcal{A}(\tau ,\ast )}+\sum _{t\in \mathcal{T}(\tau )\setminus \mathcal{A}(\tau ,\ast )}\right)\mathbb{E}\left[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}\mathbb{1}\{i_t=i_t^{\ast },{\theta }_t(i)<y_t(i)\}\right]\hfill \\ \hfill & \le |\{t:i_t=i_t^{\ast },N_t(\tau ,\ast )\le L(\tau )\}|(e^{11}+3+3log\tau )+\sum _{t\in \mathcal{T}(\tau )\setminus \mathcal{A}(\tau ,\ast )}\mathbb{E}[{\displaystyle \frac{1-p_{i,t}}{p_{i,t}}}]\hfill \\ \hfill & \le {\displaystyle \frac{T}{\tau }}L(\tau )(e^{11}+3+3log\tau )+6{\displaystyle \frac{T}{\tau }}log(\tau +e^{11})\hfill \\ \hfill & \le (e^{11}+9+3log\tau ){\displaystyle \frac{T}{\tau }}L(\tau ).\hfill \end{array}$$

(A19)