Adaptive Task Scheduling for Edge-Intelligent Systems: An Online Sleeping Restless Bandits Framework

Abstract

In edge-intelligent systems, efficient resource management and task scheduling are critical but challenging due to the dynamic and heterogeneous nature of edge nodes (e.g., IoT devices, drones). We model this dynamic resource allocation challenge as an online sleeping Restless Multi-Armed Bandits (RMAB) problem, where each edge node (arm) operates as a Markov decision process. Unlike prior RMAB frameworks assuming perpetual availability, our setting captures the stochastic availability of edge nodes across rounds. The system controller (learner) is unaware of the transition functions, reward distributions, and node availability a priori. The goal is to maximize expected cumulative rewards through adaptive node selection. To explore this target problem, we first derive an asymptotically optimal sleeping-index policy (SIP) as the oracle based on the fluid process transformation. Then we propose OSILA (Online Sleeping Index-aware Learning Algorithm), featuring a Minimum Exploration Guarantee (MEG) mechanism for efficient exploration. This is coupled with a modified Linear Programming-based exploitation mechanism to construct an online sleeping index, effectively handling dynamic node availability. To the best of our knowledge, this work is the first to provide the theoretical analysis (which achieves $\tilde{\mathcal{O}}\left(K{T}^{2/3}\sqrt{logT}\right)$ regret where K is the number of arms and T is the time horizon) to the online sleeping RMAB problem. Empirical results validate both theoretical guarantees and practical effectiveness in dynamic edge computing environments.

1. Introduction

With the rapid evolution of 5G networks, the Internet of Things (IoT), and smart manufacturing, edge computing has emerged as a transformative paradigm [1,2]. By processing tasks closer to the data source rather than relying solely on centralized cloud servers, edge computing drastically reduces latency, enhances reliability, and improves data privacy. As the volume of multi-modal data grows, and the demand for personalized, intelligent services (such as AR/VR, autonomous drones, and smart healthcare) increases, traditional cloud computing architectures struggle to handle the massive scale and maintain rigorous Quality of Service (QoS) standards. Consequently, “edge intelligence”—the seamless integration of networking, computing, storage, and AI applications at the network edge—has become a critical enabler for next-generation technological applications.

Despite its immense potential, realizing true edge intelligence is fraught with heterogeneous challenges. Edge environments are inherently dynamic and symmetric/asymmetric in topology; edge nodes (e.g., IoT devices, mobile sensors, and drones) often face severe resource constraints, including limited battery life, fluctuating communication channel qualities, and varying computational capacities. These constraints lead to the stochastic availability of resources, where edge nodes may frequently alternate between active and inactive (or unreachable) states. Therefore, designing efficient models and architecture frameworks for task scheduling, computation offloading, and resource management in such dynamic, resource-constrained systems remains an open and highly active research area.

This practical importance is also reflected in the recent edge-computing literature, where task scheduling, computation offloading, and resource allocation have been extensively studied from system, algorithmic, and application perspectives [3,4,5,6,7]. However, these works mainly focus on direct scheduling or coordination mechanisms in edge systems, rather than online restless bandit learning under stochastic arm availability.

To mathematically capture and solve this complex resource management challenge, the Restless Multi-Armed Bandits (RMAB) model [8] has been widely applied to sequential decision-making problems across various domains, including wireless networks [9,10], resource allocation [11,12], and healthcare [13]. This model typically involves a learner (e.g., a central edge controller) interacting with $K\in {\mathbb{N}}^{+}$ arms (representing edge nodes), each described as a Markov Decision Process (MDP). In each round of RMAB, the learner selects $M\le K$ arms to pull, after which each arm transits to new states based on its stationary transition kernel. Rewards (e.g., system throughput, QoS, or negative latency) are stochastically generated based on the arms’ state, and the learner’s goal is to maximize the expected cumulative reward.

The existing RMAB literature primarily focuses on static environments with a fixed decision space, where all arms are assumed to be perpetually available in every round [14,15,16,17]. However, in edge-intelligent systems, the decision space is intrinsically dynamic. Two practical edge computing applications are presented below.

(1)

Application 1: Task Offloading in Wireless Edge Networks. The decision maker (edge server) needs to select which end devices to offload tasks to or retrieve data from. Each device exhibits restless behavior: its internal state (e.g., buffer level, channel quality, energy reserve) autonomously evolves over time [18,19,20]. However, due to environmental interference or energy depletion, the availability of each device follows an unknown distribution, resulting in a dynamically changing set of selectable devices. Consequently, system transitions heavily depend on which devices are available in each round, fundamentally altering the policy optimization.

(2)

Application 2: Drone-Assisted Data Collection. In smart city or agricultural IoT scenarios, drones serve as mobile edge nodes to collect data from distributed sensors. Each sensor’s state (e.g., data freshness, storage capacity) changes continuously. Due to the drone’s mobility and varying weather conditions, a sensor’s availability to establish a reliable communication link shifts over time. This dynamic availability implies that state transitions and data collection efficiency depend not only on the chosen scheduling action but also on the real-time availability of the communication links, complicating the design of symmetric data distribution and scheduling policies.

To capture the dynamic nature of the decision space, a series of existing literature [21,22,23] modeled the dynamic decision space as sleeping bandits, where the set of available arms for the learner’s decision in each round follows an unknown distribution that varies stochastically. However, the RMAB problem under the sleeping setting has not been adequately considered or studied to address the real-world heterogeneous challenges in edge computing as mentioned above. The critical challenges mainly lie in the complex interplay between node availability and restless Markovian reward optimization. This gap motivates our work.

In this paper, we formulate a framework for edge-intelligent systems by considering the online RMAB problem under a sleeping setting. In each round, the learner (edge controller) needs to select at most M nodes from the revealed set of available nodes to interact with. Each node’s availability follows an independent, unknown distribution, and the learner is unaware of the state transition dynamics and reward distributions in advance. The goal of the learner is to minimize the expected cumulative policy regret, which is the difference between the cumulative QoS/reward under the deterministic stationary optimal policy and the proposed online algorithm. This fundamental problem presents critical and new challenges: (1) Traditional RMAB oracles (e.g., the Whittle index policy) lose their asymptotic optimality guarantees under the time-varying action space induced by stochastic node availability; (2) Efficiently learning both the restless nodes’ MDP parameters and their availability distributions requires novel, sample-efficient mechanisms due to their stochastic interdependence; (3) Achieving sub-linear regret requires new analytical tools to mathematically decouple the impact of dynamic node availability from the restless state evolution during the online learning process.

Existing studies in recent years have not been able to address the above challenges [16,17]. In the sleeping bandit setting [21], where both the availability and reward are stochastic, an efficient online policy is to pull the arm with the highest upper confidence bound on the reward from the revealed available set in each round, without considering the impact of estimating arm availability on the algorithm. In classic RMAB methods [15,24,25], the availability of arms prevents existing index-aware oracles from guaranteeing asymptotic optimality, making online algorithms infeasible under the sleeping setting, and simple greedy-based modifications cannot achieve significant performance. These online schemes actually have lower exploration efficiency, and the selection methods do not consider arm availability. The reward of each arm depends on its state and action, and actions are coupled with the availability. These factors need to be considered comprehensively. We validate these in Section 6.

To the best of our knowledge, this is the first work to study online sleeping RMAB problems and achieve the $\tilde{\mathcal{O}}\left(T^{2/3}\right)$ sub-linear policy regret. The contributions of this work are summarized below.

• We are the first to investigate the open challenge of the online sleeping Restless Multi-Armed Bandit (RMAB) problem, which demands addressing the complex interplay between arm availability and restless Markovian reward optimization.
• Given the transition dynamics, reward distributions, and availability, we design a sleeping index policy (SIP) as an oracle for the online problem by constructing a fluid process.
• We propose the Minimum Exploration Guarantee (MEG) mechanism that enables efficient exploration for online algorithms in dynamic environments (In this paper, dynamic environments refer to the time-varying available set $I_t$ induced by stochastic arm availability, together with the restless evolution of arm states under stationary transition kernels and fixed availability probabilities, rather than non-stationary model parameters drifting over time ). We employ Laplace analysis to describe the characteristic function of the exploration rounds, and apply Taylor approximation to prove their convergence to the Gumbel distribution under the MEG mechanism. In conjunction with the online SIP derived from the modified LP solution, we present a low-complexity online algorithm OSILA with an $\tilde{\mathcal{O}}\left(K{T}^{2/3}\sqrt{logT}\right)$ regret guarantee.
• Based on real-world applications, we constructed problem instances for experimental validation. The results confirm that SIP is asymptotically optimal in the offline setting. In the online scenario, the regret performance of OSILA outperforms other baseline algorithms.

2. Related Work

Since our motivating applications arise from edge-intelligent systems, we first note that there is indeed a growing body of literature on scheduling in edge computing environments. Recent surveys review resource scheduling and delay-sensitive task scheduling from broad system and application perspectives [3,4], while representative recent studies investigate time-slotted offloading, collaborative multi-server allocation, and multi-edge resource allocation under dynamic system conditions [5,6,7]. These studies are highly relevant from the application side, but they do not address the online sleeping RMAB formulation considered here, where the key challenge is to jointly handle restless state evolution, unknown model parameters, and stochastic arm availability within a regret-minimization framework.

Online learning for Sleeping Bandits. Prior to the study of sleeping bandits, earlier work addressed the sleeping expert problem [26,27,28,29], which is a variation of the sleeping bandits under a full-information feedback mechanism, where the reward of pulling arms is revealed at the end of each round. With bandit feedback taken into account, Kleinberg et al. [21] were the first to study the sleeping bandits, where both rewards and the set of available arms vary stochastically, and they proposed an online algorithm achieving $\tilde{\mathcal{O}}\left(\sqrt{TKlogT}\right)$ regret. Follow-up research can be roughly divided into two categories: one focuses on whether the rewards model and availabilities are stochastic or adversarial [22,23,30,31]. The other explores online combinatorial problems within the sleeping bandits framework [32,33,34]. For the stochastic reward model with adversarial availabilities, Saha et al. [22] achieve a regret of $\tilde{\mathcal{O}}\left(K^2\sqrt{T}\right)$ in the independent setting and $\tilde{\mathcal{O}}\left(\sqrt{2^{K}T}\right)$ in the non-independent setting. Gaillard et al. [31] addressed the scenario where both rewards and availabilities are adversarial, proposing an algorithm with $\tilde{\mathcal{O}}\left(K\sqrt{TlogK}\right)$ regret. Subsequently, Nguyen and Mehta [23] further improved the regret under this setting to $\tilde{\mathcal{O}}\left(\sqrt{TlogK}\right)$. Another research direction was by Neu and Valko [32], who explored the sleeping bandits in a combinatorial setting. Kale et al. [33] analyzed the hardness of the online sleeping combinatorial optimization problems and provided the lower bound of the per-action regret as $\Omega \left(\mathrm{poly}\left(K\right)T^{1-\delta }\right)$.

Online learning for RMAB. When the transition dynamics of each arm are given, Papadimitriou and Tsitsiklis [35] proved that the RMAB problem is P-SPACE hard. Accordingly, a series of works have proposed index-structure-based policy under conditions such as indexability and unichain, guaranteeing asymptotic optimality [24,36,37,38]. When the transition dynamics of the MDP for each arm are unknown, RMAB becomes an online learning problem. Colored-UCRL2 [14] is the first algorithm that provides a sub-linear regret but incurs exponential computational complexity. To overcome this challenge, Wang et al. [15] proposed a low-complexity online algorithm with $\tilde{\mathcal{O}}\left(T^{2/3}\right)$ regret assuming that the oracle is given. Under the assumption of indexability in RMAB, the UCWhittle [16] achieves $\tilde{\mathcal{O}}\left(\sqrt{TlogT}\right)$ regret based on Whittle’s index policy and upper confidence bound. Another category of index-aware approaches based on occupancy measures is extensively applied in finite-horizon [38], infinite-horizon [25], and adversarial reward settings [17], also achieving $\tilde{\mathcal{O}}\left(\sqrt{T}\right)$ regret. Jiang et al. [39] considered the online RMAB where the Markov chain states are unobservable and proposed a Thompson Sampling-based algorithm that achieves $\tilde{\mathcal{O}}\left(\sqrt{T}\right)$ Bayesian regret.

Due to the distinctions between stochastic reward models and restless Markovian reward models, algorithms based on the sleeping bandits framework, such as AUER [21] and LFG [34], are not applicable to the online RMAB problem. Moreover, in RMAB problems with availability constraints, existing index-based policies fail to guarantee asymptotic optimality and are unable to address the dynamic decision space in online scenarios. To the best of our knowledge, the online sleeping RMAB problem remains open, which motivates our research.

3. Model and Problem Statement

In this section, we introduce the sleeping restless bandits model and its online problem formulation.

3.1. Setup: Sleeping RMAB

Suppose that the arm space is $\left[K\right]:=\{1,2,\dots ,K\}$, where K is the number of arms. Each arm $i\in \left[K\right]$ is modeled as an independent Markov Decision Process (MDP) defined by a tuple $(\mathcal{S},\mathcal{A},r_i,P_i)$. The state space $\mathcal{S}$, the action space $\mathcal{A}$, the reward function $r_i:=\mathcal{S}\times \mathcal{A}\mapsto [0,1]$, and the transition probability $P_i:=\mathcal{S}\times \mathcal{A}\times \mathcal{S}\mapsto [0,1]$. We consider a T-round sequential game. At each round t, the learner is presented with a set of available arms, defined as $I_t\subseteq \left[K\right]$, and needs to select $m_t:=min\left\{\right|I_t|,M\}$ arms from this set to pull, where M is the maximum number of arms that can be selected in each round. The availability of arms is defined as follows.

Independent Availabilities. We assume that each arm $i\in \left[K\right]$ is independently available at each round with a probability $z_i\in [0,1]$. Specifically, for all arms $\forall i\in \left[K\right]$, $\mathbb{I}\{i\in I_t\}$∼$Ber\left(z_i\right)$, $z_i$ remains fixed over time, and $z_i$ is independent of $z_j$ for $i\ne j$.

We define an indicator variable $a_i\left(t\right)\in \{0,1\}$, such that $a_i\left(t\right)=1$ indicates the learner selects arm i to pull in round t, and $a_i\left(t\right)=0$ otherwise. The number of arms pulled is constrained by the budget M such that ${\sum }_i{a}_i\left(t\right)\le M$. Let $A_t$ denote the set of arms pulled by the algorithm at round t, where $A_t\subseteq I_t$, and $|A_t|\le M$. Each selected arm i generates a stochastic reward $r_i\left(t\right):=r(s_i\left(t\right),a_i\left(t\right))$, depending on its state $s_i\left(t\right)$ and action $a_i\left(t\right)$. Without loss of generality, we assume that $r_i\left(t\right)\in [0,1]$ with the mean denoted as $\overline{r}(s_i,a_i),\forall i$. The learner’s actions are described by a deterministic policy $\pi :=2^{\left[K\right]}\times {\mathcal{S}}^K\mapsto {\mathcal{A}}^K$. The learner’s goal is to optimize the total expected reward:

$$\begin{array}{cc}\hfill \mathtt{S-RMAB}:& \underset{\pi }{max}\underset{T\to \infty }{lim}inf{\displaystyle \frac{1}{T}}{\mathbb{E}}_{\pi }[\sum _t\sum _{i\in I_t}{r}_i\left(t\right)]\hfill \\ & \mathrm{s}.\mathrm{t}.\sum _{i\in I_t}{a}_i\left(t\right)\le M,\forall t\in \left[T\right].\hfill \end{array}$$

(1)

Case study [AoI (Age of Information (AoI), as a metric for quantifying information freshness, has been widely applied to destination-centric network systems) optimization in edge-intelligent systems]: Consider an edge server gathering real-time sensory data from K edge devices over M unreliable communication channels to support time-sensitive intelligent tasks (e.g., real-time inference or model updates). Edge device $i\in \left[K\right]$ transmits only with available energy, harvested via a Bernoulli process (probability $z_i$) [12,19,40]. The decision maker needs to select available devices for data transmission in each round. The objective in such an edge intelligent system is to minimize the time-average AoI under channel resource constraints to maintain data freshness, i.e., ${min}_{\pi }{lim}_{T\to \infty }{\displaystyle \frac{1}{T}}{\mathbb{E}}_{\pi }\left[{\sum }_i{\sum }_t{h}_i\left(t\right)\right]$, subject to ${\sum }_i{a}_i\left(t\right)\le M$. ($h_i\left(t\right)$ denotes the AoI of device i at time t. Specifically, if the device has sufficient energy and is selected for transmission successfully, then $h_i\left(t\right)=1$; otherwise, $h_i\left(t\right)=h_i(t-1)+1$). The detailed performance evaluation of this case study is given in Section 6 and Appendix C.

3.2. Objective and Online Setting

We focus on the problem of Online Sleeping Restless Multi-Armed Bandits (OS-RMAB), where the learner repeatedly interacts with the available arms $I_t\subseteq \left[K\right]$ in an episodic manner. The learner is unaware of the values of the transition probability $P_i$, the reward function $r_i$, and the availability probabilities $z_i$. Then, the definition of policy regret can be given by

$${\mathrm{Reg}}_T:={\mathbb{E}}_{\pi }[{\displaystyle \sum _{t=1}^T}{J}_{A_t^{★}}-{\displaystyle \sum _{t=1}^T}{J}_{A_t}],$$

(2)

where $J_{A_t^{★}}={\sum }_{i\in A_t^{★}}{r}_i\left(t\right)$ ($J_{A_t}={\sum }_{i\in A_t}{r}_i\left(t\right)$) represents the accumulated reward obtained under the offline optimal policy ${\pi }^{★}$ (under the algorithm $\pi$) at round t, respectively. The analysis of the offline optimal policy (as oracle of the target bandit problem) is given in Section 4.1.

Remark 1.  

The fundamental distinction between the regret defined in Equation (2) and the regret metric in a non-sleeping setting is that the set of available arms changes dynamically, where the offline optimal policy ${\pi }^{★}$ may not be optimal in each round. In this paper, we focus on policy regret [22,32]. Given the inherent hardness of combinatorial sleeping bandits and restless bandit problems, the design of online algorithms with sub-linear regret guarantees remains an open problem.

Remark 2.  

Notably, even with known Markovian dynamics, solving the S-RMAB  problem is  P-SPACE hard  [35], rendering the search for an optimal policy ${\pi }^{★}$ intractable. Thus, we aim to construct an offline policy that satisfies asymptotic optimality (see Theorem 1), acting as an oracle for the online sleeping restless bandit problem and guiding the design of online algorithms.

4. Online Learning Algorithm Design

In this section, we first analyze the complexity of the sleeping RMAB problem, design a sleeping index-based oracle by relaxing some constraints, and establish the asymptotic optimality of the sleeping index policy. Next, we introduce a Minimum Exploration Guarantee mechanism to construct estimators with probabilistic upper bounds for unknown parameters and propose an explore-then-commit based algorithm under the online sleeping index policy.

4.1. Oracle: Optimal Sleeping Index-Based Policy

To construct the oracle policy, following the approach of classical RMAB methods [8], we begin by relaxing the budget constraint as $\underset{T\to \infty }{lim}sup{\displaystyle \frac{1}{T}}{\mathbb{E}}_{\pi }\left[{\sum }_{i\in I_t}{a}_i\left(t\right)\right]\le M$. At this point, each arm can be regarded as an independent Constrained Markov Decision Process (CMDP). Then, we employ the fluid analysis to reformulate the problem and convert it into a solvable Linear Programming (LP) problem.

The core idea of fluid analysis is to construct a statistical variable whose steady-state distribution allows the original stochastic model to consider only the system’s mean drifts. For a given policy $\pi$, we define a time-dependent statistical variable $X_{s_i,a_i}^{\pi }\left(t\right)$ to describe the number of times arm i is in $(s_i,a_i)\in \mathcal{S}\times \mathcal{A}$ at round t. Under this modeling, the objective function of the relaxed S-RMAB problem can be defined as:

$$J_R^{\pi }:=\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}{\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=1}^T}\sum _{i\in I_t}\sum _{(s_i,a_i)}\overline{r}(s_i,a_i)X_{s_i,a_i}^{\pi }\left(t\right)\right],$$

(3)

where $\overline{r}(s_i,a_i)$ is the mean reward of arm i in state $s_i$ and action $a_i$. Note that the expected time average budget constraint relaxes the feasible region of the S-RMAB; hence, $J_R^{{\pi }^{★}}$ serves as an upper bound of $J^{{\pi }^{★}}={\sum }_t{J}_{A_t^{★}}$. Then, we aim to characterize the equilibrium state of the fluid formulation of the relaxed S-RMAB using a fixed probability. To ensure the existence of a stationary steady-state distribution in such a stochastic system, we adopt an assumption widely used in state-of-the-art studies [24,41].

Assumption 1.  

For $\forall i\in \left[K\right]$, the process describing the state of arm i is unichain, regardless of the policy employed.

We denote ${\tilde{x}}_{s_i,a_i}^{\pi }=\underset{T\to \infty }{lim}{\displaystyle \frac{{\mathbb{E}}_{\pi }\left[{\sum }_{t:i\in I_t}\mathbb{I}\{s_i\left(t\right)=s_i,a_i\left(t\right)=a_i\}\right]}{{\sum }_t\mathbb{I}\{i\in I_t\}}}$ as the stationary probability of all available rounds of arm i in state $s_i$ and action $a_i$. Since the availability probabilities of arms are independent, the steady-state probability of each arm i at any round can be expressed as $x_{s_i,a_i}^{\pi }=z_i{\tilde{x}}_{s_i,a_i}^{\pi }$. Furthermore, if the system achieves equilibrium, it implies that ${\sum }_{a_i\in \mathcal{A}}{\sum }_{s_i^{\prime }\in \mathcal{S}}P(s_i^{\prime },a_i,s_i)x_{s_i^{\prime },a_i}^{\pi }={\sum }_{a_i\in \mathcal{A}}{\sum }_{s_i^{\prime }\in \mathcal{S}}P(s_i,a_i,s_i^{\prime })x_{s_i,a_i}^{\pi }$.

Under Assumption 1 and the equilibrium condition, the relaxed S-RMAB can be equivalently reformulated as an $\mathrm{LP}(P_i,{\overline{r}}_i,z_i)$ problem [42]. The details of the LP can be found in Appendix B. We denote ${\tilde{x}}_{s_i,a_i}^{★}=argmax\mathrm{LP}(P_i,{\overline{r}}_i,z_i)$ as the optimal solution of LP. Thus, we can evaluate the value of each arm in different states $s_i$, based on the sleeping index $w\left(s_i\right)={\displaystyle \frac{{\tilde{x}}_{s_i,1}^{★}}{{\sum }_{a_i\in \mathcal{A}}{\tilde{x}}_{s_i,a_i}^{★}}}$. Accordingly, we formally define the sleeping index policy (SIP) below:

Definition 1 (Sleeping Index Policy (SIP)). 

Given availability probabilities $z_i$ and sleeping index $w\left(s_i\right),\forall i\in \left[K\right]$. At time slot $t\in \left[T\right]$, we activate the arms with top-$m_t:=min\left\{\right|I_t|,M\}$ sleeping indices in $I_t$.

We then show that SIP is asymptotically optimal within the same asymptotic regime as methods such as Whittle’s Index policy [8], ERC policy [25], and others [24,41]. We define a scaling factor $\alpha \in {\mathbb{R}}^{+}$, such that if there are $\alpha K$ arms, the budget per round is scaled proportionally to $\alpha M$. Under this asymptotic regime, the objective function of the relaxed S-RMAB is given by:

$$J^{\pi ,\alpha }=\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}{\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=1}^T}\sum _{i\in I_t}\sum _{(s_i,a_i)}\overline{r}(s_i,a_i){\displaystyle \frac{X_{s_i,a_i}^{\pi ,\alpha }\left(t\right)}{\alpha }}\right],$$

(4)

and we have the following theorem.

Theorem 1.  

The sleeping index policy ${\pi }^S$ is asymptotically optimal, i.e., ${lim}_{\alpha \to \infty }{J}^{{\pi }^S,\alpha }-J^{{\pi }^{★}}=0$.

Proof. 

The main idea of the proof is as follows. We first analyze the behavior of the fluid process $X_{s_i,a_i}^{\pi ,\alpha }\left(t\right)$ for any arm $i\in \left[K\right]$ under a given policy $\pi$, demonstrating that as the system grows large ($\alpha \to \infty$), it converges to a deterministic system $x_{s_i,a_i}^{\pi }\left(t\right)$ described by a set of ordinary differential equations (ODEs). Subsequently, we examine the steady-state probability distribution properties of this system. By establishing the tightness and uniform integrability of the probability distribution, we express the long-term reward represented by the fluid process $X_{s_i,a_i}^{\pi ,\alpha }\left(t\right)$ in terms of the optimal solution ${\pi }^{★}$.

To prove the asymptotic optimality of the sleeping index policy ${\pi }^S$, we first present the following necessary definition:

Definition 2 (Asymptotic optimality).

A policy π is asymptotically optimal if

$$\underset{\alpha \to \infty }{lim}{J}^{\pi ,\alpha }-J^{{\pi }^{★}}=0,$$

(5)

where the long-term reward $J^{\pi ,\alpha }$ can be given by:

$$J^{\pi ,\alpha }:=\underset{T\to \infty }{lim\; sup}{\displaystyle \frac{1}{T}}{\mathbb{E}}_{\pi ,\alpha }\left[{\int }_{t=1}^T\sum _{i\in I_t}\sum _{(s_i,a_i)\in \mathcal{S}\times \mathcal{A}}\overline{r}(s_i,a_i){\displaystyle \frac{X_{s_i,a_i}^{\pi ,\alpha }\left(t\right)}{\alpha }}dt\right],$$

(6)

Definition 3 (Density-dependent population process).

A sequence of Markov process $X^{\alpha }$ on ${\displaystyle \frac{1}{\alpha }}{\mathbb{N}}^d(d\ge 1)$ is called a density-dependent population process if there exists a finite number of transitions, say $\mathcal{L}\subset {\mathbb{N}}^d$, such that for each $l\in \mathcal{L}$, the rate of transition from $X^{\alpha }\to X^{\alpha }+{\displaystyle \frac{l}{\alpha }}$ is $\alpha c_l\left(X^{\alpha }\right)$, where $c_l(.)$ does not depend on α.

Next, we will show that the fluid process ${\displaystyle \frac{1}{\alpha }}{X}_{s_i,a_i}^{{\pi }^S,\alpha }$ belongs to the class of density-dependent population processes, as defined in Definition 3. The transition rates of the process ${\displaystyle \frac{1}{\alpha }}{X}_{s_i,a_i}^{{\pi }^S,\alpha }$ can be expressed as follows:

$$x\to x+{\displaystyle \frac{e_i^{s^{\prime }}}{\alpha }}-{\displaystyle \frac{e_i^s}{\alpha }}\mathrm{at}\mathrm{rate}\alpha {\displaystyle \sum _{a_i=0}^1}P(s_i,a_i,s_i^{\prime })x_{s_i,a_i}^{{\pi }^S},$$

(7)

where $e_i^s\in {\mathbb{R}}^{\left|\mathcal{S}\right|\times 1}$ is a vector composed of all zeros except for position s which is one and $x_{s_i,1}^{{\pi }^S}=min\{{(M-{\sum }_{k,j\in A_i^{*}\left(s\right)}{x}_{j_k,1}^{{\pi }^S})}^{+},x_{s_i,a_i}^{{\pi }^S}\}$. $A_i^{*}\left(s\right)$ is defined as the set of all combinations $(k,j),k\in \left[K\right],j\in \mathcal{S}$, such that under the sleeping index policy ${\pi }^S$, the index of the class-k arms in state j is larger than the index of the class-i arms in state s. Based on Equation (7), we have $x\to x+{\displaystyle \frac{l}{\alpha }},\forall l\in \mathcal{L}$, where $\mathcal{L}$ is a finite number of vector in ${\mathbb{N}}^{K\left|\mathcal{S}\right|}$. Thus, the fluid process ${\displaystyle \frac{1}{\alpha }}{X}_{s_i,a_i}^{{\pi }^S,\alpha }$ is a density dependent population process, and can be modeled by the continuous bounded function $c_l(.)$. We denote $p^{\alpha ,{\pi }^S}$ as the stationary invariant distribution under the sleeping index policy ${\pi }^S$. Moreover, the ODEs of the system are constructed as below:

$${\displaystyle \frac{d{X}_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)}{dt}}=\sum _{l}l{c}_l\left(X_{s_i,a_i}^{{\pi }^s,\alpha }\left(t\right)\right)\mathrm{with}{X}_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)=x\in \mathbb{N}.$$

(8)

Let ${\Psi }_t\left(x\right)$ be the value of the Equation (8) at time t. Then, the difference between the expected value of all trajectories of the ODEs system solution and the expectation of the fluid limit process $X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)$ with $\alpha \to \infty$ can be given by:

$$\begin{array}{cc}\hfill & \underset{\alpha \to \infty }{lim}|{\int }_x\mathbb{E}\left[c_l\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)\right)\right]p^{\alpha ,{\pi }^S}dx-{\int }_x{c}_l\left({\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)\right)p^{\alpha ,{\pi }^S}dx|\hfill \\ \hfill & \stackrel{\left(a\right)}{=}\underset{\alpha \to \infty }{lim}\left|{\int }_x\left(c_l\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)\right)-c_l\left({\Psi }_t(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)\right)p^{\alpha ,{\pi }^S}dx\right|\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }\underset{\alpha \to \infty }{lim}{\int }_x\left(2\underset{x}{sup}\parallel c_l\left(x\right)\parallel \mathrm{Pr}\{\parallel X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)-{\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)\parallel >ϵ\}+\delta \right)p^{\alpha ,{\pi }^S}dx\hfill \end{array}$$

(9)

where $ϵ,\delta >0$. Step (a) holds since $p^{\alpha ,{\pi }^S}$ is the stationary invariant distribution. Step (b) is derived from the fact that $c_l(.)$ is a continuous bounded function. For any variables a and b, there exists $ϵ,\delta >0$ such that $\parallel a-b\parallel \le ϵ,\parallel c_l\left(a\right)-c_l\left(b\right)\parallel \le \delta$. Thus, we have

$$\begin{array}{cc}\hfill & \mathbb{E}\left[|c_l\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)\right)-c_l({\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)|\right]\hfill \\ \hfill & =\mathbb{E}\left[|c_l\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)\right)-c_l({\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)|\right]\mathbb{I}\{\parallel X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)-{\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)\le ϵ\parallel \}\hfill \\ \hfill & +\mathbb{E}\left[|c_l\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)\right)-c_l({\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)|\right]\mathbb{I}\{\parallel X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)-{\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)>ϵ\parallel \}\hfill \\ \hfill & =\delta +2\underset{x}{sup}\parallel c_l\left(x\right)\parallel \mathrm{Pr}\{\parallel X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)-{\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)>ϵ\parallel \}\hfill \end{array}$$

(10)

Lemma 1  

([43]).  Assume that for all compact $E\subset {\mathbb{R}}^d,{\sum }_l\left|l\right|{sup}_x{c}_l\left(x\right)<\infty$, and $C\left(x\right)={\sum }_{l\in \mathcal{L}}l{c}_l\left(x\right)$ is Lipschitz on E. If ${lim}_{\alpha \to \infty }{X}^{\alpha }\left(0\right)=x_0$ in probability, then for all $t>0$:

$$\underset{\alpha \to \infty }{lim}\underset{s\le t}{sup}|X^{\alpha }\left(s\right)-x\left(s\right)|=0,inprobability.$$

(11)

According to Lemma 1, we infer $\mathrm{Pr}\{\parallel X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)-{\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)>ϵ\parallel \}=0$. Together with Equation (9), this leads to ${lim}_{\alpha \to \infty }|{\int }_x\mathbb{E}\left[c_l\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)\right)\right]p^{\alpha ,{\pi }^S}dx-{\int }_x{c}_l\left({\Psi }_t\left(X_{s_i,a_i}^{{\pi }^S,\alpha }\left(0\right)\right)\right)p^{\alpha ,{\pi }^S}dx|$$<\delta$. Hence, when $\alpha \to \infty$, the fluid process $X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)$ converges to the deterministic system described by the ODEs. Additionally, there exists a stationary distribution $p^{\alpha ,{\pi }^S}$, where any limit point of $p^{\alpha ,{\pi }^S}$ is composed of solutions to the ODEs.

Lemma 2 

([44]).  If $C(.)$ has a unique stationary point $x^{*}$ to which all trajectories converge, then the stationary measures $p^{\alpha ,\pi }$ concentrate around $x^{*}$ as $\alpha \to \infty$:

$$\underset{\alpha \to \infty }{lim}{p}^{\alpha ,\pi }\stackrel{weak}{⟶}{\delta }_{x^{*}}where{\delta }_{x^{*}}istheDiracmeasurein{x}^{*}.$$

(12)

Indeed, according to Lemma 2, the stationary distribution $p^{\alpha ,{\pi }^S}$ can be characterized as a set of Dirac-based static solutions under the sleeping index policy ${\pi }^S$. Prior to this, we examine the tightness and uniform integrability of the stationary distribution $p^{\alpha ,{\pi }^S}$, allowing us to further deduce the closed-form expression of the long-term reward with respect to the optimal solution ${\pi }^{★}$.

Since at any time t, arm $i\in \left[K\right]$ is not always available, the state space of the process $X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)$ satisfies $\left|\mathcal{X}\right|<\infty$. In this case, the state space $\mathcal{X}$ is compact. Hence, for any $ϵ>0$, we can simply take $K_{ϵ}=\mathcal{X}$, and $\mathrm{Pr}\{\mathcal{X}\setminus K_{ϵ}\}=\mathrm{Pr}\{\varnothing \}=0<ϵ$. Then, $p^{\alpha ,{\pi }^S}$ is tightness. Similarly, any function $f\left(x\right)$ defined on the state space $\mathcal{X}$. Let $M_0\ge \parallel f\parallel ={sup}_{x\in \mathcal{X}}\left|f\left(x\right)\right|$, we have ${\int }_{\mathcal{X}}\left|f\left(x\right)\right|p^{\alpha ,{\pi }^S}dx<\infty$ and ${\int }_{\{x\in \mathcal{X}:|f\left(x\right)|>M_0\}}\left|f\left(x\right)\right|p^{\alpha ,{\pi }^S}=0$. Thus, $p^{\alpha ,{\pi }^S}$ is uniform integrability. Based on this conclusion and Lemma 2, the long-term reward under the sleeping index policy can be represented as:

$$\begin{array}{cc}\hfill J^{{\pi }^S,\alpha }& =\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}{\mathbb{E}}_{{\pi }^S,\alpha }\left[{\int }_{t=1}^T\sum _{i\in I_t}\sum _{(s_i,a_i)}\overline{r}(s_i,a_i){\displaystyle \frac{X_{s_i,a_i}^{{\pi }^S,\alpha }\left(t\right)}{\alpha }}dt\right]\hfill \\ \hfill & \stackrel{\left(a\right)}{=}\sum _{i\in \left[K\right]}\sum _{(s_i,a_i)}\underset{\alpha \to \infty }{lim}\sum _{x\in \mathcal{X}}{p}^{\alpha ,{\pi }^S}\left(x\right)z_i\overline{r}(s_i,a_i){\displaystyle \frac{x_{s_i,a_i}^{{\pi }^S,\alpha }}{\alpha }}\hfill \\ \hfill & \stackrel{\left(b\right)}{=}\sum _{i\in \left[K\right]}\sum _{(s_i,a_i)}\sum _{x\in \mathcal{X}}\underset{\alpha \to \infty }{lim}{p}^{\alpha ,{\pi }^S}\left(x\right)z_i\overline{r}(s_i,a_i){\displaystyle \frac{x_{s_i,a_i}^{{\pi }^S,\alpha }}{\alpha }}\hfill \\ \hfill & \stackrel{\left(c\right)}{=}\sum _{i\in \left[K\right]}\sum _{s_i\in \mathcal{S}}{z}_i\left(\overline{r}(s_i,0){\tilde{x}}_{s_i,0}^{★}+\overline{r}(s_i,1){\tilde{x}}_{s_i,1}^{★}\right)\ge J^{{\pi }^{★}}.\hfill \end{array}$$

(13)

where ${\tilde{x}}_{s_i,a_i}^{★}=argmax\mathrm{LP}(P_i,{\overline{r}}_i,z_i)$, as shown in Equation (A1). Step (a) is satisfied because the stationary distribution $p^{\alpha ,{\pi }^S}$ and the arm available distribution are tight, which follows from the ergodic theorem. Step (b) is a consequence of uniform integrability. According to Lemma 2, step (c) shows that the stationary distribution $p^{\alpha ,{\pi }^S}$ weakly converges to the static probability ${\tilde{x}}_{s_i,a_i}^{★}$ defined on the Dirac measure. Furthermore, based on the definition of ${\pi }^{★}$, we have $J^{{\pi }^S,\alpha }\le J^{★}$, and thus ${lim}_{\alpha \to \infty }{J}^{{\pi }^S,\alpha }-J^{{\pi }^{★}}=0$.    □

Remark 3.  

As the classical RMAB methods [8,24,25,41] fail to guarantee asymptotic optimality for the  S-RMAB  problem, as shown in Section 6. The sleeping setting impacts the asymptotic properties of SIP by dynamically changing the decision space due to varying arm availability. When $|I_t|<M$, selecting all arms is optimal. For $|I_t|\ge M$, the independence of the available arms and the tightness of its stationary distribution are critical, enabling fluid analysis using the ergodic theorem.

4.2. Online Learning Algorithm: OSILA

In this subsection, we consider the sleeping RMAB problem in online scenarios and adopt the explore-then-commit framework, which separates the exploration and exploitation phases to improve learning efficiency. Following this framework, we begin by discussing the estimation of unknown system parameters in an online setting. Subsequently, we detail the design of execution strategies based on these estimates. Finally, we introduce the Online Sleeping Index-aware Learning Algorithm (OSILA).

4.2.1. Minimum Exploration Guarantee Mechanism

Under the sleeping framework, the set of available arms for the learner in each round $t\in \left[T\right]$ follows a stochastic process. To achieve a probabilistic upper-bound estimation method within the stochastic exploration rounds, we propose the MEG mechanism as outlined in Algorithm 1. The main concept of this mechanism is to guarantee a minimum selection count for each arm i, resulting in the selection period $T_i$ adhering to a negative binomial distribution. The minimum selection threshold $G\left(T\right)$ is a parameter dependent on T. As T increases, we analyze the characteristic function of the exploration phase $T_1$ modeled as the extreme value of all $T_i$ (i.e., $T_1={max}_{i\in \left[K\right]}{T}_i$), using the Laplace transform method and show that it converges to a Gumbel distribution. Combined with Hoeffding’s inequality, this yields a probabilistic upper bound for the estimated availability probability of each arm.

The intuition behind MEG is that classical forced exploration in non-sleeping RMAB fixes the exploration horizon in advance, since every arm can always be selected when needed. This strategy is no longer reliable in OS-RMAB, because low-availability arms may not appear often enough within a predetermined number of rounds. MEG therefore fixes the exploration target rather than the exploration time: it keeps tracking the number of actual pulls of each arm and stops only when every arm has been selected at least $G\left(T\right)$ times. This design directly guarantees enough samples for constructing the confidence intervals of $(\widehat{P},\widehat{r},\widehat{z})$, which is essential for controlling the exploitation regret. At the same time, it turns the exploration length $T_1$ into a random variable determined by the slowest arm to reach the target. This is precisely why the distributional analysis of $T_1$ is needed and why $G\left(T\right)$ becomes the key parameter balancing exploration regret and exploitation regret.

Algorithm 1 Minimum Exploration Guarantee Mechanism

1: Input: create a counter $C_i\left(t\right)\in \mathbb{N}$ for each arm $i\in \left[K\right]$ to keep track of the number of times it has pulled. Set $C_i\left(1\right)=0$ for all i. 2: At each round $t=1,2,\dots ,T$: 3: Identify the available set $I_t$. 4: From $I_t$, select the $min\left\{\right|I_t|,M\}$ arms with the lowest $C_i\left(t\right)$ values. If there are ties, we select randomly. 5: Pull the selected arms and increment their counters $C_i(t+1)=C_i\left(t\right)+1,\forall i\in A_t$ 6: Until: All $C_i\left(t\right)$ counters reach $G\left(T\right)$.

Lemma 3.  

We estimate each arm’s availability probability ${\widehat{z}}_i$ with empirical mean ${\widehat{z}}_i={\displaystyle \frac{{\sum }_{t=1}^{T_1}\mathbb{I}\{i\in I_t\}}{T_1}}$. We have

$$\begin{array}{cc}\hfill & \mathit{Pr}\left\{\right|{\widehat{z}}_i-z_i|\ge {ϵ}_z\}\hfill \\ \hfill & ={\int }_0^{\infty }\mathit{Pr}\left\{\right|{\widehat{z}}_i-z_i|\ge z_i|T_i=t\}{f}_{T_1}\left(t\right)dt\hfill \\ \hfill & \le 2exp\left(-2\left({\displaystyle \frac{\sqrt{G\left(T\right)(1-z_i)}(2logK+\gamma )}{z_i\sqrt{2logK}}}+{\mu }^{\prime }\right){ϵ}_z^2\right),\hfill \end{array}$$

(14)

where ${ϵ}_z$ is the confidence radius, ${\mu }^{\prime }={\displaystyle \frac{G\left(T\right)}{z_i}}$, and $\gamma =0.5772$ is the Euler–Mascheroni constant.

Proof. 

Under the sleeping framework, the exploration phase $T_1$ is actually a random variables. Assume $T_1$ is distributed according to some distribution $f_{T_1}(.)$, and we have

$$\mathrm{Pr}\left\{\right|{\widehat{z}}_i-z_i|\ge {ϵ}_z\}={\int }_{t=1}^{\infty }\mathrm{Pr}\left\{\right|{\widehat{z}}_i-z_i|\ge {ϵ}_z|T_1=t\}{f}_{T_1}\left(t\right)dt,$$

(15)

and according to Hoffeding’s inequality, we have $\mathrm{Pr}\left\{\right|{\widehat{z}}_i-z_i|\ge {ϵ}_z|T_1=t\}\le 2exp(-2{ϵ}_z^{2}t)$. Then,

$$\mathrm{Pr}\left\{\right|{\widehat{z}}_i-z_i|\ge {ϵ}_z\}\le 2{\int }_{t=1}^{\infty }exp(-2{ϵ}_z^{2}t)f_{T_1}\left(t\right)dt.$$

(16)

Next, we analyze the characteristics of the probability density function $f_{T_1}\left(t\right)$. For each arm $i\in \left[T\right]$, the total number of rounds required for it to be pulled $G\left(T\right)$ times still follows a negative binomial distribution, i.e., $T_i$∼$\mathrm{NB}(G\left(T\right),z_i)$, and the probability mass function (PMF) can be given by:

$$\mathrm{Pr}\{T_i=t\}=\left({\displaystyle \genfrac{}{}{0pt}{}{t-1}{G\left(T\right)-1}}\right)z_i^{G\left(T\right)}{(1-z_i)}^{t-G\left(T\right)}\forall t\ge G\left(T\right).$$

(17)

Subsequently, we intend to employ the Laplace transform to analyze the characteristic function of the $T_1={max}_{i\in \left[K\right]}{T}_i$, which converges to the Gumbel distribution as T becomes large. In particular, the characteristic function of $T_i$ is defined as:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{T_i}\left(s\right)& =\mathbb{E}\left[e^{-s{T}_i}\right]={\displaystyle \sum _{t=G\left(T\right)}^{\infty }}{e}^{-st}\mathrm{Pr}\{T_i=t\}={\displaystyle \sum _{t=G\left(T\right)}^{\infty }}{e}^{-st}\left({\displaystyle \genfrac{}{}{0pt}{}{t-1}{G\left(T\right)-1}}\right)z_i^{G\left(T\right)}{(1-z_i)}^{t-G\left(T\right)}\hfill \\ \hfill & \stackrel{t=G\left(T\right)+k}{=}{z}_i^{G\left(T\right)}{\displaystyle \sum _{k=0}^{\infty }}{e}^{-s\left(G\right(T)+k)}\left({\displaystyle \genfrac{}{}{0pt}{}{G\left(T\right)+k-1}{G\left(T\right)-1}}\right){(1-z_i)}^k\hfill \\ \hfill & =z_i^{G\left(T\right)}{e}^{-sG\left(T\right)}{\displaystyle \sum _{k=0}^{\infty }}\left({\displaystyle \genfrac{}{}{0pt}{}{G\left(T\right)+k-1}{G\left(T\right)-1}}\right)e^{-sk}{(1-z_i)}^k\hfill \\ & =z_i^{G\left(T\right)}{e}^{-sG\left(T\right)}{\left(1-(1-z_i)e^{-s}\right)}^{G\left(T\right)}={\left({\displaystyle \frac{z_i{e}^{-s}}{1-(1-z_i)e^{-s}}}\right)}^{G\left(T\right)}.\hfill \end{array}$$

(18)

Moreover, given that $T_1={max}_{i\in \left[K\right]}{T}_i$ and $T_i$ follows the property of independent and identical distribution, the characteristic function of $T_1$ can be explicitly expressed as:

$${\mathcal{L}}_{T_1}\left(s\right)=\mathbb{E}\left[e^{-s{max}_{i\in \left[K\right]}{T}_i}\right]={\displaystyle \prod _{i=1}^K}\mathbb{E}\left[e^{-s{T}_i}\right]={\left({\displaystyle \frac{z_i{e}^{-s}}{1-(1-z_i)e^{-s}}}\right)}^{G\left(T\right)K}.$$

(19)

Based on the Taylor series approximation theory (as $s\to 0$), it follows that ${\mathcal{L}}_{T_1}\left(s\right)={\left({\displaystyle \frac{z_i(1-s+o\left(s^2\right))}{1-(1-z_i)(1-s+o\left(s^2\right))}}\right)}^{G\left(T\right)K}=exp{(-z_{i}s)}^{G\left(T\right)K}+o\left(s^2\right)$. Thus, with T growing larger, the random variable $T_1$ is normalized, i.e., ${\displaystyle \frac{T_1-\mu }{\beta }}$. Let $\mu ={\displaystyle \frac{G\left(T\right)}{z_i}}+{\displaystyle \frac{\sqrt{G\left(T\right)(1-z_i)}}{z_i}}\sqrt{2logK},$$\beta ={\displaystyle \frac{\sqrt{G\left(T\right)(1-z_i)}}{z_i\sqrt{2logK}}}$, then ${\mathcal{L}}_{T_1}\left(s\right)=e^{-e^{-(s-\mu )/\beta }}$, which is the characteristic function of the Gumbel distribution. Substituting this result into Equation (16), we deduce

$$\begin{array}{c}\hfill \mathrm{Pr}\left\{\right|{\widehat{z}}_i-z_i|\ge {ϵ}_z\}\le 2exp\left(-2\left({\displaystyle \frac{\sqrt{G\left(T\right)(1-z_i)}(2logK+\gamma )}{z_i\sqrt{2logK}}}+{\displaystyle \frac{G\left(T\right)}{z_i}}\right){ϵ}_z^2\right).\end{array}$$

(20)

   □

Next, we define $N_{T_1}(s_i,a_i,s_i^{\prime })$ as the number of transitions from state $s_i$ to state $s_i^{\prime }$ under action $a_i$ during the exploration phase $T_1$. We estimate the unknown state transition probability with the empirical mean $\widehat{P}(s_i,a_i,s_i^{\prime })={\displaystyle \frac{N_{T_1}(s_i,a_i,s_i^{\prime })}{max\{1,{\sum }_{s_i^{\prime }\in \mathcal{S}}{N}_{T_1}(s_i,a_i,s_i^{\prime })\}}}$. Similarly, the stochastic reward can be estimated as $\widehat{r}(s_i,a_i)={\displaystyle \frac{{\sum }_{t=1}^{T_1}r(s_i,a_i)\mathbb{I}\{s_i\left(t\right)=s_i,a_i\left(t\right)=a_i\}}{max\{1,{\sum }_{s_i^{\prime }\in \mathcal{S}}{N}_{T_1}(s_i,a_i,s_i^{\prime })\}}}$. We aim to build sufficiently accurate estimates. Let events ${\mathcal{E}}_z:=\left\{\right|{\widehat{z}}_i-z_i|\le {ϵ}_z|\exists i\}$, ${\mathcal{E}}_P:=\left\{\right|\widehat{P}(s_i,a_i,s_i^{\prime })-P(s_i,a_i,s_i^{\prime })|\le {ϵ}_{P,r}|\exists i,(s,a)\}$, and ${\mathcal{E}}_r:=\left\{\right|\widehat{r}(s_i,a_i)-\overline{r}(s_i,a_i)|\le {ϵ}_{P,r}|\exists i,(s,a)\}$ represent the estimates that lies within the confidence intervals, while ${ϵ}_z$ and ${ϵ}_{P,r}$ denote the corresponding confidence radius. When the three events occur concurrently, we obtain a set of plausible events $\mathcal{B}:=\{B_{i\in \left[K\right]}={\mathcal{E}}_z\cap {\mathcal{E}}_P\cap {\mathcal{E}}_r\}$.

4.2.2. Modified LP-Based Exploitation Mechanism

In the online setting, since the learner does not have information on the available probabilities, transition probabilities, and the reward of each arm before making decisions, the sleeping index policy under the oracle cannot be directly derived by solving the LP. We construct the auxiliary variable $\tilde{y}(s_i,a_i,s_i^{\prime })=P(s_i,a_i,s_i^{\prime })z_i{\tilde{x}}_{s_i,a_i}^{\pi }$ to transform the online sleeping restless bandits problem into a Modified Linear Programming (MLP) problem with respect to $\tilde{y}$. The description of $\mathrm{MLP}({\widehat{P}}_i,{\widehat{r}}_i+{ϵ}_{P,r},{\widehat{z}}_i)$ is provided in Appendix B. Then, we define the optimal solution of MLP as ${\tilde{y}}^{★}(s_i,a_i,s_i^{\prime })$. Similarly to the sleeping index policy, under the online setting, the probability of arm i in state $s_i$ towards maximizing the total rewards can be defined as $\tilde{\omega }\left(s_i\right):={\displaystyle \frac{{\sum }_{s_i^{\prime }\in \mathcal{S}}{\tilde{y}}^{★}(s_i,1,s_i^{\prime })}{{\sum }_{(a_i,s_i^{\prime })\in \mathcal{A}\times \mathcal{S}}{\tilde{y}}^{★}(s_i,a_i,s_i^{\prime })}}$. Based on the online sleeping index, we choose the top-$m_t$ arms with index $\tilde{\omega }\left(s_i\right)$ in the availability set $I_t$ during the exploitation phase $\left(T_2\right)$.

The MLP remains polynomial in problem size. Its variable size scales as ${K\left|\mathcal{S}\right|}^2\left|\mathcal{A}\right|$, and the associated flow balance, budget, occupancy, and confidence constraints also grow polynomially with K, $\left|\mathcal{S}\right|$, and $\left|\mathcal{A}\right|$. Hence, the MLP can be solved in polynomial time by standard interior-point methods. In practice, the constraint matrix is sparse and largely arm-separable except for the global budget coupling; thus, commercial LP solvers can exploit this structure effectively. For the problem sizes considered in our experiments, the LP step is not a computational bottleneck, although its cost may become more noticeable in very large-scale instances.

4.2.3. Main Idea of OSILA

By integrating the MEG mechanism with the Modified LP-based Exploitation mechanism, we develop the Online Sleeping Index-aware Learning Algorithm (OSILA) to solve S-RMAB under the online setting.

In general, the parameter $G\left(T\right)$ in Algorithm 2 influences its performance and runtime. When $G\left(T\right)$ is large enough, the execution time of the MEG mechanism becomes longer, resulting in higher regret during this phase. Conversely, sufficient exploration during the execution period ensures that the estimate tuple lies within a smaller confidence interval with high probability, thereby minimizing regret in this segment. $G\left(T\right)$ serves to balance the exploration and exploitation trade-offs in Algorithm 2.

Before presenting Algorithm 2, we clarify its inputs. The algorithm takes the time horizon T and the exploration threshold $G\left(T\right)$ as external inputs. The budget M and the spaces $\mathcal{S}$ and $\mathcal{A}$ are assumed to be known from the problem setup. In each round, the learner observes the currently available set $I_t$ and the current arm states. In contrast, the transition kernels $P_i$, expected rewards ${\overline{r}}_i$, and availability probabilities $z_i$ are unknown model parameters and are estimated online through the MEG exploration phase before constructing the plausible set $\mathcal{B}$ and solving the modified LP.

Algorithm 2 Online Sleeping Index-aware Learning Algorithm (OSILA)

1: Input: Time horizon T and exploration threshold $G\left(T\right)$; the budget M and the spaces $\mathcal{S},\mathcal{A}$ are given by the problem setup. 2: Execute the Minimum Exploration Guarantee Mechanism, with the termination time defined as $T_1$. 3: By calculating ${\sum }_{t=1}^{T_1}\mathbb{I}\{i\in I_t\}/T_1$, along with $\widehat{P}(s_i,a_i,s_i^{\prime })$ and $\widehat{r}(s_i,a_i)$, we obtain the estimate tuple $({\widehat{z}}_i,{\widehat{P}}_i,{\widehat{r}}_i)$, and construct a set of plausible events $\mathcal{B}$. 4: for  $t=T_1+1$  to  T  do 5:    Compute the online sleeping index $\tilde{\omega }\left(s_i\right)$ by solving $\mathrm{MLP}\left(\mathcal{B}\right)$. 6:    Pull arms with top-$m_t$ online sleeping indices in $I_t$. 7: end for

5. Performance Analysis

In this section, we present our theoretical results towards the proposed online algorithm OSILA, along with discussions and a proof sketch.

5.1. Main Results and Discussions

Theorem 2.  

Under Assumptions 1, let $z_{min}={min}_{i\in \left[K\right]}{z}_i$, $z_{max}={max}_{i\in \left[K\right]}{z}_i$ and take $G\left(T\right)=T^{2/3}$, the regret of OSILA satisfies:

$$\begin{array}{c}\hfill {\mathit{Reg}}_T\le \mathcal{O}\left(\left({\displaystyle \frac{1}{z_{min}}}+{\displaystyle \frac{\sqrt{2}}{2}}\sqrt{z_{max}}{\left|\mathcal{S}\right|}^{3/2}{\left|\mathcal{A}\right|}^{1/2}\sqrt{logT}\right)K{T}^{2/3}\right)\end{array}$$

(21)

Notably, Theorem 2 shows that the proposed algorithm OSILA is the first to achieve the $\tilde{\mathcal{O}}\left(T^{2/3}\right)$ sub-linear policy regret bound in polynomial form with respect to $\left|\mathcal{S}\right|,\left|\mathcal{A}\right|$ and K for the online sleeping restless bandits problem. The main factors that enable Algorithm 2 to achieve $\tilde{\mathcal{O}}\left(T^{2/3}\right)$ regret are twofold. First, during the exploration phase, we design a novel MEG mechanism, which provides a probabilistic upper bound for the estimate in the form of $G\left(T\right)$ and ensures that the accumulated regret during exploration remains sub-linear and related to $G\left(T\right)$. Second, by solving $\mathrm{MLP}\left(\mathcal{B}\right)$, we construct the online index policy, which, after decoupling the regret, yields a polynomial bound in terms of $G\left(T\right)$.

Compared to other online restless bandit algorithms [15,25,38,39] with the same explore-then-commit framework and achieve $\tilde{\mathcal{O}}\left(\sqrt{T}\right)$ regret, our algorithm requires fewer rounds in the exploration phase to obtain accurate estimates of the unknown parameters when arm sleeping is not considered. Thus, the algorithm can achieve better regret performance guarantees. In contrast to these works, OSILA can deal with the dynamics of the available arms set in each round and make accurate estimates. Furthermore, due to the randomness of exploration rounds, the decoupling method for non-sleeping restless bandits is not directly applicable to analyze the regret bound in the exploitation phase. We thus adopted a different method to address this challenge.

Remark 4.  

Achieving $\tilde{\mathcal{O}}\left(T^{2/3}\right)$ regret is non-trivial for online sleeping RMAB due to the complex coupling between restless dynamics and stochastic availability, which necessitates a careful decoupling for effective learning. We note that similar $\tilde{\mathcal{O}}\left(T^{2/3}\right)$ bounds are standard in other challenging non-stationary settings, such as non-stationary stochastic bandits [45,46] and reinforcement learning with changing environments [47], where the dynamic nature of the problem makes the optimal $\tilde{\mathcal{O}}\left(\sqrt{T}\right)$ regret unattainable.

Discuss 1. The independent availability assumption is mainly adopted to obtain a tractable oracle and a rigorous regret analysis. Technically, it enables the factorized occupancy characterization used in the LP formulation and supports the finite-time analysis of the MEG exploration phase. If arm availabilities are correlated due to shared communication conditions, interference, or common energy patterns, the MEG mechanism still terminates in finite time as long as each arm has a positive marginal availability $z_i>0$, although the random exploration length $T_1$ may become larger, especially under negative correlations. In that case, the LP based only on marginal probabilities $z_i$ should be interpreted as an approximation rather than an exact oracle characterization, while the resulting online sleeping index may still serve as a practically useful heuristic. A rigorous treatment of correlated availability would require modeling joint availability statistics or a latent common environment state, which is beyond the scope of the current paper.

Discuss 2. Regarding non-stationary environments, the current theoretical guarantees are established for stationary transition kernels and availability probabilities. Since OSILA follows an explore-then-commit structure, if these dynamics drift significantly after the exploration phase, the learned indices may gradually become sub-optimal. A practical extension is to combine OSILA with sliding-window estimation or epoch-based restart mechanisms, so that the MEG exploration phase is periodically re-invoked to forget outdated information and refresh the estimates of $(P,r,z)$. Developing dynamic-regret guarantees under such non-stationary sleeping RMAB settings is an interesting direction for future work.

5.2. Proof Sketch of Theorem 2

In this subsection, we outline the proof sketch of the main result. The regret of the OSILA is initially decomposed into exploration phase regret and exploitation phase regret, as shown below:

$$\begin{array}{cc}\hfill & {\mathrm{Reg}}_T=\underset{{\mathrm{Reg}}_{T_1}}{\underbrace{{\mathbb{E}}_{\pi }[\sum _{t=1}^{T_1}{J}_{A_t^{★}}-J_{A_t}]}}+\underset{{\mathrm{Reg}}_{T_2}}{\underbrace{{\mathbb{E}}_{\pi }[\sum _{t=T_1+1}^T{J}_{A_t^{★}}-J_{A_t}]}}\hfill \end{array}$$

(22)

Bounding ${\mathrm{Reg}}_{T_1}$. The regret ${\mathrm{Reg}}_{T_1}$, induced by the MEG mechanism in the exploration phase $T_1$, can be simply bounded by the following Lemma.

Lemma 4.  

Let $z_{min}={min}_{i\in \left[K\right]}{z}_i$, the regret in the exploration phase can be bounded as

$${\mathit{Reg}}_{T_1}\le M\mathbb{E}\left[T_1\right]\le M\left({\displaystyle \frac{KG\left(T\right)}{M}}\times {\displaystyle \frac{1}{z_{min}}}\right).$$

(23)

Proof. 

Let $z_{min}={min}_{i\in \left[K\right]}{z}_i$, the regret of ${\mathrm{Reg}}_{T_1}$ is given by:

$$\begin{array}{cc}\hfill {\mathrm{Reg}}_{T_1}& \stackrel{\left(a\right)}{\le }M\mathbb{E}\left[T_1\right]\hfill \\ \hfill & \stackrel{\left(b\right)}{=}M\left({\displaystyle \frac{G\left(T\right)+\sqrt{2logKG\left(T\right)(1-z_i)}}{z_i}}+\gamma {\displaystyle \frac{\sqrt{G\left(T\right)(1-z_i)}}{z_i\sqrt{2logK}}}\right)\stackrel{\left(c\right)}{\le }\mathcal{O}\left({\displaystyle \frac{G\left(T\right)K}{z_{min}}}\right)\hfill \end{array}$$

(24)

The first inequality holds because at most $m_t=min\left\{\right|I_t|,M\}$ arms are selected in each round t and the reward is upper bounded by one. Thus, the upper bound of ${\mathrm{Reg}}_{T_1}$ depends on the product of the expectation of $T_1$ and the number of arms selected, i.e., ${\mathrm{Reg}}_{T_1}\le {\sum }_{t_1\in \left[T_1\right]}\mathbb{E}\left[m_t\right](1-0)\le M\mathbb{E}\left[T_1\right]$. Under the MEG mechanism, Lemma 3 proves that the random variable $T_1$ converges to the Gumbel distribution, which validates equation (b). By taking the leading term and the smallest selection probability, we derive $\mathbb{E}\left[T_1\right]\le \mathcal{O}\left({\displaystyle \frac{KG\left(T\right)}{z_{min}M}}\right)$.    □

Lemma 4 holds based on two key facts. First, the maximum reward value for each arm is one, and the learner can select at most $m_t=min\left\{\right|I_t|,M\}$ arms per round. Another influencing factor is the expected number of rounds in the exploration phase. As shown in Lemma 3, $T_1$ converges to the Gumbel distribution under the MEG mechanism. Thus, regret ${\mathrm{Reg}}_{T_1}$ is of the form $\tilde{\mathcal{O}}\left(G\left(T\right)\right)$.

Bounding ${\mathrm{Reg}}_{T_2}$. Next, we bound the regret incurred during the exploitation phase by dividing it into two components: the sum of regret from good event occurrences and that from bad event occurrences.

$${\mathrm{Reg}}_{T_2}:={\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]+{\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\notin \mathcal{B}\}\right].$$

(25)

The following lemma provides a bound for the regret under the occurrence of all failure events.

Lemma 5.  

When ${\eta }_1,{\eta }_2\in (0,1)$, the regret conditioned on the failure event is given by

$${\mathit{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\notin \mathcal{B}\}\right]\le {\displaystyle \frac{M{T}_2({\eta }_1+{\eta }_2)}{G\left(T\right)}}$$

(26)

when ${ϵ}_z=\sqrt{{\displaystyle \frac{log\left(\frac{2K}{{\eta }_1}G\left(T\right)\right)}{2(C_1+C_2)}}}$, ${ϵ}_{P,r}=\sqrt{{\displaystyle \frac{log\left(\frac{4K\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{{\eta }_2}G\left(T\right)\right)}{2max\{1,N_{T_1}(s_i,a_i)\}}}}$, $C_1={\displaystyle \frac{G\left(T\right)+\sqrt{2logKG\left(T\right)(1-z_i)}}{z_i}}$, and $C_2={\displaystyle \frac{\gamma \sqrt{G\left(T\right)(1-z_i)}}{z_i\sqrt{2logK}}}$.

Proof. 

Based on the definition of set $\mathcal{B}$, the set comprising all failure event is $\{{\mathcal{E}}_z^c\cup {\mathcal{E}}_P^c\cup {\mathcal{E}}_r^c\}$. Here, event ${\mathcal{E}}_z^c$ is the complement of event ${\mathcal{E}}_z$, that is, ${\mathcal{E}}_z^c:=\left\{\right|{\widehat{z}}_i-z_i|>{ϵ}_z|\exists i\}$. Similarly, this holds for the other two events. According to Chernoff–Hoeffding inequality [48], we have

$$\begin{array}{cc}\hfill & \mathrm{Pr}\left\{\right|{\widehat{z}}_i-z_i|>\sqrt{{\displaystyle \frac{log\left(\frac{2K}{{\eta }_1}G\left(T\right)\right)}{2(C_1+C_2)}}}\}\le {\displaystyle \frac{{\eta }_1}{KG\left(T\right)}},\hfill \\ \hfill & \mathrm{Pr}\left\{\right|{\widehat{r}}_i-{\overline{r}}_i|>\sqrt{{\displaystyle \frac{log\left(\frac{4K\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{{\eta }_2}G\left(T\right)\right)}{2max\{1,N_{T_1}(s_i,a_i)\}}}}\}\le {\displaystyle \frac{{\eta }_2}{2K\left|\mathcal{S}\right|\left|\mathcal{A}\right|G\left(T\right)}}\hfill \\ \hfill & \mathrm{Pr}\left\{\right|\widehat{P}(s_i,a_i,s_i^{\prime })-P(s_i,a_i,s_i^{\prime })|>\sqrt{{\displaystyle \frac{log\left(\frac{4K\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{{\eta }_2}G\left(T\right)\right)}{2max\{1,N_{T_1}(s_i,a_i)\}}}}\}\le {\displaystyle \frac{{\eta }_2}{2K\left|\mathcal{S}\right|\left|\mathcal{A}\right|G\left(T\right)}},\hfill \end{array}$$

(27)

where $C_1={\displaystyle \frac{G\left(T\right)+\sqrt{2logKG\left(T\right)(1-z_i)}}{z_i}}$, and $C_2={\displaystyle \frac{\gamma \sqrt{G\left(T\right)(1-z_i)}}{z_i\sqrt{2logK}}}$. For any $i\in \left[K\right]$, $B_i\notin \mathcal{B}={\mathcal{E}}_z^c\cup {\mathcal{E}}_P^c\cup {\mathcal{E}}_z^c$. By summing over all arms, states, and actions jointly, we can get

$$\begin{array}{cc}\hfill \mathrm{Pr}\{B_i\notin \mathcal{B}\}\le & \sum _{i\in \left[K\right]}\mathrm{Pr}\left\{\right|{\widehat{z}}_i-z_i|>{ϵ}_z\}+\sum _{i\in \left[K\right]}\sum _{s_i\in \mathcal{S}}\sum _{a_i\in \mathcal{A}}\mathrm{Pr}\left\{\right|\widehat{P}(s_i,a_i,s_i^{\prime })-P(s_i,a_i,s_i^{\prime })|>{ϵ}_{P,r}\}\hfill \\ \hfill & +\sum _{i\in \left[K\right]}\sum _{s_i\in \mathcal{S}}\sum _{a_i\in \mathcal{A}}\mathrm{Pr}\left\{\right|{\widehat{r}}_i-{\overline{r}}_i|>{ϵ}_{P,r}\}\hfill \\ \hfill & \le 2Kexp(-2(C_1+C_2){ϵ}_z^2)+4K\left|\mathcal{S}\right|\left|\mathcal{A}\right|exp(-2{ϵ}_{P,r}^2{N}_{T_1}(s_i,a_i)).\hfill \end{array}$$

(28)

Thus, by setting ${ϵ}_z=\sqrt{{\displaystyle \frac{log\left(\frac{2K}{{\eta }_1}G\left(T\right)\right)}{2(C_1+C_2)}}}$, ${ϵ}_{P,r}=\sqrt{{\displaystyle \frac{log\left(\frac{4K\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{{\eta }_2}G\left(T\right)\right)}{2max\{1,N_{T_1}(s_i,a_i)\}}}}$, the regret conditioned on the failure event is given by

$$\begin{array}{cc}\hfill {\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\notin \mathcal{B}\}\right]& \le M{T}_2\mathbb{E}\left[\mathbb{I}\{B_i\notin \mathcal{B}\}\right]\hfill \\ \hfill & \le M{T}_2\left(2Kexp(-2(C_1+C_2){ϵ}_z^2)+4K\left|\mathcal{S}\right|\left|\mathcal{A}\right|exp(-2{ϵ}_{P,r}^2{N}_{T_1}(s_i,a_i))\right)\hfill \\ \hfill & \le {\displaystyle \frac{M{T}_2({\eta }_1+{\eta }_2)}{G\left(T\right)}}.\hfill \end{array}$$

(29)

   □

Then, we analyze the regret associated with good event occurrences. Based on Lemma 5, we establish that $\mathrm{Pr}\{B_i\in \mathcal{B}\}\ge 1-({\eta }_1+{\eta }_2)/G\left(T\right)$. To characterize this component of regret, we define the unknown MDPs as $\Omega :=[{\omega }_1,{\omega }_2,\dots ,{\omega }_K]$, where each ${\omega }_i$ is represented as $(z_i,P(s_i,a_i,s_i^{\prime }),R(s_i,a_i))$. The value function for arm i at state $s_i$ under the policy $\pi$ is $V_{\pi }\left(s_{i,1}\right)={\mathbb{E}}_{\pi }\left[{\sum }_{t=1}^{T_2}{z}_{i}R(s_{i,t},a_{i,t})\right]$. Moreover, the Bellman equations $\Theta$ for the MDPs $\Omega$ under the policy $\pi$ can be defined as ${\Theta }^{\Omega }{V}_{\pi }\left(s_i\right)=z_i\mathbb{E}[R(s_i,a_i)+{\sum }_{s_i^{\prime }\in \mathcal{S}}P(s_i,a_i,s_i^{\prime })V_{\pi }\left(s_i^{\prime }\right)]$.

For ${\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]$, we decompose it via Bellman equations into per-round sleeping value function differences under the learning policy ${\tilde{\pi }}^S$ (derived by solving $\mathrm{MLP}\left(\mathcal{B}\right)$ in OSILA to construct online indices). Through upper confidence interval estimation, we obtain $V_{{\pi }^{★}}^{\Omega }\left(s_{i,1}\right)-V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,1}\right)\le 0$, directly yielding the following lemma.

Lemma 6.  

Let $V_{max}$ be the upper bound of the sleeping value function. The part of regret with significant event occurrences can be bounded by:

$$\begin{array}{c}\hfill {\mathit{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]\le \sum _{i\in \left[K\right]}{\displaystyle \sum _{t=1}^{T_2}}{\mathbb{E}}_{{\tilde{\pi }}^S}\left[\left(1+\left|\mathcal{S}\right|V_{max}\right)\left({ϵ}_z+{ϵ}_{P,r}\right)\right]\end{array}$$

(30)

Proof. 

We denote the optimal policy as ${\pi }^{★}$, and the learned policy ${\tilde{\pi }}^S$. Based on the definition of Bellman Equations, we have

$$\begin{array}{cc}\hfill & {\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]=\sum _{i\in \left[K\right]}\left[V_{{\pi }^{★}}^{\Omega }\left(s_{i,1}\right)-V_{{\tilde{\pi }}^S}^{\Omega }\left(s_{i,1}\right)\right]\hfill \\ \hfill & =\sum _{i\in \left[K\right]}\left[V_{{\pi }^{★}}^{\Omega }\left(s_{i,1}\right)-V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,1}\right)+V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,1}\right)-V_{{\tilde{\pi }}^S}^{\Omega }\left(s_{i,1}\right)\right]\hfill \\ \hfill & \stackrel{\left(a\right)}{\le }\sum _{i\in \left[K\right]}\left[V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,1}\right)-V_{{\tilde{\pi }}^S}^{\Omega }\left(s_{i,1}\right)\right]\stackrel{\left(b\right)}{=}\sum _{i\in \left[K\right]}\left[{\Theta }_{{\tilde{\pi }}^S}^{\widehat{\Omega }}{V}_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,1}\right)-{\Theta }_{{\tilde{\pi }}^S}^{\Omega }{V}_{{\tilde{\pi }}^S}^{\Omega }\left(s_{i,1}\right)\right]\hfill \\ \hfill & =\sum _{i\in \left[K\right]}\left[\left({\Theta }_{{\tilde{\pi }}^S}^{\widehat{\Omega }}-{\Theta }_{{\tilde{\pi }}^S}^{\Omega }\right)V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,1}\right)+{\Theta }_{{\tilde{\pi }}^S}^{\Omega }\left(V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}-V_{{\tilde{\pi }}^S}^{\Omega }\right)\left(s_{i,1}\right)\right]=\sum _{i\in \left[K\right]}{\mathbb{E}}_{{\tilde{\pi }}^S}\left[{\displaystyle \sum _{t=1}^{T_2}}\left({\Theta }_{{\tilde{\pi }}^S}^{\widehat{\Omega }}-{\Theta }_{{\tilde{\pi }}^S}^{\Omega }\right)V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,t}\right)\right]\hfill \end{array}$$

(31)

Step (a) holds because policy ${\tilde{\pi }}^S$ is obtained by solving the $\mathrm{MLP}\left(\mathcal{B}\right)$, with the upper confidence estimation of the objective function chosen, resulting in $V_{{\pi }^{★}}^{\Omega }\left(s_{i,1}\right)-V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,1}\right)\le 0$. Step (b) is valid based on the definition of the Bellman equation. Where,

$$\begin{array}{c}\hfill \left({\Theta }_{{\tilde{\pi }}^S}^{\widehat{\Omega }}-{\Theta }_{{\tilde{\pi }}^S}^{\Omega }\right)V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s_{i,t}\right)={\mathbb{E}}_{{\tilde{\pi }}^S}[{\widehat{z}}_i\widehat{P}(s_{i,t},a_{i,t})+{\widehat{z}}_i\sum _{s^{\prime }\in \mathcal{S}}\widehat{P}(s_{i,t},a_{i,t},s^{\prime })V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s^{\prime }\right)\\ \hfill -z_{i}r(s_{i,t},a_{i,t})-z_i\sum _{s^{\prime }\in \mathcal{S}}P(s_{i,t},a_{i,t})V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s^{\prime }\right)]\end{array}$$

(32)

By plugging Equation (32) into Equation (31), we obtain:

$$\begin{array}{cc}\hfill & {\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]\le \hfill \\ \hfill & \sum _{i\in \left[K\right]}{\displaystyle \sum _{t=1}^{T_2}}{\mathbb{E}}_{{\tilde{\pi }}^S}\left[\underset{Term1}{\underbrace{\sum _{s^{\prime }\in \mathcal{S}}({\widehat{z}}_i\widehat{P}(s_{i,t},a_{i,t},s^{\prime })-z_{i}P(s_{i,t},a_{i,t},s^{\prime }))V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s^{\prime }\right)}}+\underset{Term2}{\underbrace{{\widehat{z}}_i\widehat{r}(s_{i,t},a_{i,t})-z_{i}r(s_{i,t},a_{i,t})}}\right]\hfill \end{array}$$

(33)

For the part of Term 1, we have:

$$\begin{array}{cc}\hfill Term1& ={\widehat{z}}_i\widehat{r}(s_{i,t},a_{i,t})-z_{i}r(s_{i,t},a_{i,t})\hfill \\ \hfill & =({\widehat{z}}_i-z_i)\widehat{r}(s_{i,t},a_{i,t})+z_i(\widehat{r}(s_{i,t},a_{i,t})-r(s_{i,t},a_{i,t}))\le {ϵ}_z+{ϵ}_{P,r}\hfill \end{array}$$

(34)

Then, for the part of Term 2, we can obtain:

$$\begin{array}{cc}\hfill \mathrm{Term}2& =\sum _{s^{\prime }\in \mathcal{S}}\left({\widehat{z}}_i\widehat{P}(s_{i,t},a_{i,t},s^{\prime })-z_{i}P(s_{i,t},a_{i,t},s^{\prime })\right)V_{{\tilde{\pi }}^S}^{\widehat{\Omega }}\left(s^{\prime }\right)\hfill \\ \hfill & \le \sum _{s^{\prime }\in \mathcal{S}}\left(({\widehat{z}}_i-z_i)P(s_{i,t},a_{i,t},s^{\prime })+z_i(\widehat{P}(s_{i,t},a_{i,t},s^{\prime })-P(s_{i,t},a_{i,t},s^{\prime }))\right)V_{max}\hfill \\ \hfill & \le \left|\mathcal{S}\right|({ϵ}_z+{ϵ}_{P,r})V_{max}\hfill \end{array}$$

(35)

By combining Equation (34) and Equation (35), we have $term1+term2\le (1+|\mathcal{S}|V_{max})({ϵ}_z+{ϵ}_{P,r})$, and 

$${\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]\le \sum _{i\in \left[K\right]}{\mathbb{E}}_{{\tilde{\pi }}^S}\left[{\displaystyle \sum _{t=1}^{T_2}}(1+|\mathcal{S}|V_{max})({ϵ}_z+{ϵ}_{P,r})\right]$$

(36)

   □

Based on Lemma 6, the upper bound of the regret with favorable event occurrences concerning $G\left(T\right)$ is presented below:

Lemma 7.  

Let $z_{max}={max}_{i\in \left[K\right]}{z}_i$. Under the algorithm OSILA, we have

$$\begin{array}{cc}\hfill {Reg}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]& \le {\displaystyle \frac{\sqrt{2}(1+|\mathcal{S}|V_{max})K}{2}}\sqrt{{\displaystyle \frac{T^2}{G\left(T\right)}}}\hfill \\ \hfill & \left(\sqrt{log\left(2KG\left(T\right)\right)z_{max}}+\sqrt{log\left(4K\right|\mathcal{S}\left|\right|\mathcal{A}\left|G\right(T\left)\right)\left|\mathcal{S}\right|\left|\mathcal{A}\right|}\right)\hfill \end{array}$$

(37)

Proof. 

First, based on Lemma 6, part ${\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]$ can be represented as follows:

$$\begin{array}{cc}\hfill & {\mathrm{Reg}}_{T_2}\mathbb{E}\left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]\le \sum _{i\in \left[K\right]}{\mathbb{E}}_{{\tilde{\pi }}^S}\left[{\displaystyle \sum _{t=1}^{T_2}}(1+|\mathcal{S}|V_{max})({ϵ}_z+{ϵ}_{P,r})\right]=\left(1+\left|\mathcal{S}\right|V_{max}\right)\times \hfill \\ \hfill & \left(\sum _{i\in \left[K\right]}\left(\mathbb{E}\left[T_2\sqrt{{\displaystyle \frac{log\left(\frac{2K}{{\eta }_1}G\left(T\right)\right)}{2(C_1+C_2)}}}\right]+\sqrt{log\left({\displaystyle \frac{4K\left|\mathcal{S}\right|\left|\mathcal{A}\right|G\left(T\right)}{{\eta }_2}}\right)}\right)\mathbb{E}\left[{\displaystyle \sum _{t=1}^{T_2}}\sqrt{{\displaystyle \frac{1}{2max\{1,N_{T_1}(s_i,a_i)\}}}}\right]\right)\hfill \end{array}$$

(38)

where $C_1={\displaystyle \frac{G\left(T\right)+\sqrt{2logKG\left(T\right)(1-z_i)}}{z_i}}$, and $C_2={\displaystyle \frac{\gamma \sqrt{G\left(T\right)(1-z_i)}}{z_i\sqrt{2logK}}}$. And, we have

$$C_1+C_2\ge {\displaystyle \frac{\sqrt{2logK}G\left(T\right)+(2logK+\gamma )\sqrt{G\left(T\right)(1-z_{max})}}{z_{max}\sqrt{2logK}}}$$

(39)

Thus, for the term of $\mathbb{E}\left[T_2\sqrt{{\displaystyle \frac{log\left(\frac{2K}{{\eta }_1}G\left(T\right)\right)}{2(C_1+C_2)}}}\right]$,

$$\mathbb{E}\left[T_2\sqrt{{\displaystyle \frac{log\left(\frac{2K}{{\eta }_1}G\left(T\right)\right)}{2(C_1+C_2)}}}\right]\le {\displaystyle \frac{\sqrt{2}}{2}}\sqrt{{\displaystyle \frac{T^{2}log\left(\frac{2K}{{\eta }_1}G\left(T\right)\right)\sqrt{2logK}{z}_{max}}{\sqrt{2logK}G\left(T\right)+(2logK+\gamma )\sqrt{G\left(T\right)(1-z_{max})}}}}$$

(40)

Next, for the term of $\mathbb{E}\left[{\sum }_{t=1}^{T_2}\sqrt{{\displaystyle \frac{1}{2max\{1,N_{T_1}(s_i,a_i)\}}}}\right]$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\displaystyle \sum _{t=1}^{T_2}}\sqrt{{\displaystyle \frac{1}{2max\{1,N_{T_1}(s_i,a_i)\}}}}\right]& ={\displaystyle \frac{\sqrt{2}}{2}}{T}_2\mathbb{E}\left[{\displaystyle \frac{1}{\sqrt{N_{T_1}(s_i,a_i)}}}\right]\stackrel{\left(a\right)}{\le }{\displaystyle \frac{\sqrt{2}}{2}}{T}_2{\displaystyle \frac{1}{\mathbb{E}\left[\sqrt{N_{T_1}(s_i,a_i)}\right]}}\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }{\displaystyle \frac{\sqrt{2}}{2}}{T}_2{\displaystyle \frac{1}{\sqrt{\mathbb{E}\left[N_{T_1}(s_i,a_i)\right]}}}\stackrel{\left(c\right)}{\le }{\displaystyle \frac{\sqrt{2}}{2}}{T}_2\sqrt{{\displaystyle \frac{\left|\mathcal{S}\right|\left|\mathcal{A}\right|}{G\left(T\right)}}}\hfill \end{array}$$

(41)

Step (a) and (b) are based on Jensen’s inequality, and step (c) follows from $C_i={\sum }_{s\in \mathcal{S}}{\sum }_{a\in \mathcal{A}}{N}_{T_1}(s_i,a_i)\ge G\left(T\right)$. Hence, we have

$$\begin{array}{cc}\hfill {\mathrm{Reg}}_{T_2}\mathbb{E}& \left[\mathbb{I}\{B_i\in \mathcal{B}\}\right]\hfill \\ \hfill & \le {\displaystyle \frac{\sqrt{2}(1+|\mathcal{S}|V_{max})K}{2}}\sqrt{{\displaystyle \frac{T^2}{G\left(T\right)}}}\left(\sqrt{log\left(2KG\left(T\right)\right)z_{max}}+\sqrt{log\left(4K\right|\mathcal{S}\left|\right|\mathcal{A}\left|G\right(T\left)\right)\left|\mathcal{S}\right|\left|\mathcal{A}\right|}\right)\hfill \end{array}$$

(42)

This completes the proof.    □

Consequently, when $G\left(T\right)=T^{2/3}$, the claim of Theorem 2 follows directly from Lemmas 4, 5 and 7.

6. Experiments

In this section, we validate two main theoretical insights using two application instances: (1) the asymptotic optimality of the sleeping index policy in offline settings, and (2) the sublinear cumulative regret and superior performance of OSILA in online scenarios.

Application instances construction. We constructed problem instances based on two real-world RMAB applications: (1) adherence to continuous positive airway pressure therapy (CPAP) [17,41,49], and (2) AoI optimization in edge-intelligent systems [12,40]. For each CPAP instance, the state space is $S=\{1,2,3\}$, and the reward values for each arm under different states and actions follow a uniform distribution with means of 0.3, 0.6, and 0.9. The availability probabilities of the arms are independently drawn from the interval $[0.1,0.9]$. The budget is set to $M=0.3K$. The AoI instance follows settings similar to [12,40]. More details are provided in Appendix C. In all experiments, the oracle LP for SIP and the modified LP subproblems arising in OSILA were solved using Gurobi. Unless otherwise stated, the online experiments used the theoretically motivated choice $G\left(T\right)=\lceil T^{2/3}\rceil$, with $K=20$, budget $M=0.3K$, and time horizon T varying from 5000 to 50,000, and each online data point was averaged over 100 independent trials. For the offline optimality-gap study, we fixed T = 50,000 and varied the number of arms from 20 to 100. All baseline algorithms were evaluated on the same generated instances and under the same realized arm-availability process for each trial.

Baselines. In the offline scenario, we compared the sleeping index policy with the ERC policy [25] and Whittle’s index policy [8]. These two policies have been proven to be asymptotically optimal without the arm-sleeping condition. To fit our setting, we modified them into greedy-based ERC and Whittle’s index policy. In each round, the $min\left\{\right|I_t|,M\}$ arms with the highest indices under the ERC/Whittle policy are selected from the available set. In the online scenario, we evaluated our OSILA with the UCB algorithm [50], UC-Whittle [16], and GM-R2MAB [25] using these constructed problem instances. Similarly, to fit the sleeping setting, during the exploitation phase of the UC-Whittle and GM-R2MAB, we select the top-$min\left\{\right|I_t|,M\}$ arms with the highest online ERC/Whittle’s indices from the available set. The UCB algorithm, widely used for stochastic sleeping bandits, achieves a regret bound of $\tilde{\mathcal{O}}\left(\sqrt{T}\right)$. UC-Whittle and GM-R2MAB, two state-of-the-art algorithms, address the online RMAB problem in non-sleeping settings. Thus, we compare the above advanced algorithms with our proposed method.

Asymptotic optimality. In the asymptotic optimality analysis of the sleeping index policy, we conducted experiments with T = 50,000 rounds. The optimality gap is defined as the difference between the upper bound of the optimal solution for the S-RMAB problem and the cumulative reward of the index policy. The upper bound of the optimal solution can be obtained by solving the LP problem using Gurobi [51]. We evaluated the variation in the optimality gap for three policies (SIP, greedy-ERC, and greedy-Whittle) as the number of arms increased from 20 to 100. As shown in Figure 1 and Figure 2a, Whittle’s index policy and the ERC policy no longer exhibit asymptotic optimality under the sleeping settings as the number of arms grows. In contrast, the proposed sleeping index policy demonstrates an optimality gap converging to zero, confirming the validity of Theorem 1. This trend is consistent with the theory: greedy-ERC and greedy-Whittle are adapted from non-sleeping RMAB policies and therefore do not account for the structural effect of stochastic arm availability in the oracle construction, whereas SIP is derived directly from the relaxed sleeping LP and explicitly incorporates availability into the index design.

Regret. Under the online setting, we further evaluate the performance of the OSILA. The number of arms is set to $K=20$, and the rounds for each independent experiment vary from 5000 to 50,000. We repeat the independent trials 100 times, and the mean and error range of the algorithm’s cumulative reward are computed. In the regret plots, the solid lines represent the mean values over all repeated experiments, while the shaded regions indicate the fluctuation range across individual trials. As shown in Figure 2b and Figure 3, under the scenario with dynamically changing sets of available arms, the expected regret of our proposed OSILA outperforms GM-R2MAB, UC-Whittle, and UCB algorithms in both problem instances. Specifically, at T = 50,000 rounds, OSILA achieves an average cumulative regret reduction in 35.7% compared to GM-R2MAB, 55.7% against UCB, and 64.1% versus UC-Whittle across three problem instances. This significant improvement stems from OSILA’s high exploration efficiency in dynamic decision spaces — it intelligently allocates sampling resources to rapidly estimate restless state transitions while minimizing wasted explorations on unavailable arms, thereby adapting more effectively to stochastic availability patterns. Furthermore, the cumulative regret of OSILA exhibits a sub-linear growth pattern with increasing rounds. This confirms that the OSILA effectively addresses the online sleeping restless bandits problem.

From a qualitative perspective, the performance gain comes from both components of OSILA, but their roles are slightly different across baselines. Compared with UCB, the main advantage comes from the online sleeping index construction, since UCB does not exploit the restless state dynamics and therefore cannot prioritize arms according to long-term state-dependent value. In contrast, compared with UC-Whittle and GM-R2MAB, which already use RMAB-style indices but are designed for non-sleeping settings, the larger gain mainly comes from the availability-aware exploration mechanism and the sleeping-aware exploitation rule working together. MEG prevents low-availability arms from being persistently under-explored, while the online sleeping index uses the learned availability information to rank only the currently available arms more appropriately. In this sense, the strongest empirical gains of OSILA come from the joint effect of availability-aware exploration and sleeping-aware index construction, rather than from either component alone.

7. Conclusions

This paper is the first to study the online sleeping RMAB problem motivated by dynamic edge-intelligent systems with stochastic arm availability and restless state evolution. We first derive a sleeping-index policy as the oracle through fluid relaxation and establish its asymptotic optimality. Building on this oracle, we then propose OSILA, which combines the Minimum Exploration Guarantee mechanism with an online sleeping-index construction to handle unknown transitions, rewards, and availabilities. On the theoretical side, we characterize the stochastic exploration length and prove a regret bound of $\tilde{\mathcal{O}}\left(K{T}^{2/3}\sqrt{logT}\right)$. On the empirical side, experiments on CPAP intervention planning and AoI optimization in edge-intelligent systems validate both the oracle behavior and the practical effectiveness of OSILA against representative baselines. Future work includes extending the framework to correlated availabilities and non-stationary environments.