Multi-Armed Bandit-Based User Network Node Selection

Abstract

In the scenario of an integrated space–air–ground emergency communication network, users encounter the challenge of rapidly identifying the optimal network node amidst the uncertainty and stochastic fluctuations of network states. This study introduces a Multi-Armed Bandit (MAB) model and proposes an optimization algorithm leveraging dynamic variance sampling (DVS). The algorithm posits that the prior distribution of each node’s network state conforms to a normal distribution, and by constructing the distribution’s expected value and variance, it maximizes the utilization of sample data, thereby maintaining an equilibrium between data exploitation and the exploration of the unknown. Theoretical substantiation is provided to illustrate that the Bayesian regret associated with the algorithm exhibits sublinear growth. Empirical simulations corroborate that the algorithm in question outperforms traditional ε-greedy, Upper Confidence Bound (UCB), and Thompson sampling algorithms in terms of higher cumulative rewards, diminished total regret, accelerated convergence rates, and enhanced system throughput.

1. Introduction

The Space-Air-Ground Integrated Network (SAGIN) integrates satellite systems, aerial networks, and terrestrial communication infrastructures [1], thereby enabling global, uninterrupted coverage. It holds considerable potential for application in scenarios such as disaster response, intelligent transportation systems, and the evolution towards 6G communication networks [2]. The integration of cutting-edge technologies, including artificial intelligence, machine learning, and Software-Defined Networking (SDN), further augments the performance and adaptability of the SAGIN [3]. The pivotal technological hurdles encompass dynamic node management, interconnection of heterogeneous networks, resource allocation optimization, and the intelligent management of networks [4].

In the context of emergency communication, users are in constant motion, and terrestrial base stations may be insufficient to fulfill their communication demands, thus requiring the collaborative support of space and aerial networks [5]. To ensure the reliability of data transmission, it is imperative for users to swiftly connect to the network node at the highest rate within the signal range [6]. However, given the dynamic shifts in the location and status of network nodes, users must rapidly connect to the optimal network node guided by a specific algorithm, without the benefit of prior knowledge of the network conditions. Addressing how users in emergency communication scenarios can expeditiously access the most suitable network nodes has become an imperative issue.

2. Related Work

Online learning methodologies serve as efficacious algorithms for learning and forecasting within dynamic settings [7]. The Multi-Armed Bandit (MAB) problem, often referred to as the slot machine dilemma [8], is a quintessential issue in the realm of online learning. The MAB framework has garnered widespread application due to its capacity to facilitate access optimization even amidst a dearth of environmental information. In particular, ref. [9] investigates the selection process among multiple channels under the condition that channel information adheres to independent and identically distributed variations within a solitary user context, proposing a decision-making framework predicated on the Restless Multi-Armed Bandit (RMAB) model. Ref. [10] integrates the RMAB model, introducing a greedy algorithm for channel selection to augment the spectrum access efficiency for users. Ref. [11] pioneers the employment of index algorithms to address the archetypal MAB challenge, while ref. [12] refines the confidence parameter of the Upper Confidence Bound (UCB) algorithm to enhance its efficacy.

Existing studies on network selection are typically marred by two principal deficiencies: firstly, the oversight of the influence that immediate gains may exert on future earnings; secondly, the linear growth pattern of cumulative regret values yielded by current algorithms, which results in diminished learning efficiency and protracted convergence. Such outcomes are at odds with practical aspirations for achieving higher efficiency through straightforward methods.

Addressing these concerns, the present research introduces an enhanced index algorithm designed to refine the network selection process. This algorithm takes into account the interplay between immediate and prospective future gains, leveraging the strengths of both the UCB and Thompson sampling algorithms to achieve a harmonious equilibrium between exploration and exploitation. Simulation results indicate that, in contrast to extant methodologies, the dynamic variance sampling algorithm proposed herein not only escalates learning efficiency but also mitigates cumulative regret, thereby augmenting system throughput.

3. System Model

3.1. Network Architecture

The network architecture of this study is illustrated in Figure 1. The research area is conceptualized as a circular zone, equipped with satellites, unmanned aerial vehicles (UAVs), and base stations. The satellite signals ensure comprehensive coverage across the entire area, and UAVs operate on predefined circular trajectories, while the coverage of base stations is confined. Users are outfitted with multi-mode antennas capable of accessing multiple networks, yet they are restricted to connecting to a single network node at any given instant. The user communication is structured in a time-slotted manner, segmenting the user communication timeframe into T discrete time slots, with each slot separated by a relatively minor interval. The crux of this paper’s research lies in the decision-making process for an individual user to access the space–air–ground integrated emergency communication network node during each time slot.

Figure 1. Network architecture.

3.2. Channel Model

The system includes three types of links: space-to-ground, aerial-to-ground, and base station-to-ground links. According to the literature [13], the channel state information (CSI) from the satellite to the user is as follows:

$$h_{SAT}=\sqrt{C_{L}b\beta }\exp \left(-j\theta \right)$$

(1)

where $C_L$ denotes the free space loss, which can be calculated by the formula $C_L={\left(\lambda /4\pi \right)}^2/\left(d^2+l^2\right)$, where $\lambda$ denotes the signal wavelength, $l$ denotes the horizontal distance between the center of the satellite beam and the user, $d$ denotes the vertical height of the satellite relative to the ground, $\beta$ denotes the channel gain caused by the rain attenuation, which obeys the lognormal distribution, ${\beta }_{dB}$ is the $\beta$ form of $\mathrm{d}\mathrm{B}$, $\theta$ is a phase vector uniformly distributed within the range $[0,2\mathrm{\pi }]$, and $b$ indicates the satellite beam gain, which is defined as follows:

$$b=G{\left(\frac{J_1\left({\mu }_0\right)}{2{\mu }_0}-36\frac{J_3\left({\mu }_0\right)}{{{\mu }_0}^2}\right)}^2$$

(2)

where $G$ represents the maximum gain of the satellite antenna ${\mu }_0=2.07123\sin \left(\alpha \right)/\sin \left({\alpha }_{3\mathrm{d}\mathrm{B}}\right)$, $\alpha$ is the elevation angle between the center of the beam and the user, ${\alpha }_{3\mathrm{d}\mathrm{B}}$ is the $3 \mathrm{d}\mathrm{B}$ angle of the satellite beam, and $J_1(·)$ and $J_3(·)$ are the first-order and third-order Bessel functions of the first class, respectively.

According to the literature [13], the channel state information (CSI) from the aerial platform to the user is as follows:

$$a_{UAV}=\sqrt{G_L}\left(\sqrt{\frac{K}{K+1}}{a}_{LoS}+\sqrt{\frac{1}{K+1}}{a}_{Ray}\right)$$

(3)

where $G_L=C_0/\left(U_d^2+U_h^2\right)$ denotes the path loss, where $C_0$ denotes the channel power gain for a reference distance of 1 m, $U_d$ is the horizontal distance from the UAV to the target user, and $U_h$ is the height of the UAV. The small-scale fading follows the Rice (Rician) channel model, where $K$ is the Rician coefficient. $a_{LoS}$ is the line-of-sight Rician fading component and $a_{Ray}$ is the non-line-of-sight Rayleigh fading component.

According to the literature [13], the channel state information (CSI) from the base station to the user is as follows:

$$g_{BS}=\sqrt{\alpha }{g}_0$$

(4)

where $\alpha =C_0{r}^{-4}$ represents the large-scale fading, where $C_0$ denotes the channel power gain at a reference distance of 1 m, $r$ denotes the distance between the base station and the user, and g₀ represents the small-scale fading, which follows a $Nakagami\text{-}\mathrm{m}$ distribution.

3.3. Communication Model

Based on the method of calculating the channel state information (CSI) mentioned, the communication rate that can be obtained by a user when connecting to a satellite, a UAV, and a base station, respectively (without considering inter-channel interference) can be expressed by the formula

$$R_{SAT}={\log }_2\left(1+\frac{p_{SAT}{\left|h_{SAT}\right|}^2}{{{\delta }_{SAT}}^2}\right)$$

(5)

$$R_{UAV}={\log }_2\left(1+\frac{p_{UAV}{\left|a_{UAV}\right|}^2}{{{\delta }_{UAV}}^2}\right)$$

(6)

$$R_{BS}={\log }_2\left(1+\frac{p_{BS}{\left|g_{BS}\right|}^2}{{{\delta }_{BS}}^2}\right)$$

(7)

In Equation (5), $p_{SAT}$ denotes the power of the satellite, $h_{SAT}$ denotes the CSI from the satellite to the user, and ${\delta }_{SAT}$ denotes the noise power of the satellite received by the user. Similarly, the meaning of each symbol in Equations (6) and (7) can be known.

3.4. Benefit Model

The benefit of the proposed algorithm in this paper after the user selects a network node to access is represented as in Equation (8):

$$r=\frac{R_{node}}{R_{upper}}$$

(8)

where $R_{node}$ denotes the communication rate (i.e., $R_{SAT}$,$R_{UAV}$, or $R_{BS}$) that the user obtains from the network nodes, and $R_{upper}$ denotes the upper bound of the communication rate that the user is able to obtain from all the network nodes, e.g., when the maximal rate is 100, $R_{upper}\ge 100$, which ensures that the user’s gain interval is $\left[0, 1\right]$.

Meanwhile, in order to experimentally compare the algorithm proposed in this paper with the traditional ε-greedy algorithm, UCB algorithm, and Thompson sampling algorithm, according to the literature [14], the gain $r$ can be transformed into a Bernoulli random variable, and the current gain $r^{\prime }$ obeys the Bernoulli distribution with parameter $r$ (i.e., $r^{\prime }~\mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}\left(r\right)$) with $r^{\prime }\in \left\{0, 1\right\}$. The article has shown that the above two returns are algorithmically equivalent.

Furthermore, to experimentally compare the algorithm proposed in this paper with conventional algorithms, including the ε-greedy algorithm, the Upper Confidence Bound (UCB) algorithm, and the Thompson sampling algorithm, the reward $r$ can be converted into a Bernoulli random variable, as outlined in reference [14]. The instantaneous reward $r^{\prime }$ is governed by a Bernoulli distribution parameterized by $r$ (i.e., $r^{\prime }~\mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i}\left(r\right)$) with $r^{\prime }\in \left\{0, 1\right\}$. The aforementioned article has substantiated the algorithmic equivalence of these two forms of rewards.

3.5. Objective Function

The purpose of the algorithm based on the MAB model is to determine the user’s strategy for selecting a network node at each moment in time so as to maximize the expectation of the total gain over $T$ time slots, i.e.,

$$\max \mathbb{E}\left[{\displaystyle {\sum }_{t=1}^T{r}_{i\left(t\right)}}\right]$$

(9)

where $i\left(t\right)$ denotes the network node $i$ selected by the user according to the algorithm at moment $t$, and $r_{i\left(t\right)}$ denotes the gain after selecting network node $i$, corresponding to $r$ and $r^{\prime }$ defined in the gain model. In order to compare the effects of different algorithms more intuitively, this paper uses the minimization of the expected total regret as the objective function corresponding to the maximization of the expected total gain, where the so-called regret is the expected rate lost at each moment due to the failure to select the best network node. Define ${\mu }_i$ as the expected gain of network node $i$, so that ${\mu }^{*}=\max {\mu }_i$ denotes the expected gain of the optimal network node, ${∆}_i={\mu }^{*}-{\mu }_i$ denotes the suboptimal network gap, and also define $k_i\left(t\right)$ as the number of times that the network node $i$ has been selected prior to the moment of $t$. Then the total regret of the user in T time slots is denoted as

$$\mathbb{E}\left[\Re \left(T\right)\right]=\mathbb{E}\left[{\displaystyle {\sum }_{t=1}^T\left({\mu }^{*}-{\mu }_{i\left(t\right)}\right)}\right]={\displaystyle {\sum }_i{\Delta }_i\cdot \mathbb{E}\left[k_i\left(T\right)\right]}$$

(10)

4. Network Selection Mechanism Based on the MAB Model

4.1. Dynamic Variance Sampling Algorithm

The issue of user network node selection in an unknown environment has been modeled as a Multi-Armed Bandit (MAB) model. As an advanced dynamic stochastic control framework, the MAB model has excellent learning capabilities and is mainly used to solve problems of selection and resource allocation under limited resources. This includes, but is not limited to, scenarios such as channel allocation, opportunistic network access, and routing selection. Through the application of the MAB model, users can make optimal decisions in uncertain environments, thereby effectively enhancing the overall performance of the system. Reference [15] proposed an Upper Confidence Bound (UCB)-based index algorithm, which, although it reduces the algorithm complexity compared to traditional algorithms, still has a certain gap in overall revenue compared to the ideal state, and the convergence speed is slow; reference [16] studied the theoretical performance of the Thompson sampling algorithm, which, although it has a lower regret lower bound compared to the UCB index algorithm, still has a gap from the ideal state.

The index value of the UCB algorithm consists of two parts: the sample average reward of the current network and the confidence factor, which is represented as follows:

$${\theta }_i(t)={\widehat{\mu }}_i\left(t\right)+\sqrt{2\ln \left(t-1\right)/k_i(t)}$$

(11)

where $k_i\left(t\right)$ denotes the number of times a network node has been selected at moment $t$, ${\widehat{\mu }}_i\left(t\right)$ denotes the mean benefit of network node $i$ before moment $t$, which reflects how well the algorithm utilizes the data, and the confidence factor $\sqrt{2\ln \left(t-1\right)/k_i\left(t\right)}$ is a quantity inversely related to the number of times it has been selected, which reflects how well the data has been explored.

The Thompson sampling algorithm is a stochastic algorithm based on Bayesian ideas, which selects arms at each time step based on their posterior probability of being the best arm by assuming a Beta prior distribution for the reward distribution parameter of each arm. The index values are calculated as follows:

$${\theta }_i(t)=\mathrm{Sample}[Beta(1+S_i\left(t\right),1+F_i\left(t\right)]$$

(12)

where $Sample\left[Beta\left(·\right)\right]$ denotes the $Beta$ distribution sampling, $S_i\left(t\right)$ and $F_i\left(t\right)$ denote the number of times that the network node has been selected with a gain of 1 and the number of times that the network node has been selected with a gain of 0 before the moment $t$, respectively.

The advantage of the UCB algorithm is the introduction of the confidence factor related to the number of times selected, which enhances the exploratory nature of the algorithm, and the disadvantage is the low exploratory efficiency and slow convergence speed; the advantages of the Thompson sampling algorithm are the introduction of the Bayesian sampling idea, that the assumption of the prior distribution is more in line with the actual scenario, and that its convergence speed has been improved, but it is a large gap with the ideal value. At the same time, from the CSI model, it can be seen that the changes in the state of network nodes are closer to the normal distribution, so the normal distribution is considered as the prior distribution of network state changes.

Based on the characteristics of the above two algorithms, this paper considers the improvement in terms of the advantages of the two algorithms. Bayesian sampling and the confidence factor are introduced into the algorithm at the same time, which assumes that the return of each network node obeys the normal distribution, the mean value of the sample is used as the expectation (reflecting the utilization of the data), the number of times of being selected is introduced into the sample variance (reflecting the exploration of the data), and the index value is calculated as follows:

$${\theta }_i(t)=Sample\left[\mathcal{N}\left({\widehat{\mu }}_i\left(t\right),\frac{1}{k_i(t)+1}\right)\right]$$

(13)

where $Sample\left[\mathcal{N}\right(·\left)\right]$ denotes the sampling of a normal distribution, ${\widehat{\mu }}_i\left(t\right)$ denotes the mean benefit of network node $i$ before time $t$, and $k_i\left(t\right)$ denotes the number of times network node $i$ is selected at time $t$. The update rule is as follows:

$${\widehat{\mu }}_i(t+1)=\{\begin{array}{l}\frac{{\widehat{\mu }}_i(t)\times k_i(t)+r_i(t)}{k_i(t)+1}, select i\\ {\widehat{\mu }}_i(t) \hspace{1em}, other\end{array}$$

(14)

$$k_i(t+1)=\{\begin{array}{l}{k}_i(t)+1 , select i\\ k_i(t) , other\end{array}$$

(15)

4.2. Theoretical Analysis and Proof

Definition 1.

$k_i\left(t\right)$ denotes the number of times network node $i$ has been selected prior to moment t. $S_i\left(t\right)$ and $F_i\left(t\right)$ denote the number of times network node $i$ has been selected prior to moment $t$ with a gain of 1 and a gain of 0, respectively. $i\left(t\right)$ denotes the value of the index of the network node that has been selected at moment $t$.

Definition 2.

Assuming that network node 1 is the optimal network and that ${\mu }_i$ denotes the expected return of network node $i$, there is ${\mu }_i<{\mu }_1$ for $\forall i\ne 1$. Define $x_i$, $y_i$ as two real numbers that satisfy ${\mu }_i<x_i<y_i<{\mu }_1$, and obviously $x_i$ and $y_i$ must exist. Define $L_i\left(T\right)=lnT/d(x_i,y_i)$, where $d\left(x_i,y_i\right)=x_i\mathit{ln}\left(x_i/y_i\right)+(1-x_i)ln\left[(1-x_i)/(1-y_i)\right]$ denotes the KL dispersion between Bernoulli distributions with the parameters $x_i$ and $y_i$, respectively, of the KL distance.

Definition 3.

${\theta }_i\left(t\right)$ denotes the sample values sampled by the algorithm from the posterior distribution of network node $i$ at moment $t$ and ${\theta }_i\left(t\right)~\mathcal{N}\left({\widehat{\mu }}_i\left(t\right),1/\left(k_i\left(t\right)+1\right)\right)$.

Definition 4.

${\widehat{\mu }}_i\left(t\right)$ denotes the mean value of the returns of network node $i$ at moment $t$, defined as ${\widehat{\mu }}_i\left(t\right)=\sum _{\tau =1,i\left(\tau \right)=i}^{t-1}{r}_i\left(\tau \right)/\left[k_i\left(t\right)+1\right]$, for $i\ne 1$, defining $E_i^{\mu }\left(t\right)$ as the event ${\widehat{\mu }}_i\left(t\right)\le x_i$, and $E_i^{\theta }\left(t\right)$ as the event ${\theta }_i\left(t\right)\le y_i$.

It can be seen that both ${\widehat{\mu }}_i\left(t\right)$ and ${\theta }_i\left(t\right)$ are approximations of the true expected return of network node $i$. The former is an empirical estimate, the latter is a sampled sample from the posterior distribution, and $x_i$ and $y_i$ are upper bounds on the true return expectation of network node $i$. Thus, intuitively, the significance of $E_i^{\mu }\left(t\right)$ and $E_i^{\theta }\left(t\right)$ is that these two estimates should not be overestimated too much; specifically, do not exceed the thresholds $x_i$ and $y_i$.

Definition 5.

${\mathcal{F}}_t$ denotes the sequence of historical policy information prior to moment $t$, defined as ${\mathcal{F}}_t=\left\{i\left(\tau \right),r_{i\left(\tau \right)},\tau =1,\dots ,t\right\}$, where $i\left(\tau \right)$ denotes the index value of the network node that is selected at moment $\tau$, and $r_{i\left(\tau \right)}$ is its corresponding gain. Defining ${\mathcal{F}}_0=\left\{\right\}$ and satisfying ${\mathcal{F}}_0\subseteq {\mathcal{F}}_1\subseteq \dots \subseteq {\mathcal{F}}_{T-1}$, it can be seen that the distributions of $k_i\left(t\right)$, ${\widehat{\mu }}_i\left(t\right)$, and ${\theta }_i\left(t\right)$ in the above definitions, and whether or not the events $E_i^{\mu }\left(t\right)$ and $E_i^{\theta }\left(t\right)$ occur, are all determined by ${\mathcal{F}}_{t-1}$.

Definition 6.

Defining $p_{i,t}=\mathbb{P}\left({\theta }_1\left(t\right)>y_i|{\mathcal{F}}_{t-1}\right)$, it can be seen that $p_{i,t}$ is a random variable determined by ${\mathcal{F}}_{t-1}$.

Lemma 1.

According to Lemma 1 of reference [17], when  $i\ne 1$ , we have the following for any  $t$  and   ${\mathcal{F}}_{t-1}$:

$$\mathbb{P}\left(i\left(t\right)=i,E_i^{\mu }\left(t\right),E_i^{\theta }\left(t\right)|{ℱ}_{t-1}\right)\le \frac{1-p_{i,t}}{p_{i,t}}\mathbb{P}\left(i\left(t\right)=1,E_i^{\mu }\left(t\right),E_i^{\theta }\left(t\right)|{ℱ}_{t-1}\right)$$

(16)

Lemma 2.

Let ${\tau }_j$ denote the moment when network node 1 is selected for the $j$th time, which can be obtained according to Lemma 2.13 of reference [18]:

$${\displaystyle \sum _{j=0}^{T-1}\mathbb{E}\left[\frac{1}{p_{i,{\tau }_j+1}}-1\right]}\le \{\begin{array}{l}{e}^{11}+5, \forall j\\ \frac{4}{T{\Delta }_i^2} ,j>4L_i(T)\end{array}$$

(17)

 where $L_i\left(T\right)=18\mathit{ln}\left(T{∆}_i^2\right)/{∆}_i^2$.

Fact 1

(Chernoff–Hoeffding Bound). Let $X_1,\dots ,X_n$ be random variables on the interval $[0, 1]$ and $\mathbb{E}\left(X_t|X_1,\dots ,X_{t-1}\right)=\mu$, so that $S_n=X_1+\cdots +X_n$, and there is the following for any $a\ge 0$:

$$\mathbb{P}\left(S_n\ge n\mu +a\right)\le e^{-2a^2/n}$$

(18)

$$\mathbb{P}\left(S_n\le n\mu -a\right)\le e^{-2a^2/n}$$

(19)

Fact 2.

Let $Z$ denote a random variable obeying a normal distribution with mean $\mu$ and variance ${\sigma }^2$ for $\forall z\in Z$ with

$$\frac{1}{4\sqrt{\pi }}\cdot e^{-7z^2/2}\le \mathbb{P}\left(\left|Z-\mu \right|>z\sigma \right)\le \frac{1}{2}{e}^{-z^2/2}$$

(20)

Theorem 1.

The upper bound on the regret of the dynamic variance sampling algorithm is given by

$$\mathbb{E}\left[\Re \left(T\right)\right]\le O\left(\sqrt{NT\ln N}\right)$$

(21)

 where $T$ is the total time duration and $N$ is the number of network nodes.

Proof. 

According to the definition of regret in Equation (10), we can see that

$$\mathbb{E}\left[\Re \left(T\right)\right]={\displaystyle {\sum }_i{\Delta }_i\cdot \mathbb{E}\left[k_i\left(T\right)\right]}$$

Firstly, the regret is decomposed according to the events defined in Definition 4, but here instead of decomposing the regret directly, the expectation of the number of times a suboptimal network node is selected is decomposed. Because according to the definition of regret, the expectation of the number of times a network node is selected is multiplied by the suboptimal network gap and summed up to get the final regret, and the optimal network node does not contribute to the regret, it is only necessary to decompose the number of times a suboptimal network node is selected. For $\forall i\ne 1$, there is

$$\mathbb{E}\left[k_i\left(T\right)\right]=\mathbb{E}\left[{\displaystyle \sum _{t=1}^T\mathbb{I}\left(i\left(t\right)=i\right)}\right]={\displaystyle \sum _{t=1}^T\mathbb{E}\left[\mathbb{I}\left(i\left(t\right)=i\right)\right]}={\displaystyle \sum _{t=1}^T\left(i\left(t\right)=i\right)}$$

where $\mathbb{I}(·)$ denotes the indicator function, and the decomposition of the above equation using the event $E_i^{\mu }\left(t\right)$ and its complement $\overline{E_i^{\mu }\left(t\right)}$ is obtained:

$$\mathbb{E}\left[k_i\left(T\right)\right]={\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,E_i^{\mu }\left(t\right)\right)}+{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\mu }\left(t\right)}\right)}$$

Continuing to decompose the above equation by the event $E_i^{\theta }\left(t\right)$ yields

$$\begin{array}{l}\mathbb{E}\left[k_i\left(T\right)\right]=\underset{①}{\underbrace{{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)\right)}}}+\underset{②}{\underbrace{{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}}}\\ +\underset{③}{\underbrace{{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\mu }\left(t\right)}\right)}}}\end{array}$$

(22)

Next, derive the upper bounds for each of the above three terms, starting with the upper bound for Equation (1). Combined with the relationship stated in Definition 5, this is obtained from the properties of conditional probability and Lemma 1:

$$\begin{array}{ll}& {\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)\right)}\\ =& {\displaystyle \sum _{t=1}^T\mathbb{E}\left[\mathbb{P}\left(i\left(t\right)=i,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)|{ℱ}_{t-1}\right)\right]}\\ \le & {\displaystyle \sum _{t=1}^T\mathbb{E}\left[\frac{1-p_{i,t}}{p_{i,t}}\mathbb{P}\left(i\left(t\right)=1,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)|{ℱ}_{t-1}\right)\right]}\\ =& {\displaystyle \sum _{t=1}^T\mathbb{E}\left[\mathbb{E}\left[\frac{1-p_{i,t}}{p_{i,t}}\mathbb{I}\left(i\left(t\right)=1,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)|{ℱ}_{t-1}\right)\right]\right]}\\ =& {\displaystyle \sum _{t=1}^T\mathbb{E}\left[\frac{1-p_{i,t}}{p_{i,t}}\mathbb{I}\left(i\left(t\right)=1,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)\right)\right]}\end{array}$$

(23)

The second equality sign above utilizes the property that $p_{i,t}$ is invariant given ${\mathcal{F}}_{t-1}$. Combined with $p_{i,t}=\mathbb{P}\left({\theta }_1\left(t\right)>y_i|{\mathcal{F}}_{t-1}\right)$, it can be seen that the value of $p_{i,t}$ only changes with the distribution of ${\theta }_1\left(t\right)$, i.e., only after network node 1 is selected. Defining ${\tau }_j$ to denote the moment when network node 1 is selected for the jth time, the values of $p_{i,t}$ are equal at all $t\in \left\{{\tau }_j+1,\dots ,{\tau }_{j+1}\right\}$ moments. Therefore,

$$\begin{array}{ll}& {\displaystyle \sum _{t=1}^T\mathbb{E}\left[\frac{1-p_{i,t}}{p_{i,t}}\mathbb{I}\left(i\left(t\right)=1,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)\right)\right]}\\ =& {\displaystyle \sum _{j=1}^{T-1}\mathbb{E}\left[\frac{1-p_{i,{\tau }_j+1}}{p_{i,{\tau }_j+1}}{\displaystyle \sum _{t={\tau }_j+1}^{{\tau }_{j+1}}\mathbb{I}\left(i\left(t\right)=1,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)\right)}\right]}\\ \le & {\displaystyle \sum _{j=0}^{T-1}\mathbb{E}\left[\frac{1-p_{i,{\tau }_j+1}}{p_{i,{\tau }_j+1}}\right]}\end{array}$$

(24)

Let $L_i\left(T\right)=18\ln \left(T{∆}_i^2\right)/{∆}_i^2$, and by Lemma 2, when $j\ge 4L_i$, there are

$$\mathbb{E}\left[\frac{1-p_{i,{\tau }_j+1}}{p_{i,{\tau }_j+1}}\right]\le \frac{4}{T{\Delta }_i^2}$$

(25)

Substituting Equation (25) into Equation (24) gives the upper bound of Equation (1):

$${\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,E_i^{\theta }\left(t\right),E_i^{\mu }\left(t\right)\right)}\le 4L_i\left(T\right)\left(e^{64}+4\right)+\frac{4}{{\Delta }_i^2}$$

(26)

Next, the upper bound of Equation (2) is derived, and Equation (2) is decomposed into two parts according to the magnitude relationship between $k_i\left(t\right)$ and $L_i\left(T\right)$:

$$\begin{array}{ll}{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}& ={\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,k_i\left(t\right)\le L_i\left(T\right),\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}\\ & +{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,k_i\left(t\right)>L_i\left(T\right),\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}\end{array}$$

(27)

First analyzing the first term of Equation (27),we can get

$$\begin{array}{l}{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,k_i\left(t\right)\le L_i\left(T\right),\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}\\ \le \mathbb{E}\left[{\displaystyle \sum _{t=1}^T\mathbb{I}\left(i\left(t\right)=i,k_i\left(t\right)\le L_i\left(T\right)\right)}\right]\\ \le L_i\left(T\right)\end{array}$$

(28)

Next, analyzing the latter term of Equation (27), if the value of $k_i\left(t\right)$ is large and the event $E_i^{\mu }\left(t\right)$ occurs, then the probability of the event $\overline{E_i^{\theta }\left(t\right)}$ occurring will be very small, and in conjunction with the definition of the event in Definition 4, there is

$$\begin{array}{ll}& {\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,k_i\left(t\right)>L_i\left(T\right),\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}\\ \le & \mathbb{E}\left[{\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\theta }\left(t\right)}|k_i\left(t\right)>L_i\left(T\right),E_i^{\mu }\left(t\right),{ℱ}_{t-1}\right)}\right]\\ \le & \mathbb{E}\left[{\displaystyle \sum _{t=1}^T\mathbb{P}\left({\theta }_i\left(t\right)>y_i|k_i\left(t\right)>L_i\left(T\right),{\widehat{\mu }}_i\left(t\right)\le x_i,{ℱ}_{t-1}\right)}\right]\end{array}$$

(29)

From the definition, we can see that ${\theta }_i\left(t\right)~\mathcal{N}\left({\widehat{\mu }}_i\left(t\right),1/\left(k_i\left(t\right)+1\right)\right)$, and according to the normal distribution property, given ${\widehat{\mu }}_i\left(t\right)<x_i$, there are

$$\mathbb{P}\left[Sample\left[\mathcal{N}\left({\widehat{\mu }}_i\left(t\right),{\sigma }^2\right)\right]\right]\le \mathbb{P}\left[Sample\left[\mathcal{N}\left(x_i,{\sigma }^2\right)\right]\right]$$

(30)

Therefore, Equation (29) can be further scaled as

$$\begin{array}{ll}& \mathbb{E}\left[{\displaystyle \sum _{t=1}^T\mathbb{P}\left({\theta }_i\left(t\right)>y_i|k_i\left(t\right)>L_i\left(T\right),{\widehat{\mu }}_i\left(t\right)\le x_i,{ℱ}_{t-1}\right)}\right]\\ \le & \mathbb{E}\left[{\displaystyle \sum _{t=1}^T\mathbb{P}\left(Sample\left[\mathcal{N}\left(x_i,\frac{1}{k_i\left(t\right)+1}\right)>y_i\right]|k_i\left(t\right)>L_i\left(T\right),{ℱ}_{t-1}\right)}\right]\end{array}$$

(31)

From the probability density function of normal distribution and its distribution characteristics, when $k_i\left(t\right){>L}_i\left(T\right)$, there are

$$\mathbb{P}\left(Sample\left[\mathcal{N}\left(x_i,\frac{1}{k_i\left(t\right)+1}\right)>y_i\right]\right)\le \frac{1}{2}{e}^{-\frac{\left(k_i\left(t\right)+1\right){\left(y_i-x_i\right)}^2}{2}}\le \frac{1}{2}{e}^{-\frac{L_i\left(t\right)\cdot {\left(y_i-x_i\right)}^2}{2}}$$

(32)

Taking $x_i={\mu }_i+{∆}_i/3$ and $y_i={\mu }_1-{∆}_i/3$, $y_i-x_i={∆}_i/3$ and substituting in Equation (32):

$$\mathbb{P}\left(Sample\left[\mathcal{N}\left(x_i,\frac{1}{k_i\left(t\right)+1}\right)>y_i\right]\right)\le \frac{1}{2}{e}^{-\frac{L_i\left(t\right)\cdot {\left(y_i-x_i\right)}^2}{2}}=\frac{1}{2T{\Delta }_i^2}$$

(33)

Therefore,

$${\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,k_i\left(t\right)>L_i\left(T\right),\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}\le \frac{1}{2T{\Delta }_i^2}$$

(34)

Substituting Equations (28) and (34) into Equation (27) gives the upper bound of Equation (2) as

$${\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\theta }\left(t\right)},E_i^{\mu }\left(t\right)\right)}\le L_i\left(T\right)+\frac{1}{2T{\Delta }_i^2}$$

(35)

Finally, to derive the upper bound of Equation (3), define ${\tau }_k$ to denote the moment of the $k$th selection of network node $i$. Let ${\tau }_k=0$. According to the definition of $E_i^{\mu }\left(t\right)$, we can see that $\overline{E_i^{\mu }\left(t\right)}$ denotes the event ${\widehat{\mu }}_i\left(t\right)>x_i$, and then we have

$${\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\mu }\left(t\right)}\right)}\le {\displaystyle \sum _{k=0}^{T-1}\mathbb{P}\left(\overline{E_i^{\mu }\left({\tau }_{k+1}\right)}\right)}$$

(36)

When $k\ge 1$, there is

$${\widehat{\mu }}_i\left({\tau }_{k+1}\right)=\frac{S_i\left({\tau }_{k+1}\right)}{k+1}\le \frac{S_i\left({\tau }_{k+1}\right)}{k}$$

(37)

According to the Chernoff–Hoeffding Bound (Fact 2):

$$\mathbb{P}\left(\overline{E_i^{\mu }\left({\tau }_{k+1}\right)}\right)=\mathbb{P}\left({\widehat{\mu }}_i\left({\tau }_{k+1}\right)>x_i\right)\le \mathbb{P}\left(\frac{S_i\left({\tau }_{k+1}\right)}{k}>x_i\right)\le e^{-kd\left(x_i,{\mu }_i\right)}$$

(38)

Substitute Equation (38) into Equation (36):

$${\displaystyle \sum _{t=1}^T\mathbb{P}\left(i\left(t\right)=i,\overline{E_i^{\mu }\left(t\right)}\right)}\le {\displaystyle \sum _{k=1}^{T-1}{e}^{-kd\left(x_i,{\mu }_i\right)}}\le 1+\frac{1}{d\left(x_i,{\mu }_i\right)}$$

(39)

Taking $x_i={\mu }_i+{∆}_i/3$, we have by Pinsker’s inequality:

$$d\left(x_i,{\mu }_i\right)\ge 2{\left(x_i-{\mu }_i\right)}^2=\frac{2{\Delta }_i^2}{9}$$

(40)

Substituting Equations (26), (35), and (41) into Equation (22), respectively, yields

$$\begin{array}{ll}\mathbb{E}\left[k_i\left(T\right)\right]& \le 4L_i\left(T\right)\left(e^{64}+4\right)+\frac{4}{{\Delta }_i^2}+L_i\left(T\right)+\frac{1}{2T{\Delta }_i^2}+1+\frac{9}{2{\Delta }_i^2}\\ & \le \frac{72\ln \left(T{\Delta }_i^2\right)}{{\Delta }_i^2}\left(e^{64}+4\right)+\frac{4}{{\Delta }_i^2}+\frac{18\ln \left(T{\Delta }_i^2\right)}{{\Delta }_i^2}+\frac{1}{2{\Delta }_i^2}+1+\frac{9}{2{\Delta }_i^2}\end{array}$$

(41)

Therefore, the expectation of the regret upper bound for network node $i$ is

$${\Delta }_i\mathbb{E}\left[k_i\left(T\right)\right]\le \frac{72\ln \left(T{\Delta }_i^2\right)}{{\Delta }_i}\left(e^{64}+5\right)+{\Delta }_i+\frac{9}{{\Delta }_i}$$

(42)

The value of the above equation decreases with ${∆}_i$ when ${∆}_i\ge e/\sqrt{T}$, and the upper bound of the expected regret is when ${∆}_i\ge e\sqrt{NlnN/T}$ for all network nodes is

$$\mathrm{O}\left(\sqrt{\frac{T\ln N}{N}}+1\right)$$

When ${∆}_i\le e/\sqrt{T}$, the upper bound on total regret is

$$\mathbb{E}\left[\Re \left(T\right)\right]=e\sqrt{NT\ln N}=\mathrm{O}\left(\sqrt{NT\ln N}\right)$$

(43)

□

The proof is complete. This theorem shows that the algorithm proposed in this paper is able to make the system regret value grow sublinearly and eventually converge, which is better than the algorithm with linear growth and can improve the system throughput. The notation and terminology involved in the proof are recapitulated in Table 1.

Table 1. Review of the main symbols.

Symbol	Meaning of Symbol
$k_i\left(t\right)$	The number of times network node $i$has been selected before moment $t$
$S_i\left(t\right)$	The number of times network node $i$ has gained 1 after being selected before moment $t$
$F_i\left(t\right)$	The number of times network node $i$ has gained 0 after being selected before moment $t$
$i\left(t\right)$	Index of the selected network node at moment $t$
$r_{i\left(t\right)}$	Benefit of network node $i$ after being selected at moment $t$
${\mu }_i$	Expected benefit of network node $i$
${∆}_i$	${\mu }_1-{\mu }_i$
$x_i,y_i$	Real numbers satisfying ${\mu }_i<x_i<y_i<{\mu }_1$
$d\left(x_i,y_i\right)$	The KL dispersion between the Bernoulli distributions of $x_i$ and $y_i$
$T$	Total number of time slots
$\tau$	Variables in the definition of Ft that depend on the value taken at time t
$L_i\left(T\right)$	$lnT/d(x_i,y_i)$
${\theta }_i\left(t\right)$	Sample values sampled by the algorithm in the posterior distribution of network node i at moment t
${\widehat{\mu }}_i\left(t\right)$	The mean value of the benefit of network node $i$ at time $t$
$E_i^{\mu }\left(t\right)$	Event ${\widehat{\mu }}_i\left(t\right)\le x_i$
$E_i^{\theta }\left(t\right)$	Event ${\theta }_i\left(t\right)\le y_i$
${\mathcal{F}}_t$	Sequence of historical strategy information up to moment $t$
$p_{i,t}$	$\mathbb{P}\left({\theta }_1\left(t\right)>y_i|{\mathcal{F}}_{t-1}\right)$
$\mathbb{P}(·)$	Probability of an event
$\mathbb{E}(·)$	Expectation calculus
$\mathbb{I}\left(·\right)$	Indicator function
$\mathcal{N}\left(\mu ,{\sigma }^2\right)$	Normal distribution
$Sample\left[·\right]$	Sampling the distribution

4.3. Algorithm Description and Procedure

Based on the above analysis, the algorithm based on the MAB model always selects the network node access corresponding to the maximum value of the current time-slot index, so that the algorithm proceeds along the direction of minimizing the total regret value. Users can explore the node with the optimal network state in a short time by using the algorithm proposed in this paper under the condition of an unknown network state. The steps of the algorithm are as follows (Algorithm 1):

Algorithm 1. Dynamic variance sampling algorithm flow.

for $t=\mathrm{1,2},\cdots ,T$ do         for each node $i=1,\dots ,N$, sample ${\theta }_i\left(t\right)$ independently from the $\mathcal{N}\left({\widehat{\mu }}_i\left(t\right),\frac{1}{k_i\left(t\right)+1}\right)$ distribution         select node: $i\left(t\right)={argmax}_i{\theta }_i\left(t\right)$         observe reward: $r_{i\left(t\right)}$         update selected times: $k_{i\left(t\right)}\left(t+1\right)=k_{i\left(t\right)}\left(t\right)+1$         update mean benefit: ${\mu }_{i\left(t\right)}(t+1)=\frac{{\mu }_{i\left(t\right)}\left(t\right)\times k_{i\left(t\right)}\left(t\right)+r_{i\left(t\right)}}{k_{i\left(t\right)}+1}$ end for

5. Performance and Evaluation

In this paper, we consider the problem of how users choose the optimal network node access under an unknown network environment, and adopt a dynamic variance sampling algorithm to explore and learn the unknown network environment, and choose the optimal network node access based on historical experience prediction. This subsection simulates and analyzes the algorithm from the perspective of its performance parameters by establishing a simulation scenario, and compares its performance with the traditional ε-greedy algorithm, UCB algorithm, and Thompson sampling algorithm, respectively.

5.1. Simulation Settings

According to the relevant formulas in the channel model, communication model, and revenue model, the meanings of the relevant symbols and the simulation parameter settings are shown in Table 2.

Table 2. Meanings of symbols and simulation parameter settings.

Symbol	Meaning of Symbol	Value
$d$	Satellite orbital altitude	$600 \mathrm{k}\mathrm{m}$
$l$	Horizontal distance between satellite beam center and user	$0$
${\beta }_{\mathrm{d}\mathrm{B}}$	Satellite channel gain caused by rain attenuation	$\mathrm{l}\mathrm{n}\left({\beta }_{dB}\right)~\mathcal{N}(-3.125,1.6)$
$G$	Maximum gain of satellite antenna	$52 \mathrm{d}\mathrm{B}$
${\alpha }_{3\mathrm{d}\mathrm{B}}$	3 dB angle of satellite beam	$0.4$
$p_{SAT}$	Satellite downlink power	$120 \mathrm{d}\mathrm{B}\mathrm{W}$
$U_d$	Horizontal distance from UAV to user	$0~400 \mathrm{m}$
$U_h$	The height of the drone	$200~300 \mathrm{m}$
$K$	Rician coefficient of UAV small-scale fading	$10 \mathrm{d}\mathrm{B}$
$p_{UAV}$	Downlink power of UAV	$3 \mathrm{d}\mathrm{B}\mathrm{W}$
$C_0$	Channel power gain with a reference distance of 1 m	$40 \mathrm{d}\mathrm{B}$
$r$	Distance between base station and user	$0~400 \mathrm{m}$
$p_{BS}$	Downlink power of base station	$20 \mathrm{d}\mathrm{B}\mathrm{W}$
$g_0$	Small-scale fading of base station	$g_0~Nakagami-m(2,1)$
${\delta }_{SAT},{\delta }_{UAV},{\delta }_{BS}$	Noise power received by users	$1 \mathrm{d}\mathrm{B}\mathrm{W}$
$R_{upper}$	Upper bound of communication rate	$100$

The parameter settings related to the simulation scenario are shown in Table 3.

Table 3. Scene parameter settings.

Parameter	Value
Scene radius	$400 \mathrm{m}$
Number of satellites	$1$
Number of base stations	$2$
Number of UAVs	$5$
Angular velocity of UAV	$5°/\mathrm{s}$
Flight radius of UAV	$150 \mathrm{m}$
User moving radius	$200 \mathrm{m}$
User angular velocity	$0.5°/\mathrm{s}$

The scene schematic is shown in Figure 2. The reference coordinate system is the polar coordinate system with the center of the scene as the coordinate origin, the satellite is located at the origin of the coordinate system (0, 0), the coordinates of the flight centers of the UAVs are set to be $\left(250, 0\right), \left(250, \pi /2\right), \left(250, \pi \right), \left(250,-\pi /2\right), (0, 0)$, the flight altitudes are set to be $200, 225, 250, 275, 300$, respectively, the initial positions are all set to be on the positive half-axis of the x-axis with the respective flight centers as the origin, and the positions of the base stations are set to be $\left(200, 3\pi /2\right), \left(200, -\pi /2\right)$, respectively. The coordinates of the user’s motion center are (0, 0) and the initial position is set to (200, 0). The total time slot of the simulation is set to be 1000, and in order to overcome the randomness of the network environment, each group of experiments is repeated 100 times.

Figure 2. Schematic diagram of the simulation scene.

5.2. Results and Analysis

The experiment compares the dynamic variance sampling algorithm proposed in this paper with current mainstream algorithms (ε-greedy [18], UCB, and Thompson sampling, etc.) from different perspectives, ultimately verifying the superiority of this algorithm. As can be seen from Figure 3, the cumulative regret value of the algorithm proposed in this paper grows in an approximately logarithmic relationship with the time slot and ultimately converges, which confirms the deduction of Theorem 1.

Figure 3. Cumulative regret curve of different algorithms.

From Figure 4, it can be observed that as the number of time slots increases, all four algorithms are capable of gradually reducing the average regret value towards stability. However, the dynamic variance sampling algorithm proposed in this paper has a faster convergence rate compared to the other three algorithms, and the average regret value approaches closer to 0.

Figure 4. Average regret curve of different algorithms.

Figure 5 presents the comparison results of the average throughput for the four algorithms. It can be seen that the average throughput of the algorithm proposed in this paper is significantly higher than that of the other algorithms. When the number of time slots is sufficiently large, the average throughput approaches the ideal value.

Figure 5. Average throughput curve of different algorithms.

Figure 6 illustrates the mean throughput of the four algorithms following 1000 iterations. Analysis of the figure reveals that, within the simulated environment described, the mean throughput of the algorithm introduced in this study has experienced enhancements of 7.63%, 11.96%, and 3.13% relative to the ε-greedy algorithm, Upper Confidence Bound (UCB) algorithm, and Thompson sampling algorithm, respectively.

Figure 6. Average throughput after 1000 iterations.

6. Conclusions

This study, predicated on the MAB model, explores the dynamic selection of and access to network nodes by users within an integrated space–air–ground emergency communication network, given the condition of uncertain node network states. Comparative simulation experiments were conducted across various online learning algorithms. The ε-greedy algorithm, rooted in a greedy heuristic, prioritizes immediate gains; the Upper Confidence Bound (UCB) algorithm focuses on static assessments and lacks consideration of the interplay between sequential strategies, leading to reduced adaptability; the Thompson sampling algorithm, conversely, offers more consistent outcomes. The simulation outcomes underscore that the algorithm introduced in this paper adeptly harmonizes the interplay between exploration and exploitation inherent in network node states. In scenarios where the network node state information is obfuscated, the algorithm proposed herein markedly diminishes the learning regret, expedites the convergence pace, and augments system throughput. Nonetheless, the current research is primarily concentrated on the network node access mechanism for individual users, and further investigation is warranted to address the network node access mechanisms for multiple users within the context of an integrated space–air–ground emergency communication network.