Bandit Algorithm Driven by a Classical Random Walk and a Quantum Walk

Abstract

Quantum walks (QWs) have a property that classical random walks (RWs) do not possess—the coexistence of linear spreading and localization—and this property is utilized to implement various kinds of applications. This paper proposes RW- and QW-based algorithms for multi-armed-bandit (MAB) problems. We show that, under some settings, the QW-based model realizes higher performance than the corresponding RW-based one by associating the two operations that make MAB problems difficult—exploration and exploitation—with these two behaviors of QWs.

1. Introduction

Random walk (RW) is one of the most ubiquitous stochastic processes and is employed for both mathematical analyses and applications, such as describing real-world phenomena and constructing various algorithms. Meanwhile, along with the increasing interest in quantum mechanics from both theoretical and applied perspectives, the quantum counterpart of an RW, known as a quantum walk (QW), is also attracting attention [1,2,3,4]. A QW includes the effects of quantum superposition or time evolution. In classical RWs, a random walker (RWer) selects in which direction to go probabilistically at each time step, and thus, one can track where the RWer is at any time step. On the other hand, in QWs, one cannot tell where a quantum walker (QWer) exists during the time evolution, and the location is determined only after conducting the measurement.

QWs have a property that classical RWs do not possess: the coexistence of linear spreading and localization [5,6]. As a result, QWs show probability distributions that are totally different from those of random walks, which weakly converge to normal distributions. The former behavior, linear spreading, means that the standard deviation of the probability distribution of measurement of quantum walkers (QWers) grows in proportion to the run time t. In the case of discrete-time RWs on a one-dimensional lattice $\mathbb{Z}$, denoting the random variable of the position where a walker is measured at time $t\in {\mathbb{N}}_0=\mathbb{N}\cup \left\{0\right\}$ by $X_t^{\left(\mathrm{RW}\right)}$, then the standard deviation is $\mathbb{D}\left[X_t^{\left(\mathrm{RW}\right)}\right]=\mathrm{O}\left(\sqrt{t}\right)$. On the other hand, in discrete-time QWs on $\mathbb{Z}$, the standard deviation of a walker’s position at time t is $\mathbb{D}\left[X_t^{\left(\mathrm{QW}\right)}\right]=\mathrm{O}\left(t\right)$, and thus, discrete-time QWs outperform RWs in terms of the propagation velocity [7]. The latter behavior, localization, implies that the probability is distributed at a particular position no matter how long the walk runs. In the classical RWs, the probability distribution becomes flat despite keeping a bell-shaped curve; that is, localization is not observed.

QWs were first introduced in the field of quantum information theory [7,8,9]. The idea of weak convergence, which is frequently used in probability theory, was introduced to show the properties of QWs [10,11], and since then, quantum walks have been actively studied from both fundamental and applied perspectives. In fundamental fields, there have been many attempts to analyze these evolution models mathematically [6,12,13,14,15,16,17,18,19,20,21,22] due to varying behavior of QWs depending on the conditions or settings of time and space. In applied fields, their unique behavior is useful for implementing quantum structures or quantum analogs of existing models; therefore, various QW-based models have been considered for subjects such as time-series analysis [23], topological insulators [24,25], radioactive waste reduction [26,27], and optics [28,29]. In addition, the contribution to quantum information technology is becoming more prominent these days. QWs have been applied not only to the principle of technologies such as quantum search, quantum teleportation [30,31], and quantum key distribution [32] but also to the implementation of quantum gates themselves [33,34].

Throughout the extensive research conducted, considerable attention has also been devoted to the models of QWs themselves. Initially, QW models were introduced in the form of coined QWs, wherein the time evolution of walkers is considered through the utilization of unitary matrices called coin matrices, analogous to transitions in RWs [9]. According to the No-Go Lemma [35,36], we need the additional subspace, namely the coin space to each vertex, to construct a non-trivial unitary time evolution on the cycle, introduced later. This model remains one of the most intuitive models and is widely studied or applied even in contemporary times. On the other hand, depending on the research interests, some literature also explores QW models that do not incorporate coins [37,38,39], which represent a generalization of quantum cellular automata [35]. The coinless QW models seem to achieve reduced computational costs, but it is shown that such QW models are unitarily equivalent to coined QW models [40]. In this paper, we focus on using the coined QW models because adopting the coined QWs is reasonable when conducting a comparison with RWs. In the following, when referring to QWs, it indicates the coined QWs.

This paper proposes new solution schemes for multi-armed bandit (MAB) problems [41] using RWs and QWs. In the MAB problems, we consider a situation where there are multiple slot machines in an environment, each gives a reward with a probability allocated to it, and an agent iterates the selection of slot machines and probabilistic gain of the rewards and tries to maximize the total reward. Initially, the agent has no information about the probability of giving rewards, especially which slot machine has the maximum probability, which we call the best slot machine. Thus, it is required to accumulate such information through a certain number of selections, an action which we call exploration. On the other hand, too much exploration will use up the opportunities for selecting the better slot machines that have already been found; that is, it is also necessary to spend some rounds to bet on slot machines that are reliable based on the information obtained, which we call exploitation. The difficulty of MAB problems occurs under the balance between these two operations, known as the exploration–exploitation trade-off [42].

One of the purposes of this study is to show that, by utilizing QWs, we can construct an algorithm to solve the MAB problem. This algorithm can outperform models implemented by RWs in terms of the total rewards under some settings. Our idea to realize this is to address this dilemma derived from the exploration–exploitation trade-off by utilizing a unique property of QWs, i.e., the coexistence of linear spreading and localization. More precisely, we combine exploration with linear spreading, and exploitation with localization, as shown in Figure 1. By utilizing linear spreading, we intend to cover the whole environment and prevent us from missing some slot machines. In addition, by applying localization, we intend to mark slot machines that should be recommended with a high probability distribution. This paper introduces a QW-based algorithm for MAB problems, which realizes these combinations. Our study focuses on three-state site-dependent QWs on a cycle. The behavior of these walks corresponds to that of lazy random walks, with walkers moving clockwise, anti-clockwise, or staying in place in superposition. Giving three states to QWs enables us to obtain a high existence probability at the initial position of a QWer. In addition, site-dependent coin matrices make it possible to trap or dam the QWer on certain vertices. By taking advantage of these, we attempt to bring about a high probability on a vertex whose slot machine should be recommended. To facilitate a clear comparison, we also construct an RW-based model wherein lazy RWs occur on cycles, and the transition probabilities depend on the position from which the walkers depart. While the QW-based model possesses the coexistence of linear spreading and localization, the RW-one does not; while the results depend on the specifics of the MAB problem, our study reveals that, for certain settings, the different properties of QWs and RWs also lead to a significant difference in total rewards between both algorithms.

The rest of this paper is organized as follows. First, in Section 2, we present an algorithm for MAB problems based on the RW, which is more intuitive than the QW-based one. Then, in Section 3, we introduce a system of discrete-time quantum walks on a cycle and the QW-based algorithm for MAB problems. In Section 4, we show some results for numerical simulations of the RW- and QW-based models and compare the performance between the two models. Section 5 concludes this paper and discusses the future possibilities of our work.

2. Random-Walk-Based Model for MAB Problem

This section presents an MAB algorithm implemented using a discrete-time random walk (RW) on cycles. The RW model presented in this paper describes the walkers that can stay at the same position, which we often call a lazy random walk. First, we present the mathematical system of the lazy RW, and then we construct the MAB algorithm based on it.

2.1. Random Walk on Cycles

Assume that cycle ${\mathcal{C}}_N$ is composed of N vertices and edges. Here vertices are labeled by set $V_N:=\{0,1,\cdots ,N-1\}$, and the label is ordered clockwise. Thus, the set of edges is given by $E_N:=\left\{\{x,x+1\}\mid x\in V_N\right\}$, where one applies addition and subtraction modulo N to $V_N$; i.e., $(N-1)+1\equiv 0$ and $0-1\equiv N-1$. In other words, $V_N$ is isometric to $\mathbb{Z}/N\mathbb{Z}$.

We assume that the position of a walker is determined as follows:

• A walker initially exists at position $s\in V_N$.
• At each time step, a walker at position x moves one unit clockwise with probability $q\left(x\right)$, moves one unit anti-clockwise with probability $q\left(x\right)$ or stays at the current position with probability $1-2q\left(x\right)$.

Here the probabilities of moving clockwise and anti-clockwise are equal to each other in this paper. This is due to correspondence with the setting of the QW presented later; therein, the setting of the initial state of the QW gives symmetric probability distributions when the coin matrix is homogeneous for the space. Note that $q\left(x\right)$ should satisfy $0\le q\left(x\right)\le 1/2$ under this condition, and this setting is equivalent to the simple RW on cycles in the case of $q\left(x\right)=1/2$ for all $x\in V_N$.

Such an RW is mathematically constructed as follows. For ${\mathbb{N}}_0=\{0,1,\cdots \}$, let ${\left\{X_t\right\}}_{t\in {\mathbb{N}}_0}$ be the sequence of random variables that represent the position of a walker at time step t. The next position $X_{t+1}$ depends on the current one $X_t$, and the conditional probability is determined as follows:

$$\begin{array}{cc}\hfill & \mathbb{P}(X_{t+1}=x+1|X_t=x)=\mathbb{P}(X_{t+1}=x-1|X_t=x)=q\left(x\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & \mathbb{P}(X_{t+1}=x|X_t=x)=1-2q\left(x\right)\hfill \end{array}$$

(2)

with $x\in V_N$. We recall that $V_N\simeq \mathbb{Z}/N\mathbb{Z}$; the equation above includes

$$\begin{array}{cc}\hfill & \mathbb{P}(X_{t+1}=N-1|X_t=0)=q\left(0\right),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & \mathbb{P}(X_{t+1}=0|X_t=N-1)=q(N-1).\hfill \end{array}$$

(4)

Here we denote the probability that a walker is at position x at time step t by ${\nu }^{\left(t\right)}\left(x\right)$:

$$\begin{array}{c}\hfill {\nu }^{\left(t\right)}\left(x\right)=\mathbb{P}(X_t=x).\end{array}$$

(5)

Then, the relation

$$\begin{array}{cc}\hfill & {\nu }^{\left(0\right)}\left(x\right)={\delta }_s\left(x\right)\hfill \end{array}$$

(6)

holds, where ${\delta }_{x^{\prime }}\left(x\right)$ is the delta function: for $x^{\prime }\in V_N$,

$$\begin{array}{c}\hfill {\delta }_{x^{\prime }}\left(x\right)=\left\{\begin{array}{cc}1& (x=x^{\prime })\hfill \\ 0& \left(\mathrm{otherwise}\right)\hfill \end{array}\right..\end{array}$$

(7)

Moreover, by Equations (1) and (2), ${\nu }^{\left(t\right)}\left(x\right)$ varies as follows:

$$\begin{array}{cc}\hfill & {\nu }^{(t+1)}\left(x\right)=q(x+1){\nu }^{\left(t\right)}(x+1)+(1-2q\left(x\right)){\nu }^{\left(t\right)}\left(x\right)+q(x-1){\nu }^{\left(t\right)}(x-1).\hfill \end{array}$$

(8)

2.2. Random-Walk-Based Algorithm

We consider an N-armed bandit problem with cycle ${\mathcal{C}}_N$; each vertex $x\in V_N$ is given a slot machine that gives a reward with the probability $p\left(x\right)$. In the following, each slot machine is identified by the same label as the corresponding slot machine; for example, we call the slot machine on the vertex x slot machine x. In addition, we call probability $p\left(x\right)$ the success probability of slot machine x. Moreover, we denote the slot machine with the best success probability in $V_N$ by $x^{\ast }$; that is,

$$\begin{array}{c}\hfill x^{\ast }=\underset{x\in V_N}{arg\; max}p\left(x\right),\end{array}$$

(9)

and we call it the best slot machine.

The principle consists of the following four steps: [STEP 0] initializing the quantum walk settings, [STEP 1] running random walks, [STEP 2] playing the selected slot machine, and [STEP 3] updating the quantum walk settings. After finishing [STEP 3], the process returns to [STEP 1]. We call the series of the last three steps ([STEP 1–3], shown in Figure 2) a decision, and decisions are iterated J times over a run. Here, we use the following notations:

• $s_j\in V_N$: Initial position of random walk in the j-th decision.
• $q_j\left(x\right)\in [0,1/2]$: Clockwise-transition probability and anti-clockwise-transition probability in the j-th decision.
• ${\widehat{x}}_j\in V_N$: Vertex (slot machine) measured in the j-th decision.
• ${\widehat{r}}_j\in \{0,1\}$: Reward on the j-th decision. This value is probabilistically determined by the Bernoulli distribution $\mathrm{Ber}\left(p\left({\widehat{x}}_j\right)\right)$; that is,

  $$\begin{array}{c}\hfill {\widehat{r}}_j:=\left\{\begin{array}{cc}1\hfill & (\mathrm{with}\mathrm{prob}.p\left({\widehat{x}}_j\right))\hfill \\ 0\hfill & (\mathrm{with}\mathrm{prob}.1-p\left({\widehat{x}}_j\right))\hfill \end{array}\right..\end{array}$$

  (10)
• $H_j\left(x\right)$: Number of decisions where the slot machine x is selected until the j-th decision.
• $L_j\left(x\right)$: Number of decisions where the slot machine x gives the reward until the j-th decision.
• ${\widehat{p}}_j\left(x\right)$: Empirical probability that the slot machine x gives the reward on the j-th decision:

  $$\begin{array}{c}\hfill {\widehat{p}}_j\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{L_j\left(x\right)}{H_j\left(x\right)}}& (H_j\left(x\right)\ne 0)\hfill \\ 0& (H_j\left(x\right)=0)\hfill \end{array}\right..\end{array}$$

  (11)

[STEP 0] RW-setting initialization

For the first decision, the settings of the random walk are determined as follows:

• Initial position $s_1$: Probabilistically determined by the uniform distribution on $V_N$.
• Transition probability: $q_1\left(x\right)=q^{\circ }\in [0,1/2]$ for all $x\in V_N$.

After finishing this step, the process iterates the following three steps.

[STEP 1] Random walk

Random walks are run over T time steps with the initial position $s_j$ and transition probability $q_j\left(x\right)$, and the value ${\widehat{x}}_j\in V_N$ is obtained following probability distribution ${\nu }_j^{\left(T\right)}\left(x\right)$.

[STEP 2] Slot machine play

The slot machine ${\widehat{x}}_j\in V_N$ obtained at [STEP 1] is played. Then, the reward $({\widehat{r}}_j=1)$ is obtained with probability $p\left({\widehat{x}}_j\right)$.

Here H- and L-values are updated. First, the H-value on ${\widehat{x}}_j$ is incremented:

$$\begin{array}{c}\hfill H_j\left({\widehat{x}}_j\right)=H_{j-1}\left({\widehat{x}}_j\right)+1.\end{array}$$

(12)

If ${\widehat{r}}_j=1$, the L-value on ${\widehat{x}}_j$ is also incremented (otherwise, the value is maintained):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_j\left({\widehat{x}}_j\right)=\left\{\begin{array}{cc}{L}_{j-1}\left({\widehat{x}}_j\right)+1\hfill & (\mathrm{with}\mathrm{prob}.p\left({\widehat{x}}_j\right))\hfill \\ L_{j-1}\left({\widehat{x}}_j\right)\hfill & (\mathrm{with}\mathrm{prob}.1-p\left({\widehat{x}}_j\right))\hfill \end{array}\right..\end{array}\end{array}$$

(13)

For $x\ne {\widehat{x}}_j$, the H- and L-values are maintained:

$$\begin{array}{cc}\hfill H_j\left(x\right)& =H_{j-1}\left(x\right),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill L_j\left(x\right)& =L_{j-1}\left(x\right).\hfill \end{array}$$

(15)

Based on that, ${\widehat{p}}_j\left(x\right)$s are updated.

[STEP 3] RW-setting adjustment

Using the new ${\widehat{p}}_j\left(x\right)$s, the settings of quantum walks are updated for the next decision. The new initial position is defined as

$$\begin{array}{c}\hfill s_{j+1}=\underset{x\in V_N}{arg\; max}{\widehat{p}}_j\left(x\right).\end{array}$$

(16)

Moreover, the new transition probabilities are determined as

$$\begin{array}{c}\hfill q_{j+1}\left(x\right)=q^{\circ }exp(-a·{\widehat{p}}_j{\left(x\right)}^b)\end{array}$$

(17)

where $a,b\ge 1$, and $q^{\circ }$ are defined in [STEP 0]. Note that the q-value monotonically decreases about the empirical success probability; that is, if ${\widehat{p}}_j\left(x\right)$ is larger, then $q_{j+1}\left(x\right)$ is smaller. By setting the new initial state and q-value in these manners, we aim at confining walkers to the desired position while concurrently affording them opportunities to depart when the current decision is uncertain. The parameters a and b control the strength of the effect by ${\widehat{p}}_j\left(x\right)$; the details are given in Appendix A.

After this step, the process returns to [STEP 1].

3. Quantum-Walk-Based Model for MAB Problem

This section presents the MAB algorithm implemented by the discrete-time quantum walk (QW) on cycles. The difference between QW and RW lies in whether one handles the quantum superposition of states pertaining to the walker’s positions. Herein, the transition of walkers at each time step is also superposed; that is, it is uncertain even after the time step which transition occurs: moving clockwise, anti-clockwise, or staying in place. We introduce probability amplitude vectors and coin matrices in the QW to describe the quantum superposition and its time evolution.

The QW model employed in our study is a three-state QW on a cycle, which can be naturally reduced to a finite space from the one-dimensional lattice model [5]; see, for example, [18,43,44]. First, we explain the definition of our QW in detail, and then we present the MAB algorithm based on it.

3.1. Quantum Walk on Cycles

Assume that cycle ${\mathcal{C}}_N$ is constructed in the same manner as in Section 2; that is, it is the graph established by the set of vertices $V_N:=\{0,1,\cdots ,N-1\}$ and edges $E_N:=\left\{\{x,x+1\}\mid x\in V_N\right\}$, where one applies addition and subtraction modulo N to $V_N$.

The space of the probability amplitude vectors driving our QW is defined in a compound Hilbert space consisting of the position Hilbert space ${\mathcal{H}}_{\mathrm{P}}$ and the coin Hilbert space ${\mathcal{H}}_{\mathrm{C}}$. The position Hilbert space ${\mathcal{H}}_{\mathrm{P}}$ is spanned by the unit vectors corresponding to the vertices on ${\mathcal{C}}_N$; i.e., ${\mathcal{H}}_{\mathrm{P}}=span\left\{|x\rangle \right|x\in V_N\}$. Here we require them to be mutually orthogonal, which is equivalent to satisfying the relation $\langle y|x\rangle ={\delta }_y\left(x\right)$ for any $x,y\in V_N$, where ${\delta }_y$ is the delta function defined by Equation (7). Then, ${\mathcal{H}}_{\mathrm{P}}\simeq {\mathbb{C}}^N$ holds.

The coin Hilbert space ${\mathcal{H}}_{\mathrm{C}}$ pertains to the internal state of walkers. In this model, we assume that there exist three internal states: clockwise (+), anti-clockwise (−), and staying $\left(\mathrm{O}\right)$. We define the three-dimensional unit vectors corresponding to them as $|-\rangle ={\left[100\right]}^{\mathsf{T}}$, $|\mathrm{O}\rangle ={\left[010\right]}^{\mathsf{T}}$, and $|+\rangle ={\left[001\right]}^{\mathsf{T}}$, where a superscript $\mathsf{T}$ on a matrix represents its transpose, and construct the coin Hilbert space as ${\mathcal{H}}_{\mathrm{C}}=span\{|-\rangle ,|\mathrm{O}\rangle ,|+\rangle \}$. Here you see that ${\mathcal{H}}_{\mathrm{C}}={\mathbb{C}}^3$. Based on ${\mathcal{H}}_{\mathrm{P}}$ and ${\mathcal{H}}_{\mathrm{C}}$, the whole system is described by

$$\begin{array}{c}\hfill {\mathcal{H}}_{\mathrm{P}\mathrm{C}}={\mathcal{H}}_{\mathrm{P}}\otimes {\mathcal{H}}_{\mathrm{C}}=span\{|x\rangle \otimes |\epsilon \rangle |x\in V_N,\epsilon \in \{\pm ,\mathrm{O}\}\}.\end{array}$$

(18)

Then the total state of our QW at time $t\in {\mathbb{N}}_0$ is represented as follows: there exists $|{\psi }^{\left(t\right)}\left(x\right)\rangle \in {\mathbb{C}}^3$ for each $x\in V_N$ such that

$$\begin{array}{c}\hfill |{\Psi }^{\left(t\right)}\rangle =\sum _{x\in V_N}|x\rangle \otimes |{\psi }^{\left(t\right)}\left(x\right)\rangle \in {\mathcal{H}}_{\mathrm{P}\mathrm{C}}.\end{array}$$

(19)

Here, $t\in {\mathbb{N}}_0$ represents time step of QWs, and $|{\psi }^{\left(t\right)}\left(x\right)\rangle \in {\mathbb{C}}^3$ is called the probability amplitude vector at position $x\in V_N$ at run time t. We set the initial state as

$$\begin{array}{c}\hfill |{\Psi }^{\left(0\right)}\rangle =|\Phi \rangle :=|s\rangle \otimes |\phi \rangle ,\end{array}$$

(20)

where $s\in V_N$, and $|\phi \rangle \in {\mathbb{C}}^3$ is a constant vector with $\parallel \phi \parallel =1$. In this paper, we fix $|\phi \rangle$ to $|\mathrm{O}\rangle$, realizing a symmetric probability distribution about the initial position when the coin matrix defined later is homogeneous for positions.

Now, we introduce the time evolution of $|{\Psi }^{\left(t\right)}\rangle$ by

$$\begin{array}{c}\hfill |{\Psi }^{(t+1)}\rangle =U|{\Psi }^{\left(t\right)}\rangle .\end{array}$$

(21)

Here U is the unitary operator, referred to as the time evolution operator, and is composed of shift operator S and coin operator C:

$$\begin{array}{c}\hfill U=SC,\end{array}$$

(22)

and S and C are given by

$$\begin{array}{c}\hfill S=S^{†}\otimes |-\rangle \langle -|+I_N\otimes |\mathrm{O}\rangle \langle \mathrm{O}|+S\otimes |+\rangle \langle +|\end{array}$$

(23)

and

$$\begin{array}{c}\hfill C=\sum _{x\in V_N}\left(|x\rangle \langle x|\otimes C\left(x\right)\right).\end{array}$$

(24)

Here S is defined as

$$\begin{array}{c}\hfill S=\sum _{x\in V_N}|x+1\rangle \langle x|\end{array}$$

(25)

and represents the clockwise transition, and then

$$\begin{array}{c}\hfill S^{†}=\sum _{x\in V_N}|x-1\rangle \langle x|\end{array}$$

(26)

indicates the anti-clockwise transition. The identity matrix $I_N$ corresponds to staying in place. Here, note that $N\equiv 0$ on $V_N(\simeq \mathbb{Z}/N\mathbb{Z})$; that is, for example, in the case of $N=4$,

$$\begin{array}{c}\hfill S=|1\rangle \langle 0|+|2\rangle \langle 1|+|3\rangle \langle 2|+|0\rangle \langle 3|\end{array}$$

(27)

and

$$\begin{array}{c}\hfill S^{†}=|3\rangle \langle 0|+|0\rangle \langle 1|+|1\rangle \langle 2|+|2\rangle \langle 3|.\end{array}$$

(28)

$C\left(x\right)$ is a unitary matrix called a coin matrix, which is defined as follows:

$$\begin{array}{c}\hfill C\left(x\right)=\left[\begin{array}{ccc}{\displaystyle -\frac{1+cos\theta \left(x\right)}{2}}& {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}& {\displaystyle \frac{1-cos\theta \left(x\right)}{2}}\\ {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}& cos\theta \left(x\right)& {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}\\ {\displaystyle \frac{1-cos\theta \left(x\right)}{2}}& {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}& {\displaystyle -\frac{1+cos\theta \left(x\right)}{2}}\end{array}\right]\end{array}$$

(29)

with $\theta \left(x\right)\in [0,2\pi )$ for all $x\in V_N$. Note that, in the case of $cos\theta \left(x\right)=-1/3$, $C\left(x\right)$ is reduced to the Grover matrix, which is important in quantum searching [45].

Let us explain the equivalent expression for the time evolution operator U, which is useful to understand the dynamics of our QW: by applying the property of the Kronecker product, we have

$$\begin{array}{cc}\hfill & U=\sum _{x\in V_N}\left(|x-1\rangle \langle x|\otimes P\left(x\right)+|x\rangle \langle x|\otimes R\left(x\right)+|x+1\rangle \langle x|\otimes Q\left(x\right)\right),\hfill \end{array}$$

(30)

where

$$\begin{array}{cc}\hfill P\left(x\right)& =|-\rangle \langle -|C\left(x\right)=\left[\begin{array}{ccc}{\displaystyle -\frac{1+cos\theta \left(x\right)}{2}}& {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}& {\displaystyle \frac{1-cos\theta \left(x\right)}{2}}\\ 0& 0& 0\\ 0& 0& 0\end{array}\right],\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill Q\left(x\right)& =|+\rangle \langle +|C\left(x\right)=\left[\begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ {\displaystyle \frac{1-cos\theta \left(x\right)}{2}}& {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}& {\displaystyle -\frac{1+cos\theta \left(x\right)}{2}}\end{array}\right],\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill R\left(x\right)& =|\mathrm{O}\rangle \langle \mathrm{O}|C\left(x\right)=\left[\begin{array}{ccc}0& 0& 0\\ {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}& cos\theta \left(x\right)& {\displaystyle \frac{sin\theta \left(x\right)}{\sqrt{2}}}\\ 0& 0& 0\end{array}\right].\hfill \end{array}$$

(33)

The matrices $P\left(x\right)$, $Q\left(x\right)$, and $R\left(x\right)$ are considered to be the decomposition elements of $C\left(x\right)$; that is, the relation $P\left(x\right)+Q\left(x\right)+R\left(x\right)=C\left(x\right)$ holds. They describe the matrix-valued weight of a clockwise transition, an anti-clockwise transition, and staying in place, respectively, corresponding to the transition probabilities of the RW, as shown in Figure 3.

By Equations (21) and (30), we have

$$\begin{array}{cc}\hfill & |{\psi }^{(t+1)}\left(x\right)\rangle =P(x+1)|{\psi }^{\left(t\right)}(x+1)\rangle +R(x)|{\psi }^{\left(t\right)}\left(x\right)\rangle +Q(x-1)|{\psi }^{\left(t\right)}(x-1)\rangle .\hfill \end{array}$$

(34)

Moreover, from the initial state (20), there exists a 2-dimensional matrix ${\Xi }^{\left(t\right)}\left(x\right)$ such that

$$\begin{array}{c}\hfill |{\psi }^{\left(t\right)}\left(x\right)\rangle ={\Xi }^{\left(t\right)}\left(x\right)|\phi \rangle .\end{array}$$

(35)

Here ${\Xi }^{\left(t\right)}\left(x\right)$ describes the weight of all the possible paths from the origin to the position x at run time t. From Equation (34), the following relation holds:

$$\begin{array}{c}\hfill {\Xi }^{(t+1)}\left(x\right)=P(x+1){\Xi }^{\left(t\right)}(x+1)+R\left(x\right){\Xi }^{\left(t\right)}\left(x\right)+Q(x-1){\Xi }^{\left(t\right)}(x-1).\end{array}$$

(36)

Finally, the measurement probability of the particle at position x at run time t, denoted by ${\mu }^{\left(t\right)}\left(x\right)$, is given by

$$\begin{array}{c}\hfill {\mu }^{\left(t\right)}\left(x\right):={\parallel {\psi }^{\left(t\right)}\left(x\right)\parallel }^2.\end{array}$$

(37)

Setting random variable $X_t$ following the distribution ${\mu }^{\left(t\right)}$, we call $X_t$ the position of a QWer at time t. This definition is based on the Born probability interpretation in quantum mechanics. Note that for any $t\in {\mathbb{N}}_0$, the following is satisfied:

$$\begin{array}{c}\hfill \sum _{x\in V_N}{\mu }^{\left(t\right)}\left(x\right)=\sum _{x\in V_N}{\parallel {\psi }^{\left(t\right)}\left(x\right)\parallel }^2=1.\end{array}$$

(38)

3.2. Quantum-Walk-Based Algorithm

We consider an N-armed bandit problem with cycle ${\mathcal{C}}_N$; each vertex $x\in V_N$ is given a slot machine that gives a reward with the probability $p\left(x\right)$, identically to the RW-based model in Section 3.

The principle is also similar to the RW-based model: First, the QW settings are initialized ([STEP 0]), and then decisions are iterated ([STEP 1–3]) J times over a run. As shown in Figure 4, the QW-based model controls the coin matrix $C\left(x\right)$ by adjusting the value of the parameter $\theta \left(x\right)$ instead of the transition probability $q\left(x\right)$ in the RW-based model. Here we use the following notations:

• $|{\Phi }_j\rangle \in {\mathcal{H}}_{\mathrm{P}\mathrm{C}}$: Initial state of quantum walk on the j-th decision.
• $s_j\in V_N$: Initial position of quantum walk on the j-th decision.
• ${\theta }_j\left(x\right)\in [0,2\pi )$: Parameter of Equation (24) on vertex x on the j-th decision; then the coin matrix there is $C\left(x\right)$.
• ${\widehat{x}}_j\in V_N$: Vertex (slot machine) measured on the j-th decision.
• ${\widehat{r}}_j\in \{0,1\}$: Reward on the j-th decision, which follows the Bernoulli distribution $\mathrm{Ber}\left(p\left({\widehat{x}}_j\right)\right)$:

  $$\begin{array}{c}\hfill {\widehat{r}}_j:=\left\{\begin{array}{cc}1\hfill & (\mathrm{with}\mathrm{prob}.p\left({\widehat{x}}_j\right))\hfill \\ 0\hfill & (\mathrm{with}\mathrm{prob}.1-p\left({\widehat{x}}_j\right))\hfill \end{array}\right..\end{array}$$

  (39)
• $H_j\left(x\right)$: Number of decisions in which slot machine x is selected until the j-th decision.
• $L_j\left(x\right)$: Number of decisions in which slot machine x gives the reward until the j-th decision.
• ${\widehat{p}}_j\left(x\right)$: Empirical probability that slot machine x gives the reward on the j-th decision:

  $$\begin{array}{c}\hfill {\widehat{p}}_j\left(x\right)=\left\{\begin{array}{cc}{\displaystyle \frac{L_j\left(x\right)}{H_j\left(x\right)}}& (H_j\left(x\right)\ne 0)\hfill \\ 0& (H_j\left(x\right)=0)\hfill \end{array}\right..\end{array}$$

  (40)

[STEP 0] QW-setting initialization

For the first decision, the settings of the quantum walk are determined as follows:

• Initial state: $|{\Phi }_1\rangle =|s_1\rangle \otimes |\mathrm{O}\rangle$. Here the initial position $s_1$ is probabilistically determined by the uniform distribution on $V_N$.
• Parameter of coin matrices: ${\theta }_1\left(x\right)={\theta }^{\circ }\in [0,2\pi )$ for all $x\in V_N$.

After finishing this step, the run iterates the following three steps.

[STEP 1] Quantum walk

Quantum walks are run over T time steps with the initial position $s_j$ and the parameter ${\theta }_j\left(x\right)$. After running T steps of time evolution, the QWer is measured to obtain the value ${\widehat{x}}_j\in V_N$ following probability distribution ${\mu }_j^{\left(T\right)}\left(x\right)$.

[STEP 2] Slot machine play

The slot machine ${\widehat{x}}_j\in V_N$ obtained at [STEP 1] is played. Then, the reward (${\widehat{r}}_j=1$) is obtained with probability $p\left({\widehat{x}}_j\right)$.

Here H- and L-values are updated. First, the H-value on ${\widehat{x}}_j$ is incremented:

$$\begin{array}{c}\hfill H_j\left({\widehat{x}}_j\right)=H_{j-1}\left({\widehat{x}}_j\right)+1.\end{array}$$

(41)

If ${\widehat{r}}_j=1$, the L-value on ${\widehat{x}}_j$ is also incremented (otherwise, the value is maintained):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_j\left({\widehat{x}}_j\right)=\left\{\begin{array}{cc}{L}_{j-1}\left({\widehat{x}}_j\right)+1\hfill & (\mathrm{with}\mathrm{prob}.p\left({\widehat{x}}_j\right))\hfill \\ L_{j-1}\left({\widehat{x}}_j\right)\hfill & (\mathrm{with}\mathrm{prob}.1-p\left({\widehat{x}}_j\right))\hfill \end{array}\right..\end{array}\end{array}$$

(42)

For $x\ne {\widehat{x}}_j$, the H- and L-values are maintained:

$$\begin{array}{cc}\hfill H_j\left(x\right)& =H_{j-1}\left(x\right),\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill L_j\left(x\right)& =L_{j-1}\left(x\right).\hfill \end{array}$$

(44)

Based on that, the ${\widehat{p}}_j\left(x\right)$ values are updated.

[STEP 3] QW-setting adjustment

Using the new ${\widehat{p}}_j\left(x\right)$s, the settings of quantum walks for the next decision are updated. The new initial state is defined as

$$\begin{array}{c}\hfill |{\Phi }_{j+1}\rangle =|s_j\rangle \otimes |\mathrm{O}\rangle ,\end{array}$$

(45)

where $s_j$ is the provisionally best machine:

$$\begin{array}{c}\hfill s_j=\underset{x\in V_N}{arg\; max}{\widehat{p}}_j\left(x\right).\end{array}$$

(46)

Moreover, the new parameters of the coin matrices are determined as

$$\begin{array}{c}\hfill {\theta }_{j+1}\left(x\right)={\theta }^{\circ }exp(-a·{\widehat{p}}_j{\left(x\right)}^b)\end{array}$$

(47)

where $a,b\ge 1$, and ${\theta }^{\circ }$ are defined in [STEP 0]. Note that the $\theta$-value is defined similarly to the q-value in the RW-based model; that is, if ${\widehat{p}}_j\left(x\right)$ is larger, then ${\theta }_{j+1}\left(x\right)$ is smaller. When the $\theta$-value at a certain position $x_{\mathrm{L}}\in V_N$ is updated, the difference between $C\left(x_{\mathrm{L}}\right)$ and $C\left(x\right)$ with $x=x_{\mathrm{L}}\pm 1$ emerges, wherein $x_{\mathrm{L}}$ is often called a defect. If a defect exists at the initial position, the coin matrix, depending on the $\theta$-value, controls the strength of localization there. Incidentally, when ${\widehat{p}}_j\left(x\right)$ is large, ${\theta }_{j+1}\left(x\right)$ can be almost 0. If $\theta \left(x\right)$ in Equation (24) is exactly 0, then $C\left(x\right)$ is the identity matrix. This means that, if the initial position of the walker has a coin matrix with $\theta \left(x\right)=0$, the walker is completely trapped there because the internal state is set to be $|\mathrm{O}\rangle$. Thus, a larger empirical success probability indicates a strong localization if it is provisionally best. However, if ${\widehat{p}}_j\left(x\right)$ is not large, this phenomenon is relaxed. In short, this $\theta$-value plays a role corresponding to the q-value in the RW-based model; that is, it confines walkers to the desired position while concurrently affording them opportunities to depart when the current decision is uncertain. Regarding the analysis of $\theta$-values, also see Appendix A.

After this step, the process returns to [STEP 1].

4. Numerical Simulations

In this section, we give and compare simulation results for the RW- and QW-based models. Assume that the RW- and QW-based models are run in parallel K times, respectively, and each run is labeled by the set $\{1,2,\cdots ,K\}$. We indicate that the parameters are in the k-th run by a subscript next to the number of iterations; for example, the reward in the j-th decision in the k-th run is denoted by ${\widehat{r}}_{j,k}$.

As figures-of-merit, we define quantities $M\left(j\right)$, $\rho \left(j\right)$, and $\mathrm{CDR}\left(j\right)$:

$$\begin{array}{cc}\hfill M\left(j\right)& {\displaystyle :=\frac{1}{K}\sum _{k=1}^K\sum _{\ell =1}^j{\widehat{r}}_{\ell ,k},}\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill \rho \left(j\right)& {\displaystyle :=\frac{1}{K}\sum _{k=1}^K\sum _{\ell =1}^j\left(p\left(x^{\ast }\right)-p\left({\widehat{x}}_{\ell ,k}\right)\right),}\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill \mathrm{CDR}\left(j\right)& {\displaystyle :=\frac{1}{K}\sum _{k=1}^K{\delta }_{x^{\ast }}\left({\widehat{x}}_{j,k}\right).}\hfill \end{array}$$

(50)

$M\left(j\right)$ indicates the mean of total rewards until the j-th decision over K runs. The aim of the proposed models is to make $M\left(j\right)$ as large as possible. $\rho \left(j\right)$ is the mean of cumulative regret until the j-th decision over K runs. The cumulative regret is equal to the difference in expectations of total reward between the case where only the best machine is selected until the j-th decision and that of actual selections until then. $\mathrm{CDR}\left(j\right)$ is the correct decision rate of the j-th decision, which is the ratio of the number of runs in selecting the best slot machine to the total number of runs K. Herein, ${\delta }_y$ for $y\in V_N$ is the delta function defined by Equation (7).

The parameter values used for this series of simulations are summarized in

Table 1. Parameter values used for numerical simulation of decision making.

Parameter	Symbol	Value
Number of slot machines	N	32
Number of runs	K	500
Number of decisions for a single run	J	5000
Parameters for the QW-based model	$(a,b,{\theta }^{\circ })$	$(5,6,5\pi /16)$
Parameters for the RW-based model	$(a,b,q^{\circ })$	$(9,6,0.5)$

. The success probabilities of slot machines are given as shown in Figure 5; that is,

$$\begin{array}{c}\hfill p\left(x\right)=\left\{\begin{array}{cc}0.9& (x=14)\hfill \\ 0.1& (x=15)\hfill \\ 0.7& \left(x:\mathrm{even}\mathrm{except}\mathrm{for}14\right)\hfill \\ 0.5& \left(x:\mathrm{odd}\mathrm{except}\mathrm{for}15\right)\hfill \end{array}\right..\end{array}$$

(51)

Herein, the best slot machine is $x^{\ast }=14$. Recall that the agent cannot directly access all information regarding the success probabilities of the slot machines above. The tuples for the QW- and RW-based models are selected as one of the best performers in a certain range of parameters on each model. The details about parameter-dependencies of both models are found in Appendix A.

The blue and orange curves in Figure 6a–c demonstrate the performances of RW- and QW-based models as the variations of the mean of total reward $M\left(j\right)$, the cumulative regret $\rho \left(j\right)$, and the maximum value of CDR over the number T of time steps of walks for single decision-making, respectively. The total reward and the cumulative regret are taken for the final decision $J=5000$. The maximum value of CDR is taken over J decisions; that is,

$$\begin{array}{c}\hfill \max \left(\mathrm{CDR}\right):=\underset{j=1,\cdots ,J}{max}\mathrm{CDR}\left(j\right).\end{array}$$

(52)

For $T\ge 4$, we observe that $M\left(5000\right)$ and $\max \left(\mathrm{CDR}\right)$ of the QW-based model are larger than those of the RW-based model. On the other hand, for the cumulative regret $\rho \left(5000\right)$, the value for the QW-based model is lower than that for the RW-based model. Both results indicate that the performance of the QW-based model is superior to that of the RW-based model. You see the particular difference in the growth of $M\left(5000\right)$ and $\max \left(\mathrm{CDR}\right)$ over the variation of T between the QW- and RW-models; the gradient of the orange curves (QW) in the range of $2\le T\le 8$ in $M\left(5000\right)$ and $\max \left(\mathrm{CDR}\right)$ is larger than that of the blue curve (RW). In addition, the QW-based model has higher suprema of $M\left(5000\right)$ and $\max \left(\mathrm{CDR}\right)$ than the RW-based model. Similar discussions are also made for regret $\rho \left(5000\right)$. In both the RW- and QW-models $\rho \left(5000\right)$ decreases over the variation of T in the range of $2\le T\le 8$, but the gradient of the QW-model is steeper than that of the RW-one. Moreover, the QW-based model has a lower infimum than the RW-based model.

These results depend on a variety of choices; in particular, the casino setting and the parameters a and b. Indeed, the QW-based model shows a faster growth in reward for many of a and b compared to the RW-based model (see Appendix A). However, there are also settings where this relationship is reversed. We have also observed that in the limit of very large a and b, when fine-tuned to a specific casino setting, the performance of both RW and QW can grow even faster than the results shown in Figure 6. We speculate that linear spreading and localization may lose their advantage for certain settings.

You can see the contribution of linear spreading and localization from the behavior of variation of decision-making and probability distributions for making a decision, which is particularly apparent for smaller T. Figure 7 and Figure 8, respectively, show the precise performances of runs of the RW- and QW-based models whose resultant total rewards $M\left(J\right)$ were almost equal to the average value. Herein, the number of time steps T is set to be 8 for each model, and the other parameters are set as in Table 1. Figure 7a,b indicate the relationship between the decision j and the selected slot machine ${\widehat{x}}_j$ for the RW- and QW-based models, respectively. From this figure, you can see that the decision-making in the QW-based model almost converges to $x=x^{\ast }$ near $j=1200$, while that in the RW-based model does so near $j=1400$. This means that exploration in the QW-based model is more successful than that in the RW-based model. Linear spreading makes the probability distribution of QWs wider, whose variance is larger than that of RWs, which results in faster exploration of the QW-based model. Moreover, the behavior of the QW-based model after finding $x=x^{\ast }$ is more stable than that of the RW-based model, which indicates that the QW-based model also realizes more effective exploitation than the RW-based one for this set of parameters.

These behaviors are interpreted by the variations of the probability distributions of the RW(${\nu }_j^{\left(T\right)}$)- and QW(${\mu }_j^{\left(T\right)}$)-based models over decision j shown in Figure 8a,b, respectively. You see that, for smaller j, the probability of QW is more widely distributed than that of RW, although the values are quite small except for a certain position. It is important that the probabilities are distributed in a wider range even if they are quite small because it indicates that the agent has more selections, which is crucial to realize exploration. As a result, the QW-based model can obtain a high measurement probability of the best slot machine at $j=1200$ at the latest, while the RW-based model does at $j=1400$, which corresponds to the convergence of the decision-making shown in Figure 7. After beginning the concentrated investment to $x=x^{\ast }$, the measurement probability of walkers on the QW is almost 1, while the corresponding probability on the RW is around $0.9$, which shows that strong localization occurs on vertex $x^{\ast }$ after finding the slot machine there. This phenomenon contributes to exploitation; as you see in Figure 7, the QW-based model after finding the best slot machine is much more likely to select it than the RW-based one, and we can understand that it comes from the difference in probability distributions between the two models.

5. Conclusions and Discussion

This paper has proposed new solution schemes for multi-armed bandit (MAB) problems using random walks (RWs) and quantum walks (QWs). We demonstrated that we could find parameter regimes where the QW-based model performs better than the RW-based model by addressing the exploration–exploitation dilemma by utilizing a unique property of QWs, i.e., the coexistence of linear spreading and localization. Our idea was to combine exploration with linear spreading and exploitation with localization. By utilizing linear spreading, we expect the QWs to cover the whole environment to prevent it from missing some slot machines. In addition, by applying localization, we expect the QWs to identify the slot machine that should be recommended with a high probability distribution. Indeed, we showed that, under some settings, linear spreading contributes to exploring the environment and quickly finding the best slot machine, and localization contributes to exploiting the best slot machine more frequently.

The positive results obtained in this study open the possibility for further extensions of this approach. First, can we apply this algorithm to the case of multi-agent systems such as competitive or adversarial bandit problems [46,47,48] with some revision? Especially for the QW-based model, we will examine its application to the use of coin matrices implemented by multiple registers or to drive walkers on a torus. Moreover, there are possibilities for constructing application models in the single-agent case. For example, we may construct an evolved version of the QW-based model, including a quantum version of the optimal stopping problem.

Moreover, analyses of our models are also important. The performances obtained in Section 4 should depend on the number of slot machines (i.e., vertices) N, the true success probability $p\left(x\right)$, and the parameter settings $(a,b,q^{\circ })$ or $(a,b,{\theta }^{\circ })$. Obtaining theoretical formulae for the figures-of-merit would be desirable, which would make our results more confident. Specifically, it is not immediately evident, given the parameter configurations outlined in this paper, that the QW-based model outperforms the RW-based model in general. Our investigations have focused solely on pairs $(a,b)$ with small values; in the event that larger values are assigned to either a or b, the RW-based model can exhibit superior performance compared to any QW-based model (assuming the parameters are chosen correctly for a specific casino setting). Indeed, we have confirmed that the RW-based model with tuple $(a,b,q^{\circ })=(50,20,0.5)$ performs as well as the QW-based one with tuple $(a,b,{\theta }^{\circ })=(50,20,29\pi /64)$, and in both results, the mean of total reward is around 4320 even in setting the number of time steps T to 8 (better than the results for small a and b as shown in Figure 6). However, sufficiently addressing the parameter dependency and its interplay with the casino settings is a highly complex problem at this stage. We provide our existing analysis in Appendix A.

Several factors contribute to the complexity of a rigid mathematical treatment of this problem, but the one that should be remarked on is the position-dependency of the coin matrices in the QW-based model. Solving QW-based models with site-dependent coins is very difficult, in general. While some studies have addressed this matter for the case where the coin matrix only on the origin differs from the others [6,22] or that the coin matrices are controlled by the trigonometric function whose input is in proportion to the label position [12,14,21], the generalized case remains an open problem. To conduct a thorough analysis of this model, it is necessary to accumulate analytical results for site-dependent quantum walks over an extended period of time.