A Low-Complexity Algorithm for a Reinforcement Learning-Based Channel Estimator for MIMO Systems

Abstract

This paper proposes a low-complexity algorithm for a reinforcement learning-based channel estimator for multiple-input multiple-output systems. The proposed channel estimator utilizes detected symbols to reduce the channel estimation error. However, the detected data symbols may include errors at the receiver owing to the characteristics of the wireless channels. Thus, the detected data symbols are selectively used as additional pilot symbols. To this end, a Markov decision process (MDP) problem is defined to optimize the selection of the detected data symbols. Subsequently, a reinforcement learning algorithm is developed to solve the MDP problem with computational efficiency. The developed algorithm derives the optimal policy in a closed form by introducing backup samples and data subblocks, to reduce latency and complexity. Simulations are conducted, and the results show that the proposed channel estimator significantly reduces the minimum-mean square error of the channel estimates, thus improving the block error rate compared to the conventional channel estimation.

1. Introduction

Currently, multiple-input multiple-output (MIMO) is an essential technology in wireless communications [1,2,3,4,5,6]. Multiple antennas are easy to implement in wireless systems, and their use significantly increases system reliability and capacity. However, to utilize the advantages of multiple antennas, perfect channel information is required at both the transmitter and receiver. Meeting this necessity is generally impossible because of the characteristics of wireless channels.

Although perfect channel information is unavailable, many studies have been conducted to improve the accuracy of channel estimation [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21]. These investigations were mostly based on the use of pilots whose information is shared by both the transmitter and receiver and employed least-squares and linear minimum-mean square-error (LMMSE) estimations [10,11,12]. This is because the two estimation methods reasonably perform with affordable complexities for wireless systems. However, their performance strongly depends on the number of pilots, which is generally limited in wireless systems because employing several pilots as resources degrades the spectral efficiency.

This limitation can be overcome using data in channel estimation, i.e., conducting data-aided channel estimation [13,14,15,16,17,18,19,20,21]. Its concept is to exploit a detected data symbol as an additional pilot. Because a detected data symbol may have an error, the accuracy of the channel estimation may be degraded by it. An iterative turbo approach is a good method to address this degradation because the improved detection performance achieved using an iterative turbo equalizer also increases the estimation accuracy of a channel [19,20,21,22,23,24,25]. However, the use of this iterative turbo approach is limited in wireless systems because of its inherent high complexity and latency.

Recently, a reinforcement learning (RL) approach was introduced in [26] for data-aided channel estimation. In this approach, a Markov decision process (MDP) problem is described to minimize the estimation error, and an RL algorithm is used to solve the MDP problem. Without an iterative approach, the RL solution resulted in a significant improvement compared to conventional channel estimations. However, this solution is difficult to implement in practical systems because of its considerable complexity and latency in computing the optimal policy. For example, using the approach in [26] to calculate the optimal policy requires all a posteriori probabilities (APPs) in a data block. In addition, its limitation is that the optimal policy is characterized by a specific discounting factor.

In this paper, a low-complexity channel estimator using an RL approach is proposed for MIMO systems. The key concept of this estimator is the selection of the detected data symbols obtained during data detection as additional pilot symbols. To achieve this, an MDP problem is first defined to minimize the channel estimation error where the Q-value function is generalized by a discounting factor. Subsequently, an RL solution is proposed that can be easily implement in wireless systems. To this end, concepts of backup samples and data subblocks are introduced, which significantly reduce the complexity and latency. The main contributions of this study are summarized as follows:

• A data-aided channel estimator is developed to optimize the selection of detected symbols for MIMO systems. An MDP problem is defined for this selection to minimize the mean-square-error (MSE) of the channel estimates. Compared with [26], a discounting factor is introduced in the Q-value function. The discounting factor adjusts the effects of rewards after the current state.
• A low-complexity RL algorithm is proposed. To achieve this efficiently, a data block is separated into multiple data subblocks and the optimal policy for the data subblocks is characterized. In the characterization, only partial soft information obtained from data detection is utilized to reduce the calculation latency. Unlike in [26], the optimal policy is calculated using only this partially obtained information; the remaining rewards are approximated under the assumption of perfect detection. Finally, the optimal policy is obtained using a closed-form expression. Note that the conventional RL algorithm in [26] can be employed after obtaining all soft information in a data block.
• The performance enhancement achieved for MIMO systems using the developed RL algorithm is evaluated. Simulations are conducted, and the results demonstrate that the proposed algorithm significantly reduces the performance degradation of conventional channel estimation. Based on the simulations, the proposed channel estimator using an approximate MDP presents a similar performance to that of the original MDP. In addition, the proposed channel estimator provides robustness in time-varying channels.

The remainder of this paper is organized as follows. Section 2 introduces a signal model including the channel estimation and data detection considered in this study. In Section 3, an MDP problem to select detected data symbols optimally to minimize the channel estimation error is defined. A low-complexity RL algorithm is proposed in Section 4. In Section 5, simulation results are discussed, to demonstrate the effectiveness of the developed algorithm. Finally, conclusions are presented in Section 6.

Notation

Matrices ${\mathbf{0}}_m$ and ${\mathbf{I}}_m$ represent $m\times m$ all-zero and the $m\times m$ identity matrices, respectively. The superscripts ${(·)}^T$ and ${(·)}^H$ denote the transpose and the conjugate transpose, respectively. Operators $\mathbb{E}(·)$ and $\mathbb{P}(·)$ denote the expectation of a random variable and the probability of an event, respectively. Operators $|·|$ and ${\parallel ·\parallel }^2$ denote the cardinality of a set and the norm, respectively. Operators ${(·)}^{-1}$, $\mathrm{Tr}(·)$, and $\mathcal{CN}$ denote the inverse, trace, and complex normal distribution, respectively. Set $\mathbb{C}$ represents a set of complex numbers.

2. Signal Model

This section describes the signal model for a MIMO system. Based on the signal model, the channel estimator and data detector considered in this study are introduced.

2.1. Signal Model

A MIMO system is considered; in it, a transmitter with $N_{\mathrm{t}}$ antennas communicates with a receiver with $N_{\mathrm{r}}$ antennas through a wireless channel. A wireless channel is denoted as $\mathbf{H}\in {\mathbb{C}}^{N_{\mathrm{t}}\times N_{\mathrm{r}}}$, where each channel element $h_{t,r}\in \mathbb{C}$ between the t-th transmitter and r-th receiver is modeled by Rayleigh fading $h_{t,r}\sim \mathcal{CN}\left(0,1\right)$. The transmitter sends a frame consisting of one pilot block and $N_{\mathrm{d}}$ data blocks, as shown in Figure 1. During the pilot transmission, the transmitter sends a pilot symbol ${\mathbf{x}}^{\mathrm{p}}\left[n\right]\in {\mathbb{C}}^{N_{\mathrm{t}}\times 1}$ for $n\in {\mathcal{N}}_p=\{1,\dots ,T_{\mathrm{p}}\}$, where $T_{\mathrm{p}}$ is the pilot length. When the pilot symbol ${\mathbf{x}}^{\mathrm{p}}\left[n\right]$ is transmitted to the receiver, the received symbol ${\mathbf{y}}^{\mathrm{p}}\left[n\right]\in {\mathbb{C}}^{N_{\mathrm{r}}\times 1}$ at time slot n is given as

$$\begin{array}{cc}\hfill {\mathbf{y}}^{\mathrm{p}}\left[n\right]& ={\mathbf{H}}^H{\mathbf{x}}^{\mathrm{p}}\left[n\right]+{\mathbf{z}}^{\mathrm{p}}\left[n\right],\hfill \end{array}$$

(1)

where ${\mathbf{z}}^{\mathrm{p}}\left[n\right]$ is an additive white Gaussian noise (AWGN) at time slot n whose distribution follows $\mathcal{CN}\left({\mathbf{0}}_{N_{\mathrm{r}}},N_0{\mathbf{I}}_{N_{\mathrm{r}}}\right)$. After the pilot transmission is completed, the transmitter sends a data symbol ${\mathbf{x}}^{\mathrm{d}}\left[n\right]\in {\mathbb{C}}^{N_{\mathrm{t}}\times 1}$ for $n\in {\mathcal{N}}_d=\{(d-1)T_{\mathrm{d}}+1,\dots ,d{T}_{\mathrm{d}}\}$, where $T_{\mathrm{d}}$ is the data length. Supposing $\mathcal{X}$ is a constellation set, the data symbol ${\mathbf{x}}^{\mathrm{d}}\left[n\right]\in {\mathcal{X}}^{N_{\mathrm{t}}}$. After the data transmission, the received symbol ${\mathbf{y}}^{\mathrm{d}}\left[n\right]\in {\mathbb{C}}^{N_{\mathrm{r}}\times 1}$ is expressed as

$$\begin{array}{cc}\hfill {\mathbf{y}}^{\mathrm{d}}\left[n\right]& ={\mathbf{H}}^H{\mathbf{x}}^{\mathrm{d}}\left[n\right]+{\mathbf{z}}^{\mathrm{d}}\left[n\right],\hfill \end{array}$$

(2)

where ${\mathbf{z}}^{\mathrm{d}}\left[n\right]$ is also an AWGN at time slot n.

2.2. Channel Estimator and Data Detector

The LMMSE channel estimator is considered in this study because of its satisfactory performance with low complexity. Using the received symbol in (1), the LMMSE channel estimator, $\mathbf{W}\in {\mathbb{C}}^{N_{\mathrm{t}}\times T_{\mathrm{p}}}$, is expressed as follows:

$$\begin{array}{cc}\hfill \widehat{\mathbf{W}}& =\underset{\mathbf{W}}{argmin}\mathbb{E}\left[\parallel \mathbf{W}{\left({\mathbf{y}}_r^{\mathrm{p}}\right)}^H-{\mathbf{h}}_r{\parallel }^2\right]={\left({\mathbf{X}}^{\mathrm{p}}{\left({\mathbf{X}}^{\mathrm{p}}\right)}^H+N_0{\mathbf{I}}_{N_{\mathrm{t}}}\right)}^{-1}{\mathbf{X}}^{\mathrm{p}},\hfill \end{array}$$

(3)

where ${\mathbf{y}}_r^{\mathrm{p}}$ and ${\mathbf{X}}^{\mathrm{p}}$ are sets of the received and pilot symbols and are defined as ${\mathbf{y}}_r^{\mathrm{p}}=[y_r^{\mathrm{p}}\left[1\right],\cdots ,y_r^{\mathrm{p}}\left[T_{\mathrm{p}}\right]]$ and ${\mathbf{X}}^{\mathrm{p}}=[{\mathbf{x}}^{\mathrm{p}}\left[1\right],\cdots ,{\mathbf{x}}^{\mathrm{p}}\left[T_{\mathrm{p}}\right]]$, respectively. Using the channel estimator in (3), a channel estimate is expressed as

$$\begin{array}{cc}\hfill {\widehat{\mathbf{h}}}_r& =\widehat{\mathbf{W}}{\left({\mathbf{y}}_r^{\mathrm{p}}\right)}^H={\left({\mathbf{X}}^{\mathrm{p}}{\left({\mathbf{X}}^{\mathrm{p}}\right)}^H+N_0{\mathbf{I}}_{N_{\mathrm{t}}}\right)}^{-1}{\mathbf{X}}^{\mathrm{p}}{\left({\mathbf{y}}_r^{\mathrm{p}}\right)}^H,\hfill \end{array}$$

(4)

where ${\widehat{\mathbf{h}}}_r$ is the r-th row of the channel estimate matrix $\widehat{\mathbf{H}}$.

A maximum a posteriori probability (MAP) data detector is considered in this study to ensure the optimal detection performance. The APP from the MAP data detector is computed as    

$$\begin{array}{cc}\hfill {\theta }_k\left[n\right]& =\mathbb{P}\left[{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k|{\mathbf{y}}^{\mathrm{d}}\left[n\right]\right]=\frac{\mathbb{P}\left[{\mathbf{y}}^{\mathrm{d}}\left[n\right]|{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k\right]\mathbb{P}\left[{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k\right]}{{\displaystyle \sum _{j\in \mathcal{K}}}\mathbb{P}\left[{\mathbf{y}}^{\mathrm{d}}\left[n\right]|{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_j\right]\mathbb{P}\left[{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_j\right]},\hfill \end{array}$$

(5)

where ${\mathbf{x}}_k\in {\mathcal{X}}^{N_{\mathrm{t}}}$ is the k-th possible symbol for $k\in \mathcal{K}=\{1,\dots ,|\mathcal{X}{|}^{N_{\mathrm{t}}}\}$. In (5), the apriori probability, $\mathbb{P}\left[{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k\right]$, is assumed to be equal for all possible symbols ${\mathbf{x}}_k$ for $k\in \mathcal{K}$, i.e., $\mathbb{P}\left[{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k\right]=\frac{1}{{\left|\mathcal{X}\right|}^{N_{\mathrm{t}}}}$. Concurrently, under the AWGN assumption, the likelihood probability $\mathbb{P}\left[{\mathbf{y}}^{\mathrm{d}}\left[n\right]|{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k\right]$ in (5) can be expressed as

$$\begin{array}{cc}\hfill \mathbb{P}\left[{\mathbf{y}}^{\mathrm{d}}\left[n\right]|{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k\right]& =\frac{1}{{\left(\pi N_0\right)}^{N_{\mathrm{r}}}}{e}^{-\frac{\parallel {\mathbf{y}}^{\mathrm{d}}\left[n\right]-{\widehat{\mathbf{H}}}^H{\mathbf{x}}_k{\parallel }^2}{N_0}}.\hfill \end{array}$$

(6)

The MAP data detector detects the data symbol $\widehat{\mathbf{x}}\left[n\right]$ that has the best APP value at time slot n, and it is given by

$$\begin{array}{cc}\hfill \widehat{\mathbf{x}}\left[n\right]& =\underset{{\mathbf{x}}_k\in {\mathcal{X}}^{N_{\mathrm{t}}}}{argmax}{\theta }_k\left[n\right]=\underset{{\mathbf{x}}_k\in {\mathcal{X}}^{N_{\mathrm{t}}}}{argmax}\mathbb{P}\left[{\mathbf{y}}^{\mathrm{d}}\left[n\right]|{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_k\right].\hfill \end{array}$$

(7)

Note that the accuracy of the detected symbol $\widehat{\mathbf{x}}\left[n\right]$ depends on the accuracy of the channel estimator, $\widehat{\mathbf{H}}$. However, the accuracy of the channel estimator cannot be ensured in practical systems where the pilot length, $T_{\mathrm{p}}$, is limited. To address this limitation, this study focused on improving the accuracy of the channel estimator.

3. Optimization Problem

This section defines the optimization problem for the channel estimator proposed subsequently, which uses detected symbols to improve the MSE of the channel estimates. Subsequently, to solve the optimization problem, the MDP problem and the optimal policy are presented.

3.1. Optimization Problem

This study considers a channel estimator that uses the detected symbols in (7) as additional pilot symbols. However, the data detector may generate detection errors at the receiver. Consequently, the use of detected symbols with errors degrades the accuracy of the channel estimator. To overcome this problem, the detected symbols should be selectively exploited by the channel estimator.

Let $\mathbf{a}\in {\{0,1\}}^{T_{\mathrm{d}}}$ be the set of actions whose n-th component is the selection of a detected symbol of the d-th data block for $n\in {\mathcal{N}}_{\mathrm{d}}$. Specifically, when $a=1$, a detected symbol is used as an additional pilot symbol; otherwise, it is not used. By exploiting $\mathbf{a}$, the LMMSE channel estimate in (4) can be updated as

$$\begin{array}{cc}\hfill {\widehat{\mathbf{h}}}_r\left(\mathbf{a}\right)& ={\left(\mathbf{X}\left(\mathbf{a}\right)\mathbf{X}{\left(\mathbf{a}\right)}^H+N_0{\mathbf{I}}_{N_{\mathrm{t}}}\right)}^{-1}\mathbf{X}\left(\mathbf{a}\right){\overline{\mathbf{y}}}_r{\left(\mathbf{a}\right)}^H,\hfill \end{array}$$

(8)

where ${\overline{\mathbf{y}}}_r\left(\mathbf{a}\right)=[{\mathbf{y}}_r^{\mathrm{p}},y_r^{\mathrm{d}}\left[u_1\left(\mathbf{a}\right)\right],\dots ,y_r^{\mathrm{d}}\left[u_{{\parallel \mathbf{a}\parallel }_0}\left(\mathbf{a}\right)\right]]$ and $\mathbf{X}\left(\mathbf{a}\right)=[{\mathbf{X}}^{\mathrm{p}},\widehat{\mathbf{x}}\left[u_1\left(\mathbf{a}\right)\right],\dots ,\widehat{\mathbf{x}}\left[u_{{\parallel \mathbf{a}\parallel }_0}\left(\mathbf{a}\right)\right]]$.

Here, $u_i\left(\mathbf{a}\right)$ is the time slot index of the i-th nonzero element in $\mathbf{a}$. Thus, the optimization problem that maximizes the accuracy of the proposed channel estimator can be expressed as

$$\begin{array}{cc}\hfill {\mathbf{a}}^{★}& =\underset{\mathbf{a}\in {\{0,1\}}^{T_{\mathrm{d}}}}{argmax}\mathbb{E}\{\parallel \widehat{\mathbf{H}}\left(\mathbf{a}\right)-\mathbf{H}{\parallel }^2\}.\hfill \end{array}$$

(9)

Solving the optimization problem in (9) is difficult. First, the distribution of $\widehat{\mathbf{H}}\left(\mathbf{a}\right)$ requires information regarding the transmitted symbols. However, this information is generally unknown to a receiver. In addition, the number of candidates for actions $\mathbf{a}$ exponentially increases with data length $T_{\mathrm{d}}$. Accordingly, an exhaustive search for these actions is impractical because of the unsatisfactory complexity and latency for the receiver.

3.2. Markov Decision Process

To efficiently solve the problem in (9), an MDP was formulated in [26] that sequentially selected detected symbols. In this formulation, a detected symbol is selected if the updated channel estimator reduces the estimation error.

Similar to [26], for this study, the state set of the MDP at time slot n is expressed as

$$\begin{array}{cc}\hfill {\mathcal{S}}_n=\left\{\left({\mathbf{X}}_n,{\widehat{\mathbf{X}}}_n,{\mathcal{M}}_n\right)\right|& {\mathbf{X}}_n=\left[{\mathbf{X}}^{\mathrm{p}},{\mathbf{x}}_{k_{{\mathcal{M}}_n\left(1\right)}},\cdots ,{\mathbf{x}}_{k_{{\mathcal{M}}_n\left(|{\mathcal{M}}_n|\right)}}\right],k_i\in \mathcal{K},\hfill \\ \hfill & {\widehat{\mathbf{X}}}_n=\left[{\mathbf{X}}^{\mathrm{p}},\widehat{\mathbf{x}}\left[{\mathcal{M}}_n\left(1\right)\right],\cdots ,\widehat{\mathbf{x}}\left[{\mathcal{M}}_n\left(|{\mathcal{M}}_n|\right)\right]\right],\hfill \\ \hfill & {\mathcal{M}}_n\subset \left\{T_{\mathrm{p}}+1,\cdots ,n-1\right\}\},\hfill \end{array}$$

(10)

where $k_n$ denotes the transmitted symbol index at time slot n. Set ${\mathcal{M}}_n$ represents the set of time slot indices of the data symbols to be utilized as additional pilot symbols. ${\mathcal{M}}_n\left(i\right)$ is the i-th smallest element of ${\mathcal{M}}_n$. Based on the above notations, the proposed channel estimate at state ${\mathrm{S}}_n=\left({\mathbf{X}}_n,{\widehat{\mathbf{X}}}_n,{\mathcal{M}}_n\right)\in {\mathcal{S}}_n$ is expressed as

$$\begin{array}{cc}\hfill {\widehat{\mathbf{h}}}_r\left({\mathrm{S}}_n\right)& ={\left({\widehat{\mathbf{X}}}_n{\widehat{\mathbf{X}}}_n^H+N_0{\mathbf{I}}_{N_{\mathrm{t}}}\right)}^{-1}{\widehat{\mathbf{X}}}_n{\overline{\mathbf{y}}}_r^H\left({\mathrm{S}}_n\right),\hfill \end{array}$$

(11)

where ${\overline{\mathbf{y}}}_r\left({\mathrm{S}}_n\right)=[{\mathbf{y}}_r^{\mathrm{p}},y_r^{\mathrm{d}}\left[{\mathcal{M}}_n\left(1\right)\right],\dots ,y_r^{\mathrm{d}}[{\mathcal{M}}_n\left(\right|{\mathcal{M}}_n\left|\right)\left]\right]$.

The action set of the MDP is expressed as $\mathcal{A}=\{0,1\}$. An action is defined as whether to utilize a current detected symbol as an additional pilot symbol. Specifically, when $a=1\in \mathcal{A}$, the current detected symbol is used as an additional pilot symbol.

Based on the state and action sets, the state transition function of the MDP for $a\in \mathcal{A}$ and ${\mathrm{S}}_n\in {\mathcal{S}}_n$ is expressed as follows:

$$\begin{array}{cc}\hfill {\mathsf{T}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)& =\mathbb{P}\left[{\mathsf{U}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)|{\mathsf{S}}_n,a\right]=\left\{\begin{array}{cc}\mathbb{I}\left[{\mathbf{x}}^{\mathrm{d}}\left[n\right]={\mathbf{x}}_j\right],\hfill & j\in {\mathcal{J}}_a,a=1,\hfill \\ 1,\hfill & j\in {\mathcal{J}}_a,a=0.\hfill \end{array}\right.\hfill \end{array}$$

(12)

where ${\mathcal{J}}_0=\left\{0\right\}$ and ${\mathcal{J}}_1=\{1,\dots ,K\}$. State ${\mathsf{U}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)\in {\mathcal{S}}_{n+1}$ is the valid state from the current state ${\mathrm{S}}_n=\left({\mathbf{X}}_n,{\widehat{\mathbf{X}}}_n,{\mathcal{M}}_n\right)\in {\mathcal{S}}_n$, and is expressed as

$$\begin{array}{cc}\hfill {\mathsf{U}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)& =\left\{\begin{array}{cc}\left([{\mathbf{X}}_n,{\mathbf{x}}_j],[{\widehat{\mathbf{X}}}_n,\widehat{\mathbf{x}}\left[n\right]],[{\mathcal{M}}_n\cup n]\right),\hfill & j\in {\mathcal{J}}_a,a=1,\hfill \\ \left({\mathbf{X}}_n,{\widehat{\mathbf{X}}}_n,{\mathcal{M}}_n\right),\hfill & j\in {\mathcal{J}}_a,a=0.\hfill \end{array}\right.\hfill \end{array}$$

(13)

The reward function of the MDP is obtained by the MSE improvement between the channel estimates at the current state ${\mathrm{S}}_n$ and the next state ${\mathrm{S}}_{n+1}$. Thus, the reward function from ${\mathrm{S}}_n\in {\mathcal{S}}_n$ to ${\mathrm{S}}_{n+1}\in {\mathcal{S}}_{n+1}$ is defined as

$$\begin{array}{cc}\hfill \mathsf{R}\left({\mathrm{S}}_n,{\mathrm{S}}_{n+1}\right)& ={\mathcal{E}}_r\left({\mathrm{S}}_n\right)-{\mathcal{E}}_r\left({\mathrm{S}}_{n+1}\right),\hfill \end{array}$$

(14)

where ${\mathcal{E}}_r\left({\mathrm{S}}_n\right)$ is the MSE of the channel estimate for the r-th receive antenna at state ${\mathrm{S}}_n\in {\mathcal{S}}_n$, which can be computed as

$$\begin{array}{cc}\hfill {\mathcal{E}}_r\left({\mathrm{S}}_n\right)& =\mathbb{E}\left[\parallel {\widehat{\mathbf{h}}}_r\left({\mathrm{S}}_n\right)-{\mathbf{h}}_r{\parallel }^2\right]=\mathrm{Tr}\left[{\mathbf{C}}_e\left({\mathrm{S}}_n\right)\right],\hfill \end{array}$$

(15)

where the error covariance matrix ${\mathbf{C}}_e\left({\mathrm{S}}_n\right)$ is defined as $\mathbb{E}\left\{({\widehat{\mathbf{h}}}_r\left({\mathrm{S}}_n\right)-{\mathbf{h}}_r){({\widehat{\mathbf{h}}}_r\left({\mathrm{S}}_n\right)-{\mathbf{h}}_r)}^H\right\}$.

Here, ${\mathbf{C}}_e\left({\mathrm{S}}_n\right)$ is independent of the receiver antenna index, r, because the channel and noise distributions are the same for different receive antenna indices. Thus, the reward function in (14) can be simplified as

$$\begin{array}{cc}\hfill \mathsf{R}\left({\mathrm{S}}_n,{\mathrm{S}}_{n+1}\right)& =\mathrm{Tr}\left[{\mathbf{C}}_e\left({\mathrm{S}}_n\right)-{\mathbf{C}}_e\left({\mathrm{S}}_{n+1}\right)\right].\hfill \end{array}$$

(16)

The optimal policy of the MDP at time slot n is defined as

$$\begin{array}{cc}\hfill {\pi }^{★}\left({\mathrm{S}}_n\right)& =\underset{a\in \mathcal{A}}{argmax}\mathsf{Q}\left({\mathrm{S}}_n,a\right).\hfill \end{array}$$

(17)

where the Q-value function $\mathsf{Q}\left({\mathrm{S}}_n,a\right)$ is the optimal sum of the rewards. Based on the state transition function in (12), the Q-value function can be expressed as

$$\begin{array}{cc}\hfill \mathsf{Q}\left({\mathrm{S}}_n,a\right)& =\sum _{j\in {\mathcal{J}}_a}{\mathsf{T}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)\left[\mathsf{R}\left({\mathrm{S}}_n,{\mathsf{U}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)+\gamma {\mathsf{V}}^{★}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)\right],\hfill \end{array}$$

(18)

where $0\le \gamma \le 1$ is a discounting factor whose value depends on the target of the optimization problem. For example, a small value is desirable when the accuracy of the channel estimator obtained at the current state is significant. In contrast, a larger value is preferred when the accuracy of the channel estimator obtained at the ending state is significant.

${\mathsf{V}}^{★}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)$ is the optimal sum of the future rewards. The future value function ${\mathsf{V}}^{★}\left({\mathrm{S}}_m\right)$ at state ${\mathrm{S}}_m\in {\mathcal{S}}_m$ for $n+1\le m$ can be recursively computed, as follows:

$$\begin{array}{cc}\hfill {\mathsf{V}}^{★}\left({\mathrm{S}}_m\right)& =\sum _{a\in \mathcal{A}}\pi \left({\mathrm{S}}_m,a\right)\sum _{j\in {\mathcal{J}}_a}{\mathsf{T}}_{m+1}^{(a,j)}\left({\mathrm{S}}_m\right)\left[\mathsf{R}\left({\mathrm{S}}_m,{\mathsf{U}}_{m+1}^{(a,j)}\left({\mathrm{S}}_m\right)\right)+\gamma {\mathsf{V}}^{★}\left({\mathsf{U}}_{m+1}^{(a,j)}\left({\mathrm{S}}_m\right)\right)\right],\hfill \end{array}$$

(19)

where $\pi \left({\mathsf{S}}_m,a\right)$ is a state–action transition function, expressed as

$$\begin{array}{cc}\hfill \pi \left({\mathrm{S}}_m,a\right)& =\mathbb{I}\{a=\underset{a^{\prime }\in \mathcal{A}}{argmax}\mathsf{Q}\left({\mathrm{S}}_m,a^{\prime }\right)\},\hfill \end{array}$$

(20)

where $\mathsf{Q}\left({\mathrm{S}}_m,a\right)$ is the Q-value function that can be calculated as the sum of the rewards obtained after taking action $a\in \mathcal{A}$ at state ${\mathrm{S}}_m\in {\mathcal{S}}_m$.

Using the MDP in (10), (12), and (13), the state–action diagram of the original MDP is depicted in Figure 2a. In this figure, state ${\mathrm{S}}_n$ is transited to the next valid state, ${\mathsf{U}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)$, based on action a. Particularly, when $a=1$, state ${\mathrm{S}}_n$ is transited to state ${\mathsf{U}}_{n+1}^{(1,k_n)}\left({\mathrm{S}}_n\right)$ by utilizing the transmitted symbol index, $k_n$. Based on the state and state–action transition functions in (12) and (20), the state is transited to the next valid state until the end of a data block. As previously mentioned, the original MDP, which is shown in Figure 2a, cannot be solved by dynamic programming.

First, the state and state–action functions are unavailable to the receiver because the information of the transmitted symbols, ${\mathbf{x}}_{k_n}$, and the true channel information, $\mathbf{H}$, are unknown. In addition, the computational complexity and latency required to solve the original MDP are extremely high because the number of states exponentially increases with data length $T_{\mathrm{d}}$.

4. Proposed Rl-Based Channel Estimator

In this section, an RL-based channel estimator is proposed. To address the unknown state and state–action functions, an RL algorithm is adopted because it provides a solution for the partially observable MDP [27,28]. Based on this algorithm, a computationally efficient RL solution is also proposed. The key concept of the proposed solution is to approximate the state–action transition functions to determine the optimal policy by separating the cases using the APPs.

The overall procedure of the proposed RL-based channel estimator is illustrated in Figure 3. The proposed channel estimator exploits the information of ($\widehat{x}\left[m\right],{\theta }_j\left[m\right]$) obtained from the MIMO detector. In the proposed channel estimator, the optimal policy is calculated by using only N APPs $({\theta }_j\left[n\right],\dots ,{\theta }_j[n+N])$ for a computationally efficient algorithm. The channel estimate is then updated according to the optimal policy. Details of the proposed channel estimator, i.e., how to approximate the MDP and how to derive the optimal policy in a closed form, are explained in this section.

4.1. Statistical State Transition

In this section, the state transition function in (12) at time slot n is approximated using the APP ${\theta }_j\left[n\right]$. The basic concept was introduced in [26] by assuming the APP ${\theta }_j\left[n\right]$ as the probability of the event, $\{\mathbf{x}\left[n\right]={\mathbf{x}}_j\}$. Thus, the state transition function in (12) at time slot n is approximated as follows:

$$\begin{array}{cc}\hfill {\widehat{\mathsf{T}}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)& =\left\{\begin{array}{cc}{\theta }_j\left[n\right],\hfill & j\in {\mathcal{J}}_a,a=1,\hfill \\ 1,\hfill & j\in {\mathcal{J}}_a,a=0.\hfill \end{array}\right.\hfill \end{array}$$

(21)

where the detected symbol index at time slot n is denoted as ${\widehat{k}}_n$. Note that APP ${\theta }_j\left[n\right]$ can be interpreted as the probability of the event $\{\mathbf{x}\left[n\right]={\mathbf{x}}_j\}$; thus, it is called a statistical transition. In addition, when the data detection performance is improved, i.e., ${\theta }_{k_n}\left[n\right]\to 1$, the approximate state transition function in (21) approaches the true state transition function in (12).

4.2. State–Action Transition Using Backup Samples

After time slot $n+1\le m$, the state in (20) is assumed to be transited to a virtual state that mimics the possible next states by exploiting the expected transmitted symbol, $\tilde{\mathbf{x}}\left[m\right]$. The expected transmitted symbol, $\tilde{\mathbf{x}}\left[m\right]$, is defined as

$$\begin{array}{cc}\hfill \tilde{\mathbf{x}}\left[m\right]& =\sum _{j=1}^K{\theta }_j\left[m\right]{\mathbf{x}}_j.\hfill \end{array}$$

(22)

In this study, the use of the expected transmitted symbol is the same as in [26], except its use is limited to N backup samples to reduce the complexity. A backup sample is defined as APP ${\theta }_j\left[m\right]$ for $n+1\le m\le n+N$ because the expected transmitted symbol can be computed by ${\theta }_j\left[m\right]$. Thus, the Q-value function can be calculated after all ${\theta }_j\left[m\right]$ for $n+1\le m\le n+N$ values are obtained. Using a backup sample of an APP, the state–action transition is expressed as

$$\begin{array}{cc}\hfill \widehat{\pi }\left({\mathsf{S}}_m,a\right)& =1.\hfill \end{array}$$

(23)

Thus, the virtual state, ${\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)\in {\mathcal{S}}_m$, that can be transited from ${\mathrm{S}}_n\in {\mathcal{S}}_n$ is expressed as

$$\begin{array}{cc}\hfill {\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)& =\left({\mathbf{X}}_m^{(a,j)},{\widehat{\mathbf{X}}}_m^{\left(a\right)},{\mathcal{M}}_m^{\left(a\right)}\right),\hfill \end{array}$$

(24)

where their components are

$$\begin{array}{cc}\hfill {\mathbf{X}}_m^{(a,j)}& =\left\{\begin{array}{cc}\left[{\mathbf{X}}_n,{\mathbf{x}}_j,\tilde{\mathbf{x}}[n+1],\cdots ,\tilde{\mathbf{x}}[n+N]\right],\hfill & a=1,\hfill \\ \left[{\mathbf{X}}_n,\tilde{\mathbf{x}}[n+1],\cdots ,\tilde{\mathbf{x}}[n+N]\right],\hfill & a=0.\hfill \end{array}\right.\hfill \\ \hfill {\widehat{\mathbf{X}}}_m^{\left(a\right)}& =\left\{\begin{array}{cc}\left[{\widehat{\mathbf{X}}}_n,\widehat{\mathbf{x}}\left[n\right],\tilde{\mathbf{x}}[n+1],\cdots ,\tilde{\mathbf{x}}[n+N]\right],\hfill & a=1,\hfill \\ \left[{\widehat{\mathbf{X}}}_n,\tilde{\mathbf{x}}[n+1],\cdots ,\tilde{\mathbf{x}}[n+N]\right],\hfill & a=0.\hfill \end{array}\right.\hfill \\ \hfill {\mathcal{M}}_m^{\left(a\right)}& =\left\{\begin{array}{cc}\left[{\mathcal{M}}_n\cup \{n,\dots ,n+N\}\right],\hfill & a=1,\hfill \\ \left[{\mathcal{M}}_n\cup \{n+1,\dots ,n+N\}\right],\hfill & a=0.\hfill \end{array}\right.\hfill \end{array}$$

Because a virtual state mimics the transitions to the candidate symbols, state ${\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)\in {\mathcal{S}}_m$ is always transited to a virtual state ${\tilde{\mathsf{U}}}_{m+1}^{(a,j)}\left({\mathrm{S}}_n\right)\in {\mathcal{S}}_{m+1}$. Therefore, the corresponding state transition function is written as

$$\begin{array}{cc}\hfill {\widehat{\mathsf{T}}}_{m+1}^{(a,j)}\left({\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)\right)& =1,\hfill \end{array}$$

(25)

where $n+1\le m\le n+N$.

4.3. State–Action Transition after Backup Samples

In this subsection, the virtual states after $n+N$ that can be transited without the information of the backup samples, ${\theta }_j\left[m\right]$, are described for $n+N+1\le m$. To achieve this, the states, ${\widehat{\mathsf{U}}}_{m+1}^{(a,j)}\left({\mathrm{S}}_n\right)$, for $n+N+1\le m$ are assumed to optimally act when all symbols are correctly detected. By using the property of $\mathbf{x}\left[m\right]=\widehat{\mathbf{x}}\left[m\right]$ after time slot $n+N+1$, an approximate virtual state is expressed as   

$$\begin{array}{c}\hfill {\widehat{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)=\left({\mathbf{X}}_m^{(a,j)},{\widehat{\mathbf{X}}}_m^{\left(a\right)},{\mathcal{M}}_m^{\left(a\right)}\right),\end{array}$$

(26)

where its components are defined as

$$\begin{array}{cc}\hfill {\mathbf{X}}_m^{(a,j)}& =\left[{\mathbf{X}}_{n+N+1}^{(a,j)},\widehat{\mathbf{x}}[n+N+1],\cdots ,\widehat{\mathbf{x}}[m-1]\right],\hfill \\ \hfill {\widehat{\mathbf{X}}}_m^{\left(a\right)}& =\left[{\tilde{\mathbf{X}}}_{n+N+1}^{\left(a\right)},\widehat{\mathbf{x}}[n+N+1],\cdots ,\widehat{\mathbf{x}}[m-1]\right],\hfill \\ \hfill {\mathcal{M}}_m^{\left(a\right)}& =\left[{\mathcal{M}}_{n+N+1}^{\left(a\right)}\cup \{n+N+1,\dots ,m-1\}\right],\hfill \end{array}$$

where $\left({\mathbf{X}}_{n+N+1}^{(a,j)},{\widehat{\mathbf{X}}}_{n+N+1}^{\left(a\right)},{\mathcal{M}}_{n+N+1}^{\left(a\right)}\right)$ are the components of ${\tilde{\mathsf{U}}}_{n+N+1}^{(a,j)}\left({\mathrm{S}}_n\right)$.

In Figure 2b, a state–action diagram of the approximate MDP is depicted. The original MDP requires information regarding the transmitted symbols for the state transition, as shown in Figure 2a. In contrast, the approximate MDP utilizes virtual states ${\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)$ and ${\widehat{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)$, which mimic the transitions to the candidate symbols for an unknown transmitted symbol and action. Specifically, virtual state ${\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)$ is used at time slot $n+1\le m\le n+N$ and after time slot $n+N$, respectively. These two approximations decrease the number of transitions to the next state transition, so the calculation to solve the MDP is considerably reduced.

4.4. Proposed Optimal Policy

Using the approximations in (21), (23), and (24), the optimal policy can be determined. However, the calculation latency is still considerable, because the optimal policy can be computed at the end of a data block. To prevent this computational burden, the proposed solution separates a data block into $N_{\mathrm{b}}$ data subblocks and subsequently characterizes the optimal policy for each data subblock, as shown in Figure 4. Based on this characterization, the state in (10) and the corresponding channel estimate using (11) are updated for a data subblock. To realize this data subblock separation, the data subblock length is defined as $T_{\mathrm{b}}$, which satisfies $N_{\mathrm{b}}=T_{\mathrm{d}}/T_{\mathrm{b}}$. Thus, a set of time slot indices of the b-th data subblock in the d-th data block, ${\mathcal{N}}_{b,d}$, is defined as $\{T_{\mathrm{p}}+(b-1)T_{\mathrm{b}}+(d-1)T_{\mathrm{d}}+1,\dots ,T_{\mathrm{p}}+b{T}_{\mathrm{b}}+(d-1)T_{\mathrm{d}}\}$, for $b\in \{1,\dots ,N_{\mathrm{b}}\}$ and $d\in \{1,\dots ,N_{\mathrm{d}}\}$ (see Figure 4).

Using the virtual states in (24) and (26), the Q-value function is written as

$$\begin{array}{cc}\hfill \mathsf{Q}\left({\mathrm{S}}_n,a\right)& =\sum _{j\in {\mathcal{J}}_a}{\mathsf{T}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)[\mathsf{R}\left({\mathrm{S}}_n,{\tilde{\mathsf{U}}}_{n+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)+\sum _{m=n+1}^{n+N}{\gamma }^{m-n}\mathsf{R}\left({\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right),{\tilde{\mathsf{U}}}_{m+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)\hfill \\ \hfill & +{\gamma }^{N+1}{\mathsf{V}}^{★}\left({\widehat{\mathsf{U}}}_{n+N+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)],\hfill \end{array}$$

(27)

where the future value function, ${\mathsf{V}}^{★}\left({\widehat{\mathsf{U}}}_{n+N+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)$, is obtained based on the approximation of ${\widehat{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right)$ as follows:   

$${\mathsf{V}}^{★}\left({\widehat{\mathsf{U}}}_{n+N+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right)\approx \mathsf{R}\left({\tilde{\mathsf{U}}}_{n+N+1}^{(a,j)}\left({\mathrm{S}}_n\right),{\widehat{\mathsf{U}}}_{n+N+2}^{(a,j)}\left({\mathrm{S}}_n\right)\right)+\sum _{m=n+N+2}^{{\mathcal{N}}_{b,d}\left(T_{\mathrm{b}}\right)}\mathsf{R}\left({\widehat{\mathsf{U}}}_m^{(a,j)}\left({\mathrm{S}}_n\right),{\widehat{\mathsf{U}}}_{m+1}^{(a,j)}\left({\mathrm{S}}_n\right)\right).$$

(28)

In the future reward in (28), the discounting factor is assumed to be 1 to reduce the complexity by a simple calculation.

Based on (27) and (28), the optimal policy for each state is obtained as a closed-form expression, as described in the following theorem:

Theorem 1.

Under the virtual states and the use of backup samples, the optimal policy for the state ${\mathrm{S}}_n=\left({\mathbf{X}}_n,{\widehat{\mathbf{X}}}_n,{\mathcal{M}}_n\right)\in {\mathcal{S}}_n$ is

$$\begin{array}{cc}\hfill {\pi }^{★}\left({\mathrm{S}}_n\right)& =\mathbb{I}\left[\frac{{\displaystyle \sum _{m=n}^{n+N}}{\gamma }^{m-n}(1-\gamma ){\mathsf{U}}_m\left({\mathrm{S}}_n\right)+{\gamma }^{N+1}{\mathsf{U}}_{{\mathcal{N}}_{b,d}\left(T_{\mathrm{b}}\right)+1}\left({\mathrm{S}}_n\right)}{{\displaystyle \sum _{m=n}^{n+N}}{\gamma }^{m-n}(1-\gamma ){\mathsf{L}}_m\left({\mathrm{S}}_n\right)+{\gamma }^{N+1}{\mathsf{L}}_{{\mathcal{N}}_{b,d}\left(T_{\mathrm{b}}\right)+1}\left({\mathrm{S}}_n\right)}\ge 1\right],\hfill \end{array}$$

(29)

where functions ${\mathsf{U}}_m\left({\mathrm{S}}_n\right)$ and ${\mathsf{L}}_m\left({\mathrm{S}}_n\right)$ are respectively defined as

$$\begin{array}{cc}\hfill {\mathsf{U}}_m\left({\mathrm{S}}_n\right)& =\parallel {\mathbf{t}}_m{\parallel }^2\left(N_0+N_0^2\parallel {\mathbf{t}}_m{\parallel }^2+{\parallel {\mathbf{v}}_m\parallel }^2\right)\hfill \\ \hfill {\mathsf{L}}_m\left({\mathrm{S}}_n\right)& =\parallel {\mathbf{t}}_m{\parallel }^2\left(2N_0^2{\beta }_m+{\delta }_m+{\parallel {\mathbf{e}}_m-{\mathbf{u}}_m+{\mathbf{v}}_m\parallel }^2\right)\hfill \end{array}$$

All components are defined as

$$\begin{array}{cc}\hfill & {\mathbf{Q}}_m={\left({\widehat{\mathbf{X}}}_n{\widehat{\mathbf{X}}}_n^H+\sum _{l=n+1}^m\tilde{\mathbf{x}}\left[l\right]{\tilde{\mathbf{x}}}^H\left[l\right]+N_0{\mathbf{I}}_{N_{\mathrm{t}}}\right)}^{-1},{\mathbf{D}}_m={\widehat{\mathbf{X}}}_n{\left({\widehat{\mathbf{X}}}_n-{\mathbf{X}}_n\right)}^H+\sum _{l=n+1}^m\widehat{\mathbf{x}}\left[l\right]{\left(\widehat{\mathbf{x}}\left[l\right]-\tilde{\mathbf{x}}\left[l\right]\right)}^H+N_0{\mathbf{I}}_{N_{\mathrm{r}}},\hfill \\ \hfill & {\mathbf{t}}_m=\frac{1}{\sqrt{1+{\alpha }_m}}{\mathbf{Q}}_m\widehat{\mathbf{x}}\left[n\right],{\mathbf{e}}_m=\frac{1}{\sqrt{1+{\alpha }_m}}\left(\widehat{\mathbf{x}}\left[n\right]-\tilde{\mathbf{x}}\left[n\right]\right),{\mathbf{u}}_m={\mathbf{D}}_m^H{\mathbf{t}}_m,{\mathbf{v}}_m=\frac{{\mathbf{D}}_m^H{\mathbf{Q}}_m{\mathbf{t}}_m}{\parallel {\mathbf{t}}_m{\parallel }^2},\hfill \\ \hfill & {\alpha }_m={\widehat{\mathbf{x}}}^H\left[n\right]{\mathbf{Q}}_m\widehat{\mathbf{x}}\left[n\right],{\beta }_m=\frac{{\mathbf{t}}_m^H{\mathbf{Q}}_m{\mathbf{t}}_m}{\parallel {\mathbf{t}}_m{\parallel }^2},{\delta }_m=\frac{1}{1+{\alpha }_m}\left(\sum _{j=1}^K{\theta }_j\left[n\right]\parallel \widehat{\mathbf{x}}\left[n\right]-{\mathbf{x}}_j{\parallel }^2-{\parallel \widehat{\mathbf{x}}\left[n\right]-\tilde{\mathbf{x}}\left[n\right]\parallel }^2\right)\hfill \\ \hfill & {\mathbf{Q}}_{{\mathcal{N}}_{b,d}\left(T_{\mathrm{b}}\right)+1}={\left({\mathbf{Q}}_{n+N}^{-1}+\left({\mathcal{N}}_{b,d}\left(T_{\mathrm{b}}\right)-(n+N-1)\right){\mathbf{I}}_{N_{\mathrm{t}}}\right)}^{-1},{\mathbf{D}}_{{\mathcal{N}}_{b,d}\left(T_{\mathrm{b}}\right)+1}={\mathbf{D}}_{n+N}.\hfill \end{array}$$

(30)

Proof. 

See Appendix A. □

4.5. Summary: The Proposed Algorithm

The proposed channel estimator is summarized in Algorithm 1. First, the receiver initializes the state during pilot transmission. In this algorithm, the current state is updated and transited to the next state according to the optimal action obtained using (29). For example, the most probable state transition is used when ${\alpha }^{★}=1$ for the unknown transmitted symbol index. This transition ensures a true state transition as ${\theta }_j\left[n\right]$ approaches 1 in reliable communication. At the end of a data subblock, the proposed channel estimator updates the channel estimate using the current state, ${\mathrm{S}}_n$.  

Algorithm 1: The proposed channel estimator.

1  Set $\mathbf{H}\leftarrow \widehat{\mathbf{H}}=\left[{\widehat{\mathbf{h}}}_1,\cdots ,{\widehat{\mathbf{h}}}_{N_{\mathrm{r}}}\right]$ from (4)   2  Initialize ${\mathrm{S}}_1=\left({\mathbf{X}}^{\mathrm{p}},{\mathbf{X}}^{\mathrm{p}},\varphi \right)$.   3  for $d=1$ to $N_{\mathrm{d}}$do$\phantom{(}$   4   for $b=1$ to $N_{\mathrm{b}}$do   5    for $n\in {\mathcal{N}}_{b,d}$do   6       Obtain $\widehat{\mathbf{x}}\left[n\right]$ from (8) and $\{{\theta }_j\left[n\right],\cdots ,{\theta }_j[n+N]\}$ from (5) for $j\in \mathcal{K}$   7       Compute $a^{★}={\pi }^{★}\left({\mathrm{S}}_n\right)$ from (29).   8       Set $j^{★}=0$ for $a^{★}=0$ and ${\mathbf{x}}_{j^{★}}=\widehat{\mathbf{x}}\left[n\right]$ for $a^{★}=1$.   9       Update ${\mathrm{S}}_{n+1}\leftarrow {\mathsf{U}}_{n+1}^{(a^{★},j^{★})}\left({\mathrm{S}}_n\right)$ from (13). 10    end 11    Set $\mathbf{H}\leftarrow \widehat{\mathbf{H}}=\left[{\widehat{\mathbf{h}}}_1\left({\mathrm{S}}_n\right),\cdots ,{\widehat{\mathbf{h}}}_{N_{\mathrm{r}}}\left({\mathrm{S}}_n\right)\right]$ from (11). 12  end 13  end

4.6. Complexity Analysis

In this subsection, the complexity of both the proposed channel estimator and that in [26] is discussed based on the number of states visited in the calculation of the optimal policy. This is because the rewards in the optimal policy are computed based on the states, and the calculation in (29) is similar to that in [26]. First, when the current state is ${\mathbf{S}}_n\in {\mathcal{S}}_n$ in the d-th data block, the number of visiting states in [26] is exactly $d{T}_{\mathrm{d}}-n$. By contrast, the number of visiting states using the proposed channel estimator in the b-th data subblock is exactly $(b-1)T_{\mathrm{b}}+1+(d-1)T_{\mathrm{d}}-n$. Thus, the number of states ($T_{\mathrm{d}}-(b-1)T_{\mathrm{b}}-1$) is not used in the policy calculation on introducing the data subblocks. In addition to the complexity, the proposed optimal policy can be calculated after obtaining N backup samples, whereas in the approach in [26], this is possible at the end of a data block. Thus, the latency of the optimal policy by the approach in [26] is much longer than that of the proposed optimal policy.

5. Simulation Results

This section discusses the performance of the proposed channel estimator. The number of antennas in MIMO systems is $(N_{\mathrm{t}},N_{\mathrm{r}})=(4,4)$. A rate $1/2$ turbo code is adopted for channel coding, and 4-quadrature amplitude modulation (QAM) is adopted for symbol mapping. The frame consists of $(T_{\mathrm{p}},T_{\mathrm{d}},N_{\mathrm{d}})=(8,64,20)$, and the proposed channel estimator utilizes a data subblock as $(T_{\mathrm{b}},N_{\mathrm{b}})=(16,4)$. In addition, the parameters of the proposed channel estimator are $(N,\gamma )=(1,0.5)$, unless specified otherwise. The per-bit signal-to-noise ratio (SNR) is defined as $E_b/N_0=\frac{1}{{log}_2\left|\mathcal{X}\right|N_0}$.

In all figures, the performance with perfect and imperfect channel estimates using the LMMSE method are denoted as PCSI and CE, respectively. For performance benchmarking, the optimal cases of the proposed channel estimator and the expected-symbol-based channel estimator utilizing perfect knowledge of the transmitted symbol and the expected symbol in (22) as an additional pilot symbol, respectively, are compared. The performance is measured in terms of the block error rate (BLER) and the normalized MSE (NMSE). In Figure 5, the proposed channel estimator is compared with other channel estimators, and the conventional RL method used in [26] is also depicted. It shows that the BLER of the proposed estimator is better than those of the conventional and expected-symbol-based estimators regardless of the per-bit SNR. Moreover, the proposed channel estimator outperforms the conventional estimator of [26]. This is because the proposed channel estimator updates a channel estimate by $N_{\mathrm{b}}$ in a data block, whereas the method in [26] updates it once at the end of a data block.

Figure 6 compares the BLERs of the conventional and proposed channel estimators for different modulations. For 16-QAM, a MIMO system with $(N_{\mathrm{t}},N_{\mathrm{r}})=(2,4)$ is considered because of the SNR range. The proposed channel estimator achieves an improved BLER compared to the conventional LMMSE channel estimators. This result demonstrates the effectiveness of the proposed channel estimator, which optimizes the selection of detected symbols. The improvements to achieve a BLER of ${10}^{-1}$ are approximately 1.2 dB and 0.7 dB for the 4- and 16-QAM, respectively. The BLER for the 16-QAM is more improved than that of the PCSI, which is better than that of the 4-QAM. This is because in 16-QAM, the number of reliable detected symbols that can be used as additional pilot symbols is larger than in 4-QAM.

The NMSEs of the proposed channel estimator for different data subblock lengths are shown in Figure 7. The NMSE improves as $N_{\mathrm{b}}$ decreases. This is because the approximate MDP using data subblocks approaches the original MDP as $N_{\mathrm{b}}$ decreases. However, as shown in Figure 7, the NMSE improvement is insignificant, whereas the complexity exponentially increases with $T_{\mathrm{b}}$. Thus, $(T_{\mathrm{b}},N_{\mathrm{b}})=(16,4)$ is considered in this study for the simulations.

The NMSE of the proposed channel estimator based on the number of backup samples is shown in Figure 8. Noticeably, the NMSE is improved as the number of backup samples increases. This is because the accuracy of the state–action diagram model improves as the number of backup samples increases. In addition, with a small value of N, the proposed channel estimator achieves a sufficient NMSE performance. It should be noted that the complexity and latency required to determine the optimal policy increase with the number of backup samples.

Figure 9 and Figure 10 are the results obtained using the proposed channel estimator in time-varying channels. Specifically, a first-order Gaussian–Markov process used in [29,30] was adopted.In this process, the channel matrix at time slot n is defined as

$$\begin{array}{cc}\hfill {\mathbf{H}}^{\left(n\right)}& =\sqrt{1-{ϵ}^2}{\mathbf{H}}^{(n-1)}+ϵ{\mathbf{e}}^{\left(n\right)},\hfill \end{array}$$

(31)

where $n\in {\mathcal{N}}_{b,d}$ for $b\in \{1,2,\dots ,N_{\mathrm{b}}\}$ and $d\in \{1,2,\dots ,N_{\mathrm{d}}\}$. $ϵ\in [0,1]$ is a temporal correlation coefficient depending on the velocity, and ${\mathbf{H}}^{\left(0\right)}$ is an initial channel estimate. Each element in ${\mathbf{e}}^{\left(n\right)}\in {\mathbb{C}}^{N_{\mathrm{r}}\times N_{\mathrm{t}}}$ is assumed to follow $\mathcal{CN}(0,1)$. Temporal correlation coefficients $ϵ=5\times {10}^{-3}$ and $ϵ={10}^{-2}$ are used for the simulations.

Figure 9 shows the variation in the NMSE of the proposed channel estimator with the discounting factor. When a channel varies over time as $ϵ=5\times {10}^{-3}$, an NMSE with $\gamma =0.1$ is better than it is with $\gamma =0.9$. This is because the rewards at the future states in the time-varying channels are insignificant; therefore, a small value of the discounting factor is preferable. By contrast, when the channels are time-invariant, the rewards at the future states as well as those at the current state are important. Thus, the large value of $\gamma =0.9$ improves the NMSE compared to $\gamma =0.1$. Figure 10 compares the BLERs of the proposed and conventional channel estimators. When $ϵ={10}^{-2}$, the BLERs of the CE are severely degraded because the CE method cannot capture the channel variation. However, the proposed channel estimator shows robustness in time-varying channels because the channel variation can be tracked efficiently by selecting the detected symbols.

6. Conclusions

In this paper, a low-complexity algorithm for an RL-based channel estimator for MIMO systems was proposed. The proposed channel estimator adaptively selects detected symbols as additional pilot symbols to minimize the channel estimation error. In this study, an MDP problem was introduced, and a practical algorithm to solve it was developed using backup samples and data subblocks. Simulation results showed that the proposed channel estimator significantly improves the BLER and the NMSE compared to the conventional channel estimator.

A future direction of this study is to develop the RL approach for a realistic channel. The proposed method was derived based on the Rayleigh fading channel, but the realistic channel may have a line of sight. Thus, the MDP under the Rician fading channel should be investigated. Another important direction is to develop the RL approach for frequency-selective channels. In frequency-selective channels, the use of multiple sub-carriers can increase computational complexity considerably. Thus, a low-complexity algorithm in frequency-selective channels is necessary. Lastly, the RL approach can also be extended to other advanced channel estimators, such as the iterative method. In this method, the MDP should be reformulated according to the channel estimator.