Low-Complexity Joint Channel Estimation for Multi-User mmWave Massive MIMO Systems

Abstract

For multi-user millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) systems, the precise acquisition of channel state information (CSI) is a huge challenge. With the increase of the number of antennas at the base station (BS), the traditional channel estimation techniques encounter the problems of pilot training overhead and computational complexity increasing dramatically. In this paper, we develop a step-length optimization-based joint iterative scheme for multi-user mmWave massive MIMO systems to improve channel estimation performance. The proposed estimation algorithm provides the BS with full knowledge of all channel parameters involved in up- and down-links. Compared with existing algorithms, the proposed algorithm has higher channel estimation accuracy with low complexity. Moreover, the proposed scheme performs well even with a small number of training sequences and a large number of users. Simulation results are shown to demonstrate the performance of the proposed channel estimation algorithm.

1. Introduction

Millimeter wave (mmWave) massive multiple-input multiple-output (MIMO) has been generally recognized as a strong potential key technique for enhancing quality in the future wireless communication field [1]. The demand for higher speed data transmission, shorter delays, better user experience, and denser networks is increasing with the rapid development of wireless services such as virtual reality, multimedia, and the Internet of Things [2]. The mmWave band has huge unexploited spectrum resources, which can overcome the spectrum congestion of standard wireless frequency bands and achieve an orders of magnitude increase in spectral efficiency to meet these requirements [3,4]. The work in [5] expounded that the mmWave band can greatly speed up the data transmission rate. Additionally, massive MIMO refers to adding a number of antennas at the base station (BS), which can overcome the effects of fading, average out the dominant path-loss, and make up for the short wavelength of mmWave [6,7,8], and the isolation between radiating elements is a very important issue [9,10]. The combination of mmWave and massive MIMO can be used to meet increasing demand in data traffic [11,12].

Many studies have been devoted to the research of mmWave massive MIMO multi-user systems in recent years. Compared with conventional point-to-point MIMO, multi-user MIMO with simplified resource allocation has greater advantages [13,14]. The work [15] developed blind multi-user detection based on the model of [16], which can approach the lower bound of performance under certain conditions. The work in [17] summarized the benefits, challenges, and potential solutions of mmWave massive MIMO in cellular networks.

In order to achieve the theoretical performance gain and potential benefits of mmWave massive MIMO systems, accurate channel state information (CSI) is essential, which is also regarded as one of the main difficulties in mmWave communication [18,19]. However, due to the difference in channel characteristics compared to traditional sub-6 GHz, the smaller number of RF chains, and overwhelming overhead in downlink training and uplink channel feedback, accurate acquisition of CSI is challenging [20,21,22]. Various novel channel estimation approaches have been recently proposed for mmWave massive MIMO [23,24,25,26,27]. In particular, under the assumption of the BS antennas being highly correlated, the work in [23] proposed an antenna grouping method to reduce feedback overhead. By utilizing the correlation among users, the work in [24] provided a joint CSI acquisition proposal based on a low-rank matrix. To reduce errors caused by discretization and solve the line spectral estimation problem, the work in [25] designed an iterative reweighted approach for joint dictionary parameter training and sparse signal recovery, but it had the limitation that only two parameters could be estimated. The work in [26] extended the suggestion in [25] to two dimensions and represented a super-resolution channel estimation scheme based on iterative reweighting in a point-to-point scenario. It is regrettable that such a solution in [26] had the problem of slow estimation. Considering the problem of channel estimation for mmWave MIMO systems under a transmitter impairment model, a new algorithm based on Bayesian compressive sensing (BCS) was introduced in [27]. However, the assumption of BCS was relatively harsh, and the accuracy of estimation was limited.

To obtain CSI more accurately and faster in multi-user mmWave massive MIMO, we develop a step-length optimization-based joint iterative channel estimation scheme in this paper. Our main contributions are outlined as follows:

1. For multi-user mmWave massive MIMO systems, we propose a novel joint estimation scheme, which can accurately estimate the azimuth angles of departure or arrival (AoDs/AoAs) and joint gain at the BS. Specifically, the proposed algorithm does not need to estimate the channel on the user side, and all channel parameters can be estimated at the BS, which is different from the traditional channel estimation scheme.

2. We combine the advantages of the Newton method and gradient descent method and derive an iterative step-length optimization approach to reduce computational complexity. It is confirmed that the proposed scheme has low complexity without losing the accuracy of channel estimation.

3. Compared with the traditional singular-value decomposition (SVD) algorithm and BCS algorithm under different situations, the simulation results show that the proposed algorithm achieves better channel estimation performance with fewer training data blocks.

The rest of this paper is arranged as follows. Section 2 introduces the considered multi-user mmWave massive MIMO model. Section 3 formulates the proposed joint iterative channel estimation algorithm, which accounts for the preprocessing, the choice of descent direction in the iteration, the optimization of the step-length, and the method for separating the downlink channel. Some simulation results are provided to illustrate the performance of estimation in Section 4, and Section 5 concludes this paper.

Notation: In this paper, vectors and matrices are denoted by boldface lowercase and uppercase symbols, respectively; ${\mathbf{A}}^T$, ${\mathbf{A}}^H$, ${\mathbf{A}}^{-1}$, and ${\mathbf{A}}^{†}$ correspond to the transpose, conjugate transpose, inverse, and Moore–Penrose pseudo-inverse of the matrix $\mathbf{A}$; $vec\left(\mathbf{A}\right)$ is the vector stacked by the columns of the matrix $\mathbf{A}$; ⊗ is the Kronecker product; $diag\left(\mathbf{x}\right)$ denotes the diagonal matrix with the vector $\mathbf{x}$ on its diagonal; $rank\left(\mathbf{A}\right)$ is the rank of $\mathbf{A}$; ${∥\mathbf{A}∥}_F$, ${∥\mathbf{h}∥}_0$, and  ${∥\mathbf{h}∥}_2$ represent the Frobenius norm of $\mathbf{A}$, the ${\ell }_0$-norm, and Euclidean norm of the vector $\mathbf{h}$, respectively; ${\mathbf{I}}_N$ is the $N\times N$ identity matrix; the sets of real and complex numbers are represented by $\mathbb{R}$ and $\mathbb{C}$, respectively.

2. System Model

We considered a common mmWave massive MIMO system with hybrid precoding as illustrated in Figure 1, where the number of antennas at the BS is $N_{BS}$; but, only $N_{RF}$ RF chains with $N_{BS}>N_{RF}$ to support K user equipment (UE), and each UE has a single receive antenna [18,28]. In the communication process, it is assumed that there is only one transmission path via $N_S$ data streams between BS and each UE.

Let $m_T=1,\cdots ,M_T$ and $m_R=1,\cdots ,M_R$ be the time slot of the transmitter and receiver, respectively. The BS pilots ${\mathbf{S}}_{m_T,m_R}\in {\mathbb{C}}^{N_s\times L}$ represent the signal transmitted in the ${\left(m_T,m_R\right)}^{\mathrm{th}}$ time slot, where L is the number of different pilot sequences, and its design meets the orthogonality condition ${\mathbf{S}}_{m_T,m_R}·{\mathbf{S}}_{m_T,m_R}^{\mathrm{H}}={\mathbf{I}}_{N_S\times N_S}$ [29,30]. Then, the signal ${\mathbf{y}}_k\in {\mathbb{C}}^{1\times L}$ received by the $k^{\mathrm{th}}$ user can be given by:

$${\mathbf{y}}_k={\mathbf{h}}_D^k{\mathbf{F}}_{T,m_T}{\mathbf{S}}_{m_T,m_R}+n_{m_T,m_R}^{MS}$$

(1)

where ${\mathbf{F}}_{T,m_T}={\left[{\mathbf{F}}_{RF}\right]}_{T,m_T}{\left[{\mathbf{F}}_{BB}\right]}_{T,m_T}\in {\mathbb{C}}^{N_{BS}\times N_S}$ is the transmitted hybrid precoding matrix in the ${m_T}^{\mathrm{th}}$ time slot, ${\mathbf{F}}_{BB}\in {\mathbb{C}}^{N_{RF}\times N_S}$ is the baseband beamformer followed by the RF beamformer ${\mathbf{F}}_{RF}\in {\mathbb{C}}^{N_{BS}\times N_{RF}}$, $n_{m_T,m_R}^{MS}\in {\mathbb{C}}^{1\times L}$ is the received independent and identically distributed additive white Gaussian noise elements having zero mean and the variance ${\sigma }_k^2$, and ${\mathbf{h}}_D^k\in {\mathbb{C}}^{1\times N_{BS}}$ is the downlink channel vector, and by defining ${\theta }_k\triangleq dsin{\delta }_{T,k}/\lambda$ as the normalized spatial angle [31], it can be expressed as:

$${\mathbf{h}}_D^k={\alpha }_k{\mathbf{a}}^H\left({\theta }_k\right)$$

(2)

where ${\alpha }_k$, d, and $\lambda$ denote the propagation gain of the $k^{\mathrm{th}}$ downlink path and the antenna spacing at the BS and carrier wavelength, respectively. $d=\frac{1}{2}\lambda$ is widely adopted here [32]. ${\delta }_{T,k}\in [0,2\pi ]$ is the AoD of the $k^{\mathrm{th}}$ path. $\mathbf{a}\left({\theta }_k\right)\in {\mathbb{C}}^{N_{BS}\times 1}$ is the steering vector at the transmitter of the $k^{\mathrm{th}}$ path. In this work, we considered the typical uniform linear arrays (ULAs) [33]; $\mathbf{a}\left({\theta }_k\right)$ is given by:

$$\mathbf{a}\left({\theta }_k\right)={\left[1,e^{j2\pi {\theta }_k},\cdots ,e^{j2\pi \left(N_{BS}-1\right){\theta }_k}\right]}^T.$$

(3)

Denote ${\varphi }_k\triangleq dsin{\delta }_{R,k}/\lambda$ where ${\delta }_{R,k}$ is the AoA of the $k^{\mathrm{th}}$ path. ${\varphi }_k$ is the normalized spatial angle the same as ${\theta }_k$. The uplink channel vector ${\mathbf{h}}_U^k\in {\mathbb{C}}^{N_{BS}\times 1}$ can be expressed as:

$${\mathbf{h}}_U^k={\beta }_k\mathbf{b}\left({\varphi }_k\right)$$

(4)

where ${\beta }_k$ is the propagation gain of the $k^{\mathrm{th}}$ uplink path and $\mathbf{b}\left({\varphi }_k\right)\in {\mathbb{C}}^{N_{BS}\times 1}$ is the steering vector at the receiver of the $k^{\mathrm{th}}$ path. For typical ULAs, $\mathbf{b}\left({\varphi }_k\right)$ is given by:

$$\mathbf{b}\left({\varphi }_k\right)={\left[1,e^{j2\pi {\varphi }_k},\cdots ,e^{j2\pi \left(N_{BS}-1\right){\varphi }_k}\right]}^T.$$

(5)

To reduce the overhead of downlink channel training and uplink channel feedback, we propose a joint estimation scheme, that is the user does not process the received signal and directly feeds it back to the BS [24]. For all K users, the comprehensive signal ${\mathbf{X}}_{m_T,m_R}^{BS}\in {\mathbb{C}}^{N_S\times L}$ received by the BS is:

$$\begin{array}{cc}{\mathbf{X}}_{m_T,m_R}^{BS}\hfill & ={\mathbf{W}}_{R,m_R}^H{\displaystyle \sum _{k=1}^K}{\mathbf{h}}_U^k{\mathbf{y}}_k\hfill \\ \hfill & ={\mathbf{W}}_{R,m_R}^H{\displaystyle \sum _{k=1}^K}{\beta }_k\mathbf{b}\left({\varphi }_k\right){\alpha }_k{\mathbf{a}}^H\left({\theta }_k\right){\mathbf{F}}_{T,m_T}{\mathbf{S}}_{m_T,m_R}+{\mathbf{N}}_{m_T,m_R}^{BS}\hfill \end{array}$$

(6)

where $K\ll N_{BS}$ is the number of complex paths, ${\mathbf{W}}_{R,m_R}={\left[{\mathbf{W}}_{RF}\right]}_{R,m_R}{\left[{\mathbf{W}}_{BB}\right]}_{R,m_R}\in {\mathbb{C}}^{N_{BS}\times N_S}$ is the received hybrid combining matrix in the ${m_R}^{\mathrm{th}}$ time slot, ${\mathbf{W}}_{RF}$ and ${\mathbf{W}}_{BB}$ are baseband combiners, and ${\mathbf{N}}_{m_T,m_R}^{BS}\in {\mathbb{C}}^{N_S\times L}$ is the noise received by the BS after combining. By defining $r_k={\alpha }_k{\beta }_k$ and synthesizing all K users, we can further obtain:

$${\mathbf{X}}_{m_T,m_R}^{BS}={\mathbf{W}}_{R,m_R}^H\mathbf{B}\left(\mathit{\varphi }\right)diag\left(\mathbf{r}\right){\mathbf{A}}^H\left(\mathit{\theta }\right){\mathbf{F}}_{T,m_T}{\mathbf{S}}_{m_T,m_R}+{\mathbf{N}}_{m_T,m_R}^{BS}$$

(7)

where $\mathit{\varphi }={\left[{\varphi }_1,{\varphi }_2,\cdots ,{\varphi }_k\right]}^T\in {\mathbb{C}}^{K\times 1}$, $\mathit{\theta }={\left[{\theta }_1,{\theta }_2,\cdots ,{\theta }_k\right]}^T\in {\mathbb{C}}^{K\times 1}$, $\mathbf{r}={\left[r_1,r_2,\cdots ,r_k\right]}^T\in {\mathbb{C}}^{K\times 1}$, the steering matrix $\mathbf{B}\left(\mathit{\varphi }\right)\in {\mathbb{C}}^{N_{BS}\times K}$ and $\mathbf{A}\left(\mathit{\theta }\right)\in {\mathbb{C}}^{N_{BS}\times K}$ are given by:

$$\mathbf{B}\left(\mathit{\varphi }\right)=\left[\mathbf{b}\left({\varphi }_1\right),\mathbf{b}\left({\varphi }_2\right),\cdots ,\mathbf{b}\left({\varphi }_k\right)\right]$$

(8)

$$\mathbf{A}\left(\mathit{\theta }\right)=\left[\mathbf{a}\left({\theta }_1\right),\mathbf{a}\left({\theta }_2\right),\cdots ,\mathbf{a}\left({\theta }_k\right)\right].$$

(9)

We further process the signal ${\mathbf{X}}_{m_T,m_R}^{BS}$ in (7) to make it multiply right by ${\mathbf{S}}_{m_T,m_R}^H$; we have:

$${\mathbf{X}}_{m_T,m_R}={\mathbf{X}}_{m_T,m_R}^{BS}{\mathbf{S}}_{m_T,m_R}^H={\mathbf{W}}_{R,m_R}^H\mathbf{B}\left(\mathit{\varphi }\right)diag\left(\mathbf{r}\right){\mathbf{A}}^H\left(\mathit{\theta }\right){\mathbf{F}}_{T,m_T}+{\mathbf{N}}_{m_T,m_R}.$$

(10)

Next, we start to collect and arrange the signal ${\mathbf{X}}_{m_T,m_R}$ in (10). By collecting the pilots in the $M_T$ time slots of the transmitter, we can get:

$$\begin{array}{cc}{\mathbf{X}}_{m_R}\hfill & =\left[{\mathbf{X}}_{1,m_R},{\mathbf{X}}_{2,m_R},\cdots ,{\mathbf{X}}_{M_T,m_R}\right]\hfill \\ \hfill & ={\mathbf{W}}_{R,m_R}^H\mathbf{B}\left(\mathit{\varphi }\right)diag\left(\mathbf{r}\right){\mathbf{A}}^H\left(\mathit{\theta }\right){\mathbf{F}}_T+{\mathbf{N}}_{m_R}\hfill \end{array}$$

(11)

where ${\mathbf{F}}_T=\left[{\mathbf{F}}_{T,1},\cdots ,{\mathbf{F}}_{T,M_T}\right]\in {\mathbb{C}}^{N_{BS}\times N_S{M}_T}$, ${\mathbf{N}}_{m_R}=\left[{\mathbf{N}}_{1,m_R},\cdots ,{\mathbf{N}}_{M_T,m_R}\right]\in {\mathbb{C}}^{N_S\times N_S{M}_T}$. By dealing with the signal ${\mathbf{X}}_{m_R}$ in the $M_R$ time slots of the receiver, we have after arranging:

$$\begin{array}{cc}\mathbf{X}\hfill & ={\left[{\mathbf{X}}_1,{\mathbf{X}}_1,\cdots ,{\mathbf{X}}_{M_R}\right]}^T\hfill \\ \hfill & ={\mathbf{W}}_R\mathbf{B}\left(\mathit{\varphi }\right)diag\left(\mathbf{r}\right){\mathbf{A}}^H\left(\mathit{\theta }\right){\mathbf{F}}_T+\mathbf{N}\hfill \end{array}$$

(12)

where ${\mathbf{W}}_R\in {\mathbb{C}}^{N_S{M}_R\times N_{BS}}$ and $\mathbf{N}\in {\mathbb{C}}^{N_S{M}_R\times N_S{M}_T}$ are expressed as:

$${\mathbf{W}}_R=\left[\begin{array}{c}{\mathbf{W}}_{R,1}^H\\ ⋮\\ {\mathbf{W}}_{R,M_R}^H\end{array}\right],\mathbf{N}=\left[\begin{array}{c}{\mathbf{N}}_1\\ ⋮\\ {\mathbf{N}}_{M_R}\end{array}\right].$$

(13)

Now, the signal received by the BS in (10) is written in a more compact form as:

$$\mathbf{X}={\mathbf{W}}_R\mathbf{B}\left(\mathit{\varphi }\right)diag\left(\mathbf{r}\right){\mathbf{A}}^H\left(\mathit{\theta }\right){\mathbf{F}}_T+\mathbf{N}$$

(14)

where $\mathbf{X}\in {\mathbb{C}}^{N_S{M}_R\times N_S{M}_T}$; the combined channel matrix $\mathbf{H}\in {\mathbb{C}}^{N_{BS}\times N_{BS}}$ can be written as:

$$\mathbf{H}=\mathbf{B}\left(\mathit{\varphi }\right)diag\left(\mathbf{r}\right){\mathbf{A}}^H\left(\mathit{\theta }\right).$$

(15)

3. Joint Iterative Channel Estimation Scheme

In this section, we firstly transform the above channel estimation problem into a new algorithm form. Then, we adopt the SVD preconditioning proposal to reduce the computational complexity. To improve algorithm performance, we give an angle estimation optimization scheme by optimizing the step-length and using a combination of the gradient descent algorithm and Newton method. Finally, we propose a scheme for successfully separating the downlink channel from the combined channel matrix.

3.1. Algorithm Formulation

The estimation of the combined channel matrix $\mathbf{H}$ in (15) is equivalent to the estimation of combined path gain $\mathbf{r}$ and the two normalized spatial angles $\mathit{\theta }$ and $\mathit{\varphi }$ for all paths. Due to the sparseness of the combined channel matrix $\mathbf{H}$ in the angle domain [34], such a problem can be easily expressed as:

$$\begin{array}{c}\underset{\mathbf{r},\mathit{\theta },\mathit{\varphi }}{min}{∥\mathbf{r}∥}_0\hfill \\ s.t.{∥\mathbf{X}-{\mathbf{W}}_R\mathbf{H}{\mathbf{F}}_T∥}_F⩽\gamma \hfill \end{array}$$

(16)

where ${∥\mathbf{r}∥}_0$ is the number of non-zero components of $\mathbf{r}$, which also represents the number of estimated paths K, and $\gamma$ is the error tolerance parameter related to noise.

Considering the low computational efficiency of finding the optimal solution in (16), we propose to replace the ${\ell }_0$-norm with the use of the logarithmic sum function [25,35]:

$$\begin{array}{c}\underset{\mathbf{r},\mathit{\theta },\mathit{\varphi }}{min}P\left(\mathbf{r}\right)=\sum _{k=1}^{K}log({\left|r_k\right|}^2+\xi )\hfill \\ s.t.{∥\mathbf{X}-{\mathbf{W}}_R\mathbf{H}{\mathbf{F}}_T∥}_F⩽\gamma \hfill \end{array}$$

(17)

where $r_k$ is the $k^{\mathrm{th}}$ element of the gain $\mathbf{r}$ and $\xi >0$ is a positive parameter to ensure that the definition of the log-sum function is valid. By adding a data fitting parameter $\mu >0$, the difficulty in (17) can be formulated as an unconstrained optimization problem, which yields the following optimization:

$$\underset{\mathbf{r},\mathit{\theta },\mathit{\varphi }}{min}L\left(\mathbf{r},\mathit{\theta },\mathit{\varphi }\right)=P\left(\mathbf{r}\right)+\mu {∥\mathbf{X}-{\mathbf{W}}_R\mathbf{H}{\mathbf{F}}_T∥}_F^2.$$

(18)

Due to the need to ensure the monotonically decreasing characteristic, we introduce a suitable iterative function instead of the log-sum function [26,36]:

$$\underset{\mathbf{r},\mathit{\theta },\mathit{\varphi }}{min}{Z}^{\left(t\right)}\left(\mathbf{r},\mathit{\theta },\mathit{\varphi }\right)={\mu }^{-1}{\mathbf{r}}^H{\mathbf{G}}^{\left(t\right)}\mathbf{r}+{∥\mathbf{X}-{\mathbf{W}}_R\mathbf{H}{\mathbf{F}}_T∥}_F^2$$

(19)

where t is the number of iterations and ${\mathbf{G}}^{\left(t\right)}$ is a diagonal matrix, which is defined as:

$${\mathbf{G}}^{\left(t\right)}\triangleq diag\left[{\phi }_1,{\phi }_2,\cdots ,{\phi }_k\right]$$

(20)

where ${\phi }_k=\frac{1}{\left|{\tilde{r}}_k^{\left(t\right)}\right|+\xi }$, ${\tilde{r}}_k^{\left(t\right)}$ is the path gain estimate of the $k^{\mathrm{th}}$ user at the $t^{\mathrm{th}}$ iteration. The minimization of the log-sum function $L\left(\mathbf{r},\mathit{\theta },\mathit{\varphi }\right)$ in (18) is equivalent to the minimization of the surrogate function $Z^{\left(t\right)}\left(\mathbf{r},\mathit{\theta },\mathit{\varphi }\right)$ in (19), which is proven as follows. To get better estimates ${\tilde{\mathbf{r}}}^{\left(t+1\right)}$, ${\tilde{\mathit{\theta }}}^{\left(t+1\right)}$, ${\tilde{\mathit{\varphi }}}^{\left(t+1\right)}$ in the $t^{\mathrm{th}}$ iteration and ensure the convergence of the function, the following inequality can be shown:

$$Z^{\left(t\right)}\left({\tilde{\mathbf{r}}}^{\left(t+1\right)},\left.\left({\tilde{\mathit{\theta }}}^{\left(t+1\right)},{\tilde{\mathit{\varphi }}}^{\left(t+1\right)}\right)\right|{\tilde{\mathbf{r}}}^{\left(t\right)}\right)⩽Z^{\left(t\right)}\left({\tilde{\mathbf{r}}}^{\left(t\right)},\left.\left({\tilde{\mathit{\theta }}}^{\left(t\right)},{\tilde{\mathit{\varphi }}}^{\left(t\right)}\right)\right|{\tilde{\mathbf{r}}}^{\left(t\right)}\right).$$

(21)

Then, we can get:

$$L\left({\tilde{\mathbf{r}}}^{\left(t+1\right)},{\tilde{\mathit{\theta }}}^{\left(t+1\right)},{\tilde{\mathit{\varphi }}}^{\left(t+1\right)}\right)-\mu Z^{\left(t\right)}\left({\tilde{\mathbf{r}}}^{\left(t+1\right)},\left.\left({\tilde{\mathit{\theta }}}^{\left(t+1\right)},{\tilde{\mathit{\varphi }}}^{\left(t+1\right)}\right)\right|{\tilde{\mathbf{r}}}^{\left(t+1\right)}\right)=\sum _{k=1}^K\left[log\left({\left|{\tilde{r}}_k^{\left(t+1\right)}\right|}^2+\xi \right)-{\left|{\tilde{r}}_k^{\left(t+1\right)}\right|}^2{\phi }_k\right].$$

(22)

Since the maximum value is reached at ${\tilde{\mathbf{r}}}^{(t+1)}={\tilde{\mathbf{r}}}^{\left(t\right)}$, we have:

$$\sum _{k=1}^K\left[log\left({\left|{\tilde{r}}_k^{\left(t+1\right)}\right|}^2+\xi \right)-{\left|{\tilde{r}}_k^{\left(t+1\right)}\right|}^2{\phi }_k\right]⩽\sum _{k=1}^K\left[log\left({\left|{\tilde{r}}_k^{\left(t+1\right)}\right|}^2+\xi \right)-{\left|{\tilde{r}}_k^{\left(t\right)}\right|}^2{\phi }_k\right].$$

(23)

Combining (21) and (22), we can finally get:

$$L\left({\tilde{\mathbf{r}}}^{\left(t+1\right)},{\tilde{\mathit{\theta }}}^{\left(t+1\right)},{\tilde{\mathit{\varphi }}}^{\left(t+1\right)}\right)⩽L\left({\tilde{\mathbf{r}}}^{\left(t\right)},{\tilde{\mathit{\theta }}}^{\left(t\right)},{\tilde{\mathit{\varphi }}}^{\left(t\right)}\right)$$

(24)

which explains the correctness of the above assumption.

By optimizing the path gain $\mathbf{r}$ in (19) through the derivative method [26], we can obtain the optimal point of $\mathbf{r}$ and the corresponding optimal value of $Z^{\left(t\right)}$, that is,

$${\mathbf{r}}_{pe}^{\left(t\right)}(\mathit{\theta },\mathit{\varphi })={({\mu }^{-1}{\mathbf{G}}^{\left(t\right)}+\sum _{i=1}^{N_S{M}_T}{\mathbf{Q}}_i^H{\mathbf{Q}}_i)}^{-1}\left(\sum _{i=1}^{N_S{M}_T}{\mathbf{Q}}_i^H{\mathbf{x}}_i\right)$$

(25)

$$Z_{pe}^{\left(t\right)}(\mathit{\theta },\mathit{\varphi })=\sum _{i=1}^{N_s{M}_T}{\mathbf{x}}_i^H{\mathbf{x}}_i-{\left(\sum _{i=1}^{N_s{M}_T}{\mathbf{Q}}_i^H{\mathbf{x}}_i\right)}^H{({\mu }^{-1}{\mathbf{G}}^{\left(t\right)}+\sum _{i=1}^{N_s{M}_T}{\mathbf{Q}}_i^H{\mathbf{Q}}_i)}^{-1}\left(\sum _{i=1}^{N_s{M}_T}{\mathbf{Q}}_i^H{\mathbf{x}}_i\right)$$

(26)

where ${\mathbf{x}}_i$ is the column vector of the signal $\mathbf{X}$ received by the BS, ${\mathbf{Q}}_i={\mathbf{W}}_R\mathbf{B}\left(\mathit{\varphi }\right)diag\left({\mathbf{A}}^H\left(\mathit{\theta }\right){\mathbf{f}}_T\right)$, and ${\mathbf{f}}_T$ is the column vector of the hybrid precoding matrix ${\mathbf{F}}_T$. After that, we only need to consider the optimization of the normalized spatial angles $\mathit{\theta }$ and $\mathit{\varphi }$ in (26), which will be discussed in the next subsection.

3.2. Initial Angle Preprocessing

To optimize the spatial angle, we propose a preconditioning algorithm based on SVD [37,38] to select the initial value of the spatial angle effectively. Compared with using all $N_{MS}{N}_{BS}$ angle domain grids as the initial candidate value, the proposed algorithm can rapidly find the angle domain grids closest to the true orientation angle, which can significantly reduce the computational complexity of the subsequent algorithm. Specifically, we apply SVD to the matrix $\mathbf{X}$ in (14) and get $\mathbf{X}=\mathbf{U}\Sigma {\mathbf{V}}^H$, where $\mathbf{U}$ and $\mathbf{V}$ are unitary matrices that satisfy ${\mathbf{U}}^H\mathbf{U}={\mathbf{I}}_{N_S{M}_R\times N_S{M}_R}$ and ${\mathbf{V}}^H\mathbf{V}={\mathbf{I}}_{N_S{M}_T\times N_S{M}_T}$, respectively, $\Sigma =diag\left({\kappa }_1,{\kappa }_2,\cdots ,{\kappa }_{min(N_S{M}_R,N_S{M}_T)}\right)\in {\mathbb{R}}^{N_S{M}_R\times N_S{M}_T}$ is a diagonal matrix whose value on the diagonal is the singular value of $\mathbf{X}$, and ${\kappa }_1⩾{\kappa }_2⩾\cdots ⩾{\kappa }_{min(N_S{M}_R,N_S{M}_T)}⩾0$. The signal is expressed in a more compact form as:

$$\mathbf{X}=\left({\mathbf{W}}_R\mathbf{B}\left(\mathit{\varphi }\right)\right)diag\left(\mathbf{r}\right){\left({\mathbf{F}}_T^H\mathbf{A}\left(\mathit{\theta }\right)\right)}^H+\mathbf{N}.$$

(27)

Suppose $l=1,2,\cdots ,K$ the larger the K singular value and their homologous singular vectors are roughly determined by K paths, we can infer:

$${\mathbf{u}}_l=\frac{{\mathbf{W}}_R\mathbf{b}\left({\varphi }_l\right)}{{∥{\mathbf{W}}_R\mathbf{b}\left({\varphi }_l\right)∥}_2},{\mathbf{v}}_l=\frac{{\mathbf{F}}_T^H\mathbf{a}\left({\theta }_l\right)}{{∥{\mathbf{F}}_T^H\mathbf{a}\left({\theta }_l\right)∥}_2}$$

(28)

where ${\mathbf{u}}_l$ and ${\mathbf{v}}_l$ are the $l^{\mathrm{th}}$ column of $\mathbf{U}$ and $\mathbf{V}$, respectively; the singular value can be written as:

$${\kappa }_l=\left|r_l\right|{∥{\mathbf{W}}_R\mathbf{b}\left({\varphi }_l\right)∥}_2{∥{\mathbf{F}}_T^H\mathbf{a}\left({\theta }_l\right)∥}_2.$$

(29)

Then, the spatial angles are normalized by $\left({\theta }_l,{\varphi }_l\right)\in \left\{\frac{\left(i-1\right)}{N_{BS}},i=1,2,\cdots ,N_{BS}\right\}$ and assumed to lie in the quantized points; we have:

$${\mathbf{u}}_l\approx {\mathbf{W}}_R{\mathbf{E}}_R\mathbf{b}\left({\varphi }_l\right),{\mathbf{v}}_l\approx {\mathbf{F}}_T^H{\mathbf{E}}_T\mathbf{a}\left({\theta }_l\right)$$

(30)

where ${\mathbf{E}}_R\in {\mathbb{C}}^{N_{BS}\times N_{BS}}$ and ${\mathbf{E}}_T\in {\mathbb{C}}^{N_{BS}\times N_{BS}}$ are the discrete Fourier transform matrices; we can further obtain the coarse estimates of angle, which can be formulated as:

$$\begin{array}{c}{\varphi }_l^{\left(1\right)}=\frac{arg{max}_{q=1}^{N_{BS}}{\mathbf{d}}_R\left(q\right)-1}{N_{BS}}\hfill \\ {\theta }_l^{\left(1\right)}=\frac{arg{max}_{q=1}^{N_{BS}}{\mathbf{d}}_T\left(q\right)-1}{N_{BS}}\hfill \end{array}$$

(31)

where:

$$\begin{array}{c}{\mathbf{d}}_R={\left({\mathbf{W}}_R{\mathbf{E}}_R\right)}^H{\mathbf{W}}_R{\mathbf{E}}_R\mathbf{b}\left({\varphi }_l\right)\hfill \\ {\mathbf{d}}_T={\left({\mathbf{F}}_T^H{\mathbf{E}}_T\right)}^H{\mathbf{F}}_T^H{\mathbf{E}}_T\mathbf{a}\left({\theta }_l\right).\hfill \end{array}$$

(32)

At this point, we get the initial candidates for the spatial angle. Furthermore, due to the uncertainty of the number of paths of the estimated channel at the beginning, we set $L_{init}⩾K$, $l=1,2,\cdots ,L_{init}$ to ensure the accuracy of our estimated path, and some incorrect paths will be cut off in the proposed algorithm. The iterative search scheme of angles is explained in the next subsection.

3.3. Step-Length Optimization

In the previous section, it can be seen that the objective function $Z^{\left(t\right)}\left(\mathbf{r},\mathit{\theta },\mathit{\varphi }\right)$ in (19) is the weighted sum of two parts, where $\mu$ is the regularization parameter [39] that affects the tradeoff to some extent. In the proposed algorithm, $\mu$ is simplified as:

$$\mu =min\left[{\left({\omega }^{\left(t\right)}\right)}^{-1}e,{\mu }_{max}\right]$$

(33)

where e is a constant scale factor, and the squared residual at the $t^{\mathrm{th}}$ iteration ${\omega }^{\left(t\right)}$ is expressed as:

$${\omega }^{\left(t\right)}={∥\mathbf{X}-{\mathbf{W}}_R\mathbf{B}\left({\mathit{\varphi }}^{\left(t\right)}\right)diag\left({\mathbf{r}}_{pe}^{\left(t\right)}\right){\mathbf{A}}^H\left({\mathit{\theta }}^{\left(t\right)}\right){\mathbf{F}}_T∥}_F^2.$$

(34)

The main task is to search for new estimates ${\mathit{\theta }}^{\left(t+1\right)}$ and ${\mathit{\varphi }}^{\left(t+1\right)}$ from the neighborhood of ${\mathit{\theta }}^{\left(t\right)}$ and ${\mathit{\varphi }}^{\left(t\right)}$ at the $t^{\mathrm{th}}$ iteration, so that the objective function $Z^{\left(t\right)}$ becomes smaller and eventually stabilizes. Considering the characteristics of the fast convergence speed of the Newton descent method and the high accuracy of the steepest descent method, we try to suggest combining the two for better performance; this searching can be accomplished by:

$$\begin{array}{c}{\mathit{\varphi }}^{\left(t+1\right)}={\mathit{\varphi }}^{\left(t\right)}-{\rho }^{\left(t\right)}·{\mathbf{d}}^{\left(t\right)}\hfill \\ {\mathit{\theta }}^{\left(t+1\right)}={\mathit{\theta }}^{\left(t\right)}-{\rho }^{\left(t\right)}·{\mathbf{d}}^{\left(t\right)}\hfill \end{array}$$

(35)

where ${\rho }^{\left(t\right)}$ is the step-length at the $t^{\mathrm{th}}$ iteration and ${\mathbf{d}}^{\left(t\right)}$ is the corresponding descent direction. To balance the high computational complexity of the Newton iterative algorithm, we consider using the Newton direction as the descending direction to achieve fast convergence in the first iteration, that is denote ${\mathbf{d}}^{\left(t\right)}={\mathbf{d}}_N^{\left(t\right)}$ when $t=1$; the expression ${\mathbf{d}}_N^{\left(t\right)}$ is:

$${\mathbf{d}}_N^{\left(t\right)}={\nabla }^2{Z}_{pe}^{\left(t\right)}{({\mathit{\theta }}^{\left(t\right)},{\mathit{\varphi }}^{\left(t\right)})}^{-1}\nabla Z_{pe}^{\left(t\right)}({\mathit{\theta }}^{\left(t\right)},{\mathit{\varphi }}^{\left(t\right)}).$$

(36)

After that, we choose the gradient direction to achieve high accuracy and set ${\mathbf{d}}^{\left(t\right)}={\mathbf{d}}_G^{\left(t\right)}$; the expression ${\mathbf{d}}_G^{\left(t\right)}$ is:

$${\mathbf{d}}_G^{\left(t\right)}=\nabla Z_{pe}^{\left(t\right)}({\mathit{\theta }}^{\left(t\right)},{\mathit{\varphi }}^{\left(t\right)})$$

(37)

it can be seen that the search direction involves the gradient calculation of the iterative function $Z_{pe}$. Taking the normalized angle ${\varphi }_k$ as an example, the calculation method is explained as follows.

Define $\mathbf{J}={\displaystyle \sum _{i=1}^{N_S{M}_T}}{\mathbf{Q}}_i^H{\mathbf{x}}_i$, $\mathbf{D}={\mu }^{-1}\mathbf{G}+{\displaystyle \sum _{i=1}^{N_s{M}_T}}{\mathbf{Q}}_i^H{\mathbf{Q}}_i$, then $Z_{pe}$ can be expressed as:

$$Z_{pe}=\sum _{i=1}^{N_s{M}_T}{\mathbf{x}}_i^H{\mathbf{x}}_i-{\mathbf{J}}^H{\mathbf{D}}^{-1}\mathbf{J}.$$

(38)

Take the partial derivative with respect to ${\varphi }_k$, we can obtain:

$$\frac{\partial Z_{pe}}{\partial {\varphi }_k}=-\frac{\partial {\mathbf{J}}^H}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\mathbf{J}-{\mathbf{J}}^H\frac{\partial {\mathbf{D}}^{-1}}{\partial {\varphi }_k}\mathbf{J}-{\mathbf{J}}^H{\mathbf{D}}^{-1}\frac{\partial \mathbf{J}}{\partial {\varphi }_k}$$

(39)

and $\frac{\partial {\mathbf{D}}^{-1}}{\partial {\varphi }_k}=-{\mathbf{D}}^{-1}\frac{\partial \mathbf{D}}{\partial {\varphi }_k}{\mathbf{D}}^{-1}$; we have:

$$\frac{\partial Z_{pe}}{\partial {\varphi }_k}=-\frac{\partial {\mathbf{J}}^H}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\mathbf{J}+{\mathbf{J}}^H{\mathbf{D}}^{-1}\frac{\partial \mathbf{D}}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\mathbf{J}-{\mathbf{J}}^H{\mathbf{D}}^{-1}\frac{\partial \mathbf{J}}{\partial {\varphi }_k}$$

(40)

where:

$$\frac{\partial \mathbf{J}}{\partial {\varphi }_k}=\sum _{i=1}^{N_S{M}_T}\frac{\partial {\mathbf{Q}}_i^H}{\partial {\varphi }_k}{\mathbf{x}}_i$$

(41)

$$\frac{\partial \mathbf{D}}{\partial {\varphi }_k}=\sum _{i=1}^{N_S{M}_T}\left(\frac{\partial {\mathbf{Q}}_i^H}{\partial {\varphi }_k}{\mathbf{Q}}_i+{\mathbf{Q}}_i^H\frac{\partial {\mathbf{Q}}_i}{\partial {\varphi }_k}\right)$$

(42)

$$\frac{\partial {\mathbf{Q}}_i}{\partial {\varphi }_k}=\frac{\partial {\mathbf{W}}_R\mathbf{B}\left(\varphi \right)diag\left({\mathbf{A}}^H\left(\theta \right){\mathbf{f}}_{\mathbf{T}}\right)}{\partial {\varphi }_k}=\left[0\cdots 0{\mathbf{W}}_R\frac{\partial \mathbf{b}\left({\varphi }_k\right)}{\partial {\varphi }_k}{\mathbf{a}}^H\left({\theta }_k\right){\mathbf{f}}_{\mathbf{T}}0\cdots 0\right].$$

(43)

Since there is only a certain transposition relationship between the gradient and the partial derivative, we have the gradient of $Z_{pe}$, and we can further obtain the second derivative:

$$\begin{array}{cc}\hfill \frac{{\partial }^2{Z}_{pe}}{\partial {\varphi }_k^2}=& -\frac{{\partial }^2{\mathbf{J}}^H}{\partial {\varphi }_k^2}{\mathbf{D}}^{-1}\mathbf{J}+2\left(\frac{\partial {\mathbf{J}}^H}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\frac{\partial \mathbf{D}}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\mathbf{J}\right)-2\left(\frac{\partial {\mathbf{J}}^H}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\frac{\partial \mathbf{J}}{\partial {\varphi }_k}\right)\hfill \\ \hfill & -2\left({\mathbf{J}}^H{\mathbf{D}}^{-1}\frac{\partial \mathbf{D}}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\frac{\partial \mathbf{D}}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\mathbf{J}\right)+{\mathbf{J}}^H{\mathbf{D}}^{-1}\frac{{\partial }^2\mathbf{D}}{\partial {\varphi }_k^2}{\mathbf{D}}^{-1}\mathbf{J}\hfill \\ \hfill & +2\left({\mathbf{J}}^H{\mathbf{D}}^{-1}\frac{\partial \mathbf{D}}{\partial {\varphi }_k}{\mathbf{D}}^{-1}\frac{\partial \mathbf{J}}{\partial {\varphi }_k}\right)-{\mathbf{J}}^H{\mathbf{D}}^{-1}\frac{{\partial }^2\mathbf{J}}{\partial {\varphi }_k^2}\hfill \end{array}$$

(44)

where:

$$\frac{{\partial }^2\mathbf{J}}{\partial {\varphi }_k^2}=\sum _{i=1}^{N_S{M}_T}\frac{{\partial }^2{\mathbf{Q}}_i^H}{\partial {\varphi }_k^2}{\mathbf{x}}_i$$

(45)

$$\frac{{\partial }^2\mathbf{D}}{\partial {\varphi }_k^2}=\sum _{i=1}^{N_S{M}_T}\left(\frac{{\partial }^2{\mathbf{Q}}_i^H}{\partial {\varphi }_k^2}{\mathbf{Q}}_i+2\frac{\partial {\mathbf{Q}}_i^H}{\partial {\varphi }_k}\frac{\partial {\mathbf{Q}}_i}{\partial {\varphi }_k}+{\mathbf{Q}}_i^H\frac{{\partial }^2{\mathbf{Q}}_i}{\partial {\varphi }_k^2}\right)$$

(46)

$$\frac{{\partial }^2{\mathbf{Q}}_i}{\partial {\varphi }_k^2}=\left[0\cdots 0{\mathbf{W}}_R\frac{{\partial }^2\mathbf{b}\left({\varphi }_k\right)}{\partial {\varphi }_k^2}{\mathbf{a}}^H\left({\theta }_k\right){\mathbf{f}}_{\mathbf{T}}0\cdots 0\right].$$

(47)

Next, we think about whether we can optimize the iteration step-length in (35) to improve the speed of iteration. Considering that if the optimal step-length of $\mathit{\theta }$ and $\mathit{\varphi }$ is directly solved, the algorithm complexity is very high, and the channel is only related to the path gain $\mathbf{r}$ and the spatial angles $\mathit{\theta }$ and $\mathit{\varphi }$, so when the gain has been optimized, we choose to use the optimal step-length of the channel to approximate the optimal step-length of the angles. The optimization method is as follows.

Denote $\mathbf{x}=vec\left(\mathbf{X}\right)$, $\Omega ={\mathbf{F}}_T^T\otimes {\mathbf{W}}_{\mathbf{R}}$, and given the spatial angles ${\mathit{\theta }}^{\left(t\right)}$ and ${\mathit{\varphi }}^{\left(t\right)}$ at the $t^{\mathrm{th}}$ iteration, we have:

$${\mathbf{h}}^{\left(t\right)}=vec\left(\mathbf{B}\left({\mathit{\varphi }}^{\left(t\right)}\right)diag\left({\mathbf{r}}_{pe}^{\left(t\right)}\right){\mathbf{A}}^H\left({\mathit{\theta }}^{\left(t\right)}\right)\right).$$

(48)

Then, the cost function transformed from (34) is:

$$O^{\left(t\right)}=vec\left({\omega }^{\left(t\right)}\right)={∥\mathbf{x}-\Omega {\mathbf{h}}^{\left(t\right)}∥}^2.$$

(49)

Therefore, such an optimization can be reformulated as:

$${\rho }^{\left(t\right)}=\underset{{\rho }^{\left(t\right)}}{argmin}O\left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)$$

(50)

The key to this problem is how to get the suitable ${\rho }^{\left(t\right)}$; define $\lambda \left({\rho }^{\left(t\right)}\right)=O\left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)$, and take the derivative with respect to $\lambda \left({\rho }^{\left(t\right)}\right)$; we can get:

$$\frac{\partial \lambda \left({\rho }^{\left(t\right)}\right)}{\partial {\rho }^{\left(t\right)}}=\nabla O{\left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)}^H{\mathbf{d}}^{\left(t\right)}$$

(51)

where:

$$\begin{array}{cc}\nabla O\left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)\hfill & =\nabla {∥\mathbf{x}-\Omega \left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)∥}^2\hfill \\ \hfill & =\nabla {\left[\mathbf{x}-\Omega \left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)\right]}^H\left[\mathbf{x}-\Omega \left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)\right]\hfill \\ \hfill & ={\Omega }^H\left[\Omega \left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)-\mathbf{x}\right].\hfill \end{array}$$

(52)

Similarly, we have:

$$\nabla O\left({\mathbf{h}}^{\left(t\right)}\right)={\Omega }^H\left(\Omega {\mathbf{h}}^{\left(t\right)}-\mathbf{x}\right)$$

(53)

(52) can be simplified as:

$$\begin{array}{cc}\nabla O\left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)\hfill & ={\Omega }^H\Omega \left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)-{\Omega }^H\mathbf{x}\hfill \\ \hfill & ={\Omega }^H\Omega \left({\mathbf{h}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{\left(t\right)}\right)+\nabla O\left({\mathbf{h}}^{\left(t\right)}\right)-{\Omega }^H\Omega {\mathbf{h}}^{\left(t\right)}\hfill \\ \hfill & ={\rho }^{\left(t\right)}{\Omega }^H\Omega {\mathbf{d}}^{\left(t\right)}+\nabla O\left({\mathbf{h}}^{\left(t\right)}\right).\hfill \end{array}$$

(54)

Now, we can obtain:

$$\frac{\partial \lambda \left({\rho }^{\left(t\right)}\right)}{\partial {\rho }^{\left(t\right)}}=\nabla O{\left({\mathbf{h}}^{\left(t\right)}\right)}^H{\mathbf{d}}^{\left(t\right)}+{\rho }^{\left(t\right)}{\mathbf{d}}^{{\left(t\right)}^H}{\Omega }^H\Omega {\mathbf{d}}^{\left(t\right)}.$$

(55)

Considering the complexity of the algorithm, the descent direction here is the gradient direction. Set $\frac{\partial \lambda \left({\rho }^{\left(t\right)}\right)}{\partial {\rho }^{\left(t\right)}}=0$; we can finally get:

$${\rho }^{\left(t\right)}=-\frac{\nabla O{\left({\mathbf{h}}^{\left(t\right)}\right)}^H\nabla O\left({\mathbf{h}}^{\left(t\right)}\right)}{\nabla O{\left({\mathbf{h}}^{\left(t\right)}\right)}^H{\Omega }^H\Omega \nabla O\left({\mathbf{h}}^{\left(t\right)}\right)}.$$

(56)

From this, we get the approximate optimal step-length for the angle update. During the iterative searching, the spatial angle and path gain estimates can converge quickly, until the latest estimates are almost the same as the previous ones. The proposed channel estimation scheme is shown in Algorithm 1. We can achieve the purpose of accurate and fast channel estimation, which will be verified in subsequent simulations.

Since the Newton direction is used only once, the computational complexity of each iteration mainly lies in the calculation of the gradient direction and the optimization of the step-length. The complexity to compute the gradient direction is $\mathcal{O}\left(N_s^2{M}_R{M}_T\left(N_{BS}+K\right)K^2\right)$, and the complexity of the step calculation is $\mathcal{O}\left(K^2{N}_{BS}\right)$. As a result, we can conclude that the proposed algorithm has the complexity $\mathcal{O}(N_s^2{M}_R{M}_T\left(N_{BS}+K\right)K^2+K^2{N}_{BS})$.

In the above algorithm, we successfully estimated the space angle $\mathit{\theta }$, $\mathit{\varphi }$ and the combined gain $\mathbf{r}$, where $r_k={\alpha }_k{\beta }_k$. Next, the solution to the problem of how to separate the downlink gain ${\alpha }_k$ from the combined gain is revealed.

Specifically, we adopt the unit symbol pilot $s_k=1$ sent by the UEs to the BS via the uplink channel, and the signal $\mathbf{Y}\in {\mathbb{C}}^{N_S{M}_R\times 1}$ received by the BS is written as:

$$\begin{array}{cc}\mathbf{Y}\hfill & ={\mathbf{W}}_R\left[{\mathbf{h}}_U^1{s}_1+{\mathbf{h}}_U^2{s}_2+\cdots +{\mathbf{h}}_U^{\mathrm{k}}{s}_k\right]\hfill \\ \hfill & ={\mathbf{W}}_R\left[{\beta }_1\mathbf{b}\left({\varphi }_1\right)+{\beta }_2\mathbf{b}\left({\varphi }_2\right)+\cdots +{\beta }_k\mathbf{b}\left({\varphi }_k\right)\right]\hfill \\ \hfill & =\mathbf{T}{\left[{\beta }_1,{\beta }_2,\cdots ,{\beta }_k\right]}^T\hfill \end{array}$$

(57)

where $\mathbf{T}={\mathbf{W}}_R\left[\mathbf{b}\left({\varphi }_1\right),\mathbf{b}\left({\varphi }_2\right),\cdots ,\mathbf{b}\left({\varphi }_k\right)\right]\in {\mathbb{C}}^{N_S{M}_R\times K}$; when $rank\left(\mathbf{T}\right)⩾K$, we have $\mathit{\beta }={\mathbf{T}}^{†}\mathbf{Y}$, where $\mathit{\beta }={\left[{\beta }_1,{\beta }_2,\cdots ,{\beta }_k\right]}^T$, we obtain uplink gain ${\beta }_k$, and  the downlink gain ${\alpha }_k$ can be separated by utilizing ${\alpha }_k=\frac{r_k}{{\beta }_k}$. Since ${\mathbf{W}}_R$ can be easily designed as a matrix whose columns are full rank, the above conditions can be achieved.

Algorithm 1: The optimized iterative algorithm.

Input: Received signals with noise $\mathbf{X}$, orthogonal pilot signals ${\mathbf{S}}_{m_T,m_R}$, coding matrix ${\mathbf{W}}_R$ and ${\mathbf{F}}_T$, the number of detection paths $L_{init}$, threshold $\Gamma$ and $\epsilon$. Output: Estimated angles and gains of all paths. First stage • Perform $\left[\mathbf{U},\Sigma ,\mathbf{V}\right]=SVD\left(\mathbf{X}\right)$; • Compute ${\mathbf{d}}_R={\left({\mathbf{W}}_R{\mathbf{E}}_R\right)}^H{\mathbf{W}}_R{\mathbf{E}}_R\mathbf{b}\left({\varphi }_l\right)$, ${\mathbf{d}}_T={\left({\mathbf{F}}_T^H{\mathbf{E}}_T\right)}^H{\mathbf{F}}_T^H{\mathbf{E}}_T\mathbf{a}\left({\theta }_l\right)$; • Initialize ${\varphi }_l^{\left(1\right)}=\frac{arg{max}_{q=1}^{N_{BS}}{\mathbf{d}}_R\left(q\right)-1}{N_{BS}}$, ${\theta }_l^{\left(1\right)}=\frac{arg{max}_{q=1}^{N_{BS}}{\mathbf{d}}_T\left(q\right)-1}{N_{BS}}$; • Estimate gain ${\mathbf{r}}_{pe}^{\left(1\right)}$ by (25), and trim path if ${\mathbf{r}}_{pe}^{\left(1\right)}<\Gamma$. Second stage 1. Repeat 2. Update $\mu =min\left[{\left({\omega }^{\left(t\right)}\right)}^{-1}e,{\mu }_{max}\right]$; 3. Build the function $Z_{pe}^{\left(t\right)}(\mathit{\theta },\mathit{\varphi })$ by (26); 4. If $t=1$, then 5. ${\mathbf{d}}^{\left(t\right)}={\mathbf{d}}_N^{\left(t\right)}$, ${\mathbf{d}}_N^{\left(t\right)}={\nabla }^2{Z}_{pe}^{\left(t\right)}{({\mathit{\theta }}^{\left(t\right)},{\mathit{\varphi }}^{\left(t\right)})}^{-1}\nabla Z_{pe}^{\left(t\right)}({\mathit{\theta }}^{\left(t\right)},{\mathit{\varphi }}^{\left(t\right)})$; 6. Else 7. ${\mathbf{d}}^{\left(t\right)}={\mathbf{d}}_G^{\left(t\right)}$, ${\mathbf{d}}_G^{\left(t\right)}=\nabla Z_{pe}^{\left(t\right)}({\mathit{\theta }}^{\left(t\right)},{\mathit{\varphi }}^{\left(t\right)})$; 8. End if 9. Step optimization calculation: 9.1 $\mathbf{x}=vec\left(\mathbf{X}\right)$, ${\mathbf{h}}^{\left(t\right)}=vec\left({\mathbf{H}}^{\left(t\right)}\right)$; 9.2 Compute $\nabla O\left({\mathbf{h}}^{\left(t\right)}\right)={\Omega }^H\left(\Omega {\mathbf{h}}^{\left(t\right)}-\mathbf{x}\right)$; 9.3 Compute ${\rho }^{\left(t\right)}=-\frac{\nabla O{\left({\mathbf{h}}^{\left(t\right)}\right)}^H\nabla O\left({\mathbf{h}}^{\left(t\right)}\right)}{\nabla O{\left({\mathbf{h}}^{\left(t\right)}\right)}^H{\Omega }^H\Omega \nabla O\left({\mathbf{h}}^{\left(t\right)}\right)}$. 10. ${\mathit{\theta }}^{\left(t+1\right)}={\mathit{\theta }}^{\left(t\right)}-{\rho }^{\left(t\right)}·{\mathbf{d}}^{\left(t\right)}$, ${\mathit{\varphi }}^{\left(t+1\right)}={\mathit{\varphi }}^{\left(t\right)}-{\rho }^{\left(t\right)}·{\mathbf{d}}^{\left(t\right)}$; 11. Estimate the path gain ${\mathbf{r}}_{pe}^{\left(t+1\right)}$, and streamline path if ${\mathbf{r}}_{pe}^{\left(t+1\right)}<\Gamma$; 12. Until $K^{\left(t\right)}=K^{\left(t+1\right)}$, and ${∥{\mathbf{r}}^{\left(t+1\right)}-{\mathbf{r}}^{\left(t\right)}∥}_2<\epsilon$.

4. Simulation Results

In this section, we demonstrate the simulation comparisons under different parameter scenarios, which can prove the effectiveness of the proposed step-length optimization-based joint iterative scheme. In particular, simulation results are provided to verify that the proposed algorithm can outperform the previous methods in certain aspects, as in [26,27]. Some default parameters are set to $d=0.5$, $N_S=4$, and $L=24$. Moreover, the number of antennas, UEs, and time slots is set to $N_{BS}=64$, $K=3$, and $M_R=M_T=12$, respectively, which may change in the form of variables in subsequent simulations, and we set the variable $M_P$ to represent the time slot for better presentation. As in [2,40], the signal-to-noise ratio (SNR) is defined as ${\sigma }_s^2/{\sigma }_n^2$, where ${\sigma }_s^2$ is the signal power. I denotes the number of Monte Carlo runs, and the normalized mean squared error (NMSE) is defined as:

$$NMSE=\frac{1}{I}\left(\sum _{i=1}^I\frac{{∥{\widehat{\mathbf{H}}}_i-{\mathbf{H}}_i∥}_F^2}{{∥{\mathbf{H}}_i∥}_F^2}\right)$$

(58)

Figure 2 shows the convergence performance of the optimized scheme compared with the conventional super-resolution (SR) algorithm, which is reflected by the relationship between NMSE and the number of iterations for three SNR values. It is obvious that the proposed scheme can achieve convergence with a small number of iterations for each SNR value. For instance, the proposed algorithm completes convergence in about 20 iterations at an SNR of 20 dB, which also fully proves the acceleration brought by optimization.

Figure 3 compares the NMSE performance against the SNR. We assume the system design parameters $N_{BS}=32$ and $M_R=M_T=M_P$, as shown in Figure 3, and the accuracy of the proposed solution is similar to that of the SR algorithm in different time slots, which proves that the proposed optimization does not sacrifice performance indicators. We can see that there is a certain proportional relationship between the time slot and the training pilot, so the NMSE performance becomes better as the number of time slots increases. In addition, the NMSE of the proposed and SR schemes is also shown in

Table 1. NMSE of the proposed scheme and SR algorithms in different time slots.

(SNR) (dB)	0	5	10	15	20	25	30
SR,
$M_P=6$	4.4983e-3	1.2481e-3	3.4718e-4	1.0425e-4	3.3272e-5	1.1428e-5	5.9694e-6
Proposed,
$M_P=6$	4.2355e-3	1.1838e-3	3.2268e-4	1.0394e-4	3.5274e-5	1.1674e-5	5.7921e-6
SR,
$M_P=8$	2.0730e-3	6.0159e-4	1.9159e-4	5.9361e-5	1.8088e-5	5.8977e-6	2.0542e-6
Proposed,
$M_P=8$	2.1037e-3	6.0318e-4	1.8060e-4	5.7340e-5	1.7180e-5	5.5994e-6	2.1856e-6
SR,
$M_P=10$	1.3365e-3	3.9811e-4	1.2478e-4	4.1041e-5	1.2613e-5	4.1339e-6	1.3255e-6
Proposed,
$M_P=10$	1.3688e-3	3.7503e-4	1.2548e-4	3.9852e-5	1.2220e-5	3.9532e-6	1.3329e-6
SR,
$M_P=12$	8.0790e-4	2.4650e-4	8.0040e-5	2.7415e-5	8.0607e-6	2.6785e-6	8.4921e-7
Proposed,
$M_P=12$	8.1252e-4	2.2682e-4	7.9436e-5	2.5461e-5	8.3286e-6	2.5191e-6	7.7909e-7

for further comparison. It is clearly observed that accurate channel estimation can be achieved.

Figure 4 depicts the iteration time of the proposed algorithm on the basis of the hypothetical scenario in Figure 3, which can more intuitively reflect that the speed of the proposed scheme is significantly improved and the computational complexity is reduced. For example, when the SNR is 20 and the time slot is 10, the time taken by the proposed algorithm is 2.0534 s and the SR is 5.0885 s; it can be fully seen that our method achieves fast estimation. We can also see that as the SNR and time slot increase, the time consumed increase, and that is because the accuracy is also improved.

Figure 5 displays the NMSE performance comparison with BCS in the case of different numbers of UEs. Since it optimizes all the spatialization angles simultaneously and eliminates some false paths, the proposed scheme can obtain better NMSE performance, and the processing time of the BCS method is significantly higher than the proposed scheme under the same conditions. In addition, we can see that when the number of users increases, the channel estimation accuracy of the proposed algorithm decreases; but the decrease is small, and the proposed algorithm can still effectively estimate the channel.

Figure 6 compares the angular mean squared error (RMSE) performance under various UEs and antennas. The RMSE is given by:

$$\begin{array}{c}\hfill RMSE=\frac{1}{K}\sum _{k=1}^K\sqrt{\frac{1}{I}\sum _{i=1}^I\left[{\left({\widehat{\theta }}_{k,i}-{\theta }_k\right)}^2+{\left({\widehat{\varphi }}_{k,i}-{\varphi }_k\right)}^2\right]}\end{array}$$

(59)

As SNR increases, we observe that the RMSE of SVD remains constant, and its computational complexity is lower; but our algorithm yields the best performance. This is because the SNR is related to the entire angle domain grids, and our SVD method finds the best matching grid to get coarse estimates. Similarly to Figure 5, the RMSE performance of channel estimation will decrease with the increase of users, and in contrast, the number of antennas is directly proportional to the estimated accuracy.

Figure 7 shows the performance of the gain mean squared error (GMSE) of the proposed algorithm in different time slots, and GMSE is expressed as $GMSE=\frac{1}{K}{\displaystyle \sum _{k=1}^K}\sqrt{\frac{1}{I}{\displaystyle \sum _{i=1}^I}{\left({\widehat{\alpha }}_{k,j}-{\alpha }_k\right)}^2}$. Obviously, the recovery of the channel can gradually improve with the increase of time slots and SNR.

5. Conclusions

In this paper, we proposed a joint iterative channel estimation scheme based on step-length optimization for multi-user mmWave massive MIMO systems. The proposed scheme first combined the advantages of Newton method and gradient descent method, and then gave a step-length optimization approach. Compared with the conventional SR algorithm, the proposed method greatly reduced the computational complexity without losing estimation accuracy, and the proposed algorithm yielded smaller channel estimation error compared to existing BCS and SVD methods. In addition, the proposed method could effectively estimate the channel even with a certain number of UEs. The perspective of this work considered an extension to the mmWave time-varying channel for joint channel parameters estimation, which included AoDs/AoAs, path gains, and Doppler shifts.