Hybrid Wideband Beamforming for Sum Spectral Efficiency Maximization in Millimeter-Wave Relay-Assisted Multiuser MIMO Cognitive Radio Networks

Abstract

Relay-assisted hybrid beamforming plays an inevitable role in enhancing network coverage, transmission range, and spectral efficiency while simultaneously reducing hardware cost, power consumption, and hardware implementation complexity. This study investigates a cognitive radio network (CRN)-based hybrid wideband transceiver for millimeter-wave (mm-wave) decode-and-forward (DF) relay-assisted multiuser (MU) multiple-input multiple-output (MIMO) systems. It is worth mentioning that the underlying problem has not been addressed so far, which is a real motivation behind the proposed algorithm. The joint optimization of hybrid processing components and the constant amplitude constraints imposed by the analog beamforming solution make this problem non-convex and NP-hard. Furthermore, the analog beamformer common to all sub-carriers is another challenging aspect of the underlying problem. To derive the frequency-flat analog processing component in the radio frequency (RF) domain and frequency-dependent baseband processing matrices in the baseband domain, the original complicated problem is reformulated as two single-hop sum-rate maximization sub-problems. Taking advantage of this decomposition, the sum spectral efficiency is maximized through RF precoding and combining. On the other hand, the impact of interference among transmitted data streams and inter-user interference (IUI) is minimized via baseband processing matrices. Finally, computer simulations are conducted by changing system parameters, considering both perfect and imperfect channel state information (CSI). Simulation results demonstrate that the proposed algorithm achieves performance close to full-complexity precoding and outperforms other well-known hybrid beamforming techniques. Specifically, more than 95% efficiency is achieved with perfect CSI, and more than 90% efficiency is attained under the assumption of 30% error in the estimated channels.

1. Introduction

The demand for exponentially increasing data rates and the spectrum crunch in the lower parts of the frequency spectrum led to the exploration of systems and devices compatible with the millimeter wave (mm-wave) (30–300 GHz) band [1,2]. The inevitable proliferation of connected devices in advanced future wireless systems (e.g., beyond 5G (B5G) networks) must require efficient utilization of available frequency resources to avoid spectral congestion [3,4]. The conventional fixed license-based static spectrum allocation policies result in significantly low spectrum utilization efficiency, as observed in the measurement campaign [5]. To address this problem, dynamic spectrum allocation policies are suggested that allow spectrum sharing to enhance the efficiency of spectrum utilization [6]. Cognitive radio (CR) is a viable solution that supports dynamic spectrum access and permits secondary users (SUs) to employ licensed spectrum without causing adverse interference to primary users (PUs). Furthermore, it is also required to minimize the impact of interference on the SU caused by the PU in CR mode. This target can be achieved by regulating the transmit power of the SU [7]. From this perspective, it seems crucial to combine the dynamic spectrum allocation framework proposed by cognitive radio networks (CRN) with mm-wave communication to maximize spectrum utilization in B5G wireless cellular systems [8].

In mm-wave transmission, signals suffer from huge path loss, atmospheric absorption, rain attenuation, and penetration loss when compared with lower frequencies. To overcome these poor characteristics of the mm-wave channel, large-scale antenna arrays are deployed at transceivers to achieve directional propagation and significant beamforming gain [9,10]. Note that the short wavelength associated with an mm-wave signal makes it possible to install a large antenna array in a small physical area. The blockage sensitivity of mm-wave signals poses a great challenge to establishing an effective non-line-of-sight (NLOS) transmission link [11]. This problem can be resolved by exploiting the notion of the cooperative communication paradigm in mm-wave MIMO systems, where large-scale MIMO at relay stations can greatly utilize high degrees of freedom, such as multiplexing gain, array gain, and interference reduction, but at the cost of complex signal processing at relay nodes [12,13]. It is worth highlighting that the RF chain is quite expensive and power-hungry as well. Therefore, conventional fully digital beamforming for mm-wave massive MIMO systems is impractical due to high hardware complexity and power consumption since its implementation requires the same number of RF chains as the number of antennas in an array [14,15]. The cost and power consumption of full-complexity digital transceivers are predicted to drastically decrease in the future, according to some new promising results [16], but alternative strategies have also been investigated recently. Among these alternatives, hybrid beamforming is the most favorable approach so far to making the systems feasible in actual practice. Hybrid precoding partitioned the beamforming process into low-dimensional digital baseband and high-dimensional analog RF processing components. This beamforming strategy considerably reduces the number of RF chains in comparison to the number of antennas [17]. The digital baseband processing part is responsible for achieving multiplexing gain, and beamforming gain is obtained through the RF processing component of the system. Furthermore, a network of phase-shifters is used for the practical implementation of analog precoders and combiners [18].

1.1. Related Work

The authors of [19,20,21,22,23,24,25] show that the highly directional beamforming capability of mm-wave technology is quite useful for spectrum sharing. This feature of mm-wave transmission enhances user data rates while simultaneously mitigating interference in an effective manner. In [19], the authors analytically prove that the performance of mm-wave systems can be improved using license sharing among operators. Moreover, the enhancement in performance is made possible using narrow beams that facilitate increasing the per-user data rate. In [20], the authors show that performance gain through spectrum sharing may be increased up to 130% when compared with the exclusive license model. In [22], an attempt is made to develop an optimization framework for mm-wave CRN that includes base-station (BS) association, coordination, and joint beamforming to maximize the capacity and fairness of users. Additionally, the effectiveness of coordination and BS association are also analyzed for interference management. The authors of [25] consider a real propagation environment with ideal and non-ideal beamforming techniques to study the performance of spectrum-sharing systems operating in the 26 GHz and 70 GHz frequency bands. According to their findings, two to three times higher capacity may be attained using a spectrum-sharing network in comparison to an exclusive licensing network. However, these benefits are unlikely to be achieved due to poor interference mitigation mechanisms and beamforming errors, especially in low signal-to-interference plus noise ratio (SINR) regimes.

In [26], the authors propose a method for interference management in a spectrum-sharing network and show its viability through simulation results. The authors of [27,28,29,30,31,32,33,34,35,36,37] develop algorithms for hybrid broadband mm-wave MIMO transceivers. In [27], a codebook-based hybrid precoding design is presented for wideband mm-wave MIMO systems. This design is based on the idea of limited feedback in wireless communication. The authors of [28] propose a hybrid beamforming technique for mm-wave single-user MIMO and MU multi-input single-output (MU-MISO) systems. This study shows that hybrid precoding can achieve performance equal to its full-complexity digital beamforming when the number of RF chains doubles the number of transmitted data streams. In [29], the authors introduce orthogonal frequency division multiplexing (OFDM)-based hybrid precoding for large-scale MIMO systems. This algorithm is an extension of the previous work conducted in [28] to make it consistent with practical wideband systems. A practical, low-complexity hybrid beamforming scheme is developed in [33] based on statistical channel information, where beamforming vectors are selected from codebooks using an efficient searching algorithm. This design does not require CSI, which is a challenging task for wideband mm-wave MIMO systems. A hybrid broadband transceiver design is proposed for mm-wave MU-MIMO systems in [35] using the constrained Tucker2 tensor decomposition technique. In addition, the focus of this algorithm is to suppress IUI and enhance the capacity of the equivalent baseband channels. The authors of [37] attempt to develop a hybrid transceiver design for broadband mm-wave MIMO systems using Alternating Minimization (Alt-Min) algorithms. The authors in [38] propose a hybrid precoding design based on the orthogonal matching pursuit (OMP) algorithm. This technique exploits the sparsity of mm-wave channels, and it is considered the first significant contribution to the field of hybrid beamforming.

It is obvious from the aforementioned works that significant research has been conducted on spectrum sharing in mm-wave networks and hybrid transceiver design in mm-wave MIMO OFDM systems. However, hybrid beamforming incorporating the notion of CRN has not been studied widely. For instance, the authors in [36] propose a hybrid precoding design for spectral efficiency maximization in a single-user MIMO-CRN. In [39], the authors investigate hybrid transceivers for backhaul networks that use spectrum sharing. The design orientation of hybrid beamforming algorithms in [40] and [41] is to maximize the minimum secrecy rate of all SUs by taking practical constraints into consideration. The algorithm presented in [42] is an extension of the CRN-based hybrid precoding proposed in [36], considering both uplink and downlink mm-wave MU-MIMO with complete CSI. The authors of [43] extend the OMP-based hybrid precoding design in [38] to a relay-assisted single-user MIMO system. In [44], the authors try to extend the OMP-based hybrid transceiver design in [38] to a relay-assisted MU-MIMO communication network. The authors in [45] develop a hybrid beamforming algorithm for single-user MIMO systems by taking both partially connected and fully connected architectures into account.

In [46], the authors suggest a hybrid precoding algorithm for wideband mm-wave MU-MIMO CRNs using optimal power loading. The authors of [47] present a hybrid transceiver design and optimal power allocation for downlink mm-wave MU-MIMO CRNs by employing limited feedback. In [48], the authors propose a cost-effective and high-accuracy hybrid precoding technique based on ϵ-Fuzzy Pareto active learning (FPAL) for large-scale mm-wave MIMO systems. In [49], the authors develop a modified user grouping strategy and hybrid beamforming for energy harvesting and information decoding in mm-wave single-user massive MIMO non-orthogonal multiple access (NOMA) systems by considering hardware impairments. The authors of [50] present an mm-wave large-scale MIMO hybrid beamforming technique based on tensor train decomposition to overcome beam misalignment and pointing errors.

1.2. Motivation and Contribution

In contrast to the studies in [36,37,38,39,40,41,42,43,44,45,46,47,48,49,50], the proposed algorithm is an attempt to design the CRN-based hybrid wideband mm-wave transceiver for relay-assisted MU-MIMO, which has not been reported in the existing literature to the best of the author’s knowledge. The prime focus of the proposed scheme is to maximize the sum rate of multiple SUs while keeping interference at the PU within a predefined threshold. The contribution of this study is summarized as follows:

• Considering a downlink wideband mm-wave relay-assisted MU-MIMO CRN-based communication network, a hybrid transceiver design is proposed that attempts to maximize the sum rate by taking the transmitted power and interference constraints into account. The optimization problem is formulated to maximize the sum rate of relay-assisted multiple SUs. This problem is quite complicated due to the constant modulus constraints and joint optimization of several complex matrix variables. To reduce the complexity associated with the solution of this problem, it is decomposed into two single-hop sum rate maximization sub-problems by exploiting the structural characteristics of DF relays and the notion of information theory.
• A decoupled design approach is applied to derive the analog RF and digital baseband processing components, where the focus of one sub-problem is to maximize the sum rate from the source to relay decoding, while the orientation of the other sub-problem is to maximize the sum rate from relay encoding to multiple SUs. Specifically, the target of sum rate maximization in each sub-problem is achieved through the RF beamforming solution. On the other hand, interference experienced by the PU and interference among transmitted data streams is minimized through digital baseband processing matrices. Furthermore, an endeavor is made to minimize the loss of information while solving each sub-problem.
• Simulation results are obtained under imperfect channel state information (CSI) by changing system parameters over a wide range. The proposed algorithm achieves performance close to fully digital beamforming, even in the presence of channel estimation errors. Also, minor degradation in performance occurs by increasing errors in imperfect channels in a gradual fashion. Finally, the effectiveness of the proposed scheme is evident from computer simulations.

The rest of the study is organized as follows: The system model description, the frequency domain mm-wave channel model, and problem formulation are given in Section 2. The proposed hybrid transceiver design, consisting of the derivation of the analog RF and digital baseband processing components at various communicating nodes, is presented in Section 3. Complexity analysis is included in Section 4. Simulation results for the performance evaluation are described in Section 5. Concluding remarks are given in Section 6.

Notation: Lower-case and upper-case boldface letters represent vectors and matrices, respectively.

Table 1. Notation throughout this paper.

Symbol	Definition
${\mathit{A}}^T$,  ${\mathit{A}}^H$,  ${\Vert \mathit{A}\Vert }_F$,  $\measuredangle \mathit{A}$,  $\left|\mathit{A}\left(i,j\right)\right|$,  $\mathit{A}\left(:,i\right)$, and$\mathit{A}\left(:,1:j\right)$	Transpose, conjugate transpose, Frobenius norm, element-wise phase, element-wise modulus, $i^{th}$ column, and first $j$ columns of a matrix $\mathit{A}$, respectively.
${\mathit{I}}_m$	Identity matrix of order $m\times m$
$ℂ$	Field of complex numbers
$ℝ$	Field of real numbers
$\mathcal{C}\mathcal{N}\left(\mathbf{0}, {\sigma }^2{\mathit{I}}_n\right)$	Complex Gaussian distribution with mean $0$ and covariance matrix ${\sigma }^2{\mathit{I}}_n$
$\mathit{det}\left(\mathit{A}\right)$ and$Tr\left(\mathit{A}\right)$	Determinant and trace of a matrix
$bd\left({\mathit{A}}_1,\dots ,{\mathit{A}}_K\right)$	Block diagonal matrix with sub-matrices ${\mathit{A}}_1,\dots ,{\mathit{A}}_K$
$\mathbb{E}[.]$	Expectation operator

shows the notation used in this paper.

2. System Model

Figure 1 shows the system model of the proposed hybrid beamforming approach for the CRN-based relay-assisted MU-MIMO networks, where the source node communicates with the PU through a direct transmission link and the $K$ SUs via the DF relay station. Multiple antennas are deployed at each communicating node. Furthermore, a hybrid beamforming architecture at the source, relay node, and multiple destinations is taken into consideration. The direct communication link between the source and K SUs is practically infeasible due to excessive path loss and deep fading. The number of antennas at the source, relay, and the $k$-th SU is represented as $N_t, N_r$ and $N_{d_k}$, and the corresponding number of RF chains is denoted as $N_t^{RF}, N_r^{RF}$ and $N_{d_k}^{RF}$, respectively. Let $N_s$ be the number of transmitted data streams from the cognitive radio base station (CRBS) to the $k$-th SU, and therefore, the source and relay must be able to handle $K{N}_s$ data streams. To enable efficient multi-stream transmission with a considerably small number of RF chains, it is required to fulfill the following conditions:

$$K{N}_s\le \min \left(N_t^{RF}, N_r^{RF}\right)\ll \min \left(N_t, N_r\right),$$

(1)

$$N_s\le \mathit{min}\left(N_{d_k}^{RF}\right)\ll \mathit{min}\left(N_{d_k}\right).$$

(2)

It is obvious from the above-mentioned conditions that (1) the number of data streams cannot be greater than the number of RF chains at the respective node, and (2) a significantly lower number of RF chains is required in hybrid transceivers to achieve near-optimal performance. Another condition also needs to be satisfied to avoid relay-induced signal-space collisions, which is described as

$$N_t\ge N_r\ge {\displaystyle \sum }_{k=1}^K{N}_{d_k}.$$

(3)

In relay-assisted MIMO systems, it takes two-time slots to transmit data streams from the source to the intended destination. The relay node receives the source-transmitted signal in the first time slot, while the processed signal at the relay station is forwarded to the end user in the second time slot. Keeping this perspective in view, the complete transmission link from the CRBS to SUs can be divided into point-to-point and point-to-multipoint MIMO systems. Let ${\mathit{s}}_k\left[n\right]\in {ℂ}^{N_s\times 1}$ be a complex information symbol vector, corresponding to the $n$-th sub-carrier, intended to the $k$-th SU such that $\mathbb{E}\left[{\mathit{s}}_k\left[n\right]{\mathit{s}}_k^H\left[n\right]\right]={\mathit{I}}_{N_s}, \forall n\in \left\{1,\dots ,N_{\mathit{s}ub}\right\}$, and $\mathit{s}\left[n\right]\in {ℂ}^{K{N}_s\times 1}$ be a vector that contains total information symbols for all SUs, which can be expressed as

$$\mathit{s}\left[n\right]={\left[{\mathit{s}}_1^T\left[n\right],\dots ,{\mathit{s}}_K^T\left[n\right]\right]}^T.$$

(4)

The transmitted signal $\mathit{x}\left[n\right]\in {ℂ}^{N_t\times 1}$ from the source by applying a hybrid beamformer can be characterized as

$$\mathit{x}\left[n\right]={\mathit{V}}_{RF}{\mathit{V}}_{BB}\left[n\right]\mathit{s}\left[n\right]={\mathit{V}}_{RF}{\displaystyle \sum }_{k=1}^K{\mathit{V}}_{BB,k}\left[n\right]{\mathit{s}}_k\left[n\right],$$

(5)

where ${\mathit{V}}_{RF}\in {ℂ}^{N_t\times N_t^{RF}}$ is the frequency-flat analog RF beamformer at the CRBS and ${\mathit{V}}_{BB}\left[n\right]=\left[{\mathit{V}}_{BB,1}\left[n\right],\dots ,{\mathit{V}}_{BB,K}\left[n\right]\right]\in {ℂ}^{N_t^{RF}\times K{N}_s}$ is the corresponding digital baseband precoder. The signal received ${\mathit{y}}_r\left[n\right]\in {ℂ}^{N_r\times 1}$ at the relay node can be written as

$${\mathit{y}}_r\left[n\right]=\mathit{H}\left[n\right]{\mathit{V}}_{RF}{\displaystyle \sum }_{k=1}^K{\mathit{V}}_{BB,k}\left[n\right]{\mathit{s}}_k\left[n\right]+{\mathit{n}}_r\left[n\right],$$

(6)

where $\mathit{H}\left[n\right]\in {ℂ}^{N_r\times N_t}$ is the frequency-domain channel from the source to the relay node at the $n$-th sub-carrier and ${\mathit{n}}_r\left[n\right]\in {ℂ}^{N_r\times 1}$ is the zero mean circularly symmetric complex Gaussian (ZMCSCG) noise with variance ${\sigma }_r^2$, i.e., ${\mathit{n}}_r\left[n\right]~\mathcal{C}\mathcal{N}\left(0, {\sigma }_r^2{\mathit{I}}_{N_r}\right)$. The interference experienced by a PU owing to the source transmitted signal to the SUs at the $n$-th sub-carrier can be expressed as

$$d_1\left[n\right]={\Vert {\mathit{H}}_{PU}\left[n\right]{\mathit{V}}_{RF}{\displaystyle \sum }_{k=1}^K{\mathit{V}}_{BB,k}\left[n\right]\Vert }_F^2,$$

(7)

where ${\mathit{H}}_{PU}\left[n\right]$ specifies the frequency domain channel from the source to the PU. The received signal ${\mathit{y}}_{r_1}\left[n\right]\in {ℂ}^{K{N}_s\times 1}$ at the output of the relay hybrid combiner ${\mathit{F}}_1\left[n\right]={\mathit{F}}_{RF,1}{\mathit{F}}_{BB,1}\left[n\right]\in {ℂ}^{N_r\times K{N}_s}$ is given as

$${\mathit{y}}_{r_1}\left[n\right]={\left({\mathit{F}}_{RF,1}{\mathit{F}}_{BB,1}\left[n\right]\right)}^H\left(\mathit{H}\left[n\right]{\mathit{V}}_{RF}{\displaystyle \sum }_{k=1}^K{\mathit{V}}_{BB,k}\left[n\right]{\mathit{s}}_k\left[n\right]+{\mathit{n}}_r\left[n\right]\right),$$

(8)

where ${\mathit{F}}_{RF,1}\in {ℂ}^{N_r\times N_r^{RF}}$ and ${\mathit{F}}_{BB,1}\left[n\right]\in {ℂ}^{N_r^{RF}\times K{N}_s}$ are the analog combiner common to all sub-carriers and the frequency-selective digital baseband combiner at the relay station, respectively.

The transmitted signal ${\mathit{y}}_{r_2}\left[n\right]\in {ℂ}^{N_r\times 1}$ from the relay node by applying a hybrid precoder ${\mathit{F}}_2\left[n\right]={\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]\in {ℂ}^{N_r\times K{N}_s}$, can be written as

$$\begin{gathered} {\mathit{y}}_{r_2}\left[n\right]={\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]{\mathit{y}}_{r_1}\left[n\right] \\ =\left({\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]\right){\left({\mathit{F}}_{RF,1}{\mathit{F}}_{BB,1}\left[n\right]\right)}^H\left(\mathit{H}\left[n\right]{\mathit{V}}_{RF}{\displaystyle \sum }_{k=1}^K{\mathit{V}}_{BB,k}\left[n\right]{\mathit{s}}_k\left[n\right]+{\mathit{n}}_r\left[n\right]\right), \end{gathered}$$

(9)

where ${\mathit{F}}_{RF,2}\in {ℂ}^{N_r\times N_r^{RF}}$ and ${\mathit{F}}_{BB,2}\left[n\right]\in {ℂ}^{N_r^{RF}\times K{N}_s}$ are the frequency-independent analog RF precoder and the frequency-dependent digital baseband precoder at the relay node, respectively.

The interference experienced by a PU due to relay forwarded signal ${\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]{\mathit{y}}_{r_1}\left[n\right]\in {ℂ}^{N_r\times 1}$ to the SUs can be characterized as

$$d_2\left[n\right]={\Vert {\mathit{H}}_{PU}\left[n\right]{\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]{\mathit{y}}_{r_1}\left[n\right]\Vert }_F^2.$$

(10)

Using (9), the relay hybrid filter $\mathit{F}\left[n\right]\in {ℂ}^{N_r\times N_r}$ is defined as follows:

$$\mathit{F}\left[n\right]=\left({\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]\right){\left({\mathit{F}}_{RF,1}{\mathit{F}}_{BB,1}\left[n\right]\right)}^H={\mathit{F}}_{RF,2}{\mathit{F}}_{BB}\left[n\right]{\mathit{F}}_{RF,1}^H,$$

(11)

where ${\mathit{F}}_{BB}\left[n\right]={\mathit{F}}_{BB,2}\left[n\right]{\mathit{F}}_{BB,1}^H\left[n\right]\in {ℂ}^{N_r^{RF}\times N_r^{RF}}$ is the combined baseband processing component at the relay station. Under the assumption of a block-fading channel, the received signal ${\mathit{y}}_{d_k}^{+}\left[n\right]\in {ℂ}^{N_{d_k}\times 1}$ at the $k$-th SU corresponding to the $n$-th sub-carrier can be modeled as

$${\mathit{y}}_{d_k}^{+}\left[n\right]={\mathit{G}}_{d_k}\left[n\right]\mathit{F}\left[n\right]\left(\mathit{H}\left[n\right]{\mathit{V}}_{RF}{\displaystyle \sum }_{k=1}^K{\mathit{V}}_{BB,k}\left[n\right]{\mathit{s}}_k\left[n\right]+{\mathit{n}}_r\left[n\right]\right)+{\mathit{z}}_k\left[n\right],$$

(12)

where ${\mathit{G}}_{d_k}\left[n\right]\in {ℂ}^{N_{d_k}\times N_r}$ represents the frequency domain channel matrix between the relay node and the $k$-th SU at the $n$-th sub-carrier and ${\mathit{z}}_k\left[n\right]\in {ℂ}^{N_{d_k}\times 1}$ denotes the ZMCSCG noise with variance ${\sigma }_k^2$, i.e., ${\mathit{z}}_k\left[n\right]~\mathcal{C}\mathcal{N}\left(0, {\sigma }_k^2{\mathit{I}}_{N_{d_k}}\right)$. Finally, the processed signal ${\mathit{y}}_{d_k}\left[n\right]\in {ℂ}^{N_s\times 1}$ after passing through the hybrid combiner ${\mathit{W}}_{d_k}\left[n\right]={\mathit{W}}_{RF,k}{\mathit{W}}_{BB,k}\left[n\right]\in {ℂ}^{N_{d_k}\times N_s}$ at the $k$-th SU can be expressed as

$${\mathit{y}}_{d_k}\left[n\right]={\left({\mathit{W}}_{RF,k}{\mathit{W}}_{BB,k}\left[n\right]\right)}^H{\mathit{y}}_{d_k}^{+}\left[n\right]$$

$$=\{\begin{array}{c}{\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}{\mathit{F}}_{BB}\left[n\right]{\mathit{F}}_{RF,1}^H\mathit{H}\left[n\right]{\mathit{V}}_{RF}{\displaystyle {\displaystyle \sum }_{k=1}^K}{\mathit{V}}_{BB,k}\left[n\right]{\mathit{s}}_k\left[n\right]+\\ {\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}{\mathit{F}}_{BB}\left[n\right]{\mathit{F}}_{RF,1}^H{\mathit{n}}_r\left[n\right]+{\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{W}}_{RF,k}^H{\mathit{z}}_k\left[n\right]\end{array},$$

(13)

where ${\mathit{W}}_{RF,k}\in {ℂ}^{N_{d_k}\times N_{d_k}^{RF}}$, ${\mathit{W}}_{BB,k}\left[n\right]\in {ℂ}^{N_{d_k}^{RF}\times N_s}$ are the analog RF and digital baseband combiners at the $k$-th SU. The compact representation of (12) is given as

$${\mathit{y}}_{c_k}\left[n\right]=\{\begin{array}{c}{\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{G}}_{eq,k}\left[n\right]{\mathit{F}}_{BB}\left[n\right]{\mathit{H}}_{eq}\left[n\right]{\displaystyle {\displaystyle \sum }_{k=1}^K}{\mathit{V}}_{BB,k}\left[n\right]{\mathit{s}}_k\left[n\right]+\\ {\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{G}}_{eq,k}\left[n\right]{\mathit{F}}_{BB}\left[n\right]{\mathit{n}}_r^{+}\left[n\right]+{\mathit{W}}_{BB,k}^H\left[n\right]z_k^{+}\left[n\right]\end{array}$$

(14)

where ${\mathit{H}}_{eq}\left[n\right]={\mathit{F}}_{RF,1}^H\mathit{H}\left[n\right]{\mathit{V}}_{RF}\in {ℂ}^{N_r^{RF}\times N_t^{RF}}$ is the baseband equivalent channel from the source to the relay node, ${\mathit{G}}_{eq,k}\left[n\right]={\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\in {ℂ}^{N_{d_k}^{RF}\times N_r^{RF}}$ is the baseband equivalent channel from the relay station to the $k$-th SU, ${\mathit{n}}_r^{+}\left[n\right]={\mathit{F}}_{RF,1}^H{\mathit{n}}_r\left[n\right]\in {ℂ}^{N_r^{RF}\times 1}$ is the resultant noise vector after passing through the relay analog combiner and ${\mathit{z}}_k^{+}\left[n\right]={\mathit{W}}_{RF,k}^H{\mathit{z}}_k\left[n\right]\in {ℂ}^{N_{d_k}^{RF}\times 1}$ is the resultant noise vector at the output of the RF combiner at the $k$-th SU. It is important to note that the noise distribution remains unchanged after multiplication with the analog precoder/combiner, as shown in [51]. Therefore, ${\mathit{n}}_r^{+}\left[n\right]$ and ${\mathit{z}}_k^{+}\left[n\right]$ follow the same distribution as that of ${\mathit{n}}_r\left[n\right]$ and ${\mathit{z}}_k\left[n\right]$, respectively. Using (14), the capacity expression of the $k$-th SU when the $n$-th sub-carrier is allocated for signal transmission is given as

$$C_k\left[n\right]=\left(\frac{1}{2}\right){\log }_2\det \left({\mathit{I}}_{N_s}+\frac{{\mathit{X}}_k\left[n\right]{\mathit{X}}_k^H\left[n\right]}{\underset{equivalent noise covariance matrix \left(={\mathit{R}}_k\left[n\right]\right)}{\underbrace{\left({\sigma }_r^2{\mathit{Y}}_k\left[n\right]{\mathit{Y}}_k^H\left[n\right]+{\sigma }_k^2{\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{W}}_{BB,k}\left[n\right]\right)}}+{\mathit{\Xi }}_{IUI}\left[n\right]}\right)$$

(15)

where

$$\left\{\begin{array}{c}{\mathit{X}}_k\left[n\right]={\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{G}}_{eq,k}\left[n\right]{\mathit{F}}_{BB}\left[n\right]{\mathit{H}}_{eq}\left[n\right]{\mathit{V}}_{BB,k}\left[n\right]\\ {\mathit{Y}}_k\left[n\right]={\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{G}}_{eq,k}\left[n\right]{\mathit{F}}_{BB}\left[n\right]{\mathit{n}}_r^{+}\left[n\right] \\ {\mathit{Z}}_k\left[n\right]=\underset{\mathit{interuser} \mathit{interference} \left(IUI\right)}{\underbrace{{\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{G}}_{eq,k}\left[n\right]{\mathit{F}}_{BB}\left[n\right]{\mathit{H}}_{eq}\left[n\right]{\displaystyle {\displaystyle \sum }_{j=1,j\ne k}^K}{\mathit{V}}_{BB,j}\left[n\right]}},{\mathit{\Xi }}_{IUI}\left[n\right]={\mathit{Z}}_k\left[n\right]{\mathit{Z}}_k^H\left[n\right]\end{array}\right.$$

The scaling factor (1/2) in (15) indicates that signals are transmitted from the source to $K$ SUs over two-time slots. The sum rate averaged over $N_{sub}$ sub-carriers can be written as

$$R_{avg}=\left(\frac{1}{N_{\mathit{s}ub}}\right){\displaystyle \sum }_{n=1}^{N_{\mathit{s}ub}}\left({\displaystyle \sum }_{k=1}^K{C}_k\left[n\right]\right)$$

(16)

It is known that capacity expression in MIMO communication systems is directly related to signal-to-noise plus interference $\left(SNIR\right)$ ratio. Therefore, the end-to-end sum rate in (16) can also be expressed as

$$R_{SD}=\left(\frac{1}{N_{\mathit{s}ub}}\right){\displaystyle \sum }_{n=1}^{N_{\mathit{s}ub}}\left({\displaystyle \sum }_{k=1}^K{\displaystyle \sum }_{i=1}^{N_s}{\left(1+{\left(SNIR\right)}_{k_i}\left[n\right]\right)}_{SD}\right)$$

(17)

where ${\left(SNIR\right)}_{k_i}\left[n\right]$ is the signal-to-noise plus interference ratio from the source to the $k$-SU corresponding to the $i$-th data stream when the Gaussian symbols are assumed to be transmitted.

2.1. Channel Model

Contrary to the Rayleigh fading channel, the mm-wave propagation environment does not follow conventional rich scattering [29]. The sparse scattering nature of mm-wave channels is characterized using low-rank matrices. Moreover, the geometric channel based on the extended Saleh–Valenzuela model is adopted to capture the mathematical structure of the mm-wave propagation environment. Considering the uniform planar array (UPA), the mathematical formulation of the frequency domain mm-wave channel corresponding to the $n$-th sub-carrier can be expressed as [37]

$$\mathit{H}\left[n\right]=\gamma {\displaystyle \sum }_{i=0}^{N_{cl}-1}{\displaystyle \sum }_{l=1}^{N_{ray}}{\alpha }_{il}{\mathit{a}}_r\left({\phi }_{il}^r,{\theta }_{il}^r\right) {\mathit{a}}_t{\left({\phi }_{il}^t,{\theta }_{il}^t\right)}^H e^{-j2\pi \frac{i.n}{N_{\mathit{s}ub}}}$$

(18)

where $\gamma =\sqrt{\frac{N_t{N}_r}{N_{cl}{N}_{ray}}}$ represents the normalization parameter, $N_{cl}$ shows the number of clusters, $N_{ray}$ describes the number of propagation paths within each cluster. ${\alpha }_{il}$~$\mathcal{C}\mathcal{N}\left(0,{\sigma }_{\alpha ,i}^2\right)$ denotes the complex gain, ${\phi }_{il}^t$ and ${\phi }_{il}^r$ illustrate the azimuth angles of departure and arrival, ${\theta }_{il}^t$ and ${\theta }_{il}^r$ specify the elevation angles of departure and arrival of the $l$-th transmission path in the $i$-th cluster. Furthermore, ${\mathit{a}}_r\left({\phi }_{il}^r,{\theta }_{il}^r\right)$ and ${\mathit{a}}_t\left({\phi }_{il}^t,{\theta }_{il}^t\right)$ are the planar array response vectors at the receiver and transmitter, respectively. These array response vectors depend on the antenna array structure, where each element needs to follow a constant amplitude constraint. In this work, the uniform square planar array (USPA) with $\sqrt{N}\times \sqrt{N}$ antenna elements and therefore, the array response vector corresponding to the $l$-th propagation path in the $i$-th cluster can be modeled as [37]

$$\mathit{a}\left({\phi }_{il},{\theta }_{il}\right)=\left(\frac{1}{\sqrt{N}}\right){\left[1,\dots ,e^{j\frac{2\pi }{\lambda }d\left(p\sin {\phi }_{il}\sin {\theta }_{il}+q\cos {\theta }_{il}\right)},\dots , e^{j\frac{2\pi }{\lambda }d\left(\left(\sqrt{N}-1\right)\sin {\phi }_{il}\sin {\theta }_{il}+\left(\sqrt{N}-1\right)\cos {\theta }_{il}\right)}\right]}^T,$$

(19)

where $\lambda$ is the signal wavelength, $d=\frac{\lambda }{2}$ is the antenna spacing, and $0\le u<\sqrt{N}$ and $0\le v<\sqrt{N}$ are the antenna indices.

2.2. Problem Formulation

The mean squared error (MSE) and the achievable sum rate are two optimization goals that are usually considered for problems involving the DF protocol at the relay node. Note that the latter is a crucial performance evaluation criterion in mm-wave MIMO systems, where hybrid beamforming design is concerned. Therefore, the optimization problem under the desirable constraints is formulated as

$$\underset{{\left[\begin{array}{c}{V}_{RF},V_{BB}\left[n\right], F_{RF,1},F_{BB,1}\left[n\right],\\ F_{BB,2}\left[n\right],F_{RF,2}, W_{RF,k},W_{BB,k}\left[n\right]\end{array}\right]}_{k=1, n=1}^{K, N_{\mathit{s}ub}}}{\mathit{max}}{R}_{avg}$$

$$s. t. \{\begin{array}{c}{\Vert V_{RF}{V}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1, \forall n,\\ \begin{array}{c}{d}_2\left[n\right]\le J_2, \forall n,\\ \left\{\begin{array}{c}\left|V_{RF}\left(x,y\right)\right|=\frac{1}{\sqrt{N_t}} ,\left|W_{RF,k}\left(x,y\right)\right|=\frac{1}{\sqrt{N_{d_k}}},\\ \left|F_{RF,1}\left(x,y\right)\right|=\left|F_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}},\forall x,y,k\end{array}\right\}\end{array}\end{array},$$

(20)

where source transmitted power is restricted to $P_{\mathit{s}}$, and $J_1$ and $J_2$ denote the threshold of interference experienced by the PU due to the source and relay transmitted signals, respectively. Furthermore, $R_{avg}$, $d_1\left[n\right]$, and $d_2\left[n\right]$ are defined in (16), (7), and (10), respectively. It is obvious from (20) that sum rate maximization depends on the joint optimization of several complex matrix variables ${\mathit{V}}_{RF}$, ${\mathit{V}}_{BB}\left[n\right]$, ${\mathit{F}}_{RF,1}$, ${\mathit{F}}_{BB,1}\left[n\right]$, ${\mathit{F}}_{BB,2}\left[n\right]$, ${\mathit{F}}_{RF,2}$, ${\mathit{W}}_{RF,k}$, and ${\mathit{W}}_{BB,k}\left[n\right]$. The joint optimization in this case is usually NP-hard, and the element-wise constant amplitude constraints imposed by the analog RF beamformers make the problem non-convex and, hence, mathematically intractable. To reduce the complexity associated with the solution of (20) and to make the problem mathematically tractable, an attempt is made to decompose the complicated optimization problem into sub-problems. One sub-problem deals with the optimization of complex matrix variables that are related to the first transmission phase, i.e., from source to relay station, while the other sub-problem concentrates on finding the decision variables corresponding to the second transmission phase, i.e., from the relay node to multiple SUs. Finally, a decoupled design methodology is adopted for solving each sub-problem, which allows us to derive the RF and baseband processing components separately.

3. Proposed Hybrid Beamforming Design

When the DF relay node is placed between the source and the end user, this cooperative communication network can be considered a cascade of two sub-networks. One sub-network corresponds to the part of the system from the source to the relay receiver, while the other sub-network indicates the part of the system from the relay transmitter to the destination. From the perspective of the underlying system, cascading refers to the decoded signal at the output of the relay hybrid combiner, which is given to the relay hybrid precoder as input. Therefore, the part of the network from the CRBS to the SUs can be divided into two relatively independent sub-systems. Let $R_{SR}$ be the transmission rate of the first sub-section from the source to the relay node and $R_{RD}$ be the transmission rate of the second sub-section from the relay station to the SUs without considering the direct communication path from the source to multiple destinations in the SN. Therefore, the overall sum rate of the two sub-systems can be determined by maximizing the minimum between $R_{SR}$ and $R_{RD}$. Taking advantage of this important feature, it is possible to express the sum rate of the entire system as

$$R_{avg}=\left(\frac{1}{2}\right)\min (R_{SR},R_{RD}).$$

(21)

Following the mathematical formulation in (17), the expressions of $R_{SR}$ (source-to-relay) and $R_{RD}$ (relay-to-destination) can be obtained in a straightforward manner as

$$R_{SR}=\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\left({\displaystyle \sum }_{k=1}^K{\displaystyle \sum }_{i=1}^{N_s}{\left(1+{\left(SNIR\right)}_{k_i}\left[n\right]\right)}_{SR}\right),$$

(22)

$$R_{RD}=\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\left({\displaystyle \sum }_{k=1}^K{\displaystyle \sum }_{i=1}^{N_s}{\left(1+{\left(SNIR\right)}_{k_i}\left[n\right]\right)}_{RD}\right),$$

(23)

where ${\left({\left(SNIR\right)}_{k_i}\left[n\right]\right)}_{SR}$ and ${\left({\left(SNIR\right)}_{k_i}\left[n\right]\right)}_{RD}$ denote the $SNIR$ at the relay node and the $k$-th SU corresponding to the $i$-th data stream, respectively. Exploiting the separation of the sum rate into two parts, the optimization problem defined in (20) can be decomposed into two sub-problems as follows:

$$\underset{{\left\{{\mathit{V}}_{RF},{\mathit{V}}_{BB}\left[n\right],{\mathit{F}}_{RF,1}{\mathit{F}}_{BB,1}\left[n\right]\right\}}_{k=1, n=1}^{K, N_{sub}}}{\max }{R}_{SR}$$

$$s. t. \{\begin{array}{c}{\Vert V_{RF}{V}_{BB}\left[n\right]\mathrm{\Vert }}_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|V_{RF}\left(x,y\right)\right|=\frac{1}{\sqrt{N_t}} ,\left|F_{RF,1}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}}, \forall x,y\end{array},$$

(24)

$$\underset{{\left\{{\mathit{F}}_{BB,2}\left[n\right],{\mathit{F}}_{RF,2}, {\mathit{W}}_{RF,k},{\mathit{W}}_{BB,k}\left[n\right]\right\}}_{k=1, n=1}^{K, N_{\mathit{s}ub}}}{\max }{R}_{RD}$$

$$s. t. \{\begin{array}{c}{d}_2\left[n\right]\le J_2,\\ \left|{\mathit{F}}_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}} ,\left|{\mathit{W}}_{RF,k}\left(x,y\right)\right|=\frac{1}{\sqrt{N_{d_k}}}, \forall x,y\end{array}.$$

(25)

It is obvious from (24) and (25) that each sub-problem is still non-convex and NP-hard due to the constant modulus constraints and joint optimization of several complex matrix variables. The sub-problem (24) corresponds to the sum rate maximization from the source to the relay station, and this part of the underlying network is a point-to-point MIMO communication link. Therefore, this section of the SN does not play any role in IUI among SUs. Furthermore, $R_{SR}$ can be modeled by the mutual information between $\mathit{s}\left[n\right]$ (4) and ${\mathit{y}}_{r_1}\left[n\right]$ (8) as

$$\mathfrak{T}\left(\mathit{s}\left[n\right];{\mathit{y}}_{r_1}\left[n\right]\right)={\log }_2\det \left({\mathit{I}}_{K{N}_s}+\frac{1}{{\sigma }_r^2}{\mathit{R}}_{N_1}^{-1}\left({\mathit{F}}_{BB,1}^H\left[n\right]{\mathit{H}}_{eq}\left[n\right]{\mathit{V}}_{BB}\right){\left({\mathit{F}}_{BB,1}^H\left[n\right]{\mathit{H}}_{eq}\left[n\right]{\mathit{V}}_{BB}\right)}^H\right),$$

(26)

where ${\mathit{R}}_{N_1}={\mathit{F}}_{BB,1}^H\left[n\right]{\mathit{F}}_{BB,1}\left[n\right]$. Using (26), the sub-problem in (24) can be transformed into the following form:

$$\underset{{\left\{{\mathit{V}}_{R\mathit{F}},{\mathit{V}}_{BB}\left[n\right],{\mathit{F}}_{RF,1}{\mathit{F}}_{BB,1}\left[n\right]\right\}}_{k=1, n=1}^{K, N_{sub}}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\mathfrak{T}\left(\mathit{s}\left[n\right];{\mathit{y}}_{r_1}\left[n\right]\right)$$

$$s. t. \{\begin{array}{c}{\Vert {\mathit{V}}_{R\mathit{F}}{\mathit{V}}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|{\mathit{V}}_{RF}\left(x,y\right)\right|=\frac{1}{\sqrt{N_t}} ,\left|{\mathit{F}}_{RF,1}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}}, \forall x,y\end{array}.$$

(27)

Furthermore, the optimization problem in (27) to maximize the mutual information between the source and relay node can be equivalently transformed into the following simplified form:

$$\underset{{\left\{{\mathit{V}}_{RF},{\mathit{V}}_{BB}\left[n\right],{\mathit{F}}_{RF,1}{\mathit{F}}_{BB,1}\left[n\right]\right\}}_{k=1, n=1}^{K, N_{sub}}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}Tr\left\{\left({\mathit{F}}_{BB,1}^H\left[n\right]{\mathit{H}}_{eq}\left[n\right]{\mathit{V}}_{BB}\right){\left({\mathit{F}}_{BB,1}^H\left[n\right]{\mathit{H}}_{eq}\left[n\right]{\mathit{V}}_{BB}\right)}^H\right\}$$

$$s. t. \{\begin{array}{c}{\Vert {\mathit{V}}_{RF}{\mathit{V}}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|{\mathit{V}}_{RF}\left(x,y\right)\right|=\frac{1}{\sqrt{N_t}} ,\left|{\mathit{F}}_{RF,1}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}}, \forall x,y\end{array}.$$

(28)

Following a decoupled approach, the baseband processing matrices $\left({\mathit{V}}_{BB}, {\mathit{F}}_{BB,1}\right)$ in the objective function of (28) can be ignored. These digital processing components can be derived optimally using conventional techniques after finding the frequency-independent source analog precoder and relay analog combiner. Therefore, the problem in (28) can be reduced to the following form:

$$\underset{{\left\{{\mathit{V}}_{RF},{\mathit{F}}_{RF,1}\right\}}_{n=1}^{N_{sub}}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}Tr\left\{{\mathit{H}}_{eq}\left[n\right]{\mathit{H}}_{eq}^H\left[n\right]\right\}=Tr\left\{\left({\mathit{F}}_{RF,1}^H\mathit{H}\left[n\right]{\mathit{V}}_{RF}\right){\left({\mathit{F}}_{RF,1}^H\mathit{H}\left[n\right]{\mathit{V}}_{RF}\right)}^H\right\}$$

$$s. t. \{\begin{array}{c}{\Vert {\mathit{V}}_{RF}{\mathit{V}}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|{\mathit{V}}_{RF}\left(x,y\right)\right|=\frac{1}{\sqrt{N_t}} ,\left|{\mathit{F}}_{RF,1}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}}, \forall x,y\end{array}.$$

(29)

The direct optimization of the problem (29) is mathematically challenging. To reduce the complexity, one phase-only processing component can be removed temporarily from the objective function of (29) by making a valid assumption, i.e., ${\mathit{V}}_{RF}{\mathit{V}}_{RF}^H\approx {\mathit{N}}_t{\mathit{I}}_{N_t}$ under large-scale antenna arrays [29]. After designing ${\mathit{F}}_{RF,1}$, ${\mathit{V}}_{RF}$ is adjusted according to ${\mathit{F}}_{RF,1}$ to minimize the performance degradation when an arbitrary number of antennas are deployed at communicating units. This procedure facilitates the derivation of both RF processing components to achieve acceptable performance irrespective of the number of antennas and other system parameters. The above-mentioned assumption further simplifies the objective function in (29) for designing the common analog combiner at the relay node, i.e., ${\mathit{F}}_{RF,1}$.

$$\underset{{\left\{{\mathit{F}}_{RF,1}\right\}}_{n=1}^{N_{sub}}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}Tr\left\{{\mathit{F}}_{RF,1}^H\mathit{H}\left[n\right]{\mathit{H}}^H\left[n\right]{\mathit{F}}_{RF,1}\right\}$$

$$s. t. \{\begin{array}{c}{\Vert {\mathit{V}}_{RF}{\mathit{V}}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|{\mathit{F}}_{RF,1}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}}, \forall x,y\end{array}.$$

(30)

To derive ${\mathit{F}}_{RF,1}$ common to all sub-carriers, the product of $\mathit{H}\left[n\right]{\mathit{H}}^H\left[n\right]$ in the objective function of (30) can be replaced with the following matrix:

$$\mathit{A}=\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\mathit{H}\left[n\right]{\mathit{H}}^H\left[n\right],$$

(31)

where $\mathit{A}\in {ℂ}^{N_r\times N_r}$ is the average of the covariance matrices of frequency domain channels. Therefore, the optimization problem in (30) is reformulated as

$$\underset{{\mathit{F}}_{RF,1}}{\max }Tr\left\{{\mathit{F}}_{RF,1}^H\mathit{A} {\mathit{F}}_{RF,1}\right\}$$

$$s. t. \{\begin{array}{c}{\Vert {\mathit{V}}_{RF}{\mathit{V}}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|{\mathit{F}}_{RF,1}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}}, \forall x,y\end{array}.$$

(32)

The function $Tr\left\{{\mathit{F}}_{RF,1}^H\mathit{A} {\mathit{F}}_{RF,1}\right\}$ in (32) can also be written as

$$Tr\left\{{\mathit{F}}_{RF,1}^H\mathit{A} {\mathit{F}}_{RF,1}\right\}={\left({\mathit{f}}_{RF,1}^H\right)}^{\left[1\right]}\mathit{A}{\left({\mathit{f}}_{RF,1}\right)}^{\left[1\right]}+\cdots +{\left({\mathit{f}}_{RF,1}^H\right)}^{\left[N_r^{RF}\right]}\mathit{A}{\left({\mathit{f}}_{RF,1}\right)}^{\left[N_r^{RF}\right]}$$

$$={\displaystyle \sum }_{m=1}^{N_r^{RF}}{\left({\mathit{f}}_{RF,1}^H\right)}^{\left[m\right]}\mathit{A}{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]},$$

(33)

where ${\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}\in {ℂ}^{N_r\times 1}$ is the $m$-th RF beamforming vector of ${\mathit{F}}_{RF,1}$ and ${\left({\mathit{f}}_{RF,1}^H\right)}^{\left[m\right]}\mathit{A}{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}$ is the $m$-th diagonal element of ${\mathit{F}}_{RF,1}^H\mathit{A} {\mathit{F}}_{RF,1}\in {ℂ}^{N_r^{RF}\times N_r^{RF}}$. Using (33), the problem in (32) can be transformed into the vector form as

$$\underset{{\left\{{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}\right\}}_{m=1}^{N_r^{RF}}}{\max }{\displaystyle \sum }_{m=1}^{N_r^{RF}}{\left({\mathit{f}}_{RF,1}^H\right)}^{\left[m\right]}\mathit{A}{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}$$

$$s. t. \{\begin{array}{c}{\Vert {\mathit{V}}_{RF}{\mathit{V}}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}\left(l\right)\right|=\frac{1}{\sqrt{N_r}}, \forall l\in \left\{1,\dots ,N_r\right\}\end{array},$$

(34)

where ${\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}\left(l\right)$ denotes the $l$-th element of the $m$-th column in ${\mathit{F}}_{RF,1}$. To find the phase-only beamforming vector ${\mathit{f}}_{RF,1}$ in (32), eigenvalue decomposition (EVD) is applied on $\mathit{A}$ such that $\mathit{A}=\mathit{P}\mathit{\Sigma }{\mathit{P}}^H$, where $\mathit{P}\in {ℂ}^{N_r\times N_r}$ is the unitary matrix and $\mathit{\Sigma }\in {ℂ}^{N_r\times N_r}$ is a diagonal matrix that contains eigenvalues. Let ${\mathit{p}}_m^{opt}\in {ℂ}^{N_r\times 1}$ be the optimal unconstrained combining vector corresponding to the maximum eigenvalue in $\mathit{\Sigma }\in {ℂ}^{N_r\times N_r}$, which is given as

$${\mathit{p}}_m^{opt}=\mathit{P}\left(:,m\right).$$

(35)

The required RF beamforming vector ${\mathit{f}}_{RF,1}$ can be obtained by formulating a problem that facilitates determining the unknown vector as close as possible to the unconstrained optimal vector in (35). Hence,

$$\underset{{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}}{\min }{\Vert {\mathit{p}}_m^{opt}-{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}\Vert }_2^2$$

$$s. t. \left|{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}\left(l\right)\right|=\frac{1}{\sqrt{N_r}}, \forall l\in \left\{1,\dots ,N_r\right\}, m\in \left\{1,\dots ,N_r^{RF}\right\},$$

(36)

where ${\Vert .\Vert }_2$ stands for the L₂-norm of a vector. Using the property of $L_2$-norm and trace operator, the objective function in (36) can be expressed as

$${\Vert {\mathit{p}}_m^{opt}-{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}\Vert }_2^2=2-2 Tr\left(ℝ\left[{\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}{\left({\mathit{p}}_m^{opt}\right)}^H\right]\right),$$

(37)

It is evident from (36) that the minimum value can be obtained when ${\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}$ has the same phase-values as ${\mathit{p}}_m^{opt}$. This information can be exploited for designing the required RF combining vector as

$${\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}=\left(\frac{1}{\sqrt{N_r}}\right)\exp \left\{j arg\left({\mathit{p}}_m^{opt}\right)\right\},$$

(38)

where $arg(.)$ shows the argument operator. Using ${\left({\mathit{f}}_{RF,1}\right)}^{\left[m\right]}, m=\left\{1,\dots ,N_r^{RF}\right\}$, the ${\mathit{F}}_{RF,1}$ is formulated as

$${\mathit{F}}_{RF,1}=\left[{\left({\mathit{f}}_{RF,1}\right)}^{\left[1\right]},\dots ,{\left({\mathit{f}}_{RF,1}\right)}^{\left[N_r^{RF}\right]}\right].$$

(39)

This completes the design of the relay combining matrix common to all sub-carriers. Now, it is desirable to update the source analog beamformer ${\mathit{V}}_{RF}$ to compensate for the performance loss due to the assumption made based on large antenna arrays. In addition, the updated design makes the analog beamforming solution at the source node applicable to any arbitrary number of antennas. To achieve this target, the optimization problem is defined as

$$\underset{{\left\{{\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}\right\}}_{s=1}^{N_t^{RF}}}{\max } Tr\left\{{\mathit{V}}_{RF}^H \mathit{B} {\mathit{V}}_{RF}\right\}={\displaystyle \sum }_{s=1}^{N_t^{RF}}{\left({\mathit{v}}_{RF}^H\right)}^{\left[s\right]}\mathit{B}{\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}$$

$$s. t. \{\begin{array}{c}{\Vert {\mathit{V}}_{RF}{\mathit{V}}_{BB}\left[n\right]\Vert }_F^2\le P_s, \forall n,\\ d_1\left[n\right]\le J_1,\\ \left|{\mathit{V}}_{RF}\left(x,y\right)\right|=\frac{1}{\sqrt{N_t}} , \forall x,y\end{array},$$

(40)

where

$$\mathit{B}=\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}{\mathit{H}}^H\left[n\right]{\mathit{F}}_{RF,1}{\mathit{F}}_{RF,1}^H\mathit{H}\left[n\right]\in {ℂ}^{N_t\times N_t}.$$

(41)

The problem (40) is similar to (34), and hence, the same procedure can be applied for designing the ${\mathit{V}}_{RF}$. Only the main steps are given to avoid repetition. The EVD of $\mathit{B}$ in (41) is given as

$$\mathit{B}=\mathit{Q}{\mathit{\Sigma }}_1{\mathit{Q}}^H$$

(42)

where $\mathit{Q}\in {ℂ}^{N_t\times N_t}$ is the unitary matrix and ${\mathit{\Sigma }}_1\in {ℂ}^{N_t\times N_t}$ is a diagonal matrix that contains eigenvalues. Let ${\mathit{q}}_s^{opt}\in {ℂ}^{N_t\times 1}$ be the optimal unconstrained precoding vector corresponding to the maximum eigenvalue in ${\mathit{\Sigma }}_1\in {ℂ}^{N_t\times N_t}$, which can be modeled as

$${\mathit{q}}_s^{opt}=\mathit{Q}\left(:,s\right).$$

(43)

The required analog beamforming vector ${\mathit{v}}_{RF}\in {ℂ}^{N_t\times 1}$ can be determined by defining a problem that finds its projection as close as possible to the unconstrained optimal precoding vector in (43). Hence,

$$\underset{{\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}}{\min }{\Vert {\mathit{q}}_s^{opt}-{\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}\Vert }_2^2$$

$$s. t. \left|{\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}\left(x\right)\right|=\frac{1}{\sqrt{N_t}}, \forall x\in \left\{1,\dots ,N_t\right\}, s\in \left\{1,\dots ,N_t^{RF}\right\}.$$

(44)

Similar to (36), the objective function in (44) can be written as

$${\Vert {\mathit{q}}_s^{opt}-{\left({\mathit{v}}_{RF}\right)}^{\left[\mathit{s}\right]}\Vert }_2^2=2-2 Tr\left(ℝ\left[{\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}{\left({\mathit{q}}_s^{opt}\right)}^H\right]\right).$$

(45)

It is obvious from (45) that the optimal solution of (44) can be determined when ${\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}$ has the same phase-values as ${\mathit{q}}_s^{opt}$. Therefore, the required analog RF precoding vector can be obtained using these phase-values as

$${\left({\mathit{v}}_{RF}\right)}^{\left[s\right]}=\left(\frac{1}{\sqrt{N_t}}\right)\exp \left\{j arg\left({\mathit{q}}_s^{opt}\right)\right\}.$$

(46)

Finally, the ${\mathit{V}}_{RF}$ is formulated as

$${\mathit{V}}_{RF}=\left[{\left({\mathit{v}}_{RF}\right)}^{\left[1\right]},\dots ,{\left({\mathit{v}}_{RF}\right)}^{\left[N_t^{RF}\right]}\right].$$

(47)

This completes the derivation of the source analog beamformer common to all sub-carriers. It is worth mentioning that the analog beamforming vectors, so obtained, have the largest projections on the respective eigenmodes. This implies that they cast maximum energy along those eigenmode directions [52]. Furthermore, they achieve maximum beamforming gain and facilitate keeping interference experienced by PU within an acceptable limit. The frequency-selective digital baseband precoders and combiners at the source and relay node, respectively, can be obtained by diagonalizing the equivalent channel observed from the baseband processing units. Therefore, ${\mathit{H}}_{eq}\left[n\right]={\mathit{U}}_R\left[n\right]{\mathit{\Sigma }}_{SR}\left[n\right]{\mathit{V}}_S^H\left[n\right]$, ${\mathit{V}}_{BB}\left[n\right]={\mathit{V}}_S\left[n\right]\left(:,1:K{N}_s\right)$, ${\mathit{F}}_{BB,1}\left[n\right]={\mathit{U}}_R\left[n\right]\left(:,1:K{N}_{\mathit{s}}\right)$.

In the second transmission phase, the processed received signal ${\mathit{y}}_{d_k}\left[n\right]$ (13) at the $k$-th SU in terms of the baseband transmitted signal ${\mathit{y}}_{r_1}\left[n\right]$ (8) from the relay node can be characterized as

$${\mathit{y}}_{d_k}\left[n\right]={\left({\mathit{W}}_{RF,k}{\mathit{W}}_{BB,k}\left[n\right]\right)}^H\left\{{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]{\mathit{y}}_{r_1}\left[n\right]+{\mathit{z}}_k\left[n\right]\right\}.$$

(48)

The compact representation of (48) is given as

$${\mathit{y}}_{d_k}^c\left[n\right]={\mathit{W}}_{d_k}^H\left[n\right]{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_2\left[n\right]{\mathit{s}}^R\left[n\right]+{\mathit{W}}_{d_k}^H\left[n\right]{\mathit{z}}_k\left[n\right].$$

(49)

Another useful representation of (49) is characterized as

$${\overline{\mathit{y}}}_{d_k}\left[n\right]={\mathit{W}}_{d_k}^H\left[n\right]{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_2^k\left[n\right]{\mathit{s}}_k^R\left[n\right]+{\displaystyle \sum }_{j=1,j\ne k}^K{\mathit{W}}_{d_j}^H\left[n\right]{\mathit{G}}_{d_j}\left[n\right]{\mathit{F}}_2^j\left[n\right]{\mathit{s}}_j^R\left[n\right]+{\mathit{W}}_{d_k}^H\left[n\right]{\mathit{z}}_k\left[n\right],$$

(50)

where ${\mathit{W}}_{d_k}\left[n\right]={\mathit{W}}_{RF,k}{\mathit{W}}_{BB,k}\left[n\right]\in {ℂ}^{N_{d_k}\times N_{\mathit{s}}}$, ${\mathit{F}}_2\left[n\right]={\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}\left[n\right]\in {ℂ}^{N_r\times K{N}_s}$, ${\mathit{F}}_2^k\left[n\right]={\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}^k\left[n\right]$, and ${\mathit{s}}^R\left[n\right]={\mathit{y}}_{r_1}\left[n\right]\in {ℂ}^{K{N}_{\mathit{s}}\times 1}$. The mutual information between ${\mathit{y}}_{r_1}\left[n\right]$ (8) and ${\mathit{y}}_{d_k}\left[n\right]$ (13) can be expressed as

$$\overline{\mathfrak{T}}\left({\mathit{s}}^R\left[n\right]={\mathit{y}}_{r_1}\left[n\right];{\mathit{y}}_{d_k}\left[n\right]\right)$$

$$={\log }_2\det \left({\mathit{I}}_{N_s}+{\left({\mathit{R}}_{N_2}^k\right)}^{-1}\left({\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{G}}_{eq,k}\left[n\right]{\mathit{F}}_{BB,2}^{\left[k\right]}\left[n\right]\right){\left({\mathit{W}}_{BB,k}^H\left[n\right]{\mathit{G}}_{eq,k}\left[n\right]{\mathit{F}}_{BB,2}^{\left[k\right]}\left[n\right]\right)}^H\right)$$

(51)

where ${\mathit{R}}_{N_2}^k={\mathit{W}}_{d_k}^H\left[n\right]\left(\sum _{j\ne k}\left({\mathit{G}}_j\left[n\right]{\mathit{F}}_2^j\left[n\right]\right){\left({\mathit{G}}_j\left[n\right]{\mathit{F}}_2^j\left[n\right]\right)}^H+{\sigma }_k^2{\mathit{I}}_{N_{d_k}}\right){\mathit{W}}_{d_k}\left[n\right]$ is the residual IUI plus noise. To derive the hybrid beamforming components at the relay station and multiple SUs, the optimization problem defined in (25) is reformulated as

$$\underset{{\left\{{\mathit{F}}_{BB,2}\left[n\right],{\mathit{F}}_{RF,2}, {\mathit{W}}_{RF,k},{\mathit{W}}_{BB,k}\left[n\right]\right\}}_{k=1, n=1}^{K, N_{sub}}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\left({\displaystyle \sum }_{k=1}^K\overline{\mathfrak{T}}\left({\mathit{s}}^R\left[n\right]={\mathit{y}}_{r_1}\left[n\right];{\mathit{y}}_{d_k}\left[n\right]\right)\right)$$

$$s. t. \{\begin{array}{c}{d}_2\left[n\right]\le J_2,\\ \left|{\mathit{F}}_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}} ,\left|{\mathit{W}}_{RF,k}\left(x,y\right)\right|=\frac{1}{\sqrt{N_{d_k}}}, \forall x,y\end{array},$$

(52)

where $\overline{\mathfrak{T}}$ is given in (51). The joint optimization is relaxed and decouples the optimization of the analog RF and digital baseband processing components to address the problem in (52). Therefore, an endeavor is made to reduce the objective function of (52) that contains the phase-only processing matrices. The mutual information in (51), after ignoring digital baseband processing matrices, reduces to the following form:

$$\tilde{\mathfrak{T}}={\log }_2\det \left({\mathit{I}}_{N_{d_k}^{RF}}+{\left(\widetilde{{\mathit{R}}_{N_2}^k}\right)}^{-1}\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right){\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right)}^H\right),$$

(53)

where $\widetilde{{\mathit{R}}_{N_2}^k}={\mathit{W}}_{RF,k}^H\left(\sum _{j\ne k}\left({\mathit{G}}_j\left[n\right]{\mathit{F}}_{RF,2}\right){\left({\mathit{G}}_j\left[n\right]{\mathit{F}}_{RF,2}\right)}^H+{\sigma }_k^2{\mathit{I}}_{N_{d_k}}\right){\mathit{W}}_{RF,k}\in {ℂ}^{N_{d_k}^{RF}\times N_{d_k}^{RF}}$. After designing the RF beamforming matrices in (53), the transmit digital baseband processing matrices ${\mathit{F}}_{BB,2}^{\left[k\right]}\left[n\right], \forall k,$ can be determined from the equivalent baseband channels ${\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}$, $\forall k$, using the block diagonalization (BD) method. This leads to the conclusion that $\left({\mathit{G}}_k\left[n\right]{\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}^j\left[n\right]\right){\left({\mathit{G}}_k\left[n\right]{\mathit{F}}_{RF,2}{\mathit{F}}_{BB,2}^j\left[n\right]\right)}^H=\mathbf{0}$, $\forall j\ne k$. Therefore, $\widetilde{{\mathit{R}}_{N_2}^k}$ in (53) can be simplified as $\widetilde{{\mathit{R}}_{N_2}^k}={\sigma }_k^2{\mathit{W}}_{RF,k}^H{\mathit{W}}_{RF,k}$. To this end, the mutual information in (53) can be expressed as

$${\left(\tilde{\mathfrak{T}}\right)}^{\left[k\right]}={\log }_2\det \left({\mathit{I}}_{N_{d_k}^{RF}}+{\mathit{K}}^{-1}\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right){\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right)}^H\right)$$

(54)

where $\mathit{K}={\sigma }_k^2{\mathit{W}}_{RF,k}^H{\mathit{W}}_{RF,k}$.

Using (54), the optimization problem (52) can be transformed into the following form:

$$\underset{{\left\{{\mathit{F}}_{RF,2}, {\mathit{W}}_{RF,k}\right\}}_{k=1, n=1}^{K, N_{sub}}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\left({\displaystyle \sum }_{k=1}^K{\left(\tilde{\mathfrak{T}}\right)}^{\left[k\right]}\right)$$

$$s. t. \{\begin{array}{c}{d}_2\left[n\right]\le J_2,\\ \left|{\mathit{F}}_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}} ,\left|{\mathit{W}}_{RF,k}\left(x,y\right)\right|=\frac{1}{\sqrt{N_{d_k}}}, \forall x,y\end{array},$$

(55)

Notice that each term of ${\sum }_{k=1}^K{\left(\tilde{\mathfrak{T}}\right)}^{\left[k\right]}={\left(\tilde{\mathfrak{T}}\right)}^{\left[1\right]}+\cdots +{\left(\tilde{\mathfrak{T}}\right)}^{\left[K\right]}$ contains the frequency-flat analog beamformer ${\mathit{F}}_{RF,2}$ and user-specific RF combiner, i.e., ${\mathit{W}}_{RF,k}$, $\forall k\in \left\{1,\dots ,K\right\}$. Furthermore, the mutual information maximization in (55) depends on the following function.

$$F\left({\mathit{F}}_{RF,2},{\mathit{W}}_{RF,k}\right)=Tr\left\{\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right){\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right)}^H\right\}.$$

(56)

Therefore, the optimization problem for deriving the analog combiner at the $k$-th SU and the phase-only precoder at the relay node can be written as

$$\underset{{\left\{{\mathit{F}}_{RF,2}, {\mathit{W}}_{RF,k}\right\}}_{ n=1}^{N_{sub}}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\left(Tr\left\{\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right){\left({\mathit{W}}_{RF,k}^H{\mathit{G}}_{d_k}\left[n\right]{\mathit{F}}_{RF,2}\right)}^H\right\}\right)$$

$$s. t. \{\begin{array}{c}{d}_2\left[n\right]\le J_2,\\ \left|{\mathit{F}}_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}} ,\left|{\mathit{W}}_{RF,k}\left(x,y\right)\right|=\frac{1}{\sqrt{N_{d_k}}}, \forall x,y\end{array},$$

(57)

It is highly probable that ${\mathit{F}}_{RF,2}{\mathit{F}}_{RF,2}^H\approx N_r{\mathit{I}}_r$ when a large number of antennas are deployed at the relay station [29], note that mm-wave transmission can support a large number of antennas at the transmitter/receiver due to short wavelength. Moreover,

$${\mathit{C}}_k=\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}{\mathit{G}}_{d_k}\left[n\right]{\mathit{G}}_{d_k}^H\left[n\right]\in {ℂ}^{N_{d_k}\times N_{d_k}}$$

(58)

Considering the assumption of the large antenna array and the average of the covariance matrices of frequency domain channels (58), the problem in (57) can be reformulated as

$$\underset{ {\mathit{W}}_{RF,k}}{\max }Tr\left({\mathit{W}}_{RF,k}^H{\mathit{C}}_k {\mathit{W}}_{RF,k}\right)$$

$$s. t. \{\begin{array}{c}{d}_2\left[n\right]\le J_2,\\ \left|{\mathit{W}}_{RF,k}\left(x,y\right)\right|=\frac{1}{\sqrt{N_{d_k}}}, \forall x,y\end{array}.$$

(59)

This problem is the same as that of (32), and hence, the same solution methodology can be applied to determine the analog combiner at the $k$-th SU. The main design steps are shown here to avoid repetition. The EVD of ${\mathit{C}}_k$ in (58) is given as

$${\mathit{C}}_k=\mathit{L}{\mathit{\Sigma }}_2{\mathit{L}}^H,$$

(60)

where $\mathit{L}\in {ℂ}^{N_{d_k}\times N_{d_k}}$ is the unitary matrix and ${\mathit{\Sigma }}_2\in {ℂ}^{N_{d_k}\times N_{d_k}}$ is a diagonal matrix that contains eigenvalues. Let ${\mathit{l}}_g^{opt}\in {ℂ}^{N_{d_k}\times 1}$ be the optimal combining vector that corresponds to the maximum eigenvalue in ${\mathit{\Sigma }}_2$, which is given as

$${\mathit{l}}_g^{opt}=\mathit{L}\left(:,g\right).$$

(61)

The required analog combining vector ${\mathit{l}}_{RF}\in {ℂ}^{N_{d_k}\times 1}$ can be obtained by finding its projection as close as possible to the optimal combining vector in (43). Hence,

$$\underset{{\left({\mathit{l}}_{RF}\right)}^{\left[g\right]}}{\min }{\Vert {\mathit{l}}_g^{opt}-{\left({\mathit{l}}_{RF}\right)}^{\left[g\right]}\Vert }_2^2$$

$$s. t. \left|{\left({\mathit{l}}_{RF}\right)}^{\left[g\right]}\left(x\right)\right|=\frac{1}{\sqrt{N_{d_k}}}, \forall x\in \left\{1,\dots ,N_{d_k}\right\}, g\in \left\{1,\dots ,N_{d_k}^{RF}\right\},$$

(62)

The minimum of the objective function in (62) occurs when ${\left({\mathit{l}}_{RF}\right)}^{\left[g\right]}$ has the same phase-values as ${\mathit{l}}_g^{opt}$. Therefore, ${\left({\mathit{l}}_{RF}\right)}^{\left[g\right]}$ can be obtained using the following relation.

$${\left({\mathit{l}}_{RF}\right)}^{\left[g\right]}=\left(\frac{1}{\sqrt{N_t}}\right)\exp \left\{j arg\left({\mathit{l}}_g^{opt}\right)\right\}.$$

(63)

Finally, the ${\mathit{W}}_{RF, k}$ is formulated as

$${\mathit{W}}_{RF, k}=\left[{\left({\mathit{l}}_{RF}\right)}^{\left[1\right]},\dots ,{\left({\mathit{l}}_{RF}\right)}^{\left[N_{d_k}^{RF}\right]}\right].$$

(64)

To derive the common analog precoder F_RF,2, the mutual information in (54) can be characterized as

$$\stackrel{=}{\mathfrak{T}}={\log }_2\det \left({\mathit{I}}_{K{N}_s}+\left({\mathit{W}}_{RF}^H\mathit{G}\left[n\right]{\mathit{F}}_{RF,2}\right){\left({\mathit{W}}_{RF}^H\mathit{G}\left[n\right]{\mathit{F}}_{RF,2}\right)}^H\right),$$

$$={\log }_2\det \left({\mathit{I}}_{K{N}_s}+{\left({\mathit{W}}_{RF}^H\mathit{G}\left[n\right]{\mathit{F}}_{RF,2}\right)}^H\left({\mathit{W}}_{RF}^H\mathit{G}\left[n\right]{\mathit{F}}_{RF,2}\right)\right),$$

(65)

where $\mathit{G}\left[n\right]={\left[{\mathit{G}}_{d_1}^T\left[n\right],\dots ,{\mathit{G}}_{d_K}^T\left[n\right]\right]}^T\in {ℂ}^{K{N}_{d_k}\times N_r}$, ${\mathit{W}}_{RF}=bd\left[{\mathit{W}}_{RF,1},\dots ,{\mathit{W}}_{RF,K}\right]\in {ℂ}^{K{N}_{d_k}\times K{N}_{d_k}^{RF}}$, and $bd(.)$ is a block-diagonal operator of given matrices. Using (65), the optimization problem to adjust ${\mathit{F}}_{RF,2}$ in accordance with ${\mathit{W}}_{RF,k}, \forall k$ is defined as

$$\underset{{\mathit{F}}_{RF,2}}{\max }\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}\left(Tr\left\{{\left({\mathit{W}}_{RF}^H\mathit{G}\left[n\right]{\mathit{F}}_{RF,2}\right)}^H\left({\mathit{W}}_{RF}^H\mathit{G}\left[n\right]{\mathit{F}}_{RF,2}\right)\right\}\right)$$

$$s. t. \{\begin{array}{c}{d}_2\left[n\right]\le J_2,\\ \left|{\mathit{F}}_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}} ,\forall x,y\end{array}.$$

(66)

Similar to (54), this problem can also be transformed into the following form:

$$\underset{ {\mathit{F}}_{RF,2}}{\max }Tr\left({\mathit{F}}_{RF,2}^H\mathit{D} {\mathit{F}}_{RF,2}\right)$$

$$s. t. \{\begin{array}{c}{d}_2\left[n\right]\le J_2,\\ \left|{\mathit{F}}_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}} ,\forall x,y\end{array},$$

(67)

where $\mathit{D}\in {ℂ}^{N_r\times N_r}$ is defined as

$$\mathit{D}=\left(\frac{1}{N_{sub}}\right){\displaystyle \sum }_{n=1}^{N_{sub}}{\mathit{G}}^H\left[n\right]{\mathit{W}}_{RF}{\mathit{W}}_{RF}^H\mathit{G}\left[n\right].$$

(68)

When EVD is performed on $\mathit{D}$ then

$$\mathit{D}=\mathit{E}{\mathit{\Sigma }}_3{\mathit{E}}^H,$$

(69)

where $\mathit{D}\in {ℂ}^{N_r\times N_r}$ is the unitary matrix and ${\mathit{\Sigma }}_3\in {ℂ}^{N_r\times N_r}$ is a diagonal matrix that contains eigenvalues. Using (69), the optimal relay precoder ${\mathit{F}}_{opt}\in {ℂ}^{N_r\times N_r^{RF}}$ can be derived as

$${\mathit{F}}_{opt}=\mathit{E}\left(:,1:N_r^{RF}\right),$$

(70)

Now, a problem is formulated for designing the required RF precoder at the relay station as follows:

$$\underset{{\mathit{F}}_{RF,2}}{\min }{\Vert {\mathit{F}}_{opt}-{\mathit{F}}_{RF,2}\Vert }_F^2$$

$$s. t. \left|{\mathit{F}}_{RF,2}\left(x,y\right)\right|=\frac{1}{\sqrt{N_r}} ,\forall x,y.$$

(71)

It is obvious from (71) that the exact lower bound ${\Vert {\mathit{F}}_{opt}-{\mathit{F}}_{RF,2}\Vert }_F^2=0$ cannot be achieved due to the element-wise constant amplitude constraints imposed by ${\mathit{F}}_{RF,2}$. In addition, these unavoidable constraints cannot be relaxed permanently. Therefore, an attempt is made to find ${\mathit{F}}_{RF,2}$ such that ${\Vert {\mathit{F}}_{opt}-{\mathit{F}}_{RF,2}\Vert }_F^2\approx 0$. It is known that the objective function in (71) can be expressed as

$${\Vert {\mathit{F}}_{opt}-{\mathit{F}}_{RF,2}\Vert }_F^2={\Vert {\mathit{F}}_{opt}\Vert }_F^2+{\Vert {\mathit{F}}_{RF,2}\Vert }_F^2-2Tr\left\{ℝ\left({\mathit{F}}_{RF,2}{\mathit{F}}_{opt}^H\right)\right\}.$$

(72)

Again, the minimum value in (72) is achieved when ${\mathit{F}}_{RF,2}$ has the same phase-values as ${\mathit{F}}_{opt}$. Hence, ${\mathit{F}}_{RF,2}$ is obtained as

$${\mathit{F}}_{RF,2}=\left(\frac{1}{\sqrt{N_r}}\right)\exp \left\{j\mathit{arg}\left({\mathit{F}}_{opt}\right)\right\}.$$

(73)

After designing ${\mathit{F}}_{RF,2}$ and ${\mathit{W}}_{RF,k}$, the part of the secondary network (SN) from the relay station to the SUs can be described in terms of the equivalent baseband channels, i.e., ${\mathit{G}}_{eq, k}\left[n\right]={\mathit{W}}_{RF,k}^H{\mathit{G}}_k\left[n\right]{\mathit{F}}_{RF,2}, \forall k,n$. Therefore, the frequency-dependent digital baseband processing components ${\mathit{F}}_{BB,2}\left[n\right]$ and ${\mathit{W}}_{BB,k}\left[n\right]$ can be determined using conventional techniques by exploiting ${\mathit{G}}_{eq, k}\left[n\right],\forall k, n$. Note that the prime objective behind the design of baseband processing matrices is to minimize the impact of interference caused by unintended users. To achieve this goal, the BD technique is applied to derive the required baseband processing solution. The overall channel ${\mathit{G}}_{eq}^{RD}\left[n\right]\in {ℂ}^{N_r^{RF}\times K{N}_{d_k}^{RF}}$ from the relay station to K SUs can be expressed as

$${\mathit{G}}_{eq}^{RD}\left[n\right]={\left[{\mathit{G}}_{eq,1}^T\left[n\right],\dots ,{\mathit{G}}_{eq,K}^T\left[n\right]\right]}^T\in {ℂ}^{N_r^{RF}\times K{N}_{d_k}^{RF}},$$

(74)

It is desirable to satisfy the condition ${\mathit{G}}_{eq,j}^{RD}\left[n\right]{\mathit{F}}_{BB,2}^{\left[k\right]}\left[n\right]=0, j\ne k$ while designing the baseband precoding solution at the relay station. For this purpose, another matrix ${\tilde{\mathit{G}}}_{eq,k}^{RD}\left[n\right]\in {ℂ}^{N_r^{RF}\times \left(K-1\right)N_{d_k}^{RF}}$ is defined by employing the individual baseband equivalent channels except for the intended user. Therefore,

$$\begin{gathered} {\tilde{\mathit{G}}}_{eq,k}^{RD}\left[n\right]={\left[{\mathit{G}}_{eq,1}^T\left[n\right],\dots ,{\mathit{G}}_{eq,K-1}^T\left[n\right],{\mathit{G}}_{eq,K+1}^T\left[n\right],\dots ,{\mathit{G}}_{eq,K}^T\left[n\right]\right]}^T \\ =\left({\tilde{\mathit{U}}}_{eq,k}^{RD}\left[n\right]\right)\left[\begin{array}{cc}{\tilde{\mathit{\Sigma }}}_{eq,k}^{RD}\left[n\right]& 0\\ 0& 0\end{array}\right]{\left[{\left({\tilde{\mathit{V}}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(1\right)}{\left({\tilde{\mathit{V}}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(0\right)}\right]}^H, \end{gathered}$$

(75)

To enable transmission of data streams with minimum IUI, it is required to derive the baseband precoder at the relay node corresponding to the $k$-th SU ${\mathit{F}}_{BB,2}^{\left[k\right]}\left[n\right]\in {ℂ}^{N_r^{RF}\times N_s}$ that lies in the null space of (75). This null space can be found by performing SVD on (75). It is worth highlighting that the number of transmit antennas at the source and the relay station must be greater than or equal to the total number of antennas at all the SUs for applying the BD method. This is due to the fact that the dimension of the null space of unintended users must be greater than 0 to transmit data streams to the intended user. This condition leads to the following restriction [53]:

$$rank \left({\tilde{\mathit{G}}}_{eq,k}^{RD}\left[n\right]\right)={ℛ}_k<N_r,$$

(76)

where $rank (.)$ describes the rank of a matrix. It is obvious from (75) that the orthonormal bases ${\left({\tilde{\mathit{V}}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(0\right)}$ of the required null space can be determined by selecting the last $\left(N_r-{ℛ}_k\right)$ right singular vectors of ${\tilde{\mathit{G}}}_{eq,k}^{RD}\left[n\right]$. Therefore, the BD matrix can be modeled as

$${\tilde{\mathit{G}}}_{eq,k}^{RD}\left[n\right]{\left({\tilde{\mathit{V}}}_{eq,i}^{RD}\left[n\right]\right)}^{\left(0\right)}=\left\{\begin{array}{c}0, i\ne k\\ {\tilde{\mathit{G}}}_{eq,k}^{RD}\left[n\right]{\left({\tilde{\mathit{V}}}_{eq,i}^{RD}\left[n\right]\right)}^{\left(0\right)}, i=k\end{array}\right\}$$

$$=\left[\begin{array}{ccc}{\tilde{\mathit{G}}}_{eq,1}^{RD}\left[n\right]{\left({\tilde{\mathit{V}}}_{eq,1}^{RD}\left[n\right]\right)}^{\left(0\right)}& \cdots & \mathbf{0}\\ ⋮& \ddots & ⋮\\ \mathbf{0}& \cdots & {\tilde{\mathit{G}}}_{eq,K}^{RD}\left[n\right]{\left({\tilde{\mathit{V}}}_{eq,K}^{RD}\left[n\right]\right)}^{\left(0\right)}\end{array}\right].$$

(77)

Using (77), the beamforming vectors that span the sub-space in the direction of the intended user can be derived as

$${\mathit{G}}_{eq, k}\left[n\right]{\left({\tilde{\mathit{V}}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(0\right)}=\left({\mathit{U}}_{eq,k}^{RD}\left[n\right]\right)\left[\begin{array}{cc}{\mathit{\Sigma }}_{eq,k}^{RD}\left[n\right]& 0\\ 0& 0\end{array}\right]{\left[{\left({\mathit{V}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(1\right)}{\left({\mathit{V}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(0\right)}\right]}^H,\forall k.$$

(78)

This technique minimizes the impact of interference while data streams are transmitted from the relay node to the intended SU. Finally, the baseband precoding matrix for the transmission of $N_s$ data streams to the $k$-th SU are modeled as

$${\mathit{F}}_{BB,2}^{\left[k\right]}\left[n\right]={\left({\mathit{V}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(1\right)}{\left({\tilde{\mathit{V}}}_{eq,k}^{RD}\left[n\right]\right)}^{\left(0\right)}, \forall k.$$

(79)

The repetition of the same process for other SUs leads to the derivation of baseband precoding matrix by concatenating the individual digital precoders corresponding to multiple SUs as

$${\mathit{F}}_{BB,2}\left[n\right]=\left[{\mathit{F}}_{BB,2}^{\left[1\right]}\left[n\right],\dots ,{\mathit{F}}_{BB,2}^{\left[K\right]}\left[n\right]\right]$$

(80)

Note: The proposed algorithm has the potential to show sum spectral efficiency close to its fully digital counterpart at relatively higher frequencies (70/80 GHz). Also, it achieves significantly higher spectral efficiency when compared with other hybrid precoding techniques. It is worth mentioning that the proposed scheme is based on the assumption that array response vectors are frequency-independent below the THz range. From this perspective, the proposed technique can be generalized, which can show good performance for different frequency bands in the mm-wave spectrum. Although minor degradation in performance may occur at near THz frequencies, it is due to the fact that the beam squint effect begins at these frequencies, which decreases performance if not addressed properly. A summary of the proposed algorithm is given in Algorithm A1 (Appendix A).

4. Complexity Analysis

In the previous section, a hybrid wideband mm-wave transceiver design was proposed for relay-assisted MU-MIMO systems by taking the CR communication framework into consideration. The main purpose behind this design was to enhance the efficiency of spectrum utilization by avoiding spectral congestion that may occur in response to the significantly increasing number of connected devices. This section briefly analyzes the computational complexity of the proposed hybrid beamforming technique. In addition, a comparison is also made with other existing hybrid precoding techniques. The overall computational complexity is divided into two parts: (1) the analog beamforming solution and (2) the digital baseband processing design corresponding to each sub-carrier. The frequency-flat phase-only precoding or combining depends on the average of the covariance matrices of frequency domain channels. This operation requires matrix multiplication and addition. Hence, matrix multiplication can be considered a dominating factor that determines the computational complexity while designing the RF processing components. Furthermore, complex multiplication operations are computationally more expensive when compared with complex addition operations. However, they both are treated as one floating-point operation in calculating the computational burden. For instance, the product $\mathit{H}\left[n\right]{\mathit{H}}^H\left[n\right]$ in (31) requires approximately $N_r^2{N}_t$ floating-point operations to evaluate the resultant matrix. Furthermore, eigen decomposition is also required to extract the optimal phase values for designing the analog RF beamforming matrices at different communicating nodes. The computational complexity of this process is in the order of $O\left\{\max \left(N_r{}^3,N_r{}^2{N}_t{N}_{sub}\right)\right\}$. Finally, the computational cost for deriving the baseband processing components is in the order of $O\left({\left(N_r^{RF}\right)}^3\right)$. Therefore, the overall of the proposed algorithm can be approximated as

$$C^T\approx N_r^2{N}_t+N_r{}^2{N}_t{N}_{sub}+N_{d_k}^2{N}_r+N_{d_k}^2{N}_r{N}_{sub}+{\left(N_r^{RF}\right)}^3+{\left(N_{d_k}^{RF}\right)}^3.$$

(81)

Table 2. Complexity of the proposed design and other hybrid beamforming algorithms.

Algorithms	Complexity
Proposed	$N_r^2{N}_t+N_r{}^2{N}_t{N}_{sub}+N_{d_k}^2{N}_r+N_{d_k}^2{N}_r{N}_{sub}+{\left(N_r^{RF}\right)}^3+{\left(N_{d_k}^{RF}\right)}^3$
Hybrid Precoding [29]	$N_s^4+N_s^3\left(N_t+N_r\right)+{\left(N_t{N}_r{N}_s\right)}^2+N_{sub}\left(N_s^3+N_t^2{N}_r+N_r^2{N}_s\right)$
Hybrid Beamforming [33]	$N_{sub}\left(N_s^3+N_t^2{N}_r+N_t\left(N_s{}^2+N_r{}^2\right)\right)+N_s\left(N_t{}^2+N_r{}^2\right)$

summarizes the computational complexity of different hybrid beamforming schemes. On the other hand, Figure 2 demonstrates the computational burden of the proposed hybrid transceiver as a function of the number of antennas. In addition, the computational complexity of hybrid processing techniques proposed in [29,33] is also shown in Figure 2 for comparative analysis. For conducting computer simulations, the number of antennas is changed over a wide range by setting $N_{sub}=64$, $N_t^{RF}=N_r^{RF}=K{N}_s$, $N_d^{RF}=N_s=4$, and $K=4$. The obtained results indicate that the complexity of the proposed technique is less than the presented techniques in [29,33]. Specifically, a large performance gap is observed when compared with hybrid precoding design in [29]. For instance, when 100 antennas are deployed at the source and relay node, the computational complexity of the proposed algorithm is nearly 25 times less than the hybrid beamforming scheme presented in [29]. In a similar fashion, the computational cost of the proposed algorithm is 49 times less than the hybrid beamforming design given in [29] at 196 antennas.

5. Numerical Results

This section presents numerical results for evaluating the performance of the proposed hybrid beamforming design. It is important to mention that computer simulations are conducted by changing system parameters over a wide range to show the effectiveness of the proposed technique. Furthermore, two main conditions are applied regarding the generation of the channel matrix, such as perfect CSI and imperfect CSI, with different accuracy levels. Note that channel estimation error is taken into account to check the robustness of the proposed algorithm through simulation results. Let $\overline{\mathit{H}}\left[n\right]$ be the estimation channel matrix, which can be modeled as [54]

$$\overline{\mathit{H}}\left[n\right]=\delta \mathit{H}\left[n\right]+\sqrt{1-{\delta }^2}\mathit{E},$$

(82)

where $\mathit{H}\left[n\right]$ denotes the perfect channel matrix, $\delta \in \left[0, 1\right]$ indicates the accuracy factor of $\overline{\mathit{H}}\left[n\right]$, and $\mathit{E}$ describes the error matrix that contains independent and identically distributed $\left(i.i.d\right)$ Gaussian random variables, i.e., $\mathcal{C}\mathcal{N}\left(0, 1\right)$. The obtained results are compared with the corresponding fully digital solutions and other well-known hybrid transceiver designs to demonstrate the usefulness of the presented method. For the transmission of data streams, 64 sub-carriers are considered with an operating central frequency of 28 GHz. The proposed technique is also applicable to any arbitrary number of sub-carriers (e.g., 128, 256, etc.). All simulation results are averaged over 500 channel realizations.

Table 3. System parameters for computer simulations.

Parameters	Values
Number of data streams	$N_s=3~8$
Number of RF chains	$N_t^{RF}=N_r^{RF}=K{N}_s,N_{d_k}^{RF}=N_s$
Number of antennas	$N_t=N_r=100~256,N_{d_k}=9~64$
Number of data transmission paths	$N_{cl}=5,N_{ray}=10$
Number of frequency sub-carriers	$N_{sub}=64$
Carrier frequency	$f_c=28 \mathrm{GHz}$
Number of secondary users	$K=2~8$

shows the main simulation parameters.

5.1. Spectral Efficiency Evaluation by Changing Number of Antennas, SUs, and Data Streams

Figure 3 plots the sum spectral efficiency performance of the proposed algorithm when 225 antennas are installed at the source and relay station, i.e., $N_t=N_r=225$. The system is supposed to transmit data streams to K = 3 SUs, and there are 64 antennas deployed at each SU, i.e., $N_{d_k}=64$. Furthermore, the number of data streams transmitted to each SU is set as $N_s=\left\{4, 8\right\}$. In the mm-wave spectrum, the number of RF chains has a great impact on the performance of hybrid transceivers, but cost, power consumption, and hardware complexity also need to be considered for efficient design. From this perspective, computer simulations are conducted under the assumption that the number of RF chains is equal to the number of data streams at the respective communicating node, i.e., $N_t^{RF}=N_r^{RF}=K{N}_s$ and $N_{d_k}^{RF}=N_s$. The frequency domain mm-wave channel matrix is generated according to (18) by setting parameters as $N_{ray}=10, N_{cl}=5, {\sigma }_{\alpha ,i}^2=1 \forall i$, and angle-spread $AS={10}^{°}$. The unconstrained, full-complexity digital precoding solution is derived by minimizing interference among transmitted data streams. It can be seen from the obtained results that the proposed scheme achieves performance close to its fully digital counterpart. Moreover, it is also clear from the obtained curves that there is a minor increase in the performance gap between the proposed method and full-complexity beamforming when $N_s$ increases in a gradual fashion. This behavior indicates the consistent performance of the presented technique by effectively suppressing IUI, even with a large number of data streams per user. It is evident from simulation results that the performance of the proposed hybrid transceiver decreases in a gradual manner by increasing the channel estimation error. For instance, the performance curve of the proposed scheme at $\delta =0.9$ is close to the full-complexity precoding. However, the performance gap increases with an increase in the channel estimation error. Note that the estimated channel corresponding to the perfect channel can be obtained using (82).

Figure 4 describes the sum rate performance of the proposed algorithm by changing the number of antennas at communicating nodes, the number of SUs, and the number of data streams. The mm-wave channel parameters are the same as the ones considered in the previous case. In addition, the number of RF chains is equal to the number of data streams at the respective communicating units, i.e., $N_t^{RF}=N_r^{RF}=K{N}_s$ and $N_{d_k}^{RF}=N_s$ while conducting computer simulations. There are 144 antennas deployed at the source and relay, and 36 antennas are installed at each SU. The number of transmitted data streams is set as $N_s=\left\{4, 6\right\}$, and the system is assumed to serve K = 4 SUs. It is obvious from the obtained results that the proposed hybrid beamforming technique achieves performance close to fully digital precoding. It is also evident from the obtained curves that the proposed scheme shows consistent performance, irrespective of varying system parameters. Similar to the previous case, the performance is evaluated under imperfect CSI at different accuracy levels, i.e., $\delta =0.9, 0.8$. The obtained results illustrate that performance degradation occurs in a gradual manner as the accuracy factor decreases.

5.2. Impact of Number of Users

Figure 5 shows the sum spectral efficiency of the presented technique as a function of the number of users in the secondary network at SNR = 5 [dB]. The number of antennas deployed at different communicating nodes and per-user data streams are set as $N_t=N_r=225$, $N_{d_k}=25$, and $N_s=4$. The mm-wave channel matrices in the frequency domain are generated according to (18) using the same parameters as mentioned in the previous cases, and the estimated CSI is determined by employing (82). Simulation results indicate that the rate of change decreases as the number of SUs increases. This is due to the fact that the increasing number of users enhances IUI. However, the proposed scheme achieves performance close to fully digital precoding. It is also obvious from the obtained results that the channel estimation error causes minor performance degradation. The proposed algorithm outperforms hybrid beamforming schemes given in [37,38].

5.3. Impact of Number of Antennas

Figure 6 illustrates the sum spectral efficiency of the proposed hybrid transceiver as a function of the number of antennas at the source, relay station, and SUs. The other system parameters, such as the number of users and transmitted data streams, are kept constant to visualize the impact of this change. Since beams become narrow at a large number of transmit/receive antennas, this results in a high transmit/receive beamforming gain. Therefore, simulation results are obtained by deploying the number of antennas at communicating nodes in an increasing order. From this perspective, the number of antennas at the source, relay station, and each SU is set as 64 (16), 144 (36), and 256 (64). For simplicity, it is assumed that $N_t=N_r$ while $N_s$ and $K$ are set as 4 and 4, respectively. It is obvious from the obtained curves that spectral efficiency increases in a gradual manner by increasing the number of antennas at communicating units. In addition, the proposed scheme achieves performance close to fully digital precoding, irrespective of the number of antennas. It is also evident from the obtained results that minor degradation in performance occurs when an imperfect channel with a 15% error is taken into consideration.

Figure 7 plots the sum spectral efficiency of the proposed method by varying the number of antennas at the source and relay station. To visualize the impact of this change, the number of antennas at the SUs is kept constant. For simplicity, it is assumed that an equal number of antennas are deployed at the source and the relay station such that $N_t=N_r=\left\{100, 121, 144, 169, 196, 225, 256\right\}$. Simulation results are generated by setting other parameters as $N_{d_k}=16$, $K=5$, and $N_s=5$. It is assumed that $N_t^{RF}=N_r^{RF}=K{N}_s$ and $N_{d_k}^{RF}=N_s$. This selection of RF chains at different communicating nodes consumes minimum energy for signal transmission, as the number of RF chains cannot be less than the number of data streams. It is clear from the obtained curves that the rate of change decreases with an increase in the number of antennas. This is an indication that saturation will occur at a significantly large number of antennas. Again, there is a minor gap between the performance of the proposed algorithm and the corresponding fully digital precoding. This performance gap increases gradually with channel estimation error, as depicted in Figure 7.

Figure 8 plots the sum spectral efficiency of the proposed hybrid transceiver as a function of the number of antennas at the SUs. To see the effect of this change on the performance of the proposed method, the number of antennas at the source, the number of antennas at the relay station, the number of SUs, and the number of transmitted data streams per SU are kept constant. For conducting computer simulations, these parameters are set as $N_t=N_r=256$, $K=4$, and $N_{\mathit{s}}=4$. The number of antennas deployed at each SU is set as N_dk = {9, 16, 25, 36, 49, 64}. It is clear from the obtained results that spectral efficiency increases by increasing the number of antennas at SUs. This result is in accordance with the concept that a large number of antennas enhances beamforming gain without deploying expensive RF chains. The obtained curves at different SNR values are close to their full-complexity solution, and minor performance degradation occurs when imperfect CSI is considered with $\delta =0.85$.

5.4. Impact of Number of Data Streams

Figure 9 illustrates the performance of the proposed method by varying the number of data streams per user. The number of SUs and the number of antennas at communicating nodes are set as $K=4$, $N_t=N_r=225$, and $N_{d_k}=49$. The sum spectral efficiency is evaluated at SNR = −10, −5, 0 [dB] when the number of data streams changes from 1 to 10. The mm-wave channel matrix is generated using (18), while the estimated channel matrix is determined according to (82) with $\delta =0.85$. It is clear from the obtained results that spectral efficiency increases with the number of data streams. Furthermore, the performance gap between the proposed scheme and full-complexity beamforming increases at a relatively large number of data streams. It is also evident from the obtained curves that nearly consistent performance is achieved at different SNR values. There is a slight decrease in performance when imperfect CSI with a 15% error is considered.

5.5. Performance Evaluation with Well-Known Hybrid Precoding Techniques

Figure 10a–c illustrate the performance of the proposed technique in comparison to the full-complexity solution and the hybrid precoding algorithms given in [37,38]. Simulation results are obtained by changing the number of antennas, transmitted data streams, and the number of users in the secondary network. Figure 10a plots the sum spectral efficiency of different algorithms when system parameters are set as $N_t=N_r=121$, $N_{d_k}=25$, $K=3$, $N_s=6$, and $N_{sub}=64$. It is clear from the obtained curves that the proposed scheme achieves performance close to full complexity precoding even in the presence of channel estimation error. Moreover, the presented method achieves significantly better performance when compared with hybrid transceivers proposed in [37,38]. Figure 10b illustrates the sum spectral efficiency of different algorithms. To generate simulation results, the system parameters are given as $N_t=N_r=100$, $N_{d_k}=16$, $K=4$, $N_s=5$, and $N_{sub}=64$. The obtained results indicate that the performance of the proposed method is close to fully digital beamforming. In addition, a slight decrease in performance occurs due to channel imperfections. Also, the proposed algorithm outperforms well-known hybrid transceiver designs given in [37,38].

Figure 10c shows the sum spectral efficiency of different precoding methods using system configuration parameters $N_t=N_r=144$, $N_{d_k}=25$, $K=5$, $N_s=4$, and $N_{sub}=64$. It is evident from the obtained curves that the performance of the proposed design is close to fully digital beamforming. Similar to the previous two results, the proposed scheme also outperforms hybrid processing solutions in [37,38].

Table 4. Sum spectral efficiency comparison with other hybrid precoding techniques when N_t = N_r = 121, $N_{d_k}=25$, $K=3$, $N_{\mathit{s}}=6$, and $N_{\mathit{s}ub}=64$.

Algorithms	Number of Antennas	${\mathit{N}}_{\mathit{s}}$	$\mathit{K}$	Sum Spectral Efficiency (%)
Proposed (perfect CSI)	$N_t=N_r=121$,$N_{d_k}=25$	6	3	94.66
Proposed with $\delta =0.9$	$N_t=N_r=121$,$N_{d_k}=25$	6	3	90.93
Proposed in [38]	$N_t=N_r=121$,$N_{d_k}=25$	6	3	85.14
Proposed in [37]	$N_t=N_r=121$,$N_{d_k}=25$	6	3	83.41

,

Table 5. Sum spectral efficiency comparison with other hybrid precoding techniques when $N_t=N_r=100$, $N_{d_k}=16$, $K=4$, $N_{\mathit{s}}=5$, and $N_{\mathit{s}ub}=64$.

Algorithms	Number of Antennas	${\mathit{N}}_{\mathit{s}}$	$\mathit{K}$	Sum Spectral Efficiency (%)
Proposed (perfect CSI)	$N_t=N_r=100$,$N_{d_k}=16$	5	4	95.14
Proposed with $\delta =0.85$	$N_t=N_r=100$,$N_{d_k}=16$	5	4	92.62
Proposed in [38]	$N_t=N_r=100$,$N_{d_k}=16$	5	4	80.50
Proposed in [37]	$N_t=N_r=100$,$N_{d_k}=16$	5	4	78.51

and

Table 6. Sum spectral efficiency comparison with other hybrid precoding techniques when $N_t=N_r=144$, $N_{d_k}=25$, $K=5$, $N_{\mathit{s}}=4$, and $N_{\mathit{s}ub}=64$.

Algorithms	Number of Antennas	${\mathit{N}}_{\mathit{s}}$	$\mathit{K}$	Sum Spectral Efficiency (%)
Proposed (perfect CSI)	$N_t=N_r=144$,$N_{d_k}=25$	4	5	96.70
Proposed with $\delta =0.85$	$N_t=N_r=144$,$N_{d_k}=25$	4	5	93.04
Proposed in [38]	$N_t=N_r=144$,$N_{d_k}=25$	4	5	61.63
Proposed in [37]	$N_t=N_r=144$,$N_{d_k}=25$	4	5	76.26

show the performance of the proposed algorithm in comparison to other hybrid beamforming techniques.

6. Conclusions

This study proposes a hybrid wideband mm-wave transceiver for sum rate maximization in relay-assisted MU-MIMO CRN by taking the transmitted power and interference constraints into consideration. The network architecture at hand is of great practical importance, as it enhances the efficiency of spectrum utilization. Also, the proposed hybrid transceiver improves cell coverage, transmission range, link quality, and spectral efficiency due to the deployment of a relay node between the source and end users. To maximize the sum rate of relay-assisted multiple SUs, the optimization problem is formulated to derive the hybrid beamforming components at different communicating nodes. The element-wise constant amplitude constraint associated with the RF processing solution makes the optimization problem non-convex and, hence, mathematically intractable. Therefore, the global optimal solution is quite a challenging task. The structural characteristics of DF relays and the notion of information theory help reduce the complexity associated with the solution of this problem. Hence, an endeavor is made to decompose the original complicated optimization problem into two single-hop sum rate maximization problems. Since the joint optimization of hybrid processing components is generally NP-hard, a decoupled design methodology is applied to find the analog RF and digital baseband processing components. The focus of the proposed algorithm is to maximize the sum rate of SUs by keeping interference experienced by the PU within an acceptable limit. Specifically, a decoupled approach is followed to derive the RF and baseband processing parts, where an attempt is made to maximize the achievable through analog beamformers, and interference minimization is achieved via frequency-dependent digital precoders and combiners. Computer simulations are conducted by changing system parameters over a wide range under imperfect CSI. The obtained results are close to full-complexity precoding and outperform other hybrid precoding techniques, which shows the usefulness of the proposed scheme.