Beamforming for Cooperative Non-Orthogonal Multiple Access with Full-Duplex Amplify-and-Forward Relaying

Abstract

We consider cooperative non-orthogonal multiple access (CNOMA) transmission through a full-duplex (FD) amplify-and-forward (AF) relay. Two CNOMA users are served by an access point (AP) through the FD-AF relay. FD relaying can be spectrally efficient if it sufficiently suppresses the self-interference (SI) from the FD operation. The feedback structure of the FD-AF relaying makes the convergence of the SI a critical problem because catastrophic system failure may occur if the system fails to converge the SI. The two linear beamforming algorithms proposed in this paper address these challenges, where the first is based on the zero-forcing (ZF) of the SI and the other regularizes the SI to improve the first scheme. The ZF-based algorithm completely nulls out the SI and incurs no convergence problems. In contrast, the second method is based on the regularization of SI to improve the first one, and it carefully locates the operating point of the FD-AF relay such that the system avoids the SI divergence. Numerical simulation results are provided to discuss the advantages and disadvantages of the proposed algorithms by comparing their average sum-rate performances.

1. Introduction

Future wireless systems, such as 6G networks, should serve a large number of small devices in addition to smartphones, which necessitates additional wireless resources and inevitable increases in service latency. Non-orthogonal multiple access (NOMA) technologies [1] are driven by the expectation that such technical challenges can be addressed. The power-domain NOMA [2,3,4] and sparse code multiple access [5,6] are the two most popular approaches to NOMA, because neither additional spectral resources nor new multiple access mechanisms are required. In contrast to orthogonal multiple access (OMA), where the signals toward different users are allocated to separate spectral resources, the signals in the power-domain NOMA are superposed into the common spectrum; hence, it economizes spectrum usage. This saving of the spectrum in NOMA comes at the expense of introducing inter-user interference (IUI) in the superposed common signal, which forces NOMA users to combat IUI using successive interference cancellation (SIC). Resource allocation [7,8] may improve the performance of NOMA systems, and the millimeter-wave band [9] is also a possible candidate spectrum for NOMA. Various applications of NOMA [10,11], as well as the cooperation with intelligent reflecting surfaces [12], have been addressed.

The spatial dimension has been used to improve various aspects of wireless transmission. For example, the performance of NOMA transmission can be improved using multiple antenna techniques [3,4,13,14,15,16,17,18,19,20,21,22] or relay-based cooperative transmission [23,24,25,26,27,28,29]. Feng et al. [30] investigated the optimal beamforming for a multi-antenna-based access point (AP) serving NOMA transmission. In cooperative NOMA systems, some user terminals (UTs) are near the AP, and others are far away. Some near UTs assist the AP transmission by relaying the AP NOMA signal to the far UTs, either in the half-duplex (HD) or full-duplex (FD) mode, where the additional transmission by relaying aids in overcoming the attenuation of the signal power owing to the long distances between the AP and far UTs. The FD mode in cooperative NOMA [24,25,31] overlaps the AP and relaying UT transmissions into the same spectral resource; hence, it does not require additional spectral resources to improve the spectrum efficiency. However, the overlap of the two transmissions causes self-interference (SI) at the relay receiver, and handling this SI is a challenging task. The advantage of FD relaying over HD relaying in terms of spectral efficiency relies heavily on how well the SI is suppressed.

References [31,32,33] addressed a beamforming approach to suppress SI in FD-relaying-based cooperative NOMA transmissions. In these studies, the FD relays (FDRs) operate in the decode-and-forward (DF) protocol. The DF relay decodes messages from the AP and reencodes them for transmission to the destination. Another popular relaying method uses the amplify-and-forward (AF) protocol, in which the relay simply transmits the signal to the amplified version of the received AP signal. Compared with the DF protocol, the AF protocol is computationally less demanding, but the divergence of the SI power in the FD mode may drive the system into catastrophic failure unless the SI at the relay is controlled within the saturation region [34,35]. This article addresses beamformer optimization for a cooperative NOMA system with FD-AF relaying, where multiple antenna sets are adopted at the AP and FDR. To the best of our knowledge, this topic has not been addressed elsewhere; thus, the proposed schemes are novel. The AP serves two NOMA UTs, where the AP signal toward the two UTs is blocked, and the cooperative FDR within the coverage of the AP relays the NOMA signal from the AP toward the UTs. Three beamformer vectors are to be optimized in this CNOMA system: the AP transmit beamformer and the receive/transmit beamformer vectors at the FDR.

These beamformer sets in the FD-AF-based cooperative NOMA system should be designed to maximize the NOMA transmission rate, whereas the SI at the relay is efficiently suppressed such that the system does not fall into the system failure region owing to diverging SI power. With these design constraints, we propose two algorithms to determine the set of beamformer vectors for the FD-AF-based cooperative NOMA system. The first is based on the zero forcing (ZF) of the SI, which exhibits reasonable performance with easy implementation and automatic avoidance of SI divergence. This algorithm relies on one-dimensional optimization to equate the signal-to-interference and noise power ratio (SINR) levels at two NOMA UTs. However, the regularization of the SI (leaving some amount of SI rather than perfectly removing it) is more optimal than the ZF approach of the first algorithm. Thus, the second proposed algorithm of regularized beamforming attempts to allow an optimal amount of SI to improve the ZF approach. The key challenge here is to maintain the FD-AF relay system within a stable region such that the SI does not diverge. This is achieved by locating the operating point (magnitude of the FDR transmit beamformer vector) within the convergence region of the SI. Again, the relay operating point is determined by iteratively solving two equations relating the magnitude of the FDR transmit beamformer vector and the SI power. The second algorithm has two nested one-dimensional optimizations, where the outer one equates the SINRs of the two NOMA UTs, whereas the inner one determines the amount of SI such that the SI remains within the safe region.

In numerical experiments, these two algorithms for determining the beamforming vectors were analyzed in terms of their sum-rate performance under three different settings. The AP power, FDR power, and number of antennas at the FDR were the key experimental parameters for discussing the advantages and disadvantages of the proposed algorithms. Section 2 presents the system and signal model of the proposed CNOMA with an FD-AF relay. In Section 3, two algorithms for the beamformer optimization of the CNOMA channel are presented. Section 4 presents and compares the performance of the proposed algorithms through several simulations and discusses their defects and merits. Finally, Section 5 concludes the paper.

Notation: We denote the Hermitian transpose and transpose of matrix $\mathbf{A}$ by ${\mathbf{A}}^H$ and ${\mathbf{A}}^T$, respectively. The rank of matrix $\mathbf{A}$ is returned by $\mathrm{rank}\left\{\mathbf{A}\right\}$, and $\mathbf{Tr}\left[\mathbf{A}\right]$ takes the trace of matrix $\mathbf{A}$. The ${\ell }_2$-norm of vector $\mathbf{a}$ is denoted by $\parallel \mathbf{a}\parallel$. The element at the i-th row and j-th column of $\mathbf{A}$ is denoted by ${\mathbf{A}}_{i,j}$. The diagonal matrix with the elements of the vector $\mathbf{v}$ on its diagonal is produced by the operation $\mathrm{diag}\left[\mathbf{v}\right]$. All zero vectors with k elements and all vectors with j elements are denoted by ${\mathbf{0}}_k$ and ${\mathbf{1}}_j$, respectively. The complex white Gaussian distribution of the random vector with a zero mean vector $\mathbf{0}$ and covariance matrix $\mathbf{C}$ is denoted by $\mathcal{CN}(\mathbf{0},\mathbf{C})$. The N-dimensional complex vector space is represented by ${\mathbb{C}}^N$. Finally, the expectation of x with respect to n is obtained using ${\mathbb{E}}_n\left[x\right]$.

2. System Model

Consider a cooperative non-orthogonal multiple access (CNOMA) system with an FD-AF relay (Figure 1). An AP with multiple antennas transmits messages to two UTs with a single antenna via an FD-AF relay station (FDR) with multiple-antenna sets. An $M\times 1$ beamformer vector $\mathbf{w}$ is used by the AP with ${\parallel \mathbf{w}\parallel }^2=P_a$, where M denotes the number of transmit antennas at the AP. Two CNOMA UTs ($U_1$ and $U_2$) are far from the AP, and the signal from the AP is relayed by the FDR to reach the UTs. The FDR has $M_t$ transmit antennas and $M_r$ receive antennas, and the FDR reception and transmission occur simultaneously in the same spectrum. The AF protocol-based FDR applies an $M_r\times 1$ unit-norm RS receive beamforming vector ${\mathbf{w}}_R$ with a power constraint $\parallel {\mathbf{w}}_R\parallel =1$ and an $M_t\times 1$ transmit beamformer ${\mathbf{w}}_T$ sequentially to the received signal from the AP after analog interference cancellation. These two steps can be replaced by a single application of an $M_t\times M_r$ matrix $\mathbf{W}={\mathbf{w}}_T{\mathbf{w}}_R^H$.

The $M_r\times M$ matrix ${\mathbf{H}}_r$ denotes the multiple-input multiple-output (MIMO) channel from the AP to the FDR. Two $M_t\times 1$ vectors ${\mathbf{g}}_1$ and ${\mathbf{g}}_2$ denote the multiple-input single-output (MISO) channel vectors from the FDR to the two UTs ($U_1$ and $U_2$), respectively. Finally, the loop-back channel (LBC) between the transmit/receive antenna sets of the FDR is denoted by the $M_r\times M_t$ matrix ${\mathbf{G}}_{rr}$, and SI at the FDR passes through this channel. Note that all the elements of ${\mathbf{H}}_r,{\mathbf{g}}_1$ and ${\mathbf{g}}_2$ are independent and identically distributed random variables with $\mathcal{CN}(0,1)$ distribution. However, the LBC is a rank-one matrix given by ${\mathbf{G}}_{rr}=\mathbf{u}{\mathbf{v}}^H$, where the $M_R\times 1$ vector $\mathbf{u}$ and $M_T\times 1$ vector $\mathbf{v}$ are unit norm vectors and direct to random directions. We assume that the direct paths from the AP to the UTs are heavily blocked; thus, the AP signal does not arrive at the UTs. We assume that the AP or FDR has full channel state information (CSI); therefore, all beamformer designs in this paper are conducted at one of these terminals. The impact of imperfect CSI, as well as CSI feedback delay, is a crucial issue in channel information feedback-based systems (In multiuser broadcast channel with multiple-rank transmission, the channel information of finite precision is a significant obstacle to achieving the scaling of transmit information rate according to the signal-to-noise power ratio (SNR) increase. The rate saturates even if the SNR increases when the feedback rate of channel information is fixed. This saturation can be understood by the fact that the imperfect CSI hinders the system from completely nulling out the IUI among the spatially scattered UTs. In contrast, the CNOMA in this paper considers single-rank transmission with no IUI among the users and has no concern for saturation as in the multiuser channel. Only some gap in the SNR between the perfect and imperfect CSI systems is expected in the system of this paper. Note also that the delay and the limited precision in the feedback of channel information have a similar impact on the sum-rate performance. Both deteriorate channel estimation quality. Therefore, the single-rank transmission of the proposed CNOMA system helps us avoid sum-rate saturation owing to the feedback delay).

The FD capability of relay systems enables us to spend only a single time slot of duration T for reception and transmission, which causes the feedback of its transmit signal toward the reception circuit of the FDR. The signal through the feedback channel (${\mathbf{G}}_{rr}$) generates SI. We assume a quasi-static channel; thus, the channels do not change during slot time T. If we set $x_S\left(n\right)$ as the source transmission signal at time instance n, the NOMA transmission forms a superposed signal $x_S\left(n\right)=x_1\left(n\right)+x_2\left(n\right)$. Here, the two signals $x_1\left(n\right)$ and $x_2\left(n\right)$ correspond to the encoded message signals toward $U_1$ and $U_2$, respectively. With the power-division ratio $\rho (\le 1)$, we have $\mathsf{E}\left[\right|x_S\left(n\right){|}^2]=1$, $\mathsf{E}\left[\right|x_1\left(n\right){|}^2]=\rho$ and $\mathsf{E}\left[\right|x_2\left(n\right){|}^2]=1-\rho$. The FD with the AF protocol causes the relay-received signal vector at time instance n to become (1), where the path loss of the loop-back channel at the FDR is denoted by a scalar ${\alpha }_R$. In addition, the FDR transmit vector at time n is denoted by ${\mathbf{x}}_R\left(n\right)$, and the $M_R\times 1$ vector ${\mathbf{n}}_R$ is an additive noise vector with $\mathcal{CN}(\mathbf{0},{\mathbf{I}}_{M_R})$ distribution. Before being multiplied by the relay beamformer $\mathbf{W}$, the FDR reception signal passes through an analog interference cancellation circuit, and the remaining SI and desired AP signal components are reduced within the dynamic range of the digital signal processing considered in this paper. If this analog SI suppression is insufficient, non-linear distortion of the SI can be introduced, and the signal model of this paper becomes less practical. Here, we assume that analog SI suppression is sufficiently effective such that we can model the intensity of the SI power to be reduced. Subsequently, the received signal vector can be modeled as $\acute{{\mathbf{y}}_R}\left(n\right)={\mathbf{H}}_r\mathbf{w}{x}_S\left(n\right)+\acute{{\alpha }_R}{\mathbf{G}}_{rr}{\mathbf{x}}_R\left(n\right)+{\mathbf{n}}_R\left(n\right)$, where the loop-back channel has a new path-loss as $\acute{{\alpha }_R}{\mathbf{G}}_{rr}$ ($\acute{{\alpha }_R}\ll {\alpha }_R$) [34]. Considering the inherent processing delay $\tau$ in the AF FDR, the relay transmit signal vector from the RS transmit antennas at time n can be expressed as ${\mathbf{x}}_R\left(n\right)=\mathbf{W}\acute{{\mathbf{y}}_R}(n-\tau )$ and rewritten as (2).   

$$\begin{array}{cc}\hfill {\mathbf{y}}_R\left(n\right)& ={\mathbf{H}}_r\mathbf{w}{x}_S\left(n\right)+{\alpha }_R{\mathbf{G}}_{rr}{\mathbf{x}}_R\left(n\right)+{\mathbf{n}}_R\left(n\right).\hfill \\ \hfill {\mathbf{x}}_R\left(n\right)& =\mathbf{W}\left[{\mathbf{H}}_r\mathbf{w}{x}_S(n-\tau )+\acute{{\alpha }_R}{\mathbf{G}}_{rr}{\mathbf{x}}_R(n-\tau )+{\mathbf{n}}_R(n-\tau )\right]\hfill \end{array}$$

(1)

$$\begin{array}{c}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\mathbf{W}\sum _{j=0}^{\infty }{\left(\acute{{\alpha }_R}{\mathbf{G}}_{rr}\mathbf{W}\right)}^j\left[{\mathbf{H}}_r\mathbf{w}{x}_S(n-(j+1)\tau )+{\mathbf{n}}_R(n-(j+1)\tau )\right].\end{array}$$

(2)

The terms with $j\ge 1$ form a loop-back interference signal (SI).

We can constrain the relay output power as

$$Tr\left[\mathsf{E}\left[{\mathbf{x}}_R{\left(n\right)}^H{\mathbf{x}}_R\left(n\right)\right]\right]=\parallel {\mathbf{w}}_T{\parallel }^2\mathsf{E}\left[\right|{\mathbf{w}}_R^H\acute{{\mathbf{y}}_R}\left(n\right){|}^2]\le P_R.$$

(3)

Received signals at the two UTs are then given as

$$y_i\left(n\right)={\alpha }_i{\mathbf{g}}_i^T{\mathbf{x}}_R\left(n\right)+n_i\left(n\right)\mathrm{for}i=1,2,$$

(4)

where ${\alpha }_i$ reflects the path loss toward the i-th UT and the additive noise $n_i\left(n\right)$ is $\mathcal{CN}(0,1)$-distributed. Among the two superposed message signals in the NOMA transmission, the two UTs attempt to decode the message signals intended for the near UT ($U_1$). At this time, the decoding of the near UT stops, whereas the far UT continues to decode the message intended for itself after subtracting the signal component for $U_1$. This decoding method at $U_2$ is known as SIC. Denoting the SI terms at the i-th UT as $I_i={\alpha }_i{\mathbf{g}}_i^T\mathbf{W}[{\sum }_{j=1}^{\infty }{\left(\acute{{\alpha }_R}{\mathbf{G}}_{rr}\mathbf{W}\right)}^j\{{\mathbf{H}}_r\mathbf{w}{x}_S(n-(j+1)\tau )+{\mathbf{n}}_R(n-(j+1)\tau )\}+{\mathbf{n}}_R(n-\tau )]$, the end-to-end SINR for the decoding of the first message signal $x_1\left(n\right)$ at the i-th UT denoted as ${\gamma }_i^1$ is given by [36]

$${\gamma }_i^1={\displaystyle \frac{\rho {\alpha }_i^2|{\mathbf{g}}_i^T{\mathbf{w}}_T{|}^2{\left|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}\right|}^2}{\mathsf{E}\left[\right|I_i{|}^2]+{\alpha }_i^2|{\mathbf{g}}_i^T{\mathbf{w}}_T{|}^2\left[(1-\rho )\right|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2+1]+1}}.$$

(5)

The convergent form of the statistical average of the SI power $\mathsf{E}\left[\right|I_i{|}^2]$ and ${\mathsf{E}}_{{\mathbf{n}}_R}\left[\right|{\mathbf{w}}_R^H\acute{{\mathbf{y}}_R}\left(n\right){|}^2]$ can be given as in (10), defined for the convergence interval $0\le {\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^2<1$, where the average diverges outside of this interval [36]. When the decoding of $x_1\left(n\right)$ succeeds, the SINR for decoding the second message signal $x_2\left(n\right)$ at the distant user, $U_2$, denoted by ${\gamma }_2^2$, is given by

$${\gamma }_2^2={\displaystyle \frac{(1-\rho ){\alpha }_2^2|{\mathbf{g}}_2^T{\mathbf{w}}_T{|}^2{\left|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}\right|}^2}{\mathsf{E}\left[\right|I_2{|}^2]+{\alpha }_2^2{\left|{\mathbf{g}}_2^T{\mathbf{w}}_T\right|}^2+1}}.$$

(6)

3. Beamformer Design

In this work, our objective is to design the beamformer set $\mathbf{w}$ and $\mathbf{W}=\sqrt{\rho }{\mathbf{w}}_T{\mathbf{w}}_R^H$ such that the total throughput is maximized. In NOMA transmission, the sum throughput $R_{sum}$ can be expressed as

$$R_{sum}=min[log(1+{\gamma }_1^1),log(1+{\gamma }_2^1)]+log(1+{\gamma }_2^2),$$

(7)

where the minimum operation relies on the fact that the message for the first UT $x_1\left(n\right)$ should be decoded at both UTs, whereas message $x_2\left(n\right)$ is destined to be decoded at the distant UT.

3.1. Zero Forcing Beamforming

We can easily observe that the SI term is most likely against all SINR expressions and that zeroing it is reasonable for improving the SINR performance. This ZF condition (ZFC) of the SI term can be expressed as ${\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T=0$. This ZFC can be imposed on the optimization problem involved in the beamformer design to simplify the design. If an appropriate selection of beamformer set satisfies the ZFC, the SINR expressions in (5) and (6) become

$${\gamma }_{i,ZF}^1={\displaystyle \frac{\rho {\alpha }_i^2|{\mathbf{g}}_i^T{\mathbf{w}}_T{|}^2{\left|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}\right|}^2}{{\alpha }_i^2|{\mathbf{g}}_i^T{\mathbf{w}}_T{|}^2\left[(1-\rho )\right|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2+1]+1}}$$

(8)

and

$${\gamma }_{2,ZF}^2={\displaystyle \frac{(1-\rho ){\alpha }_2^2|{\mathbf{g}}_2^T{\mathbf{w}}_T{|}^2{\left|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}\right|}^2}{{\alpha }_2^2{\left|{\mathbf{g}}_2^T{\mathbf{w}}_T\right|}^2+1}},$$

(9)

respectively. With the ZFC, the relay receive signal power becomes ${\mathsf{E}}_{{\mathbf{n}}_R}\left[\right|{\mathbf{w}}_R^H\acute{{\mathbf{y}}_R}{\left(n\right)|}^2]=[1+|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2]$, which results in the FDR transmit power constraint of (3) as $\parallel {\mathbf{w}}_T{\parallel }^2[1+|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2]\le P_R$.

$$\begin{array}{cc}\hfill {\mathsf{E}}_{{\mathbf{n}}_R}\left[\right|I_i{|}^2]& ={\displaystyle \frac{{\alpha }_i^2{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^4}{1-{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^2}}{\displaystyle \frac{|{\mathbf{g}}_i^T{\mathbf{w}}_T{|}^2\left[\right|{\mathbf{w}}_T^H{\mathbf{G}}_{rr}^H{\mathbf{H}}_r{\mathbf{w}|}^2+\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^2]}{\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^4}},\hfill \\ \hfill {\mathsf{E}}_{{\mathbf{n}}_R}\left[\right|{\mathbf{w}}_R^H\acute{{\mathbf{y}}_R}\left(n\right){|}^2]& =\left[{\displaystyle \frac{{\acute{{\alpha }_R}}^4{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^4}{1-{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^2}}\left\{{\displaystyle \frac{|{\mathbf{w}}_T^H{\mathbf{G}}_{rr}^H{\mathbf{H}}_r{\mathbf{w}|}^2+{\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T\parallel }^2}{\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^4}}\right\}+1+{\left|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}\right|}^2\right].\hfill \end{array}$$

(10)

Here, we easily observe that the optimization of the FDR receive beamformer ${\mathbf{w}}_R$ and the AP beamformer $\mathbf{w}$ is equivalent to maximizing $|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}|$, which is achieved by aligning ${\mathbf{w}}_R=\mathbf{u}$ and $\mathbf{w}=\sqrt{P_a}\mathbf{v}$ when the vectors $\mathbf{u}$ and $\mathbf{v}$ are the left and right singular vectors of ${\mathbf{H}}_r$, respectively, associated with the largest singular value. Additionally, ${\mathbf{w}}_T$ maximizes ${\gamma }_2^1$ and ${\gamma }_2^2$ simultaneously, which leads us to conclude that finding ${\mathbf{w}}_T$ that maximizes $R_{sum}$ in this case is equivalent to finding the one that maximizes $min[log(1+{\gamma }_1^1),log(1+{\gamma }_2^1)]$. This observation again indicates that the optimization of $R_{sum}$ under the ZFC of the SI results in a new condition, where ${\gamma }_1^1={\gamma }_2^1$ is achieved at the optimal point. The above discussions are summarized in Algorithm 1 to determine a beamforming scheme in which an orthogonal projection operation is adopted to satisfy the ZFC. The while loop implements a one-dimensional search over a scalar parameter $\alpha$ to locate the point where the condition ${\gamma }_1^1={\gamma }_2^1$ is satisfied, whereas the FDR transmit beamformer vector ${\mathbf{w}}_T$ spans between two vectors toward UTs. Scaling of ${\mathbf{w}}_T$ is performed to satisfy the FDR transmit power constraint. Thus, the sum throughput is maximized, and all the conditions in this subsection are satisfied.

Algorithm 1 Zero forcing beamforming scheme

1: Set $\mathbf{w}=\sqrt{P_a}\mathbf{v}$ and ${\mathbf{w}}_R=\mathbf{u}$, where the vectors $\mathbf{v}$ and $\mathbf{u}$ are the right and left singular vectors of ${\mathbf{H}}_r$, respectively, associated with the largest singular value. 2: Set ${\tilde{\mathbf{g}}}_1={\mathsf{P}}_{{\left({\mathbf{G}}_{rr}^H{\mathbf{w}}_R\right)}^{\ast }}\left({\mathbf{g}}_1\right)$ and ${\tilde{\mathbf{g}}}_2={\mathsf{P}}_{{\left({\mathbf{G}}_{rr}^H{\mathbf{w}}_R\right)}^{\ast }}\left({\mathbf{g}}_2\right)$. 3: Set ${\alpha }_{min}=0$ and ${\alpha }_{max}=1$. 4: while ($(\Delta \ge ϵ)$ ($ϵ$: small threshold.) ) do 5:    Pick test values of $\alpha$ inside the range $[{\alpha }_{min},{\alpha }_{max}]$. 6:    for each of the selected $\alpha$ values do 7:        Set ${\mathbf{w}}_T=[\alpha {\tilde{\mathbf{g}}}_1+(1-\alpha ){\tilde{\mathbf{g}}}_2]$. 8:        Scale ${\mathbf{w}}_T$ such that the condition $\parallel {\mathbf{w}}_T{\parallel }^2={\displaystyle \frac{P_R}{[1+|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2]}}$ is satisfied. 9:        Calculate ${\gamma }_{1,ZF}^1$ and ${\gamma }_{2,ZF}^1$ using (8) and (9). 10:      Set $\Delta =|{\gamma }_{1,ZF}^1-{\gamma }_{2,ZF}^1|$. 11:    end for 12:    Based on the $\Delta$ values tested, adjust the ${\alpha }_{min}$ and ${\alpha }_{max}$ according to an one-dimensional search. 13: end while

3.2. Regularized Beamforming

The ZF beamforming method described in the previous subsection is known to achieve a performance close to the optimal one when the SINR is extremely high. However, for the SINR range of practical interest, the ZF scheme is also known to leave considerable room for improvement by allowing some amount of SI to survive FDR beamforming. In this subsection, we address the optimization of the remaining SI levels after the FDR transmission beamforming. Unlike the ZF beamforming described in the previous subsection, the SI may render the FDR unstable when its power falls in the diverging regime; thus, maintaining the system within the SI convergence region is crucial. Note that the decision method of the previous subsection for $\mathbf{w}$ and ${\mathbf{w}}_R$ is very useful for simplifying the problem; thus, it is adopted again in this subsection for the same reason. Therefore, we have ${\mathbf{w}}_R=\mathbf{u}$ and $\mathbf{w}=\sqrt{P_a}\mathbf{v}$, whereas the vectors $\mathbf{u}$ and $\mathbf{v}$ are defined in the previous subsection. The removal of ZFC enables us to consider the signal models in (5) and (6) to determine the remaining beamforming vector ${\mathbf{w}}_T$. The same line of reasoning from the previous subsection enables us to consider the following problem:

$$\begin{array}{cc}\hfill \underset{{\mathbf{w}}_T}{max}min& \left[log(1+{\gamma }_1^1),log(1+{\gamma }_2^1)\right]\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \parallel {\mathbf{w}}_T{\parallel }^2\le {\displaystyle \frac{P_R}{{\mathsf{E}}_{{\mathbf{n}}_R}\left[\right|{\mathbf{w}}_R^H\acute{{\mathbf{y}}_R}\left(n\right){|}^2]}}.\hfill \end{array}$$

(11)

The development of the problem in the sequel becomes convenient by introducing the definition of the SI power at the FDR as $y\left({\mathbf{w}}_T\right)={\displaystyle \frac{{\mathsf{E}}_{{\mathbf{n}}_R}\left[\right|I_i{|}^2]}{\left({\alpha }_i^2\right|{\mathbf{g}}_i^T{\mathbf{w}}_T{|}^2)}}={\displaystyle \frac{{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^4}{1-{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^2}}·{\displaystyle \frac{\left[\right|{\mathbf{w}}_T^H{\mathbf{G}}_{rr}^H{\mathbf{H}}_r\mathbf{w}{|}^2+\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^2]}{\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^4}}\mathrm{for}\mathrm{both}i=1$ and 2.

Thus, (11) can be rewritten as (12).

$$\begin{array}{cc}\hfill \underset{{\mathbf{w}}_T}{max}min& \left[{\displaystyle \frac{\rho {\alpha }_1^2|{\mathbf{g}}_1^T{\mathbf{w}}_T{|}^2{\left|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}\right|}^2}{{\alpha }_1^2|{\mathbf{g}}_1^T{\mathbf{w}}_T{|}^2[y+(1-\rho )|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2+1]+1}},{\displaystyle \frac{\rho {\alpha }_2^2|{\mathbf{g}}_2^T{\mathbf{w}}_T{|}^2{\left|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}\right|}^2}{{\alpha }_2^2|{\mathbf{g}}_2^T{\mathbf{w}}_T{|}^2[y+(1-\rho )|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2+1]+1}}\right]\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& 0\le {\acute{{\alpha }_R}}^2|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T{|}^2<1,{\parallel {\mathbf{w}}_T\parallel }^2\le {\displaystyle \frac{P_R}{[y+|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2+1]}}.\hfill \end{array}$$

(12)

The two SINR terms inside the minimum operator in the objective function of (12) are inversely proportional to the SI power y and approach zero as y approaches infinity. Similarly, the effect of y on the SINR cases can be observed for the magnitude of ${\mathbf{w}}_T$ such that the SI power y inversely impacts the SINR directly and indirectly through the power of ${\mathbf{w}}_T$. These observations may misleadingly suggest that regularization offers no potential for further SINR improvement. This misconception can be circumvented if we form the FDR transmit beamformer ${\mathbf{w}}_T$ such that the magnitudes $|{\mathbf{g}}_i^T{\mathbf{w}}_T{|}^2$ for $i=1,2$ increase, whereas it generates more SI power y because the SINRs are proportional to the magnitudes. In Algorithm 1, the FDR transmit beamformer is formed as ${\mathbf{w}}_T=[\alpha {\tilde{\mathbf{g}}}_1+(1-\alpha ){\tilde{\mathbf{g}}}_2]$, $0\le \alpha \le 1$ with a power scaling afterward to sweep the space orthogonal to the vector ${\mathbf{G}}_{rr}^H{\mathbf{w}}_R$ and generated by two channel vectors ${\mathbf{g}}_1$ and ${\mathbf{g}}_2$. This formation of the ${\mathbf{w}}_T$ vector through orthogonal projection guarantees that the ZFC is satisfied.

In contrast, the generation of SI power can be achieved by allowing the vector component ${\mathbf{G}}_{rr}^H{\mathbf{w}}_R$ to the FDR transmit beamformer. Let us set ${\mathbf{g}}_0={\mathbf{G}}_{rr}^H{\mathbf{w}}_R$ and form the FDR transmit beamformer as ${\mathbf{w}}_T\left(\theta \right)=[\theta {\mathbf{g}}_0+(1-\theta )\alpha {\tilde{\mathbf{g}}}_1+(1-\theta )(1-\alpha ){\tilde{\mathbf{g}}}_2]$, $0\le \theta ,\alpha \le 1$. This formation of ${\mathbf{w}}_T$ causes the monotonic dependence of y on the magnitude of ${\mathbf{w}}_T$ to be transferred to the monotonic dependence of y on $\theta$ as inferred from the discussion in the next section. The relation between y and $\parallel {\mathbf{w}}_T\parallel$ is dictated by the two nonlinear equations of the definition $y={\displaystyle \frac{{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^4}{1-{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^2}}·{\displaystyle \frac{\left[\right|{\mathbf{w}}_T^H{\mathbf{G}}_{rr}^H{\mathbf{H}}_r\mathbf{w}{|}^2+\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^2]}{\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^4}}$ and the FDR power constraint $\parallel {\mathbf{w}}_T{\parallel }^2={\displaystyle \frac{P_R}{[y+|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2+1]}}$. The operating point of y and $\parallel {\mathbf{w}}_T\parallel$ in an FD-AF relay is found at the point where the curves generated by the two equations above meet, as shown in Figure 2, where the instances of the two curves on the y and $\left|\right|{\mathbf{w}}_T\parallel$ planes are plotted. The curves cross each other in an “X”-shaped manner. Depending on the strength of the FDR power $P_R$, the FDR power constraint meets the vertical $\parallel {\mathbf{w}}_T\parallel$ axis below or above the dotted line (which divides the convergence and divergence regions of the interference power). When the FDR power $P_R$ is excessively high, the SI overwhelms the signal from the AP (the dashed blue line in Figure 2); the ZFBF in Section 3.2 is the optimal approach. Otherwise (solid blue line case), the optimal approach is to determine the above-mentioned operating point of the FD-AF relay and form the regularized beamformer. The search for this point in the first quarter begins with the point at which the FDR power constraint meets the $\parallel {\mathbf{w}}_T\parallel$ axis with $y=0$ (ZF beamforming). Subsequently, we alternate between the two curves (solving the two equations) until a certain convergence condition is satisfied. When the first y value from the y-definition curve is positive, we follow the above alternation along the green dash-dot arrows in the figure. Otherwise, it indicates that the first $\parallel {\mathbf{w}}_T\parallel$ value drives the FD-AF relay into the diverging regime; thus, we opt for the ZFBF strategy in Section 3.2. We can easily see that the two terms $|{\mathbf{g}}_i^T{\mathbf{w}}_T|$ for $i=1,2$ are equated by the outer one-dimensional optimization.

Considering the two SINR terms inside the minimum operator in (12), the objective function is a concave function of $\theta$ because the vector component ${\mathbf{g}}_0$ is orthogonal to the two vectors ${\tilde{\mathbf{g}}}_1$ and ${\tilde{\mathbf{g}}}_2$, and adding this component to ${\mathbf{w}}_T$ through $\theta$ increases $|{\mathbf{g}}_i^T{\mathbf{w}}_T|$ until some points for both cases of $i=1,2$. In addition, increasing y through $\theta$ causes the two SINRs to monotonically decay such that we can conclude that the two SINRs are concave functions of $\theta$. Again, the minimum operator in (12) forces us to select a point with the condition $|{\mathbf{g}}_1^T{\mathbf{w}}_T|=|{\mathbf{g}}_2^T{\mathbf{w}}_T|$. The discussion in this subsection is summarized in Algorithm 2, which determines the regularized beamformer set. Here, two one-dimensional optimizations are conducted independently in a layered manner with two nested while loops such that one optimization is embedded inside the other. The inside one is adopted to resolve the min operation in (12) by adjusting the parameter $\alpha$, whereas the outer one searches the SI level y to locate the point by maximizing (5) and controlling the $\theta$ parameter. However, if the first determined y value inside the while loop is negative, we set $\theta =0$ and escape the two nested loops to determine the ZFBF after the Label 1:

Algorithm 2 Regularized beamforming scheme

1: Set $\mathbf{w}=\sqrt{P_a}\mathbf{v}$ and ${\mathbf{w}}_R=\mathbf{u}$, where the vectors $\mathbf{v}$ and $\mathbf{u}$ are the right and left singular vectors of ${\mathbf{H}}_r$, respectively, associated with the largest singular value. 2: Set ${\tilde{\mathbf{g}}}_1={\mathsf{P}}_{{\left({\mathbf{G}}_{rr}^H{\mathbf{w}}_R\right)}^{\ast }}\left({\mathbf{g}}_1\right)$ and ${\tilde{\mathbf{g}}}_2={\mathsf{P}}_{{\left({\mathbf{G}}_{rr}^H{\mathbf{w}}_R\right)}^{\ast }}\left({\mathbf{g}}_2\right)$. 3: Set ${\theta }_{btm}=0$ and ${\theta }_{top}=1$. Here, ${\sigma }_r$ is the smallest singular value of ${\mathbf{G}}_{rr}$. 4: while (${\Gamma }_{\theta }$ does not improve more than ${ϵ}_{\theta }$ (${ϵ}_{\theta }$: small threshold.) ) do 5:    Pick test values in the range $[{\theta }_{btm},{\theta }_{top}]$. 6:    for each chosen value of $\theta$ do 7:        Set ${\alpha }_{min}=0$ and ${\alpha }_{max}=1$. 8:        while ($(\Delta \ge ϵ)$ ($ϵ$: small threshold.) ) do 9:           Pick test values of $\alpha$ in the range $[{\alpha }_{min},{\alpha }_{max}]$. 10:         for each of the selected $\alpha$ values do 11:              Set ${\mathbf{w}}_T\left(\theta \right)=[\theta {\mathbf{g}}_0+(1-\theta )\alpha {\tilde{\mathbf{g}}}_1+(1-\theta )(1-\alpha ){\tilde{\mathbf{g}}}_2]$. 12:              Set $y=0$ and find $\parallel {\mathbf{w}}_T\parallel$ with the FDR power constraint (blue lines in Figure 2. Apply this $\parallel {\mathbf{w}}_T\parallel$ to the y-definition curve. 13:              IF (the found y is negative), 14:                   Set $\theta =0$ and goto $Lable$ 1. 15:              ENDIF 16:             Find the point, where the curves of two equations FDR power constraint ($\parallel {\mathbf{w}}_T{\parallel }^2={\displaystyle \frac{P_R}{[y+|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2+1]}}$) and the y-definition ($y={\displaystyle \frac{{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^4}{1-{\acute{{\alpha }_R}}^2{\left|{\mathbf{w}}_R^H{\mathbf{G}}_{rr}{\mathbf{w}}_T\right|}^2}}{\displaystyle \frac{\left[\right|{\mathbf{w}}_T^H{\mathbf{G}}_{rr}^H{\mathbf{H}}_r\mathbf{w}{|}^2+\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^2]}{\parallel {\mathbf{G}}_{rr}{\mathbf{w}}_T{\parallel }^4}}$) meet by following the green route in Figure 2. 17:              Scale ${\mathbf{w}}_T$ according to the found $\parallel {\mathbf{w}}_T\parallel$. 18:              Calculate ${\gamma }_1^1$ and ${\gamma }_2^1$ using (5). 19:              Set $\Delta =|{\gamma }_1^1-{\gamma }_2^1|$. 20:           end for 21:           Based on the $\Delta$ values tested, adjust the ${\alpha }_{min}$ and ${\alpha }_{max}$ according to the inner one-dimensional search. 22:        end while 23:        Calculate ${\Gamma }_{\theta }=min[{\gamma }_1^1,{\gamma }_2^1]$ using (5). 24:    end for 25:    Based on the ${\Gamma }_{\theta }$ values tested, adjust the ${\theta }_{btm}$ and ${\theta }_{top}$ according to the outer one-dimensional search. 26: end while 27: Goto $Label$ 2: 28: $Label$ 1: 29: Set ${\alpha }_{min}=0$ and ${\alpha }_{max}=1$. 30: while ($(\Delta \ge ϵ)$ ($ϵ$: small threshold.) ) do 31:    Pick test values of $\alpha$ inside the range $[{\alpha }_{min},{\alpha }_{max}]$. 32:    for each of the selected $\alpha$ values do 33:        Set ${\mathbf{w}}_T=[\alpha {\tilde{\mathbf{g}}}_1+(1-\alpha ){\tilde{\mathbf{g}}}_2]$. 34:        Scale ${\mathbf{w}}_T$ such that the condition $\parallel {\mathbf{w}}_T{\parallel }^2={\displaystyle \frac{P_R}{[1+|{\mathbf{w}}_R^H{\mathbf{H}}_r\mathbf{w}{|}^2]}}$ is met. 35:        Calculate ${\gamma }_{1,ZF}^1$ and ${\gamma }_{2,ZF}^1$ using (8) and (9). 36:        Set $\Delta =|{\gamma }_{1,ZF}^1-{\gamma }_{2,ZF}^1|$. 37:    end for 38:    Based on the $\Delta$ values tested, adjust the ${\alpha }_{min}$ and ${\alpha }_{max}$ according to an one-dimensional search. 39: end while 40: $Label$ 2:

The nested optimization structure of Algorithm 2 and the iteration of the FDR operating point inside the nest can result in complexity and convergence issues. Note that there are two nested one-dimensional searches over the concave functions and iterations of the FDR operating point within the nest. Note also that all the iterations involved converge very easily and do not incur significant computations. The impact of the number of antennas on the computation is not strong either. Computational burden may result from the orthogonal projection operation in the second step of Algorithms 1 and 2 when the number of antennas increases because it depends on the order of $O\left(n^3\right)$. Note also that we may expand the problem scope of this paper such that multiple pairs of NOMA users are served by the FDR. In such cases, the scheduling of NOMA user pairs becomes a crucial technical challenge that reduces the benefit of multiuser diversity in performance improvement.

4. Numerical Results

In this section, we analyze and discuss the performance of the proposed beamforming schemes. The average sum rates of the two proposed beamforming algorithms for CNOMA systems with FD-AF relaying are plotted in Figure 3 against the relay transmission power ($P_R$). As a reference scheme, the HDR-based CNOMA system is considered, where the SI is not important at the relay; hence, the HDR beamforming vectors can be determined using Algorithm 1 without the orthogonal projection operation in line two. However, two time slots are required in the HDR case to accommodate the AP and HDR transmissions, which incurs a halved spectral efficiency compared with the FD cases in this study. Note that the value of $P_R$ considered in this paper is too high to emulate the realistic scenario of an FDR, and $\alpha$ values are introduced to compensate for these high $P_R$ values. Two values of the AP power ($P_a$) are shown in Figure 3 to show that the proposed schemes achieve nearly twice the sum rates compared with those of the HDR case. This implies that the SI at the relay is suppressed well by the proposed algorithms, and the benefit of FD is achieved. However, the advantage of regularizing Algorithm 2 over ZF beamforming in Algorithm 1 is marginal in the two considered cases.

The advantage of regularization over the ZF scheme is more clearly demonstrated when we plot the average sum rates against the AP power ($P_a$), as shown in Figure 4, or against the number of relay antenna sets, as shown in Figure 5. The relay power $P_R$ is fixed, as shown in Figure 4, and both the AP power and relay power are fixed, as shown in Figure 5. The advantage of FD-AF beamforming over HDR beamforming is consistent with the numerical results shown in these two figures. As expected, the response to an increase in $P_a$ became more apparent when $P_R=50$ (dB) than when $P_R=40$ (dB). In the latter case, the relay power $P_R$ combined with the path loss parameters ${\alpha }_i,i=1,2$ is insufficient for observing the impact of an increase in $P_a$. The effect of adding antenna resources to the relay is shown in Figure 5. It is shown that the regularization-based beamforming of Algorithm 2 is the best at utilizing the increase in antenna resources. The ZF-based Algorithm 1 follows in second place, with the HDR CNOMA in last place.

5. Conclusions

We propose linear beamforming schemes for CNOMA with FD-AF relay systems and analyze their performance through a set of computer simulations. The two proposed optimization algorithms for the beamforming schemes show that the SI of the FD-AF relay is well suppressed and the NOMA transmission is optimized, which can be verified by comparison with the HDR-based scheme in numerical simulations. Many simulation scenarios have shown that the regularization of the SI by the second algorithm provides further improvement in the performance of the ZF-based algorithm. However, the regularization of the SI depends on controlling the operating point of the FD–AF relay so that the system stays in the regime where the SI does not diverge. In cases where such control of the SI is impossible, the only option is to opt for ZF-based beamforming.