Sensing-Efficient Transmit Beamforming for ISAC with MIMO Radar and MU-MIMO Communication

Abstract

We focus on an integrated sensing and communication (ISAC) system—a single platform equipped with multiple antennas transmitting a waveform to detect targets and communicate with downlink users. Due to spectrum sharing between multiple-input–multiple-output (MIMO) radar and multiuser MIMO (MU-MIMO) communication, beamforming is becoming increasingly important as a technique that enables the creation of directional beams. In this paper, we propose a novel joint transmit beamforming design scheme that employs a beam pattern approximation strategy for radar sensing and utilizes rate-splitting for multiuser communication offering advanced interference management strategies. The optimization problems are formulated from both radar-centric and trade-off viewpoints. First, we propose a radar-centric beamforming scheme to achieve sensing efficiency through beam pattern approximation, while requiring the fairness signal-to-interference-plus-noise ratio (SINR) to be higher than a given threshold to guarantee a minimal level of communication quality, while the obtained performance for the communication system is limited in this scheme. To address this problem, we propose a beamforming design scheme from a trade-off viewpoint that flexibly optimizes both sensing and communication performances with a regularization parameter. Finally, we propose a partial rate-splitting-based beamforming design method aimed at maximizing the effective sensing power, with the constraint of a minimal sum rate for downlink users. Numerical results are provided to assess the effectiveness of all proposed schemes.

1. Introduction

Radar is now facing a severe problem with spectrum congestion, and numerous academics have suggested sharing the wireless communication spectrum with radar [1,2,3,4]. There is a highly impending need for spectrum sharing across radar and communication due to the significant rise in the number of connected devices [5,6,7,8,9,10]. Therefore, both business and academics have given integrated sensing and communication (ISAC) a lot of attention. As a single platform with two functionalities, ISAC enables sensing and communication to operate simultaneously over identical wide frequency bands [11,12,13,14,15], realizing integration and miniaturization while also meeting the demands of contemporary societal development.

Due to these significant advantages, various strategies are proposed to achieve the ISAC, involving waveform design [16,17,18,19,20,21], joint transmit beamforming design [22,23,24,25,26,27,28], etc. We focus on the joint transmit beamforming design in this paper, where spatial multiplexing of radar and communications is achieved by shipping multiple transmit beams toward targets and communication users. The authors in [22] studied the beamforming design problems for both separated and shared antenna deployments in a joint RadCom system. A joint beamforming design scheme was studied in [24], focusing on optimizing sensing performance while satisfying the requirements of communication to obtain the overall beamforming matrix. Recent studies have considered ISAC based on various multiple access techniques [25,29] to further enhance spectrum efficiency for both sensing and communication.

The authors in [25] investigated transmit beamforming for the ISAC based on nonorthogonal multiple access (NOMA) to optimize the weighted sum rate (WSR) of communication users and the radar beam patterns. They suggested a double-layer penalty-based approach to address their optimization problem. Different from the multiaccess method mentioned earlier, authors in [29] proposed a method for information embedding based on the concept of space-division multiple access (SDMA) and waveform diversity. Compared with using nonorthogonal or space-division methods, the interference in ISAC can be effectively managed by employing more powerful multiple access techniques. For rate-splitting multiple access (RSMA), part of the interference is decoded and the remainder is treated as noise, different from the two extreme interference management strategies, SDMA and NOMA. This is because of the fact that NOMA fully decodes interference, whereas SDMA entirely treats interference as noise [30,31,32,33,34,35,36,37,38]. The system’s WSR and Energy Efficiency (EE) performance are on par with or better than that of SDMA, NOMA, and multiuser linear precoding (MU-LP) [39,40,41,42,43]. Research on rate-splitting (RS)-based ISAC systems is meaningful due to its ability to dynamically manage interference [44,45].

In this paper, a joint transmit beamforming for the ISAC system is investigated from radar-centric and trade-off perspectives to ensure sensing performance through beam pattern approximation. Similar to [24], a weighted sum of communication symbols and radar waveforms is used as our transmit signal. Unlike [24], our research motivation is not solely to maximize sensing performance. To better ensure communication performance, we have also proposed a weighted optimization algorithm that allows for a flexible trade-off between sensing and communication performance by adjusting the relevant parameters. Due to the extreme handling of interference by NOMA and SDMA, their application in ISAC (i.e., NOMA-based ISAC in [25] and SDMA-ISAC in [29]) is less advantageous compared with RSMA. We use RS as a multiple access platform for multiuser communication, providing advanced interference management strategies. Unlike [45], we consider a dual-function waveform, which is the weighted sum of communication symbols and radar waveforms, transmitted by a shared uniform linear array to achieve both communication and sensing functions simultaneously. From a practical perspective, we also consider the interference of radar signals on communication. Although the common stream generated by splitting can increase the system’s degree of freedom (DoF), it still cannot sufficiently compensate for the impact of interference on private message decoding and sensing performance, as the transmit power of the common stream is limited by the user with the worst ability to decode the common part. The authors in [46] proposed a method to maximize the sum rate of downlink users by controlling the transmit power of a common stream. Building on this, we further propose a beamforming design scheme for ISAC based on partial rate-splitting (P-RS) to address the issue of limited common energy, aiming at enhancing the effective sensing power.

The transmitter in this study maximizes the DoF of multiple-input–multiple-output (MIMO) radar waveforms by using individual communication and radar waveforms. For clarity, the key contributions made by this work are summarized as follows:

• We propose a joint beamforming design scheme from a radar-centric viewpoint using beam pattern approximation and achieve optimal beamforming by solving a nonconvex optimization problem.
• We further propose a beamforming design scheme from a trade-off viewpoint to ensure communication performance and achieve flexible optimization of sensing and communication performances with a regularization parameter.
• We propose a P-RS-based beamforming design scheme aimed at maximizing the effective sensing power of MIMO radar while satisfying the minimum requirement of multiuser communication. Numerical results are provided to assess the effectiveness of all proposed schemes.

The remainder of this work is organized as follows. Section 2 describes the system model and illustrates the performance metrics of sensing and communication. Section 3 proposes beamforming design schemes from radar-centric and trade-off viewpoints and proposes a P-RS-based ISAC scheme. Section 4 provides simulation results, Section 5 points out our future work, and Section 6 concludes the paper.

Notations: Vectors and matrices are denoted by bold-face, lower-case, and uppercase letters, respectively (i.e., $\mathbf{R}$ and $\mathbf{r}$ represent matrix and vector, respectively). ${\left(·\right)}^H$, ${\left(·\right)}^T$, and tr$\left(·\right)$ refer to Hermitian transpose, transpose, and the trace of the argument, respectively. The diagonal entry vector of a matrix is denoted by diag$\left(·\right)$. We use $\mathbf{E}\left(·\right)$ for the stochastic expectation and subscripts for the columns of a matrix. The operation $\mathbf{R}\underline{\succ }$ 0 denotes that $\mathbf{R}$ is a positive semidefinite matrix. To facilitate understanding of the abbreviations used in this paper, we have compiled them into a reference table and placed it at the end of the article for convenience (see Abbreviations section).

2. Signal Model and Performance Metrics

The transmitter simultaneously transmits radar signals and communication symbols for detecting targets and communicating with downlink users. Both the sensing and communication are operated by joint beamforming. The transmitter of the ISAC system has an antenna array with N elements that can simultaneously detect targets and serve K single-antenna users. Next, we present the signal model and introduce the performance metrics for communication and radar.

2.1. Signal Model

The transmit signal includes two parts: radar waveforms ${\mathbf{s}}_r$ used for detecting targets and communication symbols ${\mathbf{s}}_c$ for delivering messages. It should be noted that the targets and users are distinct entities. Here, we assume that the communication messages are K unicast messages $W_1$, …, $W_K$ for K downlink users, respectively. The proposed joint beamforming transmitter in this paper is shown in Figure 1. Every unicast message is divided into two parts, i.e., $W_k$ is divided into common part $W_{0,k}$, which is shared by all users, and private part $W_{p,k}$, intended for user-k. The message $W_0$ is generated by combining all of these common components $W_{0,1}$, …, $W_{0,K}$, and then encoding it into a common stream $s_0$, while these private parts $W_{p,1}$, …, $W_{p,K}$ of the unicast messages are encoded into the private streams $s_0$, …, $s_K$ independently. The data stream vector ${\mathbf{s}}_c={\left[s_0,s_1,\dots ,s_K\right]}^T\in {\mathbb{C}}^{\left(K+1\right)\times 1}$ is linearly precoded using the precoder ${\mathbf{P}}_c=\left[{\mathbf{p}}_0,{\mathbf{p}}_1,\dots ,{\mathbf{p}}_K\right]\in {\mathbb{C}}^{N\times \left(K+1\right)}$, where the vector ${\mathbf{p}}_0\in {\mathbb{C}}^{N\times 1}$ is the precoder of common stream, ${\mathbf{p}}_k\in {\mathbb{C}}^{N\times 1}$ (for $k=1,\dots ,K$) is the precoder of private stream. Radar waveforms ${\mathbf{s}}_r={\left[r_1,\dots ,r_N\right]}^T\in {\mathbb{C}}^{N\times 1}$ is a vector including N individual radar waveforms; the radar waveforms are precoded using the beamforming matrix ${\mathbf{P}}_r\in {\mathbb{C}}^{N\times N}$ and transmitted together with the precoded communication streams. The signal transmitted by the dual-function platform is the superposition of the radar and communication signals, and we assume that there is no correlation between radar waveforms and communication symbols, i.e., $\mathbf{E}\left({\mathbf{s}}_c{{\mathbf{s}}_r}^H\right)=\mathbf{E}\left({\mathbf{s}}_c\right)\mathbf{E}\left({{\mathbf{s}}_r}^H\right)={\mathbf{0}}_{(K+1)\times N}$. The transmission signal $\mathbf{X}$ can be described as follows:

$$\mathbf{X}={\mathbf{P}}_c{\mathbf{s}}_c+{\mathbf{P}}_r{\mathbf{s}}_r.$$

(1)

In Figure 1, the received common message $s_0$ is first decoded using the equalizer $g_{0,k}$, while treating the private streams of all users and radar waveforms as interference. Then, the decoded common message is split to obtain the corresponding common parts, as it contains the common parts derived from each user’s unicast message. Successive interference cancellation (SIC) is adopted to remove the common message from the received signal at each receiver. Consequently, the corresponding private message can be decoded using the equalizer $g_k$. In this paper, we assume that both communication signals are zero-mean, i.e., $\mathbf{E}\left({\mathbf{s}}_c\right)={\mathbf{0}}_{(k+1)\times 1}$, where ${\mathbf{0}}_{(k+1)\times 1}$ represents a $(k+1)\times 1$ zero vector, indicating that every signal carries no energy.

2.2. Metric for Communication Performance

The received communication signal at the downlink users can be written as $\mathbf{Y}=\mathbf{HX}+\mathbf{n}$. Combine with Equation (1), the signal received at user-k can be written as follows:

$$y_k={\mathbf{h}}_k{\mathbf{P}}_r{\mathbf{s}}_r+{\mathbf{h}}_k{\mathbf{p}}_0{s}_0+{\mathbf{h}}_k{\sum }_{k=1}^K{\mathbf{p}}_k{s}_k+n_k,$$

(2)

where $\mathbf{H}={\left[{\mathbf{h}}_1,\dots ,{\mathbf{h}}_K\right]}^T\in {\mathbb{C}}^{K\times N}$ is the channel matrix from the transmit antennas to communication users, ${\mathbf{h}}_k^H\in {\mathbb{C}}^{1\times N}$ represents the channel from the transmit platform to user-k. $\mathbf{n}=[n_1,\dots ,n_K]\in {\mathbb{C}}^{1\times K}$ with $n_k\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_k^2\right)$ represent the noise received at user-k with zero mean and variance ${\sigma }_k^2$, and it obeys complex Gaussian random distribution. Without loss of generality, the noise variances of all users are assumed to be 1.

In this paper, we consider the interference from radar waveforms and among users when decoding the communication messages. Each user decodes $s_0$ first, and he fairness signal-to-interference-plus-noise ratio (SINR) at user-k for decoding the common stream is

$${\gamma }_{0,k}={\displaystyle \frac{{\left|{\mathbf{h}}_k^H{\mathbf{p}}_0\right|}^2}{{\left|{\mathbf{h}}_k^H{\mathbf{P}}_r\right|}^2+{\sum }_{i=1}^K{\left|{\mathbf{h}}_k^H{\mathbf{p}}_i\right|}^2+1}}.$$

(3)

When the common message has been eliminated using SIC at user-k from the received signal, the private message can be decoded, and the SINR at user-k for decoding $s_k$ is

$${\gamma }_k={\displaystyle \frac{{\left|{\mathbf{h}}_k^H{\mathbf{p}}_k\right|}^2}{{\left|{\mathbf{h}}_k^H{\mathbf{P}}_r\right|}^2+{\sum }_{j=1,j\ne k}^K{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}}.$$

(4)

The corresponding achievable rates of decoding $s_0$ and $s_k$ at user-k can be given as follows:

$$R_{0,k}={log}_2(1+{\gamma }_{0,k}),$$

(5)

$$R_k={log}_2(1+{\gamma }_k).$$

(6)

To ensure that each user can correctly decode $s_0$, the achievable rate of decoding $s_0$ should not exceed the rate of each user to decode $s_0$, i.e.,

$$R_0=min[R_{0,1},\dots ,R_{0,K}].$$

(7)

We have $R_{0,k}\ge R_0$, for $k\in [1,\dots ,K]$. $C_k$ is the part of $R_0$ corresponding to user-k, simply to say, $C_k$ is the achievable rate of transmitting $W_{0,k}$ [30]. We have

$$R_0=C_1+\dots +C_K.$$

(8)

Consequently, we can obtain the total rate of user-k as

$$R_{k,\mathrm{tot}}=R_k+C_k.$$

(9)

Let $\mathit{\mu }=[{\mu }_1,\dots ,{\mu }_K]$ be the weighting vector, and the WSR can be written as $WSR={\mu }_1{R}_{1,\mathrm{tot}}+\dots +{\mu }_K{R}_{K,\mathrm{tot}}$.

2.3. Metric for Radar Performance

For MIMO radar, the beam pattern is treated as the metric to measure the performance of the sensing. MIMO radar beamforming is the process of forming transmit beams in some specified directions to obtain more information about targets. As discussed in [47,48,49], the covariance matrix $\mathbf{R}$ was designed to make the obtained beam pattern approach the desired beam pattern $p_d\left({\theta }_l\right)$, and this relationship can be written mathematically as

$$L_r={\displaystyle \frac{1}{L}}\sum _{l=1}^L{\left|\alpha p_d\left({\theta }_l\right)-{\mathbf{a}}^H\left({\theta }_l\right)\mathbf{Ra}\left({\theta }_l\right)\right|}^2,$$

(10)

where $\alpha$ is a scaling factor. The desired beam pattern level at ${\theta }_l$ is $p_d\left({\theta }_l\right)$, ${\left\{\theta \right\}}_1^L$ are azimuth angle grids that cover $[-{90}^{\circ },{90}^{\circ }]$. There are L grids, which indicates that the detection angle is split into L sections equally. $\mathbf{a}\left(\theta \right)={[e^{j{\phi }_1\left(\theta \right)},e^{j{\phi }_2\left(\theta \right)},\dots ,e^{j{\phi }_N\left(\theta \right)}]}^T\in {\mathbb{C}}^{N\times 1}$ represents the steering vector at the $\theta$ angle, ${\phi }_j\left(\theta \right)$ is the electrical angle. Many researchers consider and verify that designing a beam pattern equates to designing a radar-transmitted signal’s covariance [23,24,25,50]. According to the loss function (10), the covariance matrix of radar transmitted signal can be designed as follows:

$$\begin{array}{cc}\hfill \underset{\alpha ,\mathbf{R}}{min}& {\displaystyle \frac{1}{L}}\sum _{l=1}^L{\left|\alpha p_d\left({\theta }_l\right)-{\mathbf{a}}^H\left({\theta }_l\right)\mathbf{R}\mathbf{a}\left({\theta }_l\right)\right|}^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathrm{diag}\left(\mathbf{R}\right)={\displaystyle \frac{P_t\mathbf{1}}{N}},\hfill \\ \hfill & \mathbf{R}\succ 0,\hfill \end{array}$$

(11)

where $P_t$ denotes the total transmit power, $\mathbf{1}={[1,1,\dots ,1]}^T\in {\mathbb{R}}^{N\times 1}$ is a unit vector and $\mathbf{R}$ is a positive semidefinite matrix.

3. Proposed Beamforming Design Schemes

The purpose of this section is to jointly design communication beamforming ${\mathbf{P}}_c$ and radar beamforming ${\mathbf{P}}_r$. We first propose a beamforming scheme from the radar-centric viewpoint using beam pattern approximation and achieve optimal beamforming by solving a nonconvex optimization problem in Section 3.1. In Section 3.2, we propose a trade-off scheme to ensure communication sum rate and achieve flexible optimization of sensing and communication performances using a regularization parameter. In Section 3.3, we propose a P-RS-aided beamforming design scheme aimed at maximizing the effective sensing power of MIMO radar while satisfying the minimum requirement of multiuser communication.

3.1. Radar-Centric Beamforming

In this subsection, we propose a radar-centric beamforming scheme to achieve sensing efficiency through beam pattern approximation, while requiring fairness SINR to be higher than a given threshold to guarantee a minimal level of communication quality. Here, we take $\mathbf{P}=\left[{\mathbf{P}}_c{\mathbf{P}}_r\right]\in {\mathbb{C}}^{N\times \left(K+1+N\right)}$, which is the overall precoding matrix. For ISAC, the covariance matrix $\mathbf{R}$ is defined as

$$\mathbf{R}=\mathbf{E}\left(\mathbf{X}{\mathbf{X}}^H\right).$$

(12)

Substituting (1) into (12), we have

$$\mathbf{R}=\mathbf{P}{\mathbf{P}}^H.$$

(13)

We use ${\mathbf{P}}_{\tau }$ to represent the $\tau \text{-}\mathrm{th}$ column of the matrix $\mathbf{P}$, and $\tau =1,\dots ,K+1+N$. Therefore, (13) can be rewritten as

$$\mathbf{R}=\sum _{\tau =1}^{K+1+N}{\mathbf{P}}_{\tau }{\mathbf{P}}_{\tau }^H.$$

(14)

Let ${\mathbf{R}}_{\tau }={\mathbf{P}}_{\tau }{\mathbf{P}}_{\tau }^H$, we have

$$\mathbf{R}=\sum _{\tau =1}^{K+1+N}{\mathbf{R}}_{\tau }.$$

(15)

To guarantee the minimum performance requirements of communication, the rate of decoding the common message at user-k should be greater than or equal to 0, and the rate of decoding the private message at user-k should be greater than or equal to a given threshold $R^{th}$, which is expressed as follows

$$R_{0,k}\ge 0,R_k\ge R^{th}.$$

(16)

In this work, we take $\gamma =min[{\gamma }_1,\dots ,{\gamma }_K]$ and require it to be higher than $\Gamma$, which is the SINR value corresponding to the threshold $R^{th}$. Consequently, the optimization problem can be formulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathbf{P},\alpha }{min}& L_r\hfill \\ \hfill s.t.& {\gamma }_{0,k}\ge 0,\hfill \\ \hfill & {\gamma }_k\ge \Gamma ,\hfill \\ \hfill & \mathbf{R}\underline{\succ }0,\mathbf{R}=\mathbf{P}{\mathbf{P}}^H,\hfill \\ \hfill & diag\left(\mathbf{R}\right)=P_t\mathbf{1}/N,\hfill \end{array}$$

(17)

In problem (17), the first column of the matrix $\mathbf{P}$ denotes the precoder ${\mathbf{p}}_0$ for ${\mathrm{s}}_0$, and the next K columns of the matrix $\mathbf{P}$ denote precoders of K private streams. Therefore, the individual matrices ${\left\{{\mathbf{R}}_{\tau }\right\}}_{\tau \ge K+2}$ have no effect on the sum rate of the users, and then we can remove it from the matrix $\mathbf{R}$, i.e., $\mathbf{R}-{\displaystyle \sum _{\tau =1}^{K+1}}{\mathbf{R}}_{\tau }$. Therefore, the problem (17) becomes

$$\begin{array}{cc}\hfill \underset{\Psi }{min}& L_r\hfill \\ \hfill s.t.& {\gamma }_{0,k}\ge 0,\hfill \\ \hfill & {\gamma }_k\ge \Gamma ,\hfill \\ \hfill & \mathbf{R}\underline{\succ }0,\mathbf{R}-\sum _{\tau =1}^{K+1}{\mathbf{R}}_{\tau }\underline{\succ }0,\hfill \\ \hfill & rank\left({\mathbf{R}}_{\tau }\right)=1,\hfill \\ \hfill & diag\left(\mathbf{R}\right)=P_t\mathbf{1}/N,\hfill \end{array}$$

(18)

where $\Psi =\{\mathbf{R},{\mathbf{R}}_1,\dots ,{\mathbf{R}}_{K+1},\alpha \}$ is the set of optimization variables, $rank\left({\mathbf{R}}_{\tau }\right)=1$ indicates that the rank of the matrix ${\mathbf{R}}_{\tau }$ is 1. The problem (18) is nonconvex due to the rank-one constraint. To solve problem (18), we can obtain the following relaxation by omitting these constraints.

Here, we use the semidefinite relaxation (SDR) algorithm [51] to solve the nonconvex problem (18). According to (3), (4) and (14) we have

$$\begin{array}{cc}\hfill & {\gamma }_{0,k}={\displaystyle \frac{{\mathbf{h}}_k^H{\mathbf{R}}_1{\mathbf{h}}_k}{{\mathbf{h}}_k^H\mathbf{R}{\mathbf{h}}_k-{\mathbf{h}}_k^H{\mathbf{R}}_1{\mathbf{h}}_k+{\sigma }_k^2}},\hfill \\ \hfill & {\gamma }_k={\displaystyle \frac{{\mathbf{h}}_k^H{\mathbf{R}}_k{\mathbf{h}}_k}{{\mathbf{h}}_k^H\mathbf{R}{\mathbf{h}}_k-{\mathbf{h}}_k^H{\mathbf{R}}_k{\mathbf{h}}_k+{\sigma }_k^2}}.\hfill \end{array}$$

(19)

It is not difficult to find that the optimization problem which substitutes (19) into (18) is still nonconvex. The problem can be relaxed as

$$\begin{array}{cc}\hfill \underset{\Psi }{min}& L_r\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\mathbf{h}}_k^H\mathbf{R}{\mathbf{h}}_k-{\mathbf{h}}_k^H{\mathbf{R}}_1{\mathbf{h}}_k+{\sigma }_k^2\ge 0,\hfill \\ \hfill & (1+{\Gamma }^{-1}){\mathbf{h}}_k^H{\mathbf{R}}_k{\mathbf{h}}_k\ge {\mathbf{h}}_k^H\mathbf{R}{\mathbf{h}}_k+{\sigma }_k^2,\hfill \\ \hfill & \mathbf{R}\underline{\succ }0,\mathbf{R}-\sum _{\tau =1}^{K+1}{\mathbf{R}}_{\tau }\underline{\succ }0,\hfill \\ \hfill & {\mathbf{R}}_{\tau }\underline{\succ }0,\hfill \\ \hfill & diag\left(\mathbf{R}\right)=P_t\mathbf{1}/N,\hfill \end{array}$$

(20)

Since ${\mathbf{R}}_1$, …, ${\mathbf{R}}_{K+1}$ are exactly rank-one matrices, the relaxation employed in SDR is tight. Consequently, the solution to the nonconvex problem (20) also serves as a solution to problem (18). Derivation can be found at the bottom of this subsection.

The solution ${\mathbf{R}}^{*}$, ${\mathbf{R}}_1^{*}$, …, ${\mathbf{R}}_{K+1}^{*}$ can be obtained by solving problem (20). A rank-one optimal solution $\widehat{\mathbf{R}}$, ${\widehat{\mathbf{R}}}_1$, …, ${\widehat{\mathbf{R}}}_{K+1}$ and the corresponding optimal precoding matrix ${\widehat{\mathbf{P}}}_c=[{\widehat{\mathbf{p}}}_0,{\widehat{\mathbf{p}}}_1,\dots ,{\widehat{\mathbf{p}}}_K]$ are obtained via

$$\widehat{\mathbf{R}}={\mathbf{R}}^{*},{\widehat{\mathbf{P}}}_k={\displaystyle \frac{{\mathbf{R}}_k^{*}{\mathbf{h}}_k}{\sqrt{{\mathbf{h}}_k^H{\mathbf{R}}_k^{*}{\mathbf{h}}_k}}},{\widehat{\mathbf{R}}}_k={\widehat{\mathbf{P}}}_k{\widehat{\mathbf{P}}}_k^H,$$

(21)

$${\widehat{\mathbf{p}}}_0={\widehat{\mathbf{P}}}_1,\dots ,{\widehat{\mathbf{p}}}_k={\widehat{\mathbf{P}}}_{k+1}.$$

(22)

The remaining part of the matrix $\mathbf{P}$ (i.e., ${\left\{{\mathbf{P}}_{\tau }\right\}}_{K+2\le \tau \le K+1+N}$) is the precoder ${\mathbf{P}}_r\in {\mathbb{C}}^{N\times N}$ for radar signal, and it can be obtained by using the Cholesky decomposition

$${\widehat{\mathbf{P}}}_r{\widehat{\mathbf{P}}}_r^H=\widehat{\mathbf{R}}-\sum _{k=1}^{K+1}{\widehat{\mathbf{P}}}_k{\widehat{\mathbf{P}}}_k^H.$$

(23)

Therefore, the overall precoding matrix $\widehat{\mathbf{P}}=[{\widehat{\mathbf{P}}}_1,\dots ,{\widehat{\mathbf{P}}}_{K+1},{\widehat{\mathbf{P}}}_{K+2},\dots ,{\widehat{\mathbf{P}}}_{K+1+N}]=\left[{\widehat{\mathbf{P}}}_c{\widehat{\mathbf{P}}}_r\right]$. The computational procedure is summarized in Algorithm 1.

Algorithm 1 Beamforming Design via SDR.

Input: Instantaneous channel $\mathbf{H}$, transmit power $P_t$, SINR threshold $\Gamma$; Output: Precoder matrix $\mathbf{P}$; 1: Compute the optimal solution $\widehat{\mathbf{R}},{\widehat{\mathbf{R}}}_1,\dots ,{\widehat{\mathbf{R}}}_{K+1}$ by solving problem (20); 2: Compute ${\widehat{\mathbf{p}}}_1,\dots ,{\widehat{\mathbf{p}}}_K,{\widehat{\mathbf{p}}}_0$ using (21) and (22); 3: Compute ${\widehat{\mathbf{P}}}_r$ using (23); 4: Output the precoder $\widehat{\mathbf{P}}=[{\widehat{\mathbf{p}}}_0,{\widehat{\mathbf{p}}}_1,\dots ,{\widehat{\mathbf{p}}}_K,{\widehat{\mathbf{P}}}_r]$

Proof. 

Denote ${\mathbf{R}}^{*}$, ${\mathbf{R}}_1^{*}$, …, ${\mathbf{R}}_{K+1}^{*}$ as an arbitrary global optimal solution to (20). Now, we construct $\widehat{\mathbf{R}}$, ${\widehat{\mathbf{R}}}_1$, …, ${\widehat{\mathbf{R}}}_{K+1}$ from ${\mathbf{R}}^{*}$, ${\mathbf{R}}_1^{*}$, …, ${\mathbf{R}}_{K+1}^{*}$ as

$$\widehat{\mathbf{R}}={\mathbf{R}}^{*},{\widehat{\mathbf{P}}}_k={\displaystyle \frac{{\mathbf{R}}_k^{*}{\mathbf{h}}_k}{\sqrt{{\mathbf{h}}_k^H{\mathbf{R}}_k^{*}{\mathbf{h}}_k}}},{\widehat{\mathbf{R}}}_k={\widehat{\mathbf{P}}}_k{\widehat{\mathbf{P}}}_k^H,$$

(24)

Obviously, $\widehat{\mathbf{R}}$, ${\widehat{\mathbf{R}}}_1$, …, ${\widehat{\mathbf{R}}}_{K+1}$ are rank-one and positive semidefinite. We need to prove that $\widehat{\mathbf{R}}$, ${\widehat{\mathbf{R}}}_1$, …, ${\widehat{\mathbf{R}}}_{K+1}$ is a global optimum to (20). From (24), we have

$${\mathbf{h}}_k^H{\widehat{\mathbf{R}}}_k{\mathbf{h}}_k={\mathbf{h}}_k^H{\widehat{\mathbf{P}}}_k{\widehat{\mathbf{P}}}_k^H{\mathbf{h}}_k={\mathbf{h}}_k^H{\mathbf{R}}_K^{*}{\mathbf{h}}_k,$$

(25)

for any $\mathbf{Q}\in {\mathbb{C}}^N$, it holds that

$${\mathbf{Q}}^H({\mathbf{R}}_k^{*}-{\widehat{\mathbf{R}}}_k)\mathbf{Q}={\mathbf{Q}}^H{\mathbf{R}}_K^{*}\mathbf{Q}-{\left({\mathbf{h}}_k^H{\mathbf{R}}_K^{*}{\mathbf{h}}_k\right)}^{-1}\left|{\mathbf{Q}}^H{\mathbf{R}}_K^{*}{\mathbf{h}}_k\right|.$$

(26)

Then, we can obtain that $\left({\mathbf{h}}_k^H{\mathbf{R}}_K^{*}{\mathbf{h}}_k\right)\left({\mathbf{Q}}^H{\mathbf{R}}_k^{*}\mathbf{Q}\right)\ge \left|{\mathbf{Q}}^H{\mathbf{R}}_K^{*}{\mathbf{h}}_k\right|$ on the basis of the Cauchy–Schwarz inequality. Consequently, for any $\mathbf{Q}\in {\mathbb{C}}^N$, there is ${\mathbf{Q}}^H({\mathbf{R}}_k^{*}-{\widehat{\mathbf{R}}}_k)\mathbf{Q}\ge 0$. It means that ${\mathbf{R}}_k^{*}-{\widehat{\mathbf{R}}}_k\underline{\succ }0$, and we have

$$\begin{array}{cc}\hfill \widehat{\mathbf{R}}-\sum _{k=1}^{K+1}{\widehat{\mathbf{R}}}_k& ={\mathbf{R}}^{*}-\sum _{k=1}^{K+1}{\mathbf{R}}_k^{*}+\sum _{k=1}^{K+1}{\mathbf{R}}_k^{*}-\sum _{k=1}^{K+1}{\widehat{\mathbf{R}}}_k\hfill \\ \hfill & ={\mathbf{R}}^{*}-\sum _{k=1}^{K+1}{\mathbf{R}}_k^{*}+\sum _{k=1}^{K+1}({\mathbf{R}}_k^{*}-{\widehat{\mathbf{R}}}_k)\underline{\succ }0.\hfill \end{array}$$

(27)

Since $\widehat{\mathbf{R}}={\mathbf{R}}^{*}$, this means that constraints $\mathbf{R}\underline{\succ }0,\mathbf{R}-{\displaystyle \sum _{\tau =1}^{K+1}}{\mathbf{R}}_{\tau }\underline{\succ }0$, and ${\mathbf{R}}_{\tau }\underline{\succ }0$ in problem (20) hold for $\widehat{\mathbf{R}}$, ${\widehat{\mathbf{R}}}_1$, …, ${\widehat{\mathbf{R}}}_{K+1}$. The derivation verifies that $\widehat{\mathbf{R}}$, ${\widehat{\mathbf{R}}}_1$, …, ${\widehat{\mathbf{R}}}_{K+1}$ is a set of solutions to (20), and the proof is completed.    □

3.2. Beamforming Design for Trade-Off

It should be pointed out that problem in (17) is focused on the maximization of sensing performance, while it may cause downlink communication performance loss. In this subsection, we consider another beamforming design scheme to achieve a flexible trade-off between sensing and communication performance and propose an iterative method based on weighted minimum mean square error (WMMSE) and SDR to solve the resulting nonconvex optimization problem.

In [25], the objective function of optimization problem contains two parts, WSR of communication users and radar beam patterns. We use MSE of the beam patterns and WSR as the performance metrics for MIMO radar and multiuser communication, respectively. Similarly, we introduce a regularization parameter $0\le \lambda \le 1$ to flexibly control the performance trade-off between communication and radar. Consequently, the beamforming design scheme for trade-off can be formulated as an optimization problem as follows:

$$\begin{array}{cc}\hfill \underset{\mathbf{P},\alpha ,\mathbf{c}}{max}& (1-\lambda )\sum _{k=1}^K{\mu }_k{R}_{k,\mathrm{tot}}-\lambda L_r\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \sum _{k=1}^K{C}_{k,0}\le R_{0,k},\hfill \\ \hfill & {\gamma }_k\ge \Gamma ,\mathbf{c}\ge 0,\hfill \\ \hfill & \mathbf{R}=\mathbf{P}{\mathbf{P}}^H,\hfill \\ \hfill & \mathbf{R}\underline{\succ }0,\hfill \\ \hfill & diag\left(\mathbf{P}{\mathbf{P}}^H\right)=P_t\mathbf{1}/N.\hfill \end{array}$$

(28)

Each element of $\mathbf{c}=[C_{1,0},\dots ,C_{K,0}]$ denotes the portion of $R_0$ for the corresponding user. o ensure the quality of service (QoS), constraints are imposed on ${\gamma }_k$ for decoding the private part and on $\mathbf{c}$ for the common part. Due to the presence of quadratic equality constraints in the problem (28), it exhibits non-convexity.

In the communication part, the existing literature [37,38,39,40] indicates that user-k first decodes the estimated common stream ${\widehat{s}}_0$ using an equalizer $g_{0,k}$ and remove it from received signal, then we can decode the estimated private stream ${\widehat{s}}_k$ using an equalizer $g_k$ from the remaining message, i.e., ${\widehat{s}}_0=g_{0,k}{y}_k$ and ${\widehat{s}}_k=g_k(y_k-{\mathbf{h}}_k^H{\mathbf{p}}_0{\widehat{s}}_0)$. When decoding $s_0$ and $s_k$ at user-k, the MSEs can be calculated as

$$\begin{array}{cc}\hfill & {\epsilon }_{0,k}\triangleq \mathbf{E}\left\{{\left|{\widehat{s}}_0-s_0\right|}^2\right\},\hfill \\ \hfill & {\tilde{g}}_k={\displaystyle \frac{{\mathbf{p}}_k^H{\mathbf{h}}_k}{{\displaystyle \sum _{j=1}^K}{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}},\hfill \end{array}$$

(29)

By solving ${\displaystyle \frac{\partial {\epsilon }_{0,k}}{\partial g_{0,k}}}=0$ and ${\displaystyle \frac{\partial {\epsilon }_k}{\partial g_k}}=0$, we can obtain the minimized MSE equalizers ${\tilde{g}}_{0,k}$ and ${\tilde{g}}_k$ as follows:

$$\begin{array}{cc}\hfill & {\tilde{g}}_{0,k}={\displaystyle \frac{{\mathbf{p}}_0^H{\mathbf{h}}_k}{{\left|{\mathbf{h}}_k^H{\mathbf{p}}_0\right|}^2+{\displaystyle \sum _{j=1}^K}{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}},\hfill \\ \hfill & {\tilde{g}}_k={\displaystyle \frac{{\mathbf{p}}_k^H{\mathbf{h}}_k}{{\displaystyle \sum _{j=1}^K}{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}}.\hfill \end{array}$$

(30)

Substituting (30) into (29), we have

$$\begin{array}{cc}\hfill {\tilde{\epsilon }}_{0,k}=& {\displaystyle \frac{{\left|{\mathbf{h}}_k^H{\mathbf{p}}_0\right|}^2+{\displaystyle \sum _{j=1,j\ne k}^K}{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}{{\left|{\mathbf{h}}_k^H{\mathbf{p}}_0\right|}^2+{\displaystyle \sum _{j=1}^K}{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}},\hfill \\ \hfill {\tilde{\epsilon }}_k=& {\displaystyle \frac{{\displaystyle \sum _{j=1,j\ne k}^K}{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}{{\displaystyle \sum _{j=1}^K}{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}}.\hfill \end{array}$$

(31)

At user-k, the SINRs of decoding $s_0$ and $s_k$ can be transformed as ${\gamma }_{0,k}=1/{\tilde{\epsilon }}_{0,k}-1$, ${\gamma }_k=1/{\tilde{\epsilon }}_k-1$, then its rates are

$$\begin{array}{cc}\hfill R_{0,k}& =-{log}_2(1+{\tilde{\epsilon }}_{0,k}),\hfill \\ \hfill R_k& =-{log}_2(1+{\tilde{\epsilon }}_k).\hfill \end{array}$$

(32)

The weighted MSEs for decoding $s_0$ and $s_k$ are

$$\begin{array}{cc}\hfill {\xi }_{c,k}\triangleq & {\omega }_{0,k}{\epsilon }_{0,k}-{log}_2\left({\omega }_{0,k}\right),\hfill \\ \hfill {\xi }_k\triangleq & {\omega }_k{\epsilon }_k-{log}_2\left({\omega }_k\right),\hfill \end{array}$$

(33)

where ${\omega }_{0,k}$ and ${\omega }_k$ are the positive weights for common and private rates of user-k. We can obtain relationships as follows:

$$\begin{array}{cc}\hfill {\tilde{\xi }}_{0,k}\triangleq & \underset{{\omega }_{0,k},g_{0,k}}{min}{\xi }_{0,k}=1-R_{0,k},\hfill \\ \hfill {\tilde{\xi }}_k\triangleq & \underset{{\omega }_k,g_k}{min}{\xi }_k=1-R_k,\hfill \end{array}$$

(34)

where optimal values are $g_{0,k}^{*}={\tilde{g}}_{0,k}$, $g_k^{*}={\tilde{g}}_k$, ${\omega }_{0,k}^{*}={\tilde{\omega }}_{0,k}\triangleq {\left({\tilde{\epsilon }}_{0,k}\right)}^{-1}$, ${\omega }_k^{*}={\tilde{\omega }}_k\triangleq {\left({\tilde{\epsilon }}_k\right)}^{-1}$. By satisfying the first-order optimality conditions, these optimal values are obtained. Subsequently, the problem (28) can be rewritten as

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}\mathbf{R},\left\{{\mathbf{R}}_i\right\}\\ \alpha ,\mathbf{c},\mathit{g},\mathit{\omega }\end{array}}{min}& (1-\lambda )\sum _{k=1}^K{\mu }_k{\xi }_k-\lambda L_r\hfill \\ \hfill s.t.& \sum _{k=1}^K{C}_{k,0}\le R_{0,k},\hfill \\ \hfill & {\gamma }_k\ge \Gamma ,\mathbf{c}\ge 0,\hfill \\ \hfill & \mathbf{R}=\sum _{\tau =1}^{K+1+N}{\mathbf{R}}_{\tau },\hfill \\ \hfill & {\mathbf{R}}_{\tau }\underline{\succ }0,rank\left({\mathbf{R}}_{\tau }\right)=1,\hfill \\ \hfill & diag\left(\mathbf{R}\right)=P_t\mathbf{1}/N,\hfill \end{array}$$

(35)

where ${\xi }_{k-tot}={\xi }_k-C_k$, $\mathit{g}=[g_{0,1},\dots ,g_{0,K},g_1,\dots ,g_K]$ and $\mathit{\omega }=[{\omega }_{0,1},\dots ,{\omega }_{0,K},{\omega }_1,\dots ,{\omega }_K]$ are the equalizers and weights.

Similarly, for that reason, the individual matrices ${\left\{{\mathbf{R}}_{\tau }\right\}}_{\tau \ge K+2}$ have no effect on the sum rate of the users, and we can remove it from the matrix $\mathbf{R}$. Therefore, the problem (35) is relaxed as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}\mathbf{R},{\mathbf{R}}_1,\dots ,{\mathbf{R}}_{K+1}\\ \alpha ,\mathbf{c},\mathit{g},\mathit{\omega }\end{array}}{min}& (1-\lambda )\sum _{k=1}^K{\mu }_k{\xi }_k-\lambda L_r\hfill \\ \hfill s.t.& \sum _{k=1}^K{C}_{k,0}\le R_{0,k},\hfill \\ \hfill & {\gamma }_k\ge \Gamma ,\mathbf{c}\ge 0,\hfill \\ \hfill & \mathbf{R}\underline{\succ }0,\mathbf{R}-\sum _{\tau =1}^{K+1}{\mathbf{R}}_{\tau }\underline{\succ }0,\hfill \\ \hfill & {\mathbf{R}}_{\tau }\underline{\succ }0,\hfill \\ \hfill & diag\left(\mathbf{R}\right)=P_t\mathbf{1}/N.\hfill \end{array}$$

(36)

since ${\mathbf{R}}_1$, …, ${\mathbf{R}}_{K+1}$ are exactly rank-one, the relaxation by using SDR is tight. The derivation is similar to Section 3.1.

Briefly, our proposed iterative algorithm based on WMMSE-SDR mainly involves two procedures:

(1) Given the beamforming matrix $\mathbf{P}=\left[{\mathbf{P}}_c{\mathbf{P}}_r\right]$, the optimal ${\mathit{\omega }}^{*}$ and ${\mathit{g}}^{*}$ are obtained by WMMSE;

(2) Given $\mathit{\omega }$ and $\mathbf{g}$, the suboptimal ${\mathbf{R}}_1$, …, ${\mathbf{R}}_{K+1}$, $\mathbf{c}$ are obtained by SDR.

We can obtain the optimal solution ${\mathbf{R}}^{*}$, ${\mathbf{R}}_1^{*}$, …, ${\mathbf{R}}_{K+1}^{*}$ by solving the problem (36), then we can compute a rank-one optimal solution $\widehat{\mathbf{R}}$, ${\widehat{\mathbf{R}}}_1$, …, ${\widehat{\mathbf{R}}}_{K+1}$ and the corresponding optimal precoding matrix ${\widehat{\mathbf{P}}}_c=[{\widehat{\mathbf{p}}}_0,{\widehat{\mathbf{p}}}_1,\dots ,{\widehat{\mathbf{p}}}_K]$ via (21) and (22).

The remaining part of the matrix $\mathbf{P}$ (i.e., ${\left\{{\mathbf{P}}_{\tau }\right\}}_{K+2\le \tau \le K+1+N}$) is the precoder ${\mathbf{P}}_r\in {\mathbb{C}}^{N\times N}$ for radar signal, and we can obtain it by (23). Therefore, the overall precoding matrix $\widehat{\mathbf{P}}=[{\widehat{\mathbf{P}}}_1,\dots ,{\widehat{\mathbf{P}}}_{K+1},{\widehat{\mathbf{P}}}_{K+2},\dots ,{\widehat{\mathbf{P}}}_{K+1+N}]=\left[{\widehat{\mathbf{P}}}_c{\widehat{\mathbf{P}}}_r\right]$. We summarize the method to obtain the matrix $\mathbf{P}$ as Algorithm 2.

The complexities of Algorithms 1 and 2 are discussed in this paragraph. These two algorithms are completed based on SDR, and we analyze the complexities from the term of SDR by $\mathcal{O}(\max {\{m,n\}}^4{n}^{1/2}log(1/\ell ))$ [52]. Given a solution accuracy, the number of restrictions m and the problem size n have polynomial effects on its difficulty. The number of constraints is the same for the optimization problems (20) and (36), while the number of variables of problem (20) is $(k+3)$ and that of problem (36) is $(k+6)$. Then, the complexity of Algorithm 1 and Algorithm 2 is dominated by the number of variables of problems (20) and (36), respectively. Therefore, the complexities of Algorithm 1 is $\mathcal{O}({(k+3)}^{4.5}log(1/\ell ))$ and the complexity of Algorithm 2 is $\mathcal{O}({(k+6)}^{4.5}log(1/\ell ))$. Obviously, the complexity of Algorithm 1 is relatively low.

Algorithm 2 Iterative Beamforming Algorithm.

Input: Instantaneous channel $\mathbf{H}$; Transmit power $P_t$; Precoder ${{\mathbf{P}}_c}^{\left[z\right]}$, $z\leftarrow 0$; $\epsilon \ge 0$; while $\left|\mathrm{WS}{\mathrm{R}}^{\left[z\right]}-\mathrm{WS}{\mathrm{R}}^{[z+1]}\right|\le \epsilon$ do     1: Compute the optimal solution ${\mathit{\omega }}^{*}$ and ${\mathit{g}}^{*}$ by WMMSE;     2: Compute the ${\mathbf{c}}^{\left[z\right]}$, ${\mathbf{R}}^{*\left[z\right]}$, ${\mathbf{R}}_1^{*\left[z\right]}$, ${\mathbf{R}}_{K+1}^{*\left[z\right]}$ by solving problem (36) with updated ${\mathit{\omega }}^{*}$ and ${\mathit{g}}^{*}$;     3: Update ${\widehat{\mathbf{P}}}_c^{[z+1]}=[{\widehat{\mathbf{p}}}_0^{[z+1]},{\widehat{\mathbf{p}}}_1^{[z+1]},\dots ,{\widehat{\mathbf{p}}}_K^{[z+1]}]$ via (21) and (22);     4: Update $\mathrm{WS}{\mathrm{R}}^{[z+1]}$ using ${{\widehat{\mathbf{P}}}_c}^{[z+1]}$ and ${\mathbf{c}}^{\left[z\right]}$;     5: $z\leftarrow z+1$; end while Compute ${\widehat{\mathbf{P}}}_r$ via (23); Output: The overall precoder $\widehat{\mathbf{P}}=\left[{\widehat{\mathbf{P}}}_c{\widehat{\mathbf{P}}}_r\right]$;

3.3. P-RS-Based Beamforming

We note that the achievable rate of decoding $s_0$ should not exceed the rate of each user decoding $s_0$, i.e., $R_0\le min[R_{0,1},\dots ,R_{0,K}]$, to ensure that a common stream can be decoded by every downlink user. Therefore, the achievable rate of the common stream is limited by the user with the weakest ability to decode the common part among all users making it unable to fully utilize the common stream to reduce the impact of multiuser nonorthogonality on the private streams and sensing performance. To address this issue, we propose a P-RS-based beamforming algorithm to improve the performance of the ISAC system. In our proposed scheme, not all messages are split at the transmitter. Instead, the message of the user with the worst ability to decode the common part will be transmitted directly after being precoded.

The factors that affect a user’s ability to decode the common part may not be unique, among which the most straightforward factor is the channel state. It is hypothesized that the base station determines whether the user’s message is split at the transmitter based on its corresponding channel quality. For instance, the message of a user with poor channel quality is not split but instead precoded as a private stream before transmission. Based on this, we provide an example to verify the effectiveness of our proposed algorithm in enhancing sensing performance for further in-depth research in future work.

All users are divided into two sets, namely set $\mathcal{A}$ and set $\mathcal{B}$, by the base station based on their channel status information. Set $\mathcal{A}$ comprises users whose messages are split at the transmitter, while set $\mathcal{B}$ comprises users whose messages are not split due to poor channels. As shown in Figure 2a, the messages of the users in set $\mathcal{A}$ are still split into two parts at the transmitter: a common part and a private part. Meanwhile, the messages of the users in set $\mathcal{B}$ are directly encoded as private messages. It is hypothesized that there are $K_1$ users assigned to set $\mathcal{A}$ and $K_2$ users assigned to set $\mathcal{B}$. It should be noted that $\mathcal{A}\cap \mathcal{B}=\varnothing$ and $K_1+K_2=K$, where K is the total number of downlink users. Particularly, RS is a special case of P-RS when $K_1=K$. At the user in set $\mathcal{A}$, the common message is decoded first, and then the private message is decoded after the common message is eliminated using SIC from the received signal. In contrast, users in set $\mathcal{B}$ no longer decode the common message. In other words, the part within the dashed box in Figure 2b does not work for users in set $\mathcal{B}$, and the corresponding private message is decoded directly. At the user in set $\mathcal{B}$, the interference for decoding its required private message includes not only the private messages of other users but also the interference caused by the common message.

Therefore, the SINRs for user-a ($a\in \mathcal{A}$), decoding the common message and the private message can be given as

$${\gamma }_{0,a}={\displaystyle \frac{{\left|{\mathbf{h}}_a^H{\mathbf{p}}_0\right|}^2}{{\left|{\mathbf{h}}_a^H{\mathbf{P}}_r\right|}^2+{\sum }_{i=1}^K{\left|{\mathbf{h}}_a^H{\mathbf{p}}_i\right|}^2+1}},a\in \mathcal{A}.$$

(37)

$${\gamma }_a={\displaystyle \frac{{\left|{\mathbf{h}}_a^H{\mathbf{p}}_a\right|}^2}{{\left|{\mathbf{h}}_a^H{\mathbf{P}}_r\right|}^2+{\sum }_{j\ne a}^K{\left|{\mathbf{h}}_a^H{\mathbf{p}}_j\right|}^2+1}},a\in \mathcal{A}.$$

(38)

The SINR for user-b ($b\in \mathcal{B}$) decoding the private message can be given as:

$${\gamma }_b={\displaystyle \frac{{\left|{\mathbf{h}}_b^H{\mathbf{p}}_b\right|}^2}{{\left|{\mathbf{h}}_b^H{\mathbf{P}}_r\right|}^2+{\left|{\mathbf{h}}_b^H{\mathbf{P}}_0\right|}^2+{\sum }_{j\ne b}^K{\left|{\mathbf{h}}_b^H{\mathbf{p}}_j\right|}^2+1}},b\in \mathcal{B}.$$

(39)

The achievable rates for decoding the common message and private message at user-a, and the private message at user-b are, respectively, given as

$$R_{0,a}={log}_2(1+{\gamma }_{0,a}),$$

(40)

$$R_a={log}_2(1+{\gamma }_a),$$

(41)

$$R_b={log}_2(1+{\gamma }_b).$$

(42)

Then the sum rate of downlink users is

$$R_{tot}={\sum }_{a\in \mathcal{A}}(R_{0,a}+C_a)+{\sum }_{b\in \mathcal{B}}{R}_b,$$

(43)

where $C_a$ is the part of the $R_0$ corresponding to user-a. In this subsection, we use the effective sensing power [25] as the performance metric for MIMO radar. To this end, the proposed P-RS-based transmit beamforming design scheme can be formulated as an optimization problem as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}\mathbf{P},\mathbf{c}\end{array}}{max}& {\mathbf{a}}^H\left({\theta }_l\right){\mathbf{PP}}^H\mathbf{a}\left({\theta }_l\right)\hfill \\ \hfill s.t.& \sum _{a\in \mathcal{A}}{C}_a\le min\left\{R_{0,a}\right\},a\in \mathcal{A}\hfill \\ \hfill & \mathbf{c}\ge 0,\hfill \\ \hfill & {\parallel \mathbf{P}\parallel }_F^2\ge P_t,\hfill \\ \hfill & R_{tot}\ge R_{th},\hfill \end{array}$$

(44)

where $\mathbf{P}=[{\mathbf{p}}_0,{\mathbf{p}}_1,\dots ,{\mathbf{p}}_K,{\mathbf{P}}_r]$, $\mathbf{c}=[C_1,\dots ,C_a,\dots ,C_A]$. Constraint ${\displaystyle \sum _{a\in \mathcal{A}}}{C}_a\le min\left\{R_{0,a}\right\}$ ensures that the common message can be decoded by every user in set $\mathcal{A}$, $R_{th}$ denotes the minimum required sum rate of downlink users. Problem (44) cannot be solved directly due to its nonconvexity. However, it can be solved through a series of transformations and the use of SDR. The detailed process can be found in Appendix A. The optimal covariance matrix $\left\{{\mathbf{R}}_i^{opt}\right\}$ can be obtained by solving the problem (A13) using the SDR method, and then beamformer $\mathbf{P}$ can be obtained according to (21)–(23).

4. Numerical Results

In this section, we provide simulation results to demonstrate the effectiveness and feasibility of the algorithms proposed in this work. We analyze the radar beam pattern of the proposed schemes compared with the existing scheme of [45] (i.e., WMMSE-MM), NOMA-ISAC [25], and SDMA-ISAC, achieved by allocating no power to the common stream) in Section 4.1. Section 4.2 analyzes the achievable sum rate of the proposed schemes. Meanwhile, we compare the feasibility of Algorithms 1 and 2. Finally, we validate the effectiveness of the proposed P-RS-based beamforming design scheme in Section 4.3.

In this ISAC system, we use a uniform linear array with half-wavelength spacing between adjacent antenna elements. It should be noted that the radar beam pattern is calculated by $P\left(\vartheta \right)={\sum }_{l=1}^L{\displaystyle \frac{1}{l}}\left|\mathbf{a}{\left({\theta }_l\right)}^H\mathbf{R}\mathbf{a}\left({\theta }_l\right)\right|$. In this section, we first demonstrate the superiority of RSMA for ISAC, and then we analyze the numerical results achieved by the beamforming design methods proposed in our paper. The performance difference for the proposed submodel algorithm arises from the varying focus in the dual function. Specifically, the SDR beamformer is concentrated on maximizing sensing performance while meeting the minimum requirements for multiuser communication. In contrast, the WMMSE-SDR achieves a flexible trade-off between communication and sensing performance under the total transmit power constraint. Additionally, enhancing the contribution of the achievable common rate to communication and sensing performance in ISAC is achieved through improvements in multiple access schemes, such as P-RS. We set $N=16$, $P_t=20$ dB, $\Gamma =10$ dB throughout simulations. Here, we consider three targets, and the locations of the three beams are set at $[-{50}^{\circ },0^{\circ },{50}^{\circ }]$. The channel vector ${\mathbf{h}}_k$ follows i.i.d. complex Gaussian distribution. In the following simulation, ‘Rad Only’ is the beam pattern obtained from (11), We use ‘Radar-centric’ and ‘Trade-off’ to denote the beam patterns obtained for radar-centric beamforming and trade-off beamforming proposed in Section 3.1 and Section 3.2, respectively.

4.1. Transmit Beam Pattern Comparison

To assess the effectiveness of the algorithms on sensing, we demonstrate the beam pattern obtained by the various schemes with different numbers of communication users, as shown in Figure 3. We set the weighting factor $\lambda =1\times {10}^{-1}$ and compare the beam patterns obtained by different schemes with $K=2$ and $K=4$.

In Figure 3a, it can be clearly seen that SDR is fully approaching the ideal beam pattern and has a nearly 2.17 dB gain over SDMA-based ISAC. This superiority is attributed to the flexibility of RSMA in managing interference compared with NOMA and SDMA. Comparing Figure 3a,b it is evident that the ISAC based on SDMA provides a poor beam pattern when $K=2$ compared with $K=4$ due to its lower spatial gain, indicating that SDMA achieves higher spatial gain with more users. Furthermore, the algorithms proposed in this paper outperform the WMMSE-MM algorithm attributed to the transmit signal for our system being a weighted sum of communication symbols and radar waveforms, which are transmitted by a shared uniform linear array to achieve both communication and sensing functions simultaneously to provides MIMO radar waveforms with more DoF. Moreover, the WMMSE-SDR method proposed in our paper can achieve a flexible trade-off between communication and sensing performance by adjusting the parameter $\lambda$. Meanwhile, it can be observed that the SDR method provides a better beam pattern than WMMSE-SDR. In other words, to further ensure communication performance, WMMSE improves the sum rate of communication at the expense of approximately 0.5 dB of directional gain. A detailed analysis of the communication performance can be found in Section 4.2.

Similarly, it is not difficult to observe that the power of the main beams towards $-{50}^{\circ }$, $0^{\circ }$, and ${50}^{\circ }$, steered by the SDR beamformer, exceeds that of the main beams steered by the WMMSE-SDR beamformer. This difference arises because the focuses of these two beamforming design methods are distinct. The SDR beamformer is optimized for radar performance with minimum communication quality constraints, while the WMMSE-SDR beamformer aims to achieve a performance trade-off between communication and radar. This explains why the beam pattern obtained for the SDR beamformer is closest to the Radar only.

Figure 4 is the comparison of detection probability for the design scheme proposed in this paper. The radar detection probability can be calculated by $P_d=1-{\mathfrak{F}}_{{\mathcal{X}}_2^2\left(\mu \right)}\left({\mathfrak{F}}_{{\mathcal{X}}_2^2\left(\mu \right)}(1-P_f)\right)$, where ${\mathfrak{F}}_{{\mathcal{X}}_2^2\left(\mu \right)}$ is the noncentral chi-square cumulative distribution function with two degrees of freedom, $P_f$ denotes the false alarm probability of the radar detection, the noncentrality parameter $\mu$ is given by $\mu ={\displaystyle \frac{P_t{\left|\mathbf{a}\left(\theta \right)\mathbf{R}\mathbf{a}{\left(\theta \right)}^H\right|}^2}{{\delta }^2}}$ [53]. As can be seen, the detection performance of the radar-centric scheme outperforms that of the trade-off scheme due to $P_d$ being a monotonically increasing function for $\mu$, and the radar-centric method can provide better beam pattern gains.

4.2. Achievable Sum Rate Comparison

We first show the achievable weighted sum rate obtained by different schemes in Figure 5 and Figure 6, then compare the MSE of radar beam pattern generated by different schemes in Figure 7. The MSE of the radar beam pattern can be expressed as follows:

$$MSE=\sum _{l=1}^L|\alpha p_d\left({\theta }_l\right)-{\mathbf{a}}^H\left({\theta }_l\right)\mathbf{Ra}\left({\theta }_l\right){|}^2.$$

(45)

We discuss the variation in communication performance with SINR threshold $\Gamma$ as illustrated in Figure 5. We note that the WMMSE-SDR beamforming method produces higher WSR compared with the SDR beamforming method. Meanwhile, it is not difficult to find that the almost same WSR results are obtained by different beamforming methods with the same number of users when the $\Gamma$ is high enough.

The relationship between the achievable WSR and MSE is demonstrated in Figure 6, and it reflects the trade-offs of beamforming design schemes with different numbers of transmit antennas. It is not difficult to observe that more antennas encourage a better trade-off between the WSR and MSE. Moreover, the trade-off design method proposed in this paper outperforms the WMMSE-MM beamformer owing to the transmitter in this paper making MIMO radar achieve larger DoF by using individual communication and radar waveforms. In order to analyze the performance of the ISAC system more comprehensively, the influences of $\Gamma$ on the achievable WSR of users and the MSE of radar beam patterns are explored next. The trade-off between the MSE of radar beam patterns and the SINR threshold $\Gamma$ for SDR and WMMSE-SDR beamforming is depicted in Figure 7.

We can observe that the MSE of the SDR algorithm is higher than that of the WMMSE-SDR algorithm, whether the number of users is $K=2$ or $K=4$. However, the gap between these two curves becomes closer as $\Gamma$ increases. From the MSE expression, it is evident that the MSE value is associated with $\mathbf{R}$. In other words, the MSE value is determined by the optimization variable $\mathbf{P}$. In the SDR beamforming method, the objective function of the optimization problem is the radar loss function, which is independent of the number of users.

The quantity related to the number of users is the SINR for the given thresholds in the constraints of the optimization problem. However, in the WMMSE-SDR beamforming method, apart from the SINR constraints, the sum rate component of the objective function is significantly influenced by the number of users. Therefore, the impact of the number of users on MSE in the SDR method is not as pronounced as the impact of MSE in the WMMSE-SDR method.

The proposed joint beamforming design problems (20) and (36) are not always feasible. The relationship between the number of users and the feasible probability is demonstrated in Figure 8. Here, we compare the feasibility of SDR and WMMSE-SDR for $\Gamma =10$ dB and $\Gamma =20$ dB, respectively, as illustrated in Figure 8a,b. It is noted that the feasible probability for SDR and WMMSE-SDR are almost the same, and both of them decrease as the number of users increases for the reason that a greater user base results in a decrease in the DoFs. The feasible probability decreases when $\Gamma$ is increased from 10 dB to 20 dB. Since $P_t$ is limited, the achievable SINR to users is limited. Therefore, the feasible probability of the two problems will decrease when $\Gamma$ increases.

4.3. The Effectiveness of the P-RS-Based Beamforming Design Scheme

In Section 4.1 and Section 4.2, we compared the effectiveness of various multiple access methods in ISAC applications and concluded that RSMA is a powerful multiple access and interference management strategy suitable for ISAC. Next, we explore the effectiveness of the proposed P-RS-based beamforming design scheme by comparing it with the transmit beam patterns of ISAC systems based on RS. We use ‘RS’ and ‘P-RS’ to denote the transmit beam patterns obtained by the RS-based and P-RS-based beamforming design schemes, respectively.

We conducted simulations for scenarios with 2 and 8 users, respectively. In the case of $K=2$, it is hypothesized that one user is in set $\mathcal{A}$ and another is in set $\mathcal{B}$, i.e., $K_1=1$, $K_2=1$, while in the case of $K=8$, it is assumed that five users with good channels are in set $\mathcal{A}$ and three users with poorest channels are in set $\mathcal{B}$, i.e., $K_1=5$, $K_2=3$.

As illustrated in Figure 9, the transmit beam patterns in the target directions gradually deteriorate as the number of users increases. However, it is evident that ISAC based on P-RS can maintain better performance compared with ISAC based on RS. This is attributed to P-RS no longer splitting the messages of users with a bad ability to decode the common stream, effectively removing the constraint imposed by these users on the common rate.

5. Future Work

According to Section 4.3, we verify that the proposed P-RS-based beamforming design scheme can effectively improve sensing performance. To achieve the adaptive scheme, we are committed to jointly optimizing the beamforming and the splitting decision factor (i.e., $\mathbf{t}=[t_1,t_2,\dots ,t_K]$, $t_k=1$ means that the message of user-k split at the transmitter and decodes its corresponding message by using SIC at user-k; otherwise, $t_k=0$). For instance, this algorithm can be formulated as an optimization problem as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}\mathbf{P},\mathbf{c},\mathbf{t}\end{array}}{max}& {\mathbf{a}}^H\left({\theta }_l\right){\mathbf{PP}}^H\mathbf{a}\left({\theta }_l\right)\hfill \\ \hfill s.t.& \sum _{k\in \mathcal{K}}{C}_k\le min\left\{R_{0,k}\right\},k\in [1,2,\dots ,K]\hfill \\ \hfill & \mathbf{c}\ge 0,\hfill \\ \hfill & {\parallel \mathbf{P}\parallel }_F^2\ge P_t,\hfill \\ \hfill & R_{tot}\ge R_{th},\hfill \end{array}$$

(46)

where the $R_{0,k}$ and $R_k$ are calculated from the SINR formulas as follows:

$${\gamma }_{0,k}={\displaystyle \frac{t_k{\left|{\mathbf{h}}_k^H{\mathbf{p}}_0\right|}^2}{{\left|{\mathbf{h}}_k^H{\mathbf{P}}_r\right|}^2+{\sum }_{i=1}^K{\left|{\mathbf{h}}_k^H{\mathbf{p}}_i\right|}^2+1}},$$

(47)

$${\gamma }_k={\displaystyle \frac{{\left|{\mathbf{h}}_k^H{\mathbf{p}}_k\right|}^2}{{\left|{\mathbf{h}}_k^H{\mathbf{P}}_r\right|}^2+(t_k-1){\left|{\mathbf{h}}_k^H{\mathbf{P}}_0\right|}^2+{\sum }_{j\ne k}^K{\left|{\mathbf{h}}_k^H{\mathbf{p}}_j\right|}^2+1}}.$$

(48)

The problem (46) is a nonconvex discrete optimization problem due to $t_k=1$ or 0 for $k\in [1,2,\dots ,K]$. In this paper, we do not solve this optimization problem. This will be part of our future work.

6. Conclusions

In this paper, we aimed to design transmit beamforming for ISAC that offers sensing efficiency and ensures communication. To this end, we use rate-splitting as a multiple access platform for multiuser communication, providing advanced interference management strategies and adopting a beam pattern approximation strategy in sensing. First, we propose a radar-centric beamforming scheme to achieve sensing efficiency through beam pattern approximation, while requiring the fairness SINR to be higher than a given threshold to ensure a minimal level of communication quality. Furthermore, we propose a beamforming design scheme from a trade-off viewpoint that flexibly optimizes both sensing and communication performances with a parameter. Finally, we propose a beamforming design method based on an improved RS aimed at maximizing the effective sensing power, with the constraint of a minimal sum rate for downlink users. Numerical results are provided to verify the performance of the proposed schemes by comparing them with existing schemes. Meanwhile, we examined the proposed P-RS-based beamforming design in a simple scenario, providing possibilities for its future extension.