Joint Design of a Simultaneous Reflection and Transmission RIS in Mode-Switching Mode to Assist NOMA Systems

Abstract

Simultaneous transmitting and reflecting reconfigurable intelligent surfaces (STAR-RISs) can reflect signals and transmissive signals simultaneously and can extend the coverage of signals. A conventional RIS mainly focuses on the case where the signal source and the target are on the same side. In this paper, a STAR-RIS-assisted non-orthogonal multiple access (NOMA) downlink communication system is considered to maximize the achievable rate for users by jointly optimizing the power-allocation coefficients, active beamforming, and STAR-RIS beamforming under the mode-switching (MS) protocol. The critical information of the channel is first extracted using the Uniform Manifold Approximation and Projection (UMAP) method. Based on the key extracted channel features, STAR-RIS elements and users are clustered individually using the fuzzy C-mean clustering (FCM) method. The alternating optimization method decomposes the original optimization problem into three sub-optimization problems. Finally, the sub-problems are converted to unconstrained optimization methods using penalty functions for the solution. Simulation results show that when the number of elements of RIS is 60, the achievable rate of the STAR-RIS–NOMA system is about 18% higher than that of the RIS–NOMA system.

1. Introduction

With the rapid development of modern mobile communication technologies, the capacity of communication networks will grow significantly in the next decade, and the high complexity of networks, the high cost of hardware, and the increasing consumption of resources have become the key issues facing wireless communication in the future [1]. Reconfigurable intelligent surfaces (RISs) are low-cost, programmable, easy to deploy, and low-energy, which makes RISs stand out in competition with other technologies. RISs and their variants [2,3] have become promising technologies for sixth-generation (6G) wireless networks. Simultaneous transmitting and reflecting (STAR)-RIS is a new technique in RISs; whereas a conventional RIS focuses on the case where the signal source and the target are located on the same side of the RIS [4,5,6], a STAR-RIS is capable of simultaneous transmission and reflection, and the target can be distributed on both sides of the signal source, allowing for full spatial coverage and enabling better-aided system communication [7,8,9].

Due to the significant capacity growth of future communication networks, non-orthogonal multiple access (NOMA) [10,11,12,13,14] has also received considerable attention as a promising technology to enhance spectrum efficiency and support large-scale connectivity. NOMA transmits the information of multiple users on the same sub-channel. Interference information is introduced, in contrast to orthogonal multiple access (OMA) systems, where each user has a separate resource in the time, frequency, or code domain [15]. NOMA works through sharing time-frequency resources by separating users into other domains [16]. Power-domain-based NOMA maximizes performance by assigning different power levels to users. The receiver-side detection process of power-domain NOMA highly depends on successive interference cancellation (SIC) [17]. The SIC receiver performs interference cancellation in a specific order according to different users’ signal power levels and distinguishes between other users [18]. The main advantages of NOMA are: (1) NOMA’s resource allocation method is based on a non-orthogonal, and this approach can improve spectrum efficiency. (2) NOMA uses a scheduling-free access scheme, which can better meet latency and reliability requirements. (3) NOMA can be more easily combined with other coding schemes. Combining RIS with NOMA can provide services to more users and result in higher network performance [19,20].

In the study of assisted communication, the design of the beamforming is a major concern for researchers. Ref. [21] considered joint beamforming for RIS-assisted hybrid satellite–terrestrial relay networks to minimize the total transmit power of satellites and base stations (BSs) while ensuring user rate requirements. In [22], a confidential and energy-efficient hybrid beamforming scheme for a satellite-ground integrated network was studied, and a hybrid beamformer for the base station and a digital beamformer for the satellite was jointly designed. The problem of maximizing spectral efficiency was considered in [23] for RIS-assisted single-user communication systems. In [24], the authors applied active RISs to collaboratively enhance secure transmission from the base station to the mobile user to maximize the achievable secrecy rate subject to transmit power constraints and interference thresholds. The authors in [25] used RISs to augment artificial noise to assist multiple-input multiple-output (MIMO) secure communication systems. In [26], a Compressed Sensing (CS)-based channel estimation method was proposed by exploiting the sparse representation of cascaded channels. In [27], a globally optimal solution for a single-user system’s transmit-power-minimization problem was obtained using a branch-and-bound approach. The authors of [28,29] used semidefinite relaxation and streamlined optimization, respectively, to maximize the received signal-to-noise ratio of the multiple-input single-output (MISO) communication systems for RISs. Ref. [30] solved the RIS-assisted NOMA downlink-reachability and rate-maximization problem using semi-positive definite planning (SDP). Ref. [31] derived uplink-outage probability expressions for blind RIS–NOMA using RIS as an innovative reflector and RIS as an access point (AP). The results showed that adding RIS elements enhanced the total capacity and interruption performance. Some researchers have also conducted in-depth studies on STAR-RIS-assisted communication. In [32], the authors studied STAR-RIS-assisted dual-user communication systems and proposed the problem of maximizing the total coverage of NOMA and OMA systems. In [33], the power-minimization problem for STAR-RIS-assisted unicast and multicast systems was studied for different operating protocols of STAR-RIS, and the active-passive beamforming was jointly optimized. In [34], the authors proposed the index modulation (IM)-assisted STAR-RIS–NOMA system to improve spectral efficiency. In [35], the authors considered the STAR-RIS-assisted full-duplex (FD) wireless communication system. They derived analytical closed-form expressions for the outage probability and achievable throughput for uplink and downlink communication. Ref. [36] studied the STAR-RIS-assisted NOMA downlink communication system, maximizing the signal-to-noise ratio (SNR) to design the reflected and transmitted phase-shift matrices. In [37], the power of the STAR-RIS-enabled uplink NOMA system was minimized by a joint optimization scheme. The authors of [38] investigated the energy-saving potential of STAR-RIS in a MIMO-enabled NOMA system. However, the studies mentioned above on STAR-RIS are all processed directly on the high-dimensional channel, which is challenging to solve.

Based on the above background, integrating STAR-RIS with the NOMA system is of great practical importance. The main contributions of this paper are summarized as follows.

In the STAR-RIS–NOMA system, we jointly optimize the user’s power-allocation coefficient, active beamforming at the BS, and passive beamforming at the STAR-RIS to maximize the achievable sum rate. Since the number of antennas at the BS and the number of STAR-RIS elements are large, and thus the associated channel dimension is high, we use Uniform Streaming Approximation and Projection (UMAP) to extract the key features of the channel and reduce the complexity of the solution. The elements of the STAR-RIS are clustered to improve the efficiency of the whole communication system.

We decouple the original problem into three sub-problems for solving a given decoding order. Regarding the optimization problem of the power-allocation coefficients, we use SCA with a penalty function to solve it. After adopting the SDR relaxation constraint, the block coordinate descent (BCD) method is used to solve the active beamforming at the BS and the passive beamforming at the STAR-RIS.

Simulation results show: (1) combining the STAR-RIS with NOMA can significantly improve the system’s performance, and the system performance increases with the number of STAR-RIS elements; (2) the performance of the STAR-RIS–NOMA system is better than that of the reflection/transmission RIS–NOMA-only system.

The rest of the paper is organized as follows: Section 2 presents the STAR-RIS–NOMA system’s communication model and problem representation. Section 3 performs feature extraction of the channel and clusters the users and STAR-RIS elements. A new algorithm for alternating optimization is proposed in Section 4 to solve the joint optimization problem. Numerical results are given in Section 5. Section 6 concludes the whole paper.

Notations: Scalars, vectors and matrices are represented in lowercase, bold lowercase and uppercase letters, respectively. ${ℂ}^{N_{ris}\times 1}$ is a complex vector of dimension $N_{ris}$. $diag\left(u\right)$ represents a diagonal matrix whose diagonal elements are the corresponding elements of the vector $u$. $u^{\mathrm{H}}$ denotes the conjugate transpose of the vector $u$. $Tr\left(U\right)$ and $Rank\left(U\right)$ denote the trace and rank of matrix $U$, respectively. $\mathcal{C}\mathcal{N}\left(0,{\sigma }^2\right)$ denotes a complex Gaussian distribution with mean zero and variance ${\sigma }^2$. $U\succcurlyeq 0$ denotes $U$ as a semi-positive definite matrix.

2. System Model and Problem Formulation

2.1. System Mode

As shown in Figure 1, consider the downlink transmission of the STAR-RIS–NOMA system, where the base station (BS) communicates with $M$ single-antenna users, the set of users is $\mathcal{M}\triangleq \left\{1,\cdots ,M\right\}$, and the direct link between the users and the BS is blocked. Assume that the BS is equipped with $N_t$ transmit antennas, while the STAR-RIS is equipped with $N_{ris}$ elements and the set of elements is ${\mathcal{N}}_{ris}\triangleq \left\{1,\cdots ,N_{ris}\right\}$. All perfect CSIs are available at the BS. $H\in {ℂ}^{N_{ris}\times N_t}$ is the channel from the BS to the STAR-RIS, and $g_m\in {ℂ}^{N_{ris}\times 1}$ is the channel from the STAR-RIS to the $m\text{-}\mathrm{th}$ user. We use fuzzy C-mean (FCM) clustering to group $M$ users into $L$ clusters. The set of users in the $l\text{-}\mathrm{th}$ cluster is denoted by ${\mathcal{L}}_l$. Thus, the number of users in the $l\text{-}\mathrm{th}$ cluster can be denoted by $\left|{\mathcal{L}}_l\right|$, where ${\displaystyle {\sum }_{l=1}^L\left|{\mathcal{L}}_l\right|}=M$. The detailed process on user clustering will be discussed in Section 3.

This paper considers the user-achievable sum-rate maximization problems under the mode-switching (MS) protocol. Under the MS protocol, STAR-RIS elements are divided into two groups: one working in reflection mode (R mode) and the other in transmission mode (T mode). Let $u^t={\left[\sqrt{{\beta }_1^t}{e}^{j{\varphi }_1^t},\dots ,\sqrt{{\beta }_{N_{ris}}^t}{e}^{j{\varphi }_{N_{ris}}^t}\right]}^{\mathrm{H}}$ and $u^r={\left[\sqrt{{\beta }_1^r}{e}^{j{\varphi }_1^r},\dots ,\sqrt{{\beta }_{N_{ris}}^r}{e}^{j{\varphi }_{N_{ris}}^r}\right]}^{\mathrm{H}}$, where ${\beta }_{n_{ris}}^t,{\beta }_{n_{ris}}^r\in \left\{0,1\right\}$,${\beta }_{n_{ris}}^t+{\beta }_{n_{ris}}^r=1$ and ${\varphi }_{n_{ris}}^t,{\varphi }_{n_{ris}}^r\in [0,2\pi ),\forall n_{ris}\in {\mathcal{N}}_{ris}$. The corresponding STAR-RIS transmission and reflection coefficient matrices are ${\Phi }_t=diag\left(u^t\right)$ and ${\Phi }_r=diag\left(u^r\right)$., respectively.

In the STAR-RIS–NOMA system, $v_l$ denotes the active beamforming vector of cluster $l$, and $s_{l,c}$ denotes the signal sent by the BS to the $c\text{-}\mathrm{th}$ user of the $l\text{-}\mathrm{th}$ cluster. Therefore, the received signal of the $c\text{-}\mathrm{th}$ user of the $l\text{-}\mathrm{th}$ user cluster is given by

$$\begin{array}{l}{y}_{l,c}=g_{l,c}^{\mathrm{H}}{\Phi }_{l,c}H{v}_l\sqrt{{\eta }_{l,c}}{s}_{l,c}+{\displaystyle \sum _{i\ne c,i\in {\mathcal{L}}_l}{g}_{l,c}^{\mathrm{H}}{\Phi }_{l,c}H{v}_l\sqrt{{\eta }_{l,i}}{s}_{l,i}}\\ +{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{\displaystyle \sum _{i\in {\mathcal{L}}_j}{g}_{l,c}^{\mathrm{H}}{\Phi }_{l,c}H{v}_j\sqrt{{\eta }_{j,i}}{s}_{j,i}}}+n_{l,c},\end{array}$$

(1)

where ${\eta }_{l,c}$ denotes the power-allocation coefficient of the $c\text{-}\mathrm{th}$ user of the $l\text{-}\mathrm{th}$ user -cluster, $n_{l,c}$ represents the additive Gaussian white noise, and $n_{l,c}\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }^2\right)$, ${\sigma }^2$ is the noise power. Due to the presence of multiple users, each user uses the SIC sequentially to eliminate interference. This paper considers the user-achievable sum-rate maximization problem in the case of fixed decoding order. For simplicity and without loss of generality, we assume that the equivalent channel gain order in the $l\text{-}\mathrm{th}$ user cluster is

$${\left|g_{l,1}^{\mathrm{H}}{\Phi }_{l,1}H{v}_l\right|}^2\ge {\left|g_{l,2}^{\mathrm{H}}{\Phi }_{l,2}H{v}_l\right|}^2\cdots \ge {\left|g_{l,\left|{\mathcal{L}}_l\right|}^{\mathrm{H}}{\Phi }_{l,\left|{\mathcal{L}}_l\right|}H{v}_l\right|}^2.$$

(2)

Thus, the achievable rate for the $c\text{-}\mathrm{th}$ user of the $l\text{-}\mathrm{th}$ user cluster can be given as

$$R_{l,c}={\log }_2\left(1+\frac{{\left|g_{l,c}^{\mathrm{H}}{\Phi }_{l,c}H{v}_l\right|}^2{\eta }_{l,c}}{{\left|g_{l,c}^{\mathrm{H}}{\Phi }_{l,c}H{v}_l\right|}^2{\displaystyle \sum _{i<c}{\eta }_{l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{\displaystyle \sum _{i\in {\mathcal{L}}_j}{\left|g_{l,c}^{\mathrm{H}}{\Phi }_{l,c}H{v}_j\right|}^2{\eta }_{l,i}}}+{\sigma }^2}}\right).$$

(3)

2.2. Problem Formulation

To improve the overall data rate, the joint power-allocation coefficients, active beamforming, and transmission and reflection beamforming optimization problems at the RIS are proposed to maximize the achievable rate for users. The problem formulations are

$$\underset{{\eta }_{l,c},v_l,u^o}{\max }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{R}_{l,c}}}$$

(4a)

$$\mathrm{s}.\mathrm{t}.R_{l,c}\ge R_{{}_{l,c}}^{\min },$$

(4b)

$${\displaystyle \sum _{l\in \mathcal{L}}{‖v_{{}_l}‖}^2\le P_{\max }},$$

(4c)

$${\varphi }_{n_{ris}}^o\in [0,2\pi ),\forall n_{ris}\in {\mathcal{N}}_{ris},$$

(4d)

$${\beta }_{n_{ris}}^o\in \left\{0,1\right\},{\beta }_{n_{ris}}^t+{\beta }_{n_{ris}}^r=1,$$

(4e)

where $o\in \{t,r\}$. Constraint (4b) guarantees a minimum quality of service (QoS) requirement for each user, and constraint (4c) indicates a total transmit power budget of $P_{\max }$. Since the objective function is nonconvex, the constraint (4b)–(4d) is also nonconvex. Therefore, problem (4) is a nonconvex optimization problem, which is more difficult to solve directly. Compared with reflection-only RIS, the STAR-RIS has a more complex optimization problem because of the presence of both reflection and transmission modes.

3. Channel Feature Extraction and Clustering of Users and STAR-RIS Elements

The number of transmitting antennas arranged at the BS is usually high. The number of STAR-RIS elements is significant, resulting in a high dimensionality of the relevant channels, which is challenging to optimize directly. The Uniform Manifold Approximation and Projection (UMAP) [39] can identify the critical structures in the high-dimensional channel space and preserve these structures in the low-dimensional embedding. Therefore, we first process the channel from the BS to the STAR-RIS and the STAR-RIS to the user using UMAP to extract the critical information and then perform clustering for STAR-RIS elements and system users, respectively.

3.1. Channel Feature Extraction Using UMAP

According to the three assumptions of UMAP about the data, a manifold with a fuzzy topology can be modeled. The embedding is found by searching the low-dimensional projection of the data with the closest equivalent fuzzy topology. The process of channel feature extraction using UMAP is shown in Figure 2.

The STAR-RIS-user channel matrix is $G=[g_1,\dots ,g_M]$. To extract the features of the STAR-RIS-user channel, the first step is to establish the relationship between the high-dimensional samples. In the STAR-RIS-user channel matrix $G$, the probability-density function between each point can be given as

$$p_{i|j}^{(G)}=e^{-\frac{d(g_i-g_j)-{\rho }_i^{(G)}}{{\theta }_i^{(G)}}},$$

(5)

where $d\left(\cdot \right)$ denotes the Euclidean distance between sample points and ${\rho }_i^{(G)}$ is the distance of the nearest point to point $i$. To ensure the symmetry of the distance, the joint probability is established as

$$p_{ij}^{(G)}=p_{i|j}^{(G)}+p_{j|i}^{(G)}-p_{i|j}^{(G)}{p}_{j|i}^{(G)}.$$

(6)

Fix the $R^{(G)}$ nearest neighbors of the point $i$. Solve for ${\theta }_i^{(G)}$ according to $R^{(G)}=2^{{\displaystyle \sum _i{p}_{ij}^{(G)}}}$. The relationship between the low-dimensional points can be established by Equation (7).

$$q_{ij}^{(G)}={(1+a^{(G)}{({\tilde{g}}_i-{\tilde{g}}_j)}^{b^{(G)}})}^{-1},$$

(7)

where $a^{(G)}$, $b^{(G)}$ is the hyperparameter and ${\tilde{g}}_i$ is the downscaled channel. In order to describe the relationship between the high-dimensional sample points and the reduced-dimensional sample points, the cross-entropy-loss function is constructed as

$$CE(g_{ij},{\tilde{g}}_{ij})={\displaystyle \sum _i{\displaystyle \sum _j\left[p_{ij}^{(G)}(g_{ij})\log (\frac{p_{ij}^{(G)}(g_{ij})}{q_{ij}^{(G)}({\tilde{g}}_{ij})})+(1-p_{ij}^{(G)}(g_{ij}))\log (\frac{1-p_{ij}^{(G)}(g_{ij})}{1-q_{ij}^{(G)}({\tilde{g}}_{ij})})\right]}},$$

(8)

where $g_{ij}$ denotes the distance between the high-dimensional sample points and ${\tilde{g}}_{ij}$ denotes the distance between the low-dimensional sample points. Therefore, the low-dimensional space ${\tilde{g}}_i$, corresponding to $g_i$ in the high-dimensional space, can be obtained by solving the following problem:

$$\min CE(g_{ij},{\tilde{g}}_{ij})$$

(9)

We can use (10) to solve problem (9) using a stochastic gradient descent (SGD) update to obtain the reduced channel matrix $\tilde{G}=\left[{\tilde{g}}_1,\dots ,{\tilde{g}}_M\right]$.

$${\tilde{g}}_{ij}^{t+1}={\tilde{g}}_{ij}^t-{\alpha }_1{\nabla }_{\tilde{g}}{q}_{ij}^{(G)}({\tilde{g}}_{ij}),$$

(10)

where ${\alpha }_1$ is the step size.

The channel from the BS to the STAR-RIS is $H={\left[h_1,\dots ,h_{N_{ris}}\right]}^{\mathrm{H}}$. First, it is necessary to establish the relationship between the high-dimensional samples. The probability-density function between each point in the BS to the STAR-RIS channel matrix $H$ can be given as

$$p_{i|j}^{(H)}=e^{-\frac{d(h_i-h_j)-{\rho }_i^{(H)}}{{\theta }_i^{(H)}}},$$

(11)

where $d\left(\cdot \right)$ denotes the Euclidean distance between sample points, and ${\rho }_i^{(H)}$ is fixed as the distance of the nearest point to point $i$. To ensure the symmetry of the distance, the joint probability is established:

$$p_{ij}^{(H)}=p_{i|j}^{(H)}+p_{j|i}^{(H)}-p_{i|j}^{(H)}{p}_{j|i}^{(H)}.$$

(12)

Fix the $R^{(H)}$ nearest neighbors of the point $i$. Solve for ${\theta }_i^{(H)}$ according to $R^{(H)}=2^{{\displaystyle \sum _i{p}_{ij}^{(H)}}}$. The relationship between the low-dimensional points can be established by (13).

$$q_{ij}^{(H)}={(1+a^{(H)}{({\tilde{h}}_i-{\tilde{h}}_j)}^{b^{(H)}})}^{-1},$$

(13)

where $a^{(H)}$, $b^{(H)}$ is the hyperparameter and ${\tilde{h}}_i$ is the downscaled channel. To describe the relationship between the high-dimensional sample points and the downscaled low-dimensional sample points, a cross-entropy-loss function is constructed:

$$CE(h_{ij},{\tilde{h}}_{ij})={\displaystyle \sum _i{\displaystyle \sum _j\left[p_{ij}^{(H)}(h_{ij})\log (\frac{p_{ij}^{(H)}(h_{ij})}{q_{ij}^{(H)}({\tilde{h}}_{ij})})+(1-p_{ij}^{(H)}(h_{ij}))\log (\frac{1-p_{ij}^{(H)}(h_{ij})}{1-q_{ij}^{(H)}({\tilde{h}}_{ij})})\right]}},$$

(14)

where $h_{ij}$ denotes the distance between the high-dimensional sample points and ${\tilde{h}}_{ij}$ denotes the distance between the low-dimensional sample points. Therefore, $h_i$ in the high-dimensional space corresponds to the low-dimensional space ${\tilde{h}}_i$, which can be obtained by solving the following problem:

$$\min CE(h_{ij},{\tilde{h}}_{ij})$$

(15)

Problem (15) is solved by SGD using (16) to obtain the channel matrix $\tilde{H}={\left[{\tilde{h}}_1,\dots ,{\tilde{h}}_{N_{ris}}\right]}^{{\rm T}}$ of the reduced BS to the STAR-RIS.

$${\tilde{h}}_{ij}^{t+1}={\tilde{h}}_{ij}^t-{\alpha }_2{\nabla }_{\tilde{h}}{q}_{ij}^{(H)}({\tilde{h}}_{ij}),$$

(16)

where ${\alpha }_2$ is the step size.

3.2. User Clustering

After extracting the STAR-RIS-user channel features using UMAP and then using FCM-clustering to subdivide the user clusters, it is possible to classify them more accurately with less complexity. The process of clustering users by FCM is shown in Figure 3. Using FCM, we divide the users into $L$ clusters and the sub-cluster set into $\mathcal{L}=\left\{1,\dots ,L\right\}$. The users are thus classified by the optimization problem (17).

$$\min J_{FCM}^{(user)}\left(Z,K\right)={\displaystyle \sum _{j=1}^L{\displaystyle \sum _{i=1}^M{z}_{ji}^{{\chi }^{(user)}}}{‖{\tilde{g}}_i-k_j‖}^2}$$

(17a)

$$\mathrm{s}.\mathrm{t}.{\displaystyle \sum _{j=1}^L{z}_{ji}=1},$$

(17b)

where ${\chi }^{(user)}$ is an affiliation factor showing the importance of whether a sample belongs to a class or not, $Z$ is the affiliation matrix $z_{ji}^{{\chi }^{(user)}}$ indicating the affiliation of sample $i$ to class $j$, and $K=\left[k_1,\dots ,k_L\right]$ is the cluster center. The constraint (17b) is combined into the objective function using the Lagrange multiplier method:

$$J_{FCM}^{(user)}\left(Z,K,{\vartheta }_m^{(user)}\right)={\displaystyle \sum _{j=1}^L{\displaystyle \sum _{i=1}^M{z}_{ji}^{{\chi }^{(user)}}}{‖{\tilde{g}}_i-k_j‖}^2}+{\displaystyle \sum _{m=1}^M{\vartheta }_m^{(user)}\left({\displaystyle \sum _{j=1}^L{z}_{jm}-1}\right)},$$

(18)

where ${\vartheta }_m^{(user)}$ is the Lagrange multiplier. Taking the derivative of (18) with respect to $z_{ji}$ and $k_j$, respectively, yields (19) and (20).

$$\frac{\partial J}{\partial z_{ji}}={\chi }^{(user)}{‖{\tilde{g}}_i-k_j‖}^2{z}_{ji}^{{\chi }^{(user)}-1}+{\vartheta }_i^{(user)}=0.$$

(19)

$$\frac{\partial J}{\partial k_j}={\displaystyle \sum _{i=1}^M-2z_{ji}^{{\chi }^{(user)}}}\left({\tilde{g}}_i-k_j\right)=0.$$

(20)

Solving Equations (19) and (20), we can obtain

$$z_{ji}={\left({\displaystyle \sum _{r=1}^L\left(\frac{{‖{\tilde{g}}_i-k_j‖}^{\left(\frac{2}{{\chi }^{(user)}-1}\right)}}{{‖{\tilde{g}}_i-k_r‖}^{\left(\frac{2}{{\chi }^{(user)}-1}\right)}}\right)}\right)}^{-1}.$$

(21)

$$k_j=\frac{{\displaystyle \sum _{i=1}^M\left(z_{ji}^{{\chi }^{(user)}}{\tilde{g}}_i\right)}}{{\displaystyle \sum _{i=1}^M{z}_{ji}^{{\chi }^{(user)}}}}.$$

(22)

Equations (21) and (22) are iterated and, finally, the users are divided into $L$ clusters. The set of users in the $l\text{-}\mathrm{th}$ user cluster is ${\mathcal{L}}_l$, and the number of users is $\left|{\mathcal{L}}_l\right|$.

3.3. STAR-RIS Element Clustering

Let $X={\tilde{G}}^H{\tilde{H}}^H=\left[x_1,\dots ,x_{N_{ris}}\right]$; then, also using FCM, divide the elements of the STAR-RIS into $\left|\gamma \right|$ clusters and the set of sub-clusters into $\gamma =\left\{1,\dots ,\left|\gamma \right|\right\}$. The process of FCM-clustering of STAR-RIS elements is shown in Figure 4, and the classification results of STAR-RIS elements are obtained by the optimization problem (23):

$$\min J_{FCM}^{(ris)}\left(A,D\right)={\displaystyle \sum _{j=1}^{\left|\gamma \right|}{\displaystyle \sum _{i=1}^{N_{ris}}{a}_{ji}^{{\chi }^{(ris)}}}{‖{\tilde{x}}_i-d_j‖}^2}$$

(23a)

$$\mathrm{s}.\mathrm{t}.{\displaystyle \sum _{j=1}^{\left|\gamma \right|}{a}_{ji}=1},$$

(23b)

where ${\chi }^{(ris)}$ is an affiliation factor showing the importance of whether a sample belongs to a class or not, $A$ is the affiliation matrix, $a_{ji}^{{\chi }^{(ris)}}$ indicates the affiliation of sample $i$ belonging to class $j$, and $D=\left[d_1,\dots ,d_{L_{ris}}\right]$ is the cluster center. The constraint (23b) is combined into the objective function using the Lagrange multiplier method:

$$J_{FCM}^{(ris)}\left(A,D,{\vartheta }_m^{(ris)}\right)={\displaystyle \sum _{j=1}^{\left|\gamma \right|}{\displaystyle \sum _{i=1}^{N_{ris}}{a}_{ji}^{{\chi }^{(ris)}}}{‖{\tilde{x}}_i-d_j‖}^2}+{\displaystyle \sum _{n=1}^{N_{ris}}{\vartheta }_n^{(ris)}\left({\displaystyle \sum _{j=1}^{\left|\gamma \right|}{a}_{jn}-1}\right)},$$

(24)

where ${\vartheta }_n^{(ris)}$ is the Lagrange multiplier. Equation (24) can be obtained by deriving Equations (25) and (26) for $a_{ji}$ and $d_j$ respectively.

$$\frac{\partial J}{\partial a_{ji}}={\chi }^{(ris)}{‖{\tilde{x}}_i-d_j‖}^2{z}_{ji}^{{\chi }^{(ris)}-1}+{\vartheta }_i^{(ris)}=0.$$

(25)

$$\frac{\partial J}{\partial d_j}={\displaystyle \sum _{i=1}^{N_{ris}}-2a_{ji}^{{\chi }^{(ris)}}}\left({\tilde{x}}_i-d_j\right)=0.$$

(26)

Solving Equations (25) and (26), we obtain

$$a_{ji}={\left({\displaystyle \sum _{r=1}^{\left|\gamma \right|}\left(\frac{{‖{\tilde{x}}_i-d_j‖}^{\left(\frac{2}{{\chi }^{(ris)}-1}\right)}}{{‖{\tilde{x}}_i-d_r‖}^{\left(\frac{2}{{\chi }^{(ris)}-1}\right)}}\right)}\right)}^{-1}.$$

(27)

$$d_j=\frac{{\displaystyle \sum _{i=1}^{N_{ris}}\left(a_{ji}^{{\chi }^{(ris)}}{\tilde{x}}_i\right)}}{{\displaystyle \sum _{i=1}^{N_{ris}}{a}_{ji}^{{\chi }^{(ris)}}}}.$$

(28)

Equations (27) and (28) are iterated and, finally, the $N_{ris}$ elements of the STAR-RIS are divided into $\left|\gamma \right|$ clusters. The set of elements in the $r\text{-}\mathrm{th}$ RIS component cluster is ${\gamma }_r$, and the number of elements is $\left|{\gamma }_r\right|$.

After the channel feature extraction and the clustering operation, the received signal of the $c\text{-}\mathrm{th}$ user of the $l\text{-}\mathrm{th}$ user cluster transmitted/reflected through the $r\text{-}\mathrm{th}$ STAR-RIS element cluster is

$$\begin{array}{l}{y}_{r,l,c}={\tilde{g}}_{r,l,c}^{\mathrm{H}}{\Phi }_{r,l,c}{\tilde{H}}_r{v}_{r,l}\sqrt{{\eta }_{r,l,c}}{s}_{r,l,c}+{\displaystyle \sum _{i\ne c,i\in {\mathcal{L}}_l}{\tilde{g}}_{r,l,c}^{\mathrm{H}}{\Phi }_{r,l,c}{\tilde{H}}_r{v}_{r,l}\sqrt{{\eta }_{r,l,i}}{s}_{r,l,i}}\\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{\displaystyle \sum _{i\in {\mathcal{L}}_j}{\tilde{g}}_{r,l,c}^{\mathrm{H}}{\Phi }_{r,l,c}{\tilde{H}}_r{v}_{r,j}\sqrt{{\eta }_{r,j,i}}{s}_{r,j,i}}}+n_{r,l,c}.\end{array}$$

(29)

We assume that the power-allocation coefficients satisfy ${\displaystyle \sum _{c\in {\mathcal{L}}_l}{\eta }_{r,l,c}}=1$. Thus, the achievable rate of the $c\text{-}\mathrm{th}$ user of the $l\text{-}\mathrm{th}$ user cluster transmitted/reflected through the $r\text{-}\mathrm{th}$ STAR-RIS cluster elements can be given as Equation (30).

$$R_{r,l,c}={\log }_2\left(1+\frac{{\left|{\tilde{g}}_{r,l,c}^{\mathrm{H}}{\Phi }_{r,l,c}{\tilde{H}}_r{v}_{r,l}\right|}^2{\eta }_{r,l,c}}{{\left|{\tilde{g}}_{r,l,c}^{\mathrm{H}}{\Phi }_{r,l,c}{\tilde{H}}_r{v}_{r,l}\right|}^2{\displaystyle \sum _{i<c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,l\in \mathcal{L}}{\left|{\tilde{g}}_{r,l,c}^{\mathrm{H}}{\Phi }_{r,l,c}{\tilde{H}}_r{v}_{r,j}\right|}^2+{\sigma }^2}}}\right).$$

(30)

After clustering the STAR-RIS elements and users separately, the STAR-RIS element clusters correspond to the user clusters, in which case ${\beta }_{n_{ris}}^t$ and ${\beta }_{n_{ris}}^r$ are determined. Therefore, to maximize the achievable sum rate of $M$ users, problem (4) is reformulated as

$$\underset{{\eta }_{r,l,c},v_{r,l},u^o}{\max }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{R}_{r,l,c}}}$$

(31a)

$$\mathrm{s}.\mathrm{t}.R_{{}_{r,l,c}}\ge R_{{}_{r,l,c}}^{\min },$$

(31b)

$${\displaystyle \sum _{l\in \mathcal{L}}{‖v_{{}_{r,l}}‖}^2\le P_{\max }},$$

(31c)

$${\varphi }_{n_{ris}}^t,{\varphi }_{n_{ris}}^r\in [0,2\pi ),\forall n_{ris}\in {\mathcal{N}}_{ris},$$

(31d)

$${\displaystyle \sum _{c\in {\mathcal{L}}_l}{\eta }_{r,l,c}}=1,$$

(31e)

where $o\in \{t,r\}$. Let $U_{r.l.c}=u_{r,l,c}{u}_{r,l,c}^{\mathrm{H}}$,$b_{r,l,c}=diag\left({\tilde{g}}_{r,l,c}^{\mathrm{H}}\right){\tilde{H}}_r$, $B_{r.l.c}=b_{r,l,c}{b}_{r,l,c}^{\mathrm{H}}$ and $V_{r,l}=v_{r,l}{v}_{r,l}^{\mathrm{H}}$, then, the achievable rate is

$$R_{r,l,c}={\log }_2\left(1+\frac{{\eta }_{r,l,c}Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,l}\right)}{Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,l}\right){\displaystyle \sum _{i<c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,j}\right)}}+{\sigma }^2}\right).$$

(32)

Problem (31) can be reformulated as

$$\underset{{\eta }_{r,l,c},V_{r,l},U_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)}}$$

(33a)

$$\mathrm{s}.\mathrm{t}.f_3\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)\le 0,$$

(33b)

$${\displaystyle \sum _{l\in \mathcal{L}}Tr\left(V_{r,l}\right)\le P_{\max }},$$

(33c)

$$Rank\left(V_{r,l}\right)=1,$$

(33d)

$$Rank\left(U_{r,l,c}\right)=1,$$

(33e)

$$V_{r,l}\succcurlyeq 0,$$

(33f)

$$U_{r,l,c}\succcurlyeq 0,$$

(33g)

$$\left(31\mathrm{e}\right),$$

(33h)

where $f_1\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)$ is as in (34), $f_2\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)$ is as in Equation (35), and $f_3\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)$ is as in Equation (36). Problem (33) is still nonconvex, and we use SDR to relax the rank-one constraint and then use a first-order Taylor expansion to approximate the nonconvex part of the objective function with the constraint present.

$$f_1\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)={\log }_2\left(Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,l}\right){\displaystyle \sum _{i<c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,j}\right)}}+{\sigma }^2\right).$$

(34)

$$f_2\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)=-{\log }_2\left(Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,l}\right){\displaystyle \sum _{i\le c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,j}\right)}}+{\sigma }^2\right).$$

(35)

$$\begin{array}{l}{f}_3\left({\eta }_{r,l,c},U_{r,l,c},V_{r,l}\right)=\left(Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,l}\right){\displaystyle \sum _{i<c}{\eta }_{r,l,i}}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,j}\right)}+{\sigma }^2\right)\left(2^{R_{{}_{r,l,c}}^{\min }}-1\right).\\ -{\eta }_{r,l,c}Tr\left(U_{r,l,c}{B}_{r.l.c}{V}_{r,l}\right)\end{array}$$

(36)

4. Solution of the Problem

In this section, we propose a new alternating optimization iterative algorithm to solve the joint optimization problem of power-allocation coefficients, active beamforming, and STAR-RIS passive beamforming.

4.1. Power-Allocation Coefficient Optimization

When fixing the STAR-RIS coefficients and the BS beamforming vectors, problem (33) can be reformulated as

$$\underset{{\eta }_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left({\eta }_{r,l,c}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left({\eta }_{r,l,c}\right)}}$$

(37a)

$$\mathrm{s}.\mathrm{t}.f_3\left({\eta }_{r,l,c}\right)\le 0,$$

(37b)

$$\left(31\mathrm{e}\right).$$

(37c)

At this point, the constraint is convex, and the first-order Taylor expansion of $f_1\left({\eta }_{r,l,c}\right)$ is

$$f_1\left({\eta }_{r,l,c}\right)\ge f_1\left({\eta }_{r,l,c}^{\left(\tau \right)}\right)+Tr\left({\left({\nabla }_{\eta }{f}_1\left({\eta }_{r,l,c}^{\left(\tau \right)}\right)\right)}^{\mathrm{H}}\left({\eta }_{r,l,c}-{\eta }_{r,l,c}^{\left(\tau \right)}\right)\right)\triangleq f_1\left({\eta }_{r,l,c},{\eta }_{r,l,c}^{\left(\tau \right)}\right),$$

(38)

where ${\eta }_{r,l,c}^{\left(\tau \right)}$ denotes the value of ${\eta }_{r,l,c}$ at the $\tau \text{-}\mathrm{th}$ iteration and, according to the SCA principle, Equation (37a) can be replaced by

$$\underset{{\eta }_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left({\eta }_{r,l,c},{\eta }_{r,l,c}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left({\eta }_{r,l,c}\right)}}$$

(39)

Using the penalty-function method, problem (37) can be transformed into an unconstrained optimization problem:

$$\underset{{\eta }_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left({\eta }_{r,l,c},{\eta }_{r,l,c}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left({\eta }_{r,l,c}\right)}}+{\zeta }_1\left({\displaystyle \sum _{c\in {\mathcal{L}}_l}{\eta }_{r,l,c}^2}-1+\max {\left\{0,f_3\left({\eta }_{r,l,c}\right)\right\}}^2\right)$$

(40)

where ${\zeta }_1$ is the penalty factor. Thus, the gradient-descent method can be used to solve for ${\eta }_{r,l,c}$ by using Equation (41).

$$\begin{array}{l}{\eta }_{r,l,c}^{(\tau +1)}={\eta }_{r,l,c}^{(\tau )}-{\alpha }_3\left({\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\nabla }_{\eta }{f}_1^{\left(\tau \right)}\left({\eta }_{r,l,c},{\eta }_{r,l,c}^{\left(\tau -1\right)}\right)}}\right)+{\alpha }_3\left({\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\nabla }_{\eta }{f}_2^{\left(\tau \right)}\left({\eta }_{r,l,c}\right)}}\right)\\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\alpha }_3{\zeta }_1\left(2{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\eta }_{r,l,c}^{\left(\tau \right)}}+{\nabla }_{\eta }^{\left(\tau \right)}\left(\max {\left\{0,f_3\left({\eta }_{r,l,c}\right)\right\}}^2\right)\right),\end{array}$$

(41)

where ${\alpha }_3$ is the step size.

4.2. Active Beamforming Optimization

Fixing the STAR-RIS coefficients and power-allocation coefficients, problem (33) can be formulated as

$$\underset{V_{r,l}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(V_{r,l}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(V_{r,l}\right)}}$$

(42a)

$$\mathrm{s}.\mathrm{t}.f_3\left(V_{r,l}\right)\le 0,$$

(42b)

$$(33\mathrm{c}),(33\mathrm{d}),(33\mathrm{f}).$$

(42c)

Using the SDR relaxation rank-one constraint (33d), the first-order Taylor expansion of $f_1\left(V_{r,l}\right)$ is

$$f_1\left(V_{r,l}\right)\ge f_1\left(V_{r,l}^{\left(\tau \right)}\right)+Tr\left({\left({\nabla }_V{f}_1\left(V_{r,l}^{\left(\tau \right)}\right)\right)}^{\mathrm{H}}\left(V_{r,l}-V_{r,l}^{\left(\tau \right)}\right)\right)\triangleq f_1\left(V_{r,l},V_{r,l}^{\left(\tau \right)}\right),$$

(43)

where $V_{r,l}^{\left(\tau \right)}$ denotes the value of $V_{r,l}$ at the $\tau \text{-}\mathrm{th}$ iteration. Thus, the objective function (42a) can be converted to

$$\underset{V_{r,l}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(V_{r,l},V_{r,l}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(V_{r,l}\right)}}$$

(44)

After removing the constraints (42b) and (33c) using the penalty function, problem (42) can be converted to

$$\underset{V_{r,l}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(V_{r,l},V_{r,l}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(V_{r,l}\right)}}+{\zeta }_2\left({\left[\max \left\{0,f_3\left(V_{r,l}\right),{\displaystyle \sum _{l\in \mathcal{L}}Tr\left(V_{r,l}\right)-P_{\max }}\right\}\right]}^2\right)$$

(45a)

$$\mathrm{s}.\mathrm{t}.\left(33\mathrm{f}\right),$$

(45b)

Problem (45) is a semidefinite programming problem (SDP), which is solved by using the BCD [40] method. First, problem (45) is converted into an unconstrained optimization problem:

$$\begin{array}{l}\underset{V_{r,l}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(V_{r,l},V_{r,l}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(V_{r,l}\right)}}\\ +{\zeta }_2\left({\left[\max \left\{0,f_3\left(V_{r,l}\right),{\displaystyle \sum _{l\in \mathcal{L}}Tr\left(V_{r,l}\right)-P_{\max }}\right\}\right]}^2\right)-{\varpi }_1\log \det \left(V_{r,l}\right)\end{array}$$

(46)

where ${\varpi }_1>0$ is the barrier factor. The key to applying the coordinate-descent method to the problem of (46) is the following block determinant:

$$\det \left(V_{r,l}\right)=\det \left(S\right)\det \left(\stackrel{⌢}{y}-{\stackrel{⌢}{x}}^{\mathrm{T}}{S}^{-1}\stackrel{⌢}{x}\right),$$

(47)

where $V_{r,l}=\left(\begin{array}{cc}S& \stackrel{⌢}{x}\\ {\stackrel{⌢}{x}}^{\mathrm{T}}& \stackrel{⌢}{y}\end{array}\right)$, and therefore, problem (46) can be transformed by solving

$$\begin{array}{l}\underset{\stackrel{⌢}{x}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\stackrel{⌢}{f}}_1\left(\stackrel{⌢}{x}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\stackrel{⌢}{f}}_2\left(\stackrel{⌢}{x}\right)}}\\ +{\zeta }_2\left({\left[\max \left\{0,{\stackrel{⌢}{f}}_3\left(\stackrel{⌢}{x}\right),{\displaystyle \sum _{l\in \mathcal{L}}{C}_l^{\mathrm{T}}\stackrel{⌢}{x}-P_{\max }}\right\}\right]}^2\right)-{\varpi }_1\log \left(\stackrel{⌢}{y}-{\stackrel{⌢}{x}}^{\mathrm{T}}{S}^{-1}\stackrel{⌢}{x}\right)\end{array}$$

(48)

where ${\stackrel{⌢}{f}}_1\left(\stackrel{⌢}{x}\right)$ is (49), ${\stackrel{⌢}{f}}_2\left(\stackrel{⌢}{x}\right)$ is (50), and ${\stackrel{⌢}{f}}_3\left(\stackrel{⌢}{x}\right)$ is Equation (51).

$${\stackrel{⌢}{f}}_1\left(\stackrel{⌢}{x}\right)={\log }_2\left(w_{l,c}^{\mathrm{T}}\stackrel{⌢}{x}{\displaystyle \sum _{i<c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{w}_{l,j}^{\mathrm{T}}\stackrel{⌢}{x}}}+{\sigma }^2\right).$$

(49)

$${\stackrel{⌢}{f}}_2\left(\stackrel{⌢}{x}\right)=-{\log }_2\left(w_{l,c}^{\mathrm{T}}\stackrel{⌢}{x}{\displaystyle \sum _{i\le c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{w}_{l,j}^{\mathrm{T}}\stackrel{⌢}{x}}}+{\sigma }^2\right).$$

(50)

$${\stackrel{⌢}{f}}_3\left(\stackrel{⌢}{x}\right)=\left(w_{l,c}^{\mathrm{T}}\stackrel{⌢}{x}{\displaystyle \sum _{i<c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{w}_{l,j}^{\mathrm{T}}\stackrel{⌢}{x}}}+{\sigma }^2\right)R_{{}_{r,l,c}}^{MIN}-{\eta }_{r,l,c}{w}_{l,c}^{\mathrm{T}}\stackrel{⌢}{x},$$

(51)

where $R_{{}_{r,l,c}}^{MIN}=2^{R_{{}_{r,l,c}}^{\min }}-1$.

The gradient-descent method is used to solve for $\stackrel{⌢}{x}$ according to Equation (52).

$$\begin{gathered} \begin{array}{l}{\stackrel{⌢}{x}}^{\left(\tau +1\right)}={\stackrel{⌢}{x}}^{\left(\tau \right)}-{\alpha }_4\left(\begin{array}{l}{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\nabla }_{\stackrel{⌢}{x}}{\stackrel{⌢}{f}}_1^{\left(\tau \right)}\left(\stackrel{⌢}{x},{\stackrel{⌢}{x}}^{\left(\tau -1\right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\nabla }_{\stackrel{⌢}{x}}{\stackrel{⌢}{f}}_2^{\left(\tau \right)}\left(\stackrel{⌢}{x}\right)}}\\ \end{array}\right)\\ \end{array} \\ \begin{array}{l}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\alpha }_4{\zeta }_2\left({\nabla }_{\stackrel{⌢}{x}}^{\left(\tau \right)}{\left[\max \left\{0,{\stackrel{⌢}{f}}_3\left(\stackrel{⌢}{x}\right),{\displaystyle \sum _{l\in \mathcal{L}}{w}_l^{\mathrm{T}}\stackrel{⌢}{x}-P_{\max }}\right\}\right]}^2\right)+{\alpha }_4\frac{2{\varpi }_1}{\stackrel{⌢}{y}-{\stackrel{⌢}{x}}^T{S}^{-1}\stackrel{⌢}{x}}{S}^{-1}\stackrel{⌢}{x},\end{array} \end{gathered}$$

(52)

where ${\alpha }_4$ is the step size.

4.3. STAR-RIS Coefficient Optimization

Fixing the power-allocation coefficients and BS beamforming vectors, problem (33) can be formulated as

$$\underset{U_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(U_{r,l,c}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(U_{r,l,c}\right)}}$$

(53a)

$$\mathrm{s}.\mathrm{t}.f_3\left(U_{r,l,c}\right)\le 0,$$

(53b)

$$\left(33\mathrm{e}\right),\left(33\mathrm{g}\right).$$

(53c)

Regarding the relaxation of constraint (33e) using SDR, the first-order Taylor expansion of $f_1\left(U_{r,l,c}\right)$ is

$$f_1\left(U_{r,l,c}\right)\ge f_1\left(U_{r,l,c}^{\left(\tau \right)}\right)+Tr\left(\left({\left({\nabla }_U{f}_1\left(U_{r,l,c}^{\left(\tau \right)}\right)\right)}^{\mathrm{H}}\left(U_{r,l,c}-U_{r,l,c}^{\left(\tau \right)}\right)\right)\right)\triangleq f_1\left(U_{r,l,c},U_{r,l,c}^{\left(\tau \right)}\right),$$

(54)

where $U_{r,l,c}^{\left(\tau \right)}$ denotes the value of $U_{r,l,c}$ at the $\tau \text{-}\mathrm{th}$ iteration. The objective function (53a) can be re-expressed as

$$\underset{U_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(U_{r,l,c},U_{r,l,c}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(U_{r,l,c}\right)}}$$

(55)

Convert problem (53) to problem (56) using the penalty function.

$$\underset{U_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(U_{r,l,c},U_{r,l,c}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(U_{r,l,c}\right)}}+{\zeta }_3\left({\left[\max \left\{0,f_3\left(U_{r,l,c}\right)\right\}\right]}^2\right)$$

(56a)

$$\mathrm{s}.\mathrm{t}.\left(33\mathrm{g}\right),$$

(56b)

where ${\zeta }_3$ is the penalty factor. Problem (56) is an SDP problem, and the solution using the BCD method can convert (56) into an unconstrained problem.

$$\begin{array}{l}\underset{U_{r,l,c}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_1\left(U_{r,l,c},U_{r,l,c}^{\left(\tau \right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{f}_2\left(U_{r,l,c}\right)}}\\ \hspace{1em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\zeta }_3\left({\left[\max \left\{0,f_3\left(U_{r,l,c}\right)\right\}\right]}^2\right)-{\varpi }_2\log \det \left(U_{r,l,c}\right)\end{array}$$

(57)

where ${\varpi }_2>0$ is the barrier factor. The key to applying the coordinate-descent method to the problem of (57) is the following block determinant:

$$\det \left(U_{r,l,c}\right)=\det \left(O\right)\det \left(\widehat{y}-{\widehat{x}}^{\mathrm{T}}{O}^{-1}\widehat{x}\right),$$

(58)

where $U_{r,l,c}=\left(\begin{array}{cc}O& \widehat{x}\\ {\widehat{x}}^{\mathrm{T}}& \widehat{y}\end{array}\right)$; therefore, problem (57) can be transformed by solving

$$\underset{\widehat{x}}{\min }{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\widehat{f}}_1\left(\widehat{x}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\widehat{f}}_2\left(\widehat{x}\right)}}+{\zeta }_3\left({\left[\max \left\{0,{\widehat{f}}_3\left(\widehat{x}\right)\right\}\right]}^2\right)-{\varpi }_2\log \left(\widehat{y}-{\widehat{x}}^{\mathrm{T}}{O}^{-1}\widehat{x}\right)$$

(59)

where ${\widehat{f}}_1\left(\widehat{x}\right)$ is shown in Equation (60), ${\widehat{f}}_2\left(\widehat{x}\right)$ is shown in (61), and ${\widehat{f}}_3\left(\widehat{x}\right)$ is shown in Equation (62).

$${\widehat{f}}_1\left(\widehat{x}\right)={\log }_2\left({\widehat{w}}_{l,c}^{\mathrm{T}}\widehat{x}{\displaystyle \sum _{i<c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{\widehat{w}}_{l,j}^{\mathrm{T}}\widehat{x}}}+{\sigma }^2\right).$$

(60)

$${\widehat{f}}_2\left(\widehat{x}\right)=-{\log }_2\left({\widehat{w}}_{l,c}^{\mathrm{T}}\widehat{x}{\displaystyle \sum _{i\le c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{\widehat{w}}_{l,j}^{\mathrm{T}}\widehat{x}}}+{\sigma }^2\right).$$

(61)

$${\widehat{f}}_3\left(\widehat{x}\right)=\left({\widehat{w}}_{l,c}^{\mathrm{T}}\widehat{x}{\displaystyle \sum _{i<c}{\eta }_{r,l,i}+{\displaystyle \sum _{j\ne l,j\in \mathcal{L}}{\widehat{w}}_{l,j}^{\mathrm{T}}\widehat{x}}}+{\sigma }^2\right)R_{{}_{r,l,c}}^{MIN}-{\eta }_{r,l,c}{\widehat{w}}_{l,c}^{\mathrm{T}}\widehat{x}.$$

(62)

The gradient-descent method is used to solve for $\widehat{x}$ according to Equation (63).

$$\begin{array}{l}{\widehat{x}}^{\left(\tau +1\right)}={\widehat{x}}^{\left(\tau \right)}-{\alpha }_5\left({\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\nabla }_{\widehat{x}}{\widehat{f}}_1^{\left(\tau \right)}\left(\widehat{x},{\widehat{x}}^{\left(\tau -1\right)}\right)}}+{\displaystyle \sum _{l\in \mathcal{L}}{\displaystyle \sum _{c\in {\mathcal{L}}_l}{\nabla }_{\widehat{x}}{\widehat{f}}_2^{\left(\tau \right)}\left(\widehat{x}\right)}}\right)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{\alpha }_5{\zeta }_3\left({\nabla }_{\widehat{x}}^{\left(\tau \right)}{\left[\max \left\{0,{\widehat{f}}_3\left(\widehat{x}\right)\right\}\right]}^2\right)+\frac{2{\alpha }_5{\varpi }_2}{\widehat{y}-{\widehat{x}}^T{O}^{-1}\widehat{x}}{O}^{-1}\widehat{x},\end{array}$$

(63)

where ${\alpha }_5$ is the step size factor.

In Algorithm 1, the complexity of computing the power allocation is ${\mathcal{O}}_1\triangleq \mathcal{O}\left({\tau }_1^{\max }{M}^{3.5}{\log }_2\left(\frac{1}{{\epsilon }_1}\right)\right)$, where ${\tau }_1^{\max }$ is the number of iterations of Algorithm 1, and ${\epsilon }_1$ denotes the accuracy of Algorithm 1. In Algorithm 2, the complexity of the active beam formation is calculated as ${\mathcal{O}}_2\triangleq \mathcal{O}\left({\tau }_2^{\max }{N}_t\log \left(N_t\right)\right)$, where ${\tau }_2^{\max }$ is the number of iterations of the algorithm. In Algorithm 3, the complexity of calculating the STAR-RIS passive beam formation is ${\mathcal{O}}_3\triangleq \mathcal{O}\left({\tau }_3^{\max }\left|{\gamma }_r\right|\log \left(\left|{\gamma }_r\right|\right)\right)$, where ${\tau }_3^{\max }$ is the number of iterations of the algorithm. Thus, the total complexity of the algorithm is $\mathcal{O}\left({\tau }_4^{\max }\left({\mathcal{O}}_1+{\mathcal{O}}_2+{\mathcal{O}}_3\right)\right)$, where ${\tau }_4^{\max }$ denotes the number of outer iterations.

Algorithm 1: Proposed Penalty-Based Iterative Algorithm for Solving Problem (37)

1: Initialize feasible point ${\eta }_{r,l,c}^{(0)}$$,\mathrm{and}\mathrm{penalty}\mathrm{factor}{\zeta }_1$.

2: repeat:

3:  Set the iteration index $\tau =0$.

4:  repeat:

5:  $\mathrm{Use}{f}_1\left({\eta }_{r,l,c},{\eta }_{r,l,c}^{\left(\tau \right)}\right)$$\mathrm{to}\mathrm{approximate}{f}_1\left({\eta }_{r,l,c}\right)$.

6:  $\mathrm{Update}{\eta }_{r,l,c}^{(\tau +1)}$ by solving problem (40) using (41).

7:  $\tau =\tau +1$.

8:    until the fraction of objective function values decreases below a predefined threshold or the maximum number of internal iterations is reached.

9: $\mathrm{Update}{\eta }_{r,l,c}^{(0)}$.

10:  $\mathrm{Update}\mathrm{penalty}\mathrm{factor}{\zeta }_1$.

11: until the constraint violation is below a predefined threshold.

Algorithm 2: Proposed Penalty-Based Iterative Algorithm for Solving Problem (42)

1: Initialize the feasible point a $V_{r,l}^{(0)}$$,\mathrm{penalty}\mathrm{factor}{\zeta }_2$.

2: repeat:

3: $\mathrm{Set}\mathrm{the}\mathrm{iteration}\mathrm{index}\tau =0$.

4:  repeat:

5:  $\mathrm{For}\mathrm{a}\mathrm{given}{V}_{r,l}^{(\tau )}$$,\mathrm{set}\mathrm{the}\mathrm{barrier}\mathrm{factor}{\varpi }_1$, and use BCD to convert the solution of problem (46) to problem (48).

6:  $\mathrm{Update}{\stackrel{⌢}{x}}^{\left(\tau +1\right)}$ by (52).

7:  $\mathrm{Update}{V}_{r,l}^{(\tau +1)}$ with the obtained optimal solution.

8:  $\tau =\tau +1$.

9:  until the fraction of objective function values decreases below a predefined threshold or the maximum number of internal iterations is reached.

$10:\mathrm{Update}{V}_{r,l}^{(0)}$.

12: until the constraint violation is below a predefined threshold.

Algorithm 3: Proposed Penalty-Based Iterative Algorithm for Solving Problem (53)

1: Initialize feasible point $U_{r,l,c}^{(0)}$$,\mathrm{and}\mathrm{penalty}\mathrm{factor}{\zeta }_3$.

2: repeat:

3: $\mathrm{Set}\mathrm{the}\mathrm{iteration}\mathrm{index}\tau =0$.

4:  repeat:

5:  $\mathrm{For}\mathrm{a}\mathrm{given}{U}_{r,l,c}^{(\tau )}$$,\mathrm{set}\mathrm{the}\mathrm{barrier}\mathrm{factor}{\varpi }_2$ and convert problem (57) to problem (59).

6:  $\mathrm{Update}{\widehat{x}}^{\left(\tau +1\right)}$ using (63).

7:  $\mathrm{Update}{U}_{r,l,c}^{(\tau +1)}$ with the obtained optimal solution.

8:  $\tau =\tau +1$.

9:  until the fraction of objective function values decreases below a predefined threshold or the maximum number of internal iterations is reached.

$10:\mathrm{Update}{U}_{r,l,c}^{(0)}$.

12: until the constraint violation is below a predefined threshold.

5. Numerical Results

In this section, numerical simulations are performed to evaluate the performance of the proposed algorithm. The specific simulation scenario is shown in Figure 5. Without loss of generality, it is assumed that the BS and the STAR-RIS are located at (0, 0, 50) and (0, 40, 50), respectively. The STAR-RIS–NOMA system has $M=12$ users. The users are divided into $L=4$ clusters with cluster center coordinates of (0, 20, 0), (0, 30, 15), (0, 50, 0), and (0, 60, −15), respectively, and the users are randomly distributed in clusters corresponding to a radius of 6. It is assumed that STAR-RIS is equipped with $N_{ris}=N_{RIS}^h{N}_{ris}^v$ elements, where $N_{RIS}^h=5$ denotes the number of elements along the horizontal direction, $N_{RIS}^v=5$ represents the number of elements along the vertical path, and STAR-RIS elements are divided into $\left|\gamma \right|=4$ clusters. The narrowband quasi-static fading channel from the BS to the STAR-RIS and from the STAR-RIS to the user is modeled as a Rician-fading channel as follows:

$$H=\sqrt{\frac{{\varsigma }_0}{{\xi }_{BR}^{{\omega }_{BR}}}}\left(\frac{B_{BR}}{B_{BR}+1}{H}^{Los}+\sqrt{\frac{1}{B_{BR}+1}}{H}^{NLos}\right),$$

(64)

$$g_{l,c}=\sqrt{\frac{{\varsigma }_0}{{\xi }_{RU,l,c}^{{\omega }_{RU}}}}\left(\frac{B_{RU}}{B_{RU}+1}{g}_{l,c}^{Los}+\sqrt{\frac{1}{B_{RU}+1}}{g}_{l,c}^{NLos}\right),$$

(65)

where ${\xi }_{BR}$ and ${\xi }_{RU,l,c}$ denote the distance between the BS and the STAR-RIS and the distance between the STAR-RIS and the user, respectively; ${\varsigma }_o$ denotes the path loss at a reference distance of 1 m; ${\omega }_{BR}$ and ${\omega }_{RU}$ denote the corresponding path loss index; $B_{BR}$ and $B_{RU}$ denote the Rician factors; and $H^{Los}$ and $g_{l,c}^{Los}$ are the determined line-of-sight (LoS) components. $H^{NLos}$ and $g_{l,c}^{NLos}$ are the random non-line-of-sight (NLoS) components modeled using Rician fading. In order not to lose generality, assume that users have the same QoS requirements and set $R_{l,c}^{\min }=0.1\mathrm{bits}∕\mathrm{s}∕\mathrm{Hz}$. The system parameters used are shown in

Table 1. System Parameters.

Parameter	Parameter Explanation	Value
$B_{BR}$	Rician factor for BS to STAR-RIS channels	3 dB
$B_{RU}$	Rician factor of STAR-RIS to user channel	3 dB
${\varsigma }_o$	Path loss at a reference distance of 1 m	−30 dB
${\omega }_{BR}$	Path loss index for BS to STAR-RIS channels	2.2
${\omega }_{RU}$	Path Loss Index for Path Loss Index	2.2
${\zeta }_1,{\zeta }_2,{\zeta }_3,{\zeta }_4$	Initialize penalty factor	10−4
${\tau }_{\max }$	Maximum number of inner-layer iterations	25

.

Figure 6 investigates the convergence of the power-allocation algorithm. With $P_{\max }=35\mathrm{dBm}$, $N_{ris}=25$, and $N_t=5$, the user-achievable sum rate of the algorithm converges in about 11 iterations as the number of iterations increases.

Figure 7 shows the relationship between the user-achievable sum rate and the number of RIS elements. A total of four cases are considered. The first case is the use of the STAR-RIS in the NOMA system, where the number of elements of STAR-RIS is $N_{ris}=N$. In the second case, two conventional RISs are used in the system: one for reflection and the other for transmission, with $N$ elements per RIS. The third case also uses conventional RISs, but each RIS has $\frac{N}{2}$ elements. The fourth type is the NOMA system without RIS assistance. The results show that the performance of the STAR-RIS is higher than that of the conventional RIS for the same number of RIS elements. The higher version of the dual RIS than the STAR-RIS is because the dual RIS has $N$ reflected and $N$ transmitted elements at the same moment, while the transmitted/reflected elements of the STAR-RIS add up to a total of $N$.

Figure 8 shows the user-achievable sum rate versus the BS transmit antenna in the case of conventional RIS-assisted NOMA and STAR-RIS-assisted NOMA communications. The experimental results show that the use of STAR-RIS-assisted communication is superior to a conventional RIS when the number of BS antennas is the same, given the same number of RIS elements. It can be observed that using RIS in NOMA can significantly improve the system’s performance. Furthermore, as the number of antennas at the BS increases, the performance of the RIS–NOMA system also increases.

Figure 9 shows the user-achievable sum rate versus the total transmitted power Pmax during conventional RIS and STAR-RIS-assisted NOMA communications. The user-achievable sum rate increases with the total transmission power. RIS-assisted communications significantly improve user achievability and rates over NOMA systems without RIS assistance. It was observed that the performance difference between the system using two conventional RISs and the STAR-RIS system was relatively small.

Figure 10 considers the relationship between the user-achievable sum rate and the minimum user rate for three cases: a STAR-RIS system with $N$ elements, a RIS system with $2N$ elements, and a RIS system with $N$ elements. The results show that the user-achievable sum rate decreases as the minimum user rate increases. In all three cases, the RIS system with $2N$ components has the highest user-achievable sum rate, and the STAR-RIS system with $N$ elements performs better than the RIS system with $N$ elements.

Figure 11 shows how the proposed method in this paper compares with four other methods or schemes. The first scheme is where no RIS is configured in the system. The second scheme is where each cell of the STAR-RIS is randomly configured with a phase. The third scheme uses a pairwise-rising method [41] to determine the phase shift of the STAR-RIS. The fourth scheme is to use the Riemann conjugate-gradient method [42] to solve the phase-shift matrix of the STAR-RIS. As can be seen from the results in Figure 11, when the number of RIS elements is 60, the user-achievable sum rate of the proposed method in this paper is 13.1 bits/s/Hz. The achievable sum rate of the random phase-shift method is only 10.8 bits/s/Hz. The achievable sum rates using the pairwise-ascent method and the Riemann conjugate-gradient method are 12.8 bits/s/Hz and 12 bits/s/Hz, respectively.

Figure 12 shows the variation of achievable users and rates with the number of antennas at the BS. When the number of BS transmitting antennas is 10, the user-achievable sum rate of the random phase shift is 7.3 bits/s/Hz. The user-achievable sum rates of the pairwise-ascent method and Riemann conjugate-gradient method are 10.8 bits/s/Hz and 11.5 bits/s/Hz, respectively. The achievable sum rate of the proposed method is 12.6 bits/s/Hz. Among the five schemes, the worst performance is achieved without RIS assistance, followed by the random RIS phase shift. The Riemann conjugate-gradient method is closer to the case of the pairwise-ascent method, and the proposed method in this paper has the best performance. Therefore, this shows the effectiveness of the proposed algorithm.

6. Conclusions

In this paper, we study the downlink transmission system of the STAR-RIS–NOMA. A new alternate optimization method is proposed for the MS protocol to jointly optimize the power-allocation coefficient of the user, active beamforming at the BS, and passive beamforming of the STAR-RIS so that the achievable rate of the user is maximized. The channel is first feature-extracted, and the user and STAR-RIS elements are clustered according to the extracted features to reduce the solution difficulty. The SDR technique and penalty function are used to solve the nonconvex factors present in the problem. The effectiveness of the proposed algorithm can be seen in the simulation results.

This paper considers the STAR-RIS in an ideal state, where the transmission and reflection phase-shift coefficients can be adjusted independently. However, in real applications, coupled transmission and reflection phase-shift coefficients are more common, and the related research problems are more complex. Therefore, the design of coupled STAR-RIS phase-shift coefficients is a crucial research direction for the future.