Rate-Splitting Multiple Access for Spatial Non-Stationary Extremely Large-Scale Antenna Array

Abstract

The extremely large-scale antenna array (ELAA) is recognized as a promising technology for the sixth-generation wireless communication systems. Besides the extended near-field region, the enlarged aperture introduces spatial non-stationarity, which is characterized by the visibility region (VR). When all the antenna elements in the ELAA are used indiscriminately, the spatial non-stationarity can result in the user receiving signals radiated by partial antenna elements, which cannot be ignored in designing an effective multiple access scheme. To address this, a rate-splitting multiple access (RSMA) scheme is designed for the ELAA with spatial non-stationarity in this paper, where antenna selection and RSMA are jointly exploited to alleviate the effect of the spatial non-stationarity. Then, an optimization problem (OP) is formulated to maximize the weighted sum-rate (WSR) by jointly optimizing user grouping, digital precoding, and the rate-splitting vector. To solve the formulated OP, antenna selection is initially performed, followed by the user grouping algorithm. Subsequently, given the user grouping result, the conditional optimal solutions are obtained by using the semidefinite relaxation method. Simulation results demonstrate that the proposed scheme achieves a higher WSR than the baseline schemes.

1. Introduction

The accelerated progression of mobile communication networks to the sixth generation (6G) has led to an increasing demand for enhanced throughput and reliability [1,2]. To meet these performance requirements, it is necessary to shift the operating frequency from the sub-6 GHz band to the millimeter-wave (mmWave) and terahertz bands [3]. It is evident that higher frequencies are associated with shorter wavelengths of electromagnetic radiation. In addition, they offer a substantial amount of available bandwidth, thereby facilitating the implementation of the extremely large-scale antenna array (ELAA) [4]. Due to the enlarged array aperture, the ELAA can attain an unparalleled beamforming gain, which effectively compensates for the severe path loss caused by high-frequency propagation [5], while significantly improving spatial resolution.

The increase in array aperture and spectral resources also introduces new challenges, primarily manifesting as near-field propagation and spatial non-stationarity [6]. First of all, the ELAA significantly extends the Rayleigh distance [7,8], which leads to a scenario where a larger number of users are served within the near-field region. In this region, the electromagnetic propagation cannot be approximated by far-field plane waves, but behaves as a non-uniform spherical wave. For users located in the far-field, the channel is typically characterized by planar wavefronts, meaning that the steering vector depends solely on angular parameters. In contrast, accurate modeling of the near-field steering vector necessitates both angular and distance information [9,10]. More importantly, the significant expansion of the array aperture implies that distinct subarrays may experience different propagation environments [11]. Specifically, certain regions of the antenna array may be obstructed by obstacles, while different parts of the array observe distinct scatterers [12]. This results in the user’s energy being concentrated only on specific subarrays, thus restricting the attainable transmission rate. This phenomenon is referred to as spatial non-stationarity and is described using the visibility region (VR) [13]. The existing research confirms that neglecting the spatial non-stationarity can result in severe performance degradation [14]. Moreover, considering the costs of hardware and power consumption caused by the full-digital architecture, the hybrid architecture is utilized to strike a trade-off between the performance and the cost of the ELAA [15]. Consequently, the design of efficient multiple access schemes under the constraints of spatial non-stationarity, near-field characteristics, and hardware resources is vital to the successful deployment of ELAA systems.

In terms of the advanced multiple access technologies, rate-splitting multiple access (RSMA) is recognized as a promising multiple access technology for 6G systems [16]. Specifically, RSMA operates by splitting user messages into common and private components and subsequently encoding the aggregated common parts into a unified common stream. By performing Successive Interference Cancellation (SIC), the receiver initially decodes the common stream from the received signal, while regarding all private streams as interference. Subsequently, the designated private stream is decoded from the remaining signal, with other users’ private streams considered as noise [17]. Compared to space division multiple access (SDMA) and non-orthogonal multiple access (NOMA), where SDMA treats all interference as noise and NOMA decodes all interference, RSMA can partly decode interference and partly treat interference as noise, i.e., flexible interference management. Research has shown that RSMA achieves superior performance by dynamically allocating power and rates [18,19]. RSMA is regarded as a generalized multi-user interference management framework that can find the optimal balance between total system rate and user fairness according to demand [20]. Recently, RSMA has been rigorously investigated in emerging scenarios, including intelligent reflecting surface-aided networks and integrated sensing and communication systems, where it yields substantial performance enhancements [21,22].

Recently, the application of RSMA in near-field communication has become a popular research topic. For instance, RSMA is exploited to serve users in a hybrid near-field and far-field scenario in [23], which demonstrates that RSMA can achieve improved spectral efficiency and multi-user interference suppression. By designing spatial beamforming for near-field users, the work in [24] successfully serves additional users in the near-field and far-field regions while guaranteeing the rates of existing users. However, existing studies on RSMA-enhanced ELAA systems rely on the assumption of spatially stationary channels, presupposing that the entire antenna array maintains uniform visibility for all users. Nevertheless, the impact of spatial non-stationarity has not been sufficiently explored within the current literature. In practical ELAA environments, the VRs of different users may partially overlap or be entirely disjoint. The complex nature of VRs may suggest that conventional RSMA strategies could plausibly demonstrate the potential to inflict severe interference upon these users located in the non-visible regions, which could reasonably indicate that this substantial interference appears to bring significant challenges to interference management. Moreover, the significant discrepancy between theoretical assumptions and practical propagation conditions might undermine the intended interference suppression capability of RSMA. Therefore, there is a pressing need to design novel multiple access schemes that fully exploit the performance potential of ELAA systems by accounting for non-stationarity.

1.1. Our Contributions

To address the above challenges, this paper designs an RSMA scheme for the ELAA with spatial non-stationarity. The primary contributions of this work are listed below:

• To address performance degradation caused by the spatial non-stationarity in near-field communication, this paper proposes a hybrid precoding architecture based on a switching network. The system employs a fixed subarray selection (FSS) [25] strategy to select antenna subarrays with superior channel quality. Then, an RF chain controlled by the switching network is dynamically connected to the antenna subarray set associated with the user group’s VR, thus suppressing noise and interference from antenna elements outside the VR.
• To achieve a robust RS strategy, this paper proposes a user grouping method based on channel correlation and a phase-aligned analog precoding scheme. This approach employs the agglomerative hierarchical clustering (AHC) algorithm and the average linkage criterion to group users, utilizing the silhouette coefficient, which could determine the optimal number of clusters. Based on the grouping results, the analog precoding is designed to align with the aggregated channels of user groups, thus maximizing array gain.
• This paper formulates the weighted sum-rate (WSR) maximization problem as a non-convex mixed-integer nonlinear programming (MINLP) problem subject to constraints on transmit power, user achievable data rates, and common data rates. A three-step suboptimal algorithm is designed that jointly optimizes user grouping, digital precoding, and RS vectors. First, antenna selection is performed to mitigate the impact of spatial non-stationarity. Next, users are grouped based on channel correlation. Finally, the problem is solved using a semidefinite relaxation (SDR) iterative method according to the user grouping results.
• Numerical simulation results confirm the effectiveness of the proposed RSMA scheme, which demonstrates significant WSR gains compared to the baseline schemes. Moreover, the robustness of the scheme is validated under different configurations of the number of users, number of antennas, and power budget, confirming its ability to manage spatial non-stationarity in ELAA systems.

1.2. Differences from Existing Studies

Although RSMA technology has demonstrated significant potential in near-field communication, and the FSS strategy has proven effective for ELAA systems, the interaction between spatial non-stationarity and RSMA interference management has not been fully explored. The primary distinctions of this work from the existing literature are as follows.

Existing near-field RSMA schemes [23,24] typically assume spatial stationarity. In contrast, our paper addresses the spatial non-stationarity of ELAA. To address this, our paper proposes a hybrid precoding architecture based on a switching network for spatially non-stationary scenarios. The objective is to suppress noise and interference generated by antenna units outside the visible region.

The FSS architecture was originally proposed in [25] to improve the energy efficiency of uplink XL-MIMO via linear processing (such as zero-forcing or maximum-ratio). This approach essentially relies on SDMA to eliminate inter-user interference. However, in spatially non-stationary scenarios, channel spatial features appear highly coupled. Thus, traditional linear precoding schemes may struggle to distinguish spatial features effectively. Additionally, this leads to inefficient interference suppression and causes performance loss directly. Our simulation results demonstrate that integrating RSMA with FSS enables flexible splitting and partial decoding of overlapping interference in visible regions. Thus, this integration provides more robust performance gains than NOMA and SDMA schemes when dealing with non-stationary channels.

Although [26] demonstrated the feasibility of the SDR method for addressing the RSMA hybrid precoding problem, it primarily targeted scenarios with lower channel dimensions. In ELAA systems, the direct application of such SDR algorithms faces enormous computational complexity challenges. Therefore, this paper proposes a stepwise optimization framework. By leveraging channel non-stationarity through FSS and user grouping strategies, a subset of antennas is selected, and the high-dimensional original channel is transformed into a low-dimensional effective channel. This makes RSMA precoding computationally feasible and efficient in ELAAs.

The rest of this paper is structured as follows. Section 2 describes the spatial non-stationary near-field channel model and the proposed RSMA transmission framework. In Section 3, the mathematical modeling of the optimization problem (OP) and the proposed three-step solution algorithm are described in detail. In Section 4, numerical results and performance analyses are presented. Finally, Section 5 concludes this paper.

2. Spatial Non-Stationary Near-Field Channel Model and Proposed RSMA Scheme

Before introducing the system model,

Table 1. List of notations.

Notation	Description
a	Scalar
$\mathbf{a}$	Vector
$\mathbf{A}$	Matrix
$\mathcal{A}$	Set
${(·)}^T$	Transpose of a vector or matrix
${(·)}^H$	Conjugate transpose
$\left|a\right|$	Absolute value of scalar a
$\left|\mathcal{A}\right|$	Cardinality of set $\mathcal{A}$
$\parallel ·\parallel$	Euclidean norm
⊙	Hadamard product
$\mathrm{Tr}(·)$	Trace of matrix $(·)$
$\mathrm{rank}(·)$	Rank of matrix $(·)$
$\mathcal{A}\setminus \mathcal{B}$	Set difference between $\mathcal{A}$ and $\mathcal{B}$
$\mathbf{A}⪰0$	Matrix $\mathbf{A}$ is positive semidefinite
${\mathbb{H}}^N$	$N\times N$ Hermitian matrices
${\mathbb{C}}^N$	N-dimension complex vectors
$\mathcal{CN}(\mu ,{\sigma }^2)$	Complex normal distributions with mean $\mu$ and variance ${\sigma }^2$

summarizes the key mathematical notations employed throughout this paper. With these notations established, consider a near-field downlink mmWave communication system, as depicted in Figure 1. In this system, the base station (BS) is equipped with an ELAA. The ELAA is a uniform linear array (ULA), which has M antenna elements and is partitioned into N subarrays with ${\displaystyle \frac{M}{N}}$ antenna elements per subarray. Considering the constraints on hardware cost and power budget, a dynamic switching hybrid architecture is used, with $N_{RF}=N$ radio frequency (RF) chains. By using a switching network, RF chains can be connected to any subset of the N subarrays, allowing local connections to be dynamically adjusted. This hardware flexibility enables the system to employ a user-group-based service scheme, in which each active RF chain is tasked with serving a particular user group. Specifically, the on–off dynamic configuration connects the RF chain to the candidate antenna subarrays located within the VR of its associated user group. The system supports K single-antenna users, forming a user set denoted as $\mathcal{K}=\{1,2,\dots ,K\}$.

2.1. Spatial Non-Stationary Near-Field Channel Model

The ELAA is modeled as a ULA centered on the origin of coordinates. Specifically, the location of the m-th antenna element is given by $({\delta }_{m}d,0)$, where ${\delta }_m={\displaystyle \frac{2m-M+1}{2}}$ for $m\in \{0,1,\dots ,M-1\}$, and the antenna spacing is denoted by $d={\displaystyle \frac{\lambda }{2}}$. To accurately model the spherical wave propagation features within the near-field region, the steering vector of the array is formulated as

$$\mathbf{a}(r,\theta )={\displaystyle \frac{1}{\sqrt{M}}}{\left[e^{-j{\displaystyle \frac{2\pi }{\lambda }}(r\left(0\right)-r)},\dots ,e^{-j{\displaystyle \frac{2\pi }{\lambda }}(r(M-1)-r)}\right]}^T,$$

(1)

where r and $\theta$ denote the distance and angle of departure (AoD) from the array center to the user or scatterer, respectively. Consequently, the path length $r\left(m\right)$ between the user/scatterer and the m-th antenna element is calculated as

$$\begin{array}{c}\hfill r\left(m\right)=\sqrt{r^2+{\delta }_m^2{d}^2+2{\delta }_{m}drsin\theta }.\end{array}$$

(2)

The near-field channel model comprises both the line-of-sight (LoS) path and non-line-of-sight (NLoS) paths. As illustrated in Figure 1, the distinct colored regions correspond to the VRs of specific users. Accordingly, the near-field channel vector for user k is expressed as

$$\begin{array}{cc}\hfill {\tilde{\mathbf{h}}}_k=& \sqrt{M}{\alpha }_{k,0}\mathbf{a}(r_{k,0},{\theta }_{k,0})\odot {\mathbf{v}}_{k,0}+\sqrt{{\displaystyle \frac{M}{L}}}\sum _{l=1}^L{\alpha }_{k,l}\mathbf{a}(r_{k,l},{\theta }_{k,l})\odot {\mathbf{v}}_{k,l},\hfill \end{array}$$

(3)

where the indices $l=0$ and $l\in \{1,\dots ,L\}$ represent the LoS path and the l-th NLoS path, respectively. The scalars ${\alpha }_{k,0}$ and ${\alpha }_{k,l}$ denote the complex path gains. The spatial non-stationarity is characterized by the binary visibility indicator vector ${\mathbf{v}}_{k,l}\in {\{0,1\}}^M$. Based on the assumption that antennas within the same subarray share consistent visibility with respect to the same scatterer, we model the visibility region ${\mathcal{V}}_{k,l}$ for the l-th path as a set of contiguous subarrays. Its physical extent is determined by the start index $m_{k,l}^{min}$ and the end index $m_{k,l}^{max}$. Consequently, the set of visible antenna indices is defined as ${\mathcal{V}}_{k,l}=\{m\in \{0,\dots ,M-1\}\mid m_{k,l}^{min}\le m\le m_{k,l}^{max}\}$.

Accordingly, the m-th element of the visibility indicator vector ${\mathbf{v}}_{k,l}$ is constructed as

$${\left[{\mathbf{v}}_{k,l}\right]}_m=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}m\in {\mathcal{V}}_{k,l},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(4)

By calculating the element-wise product of the steering vector and the spatial mask ${\mathbf{v}}_{k,l}$, the channel gains outside the region ${\mathcal{V}}_{k,l}$ are nullified. This mechanism ensures that the signal energy is concentrated solely within the effective visible subarray aperture, thereby mathematically reflecting the spatial non-stationary characteristics of the near field.

Acquiring accurate CSI and VR information is a prerequisite for the precoding design in this work. Although the extremely large aperture of the ELAA introduces complex near-field effects and spatial non-stationarity, the feasibility of acquiring such information with low overhead was demonstrated in [13]. Specifically, by utilizing the group-time-division-based polar-domain simultaneous iterative gridless weighted (GP-SIGW) scheme, the complex full-array non-stationary channel can be effectively decoupled into a series of stationary subarray channels for independent estimation.

Regarding pilot overhead, unlike the dilemma in conventional massive MIMO, where the pilot length scales linearly with the total number of antennas M, ELAA systems can fully exploit channel sparsity. On the one hand, spatial non-stationarity implies that channel energy is primarily concentrated on the subarrays within the user’s VR, eliminating the need for full-array estimation. On the other hand, near-field propagation exhibits significant sparsity in the polar domain. Leveraging these characteristics, the pilot overhead can be significantly reduced to be proportional to the number of dominant propagation paths L (i.e., $L\ll M$). Consequently, the aggregate overheads associated with signaling, channel estimation, and user grouping occupy only a small fraction of the channel coherence block. Accordingly, we will evaluate the impact of CSI estimation errors and VR detection errors on system robustness in Section 4.

2.2. Proposed RSMA Transmission Scheme

A novel RSMA scheme is proposed for the spatially non-stationary ELAA. Initially, antenna selection is performed to convert the downlink channel matrix, $\tilde{\mathbf{H}}={[{\tilde{\mathbf{h}}}_1,\dots ,{\tilde{\mathbf{h}}}_K]}^T\in {\mathbb{C}}^{K\times M}$, into an effective channel matrix $\mathbf{H}={[{\mathbf{h}}_1,\dots ,{\mathbf{h}}_K]}^T\in {\mathbb{C}}^{K\times M^{\prime }}$, where $M^{\prime }$ signifies the quantity of antenna elements that remain active following the selection phase. Let the user grouping result be expressed as $\mathcal{G}=\{{\mathcal{G}}_1,{\mathcal{G}}_2,\dots ,{\mathcal{G}}_{\left|\mathcal{G}\right|}\}$, where $\left|\mathcal{G}\right|$ represents the total number of groups. The set $\mathcal{G}$ is further categorized into two subsets, the set of multi-user groups ${\mathcal{G}}^m$ (where $|{\mathcal{G}}_g|>1$) and the set of single-user groups ${\mathcal{G}}^s$ (where $|{\mathcal{G}}_g|=1$), such that $\mathcal{G}={\mathcal{G}}^m\cup {\mathcal{G}}^s$. In a multi-user group ${\mathcal{G}}_g\in {\mathcal{G}}^m$, for implementing the rate-splitting (RS) strategy, each message $U_k$ for user $k\in {\mathcal{G}}_g$ is split into a private part $U_{k,k}$ and a common part $U_{{\mathcal{G}}_g,k}$. The private messages are independently encoded into private streams $s_{k,k}$. Simultaneously, the common messages from all users within ${\mathcal{G}}_g$ are aggregated into a single group-common message $U_{{\mathcal{G}}_g}$. This common message is encoded into a common stream $s_{{\mathcal{G}}_g}$ using a codebook shared by all users of ${\mathcal{G}}_g$. Note that for a single-user group, there is no need to perform RS. Finally, all generated common streams $s_{{\mathcal{G}}_g}$ and private streams $s_{k,k}$ form a transmitted stream $\mathbf{s}\in {\mathbb{C}}^{N_s\times 1}$, satisfying the power normalization condition $\mathbb{E}\left\{\mathbf{s}{\mathbf{s}}^H\right\}=\mathbf{I}$, where $N_s=\left|{\mathcal{G}}^m\right|+K$ denotes the total number of streams. In the downlink transmission, the signal received by user k is formulated as

$$\begin{array}{c}\hfill y_k={\mathbf{h}}_k{\mathbf{F}}_{RF}{\mathbf{F}}_{BB}\mathbf{s}+{ϵ}_k,\end{array}$$

(5)

where ${\mathbf{F}}_{RF}\in {\mathbb{C}}^{M^{\prime }\times N_{RF}^{\prime }}$ and ${\mathbf{F}}_{BB}\in {\mathbb{C}}^{N_{RF}^{\prime }\times N_s}$ denote the analog and digital precoding matrices, respectively, and $N_{RF}^{\prime }$ is the number of activated RF chains. ${ϵ}_k\sim \mathcal{CN}(0,{\sigma }_k^2)$ represents the additive white Gaussian noise (AWGN) at the receiver. After user grouping determines the analog precoding ${\mathbf{F}}_{RF}$, the equivalent channel matrix can be expressed as ${\mathbf{H}}_{eq}=\mathbf{H}{\mathbf{F}}_{RF}$. Consequently, the received signal is given by $y_k={\mathbf{h}}_{eq,k}{\mathbf{F}}_{BB}\mathbf{s}+{ϵ}_k$, where ${\mathbf{h}}_{eq,k}$ denotes the k-th row vector of ${\mathbf{H}}_{eq}$.

To account for hardware limitations, the transmit power of every active RF chain is restricted to the budget $P_{p,bud}$ through a per-RF chain power constraint. This constraint is formulated as

$$\begin{array}{c}\hfill \mathrm{Tr}\left({\Phi }_n\left(\sum _{{\mathcal{G}}_g\in {\mathcal{G}}^m}{\mathbf{f}}_{BB}^{{\mathcal{G}}_g}{\left({\mathbf{f}}_{BB}^{{\mathcal{G}}_g}\right)}^H+\sum _{k\in \mathcal{K}}{\mathbf{f}}_{BB}^{k,k}{\left({\mathbf{f}}_{BB}^{k,k}\right)}^H\right)\right)\\ \hfill \le P_{p,bud},\forall n\in \{1,\dots ,N_{RF}^{\prime }\},\end{array}$$

(6)

where ${\Phi }_n={\mathbf{e}}_n{\mathbf{e}}_n^H$ functions as a selection matrix, with ${\mathbf{e}}_n$ representing the n-th column of the identity matrix ${\mathbf{I}}_{N_{\mathrm{RF}}^{\prime }}$. In this expression, ${\mathbf{F}}_{BB}^{{\mathcal{G}}_g}$ is the digital precoding vector for the common stream of group ${\mathcal{G}}_g$, while ${\mathbf{F}}_{BB}^{k,k}$ is the digital precoding vector for the private stream of user k. By performing SIC, user k can decode the common stream (if it exists) and the private stream in sequence. Upon successful decoding and cancellation of the common stream, the user proceeds to decode its private stream. The computational complexity faced by each user is determined by the SIC process. Since the dimension of the equivalent channel is $\left|\mathcal{G}\right|$, the computational complexity required for a single user to decode both the common stream and the private stream is only $\mathcal{O}\left(\right|\mathcal{G}\left|\right)$, which is significantly smaller than the total number of antennas M in the array. The achievable rates of user k decoding the common stream and the private stream are written as $R_{{\mathcal{G}}_g,k}={log}_2(1+{\gamma }_{{\mathcal{G}}_g,k})$ and $R_{k,k}={log}_2(1+{\gamma }_{k,k})$, respectively. Specifically, ${\gamma }_{{\mathcal{G}}_g,k}$ is the signal-to-interference-plus-noise ratio (SINR) of decoding the common stream, which is written as

$$\begin{array}{c}\hfill {\gamma }_{{\mathcal{G}}_g,k}={\displaystyle \frac{|{\mathbf{h}}_{eq,k}{\mathbf{f}}_{BB}^{{\mathcal{G}}_g}{|}^2}{\underset{\mathrm{Inter}-\mathrm{group}\mathrm{common}\mathrm{interference}}{\underbrace{{\displaystyle \sum _{{\mathcal{G}}_p\in {\mathcal{G}}^m\setminus {\mathcal{G}}_g}}{|{\mathbf{h}}_{eq,k}{\mathbf{f}}_{BB}^{{\mathcal{G}}_p}|}^2}}+\underset{\mathrm{Total}\mathrm{private}\mathrm{stream}\mathrm{interference}}{\underbrace{{\displaystyle \sum _{q\in \mathcal{K}}}{|{\mathbf{h}}_{eq,k}{\mathbf{f}}_{BB}^{q,q}|}^2}}+{\sigma }_k^2}},\\ \hfill \forall {\mathcal{G}}_g\in {\mathcal{G}}^m,k\in {\mathcal{G}}_g.\end{array}$$

(7)

The private stream SINR then follows as

$$\begin{array}{c}\hfill {\gamma }_{k,k}={\displaystyle \frac{|{\mathbf{h}}_{eq,k}{\mathbf{f}}_{BB}^{k,k}{|}^2}{\underset{\mathrm{Inter}-\mathrm{group}\mathrm{common}\mathrm{interference}}{\underbrace{{\displaystyle \sum _{{\mathcal{G}}_p\in {\mathcal{G}}^m\setminus {\mathcal{G}}_g}}{|{\mathbf{h}}_{eq,k}{\mathbf{f}}_{BB}^{{\mathcal{G}}_p}|}^2}}+\underset{\mathrm{Inter}-\mathrm{user}\mathrm{private}\mathrm{interference}}{\underbrace{{\displaystyle \sum _{q\in \mathcal{K}\setminus k}}{|{\mathbf{h}}_{eq,k}{\mathbf{f}}_{BB}^{q,q}|}^2}}+{\sigma }_k^2}},\\ \hfill \forall {\mathcal{G}}_g\in \mathcal{G},k\in {\mathcal{G}}_g.\end{array}$$

(8)

To guarantee successful decoding of the common data stream by all users in group ${\mathcal{G}}_g$, the rate of the common message is limited to the minimum achievable rate among the group members. This condition is formulated as

$$\sum _{k\in {\mathcal{G}}_g}{C}_{{\mathcal{G}}_g,k}\le R_{{\mathcal{G}}_g}\triangleq min\{R_{{\mathcal{G}}_g,k}\mid \forall k\in {\mathcal{G}}_g\},\forall {\mathcal{G}}_g\in {\mathcal{G}}^m,$$

(9)

where $C_{{\mathcal{G}}_g,k}$ represents the rate allocated to user k from the group common message $U_{{\mathcal{G}}_g}$. The set of $C_{{\mathcal{G}}_g,k}$ is denoted as the RS vector

$${\left[\mathbf{c}\right]}_i\ge 0,\forall i\in \{1,2,\dots ,K_m\},$$

(10)

where $K_m$ represents the total number of users within multi-user groups. Let $R_k^{min}$ denote the Quality of Service (QoS) minimum rate requirement for user k. The total achievable rate $R_k$ must satisfy

$$R_k^{min}\le R_k=\left\{\begin{array}{cc}{C}_{{\mathcal{G}}_g,k}+R_{k,k},\hfill & \forall {\mathcal{G}}_g\in {\mathcal{G}}^m,k\in {\mathcal{G}}_g,\hfill \\ R_{k,k},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(11)

3. Weighted Sum-Rate Maximization

Based on the derived achievable rates and system constraints, the WSR maximization problem is formulated as OP1. Denote the priority weight associated with user k as $w_k$.

$$\begin{array}{cc}\hfill \mathrm{OP}1:& \underset{\mathcal{G},{\mathbf{F}}_{BB},\mathbf{c}}{max}\sum _{k\in \mathcal{K}}{w}_k{R}_k\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(6\right),\left(9\right),\left(10\right),\left(11\right).\hfill \end{array}$$

(12)

Constraint (6) represents the power consumption constraint; (9) governs the common rate transmission; (10) ensures the non-negativity for the RS vector; and (11) guarantees the minimum QoS requirement for each user.

OP1 is a non-convex MINLP problem. The computational intractability of this formulation arises from two primary sources: the complex coupling between the combinatorial user grouping strategy $\mathcal{G}$ and the continuous beamforming variables (${\mathbf{F}}_{BB},\mathbf{c}$), along with the non-convexity inherent in the interference-limited SINR expressions. This complexity renders the derivation of a globally optimal solution computationally prohibitive. Consequently, a three-step suboptimal algorithm is developed in the subsequent section to obtain a tractable solution.

3.1. Fixed Subarray Selection

The spatial non-stationarity of the ELAA system is evident in the fact that the user signal energy is primarily concentrated in its VR. This phenomenon results in substantial disparities in channel conditions across subarrays serving different VRs. The fixed subarray selection (FSS) algorithm in [25] is adopted to select antenna subarrays with better communication quality to serve users, thereby reducing communication system overhead and mitigating the impact of the non-stationarity.

For user k, the channel quality metric of the n-th subarray is denoted by ${\xi }_{k,n}$ and defined as

$${\xi }_{k,n}={\displaystyle \frac{{\displaystyle \sum _{m\in {\mathcal{M}}_n}}{\left|{\left[{\tilde{\mathbf{h}}}_k\right]}_m\right|}^2}{{\displaystyle \sum _{i\in \mathcal{K}\setminus k}}{\displaystyle \sum _{m\in {\mathcal{M}}_n}}{\left|{\left[{\tilde{\mathbf{h}}}_i\right]}_m\right|}^2}}.$$

(13)

A large value of ${\xi }_{k,n}$ indicates that this subarray has superior channel quality for user k and low interference channel correlation with other users. In (13), ${\mathcal{M}}_n=\left\{{\displaystyle \frac{M}{N}}(n-1)+1,\dots ,{\displaystyle \frac{M}{N}}n\right\}$ represents the set of antenna elements within the n-th subarray. The collection of quality metrics ${\xi }_{k,n}$ for $n\in \{1,\dots ,N\}$ is denoted as a vector ${\xi }_k={[{\xi }_{k,1},{\xi }_{k,2},\dots ,{\xi }_{k,N}]}^T$, where the elements are sorted in descending order. Given the antenna selection coefficient $z_0$ ($0<z_0\le 1$), the subarrays corresponding to the top $N_k$ largest values in ${\xi }_k$ are selected to satisfy

$$\sum _{i=1}^{N_k}{Z}_{k,i}\ge z_0{Z}_k.$$

(14)

$Z_{k,\left(i\right)}$ denotes the signal power of the subarray with the i-th highest quality metric. Explicitly, for user k, $Z_{k,n}={\displaystyle \sum _{m\in {\mathcal{M}}_n}}{\left|{\left[{\tilde{\mathbf{h}}}_k\right]}_m\right|}^2$ represents the cumulative signal power on the n-th subarray and $Z_k={\displaystyle \sum _{n=1}^N}{Z}_{k,n}$ denotes the total cumulative power across the entire array. By selecting an appropriate $z_0$, the FSS is achieved. Denote the union of the selected antenna indices for all users as ${\mathcal{A}}_{\mathrm{sel}}$. The effective channel matrix $\mathbf{H}$ then follows from the antenna selection set ${\mathcal{A}}_{\mathrm{sel}}$, as summarized in Algorithm 1.

Algorithm 1 Fixed subarray selection strategy

Input: channel matrix $\tilde{\mathbf{H}}$, antenna selection coefficient $z_0$, number of subarrays N. Output: $\mathbf{H}$. 1: Initialize: Selected antenna index set ${\mathcal{A}}_{\mathrm{sel}}=\varnothing$. Partition the array into sets ${\left\{{\mathcal{M}}_n\right\}}_{n=1}^N$. 2: Metric Computation: 3: For all k and n, calculate the cumulative signal power $Z_{k,n}={\displaystyle \sum _{m\in {\mathcal{M}}_n}}{\left|{\left[{\tilde{\mathbf{h}}}_k\right]}_m\right|}^2$. 4: Calculate the ${\xi }_{k,n}$ via (13). 5: Selection Process: 6: for each user $k\in \mathcal{K}$ do 7:    Sort the elements of vector ${\xi }_k$ in descending order. 8:    Find the minimum $N_k$ such that it satisfies (14). 9:    Update the activated antenna set: ${\mathcal{A}}_{\mathrm{sel}}$. 10: end for 11: Construct $\mathbf{H}$ based on ${\mathcal{A}}_{\mathrm{sel}}$. 12: return $\mathbf{H}$.

3.2. User Grouping and Analog Precoding Design

For the ELAA, the robust user grouping strategy is the key to exploring the performance potential of RSMA. Clustering based on channel correlation groups users with highly similar channel characteristics into the same cluster. This grouping approach not only maximizes beamforming gain within clusters but also effectively suppresses interference by leveraging the near-orthogonality between clusters. Research on NOMA systems has shown [27] that grouping users with strong channel correlations can enhance channel multiplexing gains while reducing interference. However, these methods are typically constrained by the number of predefined groups and lack the flexibility to adapt to the complex, non-stationary channel environment of the ELAA. To this end, we employ the AHC [28] algorithm. Unlike static grouping methods, AHC constructs a hierarchical clustering structure based on channel correlations. To autonomously determine the optimal number of groups in the hierarchical structure, the silhouette coefficient is also used as an indicator of effectiveness [29]. By maximizing the average silhouette value, the proposed method can find clusters that are highly compact within groups and sufficiently separated between groups.

The channel correlation between user i and user j is quantified by the normalized coefficient ${\rho }_{i,j}$, which is computed based on the effective channel vectors after FSS and defined as

$${\rho }_{i,j}={\displaystyle \frac{|{\mathbf{h}}_i^H{\mathbf{h}}_j|}{{\parallel {\mathbf{h}}_i\parallel }_2{\parallel {\mathbf{h}}_j\parallel }_2}},$$

(15)

where ${\mathbf{h}}_i$ and ${\mathbf{h}}_j$ denote the effective channel vectors for user i and user j, respectively, with the coefficient satisfying $0\le {\rho }_{i,j}\le 1$. A higher value of ${\rho }_{i,j}$ indicates a stronger spatial correlation between users i and j.

For performing user grouping, the correlation matrix is transformed into a distance matrix $\mathbf{D}\in {\mathbb{R}}^{K\times K}$, where the entry $d_{i,j}={\left[\mathbf{D}\right]}_{i,j}$ is defined as

$$d_{i,j}=1-{\rho }_{i,j}.$$

(16)

This formulation establishes a distance metric derived from normalized channel correlations. The underlying premise is that user pairs exhibiting high channel correlation are separated by smaller effective distances. Leveraging this metric, AHC operates as a bottom-up strategy that builds a hierarchical clustering structure by iteratively merging the two nearest clusters. Under the average linkage criterion, the distance between two clusters ${\mathcal{C}}_a$ and ${\mathcal{C}}_b$ is computed as

$$d({\mathcal{C}}_a,{\mathcal{C}}_b)={\displaystyle \frac{1}{|{\mathcal{C}}_a\left|\right|{\mathcal{C}}_b|}}\sum _{i\in {\mathcal{C}}_a}\sum _{j\in {\mathcal{C}}_b}{d}_{i,j}.$$

(17)

The AHC procedure outputs a dendrogram, which represents all possible grouping hierarchies. To objectively determine the optimal grouping result from this dendrogram, the silhouette coefficient is used as the effectiveness measure in this paper. Specifically, the profile value $\psi \left(i\right)$ of a user i quantifies how close the user is to the group to which it belongs and how separated it is from other groups, denoted by

$$\psi \left(i\right)={\displaystyle \frac{d_b\left(i\right)-d_a\left(i\right)}{max\{d_a\left(i\right),d_b\left(i\right)\}}},$$

(18)

where $d_a\left(i\right)$ and $d_b\left(i\right)$ represent the average intra-group distance and the average distance to the nearest group for user i, respectively. For the user group $\mathcal{G}$, the average silhouette coefficient is defined as

$${\overline{\Psi }}_{\left|\mathcal{G}\right|}={\displaystyle \frac{1}{K}}\sum _{k=1}^K\psi \left(k\right).$$

(19)

Therefore, the optimal grouping result ${\mathcal{G}}^{*}$ is determined by maximizing the average silhouette coefficient to achieve a balance between the closeness within the group and the separation between the groups, defined as

$$|{\mathcal{G}}^{*}|=\underset{n\in \{2,\dots ,K-1\}}{arg\; max}{\overline{\mathrm{\Psi }}}_n.$$

(20)

Algorithm 2 describes the specific process of grouping users.

Algorithm 2 Correlation-based hierarchical user grouping

Input: Effective channel matrix $\mathbf{H}$, range of group numbers. Output: Optimal user grouping strategy ${\mathcal{G}}^{*}$. 1: Step 1: Metric Construction 2: Compute correlation via (15) and distance matrix $\mathbf{D}$ via (16). 3: Step 2: Hierarchical Clustering 4: Perform AHC on $\mathbf{D}$ using average linkage (17) to generate the dendrogram $\mathcal{T}$. 5: Step 3: Autonomous Selection 6: for $n=2$ to $K-1$ do 7:    Calculate mean Silhouette value ${\overline{\mathrm{\Psi }}}_n$. 8: end for 9: Determine optimal group number via (20). 10: return ${\mathcal{G}}^{*}$.

Given the user grouping result, denoted by ${\mathcal{G}}^{*}=\{{\mathcal{G}}_1,{\mathcal{G}}_2,\dots ,{\mathcal{G}}_{|{\mathcal{G}}^{*}|}\}$, the switching network is set up so that each group ${\mathcal{G}}_g$ ($g=1,\dots ,|{\mathcal{G}}^{*}|$) is served by a dedicated RF chain. In this setup, the g-th RF chain connects to the selected subarrays that serve user group ${\mathcal{G}}_g$. Accordingly, the system activates $N_{RF}^{\prime }$ RF chains, where $N_{RF}^{\prime }=\left|{\mathcal{G}}^{*}\right|$, to support the grouped transmission. To further capture the large-scale antenna array gain, analog precoding is designed to phase-align with the aggregated channels of each group [30,31]. Specifically, the aggregated channel vector ${\overline{\mathbf{h}}}_g$ for group ${\mathcal{G}}_g$ is defined as the coherent superposition of the effective channels of all users in ${\mathcal{G}}_g$, given by

$${\overline{\mathbf{h}}}_g=\sum _{k\in {\mathcal{G}}_g}{\mathbf{h}}_k.$$

(21)

Considering the hardware constraints of finite-resolution phase shifters, the continuous phase angles must be quantized to a discrete set $\mathcal{B}={\left\{{\displaystyle \frac{2\pi n}{2^B}}\right\}}_{n=0}^{2^B-1}$, where B denotes the resolution in bits. The subset of activated antenna indices corresponding to the g-th RF chain is defined as ${\mathcal{M}}_g^{\prime }\subseteq \{1,\dots ,M^{\prime }\}$. For each activated antenna element $m\in {\mathcal{M}}_g^{\prime }$, the unquantized phase is extracted as ${\theta }_{g,m}=\angle \left({\left[{\overline{\mathbf{h}}}_g\right]}_m\right)$. The optimal discrete phase ${\widehat{\theta }}_{g,m}$ is subsequently selected to minimize the quantization error, which is defined as

$${\widehat{\theta }}_{g,m}=\underset{\varphi \in \mathcal{B}}{arg\; min}|e^{j\varphi }-e^{j{\theta }_{g,m}}|.$$

(22)

Accordingly, the elements of the analog precoding vector ${\mathbf{f}}_{RF}^g\in {\mathbb{C}}^{M^{\prime }}$ are determined as

$${\left[{\mathbf{f}}_{RF}^g\right]}_m=\left\{\begin{array}{cc}{\displaystyle \frac{1}{\sqrt{|{\mathcal{M}}_g^{\prime }|}}}{e}^{j{\widehat{\theta }}_{g,m}},\hfill & \mathrm{if}m\in {\mathcal{M}}_g^{\prime },\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(23)

The normalization factor ${\displaystyle \frac{1}{\sqrt{|{\mathcal{M}}_g^{\prime }|}}}$ ensures that each analog beamforming vector ${\mathbf{f}}_{RF}^g$ satisfies the unit-norm constraint. Following the above method, an analog precoding matrix ${\mathbf{F}}_{RF}=[{\mathbf{f}}_{RF}^1,\dots ,{\mathbf{f}}_{RF}^{|{\mathcal{G}}^{*}|}]$ can be constructed. The specific implementation details are described in Algorithm 3.

Algorithm 3 Phase-aligned analog precoding design

Input: Effective channel matrix $\mathbf{H}$, user grouping result ${\mathcal{G}}^{*}$, phase resolution B. Output: Analog precoding matrix ${\mathbf{F}}_{RF}$. 1: Initialize: ${\mathbf{F}}_{RF}={\mathbf{0}}_{M^{\prime }\times \left|{\mathcal{G}}^{*}\right|}$. 2: Generate discrete phase codebook $\mathcal{B}={\left\{{\displaystyle \frac{2\pi n}{2^B}}\right\}}_{n=0}^{2^B-1}$. 3: for $g=1$ to $|{\mathcal{G}}^{*}|$ do 4:    Compute group aggregate channel: ${\overline{\mathbf{h}}}_g={\sum }_{k\in {\mathcal{G}}_g}{\mathbf{h}}_k$. 5:    Identify activated antenna set: ${\mathcal{M}}_g^{\prime }=\{m\mid |{\left[{\overline{\mathbf{h}}}_g\right]}_m|>0\}$. 6:    for each antenna index $m\in {\mathcal{M}}_g^{\prime }$ do 7:      Extract phase: ${\theta }_{g,m}=\angle \left({\left[{\overline{\mathbf{h}}}_g\right]}_m\right)$. 8:      Quantize phase: ${\widehat{\theta }}_{g,m}=arg{min}_{\varphi \in \mathcal{B}}|e^{j\varphi }-e^{j{\theta }_{g,m}}|$. 9:      Assign precoding element: ${\left[{\mathbf{f}}_{RF}^g\right]}_m={\displaystyle \frac{1}{\sqrt{|{\mathcal{M}}_g^{\prime }|}}}{e}^{j{\widehat{\theta }}_{g,m}}$. 10:    end for 11: end for 12: return ${\mathbf{F}}_{RF}$.

3.3. Proposed SDR-Based Algorithm

Given the user grouping result ${\mathcal{G}}^{*}$, OP1 is equivalently rewritten as

$$\begin{array}{cc}\hfill \mathrm{OP}2:& \underset{{\mathbf{F}}_{BB},\mathbf{c}}{max}\sum _{k\in \mathcal{K}}{w}_k{R}_k\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(6\right),\left(9\right),\left(10\right),\left(11\right).\hfill \end{array}$$

(24)

To solve OP2, the SDR method [26] is adopted. Define variables ${\mathbf{F}}_{BB}^{{\mathcal{G}}_g}={\mathbf{f}}_{BB}^{{\mathcal{G}}_g}{\left({\mathbf{f}}_{BB}^{{\mathcal{G}}_g}\right)}^H$, $\forall {\mathcal{G}}_g\in {\mathcal{G}}^m$ and ${\mathbf{F}}_{BB}^k={\mathbf{f}}_{BB}^k{\left({\mathbf{f}}_{BB}^k\right)}^H$, $\forall k\in \mathcal{K}$, which satisfy ${\mathbf{F}}_{BB}^{{\mathcal{G}}_g}⪰\mathbf{0}$, $\mathrm{rank}\left({\mathbf{F}}_{BB}^{{\mathcal{G}}_g}\right)=1$, ${\mathbf{F}}_{BB}^{{\mathcal{G}}_g}\in {\mathbb{H}}^{N_{RF}^{\prime }}$, ${\mathbf{F}}_{BB}^k⪰\mathbf{0}$, $\mathrm{rank}\left({\mathbf{F}}_{BB}^k\right)=1$, and ${\mathbf{F}}_{BB}^k\in {\mathbb{H}}^{N_{RF}^{\prime }}$, and the set of all variables is defined as $\mathcal{F}$. To address the non-convex rate constraint, exploiting the difference-of-convex structure of the achievable rate function, the non-convex interference term is linearized via a first-order Taylor expansion. Consequently, the common rate constraint in (9) is reformulated as the following convex approximation:

$$\sum _{k\in {\mathcal{G}}_g}{C}_{{\mathcal{G}}_g,k}\le H_{{\mathcal{G}}_g,k}^1\left(\mathcal{F}\right)+H_{{\mathcal{G}}_g,k}^{2,L}\left(\mathcal{F},{\mathcal{F}}^t\right).$$

(25)

$H_{{\mathcal{G}}_{g,k}}^1\left(\mathcal{F}\right)$ is written as

$$H_{{\mathcal{G}}_{g,k}}^1\left(\mathcal{F}\right)={log}_2\left(\mathrm{Tr}\left({\mathbf{F}}_{BB}^{{\mathcal{G}}_g}{\mathbf{h}}_{eq,k}^H{\mathbf{h}}_{eq,k}\right)+I_{{\mathcal{G}}_{g,k}}\right),$$

(26)

where $I_{{\mathcal{G}}_{g,k}}$ is written as

$$\begin{array}{cc}\hfill I_{{\mathcal{G}}_{g,k}}& =\sum _{{\mathcal{G}}_p\in {\mathcal{G}}^m\setminus {\mathcal{G}}_g}\mathrm{Tr}\left({\mathbf{F}}_{BB}^{{\mathcal{G}}_p}{\mathbf{h}}_{eq,k}^H{\mathbf{h}}_{eq,k}\right)\hfill \\ \hfill & +\sum _{q\in \mathcal{K}}\mathrm{Tr}\left({\mathbf{F}}_{BB}^q{\mathbf{h}}_{eq,k}^H{\mathbf{h}}_{eq,k}\right)+{\sigma }_k^2.\hfill \end{array}$$

(27)

$H_{{\mathcal{G}}_{g,k}}^{2,L}(\mathcal{F},{\mathcal{F}}^t)$ is a lower bound of $H_{{\mathcal{G}}_{g,k}}^2\left(\mathcal{F}\right)=-{log}_2\left(I_{{\mathcal{G}}_{g,k}}\right)$, which is written as

$$H_{{\mathcal{G}}_{g,k}}^{2,L}\left(\mathcal{F},{\mathcal{F}}^t\right)=-{\displaystyle \frac{a_{{\mathcal{G}}_{g,k}}\left(I_{{\mathcal{G}}_{g,k}}\right)}{ln2}}+{log}_2\left(a_{{\mathcal{G}}_{g,k}}\right)+{\displaystyle \frac{1}{ln2}},$$

(28)

where the superscript t denotes the t-th iteration, L denotes the lower bound, $a_{{\mathcal{G}}_{g,k}}={\left(I_{{\mathcal{G}}_{g,k}}^t\right)}^{-1}$, and $I_{{\mathcal{G}}_{g,k}}^t$ denotes the result of substituting ${\mathcal{F}}^t$ into $I_{{\mathcal{G}}_{g,k}}$ for the t-th iteration. Similarly, $R_{k,k}$ is transformed into

$$R_{k,k}(\mathcal{F},{\mathcal{F}}^t)=H_{k,k}^1\left(\mathcal{F}\right)+H_{k,k}^{2,L}(\mathcal{F},{\mathcal{F}}^t).$$

(29)

The signal term $H_{k,k}^1\left(\mathcal{F}\right)$ is expressed as

$$H_{k,k}^1\left(\mathcal{F}\right)={log}_2\left(\mathrm{Tr}\left({\mathbf{F}}_{BB}^k{\mathbf{h}}_{eq,k}^H{\mathbf{h}}_{eq,k}\right)+I_{k,k}\right).$$

(30)

The interference $I_{k,k}$ is expressed as

$$\begin{array}{cc}\hfill I_{k,k}& =\sum _{{\mathcal{G}}_p\in {\mathcal{G}}^m\setminus {\mathcal{G}}_g}\mathrm{Tr}\left({\mathbf{F}}_{BB}^{{\mathcal{G}}_p}{\mathbf{h}}_{eq,k}^H{\mathbf{h}}_{eq,k}\right)\hfill \\ \hfill & +\sum _{q\in \mathcal{K}\setminus k}\mathrm{Tr}\left({\mathbf{F}}_{BB}^q{\mathbf{h}}_{eq,k}^H{\mathbf{h}}_{eq,k}\right)+{\sigma }_k^2.\hfill \end{array}$$

(31)

The linearized interference term $H_{k,k}^{2,L}$ is constructed similarly to (28),

$$H_{k,k}^{2,L}\left(\mathcal{F},{\mathcal{F}}^t\right)=-{\displaystyle \frac{a_{k,k}\left(I_{k,k}\right)}{ln2}}+{log}_2\left(a_{k,k}\right)+{\displaystyle \frac{1}{ln2}}.$$

(32)

The rate $R_k$ is transformed into

$$R_k^L=\left\{\begin{array}{cc}{C}_{{\mathcal{G}}_g,k}+R_{k,k}^L,\hfill & \forall {\mathcal{G}}_g\in {\mathcal{G}}^m,k\in {\mathcal{G}}_g,\hfill \\ R_{k,k}^L,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(33)

To simplify the optimization problem, a set of auxiliary variables $\mathcal{U}=\{u_1,\dots ,u_K\}$ is introduced, where each $u_k$ is used as a proxy for the achievable rate of user k. Based on this, the QoS and rate constraints are reformulated as

$$\begin{array}{cc}\hfill u_k& \ge R_k^{min},\forall k\in \mathcal{K},\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill u_k& \le R_k^L,\forall k\in \mathcal{K}.\hfill \end{array}$$

(35)

Based on $\mathcal{F}$, the power budget constraint in (6) is transformed into

$$\begin{array}{cc}\hfill & \mathrm{Tr}\left({\Phi }_n\left(\sum _{{\mathcal{G}}_g\in {\mathcal{G}}^m}{\mathbf{F}}_{BB}^{{\mathcal{G}}_g}+\sum _{k\in \mathcal{K}}{\mathbf{F}}_{BB}^k\right)\right)\le P_{p,bud},\hfill \\ \hfill & \forall n\in \left\{1,\dots ,N_{RF}^{\prime }\right\}.\hfill \end{array}$$

(36)

Based on the above analyses, OP2 is finally reformulated as

$$\begin{array}{cccc}\hfill \mathrm{OP}3:& \underset{\mathcal{F},\mathbf{c},\mathbf{u}}{max}\hfill & \hfill & \sum _{k\in \mathcal{K}}{w}_k{u}_k\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill & \left(10\right),\left(25\right),\left(34\right),\left(35\right),\left(36\right).\hfill \end{array}$$

(37)

Since OP3 is a convex optimization problem, it can be solved efficiently with the CVX toolbox. Algorithm 4 summarizes the iterative SDR procedure used to solve this problem, with $\epsilon$ and T representing the convergence threshold and the iteration limit, respectively. As analyzed in [32], if OP3 is solvable, the optimal solution ${\mathcal{F}}_i^{*}$ must be a Hermitian positive semidefinite matrix; i.e., it satisfies ${\mathcal{F}}_i^{*}\in {\mathbb{H}}^{N_{RF}^{\prime }},{\mathcal{F}}_i^{*}⪰0,\forall i$. Therefore, when the optimal solution ${\mathcal{F}}_i^{*}$ satisfies the rank-1 condition, the corresponding digital precoding vector ${\mathbf{f}}_i^{*}$ can be directly extracted via eigenvalue decomposition.

The computational burden at the BS is dominated by the digital precoding design. The computational complexity for solving the digital precoding at each iteration is $\mathcal{O}\left(N_s\right|\mathcal{G}\left|\right)$. Therefore, based on the interior-point method, the computational complexity of the proposed SDR iterative algorithm is approximately

$$\mathcal{O}(T · (N_s|\mathcal{G}|{)}^4·log({\displaystyle \frac{1}{\epsilon }}))$$

(38)

In contrast, the computational complexities of the FSS algorithm and the user grouping algorithm are $\mathcal{O}\left(KM\right)$ and $\mathcal{O}(K^2|\mathcal{G}|)$, respectively. These overheads are negligible compared to the SDR iterative optimization process.

Algorithm 4 SDR-based iterative algorithm

Input: ${\mathbf{H}}_{eq}$, $w_k$. Output: ${\mathbf{F}}_{BB}^{*}$, ${\mathbf{c}}^{*}$. 1: Initialize $t=0$ and initial feasible solutions ${\mathcal{F}}^{\left(0\right)}$. 2: repeat 3:    Update $t\leftarrow t+1$. 4:    Construct the convex surrogate problem OP3 using Taylor expansions around ${\mathcal{F}}^{(t-1)}$ via (28) and (32). 5:    Solve OP3 to obtain ${\mathcal{F}}^{*}$, ${\mathbf{c}}^{*}$, and ${\mathbf{u}}^{*}$. 6:    Update ${\mathcal{F}}^{\left(t\right)}\leftarrow {\mathcal{F}}^{*}$ and $R_{\mathrm{sum}}^{\left(t\right)}$. 7: until Convergence criterion $|R_{\mathrm{sum}}^{\left(t\right)}-R_{\mathrm{sum}}^{(t-1)}|\le \epsilon$ is met or max iterations T reached. 8: Extract optimal digital precoding ${\mathbf{F}}_{BB}^{*}$ from ${\mathcal{F}}^{*}$ via eigenvalue decomposition. 9: return ${\mathbf{F}}_{BB}^{*},{\mathbf{c}}^{*}$.

4. Simulation Results

The simulation environment considers a downlink mmWave system operating at a carrier frequency of $f_c=30$ GHz, so the corresponding wavelength is $\lambda =0.01$ m. The BS utilizes a ULA with M antenna elements, physically partitioned into $N=32$ disjoint subarrays. The element spacing is fixed at half-wavelength $d={\displaystyle \frac{\lambda }{2}}=0.005$ m. The K single-antenna users are distributed within a $[4,100]$ m annular region centered at the BS. The angle range considered is $[-\pi /3,\pi /3]$.

In order to accurately characterize the spatial properties of the ELAA under near-field conditions, we use a spherical wavefront model. The model includes $L=5$ NLoS paths. Small-scale fading follows a Rician distribution, where the LoS-to-NLoS power ratio is $\kappa =8$. The system bandwidth is set as $B=100$ MHz. The receiver noise power and noise figure are ${\sigma }_k^2=-84$ dBm and $N_F=10$ dB, respectively. The minimum QoS data rate requirement for each user is selected randomly from the interval $(0.5,1]$ bps/Hz. Regarding the optimization algorithm, the maximum number of iterations for the SDR method is set as $T=100$, and the convergence threshold is set as $\epsilon ={10}^{-3}$.

The performance of the proposed scheme (simply referred to as FSS-RSMA) is evaluated against the following benchmark methods:

• FSS-NOMA Scheme: In this baseline scheme, signals are transmitted via NOMA while maintaining the same FSS and user grouping strategy as FSS-RSMA.
• FSS-LDMA Scheme: This baseline scheme employs the same FSS and user grouping strategy as the proposed FSS-RSMA, but utilizes LDMA [8] for transmission.
• Full-RSMA Scheme: This scheme employs RSMA over the full array without applying the FSS strategy.
• Full-NOMA Scheme: As a counterpart to Full-RSMA, this scheme uses NOMA with all antenna elements active.
• Full-LDMA Scheme: This baseline scheme activates the entire array for LDMA transmission.

To assess the impact of spatial non-stationarity, we simulate various VR configurations, specifically 1/4 VR and 1/8 VR, representing the fraction of the array aperture visible to the user.

In the FSS-RSMA scheme, user grouping directly determines whether the user performs RS or not. To identify optimal grouping, we use the silhouette coefficient to evaluate the results. Figure 2 shows how the average silhouette value varies with the number of groups $\left|\mathcal{G}\right|$. The curve peaks at $\left|\mathcal{G}\right|=5$. This means grouping 10 users into five groups is optimal. This grouping result achieves high channel correlation within groups and sufficient separation between them. Figure 3 presents the AHC clustering process. The vertical axis measures the degree of channel divergence among users. Smaller values indicate stronger channel correlation. Users $\{2,6\}$, $\{3,9\}$, and $\{1,7,8,10\}$ merge into three groups at a low “distance”. High correlation typically shows significant overlap in users’ VRs, which can cause severe intra-group interference. By grouping these users together, RSMA can utilize the common stream to mitigate this issue. Conversely, users $\{4,5\}$ merge with others only at the final stage. Their spatial isolation means they are treated as single-user groups.

Figure 4 shows the convergence performance of the SDR-based iterative algorithm. The WSR increases sharply at the start of iterations and stabilizes after approximately 13 to 17 iterations. Specifically, the FSS-RSMA scheme consistently yields the highest WSR, followed by FSS-NOMA, while FSS-LDMA exhibits the lowest performance. This is primarily because, in a spatial non-stationary ELAA environment, overlapping user VRs result in the loss of significant inter-user channel gain differences. As a result, NOMA loses the prerequisites for achieving optimal SIC. RSMA decodes strong interference while treating weak interference as noise. The FSS-LDMA scheme performs the worst because it treats interference between users as noise. When user VRs overlap, strong interference severely limits its performance. This flexibility enables RSMA to utilize the additional degrees of freedom provided by the ELAA more effectively than NOMA and LDMA.

Figure 5 indicates that the system WSR demonstrates a trend with the antenna selection coefficient $z_0$. Moreover, when $z_0$ is small, the BS may utilize a few subarrays that correspond to the strongest signal paths. However, this might ensure high channel quality. Nevertheless, the approach may discard a large number of potentially effective antennas. Thus, the result shows that limited beamforming gain cannot support high transmission rates. As $z_0$ increases, the WSR of all schemes exhibits a significant upward trend. However, the WSR curve gradually saturates after $z_0$ reaches approximately 0.7. Given that user energy becomes highly concentrated within a limited VR range, this saturation occurs. Activating subarrays beyond the VR range may provide extremely low effective power. Moreover, activation introduces noise and exacerbates inter-user interference. In this simulation, RSMA outperforms both NOMA and LDMA across all $z_0$ settings and VR configurations. Thus, results validate RSMA’s superior interference management capability. In summary, Figure 5 demonstrates that the FSS strategy can activate only the subarrays containing the majority of energy, ensuring high transmission rates while reducing hardware overhead.

Figure 6 illustrates how the WSR varies with the transmission power budget across different schemes. The WSR of all schemes improves as the power budget increases. However, the FSS-RSMA scheme proposed in this paper shows significant performance advantages. This fully demonstrates the necessity of FSS. Without FSS, the performance of the full-array scheme would be severely compromised. A substantial increase in power would be needed to achieve data transmission rates comparable to those achieved with the FSS-based schemes. The main reason is that the spatial non-stationarity of the near-field channel results in an effective signal that is mainly confined to the user’s VR range. Activating the full array introduces noise from all M antennas into the received signal. The accumulation of noise causes a dramatic deterioration of the SINR, making full array activation extremely inefficient. Additionally, the FSS/Full-RSMA schemes outperform both the FSS/Full-NOMA and FSS/Full-LDMA schemes across all power budgets. This persistent disparity illustrates that RSMA is more robust to power constraints and makes more efficient use of resources to maximize system throughput compared to NOMA and LDMA.

Figure 7 and Figure 8 illustrate the trends of the system WSR with respect to the number of users K and the number of antenna elements M, respectively. As shown in Figure 7, the WSR of the FSS-RSMA scheme continuously improves with K and demonstrates the best performance among all comparison schemes. In multi-user scenarios, even with identical antenna selection and user grouping strategies, RSMA exhibits more efficient multiple access capabilities than NOMA and LDMA. This is primarily attributed to RSMA’s effective management of severe inter-user interference caused by VR overlap, thus achieving higher WSR. Additionally, as observed in Figure 8, the FSS-based schemes consistently outperform the full-array schemes as M increases. This indicates that under the influence of spatial non-stationarity, the user signal energy is highly concentrated, and the full-array activation strategy cannot achieve optimal system gain. This is because the antenna elements located outside the visible region cannot provide effective beamforming gain but instead introduce additional reception noise, leading to a deterioration in the SINR. Furthermore, comparing simulation results under different VR settings reveals that the overall WSR in the 1/4 VR scenario outperforms that in the 1/8 VR scenario, as demonstrated in both Figure 7 and Figure 8.

Figure 9 shows that the Jain’s fairness index of FSS-RSMA outperforms both FSS-NOMA and FSS-LDMA across the entire power range. The reason lies in RSMA splitting messages into common and private components via RS. By allocating common rates, even users with poor channel conditions receive guaranteed rates, effectively enhancing fairness among users. Due to fixed decoding sequences, NOMA struggles to effectively eliminate interference from partially overlapping regions. This forces weaker users to endure stronger interference, severely compromising system fairness. Conversely, LDMA treats interference between users as noise, leading to significant performance degradation for users with overlapping VRs. Furthermore, FSS-based schemes generally exhibit higher fairness than full-array schemes. This is because full-array activation might introduce noise from antenna elements outside the user’s VR, which exacerbates the deterioration of SINR for weak users, thereby widening performance gaps between strong and weak users. Given that such interference is eliminated, FSS can improve WSR and effectively narrow rate disparity among users, thereby enhancing the overall fairness of the system.

This paper adopts the GP-SIGW channel estimation method proposed in [13]. Although the GP-SIGW method achieves high channel estimation accuracy, channel estimation errors still exist. To evaluate the robustness of the FSS-RSMA scheme, we consider both CSI and VR errors.

Due to pilot overhead and noise, an error inevitably exists between the estimated channel $\widehat{\mathbf{H}}$ and the actual channel $\tilde{\mathbf{H}}$. Therefore, the estimated channel $\widehat{\mathbf{H}}$ is expressed as

$$\widehat{\mathbf{H}}=\tilde{\mathbf{H}}+\mathbf{E}.$$

(39)

Let $\mathbf{E}\in {\mathbb{C}}^{K\times M}$ denote the channel estimation error, where ${\left[\mathbf{E}\right]}_{k,m}\sim \mathcal{CN}(0,{\sigma }_e^2)$. The error variance is scaled by the NMSE factor $\eta$ such that ${\sigma }_e^2=\eta {\displaystyle \frac{\mathrm{Tr}(\tilde{\mathbf{H}}{\tilde{\mathbf{H}}}^H)}{KM}}$.

The GP-SIGW algorithm indicates that valid paths are identified through sparse recovery techniques. Moreover, the critical question of whether the channel estimation can recover all paths may determine whether the user’s VR contains errors. Let $P_e$ denote the visibility error probability and ${\mathbf{v}}_{k,l}\in {\{0,1\}}^M$ represent the visibility between user k and path l. Thus, the estimated visibility ${\widehat{\mathbf{v}}}_{k,l}$ can be defined as

$${\left[{\widehat{\mathbf{v}}}_{k,l}\right]}_m=\left\{\begin{array}{cc}{\left[{\mathbf{v}}_{k,l}\right]}_m,\hfill & \mathrm{with}\mathrm{probability}1-P_e\hfill \\ 1-{\left[{\mathbf{v}}_{k,l}\right]}_m,\hfill & \mathrm{with}\mathrm{probability}{P}_e.\hfill \end{array}\right.$$

(40)

In Figure 10a, the WSR of all schemes exhibits a declining trend as $\eta$ increases. Given that channel estimation errors introduce discrepancies between the precoding matrix and actual channel conditions, the near-field beamforming characteristics appear disrupted, and severe interference results. However, the proposed FSS-RSMA scheme consistently outperforms both FSS-NOMA and FSS-LDMA schemes. Thus, RSMA effectively manages interference caused by CSI errors by splitting messages into common and private parts, and higher robustness appears evident. Figure 10b shows the WSR versus the VR error probability $P_e$. It can be observed that the WSR decreases as $P_e$ increases. Due to errors in the VR, the system activates subarrays outside the VR while some subarrays with good channel quality remain inactive. The resulting array gain loss and introduction of additional noise lead to the decline in WSR. It is noted that the FSS-RSMA scheme consistently outperforms both FSS-NOMA and FSS-LDMA schemes. NOMA’s decoding relies on precise channel gain ordering, and VR errors severely degrade its performance. Additionally, the beam alignment required by LDMA is also impacted by VR errors. In contrast, the RSMA scheme exhibits the smallest decline in WSR performance, validating its robustness under VR error scenarios.

5. Conclusions

In this paper, an RSMA-based scheme is proposed to mitigate the effect of spatial non-stationarity in near-field wireless communication scenarios. The proposed scheme involves antenna selection, user grouping, and RS design. Based on the proposed scheme, a WSR OP is formulated. Then, a three-step suboptimal algorithm is designed to solve the formulated optimization problem. Firstly, the FSS strategy is employed to select antenna subarrays with superior channel quality, thereby mitigating the impact of spatial non-stationarity. Secondly, a user grouping algorithm is used to cluster users with strong channel correlation as a group. Finally, based on the grouping result, a conditional optimal solution to the OP is obtained by using the SDR-based iterative algorithm. The simulation results demonstrate that the proposed scheme significantly improves the WSR performance compared to the baseline schemes.