RSMA-Assisted Fluid Antenna ISAC via Hierarchical Deep Reinforcement Learning

Abstract

Integrated sensing and communications (ISAC) requires tight coordination between spatial signal design and multiple-access strategies to balance communication throughput and sensing accuracy under shared spectral and hardware constraints. However, existing ISAC frameworks with rate-splitting multiple access (RSMA) typically rely on fixed antenna arrays and decoupled optimization, which fundamentally limit their ability to adapt to fast channel variations and dynamic sensing requirements. This paper introduces a fluid antenna-enabled RSMA-assisted ISAC architecture, in which movable antenna ports are exploited as a new spatial degree of freedom to enhance adaptability in both communication and sensing operations. Fluid antenna systems (FAS) are deployed at both the base station and user terminals, allowing dynamic port selection that reshapes the effective channel and sensing beampattern in real time. We formulate a joint sum-rate maximization problem subject to explicit sensing-quality constraints, capturing the coupled impact of antenna port selection, RSMA rate allocation, and multi-beam transmit design. The proposed framework maximizes the communication sum-rate while ensuring that the sensing functionality satisfies a predefined sensing quality constraint. This constraint-based ISAC formulation guarantees that sufficient sensing power is directed toward the target while optimizing communication performance. The resulting optimization involves strongly coupled discrete and continuous decision variables, rendering conventional optimization methods ineffective. To address this challenge, a hierarchical deep reinforcement learning (HDRL) framework is developed, where an upper-layer deep Q-network (DQN) determines discrete antenna port selection and a lower-layer twin delayed deep deterministic policy gradient (TD3) algorithm optimizes continuous beamforming and rate-splitting parameters. Numerical results demonstrate that the proposed approach significantly improves system performance, achieving higher communication sum-rate while satisfying sensing requirements under dynamic propagation conditions.

1. Introduction

The emergence of sixth-generation (6G) wireless networks marks a paradigm shift toward massive connectivity, which is expected to be a cornerstone of future communication systems [1]. However, the persistent challenge of spectrum scarcity continues to hinder the realization of these next-generation networks. To address this limitation, Integrated Sensing and Communications (ISAC) has emerged as a compelling paradigm that enables the joint utilization of radio frequency (RF) resources for both communication and sensing tasks, particularly through the reuse of radar spectrum [2,3,4,5]. ISAC has garnered significant attention in recent years due to its potential to substantially enhance spectral efficiency, thus mitigating spectrum congestion and minimizing resource wastage, while concurrently decreasing hardware and signaling overhead [6]. Consequently, it has been incorporated into diverse wireless architectures, such as ISAC-enabled wireless power transfer (WPT) systems [7], ISAC-based non-orthogonal multiple access (NOMA) systems [8], and ISAC-assisted physical layer security (PLS) frameworks [9]. Furthermore, the inherent performance tradeoffs between sensing and communication functionalities in ISAC systems have been rigorously analyzed in [10].

With the growing number of users, inter-user interference has emerged as a critical factor constraining communication performance. Rate-splitting multiple access (RSMA) has been developed as an efficient mechanism for interference management and enhance network efficiency [11]. In the RSMA approach, the transmitter partitions information into a common stream and multiple private streams via linear precoded rate-splitting. The common stream is intended for decoding by all receivers, while each private stream is independently encoded and decoded at the corresponding receiver using successive interference cancellation (SIC) [12]. By enabling flexible resource allocation and effective interference mitigation, RSMA possesses the potential to address the limitations imposed by scarce wireless resources and multi-user communication requirements, thereby significantly improving the performance of communication systems, including integrated sensing and communication (ISAC) networks [13].

Consequently, RSMA-assisted ISAC provides a promising framework for next-generation wireless networks, offering improved spectral efficiency, robust interference mitigation, and enhanced overall system performance. However, the full potential of RSMA-assisted ISAC may be restricted due to the limited spatial flexibility of conventional fixed antennas, which can constrain interference management and overall communication performance. In this regard, movable antennas (MAs) offer a solution by enabling dynamic spatial repositioning of antennas to synthesize flexible radiation patterns, thereby introducing additional spatial degrees of freedom (DoF) that enhance desired links and effectively suppress interference [14].

Recently, MA technology has emerged as a promising paradigm to further exploit spatial DoFs through antenna position reconfiguration. Traditionally, this is achieved via mechanically movable antenna elements, where antennas are physically repositioned within a predefined region to exploit channel variations and improve system performance.

Beyond mechanical movement, electrically reconfigurable implementations of movable antennas have also been proposed, where antenna movement is emulated without physical displacement. In particular, dense antenna arrays integrated with reconfigurable devices such as pixel-based antenna architectures enable different radiating elements or ports to be activated electronically, thereby realizing virtual antenna movement. Such electrically movable antenna systems offer reduced latency and enhanced adaptability, especially in scenarios where physical movement is constrained [15].

However, electrically reconfigurable movable antenna architectures typically rely on dense arrays of reconfigurable elements and complex control circuitry, which significantly increases hardware complexity and energy consumption. In addition, antenna position adaptation is restricted to discrete switching among predefined ports or pixels, limiting spatial resolution. Furthermore, the dense arrangement of radiating elements may introduce mutual coupling effects that complicate system design and degrade performance. These limitations motivate the development of alternative architectures that can achieve efficient spatial adaptability with reduced implementation overhead.

Building on these observations, the fluid antenna system (FAS) has emerged as a practical and efficient realization of movable antenna concepts. Unlike mechanically movable antennas that rely on physical displacement and electrically reconfigurable architectures that require dense arrays and complex control circuitry, FAS achieves spatial adaptability through a single antenna element that switches among multiple predefined ports within a compact region. This approach significantly reduces hardware complexity and energy consumption, while mitigating mutual coupling effects by avoiding densely packed radiating elements.

In FAS, antenna adaptation is realized through discrete port selection, where the system leverages channel state information (CSI) at each port to determine the optimal antenna position [16], thereby enhancing communication reliability and efficiency. In FAS, each antenna element is connected to its respective RF chain via flexible cabling, allowing real-time repositioning of the antenna within a designated region. This spatial mobility introduces a new degree of freedom for system-level optimization. By adaptively configuring antenna locations based on instantaneous channel conditions, FAS technology enables dynamic channel shaping, thereby enhancing both communication and sensing performance. Hence, the integration of FAS with RSMA-assisted ISAC emerges as a natural and powerful evolution for next-generation wireless systems. By combining the near-continuous spatial flexibility of FAS with the interference-mitigation capabilities of RSMA, the system can dynamically optimize antenna positions and rate-splitting strategies to enhance communication performance while preserving sensing accuracy. This integrated framework possesses potential to achieve higher spectral efficiency, reliability, and enhanced network performance.

The integration of FAS into ISAC gives rise to a mixed discrete–continuous optimization problem that is particularly challenging. Specifically, the antenna port selection involves discrete spatial decisions, while the beamforming and rate-splitting parameters are continuous variables. These two types of decisions are strongly coupled, as any change in antenna port configuration alters the instantaneous channel states, thereby affecting optimal beamforming and power allocation strategies. This coupling leads to a high-dimensional, non-convex, and combinatorial optimization landscape, rendering traditional convex or alternating optimization methods ineffective. The deep reinforcement learning (DRL) approaches, while capable of model-free optimization, often struggle to efficiently explore and balance actions across heterogeneous discrete–continuous decision spaces, leading to suboptimal convergence and instability [17,18]. Therefore, it is imperative to adopt a hybrid deep reinforcement learning approach that can effectively handle both discrete and continuous action spaces, enabling a more flexible and efficient decision-making mechanism [19].

1.1. Related Works

In [20], the authors investigate ISAC networks and address the challenge of jointly optimizing communication and sensing performance under practical constraints, including beamforming, transmit power, and QoS requirements. To tackle the resulting non-convex and high-dimensional optimization problem, a mixture-of-experts (MoE)-based DRL framework is proposed to enable efficient policy learning, where expert networks specialize in different regions of the state space and a gating network dynamically integrates their outputs, thereby promoting stable convergence and effective exploration of complex state-action spaces. It is evident from the obtained results that the proposed DRL achieves significant improvements in sensing and communication performance, highlighting the critical role of DRL in adaptive and efficient ISAC system design.

Researchers have explored multiple-access schemes for ISAC to enhance spectrum efficiency. RSMA is particularly effective, as it enables partial decoding of interference while treating the rest as noise, offering flexible interference management. This improves spectral efficiency and robustness, making RSMA well suited for multiuser ISAC scenarios with heterogeneous channels and strict sensing–communication requirements. In [21], the authors address the challenge of interference management in multi-antenna dual-functional radar-communication (DFRC) systems. The authors propose an RSMA-assisted framework that splits messages into common and private streams and jointly optimizes message splitting, precoding, and radar sequence design. The obtained results demonstrate that RSMA significantly achieve high spectral efficiency by enabling the common stream to function simultaneously as a radar sequence and an interference mitigation tool. In [22], the authors investigate RSMA-assisted ISAC transmission design under realistic channel fading conditions. The authors develop a geometry-based 3D channel model incorporating a dual-functional base station, multiple communication users, and moving targets such as UAVs, capturing distance, velocity, and angle-dependent fading effects. The authors formulate an energy efficiency maximization problem that jointly considers transceiver beamforming, phase shifts, and quality of service (QoS) constraints for communication and sensing. To solve this highly non-convex problem, the authors deploy DRL approach based on the proximal policy optimization (PPO) algorithm, enabling efficient joint optimization of beamforming and phase shift. Simulation results demonstrate that RSMA effectively manages interference and improve spectral efficiency in multiuser ISAC scenarios.

In multiuser ISAC systems, interference between communication users and sensing signals can limit both data transmission and sensing accuracy, highlighting the need for advanced multiple access schemes that can jointly optimize these conflicting objectives. In [23], the authors propose an RSMA-enabled ISAC (RISAC) framework, where RSMA precoding and rate-splitting parameters are jointly optimized using a combination of particle swarm optimization and semidefinite relaxation (SDR) techniques. Simulation results show that by exploiting RSMA’s common stream, the proposed framework effectively mitigates both inter-user and sensing–communication interference, achieving a superior trade-off between communication rates and sensing performance and demonstrating the potential of RSMA to enhance the overall efficiency and flexibility of ISAC systems.

The integration of sensing and communication has created a demand for architectures capable of flexibly managing their strong spatial coupling. FAS-assisted ISAC has emerged as a promising solution because the spatial agility of fluid antennas enables more effective optimization of sensing and communication links than fixed-antenna systems. In [24], the authors tackle the complexity of jointly satisfying sensing and communication signal-to-noise ratio (SNR) requirements under limited antenna ports, which is an NP-hard problem arising from the strong coupling between port selection and beamforming. The authors propose an FAS-enabled ISAC framework that exploits dynamic port switching to unlock additional spatial DoFs. Simulation results demonstrate that the proposed approach achieves a 33% reduction in transmit power while ensuring sensing and communications requirements, outperforming conventional and uniformly distributed antenna schemes. However, port selection remains a fundamental challenge in FAS-assisted ISAC systems due to the combinatorial complexity of choosing the optimal subset of ports under joint sensing and communication constraints. This challenge is addressed in [25]. The authors study joint multi-port selection and precoder design for FAS-assisted multiuser MIMO downlink ISAC systems to maximize the sum-rate under sensing power constraints. Since the problem is NP-hard due to strong coupling between port selection and precoding, they propose a DRL-based solution. A constraint-aware neural precoding network is trained via primal–dual unsupervised learning, while port selection is handled by a pointer-network-based A2C framework using sum-rate as the reward. Results show near-optimal performance close to exhaustive search, over two-fold gains versus random selection, and strong robustness even with only 15% CSI, highlighting the effectiveness of DRL-enabled FAS for ISAC optimization under limited CSI.

In [26], the authors address the limitations of FPA-based ISAC systems in exploiting spatial degrees of freedom by proposing a FAS-enhanced ISAC framework. The authors jointly optimize FAS positions and dual-functional beamforming to maximize sensing SNR while ensuring minimum communication SINR per user, under FAS movement and port separation constraints. For perfect CSI, an alternating optimization (AO) algorithm using semidefinite relaxation (SDR) and successive convex approximation (SCA) is developed, while for imperfect CSI, an AO-based design with the S-Procedure and SCA handles uncertainty. Simulation results demonstrate sensing SNR gains of 8.72–178.85% while satisfying QoS. In another research effort, in [27], the authors integrate FAS with RSMA for downlink multiuser communications to improve reliability and outage performance. Users dynamically adjust FAS positions, and the BS employs RSMA signaling to flexibly manage interference. The channel gain distributions are modeled via a joint multivariate t-distribution with a copula-based formulation, enabling accurate analytical and asymptotic outage probability expressions. It is evident from the obtained results that FAS-RSMA significantly lowers outage probability and achieves higher spectral efficiency. Thus, an integration of FAS and RSMA can provide robust and efficient solution for both multi-user ISAC systems.

1.2. Contributions

Motivated by the above discussion, this paper investigates the integration of FAS into RSMA-assisted ISAC networks to overcome the challenges of spectrum scarcity, inter-user interference, and limited spatial flexibility in next-generation wireless systems. Although ISAC enhances spectrum reuse and RSMA provides robust interference management, their joint performance is still constrained by the fixed nature of conventional antennas. By offering adaptive spatial diversity through rapid port switching, FAS enables dynamic channel reshaping, stronger desired links, and more effective interference suppression. To fully exploit these advantages, we develop an FAS-enabled RSMA-ISAC framework supported by a hierarchical deep reinforcement learning (HDRL) approach that jointly optimizes beamforming, rate-splitting, and antenna port selection, thereby improving both communication efficiency and sensing performance. The key technical contributions are summarized as follows:

• We propose an FAS-assisted RSMA-enabled ISAC framework in which both the base station (BS) and users leverage FAS to dynamically optimize antenna port locations, thereby enhancing spatial diversity and interference management.
• We formulate a joint optimization problem to maximize the communication sum-rate while satisfying sensing beampattern gain and transmit power constraints by jointly designing RSMA rate-splitting ratios, transmit beamforming, and FAS port selection.
• To solve the mixed discrete–continuous optimization challenge, an HDRL framework is developed that integrates a deep Q-network (DQN) for discrete port selection with a twin delayed deep deterministic policy gradient (TD3) algorithm for continuous beamforming and rate-splitting optimization.
• Simulation results demonstrate that the proposed RSMA-based joint design substantially improves the sum rate, highlighting the powerful synergy between RSMA’s flexible interference management and the spatial adaptability of FAS in enhancing ISAC performance compared to state-of-the-art benchmark approaches.

1.3. Organization

The rest of the paper is organized as follows. Section 2 introduces the system model and the objective function. Section 3 presents the proposed algorithm. Section 4 provides analysis and insightful discussion. Finally, Section 5 concludes the paper and highlights the future research direction.

2. System Formulation

A single-cell ISAC system is considered, as illustrated in Figure 1, comprising a dual-function BS that performs both communication and sensing services, a set of K users, and a sensing target. The BS is equipped with $N\ge 2$ fluid antennas. Each is capable of switching among ${\varpi }_t$ discrete ports, denoted by ${\varsigma }_n=\{1,2,\dots ,{\varpi }_t\}$. Moreover, each user is equipped with a single fluid antenna consisting of ${\varpi }_r$ ports, where ${\varsigma }_k=\{1,2,\dots ,{\varpi }_r\}$. Both the BS and users can instantly switch their radiating elements among these ports, enabling spatial reconfigurability and improved link reliability. These fluid antennas are connected to a dedicated RF chain via integrated waveguides or flexible cables, enabling continuous reconfiguration and movement within predefined spatial regions. Let $S_t\subset {\mathbb{R}}^2$ and $S_r\subset {\mathbb{R}}^2$ denote the feasible positioning regions for the BS and users’ fluid antennas, respectively. The position of the n-th BS fluid antenna at port $\mathsf{\mu }\in {\varsigma }_n$ is denoted as $t_n^{\mathsf{\mu }}={\left[x_n^{\mathsf{\mu }},y_n^{\mathsf{\mu }}\right]}^T$ and the positions of all BS antennas are collectively represented as $\overline{t}=\left[\begin{array}{c}{t}_1^{{\mathsf{\mu }}_1},t_2^{{\mathsf{\mu }}_2},\dots ,t_N^{{\mathsf{\mu }}_N}\end{array}\right]\in {\mathbb{R}}^{2\times N}$. Similarly, the single fluid antenna at user k can switch among ${\varpi }_r$ discrete ports located within the feasible region $S_r$, where the position corresponding to port $\mathsf{\eta }\in {\varsigma }_k$ is denoted by ${\mathbf{r}}_k^{\mathsf{\eta }}={[x_k^{\mathsf{\eta }},y_k^{\mathsf{\eta }}]}^T$.

2.1. Channel Model

A far-field channel model is adopted, under the assumption that the movement range of the fluid antennas is significantly smaller than the distance between the transmitter and receiver [28]. The angles of departure (AoD) and angles of arrival (AoA) for each multipath component are considered constant.

The variation in propagation distance for the n-th BS antenna at port $\mathsf{\mu }\in {\varsigma }_n$, along the $i\in \{1,\dots ,p_t^{c_k}\}$ transmission path, is determined by

$${\mathsf{\rho }}_{t,c_k}^i\left(t_n^{\mathsf{\mu }}\right)=x_n^{\mathsf{\mu }}sin{\mathsf{\varphi }}_{t,c_k}^{i}cos{\mathsf{\psi }}_{t,c_k}^i+y_n^{\mathsf{\mu }}cos{\mathsf{\varphi }}_{t,c_k}^i,$$

(1)

where ${\mathsf{\varphi }}_{t,c_k}^i\in [0,\mathsf{\pi }]$ and ${\mathsf{\psi }}_{t,c_k}^i\in [0,\mathsf{\pi }]$ are the elevation and azimuth AoDs for the i-th path to user k, and $p_t^{c_k}$ is the total number of transmission paths from the BS to user k.

The transmit response vector for the n-th BS antenna at port $\mathsf{\mu }$ is:

$${\mathbf{e}}_{c_k}\left({\mathbf{t}}_n^{\mathsf{\mu }}\right)\triangleq \left[\begin{array}{c}{e}^{j{\displaystyle \frac{2\mathsf{\pi }}{\mathsf{\lambda }}}{\mathsf{\rho }}_{t,c_k}^1\left({\mathbf{t}}_n^{\mathsf{\mu }}\right)}\\ ⋮\\ e^{j{\displaystyle \frac{2\mathsf{\pi }}{\mathsf{\lambda }}}{\mathsf{\rho }}_{t,c_k}^{p_t^{c_k}}\left(t_n^{\mathsf{\mu }}\right)}\end{array}\right]\in {\mathbb{C}}^{p_t^{c_k}\times 1},n\in \mathcal{N}.$$

(2)

After combining the response vectors of all N BS antennas, the transmit response matrix for user k is:

$${\mathbf{E}}_{c_k}\left(\overline{\mathbf{t}}\right)\triangleq \left[\begin{array}{c}{\mathbf{e}}_{c_k}\left({\mathbf{t}}_1^{{\mathsf{\mu }}_1}\right),{\mathbf{e}}_{c_k}\left({\mathbf{t}}_2^{{\mathsf{\mu }}_2}\right),\dots ,{\mathbf{e}}_{c_k}\left({\mathbf{t}}_N^{{\mathsf{\mu }}_N}\right)\end{array}\right]\in {\mathbb{C}}^{p_t^{c_k}\times N},$$

(3)

where $p_t^{c_k}$ represents the total number of transmission paths from the BS to user k.

For the l-th receive path at user k, the propagation distance offset for user port $\mathsf{\eta }\in {\varsigma }_k$ is:

$${\mathsf{\rho }}_r^{k,l}\left({\mathbf{r}}_k^{\mathsf{\eta }}\right)=x_k^{\mathsf{\eta }}sin{\mathsf{\varphi }}_r^{k,l}cos{\mathsf{\psi }}_r^{k,l}+y_k^{\mathsf{\eta }}cos{\mathsf{\varphi }}_r^{k,l},l\in \{1,\dots ,p_r\},$$

(4)

where ${\mathsf{\varphi }}_r^{k,l}\in [0,\mathsf{\pi }]$ and ${\mathsf{\psi }}_{r_k}^l\in [0,\mathsf{\pi }]$ denote the elevation and azimuth AoA for the l-th receive path of user k, and $p_r$ is the total number of receive paths.

The corresponding receive response vector for user k at port $\mathsf{\eta }$ is:

$${\mathbf{f}}_k\left({\mathbf{r}}_k^{\mathsf{\eta }}\right)\triangleq \left[\begin{array}{c}{e}^{j{\displaystyle \frac{2\mathsf{\pi }}{\mathsf{\lambda }}}{\mathsf{\rho }}_r^{k,1}\left({\mathbf{r}}_k^{\mathsf{\eta }}\right)}\\ ⋮\\ e^{j{\displaystyle \frac{2\mathsf{\pi }}{\mathsf{\lambda }}}{\mathsf{\rho }}_r^{k,p_r}\left({\mathbf{r}}_k^{\mathsf{\eta }}\right)}\end{array}\right]\in {\mathbb{C}}^{p_r\times 1}.$$

(5)

Let ${\Sigma }_k\in {\mathbb{C}}^{p_r\times p_t^{c_k}}$ denote the path response matrix from the BS reference point ${\mathbf{t}}_0$ to user k’s reference point ${\mathbf{r}}_0$. The overall channel vector from the BS with port selection to user k at port $\mathsf{\eta }$ is:

$${\mathbf{h}}_k^{\mathsf{\eta },\mathsf{\mu }}={\mathbf{f}}_k^H\left({\mathbf{r}}_k^{\mathsf{\eta }}\right){\Sigma }_k{\mathbf{E}}_{c_k}\left(\overline{\mathbf{t}}\right),$$

(6)

where ${\mathbf{f}}_k^H\left({\mathbf{r}}_k^{\mathsf{\eta }}\right)$ is the receive steering vector and ${\mathbf{E}}_{c_k}\left(\overline{\mathbf{t}}\right)$ is the BS transmit steering matrix for the selected ports.

2.2. Communication Model

To enhance spectral efficiency and user fairness in the downlink, the system incorporates RSMA at the BS. RSMA enables joint communication and sensing by allowing flexible interference management and partial interference decoding at the receivers.

In this framework, the message $W_k$ intended for user k is split into two parts: a common message $W_{c,k}$, decodable by all users, and a private message $W_{p,k},\forall k\in \mathcal{K}$, intended specifically for user k. All common messages $\{W_{c,1},\dots ,W_{c,K}\}$ are combined into $W_c$ and encoded as $s_c$, while the private messages $\{W_{p,1},\dots ,W_{p,K}\}$ are independently encoded into private streams $\{s_{p,1},\dots ,s_{p,K}\}$.

Then, the transmit signal is defined as $s={[s_{p,1},\dots ,s_{p,K},s_c]}^T\in {\mathbb{C}}^{(K+1)\times 1}$, where $\mathbb{E}\left[s{s}^H\right]={\mathbf{I}}_{K+1}$. The signal transmitted by the BS is $x=Ws$, where $W=[{\mathrm{w}}_1,\dots ,{\mathrm{w}}_K,{\mathrm{w}}_c]\in {\mathbb{C}}^{N\times (K+1)}$ denote the beamforming matrix, which depends on the selected port. The received signal at user k port $y_k^{\mathsf{\eta }}={\left(h_k^{\mathsf{\eta },\mathsf{\mu }}\right)}^{H}x+n_k$. ${\mathbf{h}}_k^{\mathsf{\eta },\mathsf{\mu }}$ is the channel vector from the BS to user k for the selected user port $\mathsf{\eta }$ and BS port $\mathsf{\mu }$. However, for ease of notation, the channel vector $h_k^{\mathsf{\eta },\mathsf{\mu }}$ is denoted as $h_k$ throughout the remainder of the paper.

Consequently, the received signal at user k can be expressed as

$$y_k=h_k^{H}x+n_k,$$

(7)

where $h_k\in {\mathbb{C}}^{N\times 1}$ represents the channel between the BS and user k, and $n_k\sim \mathbb{C}\mathcal{N}(0,{\mathsf{\sigma }}_k^2)$ represents the AWGN at user k.

According to the RSMA decoding mechanism [29], user k first decodes the common stream $s_c$ by treating both the common and private streams as noise. Thus, SINR of the common stream at user k is evaluated by

$${\mathsf{\gamma }}_{c,k}={\displaystyle \frac{|h_k^H{\mathrm{w}}_c{|}^2}{{\displaystyle \sum _{i=1}^K}|h_k^H{\mathrm{w}}_i{|}^2+{\mathsf{\sigma }}_k^2}},$$

(8)

Then, the achievable rate for user k is evaluated by

$$R_{c,k}={log}_2\left(1+{\mathsf{\gamma }}_{c,k}\right).$$

(9)

After successfully decoding and removing $s_c$ via SIC, user k can decode the private data stream $s_{p,k}$. The SINR of user k private stream is determined as:

$${\mathsf{\gamma }}_{p,k}={\displaystyle \frac{{\left|{\mathbf{h}}_k^H{\mathbf{w}}_k\right|}^2}{{\displaystyle \sum _{i=1,i\ne k}^K}{\left|{\mathbf{h}}_k^H{\mathbf{w}}_i\right|}^2+{\mathsf{\sigma }}_k^2}}.$$

(10)

Then, the achievable rate of user k is determined by

$$R_{p,k}={log}_2\left(1+{\mathsf{\gamma }}_{p,k}\right)$$

(11)

To ensure successful decoding of the common stream $s_c$ by all users, the transmission rate of the common stream must not exceed the achievable decoding rate at any user. For instance, when decoding $s_c$, user k achieves an achievable rate $R_{c,k}$ determined by its received SINR. Since the same common stream is decoded by all users, its transmission rate must satisfy

$$R_c\le R_{c,k},\forall k\in \mathcal{K}.$$

(12)

Let $r_{c,k}$ denote the portion of the common information $W_{c,k}$ intended for user k. All such portions are multiplexed into the common stream $s_c$, whose total transmission rate is determined by

$$R_c=\sum _{k=1}^K{r}_{c,k}.$$

(13)

Accordingly, to guarantee successful decoding of the common stream at all users, the common-rate allocation must satisfy

$$\sum _{k=1}^K{r}_{c,k}\le R_{c,k},\forall k\in \mathcal{K}.$$

(14)

The achievable rate of user k is therefore expressed as

$$R_k=R_{p,k}+r_{c,k}.$$

(15)

We adopt the sum rate as the communication performance metric, which is given by

$$R=\sum _{k=1}^K(R_{p,k}+r_{c,k})=\sum _{k=1}^K{R}_{p,k}+R_c.$$

(16)

2.3. Sensing Model

As shown in Figure 1, the detection signal reaches a single target through the direct channel $h_r\in {\mathbb{C}}^{N\times 1}$. The echo signal returns to the BS along the same path. Since the BS employs an FAS with N antennas, and each antenna can select a port ${\mathsf{\mu }}_n\in {\varsigma }_n$, the equivalent target channel depends on the BS port configuration. Hence, the received signal at the BS is determined by

$${\mathrm{y}}_{\mathrm{s}}{=\mathrm{A}}_{\mathrm{r}}^{\mathsf{\mu }}x+n,$$

(17)

where $A_r^{\mathsf{\mu }}=h_r^{\mathsf{\mu }}{\left(h_r^{\mathsf{\mu }}\right)}^H\in {\mathbb{C}}^{N\times N}$ represents the sensing channel for the selected BS ports. $h_r^{\mathsf{\mu }}$ represents the BS to target channel for the selected port configuration $\mathsf{\mu }$. $n\sim \mathcal{CN}(0,{\mathsf{\sigma }}_s^2{I}_N)$ represents the additive Gaussian noise with variance ${\mathsf{\sigma }}_s^2{I}_N$. The probability of target detection improves monotonically with the sensing signal-to-noise ratio (SNR), which is evaluated as

$${\mathsf{\gamma }}_r={\displaystyle \frac{\mathrm{Tr}\left(W^H{\left(A_r^{\mathsf{\mu }}\right)}^H{A}_r^{\mathsf{\mu }}W\right)}{N{\mathsf{\sigma }}_s^2}}.$$

(18)

2.4. Objective Function

The objective is to maximize the system sum-rate in an FAS-enabled RSMA-assisted ISAC system while ensuring that the sensing power toward the target direction satisfies a minimum required threshold. This design reflects a practical ISAC operation in which communication performance is optimized under sensing quality constraints. The optimization variables include the transmit beamforming matrix W, the BS port-selection vector $\mathsf{\mu }$, and the user port-selection vector $\mathsf{\eta }$. These variables jointly determine the allocation of spatial resources and the decoding performance of both the common and private RSMA streams. Accordingly, the joint optimization problem is formulated as:

$$\begin{array}{cc}\hfill & \underset{W,\mathsf{\mu },\mathsf{\eta }}{max}R\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathsf{\mu }}_n\in {\varsigma }_n,\forall n\in \mathcal{N},\hfill \end{array}$$

(19a)

$$\begin{array}{cc}\hfill & {\mathsf{\eta }}_k\in {\varsigma }_k,\forall k\in \mathcal{K},\hfill \end{array}$$

(19b)

$$\begin{array}{cc}\hfill & \parallel {\mathbf{t}}_n^{{\mathsf{\mu }}_n}-{\mathbf{t}}_v^{{\mathsf{\mu }}_v}{\parallel }_2\ge D,\forall n,v\in \mathcal{N},n\ne v,\hfill \end{array}$$

(19c)

$$\begin{array}{cc}\hfill & \mathrm{Tr}\left(W{W}^H\right)\le P_{max},\hfill \end{array}$$

(19d)

$$\begin{array}{cc}\hfill & {\mathsf{\gamma }}_{\mathrm{r}}\ge {\mathsf{\gamma }}_{min},\hfill \end{array}$$

(19e)

$$\begin{array}{cc}\hfill & \sum _{k\in \mathcal{K}}{r}_{c,k}\le R_{c,k},\forall k\in \mathcal{K},\hfill \end{array}$$

(19f)

$$\begin{array}{cc}\hfill & r_{c,k}\ge 0,\forall k\in \mathcal{K}.\hfill \end{array}$$

(19g)

The optimization problem in (19) aims to maximize the system sum-rate by jointly optimizing the transmit beamforming matrix W, the BS port-selection vector $\mathsf{\mu }$, and the user port-selection vector $\mathsf{\eta }$. Constraint (19a) ensures that each BS fluid antenna selects a valid port from its available discrete port set ${\varsigma }_n=\{1,2,\dots ,{\varpi }_t\}$, while constraint (19b) enforces that each user selects one of its available ports from ${\varsigma }_k=\{1,2,\dots ,{\varpi }_r\}$. Constraint (19c) guarantees a minimum Euclidean separation D between any two active BS antenna ports to limit mutual coupling effects and maintain antenna performance. Constraint (19d) enforces the maximum transmit power budget $P_{max}$ at the BS by limiting the total radiated power of all beams. Constraint (19e) ensures that the sensing functionality of the ISAC system is preserved by enforcing a minimum sensing quality requirement. Constraint (19f) enforces the RSMA common-stream decodability condition by ensuring that the total allocated common rate does not exceed the achievable common-stream decoding rate at any user. This guarantees that the common stream can be successfully decoded by all users. Constraint (19g) ensures the non-negativity of the common-rate allocation variables. Due to the discrete port selections at the BS and users, combined with the non-linear dependence of the sum-rate on beamforming and port indices as well as the coupled rate and separation constraints, the optimization problem is highly non-convex. To address this challenge, an HDRL algorithm is deployed to efficiently learn policies for maximizing the system sum-rate while satisfying both communication and sensing requirements.

3. Hierarchical Deep Reinforcement Learning (HDRL) Approach

Conventional RL algorithms are insufficient in this setting due to their limited scalability when dealing with high-dimensional state–action spaces and their inability to simultaneously manage discrete and continuous decision variables. To address these limitations, we adopt an HDRL approach that integrates DQN with the TD3 algorithm. In this novel design, DQN handles discrete actions, while TD3 manages continuous and high-dimensional action spaces. The DQN manages FAS port selection by outputting discrete Q-values for all candidate ports and selecting the port that maximizes the reward, while its exploration enables efficient and robust port switching. Afterwards, the RSMA–ISAC resource-allocation variables are continuous that benefit from deterministic policy-gradient optimization. For this purpose, TD3 is employed to improve stability through its clipped double-Q learning, target policy smoothing, and delayed actor updates, enabling efficient refinement of continuous physical-layer parameters. The combination of DQN and TD3 enables shared experience replay, improved sample efficiency, and coordinated updates across decision layers. This hierarchical decomposition reduces action dimensionality, preserves the mathematical structure of both discrete and continuous decisions, stabilizes training, and prevents the quantization loss or infeasible actions that arise when continuous-control algorithms are forced onto discrete problems.

The optimization problem is reformulated as a Markov decision process (MDP), which facilitates its solution within a reinforcement learning framework. The MDP is modeled as a 4-tuple $(s_t,a_t,r_t,s_{t+1})$, where $s_t$ denotes the current state, $a_t$ the selected action, $r_t$ the immediate reward, and $s_{t+1}$ the resulting state. At each time step t, the agent observes $s_t\in S$ and selects $a_t$ based on its policy to interact with the environment.

State: Owing to the large number of available fluid antenna ports at both the BS and the users, directly incorporating the channel gains of all ports into the state representation would result in a prohibitively high-dimensional state space, thereby hindering the training efficiency and convergence of the DRL model. To mitigate this issue, we exploit the inherent spatial correlation among adjacent ports and construct a reduced state representation by uniformly sampling the channel gains from a representative subset of ports. This approach significantly lowers the dimensionality of the state space while preserving the essential channel characteristics required for accurate decision-making. The state space at time t includes the distances between the BS and users, the SINR of the common and private streams, and the active antenna port positions, and is represented as $s_t=\left[[d_1,\dots ,d_K],[{\mathsf{\gamma }}_{c,1},\dots ,{\mathsf{\gamma }}_{c,K}],[{\mathsf{\gamma }}_{p,1},\dots ,{\mathsf{\gamma }}_{p,K}],[x_n^{{\mathsf{\mu }}_n},y_n^{{\mathsf{\mu }}_n}],[x_k^{{\mathsf{\eta }}_k},y_k^{{\mathsf{\eta }}_k}]\right]$, where ${\mathsf{\mu }}_n\in {\varsigma }_n=\{1,2,\dots ,{\varpi }_t\}$ denotes the selected port index of the n-th BS fluid antenna, and ${\mathsf{\eta }}_k\in {\varsigma }_k=\{1,2,\dots ,{\varpi }_r\}$ denotes the selected port index of user k’s fluid antenna. The corresponding 2D positions of the active ports are given by ${\mathbf{t}}_n^{{\mathsf{\mu }}_n}={[x_n^{{\mathsf{\mu }}_n},y_n^{{\mathsf{\mu }}_n}]}^T$ for the BS antennas and ${\mathbf{r}}_k^{{\mathsf{\eta }}_k}={[x_k^{{\mathsf{\eta }}_k},y_k^{{\mathsf{\eta }}_k}]}^T$ for the users.

Action: The action space is designed to jointly capture the port selection of the fluid antennas at both the BS and users, as well as the transmit power allocation strategy. The first component corresponds to the port indices of the BS fluid antennas, denoted by ${\mathsf{\mu }}_{BS}=[{\mathsf{\mu }}_1,\dots ,{\mathsf{\mu }}_N]$, where ${\mathsf{\mu }}_n\in {\varsigma }_n=\{1,2,\dots ,{\varpi }_t\}$ indicates the active port of the n-th BS antenna. The second component, ${\mathsf{\eta }}_u=[{\mathsf{\eta }}_1,\dots ,{\mathsf{\eta }}_K]$, represents the port indices of the user-side fluid antennas, where ${\mathsf{\eta }}_k\in {\varsigma }_k=\{1,2,\dots ,{\varpi }_r\}$ selects the active port of user k. The third component, $[p_1^c,\dots ,p_N^c]$, specifies the power allocated to the common stream from each BS antenna port ${\mathsf{\mu }}_n$, while the fourth component, $[p_{11},\dots ,p_{NK}]$, defines the power allocation from each BS antenna port ${\mathsf{\mu }}_n$ to each user port ${\mathsf{\eta }}_k$ for the private streams. Therefore, the overall action space is given by ${\mathbf{a}}_t=\left[[{\mathsf{\mu }}_1,\dots ,{\mathsf{\mu }}_N],[{\mathsf{\eta }}_1,\dots ,{\mathsf{\eta }}_K],[p_1^c,\dots ,p_N^c],[p11,\dots ,p_{NK}]\right]$, which allows the agent to autonomously adapt both antenna port selections and power allocation, thereby enhancing spectral efficiency and overall system performance.

Reward: The reward function is designed to be directly aligned with the system objective, namely, the maximization of the sum-rate. In reinforcement learning, the reward serves as the feedback signal that guides the agent’s policy updates; hence, it must be strongly correlated with the performance metric of interest. To this end, the agent receives a positive reward proportional to the achievable system sum-rate whenever all optimization constraints (19a)–(19e) are satisfied. Conversely, if any constraint is violated, the agent is penalized with a zero reward. Formally, the reward at time step t is expressed as follows:

$$r_t=\left\{\begin{array}{cc}R,\hfill & \mathrm{if} (19a)–(19g) \mathrm{are}\mathrm{satisfied},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(20)

This formulation ensures that the agent is incentivized to learn policies that not only maximize the sum-rate but also strictly adhere to the feasibility conditions of the optimization problem. The sensing requirement is enforced through the constraint-aware reward design. Specifically, the agent receives the achieved communication sum-rate when all system constraints, including the sensing power constraint, are satisfied; otherwise, the reward is set to zero. This mechanism ensures that the learning agent favors actions that simultaneously satisfy both communication and sensing requirements while maximizing communication performance.

The core idea of the proposed HDRL approach is to decompose the system action space into discrete and continuous components. The discrete action $a_d$, corresponding to FAS port selection, is optimized using a DQN. Specifically, the DQN approximates the discrete action-value function $Q(s,a_d)$ with a neural network parameterized by ${\mathsf{\theta }}_d$, enabling the agent to evaluate and select the most promising port configuration. At each time step, the DQN updates its parameters by minimizing a TD loss that measures the discrepancy between predicted and target Q-values: The loss function for training the value network is defined as

$$L_d\left({\mathsf{\theta }}_d\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(Q_d(s_i,a_{d,i};{\mathsf{\theta }}_d)-y_i\right)}^2,$$

(21)

where $y_i$ denotes the TD target constructed from sampled transitions in the replay buffer. This loss represents the mean squared error (MSE) between the predicted Q-values from the online network and the target values $y_i$. By penalizing large deviations between predicted and target Q-values, it encourages the network to iteratively approximate the optimal action-value function.

The target value $y_i$ is evaluated using the target Q-network with parameters ${\mathsf{\theta }}_d^{\prime }$, which is a delayed copy of the online Q-network:

$$y_i=r_i+\mathsf{\gamma }\underset{a_d^{\prime }}{max}{Q}_d(s_{i+1},a_d^{\prime };{\mathsf{\theta }}_d^{\prime }),$$

(22)

where ${\mathsf{\theta }}_d^{\prime }$ is updated as ${\mathsf{\theta }}_d^{\prime }\leftarrow {\mathsf{\theta }}_d$ every $t_d$ steps, with $t_d$ being the delay interval. This delay mechanism reduces the risk of divergence by decoupling target computation from rapid online weight updates.

The TD error, defined as the discrepancy between the target and predicted Q-values, is expressed as:

$${\mathsf{\delta }}_i=y_i-Q_d(s_i,a_{d,i};{\mathsf{\theta }}_d).$$

(23)

The gradient of the loss function with respect to the network parameters ${\mathsf{\theta }}_d$ is obtained as:

$${\nabla }_{{\mathsf{\theta }}_d}{L}_d\left({\mathsf{\theta }}_d\right)={\displaystyle \frac{2}{N}}\sum _{i=1}^N{\mathsf{\delta }}_i{\nabla }_{{\mathsf{\theta }}_d}{Q}_d(s_i,a_{d,i};{\mathsf{\theta }}_d),$$

(24)

The weight update rule for the Q-network follows the standard stochastic gradient descent (SGD) form:

$${\mathsf{\theta }}_d\leftarrow {\mathsf{\theta }}_d-{\mathsf{\eta }}_d{\nabla }_{{\mathsf{\theta }}_d}{L}_d\left({\mathsf{\theta }}_d\right),$$

(25)

where ${\mathsf{\eta }}_d$ denotes the learning rate of the Q-network.

Consequently, the policy of the outer-loop DQN procedure is evaluated as:

$${\mathsf{\pi }}_d\left(s_t\right)=arg\underset{a_d}{max}{Q}_d(s_t,a_d;{\mathsf{\theta }}_d),$$

(26)

which ensures that the selected discrete action maximizes the estimated long-term return at the current state. This discrete action serves as the input for the inner-loop TD3 module, enabling coordinated optimization over both discrete and continuous control spaces.

The TD3 algorithm is an enhanced variant of the Deep Deterministic Policy Gradient (DDPG) method, designed to improve stability and reduce overestimation bias in continuous action space learning.

3.1. Critic Network Update

During training, a mini-batch of $N_B$ transition tuples ${\left\{(s_i,a_i,r_i,s_{i+1})\right\}}_{i=1}^{N_B}$ is sampled uniformly from the experience replay buffer $\mathcal{D}$. For the i-th tuple, the target Q-value is computed as:

$$y_i=r_i+\mathsf{\gamma }{Q}^{\prime }\left(s_{i+1},{\mathsf{\pi }}^{\prime }(s_{i+1};{\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }});{\mathsf{\theta }}_{Q^{\prime }}\right),$$

(27)

where $r_i$ denotes the immediate reward obtained by executing action $a_i$ in state $s_i$, $\mathsf{\gamma }\in [0,1]$ is the discount factor, and ${\mathsf{\pi }}^{\prime }(s_{i+1};{\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }})$ denotes the deterministic target policy (actor) parameterized by ${\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }}$. The function $Q^{\prime }(·,·;{\mathsf{\theta }}_{Q^{\prime }})$ represents the target critic network parameterized by ${\mathsf{\theta }}_{Q^{\prime }}$. The parameters ${\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }}$ and ${\mathsf{\theta }}_{Q^{\prime }}$ are delayed (soft-updated) copies of the corresponding online actor and critic network parameters, respectively.

The critic network parameters ${\mathsf{\theta }}_Q$ are updated by minimizing the mean squared Bellman error (MSBE) between the predicted Q-value and the target value:

$$L_Q\left({\mathsf{\theta }}_Q\right)={\displaystyle \frac{1}{N_B}}\sum _{i=1}^{N_B}{\left(Q(s_i,a_i;{\mathsf{\theta }}_Q)-y_i\right)}^2.$$

(28)

The gradient of the loss with respect to ${\mathsf{\theta }}_Q$ is:

$${\nabla }_{{\mathsf{\theta }}_Q}{L}_Q={\displaystyle \frac{2}{N_B}}\sum _{i=1}^{N_B}\left(Q(s_i,a_i;{\mathsf{\theta }}_Q)-y_i\right){\nabla }_{{\mathsf{\theta }}_Q}Q(s_i,a_i;{\mathsf{\theta }}_Q),$$

(29)

and the parameters are updated via gradient descent:

$${\mathsf{\theta }}_Q\leftarrow {\mathsf{\theta }}_Q-{\mathsf{\eta }}_Q{\nabla }_{{\mathsf{\theta }}_Q}{L}_Q,$$

(30)

where ${\mathsf{\eta }}_Q$ is the learning rate of the critic network.

3.2. Actor Network Update

To improve the quality of the learned policy, the actor network is optimized such that its output action maximizes the expected Q-value as estimated by the critic network. The actor parameters ${\mathsf{\theta }}_{\mathsf{\pi }}$ are updated by performing gradient ascent on the expected return:

$${\nabla }_{{\mathsf{\theta }}_{\mathsf{\pi }}}J\left({\mathsf{\theta }}_{\mathsf{\pi }}\right)={\displaystyle \frac{1}{N_B}}\sum _{i=1}^{N_B}{\nabla }_{a}Q(s_i,a;{\mathsf{\theta }}_Q){|}_{a=\mathsf{\pi }(s_i;{\mathsf{\theta }}_{\mathsf{\pi }})}{\nabla }_{{\mathsf{\theta }}_{\mathsf{\pi }}}\mathsf{\pi }(s_i;{\mathsf{\theta }}_{\mathsf{\pi }}),$$

(31)

where:

• $\mathsf{\pi }(s_i;{\mathsf{\theta }}_{\mathsf{\pi }})$ is the deterministic policy output by the actor network,
• $Q(s_i,a;{\mathsf{\theta }}_Q)$ is the critic’s Q-value estimation,
• ${\mathsf{\theta }}_Q$ and ${\mathsf{\theta }}_{\mathsf{\pi }}$ are the critic and actor network parameters, respectively.

The actor parameters are then updated using gradient ascent:

$${\mathsf{\theta }}_{\mathsf{\pi }}\leftarrow {\mathsf{\theta }}_{\mathsf{\pi }}+{\mathsf{\eta }}_{\mathsf{\pi }}{\nabla }_{{\mathsf{\theta }}_{\mathsf{\pi }}}J\left({\mathsf{\theta }}_{\mathsf{\pi }}\right),$$

(32)

where ${\mathsf{\eta }}_{\mathsf{\pi }}$ is the actor learning rate.

To ensure stable learning, the target critic and target actor networks are updated via a soft update mechanism, preventing large oscillations in target values. The target critic parameters ${\mathsf{\theta }}_{Q^{\prime }}$ are updated as:

$${\mathsf{\theta }}_{Q^{\prime }}\leftarrow \mathsf{\tau }{\mathsf{\theta }}_Q+(1-\mathsf{\tau }){\mathsf{\theta }}_{Q^{\prime }},$$

(33)

and the target actor parameters ${\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }}$ are updated analogously:

$${\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }}\leftarrow \mathsf{\tau }{\mathsf{\theta }}_{\mathsf{\pi }}+(1-\mathsf{\tau }){\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }},$$

(34)

where $\mathsf{\tau }\in (0,1]$ is the Polyak averaging coefficient.

3.3. Exploration Strategy

To ensure adequate exploration in the continuous action space, the deterministic policy output is perturbed with zero-mean Gaussian noise:

$$a_t=\mathsf{\pi }(s_t;{\mathsf{\theta }}_{\mathsf{\pi }})+\mathcal{N}(0,{\mathsf{\sigma }}_t^2),$$

(35)

where ${\mathsf{\sigma }}_t=\upsilon e^{-\kappa t}$ controls the exploration variance, $\upsilon$ is the initial noise scale, and $\kappa$ is a decay constant controlling the rate at which exploration noise decreases over time.

However, overestimation bias can lead to the accumulation of approximation errors over time. Specifically, inaccuracies in the Q-value function may result in the agent assigning artificially high values to suboptimal state–action pairs, thereby producing a suboptimal policy. The primary objective of TD3 is to mitigate such function approximation errors, which can lead to overestimation of Q-value functions and degrade policy performance in the DDPG algorithm [30]. TD3 employs a dual-critic architecture to address this issue, consisting of three main neural networks: one actor network and two critic networks. The actor network comprises:

• Policy network parameterized by ${\mathsf{\theta }}_{\mathsf{\pi }}$, outputs the continuous control action $a_c=\mathsf{\pi }(s;{\mathsf{\theta }}_{\mathsf{\pi }})$ for a given state s.
• Target policy network parameterized by ${\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }}$, is a delayed copy of the policy network used for stable target value computation.

Furthermore, the critic network consists of:

• Main Q-networks $Q_1(s,a;{\mathsf{\theta }}_{Q_1})$ and $Q_2(s,a;{\mathsf{\theta }}_{Q_2})$, estimate the action-value function for a given state–action pair.
• Target Q-networks $Q_1^{\prime }(s,a;{\mathsf{\theta }}_{Q_1^{\prime }})$ and $Q_2^{\prime }(s,a;{\mathsf{\theta }}_{Q_2^{\prime }})$ are delayed copies of the main Q-networks and used for computing target values in the Bellman updates.

This network structure decouples target Q-value computation from action selection, thereby reducing the correlation between estimated and target values. To further suppress overestimation bias, TD3 introduces several key modifications:

• Clipped Double Q-Learning: Uses the minimum value of the two target Q-networks, i.e.,

  $$y_i=r_i+\mathsf{\gamma }min\left(Q_1^{\prime }(s_{i+1},{\tilde{a}}_{i+1}),Q_2^{\prime }(s_{i+1},{\tilde{a}}_{i+1})\right),$$

  (36)

  where ${\tilde{a}}_{i+1}$ is the target action from the smoothed target policy.
• Target Policy Smoothing: Adds small clipped noise to the target action before evaluating it with the target critics to prevent exploitation of Q-function errors.
• Delayed Policy Updates: Updates the actor and target networks less frequently than the critics, stabilizing policy learning.

TD3 maintains two independent critic networks, $Q_1(s,a;{\mathsf{\theta }}_{Q_1})$ and $Q_2(s,a;{\mathsf{\theta }}_{Q_2})$, to reduce overestimation. The target Q-value is computed using the smaller of the two critic estimates:

$$y_i=r_i+\mathsf{\gamma }\underset{j=1,2}{min}{Q}_j^{\prime }\left(s_{i+1},{\mathsf{\pi }}^{\prime }(s_{i+1};{\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }});{\mathsf{\theta }}_{Q_j^{\prime }}\right),$$

(37)

where $({\mathsf{\theta }}_{Q_1^{\prime }},{\mathsf{\theta }}_{Q_2^{\prime }})$ are target critic parameters.

To prevent the actor from being trained on inaccurate value estimates, TD3 updates the policy network and target networks less frequently than the critics. Specifically, for every d critic updates, one policy update is performed.

To reduce exploitation of Q-function peaks caused by function approximation errors, TD3 applies clipped Gaussian noise to the target policy actions:

$$\tilde{a}={\mathsf{\pi }}^{\prime }(s_{i+1};{\mathsf{\theta }}_{{\mathsf{\pi }}^{\prime }})+\mathrm{clip}\left(ϵ,-c,c\right),$$

(38)

where $ϵ\sim \mathcal{N}(0,{\mathsf{\sigma }}^2)$ is Gaussian noise and $c>0$ is the noise clipping threshold. This regularization smooths the Q-value landscape by preventing abrupt policy changes in narrow action regions.

The proposed HDRL algorithm is summarized in Algorithm 1. The training procedure begins with the initialization phase, where the maximum number of episodes and time steps per episode are specified, a unified replay buffer for both the DQN and TD3 modules is established, and the parameters of the DQN, actor, and critic networks, together with their corresponding target networks, are randomly initialized to enable stable learning. Training proceeds over multiple episodes, each of which resets the environment with new channel conditions, antenna positions, and RSMA variables. Within each episode, the algorithm iterates through time steps, where in every step, the outer DQN module first selects a discrete action corresponding to the BS antenna port based on the observed system state by greedily choosing the action with the highest Q-value. Conditioned on this discrete choice, the inner TD3 module generates continuous control actions, including user FAS displacements, RSMA power allocation, and beamforming vectors, with additional Gaussian noise injected for exploration. These actions are executed in the environment, which produces the next state, achievable communication rates, and sensing metrics. A reward is then computed based on the achieved communication sum-rate, and it is set to zero if any feasibility constraints, such as transmit power or antenna separation, are violated. The resulting transition is stored in the replay buffer, enabling off-policy learning. Subsequently, the DQN network is updated by minimizing the temporal-difference error using sampled mini-batches, while the TD3 actor and critic networks are trained with mini-batch updates incorporating clipped double Q-learning, target policy smoothing, and delayed policy updates to enhance stability. Target networks for both modules are softly updated to track the learned networks. This process continues until the episode concludes, after which the algorithm advances to the next episode. Upon completion of all episodes, the algorithm yields a hierarchical policy consisting of the trained DQN for BS antenna port selection and the TD3 for continuous optimization of user antenna positions and RSMA power allocation, thereby solving the hybrid ISAC optimization problem.

3.4. Complexity Analysis

The computational complexity of the proposed HDRL algorithm can be analyzed by considering both the DQN and TD3 components. For the DQN, which handles discrete BS antenna port selection, the state dimension includes channel and system information, and the action space consists of selecting one of M antenna ports. Each forward pass through the DQN network has a complexity of $\mathcal{O}\left(\right|{\mathsf{\theta }}_d\left|\right)$, where $|{\mathsf{\theta }}_d|$ denotes the total number of network parameters. Performing a gradient update using a mini-batch of size B requires $\mathcal{O}(B·|{\mathsf{\theta }}_d\left|\right)$. Across N time steps in an episode, the total DQN computational cost is therefore $\mathcal{O}\left(NB\right|{\mathsf{\theta }}_d\left|\right)$. Furthermore, the TD3 component, responsible for continuous control of the users’ FAS positions and RSMA variables, processes a state vector including the positions of all K users and relevant RSMA parameters. The action dimension reflects user FAS displacements and power allocation variables. Each forward pass through the actor and twin critics has a complexity of $\mathcal{O}\left(\right|{\mathsf{\theta }}_{\mathsf{\pi }}|+|{\mathsf{\theta }}_{Q1}|+|{\mathsf{\theta }}_{Q2}\left|\right)$, and gradient updates on a mini-batch of size B require $\mathcal{O}(B·(|{\mathsf{\theta }}_{\mathsf{\pi }}|+|{\mathsf{\theta }}_{Q1}|+|{\mathsf{\theta }}_{Q2}\left|\right))$. Over N time steps, this results in a TD3 computational cost of $\mathcal{O}\left(NB\right(|{\mathsf{\theta }}_{\mathsf{\pi }}|+|{\mathsf{\theta }}_{Q1}|+|{\mathsf{\theta }}_{Q2}\left|\right))$ per episode. Combining both components, the overall computational complexity of HDRL per episode is $\mathcal{O}\left(NB\left(|{\mathsf{\theta }}_d|+|{\mathsf{\theta }}_{\mathsf{\pi }}|+|{\mathsf{\theta }}_{Q1}|+|{\mathsf{\theta }}_{Q2}|\right)\right)$ and for $T_{\max }$ episodes, it scales as $\mathcal{O}\left(T_{\max }NB\left(|{\mathsf{\theta }}_d|+|{\mathsf{\theta }}_{\mathsf{\pi }}|+|{\mathsf{\theta }}_{Q1}|+|{\mathsf{\theta }}_{Q2}|\right)\right)$. The complexity of the DQN network scales linearly with the number of BS antennas M due to the output layer, while the TD3 network scales roughly linearly with the number of users K because the action dimensions increase with user FAS displacements and power allocations. The mini-batch size B dominates the gradient computation cost in both networks. This analysis demonstrates how system parameters such as K, M, and network sizes influence the overall computational effort required for training the HDRL algorithm.

Algorithm 1 Hierarchical Deep Reinforcement Learning (HDRL) Approach

1: Initialize: DQN network weights ${\mathsf{\theta }}_d$, TD3 actor network $\mathsf{\pi }$ weights ${\mathsf{\theta }}_{\mathsf{\pi }}$, TD3 twin critic networks $Q_1,Q_2$ weights ${\mathsf{\theta }}_{Q1},{\mathsf{\theta }}_{Q2}$, Target networks ${\mathsf{\theta }}_d^{\prime }$, ${\mathsf{\theta }}_{\mathsf{\pi }}^{\prime }$, ${\mathsf{\theta }}_{Q1}^{\prime }$, ${\mathsf{\theta }}_{Q2}^{\prime }$, Replay buffer D, Exploration parameters: $ϵ$ for DQN, noise process for TD3   2: for each episode do   3:    Initialize environment and get initial state $s_0$   4:    for each time step do   5:      Compute Q-values: $Q\_values=\mathrm{DQN}(s_t;{\mathsf{\theta }}_d)$   6:      Select discrete action $a_d=arg{max}_{a}Q\left[a\right]$ (port selection)   7:      Actor continuous action $a_c=\mathsf{\pi }(s_t,a_d;{\mathsf{\theta }}_{\mathsf{\pi }})$   8:      Add exploration noise: $a_c^{noisy}=a_c+{\mathrm{noise}}_t$   9:      Full action $a_t=(a_d,a_c^{noisy})$ 10:      Apply antenna port switching per $a_d$ 11:      Apply power, beamforming, RSMA parameters per $a_c^{noisy}$ 12:      Observe next state $s_{t+1}$ and reward $r_t$ 13:      Store $(s_t,a_t,r_t,s_{t+1})$ in replay buffer D 14:      Sample minibatch $\left\{(s_i,a_i,r_i,s_{i+1})\right\}$ from D 15:      for each sample do 16:         Compute target: $$y_i=r_i+\mathsf{\gamma }\underset{a_d^{\prime }}{max}{Q}_d(s_{i+1},a_d^{\prime };{\mathsf{\theta }}_d^{\prime })$$ 17:      end for 18:      Update DQN weights ${\mathsf{\theta }}_d$ by minimizing loss: $$L_d={\displaystyle \frac{1}{N}}\sum {\left(Q_d(s_i,a_{d,i};{\mathsf{\theta }}_d)-y_i\right)}^2$$ 19:      for each sample do 20:         Compute target actions with target actor + clipped noise:         $a_{c,i}^{\prime }={\mathsf{\pi }}^{\prime }(s_{i+1},a_{d,i+1};{\mathsf{\theta }}_{\mathsf{\pi }}^{\prime })+$ clipped_noise 21          Compute target Q-values: $$\begin{array}{cc}\hfill y_i=& \hfill r_i+\mathsf{\gamma }min(Q_1^{\prime }(s_{i+1},a_{d,i+1},a_{c,i}^{\prime }),\\ & \hfill Q_2^{\prime }(s_{i+1},a_{d,i+1},a_{c,i}^{\prime }))\end{array}$$ 22:      end for 23:      Update critics ${\mathsf{\theta }}_{Q1},{\mathsf{\theta }}_{Q2}$ by minimizing MSE loss on $y_i$ 24:      Update actor ${\mathsf{\theta }}_{\mathsf{\pi }}$ by policy gradient to maximize $$Q_1(s_i,a_{d,i},\mathsf{\pi }(s_i,a_{d,i}))$$ 25:      Update target networks softly: $$\begin{array}{c}{\mathsf{\theta }}_d^{\prime }\leftarrow \mathsf{\tau }{\mathsf{\theta }}_d+(1-\mathsf{\tau }){\mathsf{\theta }}_d^{\prime }\\ {\mathsf{\theta }}_{\mathsf{\pi }}^{\prime }\leftarrow \mathsf{\tau }{\mathsf{\theta }}_{\mathsf{\pi }}+(1-\mathsf{\tau }){\mathsf{\theta }}_{\mathsf{\pi }}^{\prime }\\ \hfill {\mathsf{\theta }}_{Q1}^{\prime }\leftarrow \mathsf{\tau }{\mathsf{\theta }}_{Q1}+(1-\mathsf{\tau }){\mathsf{\theta }}_{Q1}^{\prime }\\ \hfill {\mathsf{\theta }}_{Q2}^{\prime }\leftarrow \mathsf{\tau }{\mathsf{\theta }}_{Q2}+(1-\mathsf{\tau }){\mathsf{\theta }}_{Q2}^{\prime }\end{array}$$ 26:      Update state $s_t\leftarrow s_{t+1}$ 27:      Decay $ϵ$ over time 28:    end for 29: end for

Finally, it is important to note that the inference stage is significantly less computationally demanding than training. During deployment, the learned policies are executed through forward passes of the trained networks without performing gradient updates or replay buffer operations. Specifically, the DQN module selects the antenna port configuration, and the TD3 actor network generates the continuous control variables. Therefore, the inference-time complexity can be approximated as $\mathcal{O}\left(\right|{\mathsf{\theta }}_d|+|{\mathsf{\theta }}_{\mathsf{\pi }}\left|\right)$, which is sufficiently lightweight for real-time decision-making in practical wireless communication systems.

4. Performance Evaluation

In the simulations, the elevation and azimuth angles are modeled as independent and identically distributed random variables uniformly drawn from $\left[0,\mathsf{\pi }\right]$. The distance between adjacent fluid antenna ports is fixed to $\mathsf{\lambda }/2$, while the fluid antenna movement is constrained within a square region of size $\varpi \times \varpi$, where $\varpi =4\mathsf{\lambda }$. The path response matrix is assumed to be diagonal, i.e., $\Sigma =diag({\Sigma }_{1,1},{\Sigma }_{2,2},\dots ,{\Sigma }_{p_r,p_r})$. The diagonal elements follow complex Gaussian distributions such that ${\Sigma }_{1,1}\sim \mathcal{CN}\left(0,{\displaystyle \frac{\mathsf{\tau }}{\mathsf{\tau }+1}}\right)$ and ${\Sigma }_{l,l}\sim \mathcal{CN}\left(0,{\displaystyle \frac{1}{(\mathsf{\tau }+1)(p_t-1)}}\right)$ for $l=2,3,\dots ,p_t$, where $\mathsf{\tau }\ge 0$ denotes the Rician factor representing the power ratio between the line-of-sight (LoS) and non-line-of-sight (NLoS) components. In this work, $\mathsf{\tau }$ is set to 1. The numbers of transmit and receive paths are both set to 3. The maximum transmit signal-to-noise ratio (SNR) at the base station (BS) is defined as $P_{max}/{\mathsf{\sigma }}_w^2=5\mathrm{dB}$, while the sensing constraint is characterized by $\Gamma /{\mathsf{\sigma }}_w^2=9\mathrm{dB}$. Moreover, the number of BS antennas is set to $N=4$. The proposed HDRL framework integrates a DQN and a TD3 algorithm in a hierarchical manner. The DQN consists of a three-layer fully connected neural network with two hidden layers of 128 neurons each and ReLU activation functions. The TD3 actor network employs two hidden layers with 256 neurons each, followed by a Tanh activation to constrain the continuous action space. The TD3 critic adopts twin Q-networks with identical architectures to mitigate overestimation bias. During training, the system is trained for 1000 episodes, each consisting of 12 interaction steps. A replay buffer of size 500 is utilized, and mini-batches of size 64 are sampled for network updates. The learning rates for the actor and critic networks are set to 0.02 and 0.001, respectively, while the learning rate for the DQN is set to 0.001. The soft update rate is fixed at 0.005, and the discount factor is set to 0.9. State normalization is performed using logarithmic scaling of channel gains to improve training stability, while continuous actions are clipped within predefined bounds to satisfy system constraints. The simulation parameters are presented in

Table 1. Parameter values for system setup.

Parameters	Values
Cell size	250 × 250 $m^2$
Bandwidth	6 MHz
Number of users	7
Target position	[89 36 0]
SNR of the BS	5 dB
BS power	10 W
User noise power	${10}^{-10}$ W
Target noise power	${10}^{-12}$ W
Soft update rate	0.005
Carrier frequency	3.5 GHz
Discount rate	0.9
Exploration noise	0.2
Mini-batch	64
Actor and critic networks learning rates	0.02, 0.001
Learning rate for Q-network	0.001
Experience replay buffer	500

. To evaluate the effectiveness of the proposed HDRL approach, its performance is compared against three widely used following benchmarks.

• Proximal Policy Optimization (PPO): PPO is a policy gradient method that balances exploration and stability by constraining policy updates within a trust region. It has been widely adopted due to its strong empirical performance and relative simplicity, making it a standard benchmark for continuous and discrete action-space problems.
• Advantage Actor-Critic (A2C): A2C is a synchronous actor-critic algorithm that utilizes the advantage function to reduce the variance of policy gradient estimates. It is valued for its training efficiency and its ability to stabilize learning compared to basic policy gradient methods, serving as a strong baseline for policy optimization tasks.
• Asynchronous Advantage Actor-Critic (A3C): A3C runs multiple parallel agents asynchronously. This approach decorrelates experiences and accelerates learning, particularly in complex or high-dimensional environments.

Figure 2 illustrates the convergence behavior of the proposed approach compared to the benchmark schemes. The results clearly show that the proposed hierarchical DRL framework outperforms all baselines, achieving higher rewards throughout the training process and stabilizing around episode 185 with an average reward of approximately 665, whereas the benchmarks converge more slowly and reach significantly lower reward levels. The hierarchical design of the proposed framework effectively balances exploration of promising strategies with exploitation of well-learned policies for efficient execution. The integration of TD3 further enhances stability by addressing overestimation bias through its twin-critic network, delayed policy updates, and target smoothing, thereby reducing oscillations and promoting smoother reward progression. As training progresses, the proposed approach effectively balances exploration with exploitation of well-learned policies. This eliminates the early over-optimistic actions, causing the reward to converge smoothly. In contrast, the benchmarks face inherent limitations that prevent them from achieving the performance of the proposed approach. PPO, although more stable due to its clipped surrogate objective, updates conservatively and therefore struggles to achieve peak performance in tasks requiring fine-grained exploration. A2C improves stability but remains constrained by biased value estimation and the absence of hierarchical abstraction. Moreover, A3C encourages exploration via asynchronous updates but suffers from high gradient variance, which slows convergence; furthermore, the lack of experience replay and target networks results in unstable learning in non-convex environments with complex constraints, limiting its optimization capability. Thus, the proposed approach achieves more efficient learning, smoother convergence, and superior asymptotic performance compared to benchmark approaches.

Figure 3 illustrates the training time required for convergence of different reinforcement learning algorithms, including the proposed HDRL framework, PPO, A2C, and A3C. It can be observed that the proposed HDRL requires the longest training time, approximately 1820 s, due to its hierarchical structure that integrates both discrete and continuous optimization via DQN and TD3 components. This increased complexity leads to higher computational overhead during training. In contrast, PPO converges faster with a training time of 1490 s, as it relies on a single policy network with clipped updates, reducing computational burden. The A2C and A3C algorithms exhibit significantly lower convergence times of 980 s and 760 s, respectively. This is mainly because these methods employ simpler architectures and fewer parameters, resulting in faster updates but comparatively lower learning capability. Despite the higher training time, the proposed HDRL framework achieves superior performance, as demonstrated in the sum-rate results. This highlights a trade-off between computational complexity and performance, where HDRL provides enhanced optimization capability at the cost of increased training time.

The ablation results in Figure 4 clearly show that the full HDRL framework achieves the highest average sum-rate among all considered variants. This is because the complete scheme jointly exploits the benefits of FAS, RSMA, and HDRL, where the discrete and continuous decision spaces are optimized in a coordinated manner. In particular, FAS provides additional spatial flexibility by enabling adaptive antenna/port selection, RSMA improves interference management through message splitting and partial interference decoding, and the hierarchical DQN-TD3 structure allows the system to handle hybrid optimization variables efficiently. Since the considered problem contains both discrete and continuous control dimensions, the full HDRL framework is able to search a richer solution space and identify better transmission configurations, which directly translates into improved sum-rate performance.

When FAS is removed, the performance decreases because the system loses its ability to adapt the antenna configuration according to the channel conditions. In the full scheme, FAS introduces additional spatial DoF that can be used to strengthen desired links and reduce inter-user interference by selecting more favorable antenna positions or ports. Without this spatial adaptability, the transceiver operates with a fixed antenna structure, which limits channel exploitation capability and reduces beamforming flexibility. Consequently, even if the remaining optimization and interference-management mechanisms are still active, the achievable sum-rate is lower than that of the full HDRL scheme.

A performance reduction is also observed in the without RSMA case. This degradation is expected because RSMA plays an important role in multiuser interference mitigation. By splitting each user message into common and private parts, RSMA provides an additional degree of freedom for balancing interference suppression and information delivery. This is particularly beneficial in overloaded or interference-limited multiuser downlink scenarios, where purely private-stream transmission is often less efficient. When RSMA is disabled, the system must rely only on conventional signaling, which reduces flexibility in handling coupled user interference and therefore leads to a lower sum-rate than the complete framework.

The only DQN variant further illustrates the importance of hierarchical optimization. DQN is well suited for handling discrete decision variables, such as selecting transmission modes, ports, or index-based actions. Therefore, it can still provide meaningful gains by identifying favorable structural configurations of the system. However, when only DQN is used, the framework loses the ability to finely optimize continuous control variables, such as power allocation, rate-splitting coefficients, or other analog parameters. Consequently, the obtained solution is only partially optimized continuous refinement, preventing the system from reaching its full performance potential. Therefore, it performs better than only TD3.

Among all ablated variants, the only TD3 case exhibits the lowest performance. The main reason is that TD3 is fundamentally designed for continuous optimization, whereas the considered joint design problem is inherently hybrid, involving both discrete and continuous decisions. Even if the continuous variables are optimized effectively, the system still depends on appropriate discrete decisions to determine the underlying transmission structure. Without DQN, those discrete choices cannot be properly explored or optimized, and the continuous controller is forced to operate over a restricted or suboptimal structural configuration. Hence, continuous optimization alone cannot compensate for the lack of discrete adaptation, because the quality of the final solution strongly depends on first selecting a good discrete operating point. Therefore, it achieves the lowest sum-rate, which confirms that the discrete decision-making layer is indispensable in this problem.

Overall, the ablation study demonstrates that the performance gain of the proposed method does not come from a single module in isolation. Rather, it originates from the synergistic interaction of spatial adaptability provided by FAS, interference management enabled by RSMA, and hybrid discrete–continuous optimization realized through the HDRL framework. The superiority of full HDRL therefore validates the need for jointly optimizing all components in an integrated manner.

Figure 5 illustrates the impact of increasing the user rate on the sum-rate. It is evident from the obtained results that the sum-rate values are high at low rate because the BS can flexibly allocate transmit power between the common and private streams, allowing even weak users to reliably decode through the common stream. The deployment of FAS at both the BS and user sides further enhances spatial diversity, as the fluid antenna position can be dynamically adjusted to select more favorable channels. However, as the rate threshold increases, each user’s QoS demand becomes more stringent, forcing the BS to dedicate more transmit power to guarantee per-user requirements and thereby reducing the DoF for flexible resource balancing. This leads to a saturation effect where sum-rate growth slows and eventually declines with higher thresholds. For instance, at a rate of 0.8, the proposed method achieves a 31.6%, 42.9%, and 47.1% higher sum-rate than the PPO, A2C, and A3C, respectively. This performance gain results from the proposed HDRL approach, which combines the discrete action-handling capability of DQN for FAS position selection and RSMA mode selection with the continuous control strength of TD3 for beamforming, power allocation, and common-rate optimization. Hence, the proposed approach efficiently adapts to the hybrid action space, effectively balancing exploration and exploitation, and converges toward more optimal strategies. In contrast, the benchmarks struggle with dynamic adaptation in mixed action environments, leading to suboptimal allocation strategies and reduced performance.

The impact of an increasing number of users on the system sum-rate is evaluated in Figure 6. The results show a consistent increase in sum-rate as the number of users grows. This improvement is mainly attributed to the capability of RSMA to effectively manage multiuser interference through partial interference decoding and flexible message splitting. In addition, the deployment of FAS at both the BS and user sides enables dynamic adjustment of antenna positions to better exploit favorable channel conditions, thereby reducing interference, enhancing spatial diversity, and improving the overall system performance. It can be observed that the proposed approach consistently outperforms PPO, A2C, and A3C across all user settings. For instance, when there are 20 users, then the proposed approach achieves performance gains of approximately 9.0% over PPO, 14.9% over A2C, and 14.9% over A3C. This clearly highlights the effectiveness of the proposed framework in handling interference while leveraging spatial adaptability. Furthermore, as the number of users increases, the advantages of RSMA become more pronounced in the downlink, since its flexible message-splitting strategy efficiently accommodates multiple user transmissions without significant degradation in performance. Overall, the proposed framework effectively mitigates the adverse effects of user densification, ensuring scalable and high-throughput downlink operation.

The impact of the BS transmit power on the system sum-rate is evaluated in Figure 7. The results reveal a significant increase in sum-rate with higher BS transmit power, primarily due to the enhanced channel adaptability provided by the FAS deployed at both the BS and user terminals. FAS enables dynamic reconfiguration of antenna positions, offering increased spatial diversity and improved beamforming precision, which significantly boosts the SINR across all users. Coupled with RSMA’s rate-splitting mechanism, which allows users to partially decode the common data stream and efficiently manage interference, the system can exploit FAS-enabled channel gains to maximize downlink transmission. For instance, at 8 dB transmit power, the proposed approach achieves a 4.2%, 6.4%, and 7.8% higher sum-rate than PPO, A2C, and A3C, respectively. This gain is achieved by the proposed approach through efficient exploration of the hybrid action space. Therefore, the proposed approach dynamically balances interference management and spatial diversity, enabling the agent to achieve higher SINR and more efficient resource utilization. Consequently, the sum-rate grows more rapidly with increasing transmit power compared to the benchmarks, as the proposed approach outperforms less adaptive or suboptimal policy updates in resource allocation.

In Figure 8, the sensing SINR decreases as the communication rate threshold increases. At low rate thresholds, communication QoS requirements can be satisfied with a small fraction of the transmit power, leaving more power for sensing, which enhances the sensing SINR. In the proposed approach, the common message stream acts as an interference suppression layer, allowing the system to exploit additional sensing DoF without compromising communication reliability. Private streams require less aggressive power boosting in this regime, further conserving resources for sensing. Moreover, the deployment of FAS at the BS and users dynamically reconfigures antenna positions to select favorable channel states, minimizing inter-user interference and maximizing spatial diversity, which further improves sensing performance. It can be seen that the proposed approach achieves 31.2%, 74.1%, and 88.0% higher sensing SINR than PPO, A2C, and A3C, respectively, effectively balancing the trade-off between communication and sensing. When the rate threshold increases, more power must be allocated to the RSMA common and private streams to meet communication demands, reducing the power available for sensing. However, FAS can compensate by adjusting antenna positions, but their effectiveness is limited by the reduced sensing power budget. Consequently, low rate thresholds yield higher sensing SINR and larger sensing DoF, whereas high rate thresholds prioritize communication at the expense of sensing, with FAS still providing some resilience by improving spatial channel conditions.

In Figure 9, the impact of varying bandwidth on the system sum-rate is analyzed. The results demonstrate that the sum-rate increases with bandwidth, consistent with the theoretical relationship between throughput and spectrum size. At low bandwidths, the sum-rate rises rapidly, as additional spectrum allows more bits to be transmitted per unit time while maintaining relatively high SINR per subcarrier. As the bandwidth increases further, the growth rate slows due to the fixed total transmit power being spread across more subcarriers, reducing the per-subcarrier SINR and increasing inter-user interference. The proposed RSMA-FAS framework effectively addresses these limitations. FAS at both the BS and user sides dynamically reconfigures antenna positions to exploit favorable spatial channels, reducing interference and enhancing channel diversity. Moreover, the RSMA common stream acts as an interference mitigation layer, allowing the system to leverage additional spatial DoF while meeting user-specific communication requirements. In addition, the proposed HDRL approach enables autonomous and adaptive optimization of both antenna configurations and power allocation, allowing the agent to dynamically balance spatial diversity, interference management, and per-user rate requirements. At a bandwidth of 6 MHz, the proposed method achieves 12%, 13.3%, and 14.2% higher sum-rates than PPO, A2C, and A3C, respectively, with the most pronounced gains observed at low-to-moderate bandwidths. This performance advantage arises because HDRL jointly optimizes the hybrid action space, effectively exploiting FAS-enabled channel variations and RSMA’s interference mitigation to maximize the SINR across subcarriers. These results underscore the effective interplay between dynamic antenna positioning, RSMA-based message splitting, and HDRL-driven adaptive control in enhancing spectral efficiency and sum-rate under varying bandwidth conditions.

5. Conclusions

In this work, an FAS-assisted ISAC system is investigated that integrates the advantages of RSMA for efficient interference management. The communication rate is maximized between the BS and users under constraints on BS transmit power and sensing beampattern gain. The sum-rate is maximized by jointly optimizing the beamforming vectors, fluid antenna positions, and RSMA common rate, with the design formulated as an optimization problem to balance sensing and communication requirements. To efficiently solve this complex problem, a novel HDRL approach is proposed, which integrates the DQN and the TD3 in a hierarchical manner, enabling effective exploration of the solution space and convergence to high-performance configurations. Simulation results confirm that the proposed approach significantly outperforms benchmarks approaches by achieving higher sum-rates while maintaining robust sensing capabilities, demonstrating the synergy between fluid antenna flexibility and RSMA’s interference management. In future work, we will extend the proposed framework to more realistic ISAC scenarios by incorporating imperfect CSI conditions, including partial CSI for fluid antenna ports, as well as mobility-aware environments where dynamic user and target movements together with CSI acquisition uncertainty require rapid and continuous adaptation of beamforming strategies and fluid antenna port selection to simultaneously maintain sensing accuracy and communication quality. In addition, we will investigate more sophisticated sensing models that account for multi-target environments, clutter effects, and bistatic or monostatic sensing configurations.