A DDPG-LSTM Framework for Optimizing UAV-Enabled Integrated Sensing and Communication

Highlights

• What are the main findings?

• A novel integrated sensing and communication (ISAC)-enabled unmanned aerial vehicle (UAV) architecture is proposed, enabling a single UAV to jointly perform uplink communication and radar sensing.
• A long short-term memory (LSTM)-enhanced deep deterministic policy gradient (DDPG)-based deep reinforcement learning (DRL) framework is developed to optimize UAV trajectory and uplink power control.
• The proposed approach effectively adapts to dynamic target movements, enhancing both sensing accuracy and communication reliability.

• What is the implication of the main finding?

• The framework enables energy-efficient UAV operation by minimizing movement energy consumption while maintaining sensing performance.
• The framework enables UAVs to autonomously and efficiently balance sensing and communication tasks in dynamic environments.
• The solution supports real-time adaptation to target mobility, making it suitable for practical ISAC-UAV applications such as surveillance and smart city monitoring.

Abstract

This paper proposes a novel dual-functional radar-communication (DFRC) framework that integrates unmanned aerial vehicle (UAV) communications into an integrated sensing and communication (ISAC) system, termed the ISAC-UAV architecture. In this system, the UAV’s mobility is leveraged to simultaneously serve multiple single-antenna uplink users (UEs) and perform radar-based sensing tasks. A key challenge stems from the target position uncertainty due to movement, which impairs matched filtering and beamforming, thereby degrading both uplink reception and sensing performance. Moreover, UAV energy consumption associated with mobility must be considered to ensure energy-efficient operation. We aim to jointly maximize radar sensing accuracy and minimize UAV movement energy over multiple time steps, while maintaining reliable uplink communications. To address this multi-objective optimization, we propose a deep reinforcement learning (DRL) framework based on a long short-term memory (LSTM)-enhanced deep deterministic policy gradient (DDPG) network. By leveraging historical target trajectory data, the model improves prediction of target positions, enhancing sensing accuracy. The proposed DRL-based approach enables joint optimization of UAV trajectory and uplink power control over time. Extensive simulations validate that our method significantly improves communication quality and sensing performance, while ensuring energy-efficient UAV operation. Comparative results further confirm the model’s adaptability and robustness in dynamic environments, outperforming existing UAV trajectory planning and resource allocation benchmarks.

1. Introduction

The rapid expansion of the Internet of Things (IoT) and the emergence of intelligent applications, such as autonomous driving, have significantly increased the demand for high-speed wireless communication and precise radar sensing [1]. Traditionally, communication and sensing functions have been designed and operated separately, resulting in inefficient spectrum utilization and redundant hardware resources. To overcome these limitations, integrated sensing and communication (ISAC) has emerged as a promising paradigm that co-designs communication and sensing functionality through shared waveforms and hardware [1,2]. This integration not only improves spectral efficiency and reduces system costs but also supports the development of versatile, multifunctional wireless systems.

ISAC has gained considerable attention as a key enabler for future sixth-generation (6G) wireless systems, which aim to deliver ultra-reliable, low-latency communications alongside high-resolution environmental awareness through joint use of infrastructure and spectrum [3,4]. Leveraging cutting-edge technologies such as multiple-input–multiple-output (MIMO) antenna systems and access to high-frequency bands including millimeter-wave (mmWave) and terahertz (THz), ISAC enables the simultaneous delivery of broadband communication and fine-grained sensing capabilities within a unified system [5]. A prominent implementation of ISAC is the dual-functional radar-communication (DFRC) framework, which supports concurrent sensing and communication using a common waveform. Recent research has made significant advances in this area, including coding techniques [6], waveform design methodologies [7,8], and joint beamforming techniques [9,10], all aimed at enhancing signal reuse efficiency across radar and communication subsystems. In particular, recent studies have introduced advanced hybrid beamforming methods tailored for multi-user ISAC systems, including partially connected analog–digital architectures for Cramér–Rao bound (CRB)-optimal design [11], prior-information-aware beamforming that minimizes the posterior CRB [12], and communication-centric hybrid designs that jointly optimize multi-user transmission and sensing constraints [13]. These contributions significantly improve the practicality and performance of ISAC systems in real-world deployments. Despite these advancements, terrestrial ISAC systems remain limited by factors such as fixed infrastructure and environmental obstructions. Static base station (BS) deployments often suffer from unreliable line-of-sight (LoS) links, which degrade sensing accuracy and reduce operational coverage [14].

To overcome these challenges, unmanned aerial vehicles (UAVs) have been increasingly explored as mobile ISAC platforms capable of augmenting both communication and sensing performance. UAVs offer three-dimensional mobility [15,16] allowing real-time adjustment of altitude and trajectory to optimize channel conditions and sensing geometry [17]. This spatial flexibility significantly enhances link reliability and sensing accuracy, especially in environments where terrestrial infrastructure is insufficient or obstructed. UAV-based systems, with their rapid deployment capabilities and high probability of maintaining LoS connections, are thus strong candidates for airborne ISAC applications [18]. Their mobility enables context-aware adaptation to changing user demands and environmental dynamics, effectively balancing communication throughput and sensing resolution [19]. Importantly, UAVs excel in real-time tracking of moving targets, something static ground-based ISAC systems often struggle with, thereby enabling persistent sensing and reliable operation in mobility-constrained scenarios.

1.1. Related Work and Motivation

The synergistic integration of ISAC technologies into UAV systems holds substantial promise for enhancing both wireless communication and sensing capabilities. Consequently, this research area has attracted significant attention in recent years [20,21,22]. In [20], the authors deployed a UAV-enabled ISAC system, where the UAV senses ground users and forwards the information to BS, with the dual objective of maximizing energy efficiency and minimizing the radar mutual information. In [21], the authors introduced a massive MIMO-based UAV-enabled ISAC system for the radar probing tasks with various optimization schemes. In [22], the authors proposed a multi-stage UAV-enabled ISAC system for target localization, where the UAV first performs wide-beam sensing while stationary to acquire a coarse estimate of the target’s position, followed by narrow-beam sensing with jointly optimized trajectory and beamforming to refine localization accuracy. While these studies primarily focus on fixed targets, the emergence of autonomous vehicular applications has prompted a shift toward ISAC systems capable of real-time tracking and detection of dynamic targets. In response, the authors of [23] proposed a globally optimal trajectory design for UAV-enabled ISAC with moving ground users by projecting the trajectories into a user-relative frame. The problem was reformulated as a shape optimization involving a density-varying catenary within an artificial potential field. To enhance ISAC performance with limited UAV resources, ref. [24] examined an orthogonal frequency division multiple access (OFDMA)-based cooperative multi-UAV system that enables joint multi-static sensing and coordinated multiple point transmission. Many existing studies assume known target positions when designing UAV trajectories, which limits their practicality in dynamic environments.

To address this limitation, recent research has introduced advanced signal processing techniques to estimate parameters such as target position, angle of arrival (AoA), angle of departure (AoD), time delay, and Doppler shift under uncertainty. For instance, ref. [25] proposed a networked ISAC-based UAV tracking and handover scheme using a virtual sensing cell, where one primary BS transmits while multiple BSs receive echoes. Each BS removes clutter and estimates the UAV’s position and velocity, which are then fused by a centralized extended Kalman filter (EKF) for accurate multi-UAV tracking and handover. In a similar direction, ref. [26] presented a UAV-enabled ISAC system for joint tracking and communication with a moving target, employing an EKF-based scheme and optimizing the UAV trajectory to minimize a weighted sum of the predicted posterior Cramér–Rao bound (PCRB) for position and velocity estimation. Building on this, the authors in [27] proposed an online energy-aware UAV trajectory optimization method, further minimizing the weighted sum of the predicted PCRB while satisfying UAV energy and location constraints. In [28], a UAV-enabled ISAC system was designed for vehicular networks, introducing a three-stage approach: initial state estimation, wide-beam EKF-based tracking, and adaptive transmission based on real-time sensing and communication performance. Additionally, ref. [29] developed a UAV-assisted ISAC system for IoT networks, where a UAV senses IoT nodes with unknown positions and forwards the collected data to the BS. To cope with location uncertainty and limited resources, a trajectory optimization problem was formulated to maximize the radar estimation rate and minimize flight time, using a deep reinforcement learning (DRL) approach. The authors in [30] proposed a UAV-ISAC-assisted IoT system in which a UAV detects unknown IoT node locations and relays the data to a ground BS. This task presents several challenges, including unknown node positions, non-convex optimization, limited resources, dynamic environmental conditions, and complex resource allocation. To address these issues, they proposed a multi-step dueling double deep Q-network (DDQN) based on DRL. In [31], the authors proposed an integrated sensing, communication, and computation (ISCC) system utilizing collaborative UAVs equipped with edge servers to support intelligent transportation systems. To address the architectural and coordination challenges in UAV networks, they developed the multi-UAV collaborative air-ISCC (MCAI) algorithm, which employs an asynchronous advantage actor–critic framework to collaboratively train a DRL model across UAV. This approach aims to improve ISCC service reliability while enhancing UAV energy efficiency. In [32], an ISAC system designed for the low-altitude economy (LAE) was investigated, where a ground BS simultaneously provided UAVs with communication and navigation services while sensing the airspace to detect unauthorized mobile targets. Building on this model, the authors proposed a novel LAE-specific ISAC framework, termed DeepLSC, which leverages a DRL approach to address multi-objective optimization and satisfy diverse system constraints effectively. While these studies represent significant progress, many DRL-based ISAC-UAV frameworks overlook valuable historical information, such as past target positions, which are strongly correlated with future behavior. This motivates us to integrate predictive methods into DRL frameworks to more effectively exploit this information and further enhance the learning process, aiming to achieve superior performance in ISAC-based UAV systems.

1.2. Contribution and Outline

This study tackles the challenging task of maximizing radar sensing performance while minimizing the UAV movement energy consumption over multiple time steps, all while ensuring reliable uplink communication quality. To achieve this, we jointly optimize the UAV’s trajectory and resource allocation within a UAV-enabled ISAC framework. A central challenge lies in the uncertainty of the target’s position, which undermines traditional methods such as convex optimization that require prior knowledge of the target’s location. Effective optimization in this setting demands both the predictive capability to estimate future target positions and the decision-making capability to optimize actions over time. To tackle these challenges, we propose a novel DRL framework that integrates long short-term memory (LSTM) units into the deep deterministic policy gradient (DDPG) architecture. This LSTM-DDPG network is designed to capture temporal dependencies in historical trajectory data, thereby improving both target prediction and long-term optimization performance. The main contributions of this study can be summarized as follows:

• We introduce a new DFRC framework that incorporates UAV-based communication into an ISAC system. The proposed architecture enables the UAV to simultaneously support uplink communication for ground users and perform radar-based sensing.
• To reflect realistic operating conditions, we incorporate a detailed UAV movement energy consumption model. This component directly addresses the constraint of limited onboard energy in UAV systems.
• The impact of target mobility and position uncertainty on matched filtering and beamforming in ISAC systems is analyzed and discussed. Accordingly, we formulate a multi-step joint optimization problem, where the UAV’s trajectory, uplink transmit power, and predicted target positions are optimized to balance communication efficiency and sensing performance.
• A novel DRL method, termed LSTM-DDPG, is proposed by embedding LSTM modules into both the actor and critic networks of DDPG. This integration facilitates the model to capture temporal correlations in historical target trajectories, enhancing prediction accuracy and enabling more effective long-term UAV control and resource allocation.
• Simulation results demonstrate that the proposed method significantly enhances communication efficiency, sensing robustness, and energy conservation. Comparative studies further highlight the framework’s ability to leverage historical target position data, leading to more efficient learning and superior performance compared to existing benchmarks.

Notation: Scalars are denoted by lowercase italic letters (e.g., x; y), and column vectors are represented by lowercase boldface letters (e.g., $\mathit{x}$; $\mathit{y}$). For a vector $\mathit{x}$, the Hermitian transpose and conjugate are denoted as ${\mathit{x}}^H$ and ${\mathit{x}}^{*}$, respectively. The norm, expectation, and variance are represented by $|·|$, $\mathbb{E}[·]$, and $\mathrm{Var}[·]$, respectively.

2. System Model and Problem Formulation

We consider an ISAC system built on a full-duplex (FD) UAV platform, where the UAV simultaneously performs target motion sensing and uplink communication with multiple ground users, as depicted in Figure 1. The UAV is equipped with two uniform linear arrays (ULAs): one with $N_t$ transmit antennas and the other with $N_r$ receive antennas. Leveraging its FD capabilities, the UAV can simultaneously perform uplink communication with $K_{\mathrm{ul}}$ single-antenna user equipments (UEs) and conduct radar-based sensing by transmitting probing signals, even in the presence of C clutter sources. This architecture enables concurrent communication and sensing operations. This configuration offers a flexible and mobile solution for intelligent transportation systems, enhancing terrestrial communication, particularly in applications such as smart cities and highway monitoring. Subsequently, we formulate a joint optimization problem over a finite time horizon consisting of S discrete time steps, indexed by the set $\mathcal{S}=\{1,2,\dots ,S\}$, where $s\in \mathcal{S}$. The set of uplink UEs is denoted by ${\mathcal{K}}_{\mathrm{ul}}=\{1,2,\dots ,K_{\mathrm{ul}}\}$, where each $u\in {\mathcal{K}}_{\mathrm{ul}}$ represents a single-antenna uplink, ${\mathtt{UE}}_u^{\mathrm{ul}}$. Let $\mathcal{T}=\{1,\dots ,1+C\}$ denote the set of targets, including the desired target and C clutter sources. Each element, $t\in \mathcal{T}$ corresponds to a specific target or clutter object, where $t=1$ denotes the desired target, and $t\ne 1$ denotes a clutter source.

2.1. Channel Model

Due to the elevated altitude of the UAV, the channels from both the UEs and the target/clutter to the UAV, denoted as ${\mathbf{h}}_u^{\mathrm{ul}}$ and ${\mathbf{h}}_t^{\mathrm{ut}}$, respectively, are assumed to be dominated by LoS propagation. Each channel is modeled by incorporating both large-scale path loss and small-scale fading effects. Accordingly, the channel vectors ${\mathbf{h}}_u^{\mathrm{ul}}$ and ${\mathbf{h}}_t^{\mathrm{ut}}$ at each time step are given by

$$\begin{array}{cc}\hfill {\mathbf{h}}_u^{\mathrm{ul}}& =\sqrt{{\beta }_u^{\mathrm{ul}}}\mathbf{a}\left({\varphi }_u^{\mathrm{ul}}\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathbf{h}}_t^{\mathrm{ut}}& =\sqrt{{\beta }_t^{\mathrm{ul}}}\mathbf{a}\left({\varphi }_t^{\mathrm{ut}}\right),\hfill \end{array}$$

(2)

where $\mathbf{a}\left(\varphi \right)$ is the steering vector representing the small-scale fading, defined as

$$\begin{array}{c}\hfill \mathbf{a}\left(\varphi \right)=\left[1,\dots ,e^{-j2\pi {\displaystyle \frac{d}{\lambda }}(N-1)sin\left(\varphi \right)}\right],\end{array}$$

(3)

which depends on the AoA ${\varphi }^{ut}$ and AoD ${\varphi }^{ul}$ between the UAV and a target/clutter or UE, the wavelength $\lambda$, and the antenna spacing d. The number of antenna elements N corresponds to $N_t$ or $N_r$, depending on the array. The large-scale path loss between the UAV and a target/clutter or UE is modeled as ${\beta }_t^{\mathrm{ut}}=C_0{\left(d_t^{\mathrm{ut}}\right)}^{-\alpha }$, where $C_0$ is the path loss at a reference distance $D_0=1$ m, $d_t^{\mathrm{ut}}$ is the 3D distance between the UAV and the target/clutter, and $\alpha$ is the path loss exponent. Similar notation applies for ${\beta }_u^{\mathrm{ul}}$ between the UAV and the uplink UE, ${\mathtt{UE}}_u^{\mathrm{ul}}$.

The UAV’s position is denoted by $\mathbf{u}=(x^{\mathrm{uav}},y^{\mathrm{uav}},z^{\mathrm{uav}})$, with altitude constrained by $h_{\min }\le z_{\mathrm{u}}\le h_{\max }$. The ground-level positions of the u-th UE and the t-th target/clutter are denoted as ${\mathbf{a}}_u^{\mathrm{ul}}=(x_u^{\mathrm{ul}},y_u^{\mathrm{ul}},0)$ and ${\mathbf{a}}_t^{\mathrm{target}}=(x_t^{\mathrm{target}},y_t^{\mathrm{target}},0)$, respectively. Accordingly, the distances $d_u^{\mathrm{ul}}$ and $d_t^{\mathrm{ut}}$ can be computed as

$$\begin{array}{cc}\hfill d_u^{\mathrm{ul}}& =|\mathbf{u}-{\mathbf{a}}_u^{\mathrm{ul}}|,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill d_t^{\mathrm{ut}}& =|\mathbf{u}-{\mathbf{a}}_t^{\mathrm{target}}|.\hfill \end{array}$$

(5)

A key challenge in enabling FD communication lies in self-interference (SI), the leakage of the UAV’s transmit signal into its own receiver, which can degrade the desired signal quality. Recent advances in both active and passive SI cancellation techniques have enabled suppression levels that approach the thermal noise floor. Accordingly, this work adopts the widely used assumption of perfect SI cancellation at the UAV receiver. This assumption is supported by recent FD communication systems, where experimental setups have demonstrated SI suppression exceeding 110 dB, sufficient to render residual interference negligible in many practical scenarios [33]. Such suppression is typically achieved through a combination of passive methods (e.g., antenna isolation), analog cancellation at the RF stage, and digital cancellation at the baseband level.

2.2. Communication Model

During the downlink sensing phase at the s-th time step, the transmitted probing sensing signal ${\mathbf{x}}_{\mathrm{rad}}\in {\mathbb{C}}^{N_t\times 1}$ at the UAV is expressed as

$$\begin{array}{c}\hfill {\mathbf{x}}_{\mathrm{rad}}\left(s\right)={\mathbf{z}}_{\mathrm{rad}}\left(s\right)b_{\mathrm{rad}}\left(s\right),\end{array}$$

(6)

where $b_{rad}\left(s\right)$, satisfying $\mathbb{E}\left\{\parallel b_{rad}{\left(s\right)\parallel }^2\right\}=1$, represents the radar probing symbol, and ${\mathbf{z}}_{rad}\left(s\right)\in {\mathbb{C}}^{N_t\times 1}$ is the radar beamforming vector designed to focus on the desired target at the s-th time step.

In the uplink communication phase at the s-th time step, all UEs simultaneously transmit to the UAV. To decode the signal from ${\mathtt{UE}}_u^{\mathrm{ul}}$, the UAV applies a receive beamforming vector ${\mathbf{o}}_u\left(s\right)$ (i.e., matched-filter) to the received signal $y_m^{\mathrm{ul}}$. The resulting signal for ${\mathtt{UE}}_u^{\mathrm{ul}}$ is expressed as

$$\begin{array}{cc}\hfill {\widehat{y}}_u^{\mathrm{ul}}\left(s\right)=& {\mathbf{o}}_u\left(s\right)y_m^{\mathrm{ul}}\left(s\right)\hfill \\ \hfill =& \underset{\mathrm{Desired}\mathrm{signal}{\mathrm{DU}}_u^{\mathrm{ul}}\left(s\right)}{\underbrace{{\mathbf{o}}_u\left(s\right){\mathbf{h}}_u^{\mathrm{ul}}\left(s\right){\sqrt{p}}_{\max }^{\mathrm{ul}}{\eta }_u^{\mathrm{ul}}\left(s\right)v_u\left(s\right)}}+\sum _{u^{\prime }=1|u^{\prime }\ne u}^{K_{\mathrm{ul}}}\underset{\mathrm{Inter}-\mathrm{user}\mathrm{interference}{\mathrm{IUI}}_{u^{\prime },u}^{\mathrm{ul}}\left(s\right)}{\underbrace{{\mathbf{o}}_u\left(s\right){\mathbf{h}}_{u^{\prime }}^{\mathrm{ul}}\left(s\right){\sqrt{p}}_{\max }^{\mathrm{ul}}{\eta }_{u^{\prime }}^{\mathrm{ul}}\left(s\right)v_{u^{\prime }}\left(s\right)}}\hfill \\ & +\sum _{t=1}^{C+1}\underset{\mathrm{Echo}\mathrm{signal}\mathrm{interference}{\mathrm{ESI}}_{t,u}\left(s\right)}{\underbrace{{\mathbf{o}}_u\left(s\right){\beta }_t^{\mathrm{ut}}\left(s\right){\alpha }_t\mathbf{A}\left({\varphi }_t^{\mathrm{ut}}\left(s\right)\right){\mathbf{x}}_{\mathrm{rad}}\left(s\right)}}+\underset{\mathrm{Noise}{\mathrm{N}}_u^{\mathrm{ul}}\left(s\right)}{\underbrace{{\mathbf{o}}_u\left(s\right){\mathbf{n}}_{\mathbf{0}}\left(s\right)}},\hfill \end{array}$$

(7)

where $\mathbf{A}\left({\varphi }_t^{\mathrm{ut}}\right)={\mathbf{a}}_{\mathbf{r}}\left({\varphi }_t^{\mathrm{ut}}\right){\mathbf{a}}_{\mathbf{t}}{\left({\varphi }_t^{\mathrm{ut}}\right)}^H$ represents the target response matrix, constructed from transmit steering vector ${\mathbf{a}}_{\mathbf{t}}\left({\varphi }_t^{\mathrm{ut}}\right)$ and receive steering vector ${\mathbf{a}}_{\mathbf{r}}\left({\varphi }_t^{\mathrm{ut}}\right)$ corresponding to the direction ${\varphi }_t^{\mathrm{ut}}$. The parameter ${\alpha }_t$ denotes the complex reflection coefficient, dependent on the radar cross-section (RCS) of the target/clutter. ${\eta }_u^{\mathrm{up}}$ denotes the power control coefficient for ${\mathrm{UE}}_u^{\mathrm{ul}}$, and $v_u\left(s\right)$, with $\mathbb{E}\left\{\parallel v_u{\left(s\right)\parallel }^2\right\}=1$, is the normalized data symbol, subject to the maximum uplink power constraint $p_{\max }^{\mathrm{ul}}$. ${\mathbf{n}}_{\mathbf{0}}\left(s\right)\sim \mathcal{CN}(0,{\sigma }^2\mathbf{I})$ is the additive white Gaussian noise vector at the UAV.

The signal-to-interference-plus-noise ratio (SINR) for uplink user ${\mathtt{UE}}_u^{\mathrm{ul}}$ at the s-th time step, denoted as ${\gamma }_u^{\mathrm{ul}}\left(s\right)$, is derived in (10). Consequently, the achievable uplink rate (in bps/Hz) for ${\mathtt{UE}}_u^{\mathrm{ul}}$ at the s-th time step is given by

$$\begin{array}{c}\hfill R_u^{\mathrm{ul}}\left(s\right)={log}_2(1+{\gamma }_u^{\mathrm{ul}}\left(s\right)).\end{array}$$

(8)

2.3. Sensing Radar Model

During the sensing phase, the UAV applies a matched filtering by correlating the received signal with a beamforming vector ${\mathbf{r}}_t\left(s\right)$, which is designed to suppress the interference from clutter sources and maximize the radar SINR. The received echo signal corresponding to the desired target $t=1$ at the s-th time step is expressed as

$$\begin{array}{cc}\hfill {\widehat{y}}_t^{\mathrm{rad}}\left(s\right)& ={\mathbf{r}}_t\left(s\right)y_m^{\mathrm{ul}}\left(s\right)\hfill \\ \hfill & =\underset{\mathrm{Desired}\mathrm{radar}\mathrm{signal}{\mathrm{DU}}_{t,m}^{\mathrm{rad}}\left(s\right)}{\underbrace{{\mathbf{r}}_t\left(s\right){\beta }_t^{\mathrm{ut}}\left(s\right){\alpha }_t\mathbf{A}\left({\varphi }_t^{\mathrm{ut}}\left(s\right)\right){\mathbf{x}}_{\mathrm{rad}}\left(s\right)}}+\sum _{t^{\prime }=2}^{C+1}\underset{\mathrm{Clutter}\mathrm{interference}{\mathrm{CCI}}_{t^{\prime },t}\left(s\right)}{\underbrace{{\mathbf{r}}_t\left(s\right){\beta }_{t^{\prime }}^{\mathrm{ut}}\left(s\right){\alpha }_{t^{\prime }}\mathbf{A}\left({\varphi }_{t^{\prime }}^{\mathrm{ut}}\left(s\right)\right){\mathbf{x}}_{\mathrm{rad}}\left(s\right)}}\hfill \\ & +\sum _{u=1}^{K_{\mathrm{ul}}}\underset{\mathrm{Inter}-\mathrm{user}\mathrm{interference}{\mathrm{IUI}}_{u,t}^{\mathrm{rad}}\left(s\right)}{\underbrace{{\mathbf{r}}_t\left(s\right){\mathbf{h}}_u^{\mathrm{ul}}\left(s\right){\sqrt{p}}_{\max }^{\mathrm{ul}}{\eta }_u^{\mathrm{ul}}\left(s\right)v_u\left(s\right)}}+\underset{\mathrm{Noise}{\mathrm{N}}_t^{\mathrm{rad}}\left(s\right)}{\underbrace{{\mathbf{r}}_t\left(s\right){\mathbf{n}}_0\left(s\right)}}.\hfill \end{array}$$

(9)

$$\begin{array}{c}\hfill {\gamma }_u^{\mathrm{ul}}\left(s\right)={\displaystyle \frac{|{\mathrm{DU}}_u^{\mathrm{ul}}{\left(s\right)|}^2}{{\sum }_{u^{\prime }=1|u^{\prime }\ne u}^{K_{\mathrm{ul}}}|{\mathrm{IUI}}_{u^{\prime },u}^{\mathrm{ul}}{\left(s\right)|}^2+{\sum }_{t=1}^{C+1}|{\mathrm{ESI}}_{t,u}{\left(s\right)|}^2+{\left|{\mathrm{N}}_u^{\mathrm{ul}}\left(s\right)\right|}^2}},\end{array}$$

(10)

$$\begin{array}{c}\hfill {\gamma }_t^{\mathrm{rad}}\left(s\right)={\displaystyle \frac{|{\mathrm{DU}}_t^{\mathrm{rad}}{|}^2}{{\sum }_{u=1}^{K_{\mathrm{ul}}}|{\mathrm{IUI}}_{u,t}^{\mathrm{rad}}{\left(s\right)|}^2+{\sum }_{t^{\prime }=1|t^{\prime }\ne t}^{C+1}|{\mathrm{CCI}}_{t^{\prime },t}{\left(s\right)|}^2+{\left|{\mathrm{N}}_t^{\mathrm{rad}}\left(s\right)\right|}^2}}.\end{array}$$

(11)

From an information-theoretic perspective, the mutual information (MI) between the transmitted radar waveform and the received echo signal quantifies the amount of information shared between two random variables by measuring the reduction in uncertainty of one given knowledge of the other. In radar systems, MI serves as a fundamental metric to evaluate target detection performance by capturing the correlation between the transmitted radar signal and the target response. A higher MI indicates a stronger correlation, leading to improved detection and estimation accuracy. Prior studies have demonstrated that MI effectively characterizes the accuracy of radar target impulse response estimation [34,35]. Factors such as waveform design, target scattering characteristics, clutter, and path loss all influence the achievable MI. Consequently, maximizing MI plays a critical role in enhancing radar estimation and unifying traditional detection metrics with information-theoretic insights. In particular, in MIMO radar systems detecting point targets, the detection probability is typically proportional to the radar output SINR. Accordingly, the radar performance between the UAV and the target in time step s is characterized using the radar SINR, denoted as ${\gamma }_t^{\mathrm{rad}}$, which is defined in (11).

2.4. UAV Movement Energy Consumption

We adopt an analytical energy consumption model for rotary-wing UAVs based on actuator disc theory and blade element theory, as established in the classical aeronautical literature [36,37]. This energy consumption model captures the propulsion energy required for rotary-wing UAV flight, which is a critical factor in trajectory design, battery management, and overall mission planning. The same model has been widely adopted and validated in recent studies on UAV-enabled wireless communication systems [38,39,40], further supporting its reliability for performance evaluation and optimization. In this study, we focus on a rotary-wing UAV whose movement energy consumption between two consecutive time steps, i.e., from time step $s-1$ to s, is characterized following the approach in [41] as

$$\begin{array}{cc}\hfill E_{\mathrm{uav}}^{\mathrm{mov}}\left(s\right)=& \tau \left(s\right)\times {\mathrm{P}}_{\mathrm{uav}}^{\mathrm{mov}}\triangleq \tau \left(s\right)A_0\left(1+{\displaystyle \frac{3v_h^2}{{\ell }_{\mathrm{tip}}^2}}\right)+\tau \left(s\right)A_1{\left({\left(1+{\displaystyle \frac{v_h^4}{4v_0^2}}\right)}^{1/2}-{\displaystyle \frac{v_h^2}{2v_0^4}}\right)}^{1/2}\hfill \end{array}$$

$$\begin{array}{c}\hfill +\tau \left(s\right){\displaystyle \frac{1}{2}}{\ell }_{\mathrm{fus}}{\ell }_{\mathrm{air}}{\ell }_{\mathrm{sol}}{\ell }_{\mathrm{disc}}{v}_h^3+\tau \left(s\right){\ell }_{\mathrm{wei}}{v}_t.\end{array}$$

(12)

We consider the time flight duration $\tau \left(s\right)$ required for a UAV to travel from its previous position at the $(s-1)$-th time step, $\mathbf{u}(s-1)=(x^{\mathrm{uav}}(s-1),y^{\mathrm{uav}}(s-1),z^{\mathrm{uav}}(s-1))$ to its current position at the s-th time step, $\mathbf{u}\left(s\right)=(x^{\mathrm{uav}}\left(s\right),y^{\mathrm{uav}}\left(s\right),z^{\mathrm{uav}}\left(s\right))$, which is given by $\tau \left(s\right)={\displaystyle \frac{d_{\mathrm{uav}}\left(s\right)}{v_{\mathrm{uav}}}}={\displaystyle \frac{\left|\mathbf{u}\right(s)-\mathbf{u}(s-1\left)\right|}{v_{\mathrm{uav}}}}$, where $d_{\mathrm{uav}}$ denotes the Euclidean distance traveled, and $v_{\mathrm{uav}}$ is the UAV’s constant flight speed. The UAV’s energy consumption is modeled using the analytical framework for rotary-wing UAVs, incorporating actuator disc and blade element theories. In this model, the constants $A_0$ and $A_1$ in (12) represent the blade profile power and induced power, respectively, and are defined as $A_0={\displaystyle \frac{{\ell }_{\mathrm{drag}}}{8}}{\ell }_{\mathrm{air}}{\ell }_{\mathrm{sol}}{\ell }_{\mathrm{disc}}{\ell }_{\mathrm{ang}}^3{\ell }_{\mathrm{rad}}^3$ and $A_1=(1+{\ell }_{\mathrm{incre}}){\displaystyle \frac{{\ell }_{\mathrm{wei}}^{3/2}}{\sqrt{2{\ell }_{\mathrm{air}}{\ell }_{\mathrm{disc}}}}}$. Here, ${\ell }_{\mathrm{tip}}$ is the tip speed of the rotor blade, while ${\ell }_{\mathrm{fus}}$, ${\ell }_{\mathrm{air}}$, ${\ell }_{\mathrm{sol}}$, ${\ell }_{\mathrm{disc}}$, and ${\ell }_{\mathrm{wei}}$ represent the fuselage drag ratio, air density, rotor solidity, rotor disc area, and UAV weight, respectively. The parameters ${\ell }_{\mathrm{drag}}$, ${\ell }_{\mathrm{ang}}$, and ${\ell }_{\mathrm{rad}}$ denote the profile drag coefficient, blade angular velocity, and rotor radius, respectively. The variable $v_0$ denotes the mean rotor-induced velocity during hover, and $v_{\mathrm{uav}}$ is the UAV’s flight velocity. A maximum velocity constraint $v_{\max }$ is incorporated to ensure realistic operation. Additionally, the UAV’s velocity is decomposed into horizontal and vertical components as $v_h\left(s\right)={\displaystyle \frac{\left|\mathbf{x}\right(s)-\mathbf{x}(s-1\left)\right|}{\tau \left(s\right)}}={\displaystyle \frac{v_{\mathrm{uav}}\times |\mathbf{x}\left(s\right)-\mathbf{x}(s-1)|}{d_{\mathrm{uav}}\left(s\right)}}$ and $v_t\left(s\right)={\displaystyle \frac{|z^{\mathrm{uav}}\left(s\right)-z^{\mathrm{uav}}(s-1)|}{\tau }}={\displaystyle \frac{v_{\mathrm{uav}}\times |z^{\mathrm{uav}}\left(s\right)-z^{\mathrm{uav}}(s-1)|}{d_{\mathrm{uav}}\left(s\right)}}$, where $\mathbf{x}\left(s\right)=(x^{\mathrm{uav}}\left(s\right),y^{\mathrm{uav}}\left(s\right))$.

3. Problem Formulation

Before presenting the proposed optimization problem, we first design the beamforming vectors ${\mathbf{c}}_u$ and ${\mathbf{r}}_t$ for uplink communication and sensing, respectively. Given that the UAV is equipped with an antenna array where the number of receive antennas exceeds the number of users $N_r>K_{\mathrm{ul}}$, we employ the zero-forcing (ZF) beamforming technique to effectively exploit the spatial degrees of freedom for interference mitigation and improved radar sensing accuracy. ZF effectively maximizes the SINR by mitigating multi-user interference, thereby enhancing both uplink decoding and sensing performance. Accordingly, the receive beamforming vectors ${\mathbf{c}}_u\in {\mathbb{C}}^{1\times N_r}$ and ${\mathbf{r}}_t\in {\mathbb{C}}^{1\times N_r}$ corresponds to the u-th and $(K+1)$-th row of the ZF receiver ${\mathbf{A}}^{\mathrm{ZF}}$ at the s-th time step, which is given by

$$\begin{array}{cc}\hfill {\mathbf{A}}^{\mathrm{ZF}}& ={\left({\left({\mathbf{G}}^{\mathrm{ul}}\left(s\right)\right)}^H{\mathbf{G}}^{\mathrm{ul}}\left(s\right)\right)}^{-1}{\left({\mathbf{G}}^{\mathrm{ul}}\left(s\right)\right)}^H.\hfill \end{array}$$

(13)

where ${\mathbf{G}}^{\mathrm{ul}}\left(s\right)\triangleq [{\mathbf{h}}_1^{\mathrm{ul}},\dots ,{\mathbf{h}}_{K_{\mathrm{ul}}}^{\mathrm{ul}},{\mathbf{h}}_1^{\mathrm{ut}},\dots ,{\mathbf{h}}_{C+1}^{\mathrm{ut}}]$. Moreover, to maximize the transmit gain toward the target during the radar sensing phase, we employ a maximum ratio transmission (MRT) precoder at the UAV transmitter. Accordingly, the radar beamforming vector ${\mathbf{z}}_{\mathrm{rad}}$ is given by

$$\begin{array}{c}\hfill {\mathbf{z}}_{\mathrm{rad}}(\mathbf{u}\left(s\right),{\mathbf{a}}_t^{\mathrm{target}}\left(s\right))={\displaystyle \frac{{\mathbf{a}}_t^H\left({\varphi }_t^{\mathrm{ut}}\left(s\right)\right)}{|{\mathbf{a}}_t^H\left({\varphi }_t^{\mathrm{ut}}\left(s\right)\right)|}}.\end{array}$$

(14)

Notably, a normalization factor is in (14) which ensures compliance with the radar transmit power constraint. The radar beamforming and matched filter design at the UAV transceiver relies on steering vectors derived from the relative positions of the UAV, UEs, and targets. The assumption of ideal beamforming using ZF and MRT is justified by the LoS-dominant nature of UAV communication environments. UAVs typically operate at high altitudes, ensuring strong LoS links and enabling accurate channel estimation based on geometric information. Furthermore, since UEs are assumed to be equipped with positioning capabilities (e.g., GPS), their location information can be reliably shared with the UAV to construct precise steering vectors. However, in scenarios involving target mobility, location uncertainty can significantly challenge this beamforming design, potentially degrading both sensing and communication performance. To address this, we formulate an optimization problem aimed at jointly optimizing the UAV trajectory and uplink power control at UEs while achieving optimal beamforming and matched filtering designs based on the estimated target’s position. The objective is to maximize the radar SINR across S time steps while minimizing the UAV movement energy consumption, subject to constraints on the uplink communication rate. The problem is formulated as

$$\begin{array}{cc}\hfill \underset{\mathit{\eta },\mathbf{u},{\overline{\mathbf{a}}}_t^{\mathrm{target}}\left(s\right)}{max}& \sum _{s=1}^S{\gamma }_t^{\mathrm{rad}}\left(s\right)-{ϵ}_0\sum _{s=1}^S{E}_{\mathrm{uav}}^{\mathrm{mov}}\left(s\right).\hfill \end{array}$$

(15a)

$$\begin{array}{cc}\hfill s.t.& 0\le {\eta }_u\left(s\right)\le 1,\forall u\in {\mathcal{K}}_u,\hfill \end{array}$$

(15b)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em} |\mathbf{u}\left(s\right)-\mathbf{u}(s-1)|\le L_{\max },\forall s\in \mathcal{S},\hfill \end{array}$$

(15c)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}{h}_{\min }\le z^{\mathrm{uav}}\left(s\right)\le h_{\max },\forall s\in \mathcal{S},\hfill \end{array}$$

(15d)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{R}_u^{\mathrm{ul}}\left(s\right)\ge R_{\mathrm{QoS}},\forall u\in {\mathcal{K}}_u,\forall s\in \mathcal{S},\hfill \end{array}$$

(15e)

$$\begin{array}{cc}\hfill & \hspace{1em}{\gamma }_t^{\mathrm{rad}}\left(s\right)\ge {\gamma }_0^{\mathrm{rad}},\forall s\in \mathcal{S},\hfill \end{array}$$

(15f)

where the scalar ${ϵ}_0$ is a weighting coefficient (or trade-off parameter) that balances radar performance and energy efficiency. Constraint (15b) ensures compliance with the uplink power budget, constraint (15c) limits UAV mobility per time step, and constraint (15d) ensures that the UAV’s altitude remains within an allowable range. Constraint (15e) guarantees the quality of service (QoS) requirements for uplink UEs at each time step, and constraint (15f) ensures a minimum radar SINR for reliable sensing. The optimization problem (15) is inherently non-convex due to the complex nature of the objective function (15a) and constraints (15e) and (15f), which include exponential terms, high-order non-linear functions, and fractional expressions. These features make it challenging for traditional convex optimization methods to find globally optimal solutions, even when convex approximations are applied. Moreover, convex optimization techniques lack the ability to utilize historical data for predictive learning, which is essential in our dynamic, time-varying environment. Furthermore, the continuous and high-dimensional action space of problem (15) complicates the application of heuristic methods, which are generally better suited for discrete or simplified search spaces. Consequently, DRL provides a more flexible and effective framework to handle these challenges, enabling adaptive policy learning over continuous action domains.

While many prior studies often assume perfect target location estimates (e.g., using the last known position), such assumptions fail under fast-moving targets. To address this limitation, the proposed method exploits historical trajectory information for enhanced target position prediction. This data-driven approach enables more accurate estimation of future target locations, thereby enhancing robustness and reliability of sensing in dynamic movement scenarios. To tackle the complex, sequential decision-making problem under target location uncertainty, the LSTM-DDPG learning framework is employed. This model integrates the strengths of DRL for continuous control of trajectory and power and LSTM networks for modeling temporal dependencies and predicting future target positions. The proposed LSTM-DDPG framework is trained across diverse target movement scenarios to ensure robust generalization and effective joint optimization of UAV trajectory and UE power control over multiple time steps, while maintaining energy efficiency and radar reliability.

4. LSTM-DDPG-Based Approach

In this section, we present the LSTM-DDPG framework, developed to solve the joint trajectory and power control problem formulated in (15). The proposed method builds upon the DDPG algorithm, a reinforcement learning technique well-suited for continuous action spaces, by integrating it with LSTM networks to capture temporal dependencies in the environment. We begin by briefly reviewing the DDPG algorithm, followed by a description of how LSTM is incorporated to enhance learning performance under target mobility.

4.1. Background on DDPG

The optimization problem formulated in (15) involves continuous action spaces, rendering conventional deep reinforcement learning algorithms such as the deep Q-network (DQN) ineffective, since it is designed for discrete action spaces and typically exhibits poor convergence behavior when extended to continuous domains [42]. To address this, we adopt the DDPG algorithm, a model-free, off-policy actor–critic method that leverages deep neural networks to approximate both the policy and the value functions in continuous control settings. The DDPG framework consists of four neural networks: two evaluation networks for actor ($\mu$) and critic (Q), and two target networks for actor (${\mu }^{\prime }$) and critic ($Q^{\prime }$). The actor networks generate actions and interact with the environment, while the critic networks assess the actor’s performance to guide policy updates. The goal is to optimize the parameters (${\theta }^{\mu },{\theta }^q,{\theta }^{{\mu }^{\prime }},{\theta }^{q^{\prime }}$) of these networks to learn the optimal policy. Since the critic networks estimate continuous action value functions, the actor networks directly output optimal continuous actions without requiring a discrete action–value search, making DDPG well-suited for continuous control problems. Additionally, DDPG leverages experience replay to handle correlated sequential data, enhancing training stability and performance in continuous action reinforcement learning tasks.

4.2. Background on LSTM

LSTM networks are a specialized form of recurrent neural network (RNN) designed to capture long-term dependencies in sequential data. Traditional RNNs often struggle with retaining information over long time horizons due to issues such as vanishing or exploding gradients during training. To overcome these limitations, Hochreiter and Schmidhuber introduced LSTM architecture [43], which incorporates a memory cell and three gating mechanisms, namely input gate, forget gate, and output gate, that collectively control the flow of information through the network. This design enables LSTM networks to selectively retain, update, or discard information, making them highly effective for learning temporal patterns and dependencies in complex time-series data.

In the context of the ISAC-UAV system, where the target’s trajectory is uncertain and dynamically evolving, we employ an LSTM structure to predict future positions of the target by learning patterns from historical trajectory data. In this study, LSTM is selected over gated recurrent unit (GRU) and attention-based models based on three key considerations. First, given the limited onboard computational resources of UAV systems, LSTM provides an effective trade-off between modeling capacity and computational efficiency, enabling real-time inference with moderate complexity, unlike attention-based models, which typically require higher memory and processing power. Second, LSTM’s well-established gating mechanism ensures robust learning training and stable performance in dynamic and partially observable environments, which is crucial for reliable UAV trajectory control. Third, compared to GRU, LSTM is better equipped to capture long-term temporal dependencies, leading to improved decision-making in a sequential state setting. While GRU is more lightweight, it may sacrifice learning accuracy in complex scenarios. Therefore, LSTM was selected as a practical and effective solution for integration with the DDPG framework in our ISAC-UAV system. Integrating LSTM into the DDPG algorithm enables the model to adapt more effectively to dynamic environments, enhance its ability to learn from temporally correlated data, and improve generalization performance across diverse target movement scenarios.

4.3. Proposed LSTM-DDPG-Based Algorithm Architecture

A block diagram of the proposed LSTM-DDPG-based algorithm architecture for the ISAC-UAV system with four neural networks, denoted as q, $\mu$, $q^{\prime }$, and ${\mu }^{\prime }$, is presented in Figure 2. These networks are initialized with corresponding parameters ${\theta }^q,{\theta }^{\mu },{\theta }^{q^{\prime }},$ and ${\theta }^{{\mu }^{\prime }}$, serving as the initial weights for the respective networks. The input to both the evaluation and target actor networks is the environment state observed at each time step. Specifically, the state space $s_s$ at the s-th time step in the actor networks $\mu \left(s_s\right|{\theta }^{\mu })$ and ${\mu }^{\prime }\left(s_s\right|{\theta }^{{\mu }^{\prime }})$ is defined as

$$\begin{array}{cc}\hfill {\mathbf{s}}_s=& \{\mathbf{u}(s-1),{\mathbf{a}}^{\mathrm{ul}},{\mathbf{a}}_t^{\mathrm{target}}(s-\ell ),{\mathbf{a}}_t^{\mathrm{target}}(s-\ell -1),...,{\mathbf{a}}_t^{\mathrm{target}}(s-1)\},\hfill \end{array}$$

(16)

where ℓ represents the history window size used as input to the LSTM network, and ${\mathbf{a}}^{\mathrm{ul}}\triangleq [{\mathbf{a}}_1^{\mathrm{ul}},\dots ,{\mathbf{a}}_{K_{\mathrm{ul}}}^{\mathrm{ul}}]\in {\mathbb{C}}^{1\times K_{\mathrm{ul}}}$ represents the uplink UE positions. To effectively exploit the temporal dependencies in the target’s position history, the LSTM network is leveraged in the actor network of the DRL framework, as illustrated in Figure 3. The historical sequence of target positions at the s-th time step with the history window ℓ is fed into the LSTM module to capture temporal dynamics, while the current UAV and UE positions are processed through a separate feed-forward neural network. The outputs of both the LSTM and the neural network are then concatenated and fed into another fully connected network that generates the final action output of the actor.

The outputs of the evaluation actor $\mu \left({\mathbf{s}}_s\right|{\theta }^{\mu })$ and target actor ${\mu }^{\prime }\left({\mathbf{s}}_s\right|{\theta }^{{\mu }^{\prime }})$ networks represent the actions selected by the agent based on its observations of the environment. The corresponding action space ${\mathbf{a}}_s$ at the s-th time step is defined as

$${\mathbf{a}}_s=\left\{\mathbf{u}\left(s\right),{\overline{\mathbf{a}}}_t^{\mathrm{target}}\left(s\right),\mathit{\eta }\left(s\right)\right\}.$$

(17)

At each time step s, the agent utilizes the observed state $s_t$ to determine the corresponding action ${\mathbf{a}}_s$, which includes the UAV’s optimal 3D position $\mathbf{u}\left(s\right)$, the uplink power control coefficients $\mathit{\eta }\left(s\right)$, and the estimated target position ${\overline{\mathbf{a}}}_t^{\mathrm{target}}\left(s\right)$. Based on this estimate, the UAV employs ${\overline{\mathbf{a}}}_t^{\mathrm{target}}\left(s\right)$ to configure the matched filter $r_t(\mathbf{u}\left(s\right),{\overline{\mathbf{a}}}_t^{\mathrm{target}}\left(s\right))$ and the radar beamforming $z_{\mathrm{rad}}(\mathbf{u}\left(s\right),{\overline{\mathbf{a}}}_t^{\mathrm{target}}\left(s\right))$. The objective is to maximize the radar sensing performance while minimizing UAV movement energy consumption. Consequently, the reward $r_s$ at the s-th time step in the ISAC-UAV system is defined as

$$r_s={\gamma }_t^{\mathrm{rad}}\left(s\right)-{ϵ}_0{E}_{\mathrm{uav}}^{\mathrm{mov}}\left(s\right).$$

(18)

During the training phase, the agent collects experience tuples of the form $({\mathbf{s}}_s,{\mathbf{a}}_s,r_s,{\mathbf{s}}_{s+1})$, which are stored in a replay buffer $\mathcal{B}$ with a fixed capacity. This buffer enables the agent to sample past experiences uniformly, thereby reducing the correlation between sequential data and improving training stability. The critic networks utilize samples from $\mathcal{B}$ to learn the mapping between states and actions. Specifically, the input to both the evaluation and target critic networks is the concatenation of the current state $\mathbf{s}$ and the corresponding action $\mathbf{a}$. Similar to the actor network, the critic network is enhanced with an LSTM module to capture temporal dependencies in the target’s trajectory, enabling more accurate state–action value estimation, as shown in Figure 4. Concurrently, the current positions of the UAV and uplink UEs are processed through a dedicated feed-forward neural network to extract spatial features. Additionally, the action taken by the agent is processed separately through another neural network designed to extract action-specific features.

The output of the LSTM module, the spatial feature extractor and the action feature extractor, are concatenated and passed through a final fusion neural network to produce the Q-value $Q({\mathbf{s}}_s,{\mathbf{a}}_s)$. This architecture enhances the critic’s ability to accurately estimate value functions by learning complex interactions among temporal dynamics, spatial configurations, and action decisions in highly dynamic environments.

4.4. Proposed DDPG-Based Optimization Procedure

Given a mini-batch of $N_B$ transitions ${\left\{({\mathbf{s}}_i,{\mathbf{a}}_i,r_i,{\mathbf{s}}_{i+1})\right\}}_{i=1}^{N_B}$ sampled from the replay buffer $\mathcal{B}$, the target Q-value $y_i$ is computed based on the Markov decision process framework as

$$\begin{array}{c}\hfill y_i=r_i+\xi Q^{\prime }\left(\left({\mathbf{s}}_{i+1},{\mu }^{\prime }\left(s_{i+1}\right|{\theta }^{{\mu }^{\prime }})\right)|{\theta }^{q^{\prime }}\right),\end{array}$$

(19)

where $\xi$ denotes the discount factor applied to future rewards. The following loss function is minimized to update the parameters ${\theta }^q$ of the evaluation critic network:

$$\begin{array}{c}\hfill \mathcal{L}\left({\theta }^q\right)={\displaystyle \frac{1}{N_B}}\sum _{i=1}^{N_B}{\left(y_i-Q\left((s_i,{\mathbf{a}}_i)|{\theta }^q\right)\right)}^2.\end{array}$$

(20)

At each time step, the parameters ${\theta }^q$ of the evaluation critic network are updated using the gradient descent method:

$$\begin{array}{c}\hfill {\theta }^q\leftarrow {\theta }^q-{ϵ}_2{\nabla }_{{\theta }^q}\mathcal{L}\left({\theta }^q\right),\end{array}$$

(21)

where ${ϵ}_2\in [0,1]$ denotes the learning rate of the critic networks, and $\nabla (·)$ represents the first-order partial derivative.

Once the critic network is updated, the policy network updates the parameters of the evaluation actor network by using the observed input states $s_t$ from the mini-batch $N_B$ to maximize the expected reward. Based on the deterministic policy gradient theorem [44], the following gradient is applied to update the weights ${\theta }^{\mu }$ of the evaluation actor network:

$$\begin{array}{cc}\hfill {\nabla }_{{\theta }^{\mu }}\mathcal{J}\left({\theta }^{\mu }\right)\approx & {\displaystyle \frac{1}{N_B}}\sum _{i=1}^{N_B}{\nabla }_{{\theta }^{\mu }}\mu \left(s_i\right|{\theta }^{\mu })\hfill \\ \hfill & \times {\nabla }_{a}Q(s_i,a|{\theta }^q)|a=\mu \left(s_i\right|{\theta }^{\mu }),\hfill \end{array}$$

(22)

where ${\nabla }_{a}Q(s_i,a|{\theta }^q)$ is obtained through backpropagation in the evaluation critic network Q using the actions predicted by the evaluation actor network $\mu$. The parameters ${\theta }^{\mu }$ of the evaluation actor network are updated using the gradient descent algorithm as

$$\begin{array}{c}\hfill {\theta }^{\mu }\leftarrow {\theta }^{\mu }+{ϵ}_1{\nabla }_{{\theta }^{\mu }}\mathcal{J}\left({\theta }^{\mu }\right),\end{array}$$

(23)

where ${ϵ}_1\in [0,1]$ represents the learning rate for the actor networks. To enhance training stability and mitigate divergence issues in Q-learning, DDPG employs soft updates by gradually adjusting the parameters of the target networks to follow those of the learned evaluation networks. Specifically, the target actor network $Q^{\prime }((\mathbf{s},\mathbf{a})\mid {\theta }^{q^{\prime }})$ and the target critic network ${\mu }^{\prime }(\mathbf{s}\mid {\theta }^{{\mu }^{\prime }})$ are cloned to compute their respective target values. Their parameters are then updated as

$$\left\{\begin{array}{c}{\theta }^{q^{\prime }}\leftarrow \tau {\theta }^q+(1-\tau ){\theta }^{q^{\prime }},\hfill \\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }},\hfill \end{array}\right.$$

(24)

where $\tau \ll 1$ is the soft update coefficient that controls the adaptation speed of the target networks. Exploring continuous action spaces poses unique challenges [45]. To promote effective exploration, DDPG introduces a simple yet effective method by adding random noise to the action space. The exploration action ${\mathit{a}}_t$ is formed by injecting noise sampled from ${\mathcal{N}}_t$ into the evaluation actor policy $\mu$, enhancing exploration throughout the learning process:

$$\begin{array}{c}\hfill {\mathit{a}}_t=\mu \left(s_i\right|{\theta }^{\mu })+{\mathcal{N}}_t,\end{array}$$

(25)

where the noise sample ${\mathcal{N}}_t$ follows an Ornstein–Uhlenbeck (OU) process to generate temporally correlated random noise, as described in [42]. This inclusion of time-correlated noise plays a crucial role in facilitating more efficient and effective exploration throughout the learning process.

Eventually, the training procedure for the proposed LSTM-DDPG framework is detailed in Algorithm 1.

Algorithm 1 Proposed LSTM-DDPG-based Framework to Solve (15).

Initialization: Initialize $Q({\mathbf{s}}_s,{\mathbf{a}}_s,{\theta }^q),\mu ({\mathbf{s}}_s,{\mathbf{a}}_s,{\theta }^{\mu })$, $Q^{\prime }({\mathbf{s}}_s,{\mathbf{a}}_s,{\theta }^{q^{\prime }}),$ and ${\mu }^{\prime }({\mathbf{s}}_s,{\mathbf{a}}_s,{\theta }^{{\mu }^{\prime }})$ using ${\theta }^{q^{\prime }}={\theta }^q$ and ${\theta }^{{\mu }^{\prime }}={\theta }^{\mu }$. Set up $\mathcal{B}$, ${ϵ}_1,{ϵ}_2$, $\xi$, $N_B$, S, and ${\mathrm{ep}}_{\max }$. $\mathbf{Set}$$\widehat{\mathrm{ep}}\leftarrow 1,\mathrm{ep}\leftarrow 1,\mathrm{s}\leftarrow 1.$ for $\mathrm{eq}=1$ to ${\mathrm{ep}}_{\max }$ do     Sample a dataset ${\mathcal{D}}_i\sim \{{\mathcal{D}}_1,...,{\mathcal{D}}_N\}$.     for $\overline{\mathrm{ep}}=1$ to ${\overline{\mathrm{ep}}}_{\max }$ do         Set the UAV position at time step 0 $\mathbf{u}\left(0\right)$ and gather ℓ historical target positions.         Set the initial state ${\mathbf{s}}_1$.         for $\mathrm{s}=1$ to S do            Select action ${\mathbf{a}}_s=\mu \left({\mathbf{s}}_s\right|{\theta }^{\mu })+{\mathcal{N}}_s$.            Extract $\left(\mathbf{u}\left(s\right),{\overline{\mathbf{a}}}_t^{\mathrm{target}}\left(s\right),\mathit{\eta }\left(s\right)\right)$.            Calculate ${\gamma }_t^{\mathrm{rad}}\left(s\right)$ and $E_{\mathrm{uav}}^{\mathrm{mov}}\left(s\right)$ using (11) and (12).            Determine the next state ${\mathbf{s}}_{s+1}$ based on (16).            Calculate reward $r_s$ using (18).            If constraints (15e) and (15f) violated, set $r_s\leftarrow 0$.            Store $\left({\mathbf{s}}_s,{\mathbf{a}}_s,r_s,s_{s+1}\right)$ in $\mathcal{B}$.            Randomly sample a mini-batch of size $N_B$ from $\mathcal{B}$.            Update evaluation critic network using (21).            Update evaluation actor network using (23).            Update target actor and target critic networks using (24).         end for     end for end for

5. Numerical Results

5.1. Simulation Setup

5.1.1. ISAC-UAV System Setup

To realistically simulate the UAV’s physical movement, we adopt mechanical parameters listed in

Table 1. Simulation parameters for UAV movement.

Parameter	Value
Tip speed of rotor blade (${\ell }_{\mathrm{tip}}$)	120 m/s
Fuselage equivalent flat plate area (${\ell }_{\mathrm{flat}}$)	$0.0151{\mathrm{m}}^2$
Fuselage drag ratio (${\ell }_{\mathrm{fus}}$)	$0.6$
Air density (${\ell }_{\mathrm{air}}$)	$1.225{\mathrm{kg}/\mathrm{m}}^3$
Rotor solidity (${\ell }_{\mathrm{sol}}$)	$0.05$
Rotor disc area $\left({\ell }_{\mathrm{disc}}\right)$	$0.503{\mathrm{m}}^2$
Weight of UAV (${\ell }_{\mathrm{wei}})$	20 Newton
Profile drag coefficient $\left({\ell }_{\mathrm{drag}}\right)$	$0.012$
Blade angular velocity (${\ell }_{\mathrm{ang}}$)	300 $\mathrm{rad}/\mathrm{s}$
Blade or aerofoil chord length (${\ell }_{\mathrm{aer}}$)	0.0157 $\mathrm{m}$
Number of blades (${\ell }_{\mathrm{num}}$)	4
Rotor radius $\left({\ell }_{\mathrm{rad}}\right)$	$0.4\mathrm{m}$
Incremental correction factor (${\ell }_{\mathrm{incre}}$)	0.1
Mean rotor-induced velocity in hover ($v_0$)	$4.03\mathrm{m}/\mathrm{s}$

, following the setup in [41]. Specifically, the rotor disc area, tip speed, rotor solidity, fuselage drag ratio, and mean rotor induced velocity are computed as follows: ${\ell }_{\mathrm{disc}}\triangleq \pi {\ell }_{\mathrm{rad}}^2$, ${\ell }_{\mathrm{tip}}\triangleq {\ell }_{\mathrm{ang}}{\ell }_{\mathrm{rad}}$, ${\ell }_{\mathrm{sol}}\triangleq \left({\ell }_{\mathrm{num}}{\ell }_{\mathrm{aer}}\right)/\left(\pi {\ell }_{\mathrm{rad}}\right)$, ${\ell }_{\mathrm{fus}}\triangleq {\ell }_{\mathrm{flat}}/\left({\ell }_{\mathrm{sol}}{\ell }_{\mathrm{disc}}\right)$, and $v_0\triangleq {({\ell }_{\mathrm{wei}}/\left(2{\ell }_{\mathrm{air}}{\ell }_{\mathrm{disc}}\right))}^{0.5}$, where ${\ell }_{\mathrm{num}}$ and ${\ell }_{\mathrm{aer}}$ denote the number of rotor blades and the aerofoil chord length, respectively. All relevant parameters are provided in detail in Table 1.

The UAVs, UEs, targets, and clutters are initially randomly distributed within a square area of size $L\times L$, with $L\times L$, with $L=100$ m, for simulation convenience. However, the UAV’s position is optimized dynamically over the entire time horizon without spatial constraints, allowing it to adapt its trajectory based on the target’s movement. Following the configuration in [46], the radar SINR threshold ${\gamma }_0^{\mathrm{rad}}$ is set to 12 dB to ensure a target detection probability above 90% while maintaining a false alarm rate of ${10}^{-4}$. The simulation parameters for wireless communication are listed in

Table 2. Simulation parameters for an ISAC-FD-CFMM system.

Parameter	Value
Path loss at reference distance $\left({\mathcal{C}}_0\right)$	$-30\mathrm{dB}$
The path loss exponent $\left(\alpha \right)$	2
Noise power $\left({\sigma }^2\right)$	$-104\mathrm{dBm}$
Maximum flight speed of UAV ($v_{\max }$)	$20\mathrm{m}/\mathrm{s}$
Target velocity ($v_{\mathrm{target}}$)	$20\mathrm{m}/\mathrm{s}$
Time step duration	$300\mathrm{ms}$
Required QoS threshold ($R_{\mathrm{QoS}}$)	$0.5\mathrm{bps}/\mathrm{Hz}$
Power budget at UL UEs ($p_{\max }^{\mathrm{ul}}$)	$23\mathrm{dBm}$
Power budget at DL UAVs ($p_{\max }^{\mathrm{dl}}$)	$30\mathrm{dBm}$
Radar SINR threshold (${\gamma }_0^{\mathrm{rad}}$)	$12\mathrm{dB}$

, based on [47].

5.1.2. LSTM-DDPG-Based Algorithm Configuration

Figure 5 presents the overall architecture of the actor and critic networks in the proposed LSTM-DDPG framework. The actor network processes the state ${\mathbf{s}}_t$, which includes the historical target positions and the current positions of the UAV and UEs. The target’s position history is passed through an LSTM layer, while the spatial features of the UAV/UEs are processed via a fully connected layer, both using ReLU activation. The outputs are concatenated and passed through two additional FC layers with ReLU activation, followed by a final Sigmoid-activated output layer to generate the action ${\mathbf{a}}_t$. The critic network takes both ${\mathbf{s}}_t$ and ${\mathbf{a}}_t$ as inputs, processes them through similar structures with ReLU activation, concatenates the outputs, and passes them through two fully connected layers, concluding with a linear-activated output to estimate the Q-value.

The key hyperparameters of the proposed LSTM-DDPG-based framework are as follows. The soft update coefficient $\tau$ is set to 0.005. The learning rates for the actor and critic models, ${ϵ}_1$ and ${ϵ}_2$, are both set to 0.001. The discount factor $\xi$ is 0.99. The replay buffer memory size $\mathcal{B}$ is 1,000,000, and the mini-batch size $N_B$ is 32. The number of time steps S is 30. The history window size ℓ is 5. The total number of episodes $e{p}_{\max }$ is 2000, with a pre-training phase of ${\overline{ep}}_{\max }=100$ episodes. The sizes of the fully connected layers $L_1$ through $L_9$ are all set to 128. Simulations were performed on a desktop computer equipped with an Intel Core i9-13900KF CPU running at 3.40 GHz and 32 GB of RAM, using TensorFlow 2.9.1 and Python via Visual Studio Code. To train the proposed DRL framework, we generate a dataset consisting of 500 diverse training samples. Each sample includes a unique set of zigzag target trajectories, as well as randomized initial positions for the UAV and ground UEs within a square area of size $L\times L$. This diversity enables the model to learn robust policies that generalize well to unseen scenarios. Specifically, the zigzag movement model is employed to more accurately capture the non-linear and dynamic motion patterns commonly observed in real-world ISAC-UAV applications such as surveillance, target tracking, and mobile sensing [48]. Unlike straight-line motion, this model simulates more realistic mobility by having the target move at a constant speed while periodically changing direction by a fixed angle ${\theta }_z$, resulting in a piecewise linear trajectory. Straight-line motion is a special case when ${\theta }_z=0^{\circ }$. By incorporating this model, we aim to evaluate the ISAC-UAV system under more challenging and realistic conditions, thereby improving the practical relevance and robustness of the proposed approach. In reinforcement learning, particularly in continuous control tasks such as UAV navigation and beamforming, dataset size alone does not directly determine training effectiveness as it does in supervised learning [49]. Instead, the diversity and representativeness of the experience replay buffer, combined with exploration strategies, play a critical role in learning performance. Our dataset of 500 samples was carefully generated to cover a wide range of UAV states, target positions, and movement patterns, including the zigzag mobility model, thereby ensuring sufficient coverage of the state–action space. Furthermore, the DDPG algorithm continuously interacts with the environment during training, updating policies based on newly collected experience, which enables the model to adapt well beyond the initial dataset limitations. All numerical results are averaged over 100 independent testing datasets.

To comprehensively evaluate the performance of the proposed method, we compare it with several benchmark schemes:

(1)

DDPG-based approach: This approach employs the standard DDPG algorithm to solve problem (15), without leveraging historical target position data.

(2)

KF-DDPG-based approach: In this approach, the LSTM module in the DDPG-based framework is replaced with a Kalman filter (KF) to predict the target’s future position. The KF is commonly used in traditional ISAC systems due to its simplicity and effectiveness in estimating dynamic states under linear system dynamics and Gaussian noise assumptions.

(3)

LSTM-DDGP-based approach in 2D: This approach employs the proposed LSTM-DDPG method in a fixed-altitude (2D) UAV configuration to assess the effect of UAV altitude on system performance.

(4)

LSTM-DDPG-based approach in 3D: This approach corresponds to the full implementation of the proposed LSTM-DDPG framework, solving the complete 3D optimization problem defined in (15), including the UAV altitude.

5.2. Simulation Results

As a result, the proposed learning framework enables the UAV to perform more efficient trajectory planning and resource allocation, thereby reducing movement-related energy consumption while simultaneously enhancing radar performance. Figure 6a,b show the cumulative reward curves achieved by the LSTM-DDPG model and the baseline DDPG model without LSTM integration, respectively, under the parameter setting $K_{ul}=2$ and $C=4$ for solving the problem (15) in the ISAC-UAV system. In Figure 6a, the LSTM-DDPG exhibits rapid convergence and stable performance, achieving near-optimal cumulative rewards within approximately 1000 episodes. The curve remains smooth with minimal variance, indicating the effectiveness of the LSTM module in capturing temporal dependencies and enhancing the agent’s decision-making capabilities across sequential states. In contrast, the baseline DDPG model in Figure 6b exhibits a much slower learning process. It shows significantly slower convergence, requiring over 8000 episodes to reach a comparable reward level. Moreover, its learning trajectory is marked by pronounced instability and oscillations throughout training. These findings clearly demonstrate that the proposed LSTM-DDPG framework outperforms the conventional DDPG approach in terms of both convergence speed and policy robustness. The integration of LSTM layers enables the agent to retain and leverage historical state information, thereby significantly enhancing its ability to predict future target positions and adapt to dynamic environmental conditions. Figure 7 presents a 3D visualization of the UAV’s trajectory generated by the trained LSTM-DDPG-based learning model. The target’s movement is illustrated from the first to the S-th time step. In response, the UAV adapts its trajectory, including altitude adjustments, based on the learned policy to effectively track the target, avoid clutter, and maintain reliable uplink communication with the UEs. The visualization highlights the UAV’s ability to dynamically plan efficient paths in complex and time-varying environments. By continuously adjusting its position to preserve sensing quality and communication reliability, the UAV minimizes unnecessary movements and energy consumption while ensuring high radar performance. These results confirm the effectiveness of the proposed LSTM-DDPG framework in enabling energy-efficient and adaptive trajectory planning in dynamic scenarios.

To evaluate energy efficiency, we compare the results obtained by varying the weighting coefficient ${ϵ}_0$, as shown in Figure 8. The coefficient ${ϵ}_0$ in the reward function governs the trade-off between sensing performance and UAV movement energy consumption. As ${ϵ}_0$ increases, the model increasingly prioritizes energy savings, potentially at the expense of radar sensing accuracy, and vice versa. Figure 8 shows the UAV trajectories for ${ϵ}_0=0.1$ and ${ϵ}_0=1$. It can be observed that as ${ϵ}_0$ increases, the trajectory between the initial and final steps becomes shorter, reflecting the model’s preference for energy-efficient paths. This behavior confirms the model’s adaptability in adjusting to different optimization objectives. Moreover, these results emphasize the importance of carefully selecting ${ϵ}_0$ to achieve a desirable balance between radar performance and energy efficiency. Figure 9 further supports this finding by showing a clear reduction in UAV energy consumption as ${ϵ}_0$ increases from 0.1 to 1. The model increasingly favors shorter trajectories, thereby minimizing movement-related energy usage.

Figure 10 illustrates the convergence behavior of the proposed LSTM-DDPG framework for solving problem (15) under various learning rates ${ϵ}_1$ and ${ϵ}_2$ over 1600 training episodes. The learning rate plays a crucial role in controlling the step size of weight updates during training, thereby significantly influencing both convergence speed and training stability. Among the configurations tested, ${ϵ}_1=0.001$ and ${ϵ}_2=0.001$ yielded the best most stable and efficient convergence. These values were therefore adopted as the default learning rates for all subsequent experiments.

To demonstrate the superior performance of the proposed framework, we conducted extensive comparisons with several benchmark algorithms. As discussed earlier, increasing the value of ${ϵ}_0$ places greater emphasis on energy efficiency, which may lead to degraded radar sensing performance and vice versa. This trade-off is illustrated in Figure 11, which presents the effects of varying ${ϵ}_0$ from 0.1 to 1. The proposed LSTM-DDPG framework consistently outperforms the benchmark approaches, achieving the highest radar sensing performance while simultaneously minimizing UAV movement energy consumption. Among the baselines, the KF-DDPG-based approach performs the worst in radar performance, particularly when compared to the framework that employs DDPG solely. This stems from its reliance on a KF to predict the target’s next position. While KF is effective for linear motion under Gaussian noise, it is ill-suited for non-linear target trajectories, resulting in large prediction errors. These inaccuracies hinder the learning process when integrated with DDPG, ultimately degrading overall system performance. In contrast, the integration of LSTM into the DDPG-based model allows the learning framework to effectively capture and adapt to complex, non-linear motion patterns. Furthermore, the proposed framework also outperforms a variant of the LSTM-DDPG model that does not optimize the UAV’s altitude, underscoring the importance of full 3D trajectory optimization. By enabling the UAV to dynamically adjust its altitude, the proposed model achieves substantial performance gains over all benchmark algorithms. A similar trend is observed in Figure 12, which presents the comparison results for different values of the number of UEs K. As K increases, the radar sensing performance of the ISAC-UAV system tends to degrade due to increased uplink interference. Despite this, the proposed LSTM-DDPG framework consistently achieves the best performance, demonstrating its robustness and effectiveness in more densely populated and interference-limited scenarios.

Finally, we evaluate the model complexity by measuring the processing time required for UAV trajectory and resource allocation optimization in a representative test scenario. As shown in

Table 3. The processing time for the methods.

Method	Latency (ms)
DDPG approach	$12.21$
KF-DDPG approach	$16.36$
LSTM-DDPG approach in 2D	$40.23$
LSTM-DDPG approach in 3D	$41.76$

, the results highlight a trade-off between performance and computational complexity.

6. Conclusions

In this study, we investigated an ISAC-UAV system and designed a novel DRL framework to jointly optimize the UAV’s trajectory and uplink power, alongside beamforming and matched filtering design, with the goal of maximizing performance while minimizing the UAV’s energy consumption. To address the complexity of this optimization problem, we adopted a DDPG-based DRL approach. However, a primary challenge in this setting lies in the uncertainty of the target’s position due to its movement, which can hinder the effectiveness of the learning process. To overcome this limitation, we integrated an LSTM model into both the actor and critic networks of the DDPG framework, enabling the system to better capture temporal dependencies and accurately estimate the target’s motion. The resulting LSTM-DDPG framework demonstrated strong robustness and significantly improved learning stability and performance compared to benchmark algorithms. Our experimental results show that the proposed framework effectively handles non-linear target trajectories, ensuring reliable uplink communication for UEs while optimizing radar sensing and minimizing the UAV’s energy consumption in a dynamic environment. In the future, we will extend the learning model structure to handle more complex target motion patterns. Additionally, we aim to scale the ISAC-UAV system to accommodate multiple UAVs and multiple moving targets, further enhancing its applicability to real-world scenarios. Future work will extend the current model by accounting for residual SI after cancellation and beamforming imperfections caused by practical factors such as channel estimation errors and hardware impairments. Incorporating these aspects will enable a more realistic and robust performance analysis, providing deeper insights into the practical design and deployment of ISAC-enabled UAV systems. Future work will focus on developing a theoretical analysis to characterize the convergence and performance bounds of the proposed LSTM-DDPG framework. This endeavor is challenging due to the inherent non-convexity of DRL algorithms and the non-Markovian properties of LSTM networks. Eventually, future work will focus on extending the proposed framework to multi-UAV scenarios to improve scalability, coverage, and robustness. This expansion will address new challenges including UAV coordination, interference mitigation, and distributed cooperative trajectory optimization.