UAV-Enabled Diverse Data Collection via Integrated Sensing and Communication Functions Based on Deep Reinforcement Learning

Abstract

Unmanned aerial vehicles (UAVs) and drones are considered to represent a flexible mobile aerial platform to collect data in various applications. However, the existing data collection methods mainly consider uplink communication. The burgeoning development of integrated sensing and communication (ISAC) provides a new paradigm for data collection. A diverse data collection framework is established where the uplink communication and sensing functions are both considered, which can also be referred to as the uplink ISAC system. An optimization is formulated to minimize the data freshness indicator for communication and the detection freshness indicator for sensing by optimizing the UAV paths, the transmitted power of IoT devices and UAVs, and the transmission allocation indicators. Three state-of-the-art deep reinforcement learning (DRL) algorithms are utilized to solve this optimization. Experiments are conducted in both single-UAV and multi-UAV scenarios, and the results demonstrate the effectiveness of the proposed algorithms. In addition, the proposed algorithms outperform the benchmark in terms of accuracy and efficiency. Moreover, the effectiveness of the data collection mode with only communication or sensing functions is also verified. Also, the numerical Pareto front between communication and sensing performance is obtained by adjusting the importance parameter.

1. Introduction

1.1. Background

Data collection is an essential task in the modern decision-making process. In numerous fields, from environmental monitoring to urban planning and disaster management, accurate and timely data are the foundation of formulating effective strategies and responses [1]. High-quality data collection can reveal hidden patterns, predict future trends, and allow for real-time monitoring, leading to better resource allocation and risk mitigation. Without the support of reliable data, policy-making might be based on incorrect assumptions, leading to a waste of resources and a loss of opportunities [2]. Therefore, accurate, efficient, and massive data collection is crucial for enhancing decision-making’s effectiveness in various applications, especially in disaster management [3].

Unmanned aerial vehicles (UAVs)/drones offer an extremely flexible solution for data collection, capable of rapid deployment to areas that are difficult to reach or hazardous and being able to adjust their flight paths on the fly to meet varying data collection needs [4,5,6]. Compared to establishing fixed stations, UAVs are more cost-effective, particularly for short-term or temporary projects [7,8]. They can also be loaded with various sensors [9,10]. These sensors enable the Internet of Things (IoT) to provide real-time data, which are crucial for emergency responses and live monitoring [11,12]. Moreover, UAVs reduce safety risks for personnel, especially when operating in dangerous environments like volcanoes, areas with nuclear radiation, and conflict zones. They can cover vast geographical areas, respond swiftly to natural disasters, and collect essential information to support rescue and recovery efforts, all while minimizing environmental disruption [13]. Although the use of UAVs is limited by factors like battery life, weather conditions, and regulations, they offer clear advantages in terms of dynamism, flexibility, and cost-effectiveness compared to fixed stations [14].

1.2. Related Works

The existing data collection methods mainly consider the uplink communication, where UAVs serve as a key receiving aerial platform, responsible for collecting data transmitted by IoT devices in the region of interest [15,16,17]. This scheme enables UAVs to rapidly gather information from distributed sensor networks across vast areas. The high maneuverability and deployability of UAVs make them ideal relay nodes for connecting ground-based IoT devices with larger network infrastructures [18,19]. Through the uplink communication link, UAVs can effectively collect various types of data, including environmental monitoring, facility status, real-time events, etc., and then transmit these data to the cloud or ground base stations or servers for further analysis and decision-making support [20,21].

Some research has focused on optimizing the deployment, resource allocation, and scheduling of UAVs to enhance communication efficiency and service quality in IoT networks. These studies address the balance between different performance metrics, such as effective rate, transmission delay, spectral efficiency, and interference management. Hellaoui et al. [20] proposed optimizing multi-service aerial communication in an IoT network based on UAVs by allocating resources and deploying UAVs to balance the trade-off between effective rate and transmission delay. Feng et al. [22] proposed an IoT communication model that used UAVs as aerial base stations in the absence or overload of ground base stations. By optimizing communication scheduling, the public rate, transmission power, and UAV flight trajectories, the system aimed to maximize throughput. An alternating iterative algorithm was employed to solve the non-convex optimization problem, effectively enhancing the communication rate for ground users. Cai et al. [23] proposed different power allocation algorithms for the uplink communication of numerous cellular-connected UAVs. The objective was to optimize the minimum spectral efficiency or overall spectral efficiency based on the principle of successive convex approximation, to address the severe interference issues in high-density UAV deployment scenarios. Chen et al. [24] proposed a quality of service guaranteed multi-UAV coverage scheme, which optimized the deployment of UAVs through interference management and spectrum resource allocation to enhance the average UAV capacity within IoT communication hotspots, meeting various service quality requirements. Duan et al. [25] studied a multi-UAV-assisted IoT non-orthogonal multiple access uplink transmission system, where the system capacity was maximized by jointly optimizing sub-channel allocation, the uplink transmission power of the IoT nodes, and the flight altitude of UAVs. They proposed an algorithm based on K-means clustering and matching theory, which was suitable for scenarios requiring rapid deployment and efficient data collection.

Some researchers have focused on the optimization of drone trajectories and positioning to improve various aspects of communication, such as latency, power efficiency, and secrecy performance. Eldeeb et al. [26] utilized a deep reinforcement learning (DRL) framework, combined with traffic prediction techniques, to optimize the trajectory and scheduling strategies of multiple UAVs, thereby reducing the information update delay, the average transmission power of devices, and the cumulative regret in the IoT network. Yin et al. [27] studied how to enhance the secrecy performance of the uplink in satellite-supported IoT scenarios through UAV-assisted communication, aiming to achieve confidentiality and fairness among IoT users. This was primarily accomplished by optimizing the uplink power of IoT users, the position of the UAV, and beamforming.

Some research has been dedicated to analyzing and optimizing the communication performance of UAVs in specific application scenarios (such as rural areas, disaster zones, or equipment-dense networks), including reliability, efficiency, and connectivity. Liu et al. [18] investigated the uplink transmission performance of a wireless backhaul network, where UAVs were used as relays to assist user devices in transmitting data to remote ground base stations in scenarios with a lack of network coverage, such as rural areas or disaster zones. By considering the distribution of user devices and the positional changes of the UAVs, the authors developed a theoretical model to analyze the connectivity of this two-hop uplink transmission path. Nabil et al. [21] proposed a communication scheme to enhance the reliability and efficiency of device-centric uplink communications in UAV-supported aerial networks by optimizing the ratio of hovering to moving time of the UAVs.

In recent years, integrated sensing and communication (ISAC) has emerged as a new paradigm for data collection. ISAC integrates communication and sensing functions by utilizing shared hardware platforms and spectrum resources, which enhances spectral efficiency and reduces resource costs [28,29]. It not only optimizes the utilization of hardware but also enhances the flexibility of the system, allowing for both data transmission and environmental sensing to occur simultaneously on the same resource blocks [30,31,32,33,34].

In an ISAC system, Liu et al. [35] addressed the issue of multi-user uplink communication security against mobile aerial eavesdroppers by predicting the movement state of the aerial eavesdroppers and jointly designing radar signals and receive beamformers, thereby achieving an enhancement in the system’s minimum signal-to-noise ratio while ensuring the security and fairness of multi-user communication. Zhou et al. [36] proposed an integrated system of sensing, computing, and communication for drone-enabled IoT, enabling sensing of remote user devices, task computation, and data processing collaboration with access points. By jointly optimizing the CPU frequency of the drone, the radar sensing power, the transmission power of user devices, and the flight trajectory, the goal was to minimize the weighted total energy consumption of both the drone and user devices. In the context of emergency IoT networks, Zhu et al. [37] proposed an ISAC pilot optimization scheme. By employing a particle swarm optimization algorithm to allocate time–frequency resources, they aimed to minimize the bit error rate and average ranging error, thereby enhancing communication and positioning performance under dynamic channel conditions. In an IoT environment without infrastructure support, Liu et al. [38] optimized the three-dimensional flight trajectories and resource allocation of UAVs to enhance radar estimation in integrated sensing and communication systems.

However, as far as we know, the existing works present several gaps and have some drawbacks as follows. First, the existing works usually focus on traditional data collection methods with only communication, but rarely exploit the radar sensing capabilities of UAVs for data collection. Another gap exists in the consideration of the differences in source, type, and collection methods between data collected by IoT devices and radar systems. Moreover, there is a lack of effective resource sharing and interference management mechanisms to handle the simultaneous execution of communication and sensing functions, thereby achieving efficient, accurate, and comprehensive data collection. Additionally, there is a gap in the assessment of fairness during the data collection process, and the existing works often consider only the performance of communication and sensing functions.

1.3. Contributions of This Paper

Motivated by ISAC, to tackle the above drawbacks in the existing works, we propose a diverse data collection framework where uplink communication and sensing are simultaneously considered, as depicted in Figure 1. Differing from the existing works, UAVs serve not only as receiving devices for collecting uplink communication data from IoT devices but also execute active sensing tasks by emitting radar pulses and receiving the echoes. Such a framework can also be referred to as an uplink ISAC system, where the uplink communication (from IoT device to UAV) and the sensing (between UAV and sensing targets) are both considered with a time division scheme to avoid mutual interference. Compared to traditional passive sensing methods that use only optical or infrared cameras, radar sensing allows UAVs to gather information in complex environments and operate around the clock, especially in environments with limited line-of-sight or poor lighting conditions. This framework provides an accurate, reliable, and comprehensive data collection scheme, supporting the realization of autonomous decision-making and response.

In the proposed scheme, besides the data provided by optical cameras, such as the images, videos, etc., the data collected by IoT devices and the data gathered by radar pulses also have distinctly different characteristics in terms of their sources, types, and collection methods. IoT devices, as physical devices connected to the Internet, passively monitor and record digital signals of environmental changes such as temperature, humidity, and light levels through built-in sensors. The data are typically structured, for instance as time-series data, images, and sound waveforms, which are easier to store and process compared to unstructured data. In contrast, radar systems actively emit radio waves and analyze the signals reflected to detect the position, speed, and direction of objects. The data collected include information on distance, speed, angle, and size, often analog signals that require conversion and complex processing to extract useful information and are typically unstructured.

The proposed framework introduces some new challenges. First, the shared resources between communication and sensing cause congestion and interference. This is manifested not only in the competition for scarce resources between the two functions but also in the mutual interference (i.e., interference of IoT signals to sensing and interference of echo signals to communication). In addition, the complex coupled objectives and various decision variables increase the complexity of the problem, where not only the complicated relationships between the communication and sensing dual functions but also the relationships between the IoT devices influence the system’s performance. Moreover, the maneuverability of UAVs adds another layer of difficulty to this issue. This involves a complex optimization problem to find the optimal UAV position, power allocation, and association allocation.

Such a problem is complex and hard to solve because it involves many different variables and the relationships between these variables and the goals are complicated. Classic convex optimization is often good for smaller problems with simple goals. Also, the proposed two indicators, the data freshness indicator and the detection freshness indicator, are hard to reshape into a form that works well with convex optimization. The evolutionary algorithm and the gradient algorithm, other common methods, have trouble dealing with dynamic problems and have many time points to consider.

Therefore, considering the large number of time nodes and their dynamic nature, we adopt the DRL to solve this optimization. To implement the DRL, the optimization is converted into a decision-making problem regarding UAV path planning: when and where a UAV should fly to provide communication services to a particular IoT device, or when and where a UAV should move to sense specific sensing targets.

Employing DRL to tackle the problem has some advantages. It makes choices over time and discovers the best long-term plan through a series of decisions, which is well-suited for working out the best long-term solutions. In addition, since the DRL keeps working with the environment, it can get used to changes and unknowns in complex situations.

To address the above difficulties, we establish a diverse data collection framework in which the DRL algorithms are utilized to optimize the overall performance of the system. Our contributions mainly lie in five aspects, summarized as follows.

• We establish a diverse data collection scenario where the UAVs not only collect data from IoT devices (communication functionality) but also transmit radar pulses and receive echo signals (sensing functionality). To cancel the interference between communication and sensing, we adopt the time division scheme to perform communication and sensing functions sequentially. To cancel the mutual interference among IoT devices, we adopt the frequency division scheme to allocate different sub-carriers to different IoT devices.
• We define the successful transmission state to judge whether an IoT device successfully transmits its data to a UAV. Then, we define the data freshness indicator to characterize both the communication coverage and the fairness among IoT devices. Similarly, we define the successful sensing state and the detection freshness indicator, indicating the sensing coverage and the fairness among sensing targets, which further reflects the time-sensitivity and fairness of the system.
• We formulate an optimization for diverse data collection to minimize the summation of the data freshness indicator and detection freshness indicator by optimizing the UAV paths, the transmitted power of IoT devices and UAVs, and the transmission allocation indicators. Such an objective reflects the communication performance, the sensing performance, and their corresponding fairness. In addition, we reformulate the problem as a Markov decision process (MDP) to decide the movement distance of the UAVs, the power variations of the UAVs and IoT devices, and the allocation variations of tasks.
• To solve the formulated MDP, we propose three state-of-the-art DRL algorithms, namely twin delayed deep deterministic policy gradient (TD3), soft actor-critic (SAC), and proximal policy optimization (PPO). These are all mature algorithms designed to solve the problem with continuous action space and have been widely used in UAV trajectory planning. We provide the algorithm process to solve the established UAV-enabled diverse data collection optimization.
• Experiments are conducted in both single-UAV and multi-UAV scenarios, and the results verify the effectiveness of the algorithms. In addition, besides the joint mode, we also consider the other two data collection modes, i.e., data collection with only communication or sensing function. Moreover, we adopt the random strategy as the benchmark, and the results show that our methods outperform the benchmark. Furthermore, the numerical Pareto front between communication and sensing performance is obtained to reflect the inherent trade-offs.

The rest of this article is organized as follows. Section 2 establishes the system model, introduces the communication and sensing performance indicators, and formulates the UAV-enabled diverse data collection problem. Section 3 provides the solution to this problem, where the DRL algorithms are utilized to make decisions. Section 4 conducts the experiments to verify the algorithms’ effectiveness and discusses the results. Finally, Section 5 concludes this paper.

2. System Model and Problem Formulation

In this section, we establish the system model, introduce the communication and sensing performance indicators, and formulate the UAV-enabled diverse data collection problem.

2.1. System Model

Assume that there are M UAVs flying in a service region $\mathcal{R}$ to collect the information from I ground IoT devices and simultaneously provide sensing service to acquire information about J sensing targets, as depicted in Figure 2. Let $\mathit{U}=\left\{U_1,U_2,\dots ,U_M\right\}$ denote the UAV set, $\mathit{C}=\left\{C_1,C_2,\dots ,C_I\right\}$ denote the IoT device set where the IoT devices transmit information to UAVs, and $\mathit{S}=\left\{S_1,S_2,\dots ,S_J\right\}$ denote the sensing target set where the sensing targets are intended to be detected by the UAVs.

Assume that the UAVs are fully charged, so there is no need to consider landing to recharge during the entire flight. These UAVs all communicate with satellites, where the UAVs report their real-time positions to the satellites, and the satellites transmit the flight strategies to all the UAVs. Then, the satellite is aware of the positions of ground IoT devices and potential sensing target locations. The satellite sends the corresponding collected information from UAVs, IoT devices, and sensing targets to the ground base station control center. The ground control center runs algorithms and decides on the optimal trajectory for UAVs and the power transmission strategies for UAVs and IoT devices, and then relays these strategies back to the satellite. The illustration of the workflow diagram of the system is depicted in Figure 3.

We split the whole service time into N time slots, denoted as $t_1,t_2,\dots ,t_N$. In each time slot, the UAVs first fly to the preset locations, and then provide communication services (i.e., collect data from IoT devices), followed by sensing services (i.e., sensing the sensing targets). The illustration of the time scheduling of the UAV in such a scenario is depicted in Figure 4. Let $t_{n,\mathrm{C}}$ and $t_{n,\mathrm{S}},\forall n$ denote the communication time and sensing time in $t_n$. Let ${\mathit{L}}_{n,i}^{\mathrm{C}}={\left[x_{n,i}^{\mathrm{C}},y_{n,i}^{\mathrm{C}},z_{n,i}^{\mathrm{C}}\right]}^T$ denote the three-dimensional location of the IoT device $C_i,\forall i$ where ${\mathit{A}}^T$ represents the transposition of matrix $\mathit{A}$. Similarly, let ${\mathit{L}}_{n,j}^{\mathrm{S}}={\left[x_{n,j}^{\mathrm{S}},y_{n,j}^{\mathrm{S}},z_{n,j}^{\mathrm{S}}\right]}^T$ denote the three-dimensional location of the sensing target $S_j,\forall j$. Since the IoT devices and the sensing targets are often on the ground, we consider that $z_{n,i}^{\mathrm{C}}=0,\forall n,i$ and $z_{n,j}^{\mathrm{S}},\forall n,j$. Let ${\mathit{L}}_{n,m}={\left[x_{n,m},y_{n,m},z_{n,m}\right]}^T$ denote the three-dimensional location of the UAV $U_m$.

2.2. Communication Performance

In $t_{n,\mathrm{C}}$, the UAVs hover at the predesigned location, and there exists uplink communication between UAVs and IoT devices, where the IoT devices transmit data to UAVs. Not all IoT devices have the opportunity to transmit data to UAVs. The UAVs will decide which IoT devices’ data to collect, and will only collect data from those IoT devices that are within their communication range. Let $\mathit{A}=\left\{a_{n,m,i}|\forall n,m,i\right\}$ denote the transmission allocation set where the binary variable $a_{n,m,i}\in \left\{0,1\right\}$ represents whether the UAV $U_m$ collects data from IoT $C_i$ in $t_{n,\mathrm{C}}$. If $U_m$ collects data from $C_i$ in $t_{n,\mathrm{C}}$, we take $a_{n,m,i}=1$; otherwise, we take $a_{n,m,i}=0$.

We adopt a data freshness index to characterize whether the transmission between a UAV and an IoT device is successful and the freshness of the data. We consider that if the data from a specific IoT device have only been collected recently, then the previous data collected from that IoT device are considered fresh. Conversely, if the data from an IoT device have not been collected for a long time, then its previously collected data are considered not fresh. The data freshness index of $C_i$ in $t_n$ is given by

$$D{F}_{n,i}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{\displaystyle {\displaystyle \sum _{m=1}^M}}{a}_{n,m,i}=1\mathrm{and}{\mathbb{I}}_{n,i}^{\mathrm{C}}=1,\\ min\left(D{F}_{n,i}+1,A^{\max }\right),\hfill & \mathrm{Otherwise}.\end{array}\right.$$

(1)

Here, one IoT device only transmits its data to one UAV in a time slot, and thus ${\displaystyle {\displaystyle \sum _{m=1}^M}}{a}_{n,m,i}\le 1$ always holds. More precisely, ${\displaystyle {\displaystyle \sum _{m=1}^M}}{a}_{n,m,i}\in \left\{0,1\right\}$. If ${\displaystyle {\displaystyle \sum _{m=1}^M}}{a}_{n,m,i}=1$, this IoT device transmits data to a specific UAV, and vice versa. However, although the IoT device intends to transmit the data, the data may not be transmitted successfully due to the limited time delay. Hence, we define ${\mathbb{I}}_{n,i}^{\mathrm{C}}$ as the successful transmission state, and if ${\mathbb{I}}_{n,i}^{\mathrm{C}}=1$, this IoT device successfully transmits its data, and vice versa. On one hand, the data freshness index of an IoT device is reset to zero after its data are collected, indicating that a smaller index corresponds to fresher data. On the other hand, if an IoT device’s data have not been collected in the current time slot, its data freshness index is incremented by one to reflect and penalize this lack of collection. Establishing a maximum threshold $A^{\max }$ for the data freshness index ensures that even if certain IoT devices are continuously unable to transmit data, these IoT devices are not penalized indefinitely. This is because excessive penalties could cause UAVs to deviate from their appropriate planned routes abruptly to collect data from devices with high penalties, which is not only energy-inefficient but also detrimental to the overall system performance.

We next introduce how to characterize the successful state ${\mathbb{I}}_{n,i}^{\mathrm{C}}\in \left\{0,1\right\}$ for IoT device $C_i$ in $t_n$. If an IoT device can complete its transmission task within its maximum latency limit, then we consider the data transmission to be successful. Let $\mathsf{\Delta }{T}_{n,i}$ and $T^{\mathrm{Max}}$ denote the data transmission latency of $C_i$ in $t_{n,\mathrm{C}}$ and its limited maximum threshold, respectively. If $\mathsf{\Delta }{T}_{n,i}\le T^{\mathrm{Max}}$, we take ${\mathbb{I}}_{n,i}^{\mathrm{C}}=1$; otherwise, we take ${\mathbb{I}}_{n,i}^{\mathrm{C}}=0$. The data transmission latency is computed by

$$\mathsf{\Delta }{T}_{n,i}={\displaystyle \frac{W_{n,i}}{R_{n,i}^{\mathrm{C}}}},$$

(2)

where $W_{n,i}$ and $R_{n,i}^{\mathrm{C}}$ represent the total data amount generated by $C_i$ before $t_{n,\mathrm{C}}$ and the data transmission rate, respectively. We assume that in each time slot, IoT devices will delete their previous data and generate new data due to the constant changes in the dynamic environment. The transmission rate between $C_i$ and $U_m$ is calculated by

$$R_{n,i}^{\mathrm{C}}=B{log}_2\left(1+{\gamma }_{n,i}^{\mathrm{C}}\right),$$

(3)

where B denotes the bandwidth and ${\gamma }_{n,i}^{\mathrm{C}}$ represents the signal-interference-noise-ratio (SINR) of $C_i$ at $t_{n,\mathrm{C}}$.

SINR is the ratio of the power of a desired signal (the signal from $C_i$ to its served UAV $U_m$) to the sum of the power of interference signals (the signal from other IoT devices served by other UAVs to the served UAV $U_m$) and the power of background noise (additive white Gaussian noise), given by [39]

$${\gamma }_{n,i}^{\mathrm{C}}={\displaystyle \frac{p_{n,i}^{\mathrm{C}}{h}_{n,i,m}^{\mathrm{C}}}{{\displaystyle {\displaystyle \sum _{\widehat{i}=1}^I}}(1-a_{n,\widehat{i},m})p_{n,\widehat{i}}^{\mathrm{C}}{h}_{n,\widehat{i},m}^{\mathrm{C}}+{\sigma }^2}},$$

(4)

where ${\sigma }^2$ denotes the additive white Gaussian noise; $p_{n,i}^{\mathrm{C}}$ denotes the transmitted power of $C_i$ to the transmitted data in $t_{n,\mathrm{C}}$; and $h_{n,i,m}^{\mathrm{C}}$ denotes the channel between $C_i$ to $U_m$ in $t_{n,\mathrm{C}}$, typically expressed as

$$h_{n,i,m}^{\mathrm{C}}={\displaystyle \frac{g^{\mathrm{C}}{G}^{\mathrm{C}}{{\lambda }^{\mathrm{C}}}^2}{{\left(4\pi \right)}^2{∥{\mathit{L}}_{n,m}-{\mathit{L}}_{n,i}^{\mathrm{C}}∥}_2^2}}.$$

(5)

Here, $g^{\mathrm{C}}$ represents the transmitting antenna gain, which refers to the gain of the transmitting antenna in a certain direction relative to an ideal point source antenna; $G^{\mathrm{C}}$ represents the receiving antenna gain, which is the gain of the receiving antenna in its optimal reception direction relative to an ideal point source antenna; and ${\lambda }^{\mathrm{C}}$ represents the signal wavelength, which is the distance that a radio wave travels in one period. The term ${∥{\mathit{L}}_{n,m}-{\mathit{L}}_{n,i}^{\mathrm{C}}∥}_2$ denotes the distance between $C_i$ and $U_m$, expressed as an L2 norm term.

The interference term ${\displaystyle {\displaystyle \sum _{\widehat{i}=1}^I}}(1-a_{n,\widehat{i},m})p_{n,\widehat{i}}^{\mathrm{C}}{h}_{n,\widehat{i},m}^{\mathrm{C}}$ only comprises the interference signal from IoT devices served by other UAVs, and does not include that from IoT devices served by UAV $U_m$. This is because the orthogonal frequency-division multiple (OFDM) scheme allows the IoT devices to share the same UAV by utilizing different orthogonal sub-carriers to transmit their data. We consider that the interference between two IoT devices that are geographically close to each other is much stronger than the interference from IoT devices that are further apart. Therefore, we adopt the OFDM scheme only between devices with strong interference to achieve the maximum benefit for interference nullification. In addition, the term $1-a_{n,\widehat{i},m}$ eliminates the interference from the IoT devices served by $U_m$.

2.3. Sensing Performance

In $t_{n,\mathrm{S}}$, the UAVs also hover at the predesigned location to provide the sensing service for the potential sensing targets (usually the interest of specific spots). UAVs continuously transmit radar pulse signals to different sensing targets and receive the corresponding radar echoes to extract environmental information further. The echo data of radar pulse signals fundamentally differ from the environmental data collected by IoT devices, expanding environmental data’s diversity.

Also, not all sensing targets have the opportunity to be sensed by the UAVs. The UAVs decide which sensing target to be sensed, and the target allocation set is then denoted by $\mathit{B}=\left\{b_{n,m,j}|\forall n,m,j\right\}$, where the variable $b_{n,m,j}\in \left\{0,1\right\}$ represents whether the UAV $U_m$ senses the sensing target $S_j$. Here, we design a mechanism to determine the target allocation variable where the sensing targets within the UAV’s maximum sensing range $D^{\mathrm{S}}$ correspond to the variable value of 1; otherwise, the value is set to 0. In other words, if ${∥{\mathit{L}}_{n,m}-{\mathit{L}}_{n,j}^{\mathrm{S}}∥}_2\le D^{\mathrm{S}}$, we take $b_{n,m,j}=1$; otherwise, we take $b_{n,m,j}=0$.

Similar to communication performance, we adopt a detection freshness index to characterize whether the radar detection between a UAV and a sensing target is successful and the freshness of the detection activity, defined as

$$S{F}_{n,j}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{\displaystyle \sum _{m=1}^M}{b}_{n,m,j}=1\mathrm{and}{\mathbb{I}}_{n,j}^{\mathrm{S}}=1,\\ min\left(S{F}_{n,j}+1,A^{\max }\right),\hfill & \mathrm{Otherwise}.\end{array}\right.$$

(6)

It is worth noting that ${\displaystyle \sum _{m=1}^M}{b}_{n,m,j}\in \left\{0,1\right\}$, where a one indicates that $S_j$ is sensed by a UAV, and a zero indicates that it is not. Although a sensing target tends to be sensed by the UAV, such a sensing activity may not be successful. We further define the binary variable ${\mathbb{I}}_{n,j}^{\mathrm{S}}\in \left\{0,1\right\}$ to characterize the successful sensing state, and if ${\mathbb{I}}_{n,j}^{\mathrm{S}}=1$, the sensing target $S_j$ is successfully sensed by any UAV, and vice versa. If a sensing target is detected in $t_{n,\mathrm{S}}$, we consider that $S{F}_{n,j}$ is suddenly reduced to zero, which means fresher detection data. If a sensing target does not have the opportunity to be detected or is not successfully detected in $t_{n,\mathrm{S}}$, we consider that $S{F}_{n,j}$ is accumulated by one as an incremental penalty score.

Next, we introduce how to define a successful sensing state. The radar estimation information rate (REIR) is a commonly used metric in radar detection tasks [40]. It assesses and quantifies the radar system’s ability to sense static targets, which do not actively emit signals. In the field of radar, the detection and identification of targets is one of the core tasks, and this metric provides a standard for determining how much useful information a UAV can extract from the echo signals detected by its radar. Although static targets, such as communication devices, do not actively emit signals, their presence and characteristics (such as location, shape, material, etc.) affect the radio waves emitted by the radar and return information in the form of reflected waves. This metric can help us to optimize the design and performance of radar systems, by improving target detection accuracy and refining identification algorithms.

We assume that if the REIR of $S_j$ in $t_{n,\mathrm{S}}$ exceeds its required minimum threshold, denoted as $R_{n,j}^{\mathrm{S}}$ and $R^{\mathrm{Min}}$, respectively, this sensing activity is considered successful, and vice versa. In other words, if $R_{n,j}^{\mathrm{S}}\ge R^{\mathrm{Min}}$, we take ${\mathbb{I}}_{n,j}^{\mathrm{S}}=1$; otherwise, we take ${\mathbb{I}}_{n,j}^{\mathrm{S}}=0$. The REIR of $S_j$ in $t_{n,\mathrm{S}}$ is

$$R_{n,j}^{\mathrm{S}}=B{log}_2\left(1+{\displaystyle \frac{p_{n,m}^{\mathrm{S}}{h}_{n,m,j}^{\mathrm{S}}}{{\sigma }^2}}\right),$$

(7)

where $p_{n,m}^{\mathrm{S}}$ denotes the transmitted power of the served UAV $U_m$ and $h_{n,m,j}^{\mathrm{S}}$ denotes the round-trip channel between $U_m$ and $S_j$ (indicating not only the path from $U_m$ to $S_j$ but also the back path from $S_j$ to $U_m$), expressed as

$$h_{n,m,j}^{\mathrm{S}}={\displaystyle \frac{g^{\mathrm{S}}{G}^{\mathrm{S}}{\sigma }_{\mathrm{rcs}}{{\lambda }^{\mathrm{S}}}^2}{{\left(4\pi \right)}^2{∥{\mathit{L}}_{n,m}-{\mathit{L}}_{n,j}^{\mathrm{S}}∥}_2^4}}.$$

(8)

Here, $g^{\mathrm{S}}$, $G^{\mathrm{S}}$, and ${\lambda }^{\mathrm{S}}$ also denote the transmitting antenna gain, the receiving antenna gain, and the signal wavelength for sensing, respectively. The parameter ${\sigma }_{\mathrm{rcs}}$ represents the radar cross-section, which is used to describe the ability of a target to reflect radar waves—that is, the visibility of the target to radar detection. The larger the radar cross-section of a target, the stronger the signal it reflects back to the radar, and therefore, it is easier to be detected by radar. The term ${∥{\mathit{L}}_{n,m}-{\mathit{L}}_{n,j}^{\mathrm{S}}∥}_2$ denotes the distance between the UAV and the sensing target.

Notice that there exists no interference (communication interference and sensing interference) for $S_j$. This is due to the utilization of the TDMA scheme between communication and sensing, depicted in Figure 4, where the communication and sensing functions are executed using different sub-frames. Additionally, due to the varying distances between the UAV and each sensing target, as well as the different environments at each sensing target, even if the UAV sends radar pulse signals to all sensing targets simultaneously, the echo signals reflected from each sensing target will not arrive at the receiver on the UAV at the same time. Thereafter, the radar echo signals are free from interference from other echo signals.

2.4. Problem Formulation

Our goal is to simultaneously maximize the communication and sensing performance, i.e., to minimize the summation of the data freshness indicator for communication and the detection freshness indicator for sensing, by optimizing the UAV paths, the transmitted power of UAVs, the transmitted power of IoT devices, and the transmission allocation indicators in uplink ISAC system. The data freshness indicator for communication is defined as

$$DF={\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}D{F}_n={\displaystyle \frac{1}{NI}}{\displaystyle \sum _{n=1}^N}{\displaystyle \sum _{i=1}^I}D{F}_{n,i}.$$

(9)

The detection freshness indicator for sensing is defined as

$$SF={\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}S{F}_n={\displaystyle \frac{1}{NJ}}{\displaystyle \sum _{n=1}^N}{\displaystyle \sum _{j=1}^J}S{F}_{n,j}.$$

(10)

The data freshness indicator and the detection freshness indicator are both related to the UAV path, the transmitted power of UAVs and IoT devices, and the transmission allocation indicators. These two indicators are both dimensionless. 

Let $\mathit{L}=\left\{{\mathit{L}}_{n,m}|\forall n,m\right\}$ denote the UAV path set; ${\mathit{P}}^{\mathrm{C}}=\left\{p_{n,i}^{\mathrm{C}}|\forall n,i\right\}$ denote the IoT transmitted power set; and ${\mathit{P}}^{\mathrm{S}}=\left\{p_{n,m}^{\mathrm{S}}|\forall n,m\right\}$ denote the UAV transmitted power set. Then, the UAV-enabled diverse data collection problem is formulated as

$$\begin{array}{cc}\hfill \underset{\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}}{min}& DF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)+SF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.C1:& {\mathit{L}}_{n,m}\in \mathcal{R},\forall n,m,\hfill \\ \hfill C2:& \left|x_{n,m}-x_{n-1,m}\right|\le X^{\mathrm{M}},n\in \left\{2,\cdots ,N\right\},\forall m,\hfill \\ \hfill C3:& \left|y_{n,m}-y_{n-1,m}\right|\le Y^{\mathrm{M}},n\in \left\{2,\cdots ,N\right\},\forall m,\hfill \\ \hfill C4:& \left|z_{n,m}-z_{n-1,m}\right|\le Z^{\mathrm{M}},n\in \left\{2,\cdots ,N\right\},\forall m,\hfill \\ \hfill C5:& a_{n,m,i}\in \left\{0,1\right\},\forall n,m,i,\hfill \\ \hfill C6:& {\displaystyle \sum _{m=1}^M}{a}_{n,m,i}\le 1,\forall n,i,\hfill \\ \hfill C7:& p_{\mathrm{Min}}^{\mathrm{S}}\le p_{n,m}^{\mathrm{S}}\le p_{\mathrm{Max}}^{\mathrm{S}},\forall n,m,\hfill \\ \hfill C8:& p_{\mathrm{Min}}^{\mathrm{C}}\le p_{n,i}^{\mathrm{C}}\le p_{\mathrm{Max}}^{\mathrm{C}},\forall n,i,\hfill \end{array}$$

(11b)

where C1 is the boundary constraint for UAVs; C2–C4 are the maximum flight range constraints of UAVs for each pair of consecutive time slots; C5–C6 is the transmission allocation constraint; C7 is the UAV transmitted power constraint; and C8 is the IoT transmitted power constraint. The parameters $X^{\mathrm{M}}$, $Y^{\mathrm{M}}$, and $Z^{\mathrm{M}}$ denote the maximum flight range in the x-axis, y-axis, and z-axis, respectively; the parameters $p_{\mathrm{Min}}^{\mathrm{S}}$ and $p_{\mathrm{Max}}^{\mathrm{S}}$ denote the minimum and maximum transmitted power of the UAV, respectively; and the parameters $p_{\mathrm{Min}}^{\mathrm{C}}$ and $p_{\mathrm{Max}}^{\mathrm{C}}$ denote the minimum and maximum transmitted power of IoT devices, respectively.

3. Deep Reinforcement Learning for UAV-Enabled Diverse Data Collection

In this section, we first reformulate the original UAV-enabled diverse data collection problem into an MDP. Then, we introduce three state-of-the-art deep reinforcement learning algorithms, namely TD3, SAC, and PPO, to solve such an MDP.

3.1. Markov Decision Process

To execute the DRL, we first reformulate the problem (11) into an MDP where some key components are included, namely the agent, environment, state, action, and reward. The MDP is the mathematical framework for the modeling of sequential decision-making problems by decision-makers in uncertain environments. We introduce these key components in the following context.

Agent: The agent is the ground base station control center that determines how to allocate resources for all UAVs and IoT devices.

Environment: The environment is the UAV-enabled diverse data collection scenario.

State: In $t_n$, the state set ${\mathrm{S}}_n$ consists of the following variables: (1) the locations of all UAVs, IoT devices, and sensing targets; (2) the data and target allocation indicators for all IoT devices and sensing targets; (3) the total data amount generated by all IoT devices; (4) the transmitted power of all UAVs and IoT devices; (5) the data transmission latency of all IoT devices; (6) the successful transmission state of all IoT devices and the successful sensing state of all sensing targets; and (7) the transmission rate of all IoT devices and the REIR of all sensing targets, expressed as

$${\mathrm{S}}_n=\left\{{\mathit{L}}_{n,i}^{\mathrm{C}},{\mathit{L}}_{n,j}^{\mathrm{S}},{\mathit{L}}_{n,m},a_{n,m,i},b_{n,m,j},W_{n,i},p_{n,i}^{\mathrm{C}},p_{n,m}^{\mathrm{S}},\mathsf{\Delta }{T}_{n,i},{\mathbb{I}}_{n,i}^{\mathrm{C}},{\mathbb{I}}_{n,j}^{\mathrm{S}},D{F}_{n,i},S{F}_{n,j}|\forall i,j,m\right\}.$$

(12)

Action: In $t_n$, the action set ${\mathrm{A}}_n$ consists of the flight range of all UAVs in the x-axis, y-axis, and z-axis; the transmitted power of all UAVs and IoT devices; and all transmission allocation indicators, given by

$${\mathrm{A}}_n=\left\{\mathsf{\Delta }{x}_{n,m},\mathsf{\Delta }{y}_{n,m},\mathsf{\Delta }{z}_{n,m},p_{n,i}^{\mathrm{C}},p_{n,m}^{\mathrm{S}},a_{n,m,i}|\forall m,i\right\},$$

(13)

where $\mathsf{\Delta }{x}_{n,m}=x_{n,m}-x_{n-1,m}$, $\mathsf{\Delta }{y}_{n,m}=y_{n,m}-y_{n-1,m}$, and $\mathsf{\Delta }{z}_{n,m}=z_{n,m}-z_{n-1,m}$ denote the flight range of $U_m$ in the x-axis, y-axis, and z-axis, respectively. These three types of action variables correspond to three optimization variables.

Reward: In $t_n$, we define the reward function as

$${\mathrm{R}}_n=-k\left(D{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)+S{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)\right)+b,$$

(14)

where k and b are the coefficients to control the convergence speed and the stability of the DRL. It is worth noting that we adopt the penalty method to ensure that the UAVs tend to satisfy their boundary constraints. If the boundary constraints are not satisfied, an extremely large penalty number is subtracted from the reward function to avoid this circumstance.

We adopt DRL to solve the established MDP. DRL is a machine learning method that learns the optimal strategies through environmental interaction. In DRL, an agent receives states and rewards by performing actions, aiming to learn a policy that maximizes cumulative long-term rewards.

Within the MDP, each state transition and reward depends only on the current state and the action taken, known as the Markov property. This property simplifies the modeling of the decision-making process, since the agent does not need to consider the entire history, only the current state. This is one of the key reasons why DRL can effectively solve MDPs.

The core of solving MDPs with DRL lies in learning a policy that can tell the agent what action to take in a given state. To learn such a policy, various DRL algorithms suitable for solving different scenario problems have been proposed. These algorithms typically involve estimating state-value functions or action-value functions, which represent the expected return of taking a specific action in a specific state. By optimizing these value functions, agents can learn which actions will bring greater long-term returns in any given state. The illustration of the DRL framework for solving MDP is depicted in Figure 5.

3.2. Twin Delayed Deep Deterministic Policy Gradient

TD3 is a reinforcement learning method for continuous action spaces. It is an improved version of the deep deterministic policy gradient (DDPG) algorithm. DDPG combines the ideas of deep Q-Network (DQN) with the actor-critic framework. However, in practice, DDPG tends to overestimate the value function, leading to unstable training and decreased performance. TD3 addresses these issues with several key technical innovations, enhancing the stability and performance of the algorithm.

The TD3 algorithm has become one of the most advanced DRL algorithms due to the following advantages. First, TD3 uses two value functions (also known as Q functions) and takes the minimum of the two to reduce the overestimation of value estimates. This is inspired by Double Q-Learning, which has been shown to reduce overestimation in algorithms for discrete action spaces. In addition, the policy update frequency is less than the value function update frequency. This means that for every update of the policy network, the value network is updated multiple times. This delayed update approach allows the value estimates to stabilize, leading to a more stable learning process. Furthermore, TD3 adds noise to the target policy calculation to smooth out the target policy, avoiding excessive sensitivity to noise, which helps with the stability of the learning process.

The algorithm process step of TD3 is provided as follows.

• Initialization: Randomly initialize two value networks (critics, $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ with parameters ${\varphi }_1$ and ${\varphi }_2$) and a policy network (actor, ${\pi }_{\theta }$ with parameter $\theta$). Also, initialize the corresponding target networks ($Q_{{\varphi }_{\mathrm{target},1}}$, $Q_{{\varphi }_{\mathrm{target},1}}$, and ${\pi }_{{\theta }_{\mathrm{target}}}$ with parameters ${\varphi }_{\mathrm{target},1}$, ${\varphi }_{\mathrm{target},1}$, and ${\theta }_{\mathrm{target}}$) with the same parameters as the original networks.
• Data Collection: Execute the current policy ${\pi }_{\theta }$, take actions in the environment, collect transitions (state $\mathrm{S}$, action $\mathrm{A}$, reward $\mathrm{R}$, next state ${\mathrm{S}}^{\prime }$), and store them in D.
• Sample Extraction: Randomly sample a batch of experiences $\left(\mathrm{S},\mathrm{A},\mathrm{R},{\mathrm{S}}^{\prime }\right)$ from D.
• Value Network Update: Calculate the target value using the minimum Q value and the action produced by the target policy network, and update both value networks using the mean squared error, expressed as

  $$J_Q\left({\varphi }_i\right)={\mathbb{E}}_{\left(\mathrm{S},\mathrm{A},\mathrm{R},{\mathrm{S}}^{\prime }\right)\sim D}\left[{\left(Q_{{\varphi }_i}(\mathrm{S},\mathrm{A})-\left(\mathrm{R}+\gamma \underset{i=1,2}{min}{Q}_{{\varphi }_{\mathrm{target},\mathrm{j}}}\left({\mathrm{S}}^{\prime },{\pi }_{{\theta }_{\mathrm{target}}}\left({\mathrm{S}}^{\prime }\right)+ϵ\right)\right)\right)}^2\right],$$

  (15)

  where $i\in \left\{1,2\right\}$; ${\mathrm{A}}^{\prime }$ is the action sampled from state ${\mathrm{S}}^{\prime }$ according to the current policy; $\gamma$ is the discount factor; and $ϵ\sim \mathcal{N}\left(0,{\sigma }^2\right)$ is the noise added to the target policy action to ensure exploration and smooth out the value estimates.
• Policy Network Update: The policy network is updated using the policy gradient, which is computed using one of the value networks (typically $Q_{{\varphi }_1}$):

  $${\nabla }_{\theta }{J}_{\pi }\left(\theta \right)={\mathbb{E}}_{\mathrm{S}\sim D}\left[{\nabla }_{\mathrm{A}}{Q}_{{\varphi }_1}\left(\mathrm{S},\mathrm{A}\right){|}_{\mathrm{A}={\pi }_{\theta }\left(\mathrm{S}\right)}{\nabla }_{\theta }{\pi }_{\theta }\left(\mathrm{S}\right)\right].$$

  (16)
• Target Network Soft Update: The parameters of the target networks are slowly adjusted towards those of the main networks, with a small update magnitude, typically using a small learning rate, given by

  $$\begin{array}{cc}\hfill {\theta }_{\mathrm{target}}\leftarrow & \tau \theta +(1-\tau ){\theta }_{\mathrm{target}},\hfill \end{array}$$

  (17)

  $$\begin{array}{cc}\hfill {\varphi }_{\mathrm{target},i}\leftarrow & \tau {\varphi }_i+(1-\tau ){\varphi }_{\mathrm{target},i},i\in \left\{1,2\right\},\hfill \end{array}$$

  (18)

  where $\tau$ is the soft update coefficient.
• Repeat steps 2–6 until the training conditions are satisfied.

Let $\mathit{V}=\left\{W_{n,i},D^{\mathrm{S}},B,G^{\mathrm{C}},g^{\mathrm{C}},{\lambda }^{\mathrm{C}},g^{\mathrm{S}},G^{\mathrm{S}},{\lambda }^{\mathrm{S}},{\sigma }_{\mathrm{rcs}},{\sigma }^2,X^{\mathrm{M}},Y^{\mathrm{M}},Z^{\mathrm{M}},p_{\mathrm{Min}}^{\mathrm{S}},p_{\mathrm{Max}}^{\mathrm{S}},p_{\mathrm{Min}}^{\mathrm{C}},p_{\mathrm{Max}}^{\mathrm{C}}\right\}$ denote the variable set related to the system parameters. The pseudocode of TD3 for UAV-enabled diverse data collection optimization is provided in Algorithm 1.

Algorithm 1 Twin delayed deep deterministic policy gradient for UAV-enabled diverse data collection optimization.

1: Initialize $\mathit{V}$, $Q_{{\varphi }_1}$, $Q_{{\varphi }_2}$, ${\pi }_{\theta }$, $Q_{{\varphi }_{\mathrm{target},1}}$, $Q_{{\varphi }_{\mathrm{target},1}}$, ${\pi }_{{\theta }_{\mathrm{target}}}$, $\gamma$, $\tau$, D. 2: ${\mathrm{A}}_n\leftarrow \left\{\mathsf{\Delta }{x}_{n,m},\mathsf{\Delta }{y}_{n,m},\mathsf{\Delta }{z}_{n,m},p_{n,i}^{\mathrm{C}},p_{n,m}^{\mathrm{S}},a_{n,m,i}|\forall m,i\right\}$. 3: ${\mathrm{R}}_n\leftarrow -k\left(D{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)+S{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)\right)+b$. 4: Collect transitions $\left(\mathrm{S},\mathrm{A},\mathrm{R},{\mathrm{S}}^{\prime }\right)$ and store them in D. 5: for Each update episode l do 6:    Sample a batch of $\left({\mathrm{S}}_l,{\mathrm{A}}_l,{\mathrm{R}}_l,{\mathrm{S}}_{l+1}\right)$. 7:    Update $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ according to Equation (15). 8:    Update ${\pi }_{\theta }$ according to Equation (16). 9:    Update target policy ${\pi }_{{\theta }_{\mathrm{target}}}$ according to Equation (17). 10:    Update target value $Q_{{\varphi }_{\mathrm{target},1}}$ and $Q_{{\varphi }_{\mathrm{target},2}}$ according to Equation (18). 11: end for

3.3. Soft Actor-Critic

SAC is a deep reinforcement learning algorithm based on the actor-critic architecture that incorporates the maximum entropy reinforcement learning framework to improve the algorithm’s sampling efficiency, robustness, and exploration capabilities. The goal of maximum entropy reinforcement learning is to maximize cumulative rewards and increase the entropy of the policy, encouraging the policy to take a diverse set of actions, thereby improving exploration efficiency.

SAC has several advantages over existing DRL algorithms. First, by using off-policy data and a replay buffer, SAC can use experiences more efficiently. Second, SAC employs a soft update mechanism for the value function, which can reduce the variance during training and improve learning stability. Third, the maximum entropy framework naturally encourages the policy to explore, which is particularly beneficial in complex or sparse reward environments. Fourth, as the policy is stochastic, SAC tends to be more robust when facing changes in the environment.

The algorithm process step of SAC is provided as follows.

• Initialization: Initialize the policy network (actor, ${\pi }_{\theta }$ with parameter $\theta$), two value networks (critics, $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ with parameters ${\varphi }_1$ and ${\varphi }_2$), their target networks ($Q_{{\varphi }_{\mathrm{target},1}}$ and $Q_{{\varphi }_{\mathrm{target},2}}$ with parameters $Q_{{\varphi }_{\mathrm{target},1}}$ and $Q_{{\varphi }_{\mathrm{target},2}}$), and the replay buffer D.
• Data Collection and Sample Extraction
• Value Network Update: Update the value networks using the sampled experiences, usually by minimizing the mean squared error between the predicted and target values, given by

  $$\begin{array}{cc}\hfill J_Q\left({\varphi }_i\right)={\mathbb{E}}_{\left(\mathrm{S},\mathrm{A},\mathrm{R},{\mathrm{S}}^{\prime }\right)\sim D}\left[{\displaystyle \frac{1}{2}}\right(Q_{{\varphi }_i}\left(\mathrm{S},\mathrm{A}\right)& -(\mathrm{R}+\gamma (1-d)\underset{j=1,2}{min}{Q}_{{\varphi }_{\mathrm{target},j}}({\mathrm{S}}^{\prime },{\mathrm{A}}^{\prime })\hfill \\ \hfill & -\alpha log{\pi }_{\theta }\left({\mathrm{A}}^{\prime }\right|{\mathrm{S}}^{\prime })){)}^2],i\in \left\{1,2\right\},\hfill \end{array}$$

  (19)

  where d represents the termination flag and $\alpha$ is the entropy regularization coefficient. Here, the entropy regularization coefficient is adaptively adjusted.
• Soft Value Update: Update the soft value function using the outputs of the value networks and the rewards.
• Policy Network Update: Update the policy network to maximize the soft value function while also increasing the entropy of the policy, given by

  $$J_{\pi }\left(\theta \right)={\mathbb{E}}_{\mathrm{S}\sim D,\mathrm{A}\sim {\pi }_{\theta }}\left[\alpha log\left({\pi }_{\theta }\left(\mathrm{A}\right|\mathrm{S})\right)-Q_{\varphi }(\mathrm{S},\mathrm{A})\right].$$

  (20)
  Here, $Q_{\varphi }$ is the smaller of the two value networks, $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$.
• Target Network Soft Update: Periodically soft update the weights of the value networks to the target networks to stabilize the training process, given by

  $${\varphi }_{\mathrm{target},i}\leftarrow \tau {\varphi }_i+(1-\tau ){\varphi }_{\mathrm{target},i},i\in \left\{1,2\right\}.$$

  (21)
• Repeat steps 2–6 until the policy performance meets expectations or the training reaches a predefined number of iterations.

The pseudocode of SAC for UAV-enabled diverse data collection optimization is provided in Algorithm 2.

Algorithm 2 Soft actor-critic for UAV-enabled diverse data collection optimization.

1: Initialize $\mathit{V}$, ${\pi }_{\theta }$, $Q_{{\varphi }_1}$, $Q_{{\varphi }_2}$, $Q_{{\varphi }_{\mathrm{target},1}}$, $Q_{{\varphi }_{\mathrm{target},2}}$, $\alpha$, $\gamma$, $\tau$, d, D. 2: ${\mathrm{A}}_n\leftarrow \left\{\mathsf{\Delta }{x}_{n,m},\mathsf{\Delta }{y}_{n,m},\mathsf{\Delta }{z}_{n,m},p_{n,i}^{\mathrm{C}},p_{n,m}^{\mathrm{S}},a_{n,m,i}|\forall m,i\right\}$. 3: ${\mathrm{R}}_n\leftarrow -k\left(D{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)+S{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)\right)+b$. 4: Collect transitions $\left(\mathrm{S},\mathrm{A},\mathrm{R},{\mathrm{S}}^{\prime }\right)$ and store them in D. 5: for Each update episode l do 6:    Sample a batch of $\left({\mathrm{S}}_l,{\mathrm{A}}_l,{\mathrm{R}}_l,{\mathrm{S}}_{l+1}\right)$. 7:    Update $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ according to Equation (19). 8:    Update soft value function. 9:    Adaptively adjust entropy regularization coefficient $\alpha$. 10:    Update ${\pi }_{\theta }$ according to Equation (20). 11:    Softly update target value $Q_{{\varphi }_{\mathrm{target},1}}$ and $Q_{{\varphi }_{\mathrm{target},2}}$ according to Equation (21). 12: end for

3.4. Proximal Policy Optimization

PPO is an algorithm widely used in reinforcement learning that maintains training stability and prevents performance collapse due to excessive changes in the policy during updates. Policy gradient methods directly adjust policy parameters by optimizing an objective function to obtain greater cumulative rewards.

Compared to other state-of-the-art DRL algorithms, the characteristics and advantages of PPO are primarily as follows. First, PPO is simpler than its earlier version trust region policy optimization (TRPO), as it avoids complex second-order computations, i.e., the Hessian matrix, and instead relies on first-order optimization methods. Second, PPO is more sample-efficient than some other on-policy algorithms because it can reuse sampled data multiple times for each update. Third, PPO ensures training stability and robustness by limiting the magnitude of policy updates, preventing large policy shifts that could lead to performance collapse. Fourth, the PPO algorithm encourages exploration while learning by limiting the size of policy updates.

The core idea of the PPO algorithm is to introduce a "proximity” constraint during policy updates to ensure that the new policy does not deviate too far from the old policy. This is achieved by clipping the probability ratio, which describes the ratio of the probabilities of taking the same action under the new policy to that under the old policy.

The steps of the PPO algorithm are roughly as follows:

• Initialization: Initialize the policy network (actor, ${\pi }_{\theta }$ with parameter $\theta$) and the value network (critic, $Q_{\varphi }$ with parameter $\varphi$).
• Data Collection
• Advantage Estimation: Estimate the advantage function at each time step using the collected data, which typically involves calculating the discounted sum of rewards and a baseline to reduce variance. The temporal difference (TD) error is a method to evaluate the accuracy of the value function. At a given time step l, the TD error is given by

  $${\delta }_l={\mathrm{R}}_l+\gamma V_{\varphi }\left({\mathrm{S}}_{l+1}\right)-V_{\varphi }\left({\mathrm{S}}_l\right),$$

  (22)

  where $V_{\varphi }$ is the value function. Generalized advantage estimation (GAE) is a technique to reduce the variance of policy gradient estimates while maintaining a balance with bias, given by

  $$GA{E}_l(\gamma ,\lambda )={\displaystyle \sum _{k=0}^{\infty }}{\left(\gamma \lambda \right)}^k{\delta }_{l+k}$$

  (23)

  where $\lambda$ is a parameter between 0 and 1 used to balance bias and variance. When $\lambda =0$, GAE reduces to the single-step TD error, and when $\lambda =1$, it approaches the Monte Carlo method.
• Policy Optimization: Optimize the policy by maximizing a specific objective function that includes a clipped ratio of probabilities and the estimated advantage function, along with an entropy bonus to encourage exploration, given by

  $$\begin{array}{cc}\hfill L^{\mathrm{CLIP}}\left(\theta \right)={\mathbb{E}}_l[min& ({\displaystyle \frac{{\pi }_{\theta }\left({\mathrm{A}}_l|{\mathrm{S}}_l\right)}{{\pi }_{{\theta }_{\mathrm{previous}}}\left({\mathrm{A}}_l|{\mathrm{S}}_l\right)}}GA{E}_l,\mathrm{clip}({\displaystyle \frac{{\pi }_{\theta }\left({\mathrm{A}}_l|{\mathrm{S}}_l\right)}{{\pi }_{{\theta }_{\mathrm{previous}}}\left({\mathrm{A}}_l|{\mathrm{S}}_l\right)}},\hfill \\ \hfill & 1-\xi ,1+\xi \left)GA{E}_l\right)],\hfill \end{array}$$

  (24)

  where $\xi$ is the clipping parameter.
• Probability Ratio Clipping: Clip the probability ratio if it goes beyond a predefined interval to limit the magnitude of policy updates.
• Value Function Update: Update the value function using the same trajectory data, usually by minimizing the mean squared error between the value predictions and the actual returns, given by

  $$L^{\mathrm{VF}}\left(\varphi \right)={\left(V_{\varphi }\left({\mathrm{S}}_l\right)-{\displaystyle \sum _{k=0}^{\infty }}{\gamma }^k{\displaystyle \frac{{\pi }_{\theta }\left({\mathrm{A}}_{l+k}|{\mathrm{S}}_{l+k}\right)}{{\pi }_{{\theta }_{\mathrm{previous}}}\left({\mathrm{A}}_{l+k}|{\mathrm{S}}_{l+k}\right)}}\right)}^2.$$

  (25)
• Repeat steps 2–6 using the updated policy and value function until a termination condition is met, such as achieving a predetermined performance standard or completing a certain number of iterations.

The pseudocode of PPO for UAV-enabled diverse data collection optimization is provided in Algorithm 3.

Algorithm 3 Proximal policy optimization for UAV-enabled diverse data collection optimization.

1: Initialize $\mathit{V}$, $Q_{\varphi }$, ${\pi }_{\theta }$, $\gamma$, $\lambda$, $ϵ$, $\xi$D. 2: ${\mathrm{A}}_n\leftarrow \left\{\mathsf{\Delta }{x}_{n,m},\mathsf{\Delta }{y}_{n,m},\mathsf{\Delta }{z}_{n,m},p_{n,i}^{\mathrm{C}},p_{n,m}^{\mathrm{S}},a_{n,m,i}|\forall m,i\right\}$. 3: ${\mathrm{R}}_n\leftarrow -k\left(D{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)+S{F}_n\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)\right)+b$. 4: Collect transitions $\left(\mathrm{S},\mathrm{A},\mathrm{R},{\mathrm{S}}^{\prime }\right)$ and store them in D. 5: for Each update episode do 6:    for each time step L do 7:      Calculate temporal difference error $\delta l$ according to Equation (22). 8:      Calculate advantage estimate $GA{E}_l(\gamma ,\lambda )$ using GAE according to Equation (23). 9:    end for 10:    for each epoch do 11:      for each minibatch do 12:         Update policy parameter $\theta$ according to clipped objective function $L^{\mathrm{CLIP}}\left(\theta \right)$ in Equation (24). 13:      end for 14:    end for 15:    for each epoch do 16:      for each minibatch do 17:         Update value function parameter $\varphi$ according to value function loss $L^{\mathrm{VF}}\left(\varphi \right)$ in Equation (25). 18:      end for 19:    end for 20:    ${\theta }_{\mathrm{previous}}\leftarrow \theta$. 21: end for

4. Experimental Results and Discussions

4.1. Algorithm Effectiveness in a Single-UAV Scenario

In this subsection, we testify the algorithms’ effectiveness in a synthetic single-UAV scenario where $M=1$ UAV collects the data from $I=20$ IoT devices and simultaneously senses $J=4$ sensing targets within $N=100$ time nodes in region $\mathcal{R}\in [-500\mathrm{m},500\mathrm{m}]\times [-500\mathrm{m},500\mathrm{m}]\times [100\mathrm{m},200\mathrm{m}]$. The locations of IoT devices and sensing targets are randomly generated on the ground. The initial location of the UAV is randomly generated in the three-dimensional region, as depicted in Figure 6. In Figure 6, the blue solid diamonds represent the IoT devices, the green solid squares represent the sensing targets, and the red solid pentagons represent the UAV.

The total data amount provided by each IoT device in each time slot is randomly generated at between 20 kb and 30 kb. The maximum sensing range $D^{\mathrm{S}}$ is set to 180 m. The bandwidth B is set to 1 MHz. The variance of Gaussian noise ${\sigma }^2$ is sampled in $[{10}^{-16},{10}^{-15}]$. The maximum flight range in the x-axis, y-axis, and z-axis is set to $X^{\mathrm{M}}=50\mathrm{m}$, $Y^{\mathrm{M}}=50\mathrm{m}$, and $Z^{\mathrm{M}}=5\mathrm{m}$, respectively. The minimum and maximum transmitted power of UAVs are set to $p_{\mathrm{Min}}^{\mathrm{S}}=0.1\mathrm{W}$ and $p_{\mathrm{Max}}^{\mathrm{S}}=1\mathrm{W}$, respectively. The minimum and maximum transmitted power of IoT devices are set to $p_{\mathrm{Min}}^{\mathrm{C}}=0.01\mathrm{W}$ and $p_{\mathrm{Min}}^{\mathrm{C}}=0.1\mathrm{W}$, respectively. The minimum threshold of REIR is set to $R^{\mathrm{Min}}={10}^5$ bit/s. The settings of the above parameters are all based on a reasonable simulation of practical scenarios, listed in

Table 1. Settings of parameters.

Variable	Value	Variable	Value
$g^{\mathrm{C}}$	5 dB	$G^{\mathrm{C}}$	15 dB
${\lambda }^{\mathrm{C}}$	0.1 m	$g^{\mathrm{S}}$	15 dB
$G^{\mathrm{S}}$	15 dB	${\lambda }^{\mathrm{S}}$	0.35 m
${\sigma }_{\mathrm{rcs}}$	1 ${\mathrm{m}}^2$	k	$1/50$
b	1.3	$\gamma$	0.98
$\tau$	0.01	$\xi$	0.2

.

To verify the effectiveness, the algorithm convergence of the proposed TD3, SAC, and PPO algorithms is depicted in Figure 7. Figure 7b,c show the data freshness indicator for communication and the detection freshness indicator for sensing, respectively. It is worth noting that the summation of these two indicators is exactly the optimization objective, also referred to as the fitness function in the optimization. The illustration of the fitness function value versus episode is depicted in Figure 7a. It can be seen from Figure 7a that as the number of episodes increases, the fitness function value decreases, which demonstrates the effectiveness of the proposed algorithm. Also, we can see from Figure 7b,c that as the number of episodes increases, both indicators decrease for TD3, SAC, and PPO, meaning that communication and sensing performance improve. It is worth noting that these two indicators are both dimensionless. This indicates that the proposed three DRL algorithms are all valid in UAV-enabled diverse data collection optimization. In addition, the PPO algorithm exhibits better optimization performance in the initial stages and tends to converge faster. This is due to the high universality of the PPO algorithm. Although TD3 and SAC converge slower in the early stages, their final convergence outcomes are superior to those of the PPO algorithm in the later stages. The PPO algorithm performs slightly worse than TD3 and SAC when dealing with high-dimensional continuous action spaces. Furthermore, the SAC performs better than TD3 regarding the convergence value. SAC has better exploration capabilities and robustness with its entropy regularization, making it particularly suitable for complex continuous action space problems. However, SAC may have higher computational overheasd than TD3 and PPO due to the maintenance cost of an additional entropy term.

Then, we depict the UAV paths obtained by the SAC algorithm to further testify this algorithm’s effectiveness. The patterns of the UAV paths optimized by the three algorithms are almost identical. However, SAC exhibits the best convergence results, and thus we show the optimal UAV path obtained under this optimum. Two-dimensional and three-dimensional UAV paths are depicted in Figure 8a and Figure 8b, respectively. First, the UAV tends to fly over the entire area to provide communication services for all IoT devices and sensing services for all sensing targets. This demonstrates the effectiveness of the algorithm in terms of communication and sensing capabilities. Then, the IoT device in the bottom right corner has no opportunity to communicate during the whole service period. This is because letting the UAV specifically fly to serve that IoT device would reduce the support for other devices located in the dense region. When considering the balance between supporting frequent data collection in denser areas and collecting data from single IoT devices in remote, sparse regions, the UAV’s strategy favors the former. However, if the penalty for the IoT device not communicating gradually increases, the optimization also tends to avoid being penalized by such circumstances. Furthermore, we can observe that the UAV flies in circles over the entire area. This is to provide services to IoT devices that have not had their data collected for a long time, reflecting fairness in communication and sensing.

4.2. Algorithm Effectiveness in a Multi-UAV Scenario

We further verify the effectiveness of the algorithm in a multi-UAV scenario where $M=3$ UAVs are considered to provide diverse data collection functions in a larger-range area $\mathcal{R}\in [-1000\mathrm{m},1000\mathrm{m}]\times [-1000\mathrm{m},1000\mathrm{m}]\times [100\mathrm{m},200\mathrm{m}]$. In practical large-scale scenarios, the number of IoT devices and sensing targets does not affect the effectiveness of the algorithm, but affects its convergence speed. To characterize the effectiveness of the algorithm, we consider a smaller number of IoT devices and sensing targets to ensure faster convergence, where $I=20$ IoT devices and $J=4$ sensing targets are considered. The maximum sensing range $D^{\mathrm{S}}$ is set to 250 m for the larger area. The other settings of parameters are consistent with those in the single-UAV scenario. The locations of IoT devices are randomly generated in the area, but we consider that there are no IoT devices in the top right corner of the area. Sensing targets are distributed in the top right corner of the area. The initial positions of the three UAVs are also randomly generated. The illustration of the initial locations of the UAVs, IoT devices, and sensing targets in the three-UAV scenario is depicted in Figure 9, where the blue solid diamonds, the green solid squares, and the red solid pentagons represent the IoT devices, the sensing targets, and the UAVs, respectively.

To verify the effectiveness in the multi-UAV scenario, the algorithm convergence of the proposed TD3, SAC, and PPO algorithms is depicted in Figure 10. Figure 10a–c show the fitness function value, the data freshness indicator for communication, and the detection freshness indicator for sensing, respectively. It can be seen from Figure 10 that as the number of episodes increases, the fitness function value of the optimization gradually decreases, and the communication and sensing indicators also decrease, (indicating improved performance in communication and sensing), which demonstrates the effectiveness of the algorithm. In addition, the proposed SAC and TD3 algorithms outperform the proposed PPO algorithm in terms of the final result quality. However, the proposed PPO has higher stability.

To verify the effectiveness of the proposed algorithms, we consider the random strategy to compare the performance. A random strategy, serving as a benchmark, provides a simple contrast that can be used to verify and demonstrate the performance improvement of deep reinforcement learning algorithms over scenarios with no intelligence. It makes decisions by randomly selecting actions without any prior knowledge or intelligent decision-making algorithms. Specifically, the optimization variables, including the UAV path, the transmitted power of UAVs and IoT devices, and transmission allocation indicators, are randomly selected in their allowable range. For example, the moving distances of the UAVs are selected between the minimum and the maximum flight range in the x-axis, y-axis, and z-axis. The same strategy is also suitable for the other two variables. In Figure 10, the random strategy performs worse than the proposed DRL algorithms in terms of convergence value and efficiency.

The two-dimensional and three-dimensional illustrations of the UAV path obtained using the SAC algorithm in the multi-UAV scenario are depicted in Figure 11a and Figure 11b, respectively. It can be concluded from Figure 11 that the three UAVs each fly in circles within their respective areas to provide dual functions for the IoT devices and sensing targets. Their trajectories have little overlap, and they manage almost entirely different regions, which allows the UAVs to collect IoT data and perform sensing tasks in a timely manner. In the same time slot, the three UAVs tend to be dispersed to provide wider coverage. In addition, the UAVs not only collect data over the IoT devices but also perform sensing over the sensing targets.

4.3. Individual Communication and Sensing Functions for Data Collection

We consider two methods of information collection: one involves the collection of data from IoT devices via uplink communication, and the other involves receiving echo data through the transmission of radar pulses. Each of these collection methods corresponds to a different performance metric (i.e., the data freshness indicator and the detection freshness indicator), both of which are part of the optimization objectives. Then, we may consider three modes:

• Data collection with joint communication and sensing functions: The objective function is defined as

  $$O=DF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)+SF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right),$$

  (26)

  where $DF$ represents the data freshness indicator, measuring communication performance, and $SF$ represents the detection freshness indicator, measuring sensing performance.
• Data collection with only the communication function: The objective function is defined as

  $$O=DF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right).$$

  (27)
• Data collection with only the sensing function: The objective function is defined as

  $$O=SF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right).$$

  (28)

We verify the effectiveness of the first joint mode in Section 4.1 under the single-UAV scenario and in Section 4.2 under the multi-UAV scenario. Here, we verify the effectiveness of the data collection with a individual function (communication/sensing function). The time scheduling of a specific UAV in a time slot with an individual communication/sensing function is depicted in Figure 12.

Here, we verify the effectiveness of modes 2 and 3 under the single-UAV scenario in the larger-range area on account of the convergence speed and the geographical distinction between IoT devices and sensing targets. Similar conclusions can also be drawn in such a scenario. We also adopt the SAC algorithm for this validation in consideration of the quality of results. Similar conclusions can also be drawn for other proposed DRL algorithms. To boost the convergence, we take $b=0.5$ for these two modes in the following experiments.

To compare the two individual function modes 2 and 3, we depict the UAV paths of these two modes in Figure 13 (communication only in Figure 13a and sensing only in Figure 13b). We can conclude from Figure 13a that the UAV flies solely above the IoT devices (blue solid diamonds) to provide uplink communication for data collection without approaching the sensing targets (green solid squares) to perform sensing functions. We can conclude from Figure 13b that the UAV flies exclusively over the sensing targets to transmit radar pulses and receive echo signals, without getting close to the IoT devices to provide uplink communication capabilities. Additionally, for communication-only or sensing-only tasks, the UAV hovers above the IoT devices or sensing targets to ensure fairness among them. Moreover, the UAV flies over each IoT device/sensing target as much as possible to provide broad coverage for individual functions.

To further quantitatively analyze this comparison, we depict the data freshness indicator for communication and the detection freshness indicator for sensing versus episodes of mode 2 and mode 3 in Figure 14a and Figure 14b, respectively. It can be concluded from Figure 14a that as the number of episodes increases, the data freshness indicator gradually decreases (indicating improved communication performance) and then converges; however, the detection freshness indicator (representing sensing performance) changes little, or even increases slightly (indicating a slight deterioration in sensing performance). This is because the optimization objective is only the communication performance rather than the sensing performance. Moreover, the initial position of the UAV may achieve relatively good sensing performance, but as the optimizer iterates, it tends to provide better communication performance at the slight expense of sensing performance. It can be concluded from Figure 14b that as the number of episodes increases, the detection freshness indicator gradually decreases (indicating improved sensing performance) and then converges, while the data freshness indicator remains almost unchanged (indicating stable communication performance). This is because the optimization tends to enhance sensing performance rather than the communication performance, which is in accordance with our expectations.

4.4. Trade-Off Between Communication and Sensing Performance

There exists an inherent trade-off between the communication and sensing functionalities in the ISAC system, and researchers have widely explored it. A well-known trade-off in the ISAC system is the deterministic-random trade-off [28,41,42], where a competitive relationship between communication and sensing performance exists. This illustrates that communication signals are generally better when they are more random, as this attempts to approach the Shannon theoretical limit; however, sensing signals are usually better when they are more deterministic, as this helps in the accurate extraction of channel information. Therefore, the optimal waveforms for communication and sensing are different. If the limited resources are allocated more toward communication, then the system tends to optimize communication performance; conversely, it tends to achieve better sensing performance. Generally speaking, when communication users and sensing targets are not the same, there is usually a competitive relationship between communication and sensing. Different communication-sensing performance pairs can be achieved by adjusting their relative importance, ultimately forming a Pareto front between communication and sensing.

To balance the trade-off between communication and sensing, we redefine the optimization problem (11) into the following form:

$$\begin{array}{cc}\hfill \underset{\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}}{min}& DF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)+\kappa SF\left(\mathit{L},{\mathit{P}}^{\mathrm{C}},{\mathit{P}}^{\mathrm{S}},\mathit{A}\right)\hfill \end{array}$$

(29a)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.C1--C8.\hfill \end{array}$$

(29b)

where the newly introduced parameter $\kappa \in [0,+\infty )$ integrated with the sensing performance term is the importance parameter that reflects how important the sensing is compared with the communication requirements. If $\kappa$ is larger, then sensing performance becomes more important; conversely, communication performance becomes more important. Then, by selecting different values of $\kappa$, it is possible to obtain various communication-sensing performance pairs, which are exactly the points on the Pareto front. The communication–sensing performance pairs on the Pareto front are all dominated where they are not worse than another solution in the two objectives (i.e., the data freshness indicator for communication and the detection freshness indicator for sensing), and they are better than another solution in at least one objective. A better solution means at least an improvement in communication or sensing performance. It is worth noting that the obtained Pareto front is not its exact solution, and it is merely an approximate Pareto front due to the adoption of numerical methods (although all numerical methods are approximate solutions, such as meta-heuristic algorithms, gradient methods, learning approaches, convex approximation methods, etc.).

The illustration of the Pareto front between communication and sensing performance after 10,000 episodes by SAC is depicted in Figure 15. In Figure 15, the x-axis represents communication performance, while the y-axis represents sensing performance. Both communication and sensing performance are better when they are higher, so the nearer the Pareto front is to the upper right corner, the better. It is worth noting that the Pareto front slightly fluctuates. If more settings of $\kappa$ are simulated or more Monte Carlo experiments for a specific setting are conducted, the Pareto front will become smoother and closer to the true analytical Pareto front.

To further explore the relationship between communication and sensing, we can conclude from Figure 15 that as $\kappa$ increases, the detection freshness indicator increases, while the data freshness indicator decreases, which is in accordance with our expectation. A larger $\kappa$ indicates better sensing performance and leads to a larger detection freshness indicator value.

To further quantitatively analyze the trade-off between these two functions, the values of $\kappa$ and its corresponding data/detection freshness indicator values are listed in

Table 2. Different values of $\kappa$ and its corresponding data/detection freshness indicator values after 10,000 episodes in SAC.

$\mathit{\kappa }$	Data Freshness Indicator	Detection Freshness Indicator
0	22.58	51.49
0.1	24.49	37.43
0.3	25.44	34.36
0.5	26.99	20.53
1.0	34.53	15.98
5.0	38.40	13.05
10.0	48.48	8.51

. We can conclude from Table 2 that as the value of $\kappa$ increases, the data freshness indicator also increases, while the detection freshness indicator decreases, which is consistent with the results in Figure 15. Also, when $\kappa =1$, the converged data freshness indicator is 34.53 and the converged detection freshness indicator is 15.98.

Above all, the proposed algorithms are valid for diverse data collection optimization in both single-UAV and multi-UAV scenarios. In addition, they are applicable for data collection with either solely communication or sensing functions, or with both functions jointly.

5. Conclusions

We establish a UAV-enabled diverse data collection framework where the uplink communication from IoT devices to UAVs and the sensing from UAVs to sensing targets are simultaneously considered. We utilize three state-of-the-art DRL algorithms, namely TD3, SAC, and PPO, to minimize the summation of the data freshness indicator and the detection freshness indicator, and the optimal UAV paths, the optimal transmitted power of UAVs and IoT devices, and the optimal transmission allocation indicators can be obtained. Experiments are conducted in both the single-UAV and multi-UAV scenarios. The objective functions gradually converge with the increase in the episode number, which proves the effectiveness. Additionally, the optimal UAV paths are provided to further verify the effectiveness. In addition, we adopt the random strategy as the benchmark, and the results demonstrate that our method outperforms the benchmark. Moreover, the data collection modes with only communication/sensing functions are both valid with reasonable UAV paths and objective function values. Furthermore, the numerical Pareto front that reflects the inherent trade-offs between communication and sensing can be obtained.