Reinforcement Learning-Based Resource Allocation for Multiple Vehicles with Communication-Assisted Sensing Mechanism

Abstract

Autonomous vehicles (AVs) can be equipped with Integrated sensing and communications (ISAC) devices to realize sensing and communication functions simultaneously. Time-division ISAC (TD-ISAC) is advantageous due to its ease of implementation, efficient deployment and integration into any system. TD-ISAC greatly enhances spectrum efficiency and equipment utilization and reduces system energy consumption. In this paper, we propose a communication-assisted sensing mechanism based on TD-ISAC to support multi-vehicle collaborative sensing. However, there are some challenges in applying TD-ISAC to AVs. First, AVs should allocate resources for sensing and communication in a dynamically changing environment. Second, the limited spectrum resources bring the problem of mutual interference of multi-vehicle signals. To address these issues, we construct a multi-vehicle signal interference model, formulate an optimization problem based on the partially observable Markov decision process (POMDP) framework and design a decentralized dynamic allocation scheme for multi-vehicle time–frequency resources based on a deep reinforcement learning (DRL) algorithm. Simulation results show that the proposed scheme performs better in miss detection probability and average system interference power compared to the DRQN algorithm without the communication-assisted sensing mechanism and the random algorithm without reinforcement learning. We can conclude that the proposed scheme can effectively allocate the resources of the TD-ISAC system and reduce interference between multiple vehicles.

1. Introduction

1.1. Background

With the development of communication technology, sensing technology, artificial intelligence and the automotive industry, the level of automotive intelligence is rapidly increasing. Autonomous vehicles (AVs), which will improve road safety and reshape the way we travel, are steering the future of transportation. Sensing and communication are two essential functions of AVs [1]. AVs have many sensors, such as LiDAR, which acquires information about the surroundings and avoids collisions by detecting the presence of obstacles [2,3]. In the meantime, AVs are equipped with wireless communication transceivers to achieve large-scale and high-capacity data exchange in vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) communication. AVs’ sensing and communication functions can enhance the safety and smoothness of the transportation system.

Radar detection and wireless communication both transmit or obtain information through the transmission of radio waves, and in some scenarios, sensing and communication are highly coupled. First, the operating frequency bands for both sensing and communication are moving closer to the millimeter-wave frequency range [4]. Millimeter-wave radar offers advantages of extended detection range, high precision, weather independence and the ability to operate all day compared with cameras and LiDAR. Millimeter-wave communication technology, with its narrow beamwidth and large bandwidth, can significantly improve the communication capability of vehicles. In addition, the similarities between radar and communication in system design, signal processing and data processing create conditions for sharing hardware equipment, such as transceiver systems [5].

Integrated sensing and communications (ISAC) technology aims to unify these two functions and pursue direct trade-offs and mutual performance gains [4]. Enabled by ISAC technology, AVs can simultaneously realize sensing and communication by carrying ISAC radio-frequency (RF) transceivers, which alleviates concerns for electromagnetic compatibility as well as reduces system size, weight, and energy consumption [6,7]. Currently, ISAC has five main realization methods: time division, frequency division, space division, code division and waveform sharing [1]. The time division ISAC (TD-ISAC) method is based on the fact that an RF device occupies different time resources to transmit sensing and communication signals, respectively [8]. The critical advantage of TD-ISAC is its easy implementation and seamless integration into any system [9]. Furthermore, TD-ISAC can utilize specific waveforms for sensing and communication according to their respective needs and improve system spectrum efficiency.

However, AVs equipped with TD-ISAC systems face some technical challenges. First, only one function of sensing and communication can operate at a given time, leading to a trade-off in the performance of both functions, particularly in dynamic environments [10]. Second, as the number of TD-ISAC vehicles increases, it is necessary for vehicles to learn how to coordinate the use of ISAC to minimize mutual interference and improve overall performance [11].

1.2. Related Work

Some scholars have conducted research on time resource allocation for sensing and communication functions. The traditional time division method uses an arbitrary or fixed schedule to switch between sensing and communication. The authors in [12] use the preamble of IEEE 802.11ad standard frames for radar detection and data blocks for communication data transmission. Based on this point, Ref. [13] performs adaptive preamble design, allowing for a trade-off between radar estimation accuracy and communication rate. In addition, Refs. [14,15] study the fundamental trade-offs of radar–communication coexistence with flexible configuration. Nevertheless, the road traffic environment is dynamic and uncertain, and these solutions are fixed during the runtime and unsuitable for implementation in real-time systems. Deep reinforcement learning (DRL) has been recently introduced to find the optimal decision quickly in real time. Some studies use DRL to optimize the allocation of communication and radar time resources for individual vehicles in dynamic environments. The authors of [16,17,18] use the DRL algorithm to decide when to use the communication mode or the radar mode to maximize the data throughput while minimizing the miss detection probability of unexpected events given the uncertainty of the surrounding environment. The AV can quickly obtain the optimal policy without requiring any prior information about the environment. Ref. [19] jointly coordinates radar and communication and minimizes the age of information (AoI) of its communication function.

As the number of TD-ISAC vehicles increases, vehicles need to learn how to coordinate their use of ISAC to minimize mutual interference. In traditional radar and communication systems, interference avoidance can be used to reduce mutual interference. Interference avoidance is achieved by coordinating signal transmission in the time, frequency and space domains to reduce interference [20]. One representative method for interference avoidance in the frequency domain is spectrum allocation. The authors of [21,22] propose a centralized framework to enable multiple access to a shared spectrum among radars relying on vehicular ad hoc networking (VANET) and cellular-vehicle-to-everything (C-V2X) communication, respectively. However, centralized allocation typically involves frequent transmission of large amounts of data between vehicles and the central node, leading to increased communication costs and decision latency. Decentralized allocation, in which each AV autonomously selects its frequency sub-band, can be adopted to reduce communication costs and ensure the timeliness of spectrum resource scheduling. A direct method is random selection, which is easy to implement but cannot suppress interference [23]. In [24], the author proposes a decentralized spectrum allocation approach based on DRL, effectively avoiding mutual interference among automotive radars. In the communication system, the issue of multi-user interference has also been studied using DRL by learning sub-band access strategies to avoid users colliding with other users in the same sub-band [25,26]. However, these studies do not use the ISAC mechanism or consider interference between communication and radar signals. Ref. [27] considers multiple vehicles based on [19] and proposes a medium access control (MAC) protocol that manages contention-free access to the communication–radar channel. However, in each time step, only one vehicle can communicate or sense. Therefore, we need a more efficient decentralized spectrum allocation method suitable for the ISAC mechanism.

1.3. Contributions

This paper focuses on a time division integrated sensing and communication scenario in intelligent transportation systems. In this scenario, AVs on the road are equipped with both communication and sensing functions, sharing the same set of transceiver equipment and switching in a time division manner, occupying the same frequency resources. The environment in which the AVs operate is highly dynamic, with environmental conditions and communication demands constantly changing, and there is interference between signals from different AVs. In practical applications, the trained model’s ability to generalize across different road environments must be rigorously tested to ensure that AVs can make safe and reliable decisions under various circumstances. We propose an intelligent time–frequency resource allocation algorithm to mitigate multi-vehicle mutual interference in dynamic road environments. The proposed algorithm allows AVs to make decentralized decisions based on local observations to avoid significant time overheads caused by centralized scheduling. The main contributions of this paper can be summarized as follows:

• We propose a communication-assisted sensing mechanism based on TD-ISAC. By transmitting sensing information through communication, we can effectively reduce the number of active radars in the system. Simultaneously, by employing time division between sensing and communication, we can enhance spectrum utilization and lower the probability of multiple vehicles’ transmitted signals colliding within the same sub-band, thereby further improving system performance.
• We construct a multi-vehicle sensing and communication interference model. Building this model contributes to a comprehensive understanding of the characteristics and sources of interference, enabling us to take appropriate measures to manage system interference.
• We formulate a multi-vehicle optimization problem based on the partially observable Markov decision process (POMDP) framework. With the POMDP framework, vehicles can choose sensing or communication operations based on dynamic environments adaptively and select different sub-bands to reduce multi-vehicle interference.
• To solve the optimization problem, we design a DRL algorithm using a target network and a prioritized experience replay (PER) scheme to enable multiple vehicles to better obtain the optimal strategy for time–frequency resource allocation under uncertain environmental factors.

The rest of this paper is organized as follows. We describe the problem formulation in Section 2. Section 3 builds the POMDP framework. Furthermore, in Section 4, we present a DRL algorithm. Section 5 evaluates the performance of the proposed approach. Finally, conclusions are drawn in Section 6.

2. Problem Formulation

2.1. Environment Model

This paper considers a two-lane road traffic scenario in both directions with a dynamically changing environment, as shown in Figure 1. Environmental conditions are influenced by road condition, weather condition and moving object state, and the values of these factors can be obtained through a vehicle sensor system, such as a road friction sensor, weather station instrument and camera [16]. Environmental conditions are not the same for different AVs at the same time. In the following text, we provide the modeling of the relationship between environmental conditions and risk levels.

$e\in \mathcal{E}$ represents a particular environmental factor, where $\mathcal{E}=\left\{\rho ,w,m\right\}$, $\rho$, w and m denote the road condition, the weather condition and the moving object state, respectively. $f\in \mathcal{F}$ represents the degree of influence of the environmental factor on the vehicle’s driving, where $\mathcal{F}=\left\{0,1,\dots ,F\right\}$. For simplicity, we can assume that $\mathcal{F}=\left\{0,1\right\}$. When $f=0$, the environmental factor is good; when $f=1$, the environmental factor is poor. ${\tau }_f^e$ represents the probability of the occurrence of the condition $e=f$,

$$\sum _{f\in \mathcal{F}}{\tau }_f^e=1.$$

(1)

$p_f^e$ denotes the probability of the occurrence of an unexpected event under condition $e=f$. Using Bayes’s theorem, the average probability of an unexpected event occurring in the current environment is given by [16]

$$p_u=\sum _{e\in \mathcal{E}}\sum _{f\in \mathcal{F}}{\tau }_f^e{p}_f^e.$$

(2)

The relationship between environmental conditions and risk levels can be provided to AVs by accessing digital map databases [27].

2.2. Signal Model

This paper assumes that all AVs are equipped with TD-ISAC systems, i.e., RF devices occupy different time resources for sensing and communication, respectively. We propose a communication-assisted sensing mechanism. Every two adjacent vehicles in the same lane are paired through V2V communication technology, and this vehicle pair is referred to as a communication-assisted vehicle pair (CAVP). The front vehicle (FV) and the back vehicle (BV) can maintain a minimal distance (several meters or even tens of centimeters) between them by sharing speed, location, status, and other information [28].

While driving, a CAVP can choose to sense or communicate based on the dynamic environment. The frame structure of sensing and communication for the FV and BV is illustrated in Figure 2. During the sensing frame, the FV performs forward sensing using long-range radar (LRR) while the BV awaits communication data. During the communication frame, the FV performs backward communication, and the BV receives communication data from the FV. Additionally, a specific subframe in the communication frame is allocated to the BV for backward sensing using short-range radar (SRR). The direction adjustment between sensing and communication functions is achieved through beam angle adjustment [8].

Let the set of all AVs on the road be $\mathcal{N}=\left\{1,2,\dots ,N\right\}$ and the set of CAVPs be ${\mathcal{N}}_p=\left\{1,2,\dots ,N_p\right\}$, where $N=2N_p$. The total frequency band is divided equally into multiple orthogonal sub-bands, and the set of sub-bands is $\mathcal{M}=\left\{1,2,\dots ,M\right\}$ [21]. A CAVP can freely select a sub-band in the sensing or communication frames. The number of AVs is rapidly increasing, while the number of sub-bands is limited, making it impossible to allocate non-reused orthogonal sub-bands for each CAVP. Multiple CAVPs will inevitably collide in the same sub-band, creating interference [21,24]. Thus, we assume that $N_p\ge M$, and multiple CAVPs can use the same sub-band at the same time.

Due to the proximity of paired vehicles and the relatively low importance of backward sensing, only accidents occurring in the overall front of the CAVP are considered, without considering accidents between the FV and BV or behind the BV. When the forward sensing is successful, the FV can know whether an unexpected event occurs in the environment, represented by 1 and 0, respectively, and the sensory result is stored in the data queue. However, if the FV cannot communicate with the BV in time, the sensory result will fail, which will seriously affect the safety of the BV. We use AoI [19] as a metric to characterize the timeliness of information. AoI is defined as the duration of a sensory result in the vehicle’s queue since it was acquired, denoted by $\Delta$. If $\Delta$ is greater than or equal to the threshold ${\Delta }_0$, we consider the sensory result to have failed. The data will be removed from the queue. It is assumed that when the backward communication is successful, the FV can transmit all the non-failed sensory results. If the communication fails, all the sensory results in the queue of the FV are lost.

2.3. Interference Model

Different CAVPs may use the same sub-band for the sensing or communication frame, generating multi-vehicle mutual interference. Because this paper primarily focuses on reducing unexpected events occurring in front of the CAVP, we model the interference suffered by forward sensing and backward communication.

As shown in Figure 1 and Figure 2, when a CAVP is in the sensing frame, the FV performs forward sensing. At this time, the radar receiver of the FV will be subjected to the same-frequency interference from the LRR signals in the opposite lane and the communication/SRR signals in the same lane. When the CAVP is in the communication frame, the FV performs backward communication. At this time, the communication receiver of the BV will be subjected to the same-frequency interference from the LRR signals in the opposite lane and the communication signals in the same lane.

The received interference power depends on the relative positions of CAVPs. The interference power of the LRR signal from CAVP n in the opposite lane can be expressed as

$$P_{R,I}^n=\frac{P_{R}G{A}_{e}g}{4\pi \left(L^2+d^2\right)}·{\left[p_a\left(\theta \left(d\right)\right)\right]}^2,$$

(3)

where $P_R$ is the LRR transmitting power, G is the antenna gain, $A_e$ is the antenna effective area, g is the propagation decaying factor, $p_a\left(·\right)$ is the normalized antenna beam patter [24], L is the vertical distance between two lanes, d is the horizontal distance between two CAVPs and $\theta \left(d\right)=arctan\left(\frac{L}{d}\right)$ is the radiation direction between two CAVPs. For simplicity, we assume that the power of the communication and SRR signals is the same. The interference power of the communication/SRR signal from CAVP n in the same lane can be expressed as

$$P_{C,I}^n=\frac{P_{C}G{A}_{e}g}{4\pi d^2},$$

(4)

where $P_C$ is the communication/SRR transmitting power.

In conclusion, the total interference power suffered by a CAVP when it performs forward sensing or backward communication is

$$P_I=\sum _{n=1}^{N_R}{P}_{R,I}^n+\sum _{n=1}^{N_C}{P}_{C,I}^n,$$

(5)

where $N_R$ is the number of CAVPs in the opposite lane choosing the same frequency band for the sensing frame and $N_C$ is the number of CAVPs in the same lane choosing the same frequency band for the communication frame.

We use the interference-to-noise ratio (INR) to measure the level of interference to the CAVP [24]. The INR is defined as

$$\eta =\frac{P_I}{{\sigma }^2},$$

(6)

where ${\sigma }^2$ is the receiver noise power. Similar to [24], if $\eta$ is lower than the threshold ${\eta }_0$, we consider the forward sensing or backward communication successful.

3. Pomdp Framework

Due to the high mobility of autonomous vehicles, there is a strict requirement for the latency of information transmission and processing to ensure the safety of autonomous driving. Each CAVP must be able to make decentralized decisions based on its local observations to avoid large time overheads caused by centralized scheduling. However, decentralized decision making leads to CAVPs having access to only the partial environmental information observed by themselves, without the ability to obtain the global information. Based on this, we model multi-agent decision problems as individual POMDP problems for each single agent [29]. A POMDP models an agent decision process in which it is assumed that the system dynamics are determined by an MDP, but the agent cannot directly observe the underlying state. In this approach, each CAVP considers the influence of other CAVPs as part of the environment and makes decisions based on its local observations.

During a CAVP’s driving, it continuously interacts with the dynamic environment. At each discrete time step t, each CAVP i observes the local environmental state ${\mathbf{o}}_t^i$. The global environmental state ${\mathbf{s}}_t$ of the system is a simple superposition of the local observations of the $N_p$ CAVPs, i.e., ${\mathbf{s}}_t=\left[{\mathbf{o}}_t^1,{\mathbf{o}}_t^2,\dots ,{\mathbf{o}}_t^{N_p}\right]$. Based on the observation ${\mathbf{o}}_t^i$, each CAVP takes an action $a_t^i$ according to a policy ${\pi }^i$. Furthermore, the CAVP adjusts the policy ${\pi }^i$ based on the environmental feedback $r_t^i$.

3.1. Observation Space

At each discrete time step t, a CAVP observes its surrounding environmental conditions as well as the state of the data queue:

$${\mathbf{o}}_t^i=\left[d_{S,t}^i,d_{D,t}^i,{\mathbf{e}}_t^i,R_t^i\left({\Delta }_0\right)\right],$$

(7)

where $d_{S,t}^i=\left|p_{S,t}^i-p_t^i\right|$, $d_{D,t}^i=\sqrt{{\left(p_{D,t}^i-p_t^i\right)}^2+L^2}$, $p_t^i$ is the position (one-dimensional coordinate) of CAVP i, and $p_{S,t}^i$ and $p_{D,t}^i$ are the positions of the nearest CAVPs to CAVP i in the same and different lanes, respectively. CAVP i obtains these values through the global navigation satellite system and camera. ${\mathbf{e}}_t^i=\left[\rho ,w,m\right]$; these features are identified in Section 2 to be associated with the risk level. $R_t^i\left({\Delta }_0\right)$ denotes the number of sensory results with ${\Delta }_t^i={\Delta }_0$ in the queue, i.e., data that fail at time step t.

3.2. Action Space

According to the system model, a CAVP can perform flexible time–frequency resource configuration according to the dynamic time-varying characteristics of the vehicle location and road environment. Therefore, the action space of CAVP i at time step t can be defined as

$$a_t^i\in \left\{a_C^{<1>},a_C^{<2>},\dots ,a_C^{<M>},a_R^{<1>},a_R^{<2>},\dots ,a_R^{<M>}\right\},$$

(8)

where $a_C^{<m>}$ and $a_R^{<m>}$ denote a CAVP using sub-band m for the sensing frame and communication frame, respectively.

3.3. Reward Function

As described in Section 2, when a CAVP is in the communication frame, the FV performs backward communication. The total number of unexpected events at that moment is the sum of the number of unexpected events that occurred in the environment and the number of sensory results lost due to communication failures. When a CAVP is in the sensing frame, the FV performs forward sensing. The total number of unexpected events at that moment is the sum of the number of invalid sensory results in the queue and the number of unexpected events missed due to sensing failures. The reward the CAVP receives at each time step is used to balance the sensing and communication functions, encouraging the agent to minimize the number of unexpected events that are not successfully sensed or transmitted. Rewards should be negatively correlated with the total number of unexpected events. So we design the reward function of CAVP i at time step t as

$$r_t^i=\left\{\begin{array}{cc}-\omega \left(u_{C,t}^i+I_t^i\left(C\right)\right)& a_t^i=a_C\\ -\omega \left(n_{over,t}^i+I_t^i\left(R\right)\right)& a_t^i=a_R\end{array}\right.,$$

(9)

$$I_t^i\left(C\right)=\left\{\begin{array}{cc}0& {\eta }_t^i<{\eta }_0\\ n_{all,t}^i& {\eta }_t^i\ge {\eta }_0\end{array}\right.,$$

(10)

$$I_t^i\left(R\right)=\left\{\begin{array}{cc}0& {\eta }_t^i<{\eta }_0\\ u_{R,t}^i& {\eta }_t^i\ge {\eta }_0\end{array}\right.,$$

(11)

where $\omega$ is the weight of the reward, $u_{C,t}^i$ and $u_{R,t}^i$ denote the number of unexpected events in the environment when CAVP i is in the sensing frame and communication frame, respectively, $n_{all,t}^i$ denotes all sensory results in the queue of CAVP i and $n_{over,t}^i$ denotes the number of sensory results with ${\Delta }_t^i={\Delta }_0$ in CAVP i’s queue.

3.4. Optimal Planning

In reinforcement learning, the merit of a policy depends on the cumulative reward obtained from performing the policy over time, and the aim of learning is to find the policy that maximizes the cumulative reward over time [30]. In this paper, we use the $\gamma$ cumulative discount reward, and the optimization issue is represented as follows:

$$\underset{{\pi }^i}{max}R\left({\pi }^i\right)=\mathbb{E}\left[\sum _{t=0}^{N_t}{\gamma }^t{r}_{t+1}^i\left({\pi }^i\right)\right],$$

(12)

where $N_t$ is the number of training steps per episode, $\gamma$ is the discount factor and $\gamma \in \left[0,1\right]$.

4. Ddqn-Per Algorithm

Even though we defined the observation space and reward function in the previous discussion, vehicles navigating in the environment cannot directly access this information. Instead, they need to learn strategies through continuous interaction with the environment. Therefore, the model-free learning algorithm of reinforcement learning should be used. In addition, each CAVP performs its decisions in a decentralized manner. Still, its strategies are influenced by the strategies of other CAVPs, leading to a non-stationary environment and difficulties in convergence. We develop an improved algorithm based on a double deep Q-network (DDQN) [31] named DDQN-PER (double deep Q-network with prioritized experience replay) to deal with the dynamics and uncertainty in a multi-agent environment. The algorithm proposed in this paper uses a deep neural network as a function approximator while using the target network and the PER [32] scheme to enable multiple CAVPs to better obtain the optimal strategy for time–frequency resource allocation under uncertain environmental factors. The algorithm flow is shown in Figure 3, and the algorithm pseudocode is shown as Algorithm 1.

Algorithm 1 DDQN-PER

1: Inputs: 2: $D^i$: empty replay memory with capacity $\left|D^i\right|$ 3: ${\mathit{\theta }}^i$: initial Q-network parameters. 4: ${\hat{\mathit{\theta }}}^i$: initial target Q-network parameters 5: for $episode=1:E$ do 6:    Initialize observations for all vehicles 7:    for $t=1:T$ do 8:       for $i=1:N$ do 9:            Feed observation ${\mathbf{o}}_t^i$ to Q-network to get $Q^i$ 10:          Choose an action $a_t^i$ as (13) 11:          Get a reward $r_t^i$ 12:          Obtain new observation ${\mathbf{o}}_{t+1}^i$ 13:          Store experience $\left({\mathbf{o}}_t^i,a_t^i,r_t^i,{\mathbf{o}}_{t+1}^i\right)$ in $D^i$ 14:          Sample a mini-batch of experiences $\left({\mathbf{o}}_j^i,a_j^i,r_j^i,{\mathbf{o}}_{j+1}^i\right)$ from $D^i$ with prioritization as (15) 15:          Compute importance sampling weights $w_j$ as (16) 16:          Calculate target Q-values $y_j^i$ as (17) 17:          Update priorities of experiences as (14) 18:          Update Q-network parameters ${\mathit{\theta }}_{t+1}^i$ as (20), ${\mathit{\theta }}_t^i\leftarrow {\mathit{\theta }}_{t+1}^i$ 19:          if $tmod{N}_u=0$ then 20:             ${\hat{\mathit{\theta }}}_t^i\leftarrow {\mathit{\theta }}_t^i$ 21:          end if 22:          ${\mathbf{o}}_t^i\leftarrow {\mathbf{o}}_{t+1}^i$ 23:       end for 24:    end for 25: end for

Each CAVP needs to learn its own policy; thus, each CAVP has a separate Q-network. The inputs of the Q-network include relative distances of neighboring CAVPs, the road condition, the weather condition, the moving object state and the data queue state. The outputs are the Q-values of sub-bands for the sensing or communication frame. At each training time step, CAVP i chooses an action $a_t^i$ following the $\epsilon$-greedy policy when the observation is ${\mathbf{o}}_t^i$. The $\epsilon$-greedy policy is defined as follows:

$$a_t^i=\left\{\begin{array}{cc}random& a<\epsilon \\ \underset{a}{arg max}{Q}^i\left({\mathbf{o}}_t^i,a;{\mathit{\theta }}_t^i\right)& a\ge \epsilon \end{array}\right.,$$

(13)

where $Q^i$ are the Q-values of the Q-network of CAVP i, and ${\mathit{\theta }}_t^i$ are the parameters of the Q-network. CAVP i executes the action $a_t^i$ and obtains a reward $r_t^i$ and the next observation ${\mathbf{o}}_{t+1}^i$. The algorithm uses an experience replay scheme to store the experience ${\xi }_t^i=\left({\mathbf{o}}_t^i,a_t^i,r_t^i,{\mathbf{o}}_{t+1}^i\right)$ obtained from the interaction of CAVP i with the environment into the memory $D^i$. The storage capacity of $D^i$ is $\left|D^i\right|$. At each time step, CAVP i randomly samples a mini-batch of experiences ${\xi }_j^i=\left({\mathbf{o}}_j^i,a_j^i,r_j^i,{\mathbf{o}}_{j+1}^i\right)$ from $D^i$ for training.

The experience replay scheme improves data utilization by breaking the correlation between data through linear storage and random sampling of experiences. However, its equal probability replay ignores the importance of experiences, resulting in valuable experiences not being used efficiently. Therefore, this paper introduces the PER scheme. PER uses the temporal difference (TD) error, which represents the difference between the current Q-value and the target Q-value, to measure the importance of each experience. Express the sampling priority of the jth experience as follows:

$$p_j=\left|{\delta }_j\right|+\mu ,$$

(14)

where ${\delta }_j$ is the TD-error of the jth experience, and $\mu$ is a very small positive constant to avoid the priority being 0. The probability that the jth experience is sampled is defined as

$$p\left(j\right)=\frac{p_j^{\alpha }}{{\sum }_k^{k\in \left|D^i\right|}{p}_k^{\alpha }},$$

(15)

where $\alpha \in \left[0,1\right]$ is used to regulate the priority of experience. When $\alpha =0$, priority sampling will degrade to uniform sampling. Before priority sampling, it is necessary to sort the data according to their priority. However, sorting large amounts of data can be memory-intensive and have high time complexity. Therefore, PER uses a sum tree data structure for sampling to improve sampling efficiency [32].

However, when sampling according to importance, the sample distribution is altered. Therefore, the bias must be compensated for according to the importance sampling weights, otherwise the model will overfit the important experiences. Express the importance sampling weights as follows:

$$w_j={(\frac{1}{\left|D^i\right|}·\frac{1}{p\left(j\right)})}^{\beta },$$

(16)

where $\beta \in \left[0,1\right]$ is used to regulate the degree of compensation for bias. When $\beta =1$, the bias introduced by sampling with non-uniform probability will be entirely eliminated. $\beta$ is annealed linearly from ${\beta }_0$ to 1.

Next, we use the sampled experience ${\xi }_j^i=\left({\mathbf{o}}_j^i,a_j^i,r_j^i,{\mathbf{o}}_{j+1}^i\right)$ to calculate the target Q-values. To improve the stability of the algorithm, another network with the same structure and initial parameters as the original Q-network is introduced, called the target Q-network, which is specially used to compute target Q-values. The parameters of the Q-network are updated at each training and passed to the target Q-network after $N_u$ steps. The target Q-value at time step t is calculated as follows:

$$y_j^i=r_j^i+\gamma {\hat{Q}}^i\left({\mathbf{o}}_{j+1}^i,\underset{a^{\prime }}{argmax}{Q}^i\left({\mathbf{o}}_{j+1}^i,a^{\prime };{\mathit{\theta }}_t^i\right);{\hat{\mathit{\theta }}}_t^i\right),$$

(17)

where ${\hat{Q}}^i$ are the Q-values of the target Q-network of CAVP i, and ${\hat{\mathit{\theta }}}_t^i$ are the parameters of the target Q-network.

The DDQN algorithm updates the Q-network parameters by minimizing the mean square error between the current Q-value and the target Q-value, and the loss function can be expressed as

$$L_t^i\left({\mathit{\theta }}_t^i\right)=\mathbb{E}\left[{\left(y_j^i-Q^i\left({\mathbf{o}}_j^i,a_j^i;{\mathit{\theta }}_t^i\right)\right)}^2\right].$$

(18)

The gradient of the loss function is

$${\nabla }_{{\mathit{\theta }}_t^i}{L}_t^i\left({\mathit{\theta }}_t^i\right)=\mathbb{E}\left[\left(y_j^i-Q^i\left({\mathbf{o}}_j^i,a_j^i;{\mathit{\theta }}_t^i\right)\right){\nabla }_{{\mathit{\theta }}_t^i}{Q}^i\left({\mathbf{o}}_j^i,a_j^i;{\mathit{\theta }}_t^i\right)\right].$$

(19)

The Q-network parameters are updated according to the stochastic gradient descent (SGD) algorithm:

$${\mathit{\theta }}_{t+1}^i={\mathit{\theta }}_t^i+w_j{\alpha }_l\left(y_j^i-Q^i\left({\mathbf{o}}_j^i,a_j^i;{\mathit{\theta }}_t^i\right)\right){\nabla }_{{\mathit{\theta }}_t^i}{Q}^i\left({\mathbf{o}}_j^i,a_j^i;{\mathit{\theta }}_t^i\right),$$

(20)

where ${\alpha }_l$ is the learning rate.

5. Simulation Analysis

In this section, we evaluate the performance of the proposed approach through simulation experiments. First, we introduce the simulation setup, performance metrics and contrasting approaches. Then, we analyze the simulation results. Simulation results show that the time–frequency resource allocation problem for radar and communication in the ISAC scenario can be effectively solved within the deep Q-learning framework.

5.1. Simulation Setup

We consider a road traffic scenario as described in Section 2. Vehicles travel at a uniform speed, and the distance $d_n$ between adjacent CAVPs in the same lane follows a uniform distribution U. The frame length is T. We consider the spatial location and environment around the vehicle to be semi-static, i.e., not changing significantly within frame T but changing dynamically over multiple frames [33]. The detailed scenario parameters are set as shown in

Table 1. Scenario parameters setting.

Notation	Value
$\left({\tau }_0^e,{\tau }_1^e\right),e\in \mathcal{E}$	$\left(0.7,0.3\right)$
$\left(p_0^e,p_1^e\right),e\in \mathcal{E}$	$\left(0.01,0.1\right)$
v	10 m/s
${\Delta }_0$	4
T	$0.02$ s
$P_R$	25 dBmW
$P_C$	15 dBmW
G	48 dB
$A_e$	5 mm2
g	$0.1$
L	3 m
${\eta }_0$	10

.

The numbers of neurons in the input, hidden (two layers) and output layers of the network are set to be 6, 24, 24 and 2, respectively. The hidden layer utilizes a rectified linear unit (ReLU) as the activation function, and the output layer has no activation function. In the training, a time step equals the frame length T. Each episode contains multiple time steps. The hyperparameters used in training the Q-networks are shown in

Table 2. Hyperparameters Setting.

Notation	Value
$\omega$	200
$\epsilon$	$0.9\to 0.01$
$N_b$	64
$\mu$	$0.01$
$\alpha$	$0.6$
$\beta$	$0.4\to 1.0$
$N_u$	20
$\gamma$	$0.9$
${\alpha }_l$	$0.0001$

.

5.2. Performance Metrics

We propose two key metrics to evaluate the performance of the proposed approach:

• Miss detection probability: defined as the ratio of the number of unsuccessfully sensed or transmitted unexpected events to the total number of unexpected events in each episode:

  $$p_{miss}=\sum _{i=1}^{N_p}\sum _{t=0}^{N_t}\frac{u_{C,t}^i+I_t^i\left(C\right)+n_{over,t}^i+I_t^i\left(R\right)}{u_{C,t}^i+u_{R,t}^i}.$$

  (21)
• Average system interference power: defined as the ratio of the sum of interference power of all CAVPs per episode to the number of training steps per episode:

  $$P_{ave}=\frac{1}{N_t}\sum _{i=1}^{N_p}\sum _{t=0}^{N_t}{P}_{I,t}^i.$$

  (22)

5.3. Contrasting Approaches

There are three contrasting approaches:

• DRQN: In [24], each vehicle performs sensing independently and does not communicate with other AVs. AVs only need to allocate sensing frequency resources. The authors in [24] introduce a long short-term memory (LSTM) network into the DQN algorithm, enabling the vehicle to learn how to select sensing sub-bands by incorporating both its current and past observations. In this case, miss detection occurs only when the vehicle does not sense successfully.
• Random (CAVPs): We use the random policy to allocate time–frequency resources for the sensing and communication frames of CAVPs. i.e., CAVPs randomly select an action from the action space mentioned in Equation (8) with equal probability.
• DDQN (CAVPs): We use the DDQN algorithm proposed in [31] to allocate time–frequency resources for the sensing and communication frames of CAVPs.
• DQN (single CAVP): In [16], the DQN algorithm was introduced to allocate time resources for both sensing and communication functions of a single vehicle in a dynamic environment. Nevertheless, Ref. [16] did not account for interference between vehicles, implying that both sensing and communication could be successful regardless of the chosen strategy. We apply this algorithm to CAVPs, considering it as an upper-bound performance to evaluate the effectiveness of other algorithms.

5.4. Results Analysis

We first verify the algorithm’s convergence by using the average system reward as a measure of convergence. The average system reward is calculated as follows:

$$R_{ave}=\frac{1}{N_t}\sum _{i=1}^{N_p}\sum _{t=0}^{N_t}{r}_t^i.$$

(23)

We selected five random seed values, namely, 10, 20, 30, 40 and 50, and calculated the average $R_{ave}$ of the algorithms under these five random seeds. As shown in Figure 4, the $R_{ave}$ of DDQN-PER gradually increases with the number of episodes and eventually stabilizes around 800 episodes. Compared with DDQN and random, DDQN-PER has better convergence performance.

Then, we evaluate the performance of the proposed approach and the contrasting approaches through the following four aspects:

• The number of AVs changes, other conditions are fixed: In Figure 5, as N increases, $p_{miss}$ and $P_{ave}$ increase for all algorithms except DQN. The reason is that as N increases, the mutual interference between the AVs increases, while DQN is used in a single-CAVP scenario, and the effect of interference is not considered. In addition, compared with DRQN, random has a lower $P_{ave}$. This is because the proposed communication-assisted sensing mechanism allows some vehicles to acquire sensory data through communication, eliminating the requirement for all vehicles to perform sensing tasks simultaneously, effectively reducing inter-vehicle interference. Nevertheless, random has a higher $p_{miss}$ than DRQN. This is because the $p_{miss}$ of random is not only affected by system interference but also influenced by the dynamic environment. Random cannot choose sensing or communication modes based on the environment, leading to a higher rate of $p_{miss}$. As DDQN-PER utilizes the communication-assisted sensing mechanism and adaptive time–frequency resource allocation algorithm, it can effectively control the mutual interference between vehicles and reduce the miss detection probability. Thus, compared with other algorithms, $p_{miss}$ and $P_{ave}$ of DDQN-PER are closer to DQN, and the performance does not deteriorate significantly as N increases. In conclusion, DDQN-PER proves to be more advantageous than other algorithms, particularly in scenarios with a higher volume of vehicles.
• The number of sub-bands changes, other conditions are fixed: In Figure 6, as M increases, $p_{miss}$ and $P_{ave}$ decrease for all algorithms except DQN. The reason is that as M increases, the probability of two signals switching to the same sub-band becomes small. Compared with other algorithms, $p_{miss}$ and $P_{ave}$ of DDQN-PER are closer to DQN. It can be deduced that DDQN-PER holds a comparative advantage over other algorithms, especially when the number of sub-bands is limited.
• The interval changes, other conditions are fixed: In Figure 7, when $d_n\sim U$(50 m, 60 m), $p_{miss}$ of DDQN-PER is higher than DRQN. However, when $d_n$ is small, DDQN-PER has better performance. In addition, compared with other algorithms, the $P_{ave}$ of DDQN-PER is closer to 0. This is because when the interval is small, DRQN is greatly affected by interference and cannot effectively allocate frequency resources, while DDQN-PER can effectively control mutual interference and reduce the miss detection probability by using the communication-assisted sensing mechanism and adaptive time–frequency resource allocation algorithm. It can be concluded that DDQN-PER is more advantageous than other algorithms when the interval is smaller.
• The $p_1^m$ (i.e., the probability of the occurrence of an unexpected event under condition $m=1$) changes, other conditions are fixed: In Figure 8, as $p_1^m$ changes, $p_{miss}$ of DDQN-PER and DQN changes, while other algorithms are almost unchanged. This is because DDQN-PER and DQN can adaptively allocate time–frequency resources according to the environment. As DDQN-PER is affected by multi-vehicle interference, its $p_{miss}$ does not continue to decrease as $p_1^m$ increases like DQN. Furthermore, compared with other algorithms, the $P_{ave}$ of DDQN-PER is closer to 0.

6. Conclusions

In this paper, we propose a communication-assisted sensing mechanism based on TD-ISAC, which uses communication to transmit sensing information effectively. Moreover, we construct an interference model for multi-vehicle sensing and communication signals and formulate a multi-vehicle optimization problem based on the POMDP framework. We develop the DDQN-PER algorithm to enable vehicles to adjust their policies adaptively for decentralized time–frequency resource allocation in the dynamic environment. Simulation results show that the proposed scheme can significantly reduce miss detection probability and average system interference power. Furthermore, the method proposed in this paper also applies to the problem of multi-vehicle mutual interference where a TD-ISAC vehicle assists a radar-free vehicle.