Digital Twin-Assisted Deep Reinforcement Learning for Joint Caching and Power Allocation in Vehicular Networks

Abstract

In recent years, digital twin technology has demonstrated remarkable potential in intelligent transportation systems, leveraging its capabilities of high-precision virtual mapping and real-time dynamic simulation of physical entities. By integrating multi-source data, it constructs virtual replicas of vehicles, roads, and infrastructure, enabling in-depth analysis and optimal decision-making for traffic scenarios. In vehicular networks, existing information caching and transmission systems suffer from low real-time information update and serious transmission delay accumulation due to outdated storage mechanism and insufficient interference coordination, thus leading to a high age of information (AoI). In response to this issue, we focus on pairwise road side unit (RSU) collaboration and propose a digital twin-integrated framework to jointly optimize information caching and communication power allocation. We model the tradeoff between information freshness and resource utilization to formulate an AoI-minimization problem with energy consumption and communication rate constraints, which is solved through deep reinforcement learning within digital twin systems. Simulation results show that our approach reduces the AoI by more than 12 percent compared with baseline methods, validating its effectiveness in balancing information freshness and communication efficiency.

1. Introduction

With the development of Internet of Vehicles (IoV) technology and its increasingly broad application prospects, IoV and its related technologies have garnered significant attention from both the automotive industry and academia. IoV technology enhances transportation safety, efficiency, and intelligence, serving as a foundational technology for vehicle dispatching and autonomous driving while delivering diverse services and application experiences to passengers [1]. In the IoV system, vehicles acquire high-frequency and heterogeneous data from road unit side (RSU) and cloud platforms to obtain traffic condition and entertainment data. Therefore, information cache stands as a core technology for ensuring real-time response and high reliability by enabling seamless data flow among vehicles, roads, and clouds [2,3]. To more effectively support IoV applications and improve the quality of service (QoS), it is necessary to optimize the cache and communication resource allocation, which can synergistically achieve a multi-objective enhancement of system throughput, information freshness, and energy efficiency, thus supporting the large-scale intelligent development of the IoV [4].

Digital twin, alternatively referred to as a cyber twin, has emerged as a pivotal technology for constructing future smart cities and the industrial metaverse, garnering escalating global attention from both industry and academic circles. At its core, a digital twin is a virtual replica of a real-world physical system, process, or various abstract concepts. It can be implemented through a computer program or encapsulated within a software model that enables real-time interaction and synchronization with its physical counterpart [5]. The integration of digital twin with deep learning and generative artificial intelligence (AI) has become mainstream, addressing dynamic resource optimization and intelligent decision-making in scenarios such as vehicle networking, spacecrafts, and industrial equipments [6,7].

The application of digital twins in vehicle networking remains in the exploratory stage. The authors of [8] constructed a digital twin-based IoV framework and assigned nearby Internet-of-Things gateways for vehicular communication, addressing the issues of insufficient RSU coverage and high latency. A digital twin-driven remote vehicle data-sharing platform was built in [9], which integrated federated learning and blockchain to design privacy-preserving mechanisms and dynamic incentive contracts. Digital twin was utilized in [10] to introduce a data-driven framework to improve traffic efficiency and minimize flow-table overflow in software-defined vehicular networks. Furthermore, existing studies have begun to explore the collaboration between digital twin and multi-dimensional resources. In [11], the authors introduce a two-tier digital twin architecture with multi-agent reinforcement learning to optimize edge computing resource allocation, and the authors of [12] proposed a digital twin-assisted framework of edge resource allocation with adaptive federated multi-agent reinforcement learning in IoV to minimize the energy consumption and service cost. For dynamic IoV environments, the researchers of [13] combined semantic sensing and digital twin to optimize the QoS with a reduction in system cost, ensuring stable and efficient services with high vehicle mobility and fast-changing network topology. In [14], the authors proposed an edge intelligence collaboration scheme based on digital twin, which optimized task offloading decisions through a virtual mirror, but only considered communication delay optimization in a single-RSU scenario. Moreover, actor–critic learning was used in [15] to solve the joint network selection and power level allocation problem in digital twin data synchronization, minimizing the long-term cost of delays and energy consumption. Although these studies built digital twin-based frameworks in vehicular networks for resource optimization, they did not consider the cooperation of RSUs and joint optimization of multiple resources.

To quantify the freshness of information, the authors of [16] proposed the concept of age of information (AoI) and designed an application-layer broadcast rate adaptation algorithm to minimize the system AoI. The AoI of an information block at a certain moment refers to the time elapsed from the generation of that information block to that moment. The authors of [17] pointed out that minimizing the AoI of the information acquired by the receiver was neither equivalent to maximizing the utilization of communication systems nor equivalent to minimizing transmission delay. Numerous studies have explored the application of AoI in vehicular networks. The authors of [18] proposed a greedy algorithm for vehicular beacon scheduling to minimize expected sum AoI. In [19], the authors investigated the problem of minimizing vehicle power under stringent latency and reliability constraints in terms of probabilistic AoI, and derived the relationship between AoI and the data queue length distribution. The authors of [20] proposed a cache-aided delayed update and delivery scheme for the joint optimization of content update, delivery, and radio resource allocation to meet the AoI requirements of various applications, while in [21], the researchers built an AoI analysis framework for vehicular social networks based on mean field theory and jointly optimized the information update frequency and transmission probability to minimize the average peak AoI. Ref. [22] conducted a comprehensive performance evaluation of the main broadcast rate control algorithm from the aspects of channel load, throughput, and AoI, and revealed that controlling congestion only based on channel load or AoI would lead to suboptimal performance. Moreover, the AoI for vehicular networks was analyzed in [23] through a spatiotemporal model, which revealed the correlation between communication and computing capabilities but did not involve information cache allocation.

With the objectives of minimizing latency, AoI, or maximizing throughput, extensive research has been conducted on optimizing communication power in vehicular networks. The authors of [24] proposed a joint scheduling and power control scheme based on greedy algorithms for vehicular links, reducing the service interruption rate and maintaining throughput fairness among vehicles. Based on the analysis of the queuing system for vehicle-to-vehicle (V2V) links, Ref. [25] put forward an optimal power allocation scheme for each pair of spectrum-reusable links. In vehicular networks, the rapid mobility of vehicles leads to dynamic channel variations, making it difficult to obtain real-time channel state information and thus posing challenges to wireless resource allocation. To address this issue, the authors of [26] proposed a power allocation scheme to reduce the interference in V2V communication, maximizing the number of communicating vehicles while meeting latency and reliability requirements. In [27], the researchers achieved optimal power allocation in vehicular networks by minimizing system outage probability to enhance cooperative communication performance. The authors of [28] proposed an optimization method for energy consumption and AoI in the IoV system by adjusting vehicular transmission intervals and power via a deep reinforcement learning (DRL) algorithm. In spite of the consideration of channel changes caused by vehicle movement, the existing studies mostly relied on traditional algorithms or simplified models for power adjustment, making it difficult to adapt to the rapidly changing vehicular environment. Moreover, most of them optimized power for local links or partial regions, lacking global mechanisms such as multi-link collaboration and interference coordination, which easily led to suboptimal power allocation due to interference accumulation in dense scenarios. Different from these works [19,20,25,28], we combine a digital twin for joint caching and power allocation with a DRL approach to achieve a global and adaptive solution in vehicular networks.

Despite the progress made in existing research regarding the integration of digital twin with IoV, notable gaps persist, particularly in supporting the highly efficient utilization of caching space and communication resource. Firstly, there is a lack of focus on the collaborative scenario of RSUs, which hinders the support for short-range cooperative services among adjacent RSUs. Secondly, communication power and information caching are seldom regarded as joint optimization objectives in the resource allocation of vehicular networks, and digital twin technology has not been incorporated to realize dynamic adaptation between these factors. In response to these issues, we propose a digital twin-integrated framework for pairwise RSU collaboration, which jointly optimizes information caching and communication power allocation to minimize the total AoI in vehicular networks. This framework leverages a multi-agent DRL approach with the multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm to address the dynamic and non-convex nature of the optimization problem and enable efficient caching and power allocation strategies through virtual interactions within the digital twin environment. The main contributions of this work are summarized as follows:

• We establish a system where the vehicles obtain information from edge storage servers and develop a queuing model for information transmission from RSUs to vehicles. A scalable pairwise RSU collaboration model is proposed, leveraging digital twins for real-time scenario-mirroring of RSUs. Based on these models, we derive the expression of the total AoI in the system.
• To boost real-time information acquirement in vehicular networks, we formulate an optimization problem for minimizing the total AoI by allocating information cache and controlling transmission power, constrained by energy consumption and data rate requirement. To solve this problem, a multi-agent DRL approach is developed with a double Q-network-based MATD3 algorithm, mitigating overestimation bias in the training.
• The simulations demonstrate superior performance of the proposed approach in terms of convergence speed, AoI optimization, and resource efficiency compared with baseline methods, underscoring the framework’s adaptability to real-world vehicular network dynamics.

The remainder of this paper is organized as follows. Section 2 builds a vehicular network and AoI model, Section 3 formulates an optimization problem to minimize total AoI via RSU power control and cache allocation, and Section 4 proposes a MATD3-based multi-agent deep reinforcement learning method for obtaining solutions. Section 5 sets the simulation scenario parameters and analyzes the experiment results. Finally, the paper concludes in Section 6.

2. System Model

2.1. Scenario Description

We constructed a vehicular network model, as shown in Figure 1. There are U pairs of RSUs in the system, i.e., there are $2U$ RSUs. The edge storage server connected to each pair of RSUs stores I types of real-time information with different functions and in the form of information blocks. Each type of information is updated at a fixed time interval in the server. Assume that the sizes of information blocks of the same type are equal. The number of vehicles in the system model is V, and each vehicle is simultaneously connected to the nearest pair of RSUs, through which the vehicle obtains the information in the storage server. Let $\mathcal{U}=\{1,2,\dots ,U\}$, $\mathcal{V}=\{1,2,\dots ,V\}$ and $\mathcal{I}=\{1,2,\dots ,I\}$ denote the set of RSU pairs, the set of vehicles, and the set of information types, respectively. Let ${\mathcal{V}}_u$ be the set of vehicles connected to the u-th pair of RSUs. Each pair of RSUs is connected to a fog server. Therefore, there are a total of U fog servers in the system model. The fog server can be used to control and manage the communication power and information storage allocation of the two connected RSUs. There is a base station in the system, and it is connected to a server, which is used to run the digital twin system. Each fog server in the physical space has a corresponding digital mirror in the digital twin. The fog server maintains contact with its digital mirror through the base station.

2.2. Channel Model

Since the vehicular network is dynamic, the communication channel is considered to be time-varying [29] and subject to block fading. The block fading is composed of path loss, frequency-selective Rayleigh fading, and shadowing fading. The path loss $h_p$ is modeled as $h_p\propto {(d/d_0)}^{\eta }$, where $\eta$ is the path loss exponent, d denotes the distance between the vehicle and RSU, and $d_0$ is the preferred distance. The frequency-selective Rayleigh fading follows a Rayleigh distribution with a standard deviation of ${\sigma }_f^2$. The shadowing fading takes the form of ${10}^{-{\displaystyle \frac{\xi }{10}}}$, where $\xi$ is a random variable that follows a normal distribution with a mean of 0 and a variance of ${\sigma }_s^2$ [30].

2.3. Age of Information Model

Assume that the message blocks are instantly arranged to be transmitted from the server connected to the RSU to the vehicles connected to that RSU after its generation. The messages undergo a queueing process, and each message waits for the prior messages to be sent first. The AoI of a message at time slot t is defined as the elapsed time from the generation to time t. Let ${\delta }_{u,v,i}\left(t\right)$ denote the AoI at time t for the i-th type of information downloaded by vehicle v from the u-th pair of RSUs. To facilitate the calculation and optimization of the AoI of the total system, the AoI is decomposed into three components, including the average queuing time $t_{u,k,v}^{\mathrm{q}}$ before transmission in the u-th pair of RSUs, the time $t_{u,k,v,i}^{\mathrm{d}}$ for downloading the message block, and the elapsed time $t_{u,v,i}^{\mathrm{s}}\left(t\right)$ from the moment the message block is received by vehicle v to time t. The AoI is expressed as

$${\delta }_{u,v,i}\left(t\right)=x_{u,i}\left(t_{u,1,v}^{\mathrm{q}}+t_{u,1,v,i}^{\mathrm{d}}+t_{u,v,i}^{\mathrm{s}}\left(t\right)\right)+(1-x_{u,i})\left(t_{u,2,v}^{\mathrm{q}}+t_{u,2,v,i}^{\mathrm{d}}+t_{u,v,i}^{\mathrm{s}}\left(t\right)\right),$$

(1)

where $x_{u,i}=1$ denotes that the i-th type of information is stored in RSU 1 of the u-th pair of RSUs, and $x_{u,i}=0$ denotes that it is stored in RSU 2. Figure 2 shows the curve of ${\delta }_{u,v,i}\left(t\right)$ over time, where $Q_j$, $\forall j\in \{1,2,\dots ,N\}$, represents the area of the corresponding regions in the figure; $t_j$, $\forall j\in \{1,2,\dots ,N\}$, indicates the moment when the j-th block of information type i is generated in the storage server, and $t_j^{\prime }$ represents the moment when this message block is received by vehicle v. Let $X_j=t_j-t_{j-1}$ and $Y_j=t_j^{\prime }-t_j$. Assume that vehicles download information at a fixed rate. Since each type of information is updated at fixed time intervals in the server and the message blocks of the same type of information are of equal size, we can deduce

$$X_1=X_2=\cdots =X_N={\displaystyle \frac{1}{{\lambda }_i}}$$

(2)

$$Y_1=Y_2=\cdots =Y_N=t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}},$$

(3)

where ${\lambda }_i$ represents the update frequency of information type i. As illustrated in Figure 2, $Q_j$ can be expressed as

$$Q_j={\displaystyle \frac{1}{2}}{(X_j+Y_j)}^2-{\displaystyle \frac{1}{2}}{Y}_j^2,j=2,\dots ,N.$$

(4)

Consequently, $Q_2=\cdots =Q_N$ is satisfied. The average AoI for information type i downloaded by vehicle v over the time T is calculated by

$${\Delta }_{u,v,i}={\displaystyle \frac{1}{T}}{\int }_0^T{\delta }_{u,v,i}\left(t\right)dt.$$

(5)

Substituting (1) into (5), we get

$$\begin{array}{cc}\hfill {\Delta }_{u,v,i}& =x_{u,i}·{\displaystyle \frac{1}{T}}{\int }_0^T\left(t_{u,1,v}^{\mathrm{q}}+t_{u,1,v,i}^{\mathrm{d}}+t_{u,v,i}^{\mathrm{s}}\left(t\right)\right)dt\hfill \\ \hfill & +(1-x_{u,i})·{\displaystyle \frac{1}{T}}{\int }_0^T\left(t_{u,2,v}^{\mathrm{q}}+t_{u,2,v,i}^{\mathrm{d}}+t_{u,v,i}^{\mathrm{s}}\left(t\right)\right)dt.\hfill \end{array}$$

(6)

By the geometric meaning of a definite integral, the expression in (6) satisfies

$${\displaystyle \frac{1}{T}}{\int }_0^T\left(t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}+t_{u,v,i}^{\mathrm{s}}\left(t\right)\right)dt={\displaystyle \frac{1}{T}}\left(Q_1+\sum _{j=2}^{N_{v,i}\left(T\right)}{Q}_j+{\displaystyle \frac{1}{2}}{Y}_N^2\right),$$

(7)

where $N_{v,i}\left(T\right)$ represents the number of blocks of information type i received by vehicle v up to time T, i.e., $N_{v,i}\left(T\right)=max\left\{n\right|t_n\le T\}$. Substituting (2)–(4) into (7) gives

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{T}}{\int }_0^T\left(t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}+t_{u,v,i}^{\mathrm{s}}\left(t\right)\right)dt\hfill \\ \hfill & ={\displaystyle \frac{1}{T}}\{Q_1+\left[N_{v,i}\left(T\right)-1\right]\times [{\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{1}{{\lambda }_i}}+t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}\right)}^2\hfill \\ \hfill & -{\displaystyle \frac{1}{2}}{\left(t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}\right)}^2+{\displaystyle \frac{1}{2}}{\left(t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}\right)}^2\left]\right\}.\hfill \end{array}$$

(8)

Consider the practical scenario that the information is continuously downloaded during long-term operation; time T is assumed to be infinite for achieving the steady and statistical character of AoI. Since $\underset{T\to +\infty }{lim}{\displaystyle \frac{Q_1}{T}}=0$, $\underset{T\to +\infty }{lim}{\displaystyle \frac{N_{v,i}\left(T\right)-1}{T}}={\lambda }_i$, and $\underset{T\to +\infty }{lim}{\displaystyle \frac{1}{2T}}{\left(t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}\right)}^2=0$, we have

$$\begin{array}{cc}\hfill & \underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}{\int }_0^T(t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}+t_{u,v,i}^{\mathrm{s}}\left(t\right))dt\hfill \\ \hfill & ={\displaystyle \frac{{\lambda }_i}{2}}\left[{\left({\displaystyle \frac{1}{{\lambda }_i}}+t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}\right)}^2-{(t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}})}^2\right]\hfill \\ \hfill & ={\displaystyle \frac{1}{2{\lambda }_i}}+t_{u,k,v}^{\mathrm{q}}+t_{u,k,v,i}^{\mathrm{d}}.\hfill \end{array}$$

(9)

Therefore, as T is enlarged infinitely, AoI for the information type i downloaded by vehicle v is calculated by

$$\begin{array}{cc}\hfill {\overline{\Delta }}_{u,v,i}& =\underset{T\to \infty }{lim}{\Delta }_{u,v,i}\hfill \\ \hfill & =x_{u,i}\left({\displaystyle \frac{1}{2{\lambda }_i}}+t_{u,1,v}^{\mathrm{q}}+t_{u,1,v,i}^{\mathrm{d}}\right)+(1-x_{u,i})\left({\displaystyle \frac{1}{2{\lambda }_i}}+t_{u,2,v}^{\mathrm{q}}+t_{u,2,v,i}^{\mathrm{d}}\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{2{\lambda }_i}}+x_{u,i}\left(t_{u,1,v}^{\mathrm{q}}+t_{u,1,v,i}^{\mathrm{d}}\right)+(1-x_{u,i})\left(t_{u,2,v}^{\mathrm{q}}+t_{u,2,v,i}^{\mathrm{d}}\right).\hfill \end{array}$$

(10)

Due to the particular delay between the digital twin and the physical environment, there is a deviation between the data in the digital twin and the physical environment. Let $n_{u,k,v}$ be the channel used for communication between vehicle v and RSU k of the u-th pair of RSUs. Assume that ${\tilde{h}}_{u,k,v,n_{u,k,v}}$ is the estimated value of the channel gain in the digital twin when vehicle v is connected to RSU k of the u-th pair of RSUs, and $\Delta h_{u,k,v,n_{u,k,v}}$ is the deviation between the estimated value and the actual value. Then, the actual value of the channel gain is expressed as

$$h_{u,k,v,n_{u,k,v}}={\tilde{h}}_{u,k,v,n_{u,k,v}}+\Delta h_{u,k,v,n_{u,k,v}}.$$

(11)

Let $p_{u,k,v,n_{u,k,v}}$ be the power for communication between RSU k of the u-th pair of RSUs and vehicle v. $D_{u,k,v}$ is the interference that vehicle v receives from other vehicles and RSUs when connected to RSU k of the u-th pair of RSUs, calculated by

$$D_{u,k,v}=\sum _{u^{\prime }\ne u}\sum _{v^{\prime }\ne v}\sum _{k^{\prime }=1}^2{1}_{n_{u,k,v}=n_{u^{\prime },k^{\prime },v^{\prime }}}{p}_{u^{\prime },k^{\prime },v^{\prime },n_{u^{\prime },k^{\prime },v^{\prime }}}{h}_{u^{\prime },k^{\prime },v,n_{u,k,v}}.$$

(12)

Therefore, the actual value of the signal-to-interference-plus-noise ratio when vehicle v is connected to RSU k of the u-th pair of RSUs is expressed by

$${\Gamma }_{u,k,v}={\displaystyle \frac{p_{u,k,v,n_{u,k,v}}{h}_{u,k,v,n_{u,k,v}}}{{\sigma }^2+D_{u,k,v}}},$$

(13)

where ${\sigma }^2$ represents the power of additive white Gaussian noise (AWGN). Let the system channel bandwidth be W; then the corresponding download rate is calculated by

$$R_{u,k,v}=W{log}_2(1+{\Gamma }_{u,k,v}).$$

(14)

Therefore, the downloading time can be calculated by

$$t_{u,k,v,i}^{\mathrm{d}}={\displaystyle \frac{c_i}{R_{u,k,v}}},$$

(15)

where $c_i$ represents the size of the information block of the i-th type of information.

According to queuing theory, the sum of the average number of information blocks generated per unit time for all types of information in RSU 1 and RSU 2 of the u-th pair of RSUs is represented as

$${\lambda }_{u,1}=\sum _{i=1}^I{x}_{u,i}{\lambda }_i,$$

(16)

$${\lambda }_{u,2}=\sum _{i=1}^I(1-x_{u,i}){\lambda }_i.$$

(17)

Assume that the information transmitted from the same RSU to the vehicle is in the same queue when information is queuing for transmission. It can be obtained that

$$t_{u,k,v}^{\mathrm{q}}={\displaystyle \frac{{\lambda }_{u,k}}{({\mu }_{u,k,v}-{\lambda }_{u,k}){\mu }_{u,k,v}}},\forall k\in \{1,2\}$$

(18)

where ${\mu }_{u,k,v}$ is the average number of information blocks that RSU k transmits to vehicle v per unit time. The expression is given as

$${\mu }_{u,k,v}={\displaystyle \frac{R_{u,k,v}}{{\overline{c}}_{u,k}}}.$$

(19)

In (19), ${\overline{c}}_{u,k}$ is the average size of information blocks in RSU k of the u-th pair of RSUs, which is calculated by

$${\overline{c}}_{u,1}={\displaystyle \frac{{\sum }_{i=1}^I{x}_{u,i}{\lambda }_i{c}_i}{{\lambda }_{u,1}}},$$

(20)

$${\overline{c}}_{u,2}={\displaystyle \frac{{\sum }_{i=1}^I(1-x_{u,i}){\lambda }_i{c}_i}{{\lambda }_{u,2}}}.$$

(21)

We can express the sum of the average AoI of all types downloaded by vehicle v through the u-th pair of RSUs as

$${\Delta }_{u,v}=\sum _{i=1}^I{a}_{u,v}{\overline{\Delta }}_{u,v,i},$$

(22)

where $a_{u,v}=1$ indicates that vehicle v is connected to the u-th pair of RSUs, otherwise $a_{u,v}=0$.

3. Problem Formulation

Aiming to enhance the real-time performance of information retrieval in the vehicular network, an optimization problem of minimizing the total AoI in the system by allocating information cache and controlling the transmission power of RSUs is formulated as

$$\begin{array}{cc}\hfill \underset{\mathcal{X},\mathcal{P}}{min}& \sum _{u=1}^U\sum _{v=1}^V{\Delta }_{u,v}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathrm{C}1:R_{u,k,v}\ge R^{min},\forall u\in \mathcal{U},\forall k\in \{1,2\},\forall v\in {\mathcal{V}}_u\hfill \\ \hfill & \mathrm{C}2:\sum _{v=1}^V{a}_{u,v}{p}_{u,k,v,n_{u,k,v}}\le P_{u,k}^{max},\forall u\in \mathcal{U},\forall k\in \{1,2\}\hfill \\ \hfill & \mathrm{C}3:\sum _{i=1}^I{x}_{u,i}{c}_i\le C_{u,1},\forall u\in \mathcal{U}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & \mathrm{C}4:\sum _{i=1}^I(1-x_{u,i})c_i\le C_{u,2},\forall u\in \mathcal{U}\hfill \\ \hfill & \mathrm{C}5:{\mu }_{u,k,v}>{\lambda }_{u,k},\forall u\in \mathcal{U},\forall k\in \{1,2\},\forall v\in {\mathcal{V}}_u\hfill \end{array}$$

where $\mathcal{X}=\{x_{u,i}\mid \forall u\in \mathcal{U},\forall i\in \mathcal{I}\}$ is the set of information cache allocation variable, and $\mathcal{P}=\{p_{u,k,v,n_{u,k,v}}and\mid \forall u\in \mathcal{U},\forall v\in \mathcal{V},\forall k\in \{1,2\}\}$ represents the set of transmission power between RSUs and vehicles. C1 indicates that the download rate of each vehicle should not be less than $R^{min}$ to ensure the quality of service. C2 means that the sum of transmission power allocated by each RSU should not exceed the maximum power. C3 and C4 indicate that the total size of information in the caching server should not exceed the overall capacity of the server. C5 is formulated to prevent the queue from becoming infinitely long.

This optimization problem is non-linear and non-convex, and the power control of RSUs is interrelated. Therefore, it is difficult to solve the power and storage allocation of each RSU by traditional optimization methods. In contrast, RL can learn optimal multi-variable collaborative decision-making strategies through interactive trial-and-error attempts between agents and the environment. It is particularly suitable for such dynamic, non-convex, and high-dimensional coupled optimization scenarios.

4. Multi-Agent Deep Reinforcement Learning Method

Due to the existence of multiple pairs of RSUs, a multi-agent DRL-based method is proposed to solve problem (23). The process of this algorithm is described in detail below.

4.1. Markov Decision Process

Problem (23) is formulated as a multi-agent extended Markov decision process to describe the decision-making on information cache and communication power allocation based on the MATD3 algorithm. Below, we introduce the settings of the state space, action space, and reward function of the Markov decision process, respectively.

4.1.1. State Space

We define the state space $s_u$ of fog server u as

$${\mathit{s}}_u=({\Delta }_{u,1},{\Delta }_{u,2},\cdots ,{\Delta }_{u,v},\cdots ,{\Delta }_{u,V_u}),v\in {\mathcal{V}}_u$$

(24)

where $V_u$ is the number of vehicles in the set ${\mathcal{V}}_u$. Therefore, the global state of the system is represented as

$$\mathit{s}=({\mathit{s}}_1,{\mathit{s}}_2,\cdots ,{\mathit{s}}_U).$$

(25)

4.1.2. Action Space

Since the MATD3 algorithm is only applicable to a continuous action space, a continuously valued variable in (0, 1) is defined as $z_{u,i}$. Let

$$x_{u,i}=\left\{\begin{array}{cc}0,\hfill & 0<z_{u,i}\le 0.5\hfill \\ 1,\hfill & 0.5<z_{u,i}<1.\hfill \end{array}\right.$$

(26)

The action space of fog server u is defined as

$${\mathit{a}}_u=\left(z_{u,1},\cdots ,z_{u,I},p_{u,1,1,n_{u,1,1}},\cdots ,p_{u,1,V_u,n_{u,1,V_u}},p_{u,2,1,n_{u,2,1}},\cdots ,p_{u,2,V_u,n_{u,2,V_u}}\right).$$

(27)

Therefore, the joint action space of all fog servers is defined as

$$\mathit{a}=({\mathit{a}}_1,{\mathit{a}}_2,\cdots ,{\mathit{a}}_U).$$

(28)

4.1.3. Reward Setting

A reward function for each fog server is set to guide the algorithm to minimize the total AoI in the system under the constraints given in problem (23). In the reward function, constraints C1 to C4 are designed as penalty terms. For fog server u, its reward function is designed as

$$\begin{array}{cc}\hfill {\mathit{r}}_u=& -\sum _{u=1}^U\sum _{v=1}^V{\Delta }_{u,v}\hfill \\ \hfill & +{\phi }_1\sum _{u=1}^U\sum _{k=1}^2\sum _{v=1}^{V}min(R_{u,k,v}-R^{min},0)\hfill \\ \hfill & +{\phi }_2\sum _{u=1}^U\sum _{k=1}^{2}min\left(P_{u,k}^{max}-\sum _{v=1}^V{a}_{u,v}{p}_{u,k,v,n_{u,k,v}},0\right)\hfill \\ \hfill & +{\phi }_3\sum _{u=1}^U\left[min\left(C_{u,1}-\sum _{i=1}^I{x}_{u,i}{c}_i,0\right)\right.\hfill \\ \hfill & \left.+min\left(C_{u,2}-\sum _{i=1}^I(1-x_{u,i})c_i,0\right)\right],\hfill \end{array}$$

(29)

where ${\phi }_1$, ${\phi }_2$, and ${\phi }_3$ are penalty coefficients. When constraint C5 is not satisfied, $t_{u,k,v}^{\mathrm{q}}$ becomes negative or infinite. To avoid it, a penalty is enforced by

$$t_{u,k,v}^{\mathrm{q}}=L,{\mu }_{u,k,v}\le {\lambda }_{u,k}$$

(30)

where L is a relatively large value. The optimization problem (23) aims to minimize the total AoI. Therefore, it is a cooperative problem for all fog servers. Let the reward functions of all fog servers be the same, i.e., ${\mathit{r}}_1={\mathit{r}}_2=\cdots ={\mathit{r}}_U$. The total reward of all fog servers in the system is defined by

$$\mathit{r}=({\mathit{r}}_1,{\mathit{r}}_2,\cdots ,{\mathit{r}}_U).$$

(31)

4.2. MATD3-Based Cache and Power Allocation Algorithm

The MATD3 algorithm is a multi-agent extension of the TD3 algorithm, employing centralized training with a decentralized execution (CTDE) framework. It inherits TD3’s enhancements that mitigate DDPG’s Q-value overestimation bias via clipped double-Q learning, target policy smoothing, and delayed policy updates. Compared to the DDPG algorithm, the MATD3 algorithm converges faster and achieves more stable and higher rewards.

The framework of the power and storage allocation algorithm based on MATD3 is shown in Figure 3. Each fog server’s digital mirror acts as an agent, yielding U agents. They interact with the virtual environment in the digital twin during training to learn optimal strategies. Agent u comprises an actor network ${\mu }_{{\theta }_u}$ (parameterized by ${\theta }_u$) and two critic networks $Q_{{\varphi }_{u,1}},Q_{{\varphi }_{u,2}}$ (parameterized by ${\varphi }_{u,1},{\varphi }_{u,2}$), with corresponding target networks ${\mu }_{{\theta }_u}^{\prime },Q_{{\varphi }_{u,1}}^{\prime },Q_{{\varphi }_{u,2}}^{\prime }$. An experience replay buffer $\mathcal{D}$ stores tuples $(\mathit{s},\mathit{a},\mathit{r},{\mathit{s}}^{\prime })$, and mini-batches are sampled to decorrelate sequential dependencies for training stability.

The agents cooperate to achieve global optimization through environment information sharing and local policy update. During training of the CTDE paradigm, critic networks leverage global state and joint actions of all agents to estimate the Q-value, and the actor network is updated using gradients computed from critic networks. At execution time, each agent learns a local actor policy based on its own state and the evaluation from critic networks. The dual-critic networks are employed to mitigate overestimation bias, and the unified reward with a global goal is taken to transform the independent decision to coordination.

The training mechanisms include the critic update and actor update. The critic networks $Q_{u,k}$, $\forall k\in \{1,2\}$, are optimized by minimizing the loss function

$$\mathcal{L}\left({\varphi }_{u,k}\right)=\mathbb{E}\left[{\left(y-Q_{u,k}\left(\mathit{s},{\mathit{a}}_1,\dots ,{\mathit{a}}_U\right)\right)}^2\right],$$

(32)

where $Q_{u,k}(\mathit{s},{\mathit{a}}_1,\dots ,{\mathit{a}}_U)$ represents the Q-value estimate for the state–action pair in the k-th critic network of the u-th agent. With the definition of $\gamma$ as the discount factor, y is expressed as

$$\begin{array}{cc}\hfill y& =r_u+\gamma \underset{k=1,2}{min}{Q}_{u,k}^{\prime }\left({\mathit{s}}^{\prime },{\tilde{\mathit{a}}}_1,\dots ,{\tilde{\mathit{a}}}_U\right).\hfill \end{array}$$

(33)

The action space in (33) is obtained by

$$\begin{array}{cc}\hfill {\tilde{\mathit{a}}}_u& =\mathrm{clip}\left({\mu }_u^{\prime }\left({\mathit{s}}_u^{\prime }\right)+\mathrm{clip}\left(\tilde{\epsilon },-h,h\right),a_{\mathrm{low}},a_{\mathrm{high}}\right),\hfill \end{array}$$

(34)

where $a_{\mathrm{low}}$ and $a_{\mathrm{high}}$ denote the lower and upper bounds of the action, respectively. $-h$ and h represent the lower and upper bounds of the noise $\tilde{\epsilon }$, $\tilde{\epsilon }\sim \mathcal{N}(0,{\sigma }^2)$.

The cumulative reward of agent u is expressed by $J\left({\theta }_u\right)=\mathbb{E}\left[{\sum }_{t=0}^{\infty }{\gamma }^t{r}_u^t\right]$. The actor network $\mu$ is updated via a policy gradient to maximize the expected cumulative reward by

$$\begin{array}{c}\hfill {\nabla }_{{\theta }_u}J\left({\theta }_u\right)=\mathbb{E}{\displaystyle \left[{\nabla }_{{\theta }_u}{\mu }_u\left({\mathit{s}}_u\right){\nabla }_{{\mathit{a}}_u}{Q}_{u,1}\left(\mathit{s},{\mathit{a}}_1,\dots ,{\mathit{a}}_U\right){|}_{{\mathit{a}}_u={\mu }_u\left({\mathit{s}}_u\right)}\right]}.\end{array}$$

(35)

Assume that M denotes the number of training iteration and F represents the training step number. The centralized training procedure is presented in Algorithm 1, where d indicates the update frequency of the actor network and $\epsilon$ represents the action noise with $\epsilon \sim \mathcal{N}(0,{\sigma }^2)$. Algorithm 2 illustrates the distributed execution process after centralized training. In this algorithm, each agent only needs to use its actor network, and each agent u can quickly obtain the optimal action ${\mathit{a}}_u$ solely based on its observed state ${\mathit{s}}_u$.

Algorithm 1 Centralized training for cache and power allocation based on MATD3

1: for $m=1$ to M do   2:       Initialize the environment and obtain the initial state $\mathit{s}$;   3:       for $f=1$ to F do   4:            Each agent u selects an action: ${\mathit{a}}_u={\mu }_{{\theta }_u}\left({\mathit{s}}_u\right)+\epsilon$;   5:            Execute $\mathit{a}=({\mathit{a}}_1,\dots ,{\mathit{a}}_U)$, then observe reward r and next state ${\mathit{s}}^{\prime }$;   6:            Store $(\mathit{s},\mathit{a},r,{\mathit{s}}^{\prime })$ into replay buffer $\mathcal{D}$;   7:            Update state: $\mathit{s}\leftarrow {\mathit{s}}^{\prime }$;   8:            for $u=1$ to U do   9:                  Randomly sample a mini-batch of S samples from $\mathcal{D}$; 10:                  Update critic networks $Q_{u,1}$ and $Q_{u,2}$ by minimizing (32); 11:                  if $fmodd=0$ then 12:                       Update actor network ${\mu }_u$ via the policy gradient in (33); 13:                       Update target network parameters: 14:                       ${\theta }_u^{\prime }\leftarrow \tau {\theta }_u+(1-\tau ){\theta }_u^{\prime }$; 15:                       ${\varphi }_{u,k}^{\prime }\leftarrow \tau {\varphi }_{u,k}+(1-\tau ){\varphi }_{u,k}^{\prime }$. 16:                 end if 17:             end for 18:        end for 19: end for

Algorithm 2 Distributed execution for cache and power allocation based on MATD3

1: Each agent loads the trained actor network ${\mu }_1,\dots ,{\mu }_U$;   2: for $n=1$ to N do   3:       for $f=1$ to F do   4:            ${\mathit{a}}_u={\mu }_u\left({\mathit{s}}_u\right)$;   5:       end for   6:       Execute the joint action $\mathit{a}=({\mathit{a}}_1,\dots ,{\mathit{a}}_U)$ and obtain the next state ${\mathit{s}}^{\prime }$.   7: end for

4.2.1. Computational Complexity Analysis

In Algorithm 1, the core computations originate from the updates of the actor and critic networks. Let $L_a$ and $L_c$ denote the number of layers in the actor network and the critic network, respectively. Let $N_{l_a}^a$, $\forall l_a\in \{1,2,\dots ,L_a\}$, and $N_{l_c}^c$, $\forall l_c\in \{1,2,\dots ,L_c\}$, represent the number of neurons in the $l_a$-th layer of the actor network and the $l_c$-th layer of the critic network. The computational complexity of the $l_a$-th layer in the actor network is $\mathcal{O}\left(N_{l_a-1}^a{N}_{l_a}^a\right)$. Thus, the total computational complexity of the entire actor network is $\mathcal{O}\left({\sum }_{l_c=2}^{L_c}{N}_{l_c-1}^c{N}_{l_c}^c\right)$. Similarly, the total computational complexity of the critic network is $\mathcal{O}\left({\sum }_{l_c=2}^{L_c}{N}_{l_c-1}^c{N}_{l_c}^c\right)$. The overall computational complexity of Algorithm 1 obeys $\mathcal{O}\left(M·F·U\left[{\displaystyle \frac{1}{d}}{\sum }_{l_a=2}^{L_a}{N}_{l_a-1}^a{N}_{l_a}^a+{\sum }_{l_c=2}^{L_c}{N}_{l_c-1}^c{N}_{l_c}^c\right]\right)$. Since only the actor networks are applied in Algorithm 2, its total computational complexity amounts to $\mathcal{O}\left(F·U\left({\sum }_{l_a=2}^{L_a}{N}_{l_a-1}^a{N}_{l_a}^a\right)\right)$.

4.2.2. Scalability and Response Capability Analysis

The MATD3 model demonstrates excellent scalability through its distributed architecture. This framework decomposes the centralized solution into a collaborative solving process involving multiple distributed agents. Each agent generates decisions through a policy network based on local state information, which not only reduces computational complexity but also supports parallel computation among agents. Coupled with environmental information sharing within the digital twin system, the design effectively balances global coordination and local execution so as to adapt to the large city-wide deployment of vehicular networks.

Moreover, the MATD3 algorithm ensures a real-time response capability through the cooperation mechanism. Due to the adoption of CTDE architecture, each agent generates action independently through a lightweight actor network based on observed state and asynchronous parameter updates, avoiding the large communication overhead of centralized decision-making. Furthermore, the adoption of dual-critic networks effectively mitigate the Q-value overestimation, suppressing the value function fluctuation and significantly accelerating the policy optimization process.

5. Simulation Results

5.1. Simulation Scenario Setup

In the simulation, the scenario shown in Figure 1 is set up. First, the RSUs and vehicles are configured to be randomly and uniformly distributed in the area. Then, an IoV model based on a realistic road structure and vehicle mobility is constructed with non-uniform deployment of RSUs. Referring to [31,32], a road network covering a square area of 2 km is built with urban traffic characteristics, dynamic vehicular flow, and varying vehicle density. The traffic density ranges from 30 to 60 vehicles per ${\mathrm{km}}^2$, with vehicle speed varying from 20 km/h to 60 km/h. Non-uniform RSU deployment is achieved through variance spacing with 200~300 m for high-density segments and 500~800 m for low-density segments. During the training process of the algorithms, each episode consists of 200 steps. Both the actor network and the critic network contain three fully connected layers and two hidden layers, each having 256 neurons. All linear rectification functions are used as the activation function for all hidden layers, and all neural networks are updated using the adaptive moment estimation optimizer. The warm-up size of the experience replay buffer $\mathcal{D}$ is 25,600. Other parameter settings are summarized in

Table 1. Simulation parameter settings.

Parameter	Symbol	Value
Power of additive white Gaussian noise	${\sigma }^2$	$-100\mathrm{dBm}$
System bandwidth	W	$8\mathrm{MHz}$
Minimal required rate	$R^{min}$	$2\mathrm{Mbit}/\mathrm{s}$
Maximal transmission power	$P_{u,k}^{max}$	$1\mathrm{W}$
Discount factor	$\gamma$	$0.95$
Soft-update coefficient	$\tau$	$0.01$
Size of experience replay buffer	D	100,000
Mini-batch size	S	1024
Update frequency of actor network	d	2
Path loss reference distance	$d_0$	$1000\mathrm{m}$
Path loss exponent	$\eta$	$3.67$
Shadowing fading standard deviation	${\sigma }_s^2$	$5\mathrm{dB}$
Frequency-selective Rayleigh fading parameter	${\sigma }_f^2$	$0.5$
Bound of noise	h	$0.5$
Reward parameters	${\phi }_1,{\phi }_2,{\phi }_3,L$	${\phi }_1=100,{\phi }_2=10,{\phi }_3=1,L=10\mathrm{s}$

.

5.2. Analysis of Simulation Results

We analyze the convergence performance of the MATD3 and DDPG algorithms in Figure 4. In the simulation scenario, there are 12 RSUs, 120 vehicles, and 60 channels, and the number of information types is 10. The common hype parameters of the two algorithms are set to be the same, and the learning rate of the critic network is 0.01 and 0.005, respectively. It can be seen from the figure that the rewards obtained by the algorithms generally increase with the episode number. When the learning rate of the critic network is 0.01, the MATD3 algorithm converges after about 370 episodes of training, while the DDPG algorithm requires about 650 episodes of training to converge. When the learning rate of the critic network is 0.005, the MATD3 algorithm converges after about 600 episodes of training, while the DDPG algorithm requires about 750 episodes of training to converge. Thus, it can be obtained that the MATD3 algorithm converges faster. The figure also shows that the convergence curve of the MATD3 algorithm has less fluctuation and achieves a higher final reward.

Figure 5 illustrates the trend of the total AoI in the system, which is obtained by the MATD3 and DDPG algorithms and in relation to an increasing channel gain prediction deviation of the digital twin. In this simulation, each vehicle occupies a distinct channel, and the communication between vehicles and RSUs is free from interference by other communication links. It can be seen from the figure that the total AoI decreases with increasing deviation. This is because $D_{u,k,v}=0$ in (13) in the absence of interference, and the value of (13) changes as $\Delta h_{u,k,v,n_{u,k,v}}$ increases. Consequently, the download rate of the vehicle increases, which reduces both the information download time and the queuing time for information transmission, thereby decreasing the AoI. In addition, Figure 5 demonstrates that the proposed MATD3 algorithm achieves superior AoI performance compared to the DDPG algorithm in this scenario. Furthermore, as the number of vehicles increases, the performance gap between the two algorithms widens. The MATD3 algorithm maintains stable AoI performance, while the DDPG algorithm exhibits a more significant degradation in AoI.

The total AoI of the system achieved by the MATD3 algorithm and that by the DDPG and greedy algorithms are shown in Figure 6, in relation to different vehicle numbers versus the number of system channels. For the greedy algorithm, the RSUs select the maximal transmission power, while a random strategy is adopted for information cache allocation. It is observed that the total AoI decreases as the number of system channels increases.

Figure 7 depicts the total AoI variation versus the number of vehicles with a channel number of 100, revealing that the total AoI rises alongside both the growing count of vehicles and the increasing number of information type. This occurs because a greater number of information type leads to more queued information, which extends the queuing waiting time. Moreover, it places a heavier download load on vehicles, lengthening the overall download time and thus driving up the total AoI.

Both Figure 6 and Figure 7 demonstrate that the MATD3 algorithm exhibits better AoI performance than the DDPG and greedy algorithms. Since the greedy algorithm not only selects the total maximal transmission power of RSUs, increasing communication interference, but also uses random information storage allocation, a worse storage location is obtained for certain information types. This reduces the information transmission rate and increases both transmission delay and queuing waiting time. In contrast, the MATD3 algorithm obtains transmission power and information cache allocation through training so as to reduce the queuing time and transmission delay.

We compare AoI performance under the uniform and non-uniform deployment of RSUs in Figure 8. It can be seen that the AoI is smaller under non-uniform RSU distribution than uniform RSU distribution. This is because the non-uniform configuration adjusts the distribution interval according to the vehicle density so as to reduce the AoI of the vehicles in a high-density area. Moreover, the AoI decreases with the RSU number because the distance from the vehicle to the RSU becomes smaller with increasing RSU configuration.

6. Conclusions

We established a vehicular network model where vehicles obtained information through paired RSUs and the resource allocation of RSUs was controlled through a fog server and digital twin system. An optimization problem for information cache and communication power was formulated with the aim to minimize the AoI of the system. Subsequently, a MATD3-based algorithm was proposed to solve this problem. Finally, simulation results showed that the proposed method improved the system AoI performance compared with baseline approaches. Additionally, they revealed that the total system AoI increased with the number of vehicles, channels, and information types, which gave significant guidance on network construction and configuration.