Research on Optimization of RIS-Assisted Air-Ground Communication System Based on Reinforcement Learning

Abstract

In urban emergency communication scenarios, building obstructions can reduce the performance of base station (BS) communication networks. To address such issues, this paper proposes an air-ground wireless network enabled by an unmanned aerial vehicle (UAV) and assisted by reconfigurable intelligent surfaces (RIS). This system enhances the efficacy of UAV-enabled MISO networks. Treating the UAV as an intelligent agent moving in 3D space, sensing changes in the channel environment, and adopting zero-forcing (ZF) precoding to eliminate interference from ground users. Meanwhile, joint design is performed for UAV movement, RIS phase shifts, and power allocation for users. We propose two deep reinforcement learning (DRL) algorithms, which are termed D3QN-WF and DDQN-WF, respectively. Simulation results indicate that D3QN-WF achieves a 15.9% higher sum rate and 50.1% greater throughput than the DDQN-WF baseline, while also demonstrating significantly faster convergence.

1. Introduction

One of the visions of sixth Generation (6G) is to realize the evolution from the Internet of Everything to the Intelligent Internet of Everything on the basis of Fifth Generation (5G), which poses great challenges to traditional terrestrial cellular networks [1]. Compared with conventional terrestrial wireless communication, Unmanned Aerial Vehicle (UAV) communication has significant advantages, such as high mobility, uninterrupted line-of-sight connection, and strong perception ability. The wireless network empowered by UAV is one of the key technologies to improve communication quality [2,3,4]. Meanwhile, UAVs can be equipped with sensing and ranging modules to accurately measure the distance between obstacles and ground equipment, ensuring flight safety, providing a sensing range, and delivering high-quality communication services [5,6].

The upcoming sixth generation (6G) driven Internet-of-Things (IoT) will face the great challenges of extremely low power demand, high transmission reliability, massive connectivities, and physical layer security [7,8,9]. To effectively address the problems of traditional communication technologies being limited by hardware costs and channel fading, Reconfigurable Intelligent Surface (RIS) has become one of the potential applications as a feasible solution to various challenges in future wireless networks. The breakthrough of RIS technology lies in transforming the communication environment into a programmable medium. Its two-dimensional metamaterial surface reconstructs electromagnetic field distribution via dynamic phase adjustment. It also dynamically regulates electromagnetic waves in a programmable manner [10]. Studies have confirmed that compared with traditional AF relays, RIS can provide substantial energy efficiency improvements for wireless networks [11,12,13]. The characteristics of RIS with low cost and UAV with high mobility have the advantages of low power consumption and high spectral efficiency. They have become important components of future communication networks. When RIS is deployed on buildings or UAV platforms, it can regulate reflection paths, change the radio environment in three-dimensional space, and achieve signal enhancement [14,15]. Moreover, studies have shown that the system network composed of RIS and UAV can obtain a more flexible network structure and a higher communication rate [16,17]. However, traditional studies have focused on the offline optimization of RIS static reflection parameters and the optimization of UAV hovering positions. With the evolution of future wireless networks, RIS-UAV communication systems need to have the ability to actively sense and reconstruct the environment.

Future communication networks will integrate various emerging technologies, which will increase the complexity of optimizing throughput and energy efficiency in the network. This complexity will bring about challenges where traditional algorithms are challenged and struggle to find optimal solutions or prove infeasible. Machine learning (ML) is widely used as a powerful tool to enhance the performance of wireless networks, especially for large-scale networks, enabling them to conduct effective optimization in dynamic environments. Due to the rapid development of ML, Deep Reinforcement Learning (DRL), as a branch in this field, provides an alternative approach to solving complex optimization problems [18,19]. DRL is a combination of neural networks and Reinforcement Learning (RL). It realizes algorithm design by using rewards provided in the environment as output values. Therefore, UAVs can be regarded as agents that acquire optimal strategies through learning from feedback, which is obtained through continuous interaction with the environment [20,21]. Therefore, the vision of future 6G and its network architecture aims to establish an intelligent communication ecosystem. By deeply integrating RIS-UAV technology with DRL, a high-efficiency air-ground integrated wireless communication network can be built to meet the needs of future intelligent communications.

1.1. Related Work

In recent years, research on RIS and UAV-assisted communication technologies can be mainly divided into two types: one type deploys RIS directly on UAVs as payloads to form RIS-UAV systems. The air-ground network with RIS-UAV-assisted base stations (BS) is introduced in detail in references [22,23,24,25,26]; such RIS-UAVs act as mobile airborne relays. Although the RIS-UAV scheme achieves blind area coverage, it faces significant challenges in terms of communication quality in urban emergency communications and places with high population density. The other type deploys RIS on ground buildings and uses UAVs as aerial BS. This configuration can provide ground users with better information transmission or data collection services. In comparison, through flexible deployment, the RIS-assisted BS-UAV significantly reduces the pressure on ground BS and improves the communication quality in densely populated areas. Therefore, this paper adopts the RIS-assisted BS-UAV system model to improve the performance of the system network.

Compared to traditional ground BS, UAVs are more convenient to deploy. Liu et al. utilized RIS-assisted BS-UAV and adopted a DRL approach to optimize the 2D coordinates of UAVs and reduce UAV energy consumption. This overcomes the limitation of traditional algorithms that struggle to find or cannot obtain optimal solutions in high-dimensional convex optimization [27]. Ahmad et al. adopted the prioritized experience replay method to optimize phase angles, so as to improve users’ satisfaction with the service quality of communication systems [28]. Hu et al. utilized the Double Deep Q-Network (DDQN) to optimize the 3D trajectory of UAVs, aiming to maximize the system data throughput [29]. Mei et al. proposed a method that does not require pre-prepared training data; instead, it uses environmental modeling to provide reward feedback to BS-UAVs, to optimize the UAV trajectory, and improve the communication rate [30]. Khalili et al. in [31] optimized UAV trajectory and subcarrier allocation with the help of Dueling DQN to improve the performance of heterogeneous networks supported by dual connectivity. Regarding the complex optimization problems in RIS and UAV-assisted air-ground wireless communication networks, Nguyen et al. summarized the existing DRL-based solving methods and pointed out the challenges faced by future research [32].

In the aforementioned research works, reference [25] optimize the coordinates of UAVs in a two-dimensional space, without consideration given to three-dimensional coordinate optimization. As a result, the advantages of UAVs, such as high mobility, have not been fully demonstrated. In the literature [26], although the problem of RIS phase shift optimization has been addressed, the potential of RIS has not been fully exploited by leveraging DRL. In one study [27,28], although DQN and DDQN are used to solve the problems of RIS phase shift and UAV trajectory, the optimized coordinates are still limited to two-dimensional spatial coordinates. In another work [29,30,31], although the DRL method is adopted, power discretization or the failure to consider power allocation will lead to an overall decline in network performance. However, traditional DRL algorithms with discrete action spaces have significant limitations: the value overestimation phenomenon in DQN and Dueling DQN can cause decision biases in agents when channel states change abruptly, while the Q-value oscillation problem in DDQN restricts the stability of UAV trajectory planning. To address these bottlenecks in agent actions, the Dueling Double Deep Q-Network (D3QN) has achieved breakthroughs through innovative architectural integration. Its core idea is to combine the decomposed structure of state value and advantage value from Dueling DQN with the dual-network structure of DDQN. This enables it to accurately estimate the Q-values of all actions of the agent, effectively mitigating the overestimation problem, and boasts the advantages of fast convergence speed and stable convergence effect.

Therefore, the deep reinforcement learning framework is integrated into the RIS-assisted UAV communication model to construct collaborative optimization under a multi-dimensional action space. The sum-rate expression for multiple users under this model is established, and an optimization variable algorithm based on the D3QN framework (Dueling Double Deep Q-Network water-filling algorithm, D3QN-WF) is proposed. RIS changes its phase to reconstruct the channel state; UAVs perceive environmental changes and adjust their 3D coordinates through decision-making; and by optimizing the transmission power of UAVs, the system sum-rate is maximized, and the throughput during communication is improved.

1.2. Contributions

• A joint optimization framework for RIS-assisted UAV air-to-ground wireless communication networks under 3D spatial coordinates is proposed. This framework significantly improves the network system sum-rate and throughput during communication by collaboratively optimizing the 3D spatial coordinates of UAVs, the RIS phase shift matrix, and the UAV transmission power, thus giving full play to the synergistic potential of RIS and UAVs;
• A joint optimization method combining BS transmission power optimization based on the water-filling algorithm and D3QN. Aiming at the convexity problem of BS transmission power optimization, the water-filling algorithm is adopted for an efficient solution; meanwhile, the D3QN algorithm is innovatively used to solve the optimization problem in the discrete action space. It jointly optimizes the 3D coordinates of UAVs and RIS phases, effectively overcoming the limitations of traditional methods in high-dimensional non-convex optimization problems;
• Detailed verification results are provided to demonstrate the effectiveness of the proposed algorithm in improving the system sum-rate and throughput. Simulation results show that the proposed algorithm has significant advantages in rate improvement. In addition, compared with the DDQN-WF, the D3QN-WF algorithm shows obvious advantages in handling multi-dimensional action spaces, with faster convergence and higher stability. This method increases the system sum-rate by 15.9% and the throughput by 50.1%, providing new ideas for the dynamic optimization of future intelligent communication networks.

2. System Model

Figure 1 depicts a RIS-assisted BS-UAV air-to-ground intelligent communication network system. An RIS with N reflecting elements is deployed on a high-rise building to facilitate the reconstruction of the channel environment. The UAV equipped with M antennas is regarded as BS, which communicates with single-antenna users, and it has a sensing antenna at its bottom to sense ground users, with a maximum sensing elevation angle of $\omega$. By perceiving the environment, associating with users, and collecting real-time channel state information (CSI), the UAV makes adjustments to its 3D position. When a UAV detects users, there are direct links and cascaded links between the UAV and the users. The direct link includes both line-of-sight (LoS) and non-line-of-sight (NLoS) links. For the cascaded link, only LoS link exists between the UAV and the RIS, while both LoS and NLoS links exist between the RIS and the users. When the users are outside the detection range of the UAV, the RIS is used to cover the blind area. In this case, only the cascaded link exists between the UAV and the ground users.

It is assumed that the system’s Area of Interest (AoI) is discretized into multiple cells of equal size, and the coordinates of the center of cell i can be expressed as $L_i^c=[x_i,y_i,z_i]\in {\mathbb{R}}^{3\times 1}$. The variables $x_b$,$y_b$, and $z_b$ represent the distances between adjacent cells along the x-axis, y-axis, and z-axis, respectively. The total time slots of the system can be expressed as T, and the three-dimensional coordinates of the UAV at time slot t can be expressed as $Q_t=[x_t,y_t,z_t]\in L^c$, where $t\in \{1,2,\dots ,\mathit{T}\}$. Ground users move within a small range and are evenly distributed on one side of the RIS. Their coordinates can be expressed as ($x_k^t$, $y_k^t$, $z_k^t$). In communication networks under multi-target scenarios, the constraint on the UAV’s sensing range is a key factor to be considered in practice. In each time slot, the UAV needs to select targets within its coverage area from all potential IoT devices according to its current flight path for data transmission operations. A sensing communication variable $a_k$ is defined, and its sensing coverage is related to the height and angle of the UAV.

$$a_k=\left\{\begin{array}{cc}1\hfill & R\le Htan\omega \hfill \\ 0\hfill & R>Htan\omega \hfill \end{array}\right.$$

(1)

where R is the horizontal distance between the UAV and the user, $\omega$ represents the sensing angle and H represents the height of the UAV. When $a_k=1$, the user is within the sensing range.

2.1. Channel Model

The UAV acting as a BS moves gradually toward the users. When the user is within the UAV’s sensing range, the channel gain of the direct link between the aerial BS and the ground user k can be expressed as ${\mathbf{h}}_{u,k}^H\in {\mathbb{C}}^{1\times M}$ which can be modeled as a Rician channel, expressed as:

$${\mathbf{h}}_{u,k}^H=\sqrt{\beta d_{u,k}^{-\alpha }}\left(\sqrt{{\displaystyle \frac{\widehat{R}}{1+\widehat{R}}}}{\mathbf{h}}_{u,k}^{Lo{S}^H}+\sqrt{{\displaystyle \frac{1}{1+\widehat{R}}}}{\mathbf{h}}_{u,k}^{NLo{S}^H}\right)$$

(2)

where ${\mathbf{h}}_{u,k}^{Lo{S}^H}$ and ${\mathbf{h}}_{u,k}^{NLo{S}^H}$ are used to represent the fast fading component for LoS path and NLoS path between the UAV and the user k, respectively. $\widehat{R}$ is the Rician factor. The path loss between the BS and k-th user is given by $\sqrt{\beta d_{u,k}^{-\alpha }}$, where $\beta$ denotes the channel gain at a 1 m reference distance, $\alpha \ge 2$ is the path loss exponent, and $d_{u,k}$ is the distance between the UAV and user k. For the cascaded channel, there are two links: the UAV-RIS link and the RIS-User link. In the UAV-RIS link, only the LoS link ${\mathbf{H}}_{u,r}^H\in {\mathbb{C}}^{N\times M}$ exists, which can be expressed as:

$${\mathbf{H}}_{u,r}^H=\sqrt{\beta d_{u,r}^{-\alpha }}\sqrt{{\displaystyle \frac{\widehat{R}}{1+\widehat{R}}}}{\mathbf{H}}_{u,r}^{Lo{S}^H}$$

(3)

${\mathbf{H}}_{u,r}^{Lo{S}^H}\in {\mathbb{C}}^{N\times M}$ is the fast fading component of the LoS channel between UAV and RIS; $d_{u,r}$ is the Euclidean distance. Secondly, both LoS and NLoS propagations exist in the RIS-User link. Therefore, it is also modeled using a Rician channel, denoted ${\mathbf{h}}_{r,k}^H\in {\mathbb{R}}^{1\times N}$. The channel gain of the RIS-User link is as follows:

$${\mathbf{h}}_{r,k}^H=\sqrt{\beta d_{r,k}^{-\alpha }}\left(\sqrt{{\displaystyle \frac{\widehat{R}}{1+\widehat{R}}}}{\mathbf{h}}_{r,k}^{Lo{S}^H}+\sqrt{{\displaystyle \frac{1}{1+\widehat{R}}}}{\mathbf{h}}_{r,k}^{NLo{S}^H}\right)$$

(4)

2.2. Downlink Signal Transmission Modeling and Optimization

Consider that the UAV transmits linear preambles, and the transmitted signal is:

$$\mathbf{x}=\sum _{k=1}^K\sqrt{p_k}{\mathbf{w}}_k{s}_k$$

(5)

where $p_k$, $k\in K$ is the transmit power allocated by the UAV to user k, $s_k$ is the unit-power complex transmit symbol, and ${\mathbf{w}}_k\in {\mathbb{C}}^{M\times 1}$ is the precoding direction vector of the k-th user. When the UAV can detect the user, the communication quality is improved through the direct link. When the user is not within the UAV’s sensing range, the blind area is compensated through the cascaded link of the RIS. Therefore, the signal received by the k-th user can be expressed as:

$$y_k=\left(a_k{\mathbf{h}}_{u,k}^H+{\mathbf{h}}_{r,k}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H\right)\sum _{i=1}^K\sqrt{p_i}{\mathbf{w}}_i{s}_i+{\sigma }^2$$

(6)

where the RIS phase shift matrix is denoted by $\mathbf{\Phi }\triangleq diag[{\varphi }_1,{\varphi }_2,\dots ,{\varphi }_N]$. For each element in the diagonal matrix $\mathbf{\Phi }$, that is, the phase angle of the n-th reflecting element is denoted as ${\varphi }_n=e^{j{\theta }_n},\forall n=1,2,\dots ,N$,${\theta }_{\mathrm{n}}\in \left[0,2\pi \right]$. Where ${\sigma }^2\sim \mathcal{C}\mathcal{N}(0,{\sigma }^2)$ is the additive white Gaussian noise. According to Formula (6), the received signal interference noise ratio (SINR) at the k-th user in the system can be expressed as:

$${\widehat{\gamma }}_k\triangleq {\displaystyle \frac{p_k{\left|\left(a_k{\mathbf{h}}_{u,k}^H+{\mathbf{h}}_{r,k}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H\right){\mathbf{w}}_k\right|}^2}{{\displaystyle \sum _{i=1,i\ne k}^K}{p}_i{\left|\left(a_i{\mathbf{h}}_{u,i}^H+{\mathbf{h}}_{r,i}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H\right){\mathbf{w}}_i\right|}^2+{\sigma }^2}}$$

(7)

where ${\displaystyle \sum _{i=1,i\ne k}^K}{p}_i{\left|\left(a_i{\mathbf{h}}_{u,i}^H+{\mathbf{h}}_{r,i}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H\right){\mathbf{w}}_i\right|}^2$ is the interference signal power received by user k. The UAV departs from the starting point, plans its trajectory, and searches for the optimal deployment position that maximizes the system communication performance. For the sake of fairness, each user is allocated a bandwidth of B. Therefore, the objective function for optimizing the system sum rate under the UAV’s deployment position can be expressed as:

$$\begin{array}{}\hfill \mathrm{(8)}& \hfill P_1:& \underset{\mathbf{P},\mathbf{\Phi },\mathbf{Q}}{max}R\triangleq \sum _{k=1}^{K}B{log}_2\left(1+{\widehat{\gamma }}_k\right)\hfill \hfill \mathrm{(8a)}& \hfill s.t.& a_k\in \left\{0,1\right\}\hfill \hfill \mathrm{(8b)}& & \left|e^{j{\theta }_n}\right|=1,\forall n=1,2,\dots ,N\hfill \hfill \mathrm{(8c)}& & \sum _{k=1}^K{p}_k=P\hfill \hfill \mathrm{(8d)}& & p_k\ge 0,\forall k=1,2,\dots ,K\hfill \hfill \mathrm{(8e)}& & x_{min}\le x_t\le x_{max},\forall t=1,2,\dots ,T\hfill \hfill \mathrm{(8f)}& & y_{min}\le y_t\le y_{max},\forall t=1,2,\dots ,T\hfill \hfill \mathrm{(8g)}& & z_{min}\le z_t\le z_{max},\forall t=1,2,\dots ,T\hfill \hfill \mathrm{(8h)}& & \parallel x_{t+1}-x_t{\parallel }_2\le x_b,\forall t=1,2,\dots ,T-1\hfill \hfill \mathrm{(8i)}& & \parallel y_{t+1}-y_t{\parallel }_2\le y_b,\forall t=1,2,\dots ,T-1\hfill \hfill \mathrm{(8j)}& & \parallel z_{t+1}-z_t{\parallel }_2\le z_b,\forall t=1,2,\dots ,T-1\hfill \end{array}$$

where the constraint (8a) represents the sensing state of the direct link between the UAV and the k-th user; (8b) ensures that the RIS reflection unit only changes the phase of the signal without altering its amplitude; (8c) ensures that the transmit power of the BS after encoding is equal to P; (8d) ensures that when optimizing power, the user’s power is not lower than the minimum value of 0; (8e) to (8g) limit the 3D coordinates of the UAV’s flight to prevent it from flying out of the boundary; (8h) to (8j) ensure the displacement limit of the UAV in each time slot. Since the optimization of the UAV trajectory coordinates $\mathbf{Q}$ and the RIS phase matrix $\theta$ is covered in Section 3, this subsection only discusses in detail the user’s power $\mathbf{P}$ in Formula (8); in order to suppress channel interference in multi-user scenarios, the zero-forcing (ZF) coding algorithm in linear coding is adopted to eliminate signal interference between users [27]:

$$\left(a_i{\mathbf{h}}_{u,i}^H+{\mathbf{h}}_{r,i}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H\right){\mathbf{w}}_k=0,\forall i\ne k,i\in K.$$

(9)

${\mathbf{h}}_k^H=a_k{\mathbf{h}}_{u,k}^H+{\mathbf{h}}_{r,k}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H$ is used to represent the direct and cascaded channel gains of user k, where ${\mathbf{h}}_k^H\in {\mathbb{C}}^{1\times M}$. Under this condition, the global channel matrix can be written as ${\mathbf{H}}^H={({\mathbf{h}}_1,\dots ,{\mathbf{h}}_k)}^H$, where $\mathbf{H}\in {\mathbb{C}}^{M\times K}$. Therefore, the transmit precoding matrix can be expressed as $\mathbf{W}=\left({\mathbf{w}}_1,\dots ,{\mathbf{w}}_k\right)$, where $\mathbf{W}\in {\mathbb{C}}^{M\times K}$. Under the condition of (9), the zero-force(ZF) precoding matrix can be solved through ${\mathbf{H}}^H\mathbf{W}=\mathbf{E}$, and $\mathbf{W}$ can be expressed as:

$$\mathbf{W}=\mathbf{H}{\left({\mathbf{H}}^H\mathbf{H}\right)}^{-1}$$

(10)

$\mathbf{W}$ is the right pseudoinverse of ${\mathbf{H}}^H$. After ZF precoding, there is no interference between users. To solve the optimal power allocation, the beam vector ${\mathbf{w}}_k$ of the k-th user is normalized in the transmit precoding matrix $\mathbf{W}$.

$${\mathbf{w}}_k^{*}={\displaystyle \frac{{\mathbf{w}}_k}{{∥{\mathbf{w}}_k∥}^2}}$$

(11)

according to Formula (11), ${\mathbf{w}}_k^{*}$ is the beam direction of user k, and $p_k$ is the transmit power allocated to user k, therefore ${\displaystyle \sum _{k=1}^K}{∥\sqrt{p_k}{\mathbf{w}}_k^{*}∥}^2=P$. Then, the SINR of the k-th user is ${\gamma }_k={\displaystyle \frac{p_k{\left|\left(a_k{\mathbf{h}}_{u,k}^H+{\mathbf{h}}_{r,k}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H\right){\mathbf{w}}_k^{*}\right|}^2}{{\sigma }^2}}$ Since the bandwidth allocated among users is the same, the power allocation part can further equate the optimization problem to:

$$\begin{array}{}\hfill \mathrm{(12)}& \hfill {\mathrm{P}}_2:& -\sum _{k=1}^{K}ln\left(1+{\displaystyle \frac{p_k{\xi }_k}{{\sigma }^2}}\right)\hfill \hfill \mathrm{(12a)}& \hfill s.t.& \sum _{k=1}^K{p}_k=P\hfill \hfill \mathrm{(12b)}& & p_k\ge 0,\forall k=1,2,\dots ,K\hfill \end{array}$$

where ${\xi }_k={\left|\left(a_k{\mathbf{h}}_{u,k}^H+{\mathbf{h}}_{r,k}^H\mathbf{\Phi }{\mathbf{H}}_{u,r}^H\right){\mathbf{w}}_k^{*}\right|}^2$, this problem can be solved using the water-filling algorithm to allocate the transmit power to each user, and its Lagrangian function is:

$$L\left(p_k,v_k,\mu \right)=-\sum _{k=1}^{K}ln\left(1+{\displaystyle \frac{P_k{\xi }_k}{{\sigma }^2}}\right)-\sum _k^K{v}_k{p}_k+\mu \left(\sum _{k=1}^K{p}_k-P\right)$$

(13)

$v_k$ is the Lagrange multiplier corresponding to the inequality constraint $p_k\ge 0$, and $\mu$ is the Lagrange multiplier corresponding to the equality ${\displaystyle \sum _{k=1}^K}{p}_k=P$. The KKT necessary conditions for this optimization problem can be expressed as:

$$\begin{array}{}\hfill \mathrm{(13a)}& p_k^{*}\ge 0\hfill \hfill \mathrm{(13b)}& \sum _{k=1}^K{p}_k^{*}=P\hfill \hfill \mathrm{(13c)}& v_k^{*}\ge 0\hfill \hfill \mathrm{(13d)}& v_k^{*}{p}_k^{*}=0\hfill \hfill \mathrm{(13e)}& {\left.{\displaystyle \frac{\partial L\left(p_k,v_k,\mu \right)}{\partial p_k}}\right|}_{p_k^{*},v_k^{*},{\mu }^{*}}=-{\displaystyle \frac{{\xi }_k}{{\sigma }^2+{\xi }_k{p}_k^{*}}}-v_k^{*}+{\mu }^{*}=0\hfill \end{array}$$

where $p_k^{*}$ is the optimal power allocation, $v_k^{*}$ and ${\mu }^{*}$ are the optimal Lagrange multipliers. According to (13c) and (13e), the following inequalities can be obtained:

$${\mu }^{*}\ge {\displaystyle \frac{{\xi }_k}{{\sigma }^2+{\xi }_k{p}_k^{*}}},k=1,\dots ,K$$

(14)

When ${\mu }^{*}<{\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$, it can be derived by combining (14) that

$${\displaystyle \frac{{\xi }_k}{{\sigma }^2+{\xi }_k{p}_k^{*}}}\le {\mu }^{*}<{\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$$

(15)

It can be solved from (13a) and (15) that:

$$p_k^{*}\ge {\displaystyle \frac{1}{{\mu }^{*}}}-{\displaystyle \frac{{\sigma }^2}{{\xi }_k}}$$

(16)

According to the complementary slackness condition in (13d), if $p_k^{*}=0$, the derived formula from (14) would contradict ${\mu }^{*}<{\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$, Therefore, when $p_k^{*}>0$, it can be obtained from (13e) that $v_k^{*}=0$, so the following equality holds:

$${\mu }^{*}-{\displaystyle \frac{{\xi }_k}{{\sigma }^2+{\xi }_k{p}_k^{*}}}=0$$

(17)

Therefore, when ${\mu }^{*}<{\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$, there is $p_k^{*}>0$, and the closed-form solution can be obtained from (16) and (17):

$$p_k^{*}={\displaystyle \frac{1}{{\mu }^{*}}}-{\displaystyle \frac{{\sigma }^2}{{\xi }_k}},{\mu }^{*}<{\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$$

(18)

When ${\mu }^{*}\ge {\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$, if $p_k^{*}>0$, according to the complementary slackness condition in (13d), there is $v_k^{*}=0$. From (17) and ${\mu }^{*}\ge {\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$, it can be solved that $p_k^{*}\le 0$, which contradicts $p_k^{*}>0$. According to (13a), when ${\mu }^{*}\ge {\displaystyle \frac{{\xi }_k}{{\sigma }^2}}$ the only feasible solution is $p_k^{*}=0$. Therefore, the optimal power allocation can be expressed as:

$$p_k^{*}=\left\{\begin{array}{cccc}\hfill & {\displaystyle \frac{1}{{\mu }^{*}}}-{\displaystyle \frac{{\sigma }^2}{{\xi }_k}},\hfill & \hfill & \mathrm{if}{\mu }^{*}<{\displaystyle \frac{{\xi }_k}{{\sigma }^2}}\hfill \\ \hfill & 0,\hfill & \hfill & \mathrm{if}{\mu }^{*}\ge {\displaystyle \frac{{\xi }_k}{{\sigma }^2}}\hfill \end{array}\right.$$

(19)

The above process can be described as pouring the total power P into a water tank, and the classic method shown in Figure 2 can be used for power allocation.

Where ${\displaystyle \frac{1}{{\mu }^{*}\left(i\right)}}$ represents the water level. The channel gains of the K users are arranged in descending order. The water level can be derived from (13a) and the channel state information as follows:

$${\displaystyle \frac{1}{{\mu }^{*}\left(i\right)}}={\displaystyle \frac{P+{\displaystyle \sum _{k=1}^{K-i+1}}\frac{{\sigma }^2}{{\xi }_k}}{K-i+1}}.$$

(20)

where $i=1\dots K$; $k=1\dots K-i+1$. After obtaining the water level from (20), the power allocation for each user can be calculated using (19). By adjusting the UAV transmit power $\mathbf{P}=(p_1,\dots ,p_k)$, the RIS unit phase shift matrix $\mathbf{\Phi }$, and the UAV trajectory $\mathbf{Q}$, the maximum system sum rate can be achieved. First, the UAV perceives channel changes and uses ZF coding to eliminate inter-user interference, then employs the water-filling algorithm to solve for the optimal power allocation $\mathbf{P}$ among users. Then the RIS unit phase shift $\theta$ and UAV trajectory Q are included in Equation (8). This problem is non-convex; therefore, a DRL-based D3QN-WF algorithm is designed in the third subsection to solve it.

2.3. Throughput Model

To fully demonstrate the advantages and potential of the D3QN-WF algorithm, this study will calculate the total throughput of the UAV during the entire communication period based on the system sum rate analyzed in Formula (8). The throughput over all T time slots is defined as follows [24]:

$$R_{sum}=\sum _{t=1}^{T}R\left(t\right)$$

(21)

where R(t) is the system sum rate under the UAV’s deployment position in the t time slot.

3. DRL-Based Algorithms

Jointly adjusting the 3D coordinates $\mathbf{Q}$ of the UAV and the phase angles $\theta$ of the RIS to maximize the system sum rate can be categorized as a Markov Decision Process (MDP), which consists of states, actions, rewards, and state transition probabilities:

• State $s_t$: The state of the system at time slot t is defined as $s_t=\left({\mathbf{Q}}_t,{\mathbf{\Phi }}_t\right)$, $s_t\in S$, S is denoted as the state space. where ${\mathbf{Q}}_t$ represents the 3D coordinates of the UAV at time slot t. ${\mathbf{\Phi }}_t$ denotes the phase shift matrix of the RIS at time slot t.
• Action $a_t$: Define the spatial action of the UAV in the system at time slot t: The spatial action of the UAV includes horizontal movement and vertical movement. The phase matrix of the RIS elements is dynamically optimized, with the phase parameter of each RIS element being discretely adjusted within a predetermined discretization range. The action $a_t$ belongs to the action space A.
• State Transition Probability $p\left(s_{t+1}\left|s_t,a_t\right.\right)$: The optimization problem of UAV path planning and RIS phase angle adjustment can be simplified as an MDP. The state transition probability depends only on the current state $s_t$ and action $a_t$.
• Reward $r_t$: The UAV receives feedback on its actions from the environment. Rewards can help it evaluate the quality of its actions and adjust its strategy accordingly. This strategy is adjusted according to the magnitude of the reward to obtain higher returns, thereby improving the sum rate of the system. $r_t$ denotes the reward obtained by the system after taking action $a_t$ under state $s_t$ at time slot t. Thus, the reward $r_t$ is the downlink sum rate $R\left(t\right)$ of the system at time slot t, as shown follows:

  $$r_t=R\left(t\right)$$

  (22)
  The goal of MDP is to maximize the cumulative expected return. When the system is in the optimal state, when the maximum rate is achieved, the maximum return is obtained.

To provide a detailed introduction to D3QN, we first present the basic structure of DRL algorithms. In reinforcement learning, Q-learning is an effective method for solving an MDP. It constructs an action-value function $Q(s_t,a_t)$ to evaluate the quality of taking action $a_t$ in state $s_t$. For a state: under a given policy, $Q(s_t,a_t)$ is defined as the sum of future rewards:

$$Q\left(s_t,a_t\right)=r_t+\gamma ·r_{t+1}+\dots +{\gamma }^{T-t-1}·r_{T-1},0\le t\le T-1,$$

(23)

where $\gamma \in [0,1]$ is a discount factor that measures the importance of current and future rewards. $Q(s_t,a_t)$ represents the expected cumulative reward obtained by taking action $a_t$; in state $a_t$, the optimal action-value function $Q^{*}(s_t,a_t)$ is defined as:

$$Q^{*}\left(s_t,a_t\right)=maxQ\left(s_t,a_t\right)$$

(24)

and it satisfies the Bellman optimality equation:

$$Q^{*}\left(s_t,a_t\right)=r_t+\gamma ·\underset{a_{t+1}\in A}{\max }{Q}^{*}\left(s_{t+1},a_{t+1}\right)$$

(25)

by selecting the action with the highest value in state $s_t$, the optimal policy can be derived from the optimal action-value function. The update rule for $Q(s_t,a_t)$ is as follows:

$$Q\left(s_t,a_t\right)=Q\left(s_t,a_t\right)+\overline{\alpha }·\left[r_t+\underset{a_{t+1}\in A}{max}Q\left(s_{t+1},a_{t+1}\right)-Q\left(s_t,a_t\right)\right]$$

(26)

where $\overline{\alpha }$ denotes the learning rate. $\Theta$ and ${\Theta }^{-}$ represent the weights of the estimation network and the target network, respectively. The estimation network is used to obtain $Q\left(s_t,a_t\left|\Theta \right.\right)$, which approximates $Q(s_t,a_t)$. During the training process, the weights $\Theta$ of the estimation network only need to be updated by minimizing the loss function $L(\Theta )$:

$$L\left(\Theta \right)={\left({y_t}^{DQN}-Q\left(s_t,a_t\left|\Theta \right.\right)\right)}^2$$

(27)

$${y_t}^{DQN}=r_t+\underset{a_{t+1}\in A}{max}Q\left(s_{t+1},a_{t+1}{\left|\Theta \right.}^{-}\right)$$

(28)

where ${y_t}^{DQN}$ is the target value. The weights ${\Theta }^{-}$ of the target network are updated periodically with the weights $\Theta$ of the estimation network, with an update interval of O steps. Since DQN directly selects target actions based on target Q-values, it has the problem of overestimation. To solve this problem, DDQN decouples the two steps of selecting target actions and calculating target Q-values through the loss function $L\left(\theta \right)$:

$$L\left(\Theta \right)={\left({y_t}^{DDQN}-Q\left(s_t,a_t\left|\Theta \right.\right)\right)}^2$$

(29)

$$L\left(\Theta \right)={y_t}^{DDQN}=r_t+Q\left(s_{t+1},\underset{a_{t+1}\in A}{argmax}Q\left(s_{t+1},a_{t+1}\left|\Theta \right.\right){\left|\Theta \right.}^{-}\right)$$

(30)

when the state is at $s_t$, the operation of estimating the action value function $Q(s_t,a_t)$ using DQN or DDQN will lead to unstable output of the value function. To solve the MDP problem, D3QN is used to improve training results. The architecture of Dueling DQN utilizes a sequence of two fully connected layers, which allows the state value and the optimal action to be estimated, respectively. This design allows the neural network to make a basic judgment on a given state, and then revise its judgment through different actions $a_t$. On the other hand, Dueling DQN outputs a Q function, which can be combined with DDQN to eliminate the widespread overestimation problem in traditional DQN algorithms.

The network structure of D3QN is shown in Figure 3. First, empirical parameters of size F are placed in the input layer. After passing through the hidden layer, effective features are extracted and sent to two separate paths, respectively. One path is the state value, which is irrelevant to the actions to be taken by the UAV and the RIS. The output of this part represents the value reflecting the current environment. This function is used to evaluate the value of UAV and RIS in a specific environment, and is called the value function $V\left(s\right)$. The other path focuses on action value, expressed as the advantage function $A(s,a)$; it is used to measure the relative advantage of each action selected by UAV and RIS from A. These two paths are denoted as $V\left(s\left|\Theta ,\right.\tilde{\beta }\right)$ and $A(s,a\left|\Theta ,\tilde{a}\right.)$, where $\theta$, $\tilde{\alpha }$ and $\tilde{\beta }$ represent the parameters of the hidden layer (DNNs), the value function network, and the advantage function network, respectively. D3QN independently learns the value and the advantage of actions, then combines them to form the output layer, which outputs $Q(s_t,a_t)$. This can be further expressed as $Q\left(s,a\left|\Theta ,\tilde{\alpha },\tilde{\beta }\right.\right)$. Combining the action advantage $A(s,a)$ and the state value $V\left(s\right)$, while subtracting the average value of the action advantage $A(s,a)$, the equation is as follows:

$$Q\left(s_t,a_t\left|\Theta ,\tilde{\alpha },\tilde{\beta }\right.\right)=V\left(s_t\left|\Theta ,\right.\tilde{\beta }\right)+\left(A\left(s_t,a_t\left|\Theta ,\tilde{\alpha }\right.\right)-{\displaystyle \frac{1}{\left|A\right|}}\sum _{a_{t+1\in A}}A\left(s_t,a_{t+1}\left|\Theta ,\tilde{\alpha }\right.\right)\right)$$

(31)

where $\left|A\right|$ represents the dimension of the action space A. Additionally, the loss function of D3QN can be expressed as:

$$L\left(\Theta ,\tilde{\alpha },\tilde{\beta }\right)={\left({y_t}^{D3QN}-Q\left(s_t,a_t\left|\Theta \right.,\tilde{\alpha },\tilde{\beta }\right)\right)}^2$$

(32)

$${y_t}^{D3QN}=r_t+\gamma Q\left(s_{t+1},\underset{a_{t+1}\in A}{argmax}Q\left(s_{t+1},a_{t+1}\left|\Theta ,\tilde{\alpha },\tilde{\beta }\right.\right)\left|{\Theta }^{-},{\tilde{\alpha }}^{-},{\tilde{\beta }}^{-}\right.\right)$$

(33)

where ${\Theta }^{-}$, ${\tilde{\alpha }}^{-}$, and ${\tilde{\beta }}^{-}$ represent the parameters of the target network, which are periodically copied and updated according to $\theta$, $\tilde{\alpha }$, and $\tilde{\beta }$ in the estimation network. The process of D3QN is shown in Figure 4.

The process of the D3QN algorithm is illustrated in Figure 4. Each episode consists of two stages: the exploration stage (Steps 3 to 24 in Algorithm 1) and the training stage (Steps 25 to 27). In the exploration stage, Action $a\left(n\right)$ is selected via a random approach (with probability $\epsilon$) or the greedy approach in Step 5 to obtain the UAV trajectory and RIS phase shifts. Meanwhile, the UAV is restricted from flying out of the defined range, and the next state $s(n+1)$ is acquired. Then, ZF precoding and the WF algorithm are used for power allocation (Steps 8 to 20), and the corresponding reward $r\left(a\right(n),s(n\left)\right)$ is calculated. Subsequently, the newly generated sample $\left(s\right(n),a(n),r(n),s(n+1\left)\right)$ is stored in the experience replay buffer F in Step 23. In the training stage, a random mini-batch of samples is selected from the experience replay buffer to train the online network $Q(·)$ and the target network $Q^{-}(·)$. After that, the target value $y_t^{D3QN}$ from Equation (33) is used to update the weights $\Theta$ of the online network by minimizing the loss $L\left(\Theta \right)$, which follows the general definition in Equation (32).

The state space: The system state $s_t\in S$ at time slot t includes the 3D coordinates $(x_t,y_t,z_t)$ of the UAV, the phase ${\theta }_{nt}$ of the n-th reflecting element in the RIS. Thus, $S_t$ can be defined as: $S_t=\left(x_t,y_t,z_t,{\theta }_{nt}\right)$

Action: Based on the observed environmental state $S_t$, the UAV selects an action from the action space A to execute. Among them, A consists of three parts: (1) the horizontal movement of the UAV $\Delta L_{UAV}\in \left\{\left(x_b,0\right),\left(-x_b,0\right),\left(0,y_b\right),\left(0,-y_b\right),\left(0,0\right)\right\}$; (2) The vertical movement of the UAV: $\Delta H_{UAV}\in \left\{\left(z_b,-z_b,0\right)\right\}$. Since the water-filling algorithm can obtain the optimal UAV transmission power, the action space A does not include the BS transmission power. Instead, the BS transmission power is placed in the environment, and after being solved, it is treated as the observed value of the environment. (3) The phase shift of each reflective element in the RIS $\Delta {\theta }_n\in \left\{{\displaystyle \frac{\pi }{80}},-{\displaystyle \frac{\pi }{80}},0\right\}$.

Algorithm 1: D3QN-WF algorithm

Reward: By taking actions, the 3D coordinates of the UAV and the phase shift matrix of the RIS are changed. Then, the objective function (12) is solved in the environment, the optimal power allocation (19) is obtained using the water-filling algorithm, and the optimal system sum rate of time slot t is calculated. The reward of time slot t is the same as that in Equation (22). The pseudocode of the algorithm D3QN-WF is shown in Algorithm 1.

4. Complexity of Simulation Analysis

The complexity of the D3QN-WF algorithm can be attributed to two aspects: environmental complexity and training process complexity. In this environment, the water-filling algorithm handles the allocation of BS transmission power to K users, with the number of optimization variables being K. The complexity of the water-filling algorithm is denoted as $f_{\mathrm{wf}}$. Then, the computational complexity for the BS to calculate the transmission power in each iteration is $f_{\mathrm{wf}}K$. During the training process, the complexity of the D3QN-WF algorithm is $O\left(\left|S\right|·\left|A\right|\right)$. Where $\left|S\right|$ and $\left|A\right|$ denote the sizes of the state space and action space, respectively. Therefore, the computational complexity of D3QN-WF is: $O\left(\left|S\right|·\left|A\right|·f_{\mathrm{wf}}K\right)$.

5. Simulation Result

In this section, the D3QN-WF algorithm is applied to an RIS-assisted BS-UAV wireless communication network. Comparisons are made with methods such as D3QN with average power, D3QN with unoptimized phase angles, DDQN-WF, and the D3QN-Majorize-Minimization algorithm(D3QN-MM) [11] that optimizes phase using convex optimization methods. Furthermore, the impact of RIS with different numbers of reflecting elements on the system rate is analyzed within the D3QN-WF algorithm. The system simulation parameters are presented in

Table 1. System Simulation Parameters.

Physical Meaning	Parameter	Value
Noise Power	${\sigma }^2$	—80 dBm
Bandwidth	B	1 MHz
Path Loss Exponent	$\alpha$	4
Path Channel Gain	$\beta$	—40 dBm
UAV Transmit Power	$P_{\mathrm{UAV}}$	30 dBm
Rice Factor	$\widehat{R}$	${10}^{3.3}$

. The hyperparameters in DRL are shown in

Table 2. D3QN-WF Algorithm Hyperparameters.

Physical Meaning	Parameter	Value
Learning Rate	$\alpha$	${10}^{-3}$
Decay Factor	$\gamma$	0.9
Replay Buffer	F	1500
Greedy Policy	$\epsilon$	0.1
Update Step	O	750
Mini-batch Size	Mini-batch size	75
Activation Function	Activation function	ReLU
Optimizer	Optimizer	RMSProp

. Python 3.7.12 is used to build a DNN based on TensorFlow 2.0.0 for the simulations.

Assume the UAV starts at (—300 m, —300 m, 150 m), UEs are randomly distributed on the ground within (0 m~100 m, 0 m~200 m, 0 m), and the RIS is located at (0 m, 150 m, 20 m). The UAV’s area of interest (AoI) spans (—300 m~300 m, —300 m~300 m, 50 m~150 m), and the grid step sizes are $x_b$ = $y_b$ = $z_b$ = 5 m.

Figure 5 and Figure 6 compare the 3D flight trajectories of the UAV and their corresponding 2D projections for three algorithms: DDQN-WF, D3QN-WF, and D3QN with average power. Starting from the initial position (—300 m, —300 m, 150 m), the UAV perceives communication users gradually, searches for the optimal trajectory, and globally optimal deployment position, then hovers near this position until the end of the time slot. This hovering and lingering behavior occurs because no termination condition is set in the DRL algorithm, leading to unrestricted movement throughout the entire time slot period T. If the total number of time slots T is small, it may not be possible to find the optimal deployment position in the end. When T is large, due to the adoption of the greedy strategy in DRL and the training error of the DNN, the UAV’s ability to maintain the optimal deployment position will be affected. This may cause the UAV to deviate from the optimal position and make exploratory movements to nearby positions. Suppose T has 200 time slots. After multiple rounds of training, the UAV identifies the optimal deployment position within 90–100 time slots. In the subsequent time slots, the UAV will either remain stationary at the optimal deployment position or exhibit slight fluctuations. However, when the reward obtained by the agent decreases, the UAV will try to return to the position with the maximum reward, resulting in the above-mentioned lingering phenomenon.

Figure 7 shows the variation of rewards with the number of training rounds for four algorithms: DDQN-WF, D3QN with average power, D3QN-MM, and D3QN-WF. It can be seen from the figure that the reward value of the D3QN-WF algorithm tends to converge after approximately 500 training rounds, with only slight fluctuations occurring thereafter. This phenomenon arises due to minor deviations when the UAV reaches the optimal deployment position, as well as the continuous impact of dynamic environmental parameters on system performance. It is worth noting that compared to DDQN-WF, D3QN with average power and D3QN-MM, D3QN-WF exhibits a smaller convergence fluctuation range and more stable convergence characteristics. Comparative experiments with D3QN-WF without optimized phase angles further validate that optimizing the RIS phase parameters can significantly improve the system communication rate.

Figure 8 and Figure 9, respectively, present the optimization results of two algorithms (DDQN-WF and D3QN-WF) for 8 RIS reflecting elements. Experiments show that the stability of D3QN-WF in dynamic environments is significantly superior to that of DDQN-WF. It can be seen from Figure 6 that multiple users are concentrated in the negative direction of the x-axis. Therefore, in comparison with these two algorithms, the phases optimized by D3QN-WF are more concentrated.

Figure 10 presents the cumulative distribution function (CDF) of the system sum rate during the training process for several algorithms, including DDQN-WF, D3QN-WF with different numbers of RIS elements, D3QN-WF with unoptimized phase angles, D3QN-WF with average power, and D3QN-MM. Among them, a CDF value of 1 corresponds to the rate under the optimal 3D deployment coordinates of the UAV. The optimal deployment positions obtained by the algorithms are shown in Figure 6 as follows: the optimal deployment of the D3QN with average power algorithm is at (0 m, 130 m, 50 m); the optimal deployment of the DDQN-WF algorithm is at (70 m, 185 m, 50 m); and the optimal deployment of the D3QN-WF algorithm is at (0 m, 155 m, 50 m). In Figure 10, the CDF curves of DDQN-WF, D3QN with average power, and D3QN without optimized phase angles are located to the left of that of D3QN-WF, which illustrates the superiority of D3QN-WF and indicates that it has obvious advantages in the statistical distribution characteristics of the sum rate. The simulation results show that compared with DDQN-WF and D3QN-MM, the D3QN-WF algorithm increases the system sum rate at the optimal position by 15.9% and 17.6%, respectively. This is because the essence of D3QN lies in combining the decomposed structure of state value and advantage value in Dueling DQN with the dual-network structure of DDQN. This combination improves the accuracy of each action value estimation in D3QN and mitigates the overestimation issue inherent in traditional Q-learning. Therefore, D3QN-WF optimizes the RIS phase matrix and the UAV’s position more effectively. Meanwhile, it can be observed that reasonable power allocation also improves the overall system sum rate.

According to Equation (21), Figure 11 depicts the cumulative distribution function (CDF) of the system throughput $R_{sum}$ during the communication period. It can be seen that D3QN-WF improves throughput throughout the entire communication period compared to other algorithms. In Figure 6, for the DDQN-WF algorithm, although it can find a deployment position close to users during the optimization process, it overestimates the value of some actions. This overestimation leads to a longer time spent searching for the optimal position throughout the entire time slot T. As a result, the throughput during the entire time slot T is significantly lower compared to D3QN-WF. Therefore, it can be observed that throughout the entire communication period, due to the reasonable power allocation of the D3QN-WF algorithm, its throughput is 20% higher than that of D3QN with average power allocation, and total throughput is increased by 50.1% and 55.6% compared with DDQN-WF and D3QN-MM, respectively.

6. Conclusions

In this paper, research on the optimization of the RIS-assisted BS-UAV aerial-ground communication system is conducted. DRL is used to construct autonomous decision-making with deep interaction between the agent and the environment. The D3QN-WF and DDQN-WF algorithms were proposed to adjust the transmission power, reconstruct the RIS phase, and optimize the 3D coordinates to maximize the system sum rate. Simulation results show that, compared with the DDQN-WF algorithm, D3QN-WF has a faster convergence speed, which improves the system sum rate under the optimal deployment position. The total throughput during communication is increased by 50.1%, verifying the significant advantages of D3QN in dynamic environments and providing theoretical support for the optimization of future intelligent communication systems.