Energy Efficiency Maximization for Multi-UAV-IRS-Assisted Marine Vehicle Systems

Abstract

Mobile edge computing is envisioned as a prospective technology for supporting time-sensitive and computation-intensive applications in marine vehicle systems. However, the offloading performance is highly impacted by the poor wireless channel. Recently, an Unmanned Aerial Vehicle (UAV) equipped with an Intelligent Reflecting Surface (IRS), i.e., UIRS, has drawn attention due to its capability to control wireless signals so as to improve the data rate. In this paper, we consider a multi-UIRS-assisted marine vehicle system where UIRSs are deployed to assist in the computation offloading of Unmanned Surface Vehicles (USVs). To improve energy efficiency, the optimization problem of the association relationships, computation resources of USVs, multi-UIRS phase shifts, and multi-UIRS trajectories is formulated. To solve the mixed-integer nonlinear programming problem, we decompose it into two layers and propose an integrated convex optimization and deep reinforcement learning algorithm to attain the near-optimal solution. Specifically, the inner layer solves the discrete variables by using the convex optimization based on Dinkelbach and relaxation methods, and the outer layer optimizes the continuous variables based on the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3). The numerical results demonstrate that the proposed algorithm can effectively improve the energy efficiency of the multi-UIRS-assisted marine vehicle system in comparison with the benchmarks.

1. Introduction

Marine autonomous systems and wireless technologies have promoted the development of many new marine applications, such as marine monitoring, and intelligent navigation, posing challenges in processing computation-intensive tasks [1,2]. Unmanned Surface Vehicles (USVs) and Unmanned Aerial Vehicles (UAVs) have emerged as advanced marine intelligent devices equipped with sensing modules, computing modules, and communication modules to collect, analyze, and process marine data, including hydrological and meteorological information, image/voice/video, and obstacle detection, to make real-time prediction on vessel behaviors, and provide accurate and efficient navigation services for vessels. However, these marine devices may suffer severe energy capacity and computation resource limitations for processing lots of computation-intensive tasks. In this context, edge computing technology provides a promising paradigm for resource-limited marine devices, which has attracted significant attention in academia and industry [3,4,5]. In particular, the computation tasks can be offloaded to and processed at the edge servers with computation capability, which are usually installed on the communication infrastructures. However, due to the limitation of marine infrastructures and the special characteristics of marine environments, the direct communication links between marine devices and edge servers are highly impacted by obstacles, making it difficult to support massive data offloading in time [6,7].

Recently, an Intelligent Reflecting Surface (IRS) has shown significant advantages in enhancing signal transmission quality by reshaping the wireless propagation channel [8,9,10,11]. Specifically, the IRS is a digitally-controlled metasurface composed of a large number of passive reflection elements with adjustable phase shifts [12,13]. Existing works investigate the benefits of utilizing an IRS to assist computation offloading from users to edge servers [14,15]. For instance, Chen et al. [16] propose a sum computation rate maximizing problem by optimizing the computational model decision, time allocation, and IRS beamforming, which is solved by a novel penalty-based successive convex approximation method. Yang et al. [17] propose a total energy consumption minimization problem by optimizing the binary offloading mode, computation resources, offloading power, offloading time, and IRS phase shifts in the IRS-aided mobile edge computing system, which is solved by greedy-based and penalty-based algorithms. The aforementioned works consider the IRS-assisted edge computing network, where the IRS is deployed on the building facade or base station, which is still challenging to construct the dominant Line-of-Sight (LoS) channel links.

Thanks to the advantages of the high mobility of UAVs, the IRS mounted on a UAV (UIRS) can provide on-demand deployment in a Three-Dimensional (3D) space [18,19]. In addition, the channel in which the UIRS participates has a high LoS probability, further enhancing the signal quality. According to the advantages of the UIRS, there have been existing works on UIRS-assisted edge computing systems [20,21]. Zhai et al. [22] propose an energy efficiency maximization problem by optimizing the UIRS trajectory, passive beamforming, and computation resource allocation, and apply the successive convex approximation and Dinkelbach method to solve the optimization problem. Ai et al. [23] propose an IRS-aided wireless inland ship mobile edge computing network that minimizes energy consumption through the joint design of the offloading percentage of USVs, transmission power, UIRS trajectory, and phase shifts, and develop an enhanced block coordinate descent method to solve the optimization problem.

In marine vehicle systems, insufficient spectrum resources and huge computation demands cause several challenges in the energy consumption of USVs and UAVs as they are resource-constrained devices. USVs offload partial workloads to the shore BS, with powerful computation resources to reduce the pressure of local computing, but increase the transmission energy consumption and transmission time. In this process, the UIRS is expected to adjust its positions and phase shifts to obtain better channel conditions between USVs and the shore BS for improving the data rate. In addition, the works in [18,24,25,26] demonstrate that multiple IRSs in the wireless communication system can achieve performance improvement in the data rate compared to the single IRS-assisted system. However, these works focus on the communication resource allocation and phase shift design with the assistance of multiple static IRSs. Note that multi-UIRS studies are difficult to extend directly from the single UIRS due to complex coordination and cooperation, distributed deployment, and communication resource sharing. Additionally, the total energy consumption increases as the number of UAVs increases correspondingly [27]. In this context, to consider both the computation bits and energy consumption, there have been studies on energy efficiency, which is defined as the ratio of the computation bits to the energy consumption [28]. In this paper, energy efficiency is a critical issue in the marine vehicle system owing to the fact that USVs and UIRSs are generally energy-constrained and computation resource-constrained. To the best of our knowledge, there have been no prior studies of multi-UIRS-assisted computation offloading on providing an energy efficiency evaluation.

In this paper, we propose a resource management and trajectory design scheme in the multi-UIRS-assisted marine vehicle system. Each UAV installed with an IRS is deployed to assist the USVs in offloading partial workloads to the shore BS equipped with the edge server. To enhance energy efficiency, the association relationships, computation resources, multi-UIRS phase shifts, and trajectories are jointly optimized. In terms of optimization methods, existing works mainly rely on convex optimization theories, such as, [16,17,22,23]. These traditional methods pose design challenges in the complex and dynamic channel environment that incur high computational costs [29]. Fortunately, reinforcement learning methods can be used to solve most complex optimization problems [30,31]. However, the existing reinforcement learning algorithms cannot solve the hybrid continuous and discrete variables optimization problem directly. Therefore, we propose an integrated Convex Optimization and Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (CO-MATD3) algorithm to solve the formulated optimization problem with mixed continuous and discrete variables. Finally, extensive simulation results demonstrate that the proposed CO-MATD3 algorithm can improve energy efficiency, in comparison with the benchmarks. The main contributions are summarized as follows:

• In the multi-UIRS-assisted marine vehicle system, an energy efficiency maximization (EEM) optimization problem is formulated by jointly optimizing the association relationships between UIRSs and USVs, computation resources of USVs, multi-UIRS phase shifts, and multi-UIRS 3D trajectories, subject to the computation data demand threshold constraints.
• The formulated optimization problem is a mixed integer nonlinear programming problem with discrete variables, i.e., association relationships, and continuous variables, i.e., computation resources, phase shifts, and trajectories, which is well known to be NP-hard. To efficiently solve the challenging problem, we decompose the original problem into two layers to solve discrete and continuous variables, respectively.
• Then, we propose a CO-MATD3 algorithm, which is an integrated convex optimization and deep reinforcement learning algorithm designed to facilitate collaborative optimization. Specifically, in the inner layer, the Dinkelbach method and relaxation method are applied to optimize the association relationships. In the outer layer, a distributed cooperative deep reinforcement learning algorithm, i.e., the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), is developed to optimize continuous variables.
• Finally, the simulation results demonstrate the efficient training convergence and effectiveness of the proposed CO-MATD3 algorithm in optimizing energy efficiency. In addition, we find that our proposed CO-MATD3 algorithm has the capability to optimize multi-UIRS trajectories according to the dynamic locations of USVs to improve energy efficiency. The simulation results show that the performance of the proposed algorithm is superior to other benchmarks under different simulation conditions, such as the number of UIRSs, the number of USVs, computation resources, and transmission power.

The remainder of this paper is organized as follows. A multi-UIRS-assisted marine vehicle system model and energy efficiency maximization problem are introduced in Section 2. The CO-MATD3 algorithm to solve the optimization problem is proposed in Section 3. The numerical results are presented in Section 4, and finally, the paper is concluded in Section 5.

Notations: ${\mathbf{x}}^{†}$ denotes the transpose matrix of vector $\mathbf{x}$; ${\mathbf{x}}^{\mathrm{H}}$ denotes the conjugate transpose matrix of vector $\mathbf{x}$; $\mathbf{X}\otimes \mathbf{Y}$ denotes the Kronecker product of two matrices $\mathbf{X}$ and $\mathbf{Y}$; and ${\mathbb{C}}^{X\times Y}$ denotes the space of the $X\times Y$ complex matrix.

2. System Model And Problem Formulation

2.1. Network Model

We consider a multi-UIRS-assisted marine vehicle system as illustrated in Figure 1, which consists of a group of USVs denoted as $\mathcal{M}=\{1,2,\dots ,M\}$, a group of UIRSs denoted as $\mathcal{N}=\{1,2,\dots ,N\}$, and a shore BS. The shore BS equipped with edge servers has powerful computation capability so that it can execute offloading data from USVs. The resource-constrained USVs can either process their computation tasks locally or offload partial workloads to the shore BS. Considering the complex marine environment, the communication channels between USVs and the shore BS are often blocked by vessels in offshore areas. Multiple UIRSs are employed to provide reflecting links between USVs and the shore BS. We suppose that the perfect channel state information can be obtained by the existing channel estimation methods. The UIRS n is composed of $K_n=K_{n,x}\times K_{n,y}$ reflection elements, where $K_{n,x}$ and $K_{n,y}$ denote the number of elements of the nth UIRS along the x-axis and y-axis, respectively. The reflection coefficient matrix of the UIRS n is denoted as ${\Theta }_n=\mathrm{diag}\{e^{\mathrm{j}{\phi }_{1,1}},\dots ,e^{\mathrm{j}{\phi }_{k_{n,x},k_{n,y}}},\dots ,e^{\mathrm{j}{\phi }_{K_{n,x},K_{n,y}}}\}$, where ${\phi }_{k_{n,x},k_{n,y}}\in [0,2\mathsf{\pi }], k_{n,x}=\{1,\dots ,K_{n,x}\}, k_{n,y}=\{1,\dots ,K_{n,y}\}$ denotes the $(k_{n,x},k_{n,y})$th reflection element of the nth UIRS programmable phase shift, and $\mathrm{j}$ denotes the imaginary unit. We adopt that several USVs form a Non-Orthogonal Multiple Access (NOMA) cluster to offload workloads to the shore BS via the associated UIRS by the NOMA protocol.

In the 3D Cartesian coordinate system, the x-axis and y-axis denote the horizontal coordinate, respectively, and the z-axis denotes the vertical coordinate over the sea level. The location of the USV m is denoted by ${\mathbf{L}}_m=\{x_m,y_m,0\},\forall m\in \mathcal{M}$. The location of the shore BS is denoted by ${\mathbf{L}}_0=\{x_0,y_0,z_0\}$. For convenience, the period is divided into T equal time slots denoted as $\mathcal{T}=\{1,2,\dots ,T\}$, with the duration of $\tau$. The location of the UIRS n at time slot t is denoted by ${\mathbf{L}}_n\left(t\right)=\{x_n\left(t\right),y_n\left(t\right),z_n\left(t\right)\}$. The distance between the USV m and the UIRS n at time slot t is $d_{m,n}\left(t\right)=\parallel {\mathbf{L}}_m-{\mathbf{L}}_n\left(t\right)\parallel$; the distance between the UIRS n and the shore BS at time slot t is $d_{n,0}\left(t\right)=\parallel {\mathbf{L}}_n\left(t\right)-{\mathbf{L}}_0\parallel$; the distance between the USV m and the BS is $d_{m,0}=\parallel {\mathbf{L}}_m-{\mathbf{L}}_0\parallel$.

2.2. Channel Model

The Rician fading is adopted for both channel models from the USV to the UIRS (USV-UIRS link) and from the UIRS to the shore BS (UIRS-BS link). Thus, the channel gain ${\mathbf{h}}_{m,n}\left(t\right)\in {\mathbb{C}}^{1\times K_n}$ between the USV m and the UIRS n, and ${\mathbf{h}}_{n,0}\left(t\right)\in {\mathbb{C}}^{K_n\times 1}$ between the UIRS n and the shore BS at time slot t are

$${\mathbf{h}}_{m,n}\left(t\right)=\sqrt{{\rho }_0{d}_{m,n}^{-{\beta }_{m,n}}\left(t\right)}{\tilde{\mathbf{h}}}_{m,n}\left(t\right),\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T},$$

(1)

and

$${\mathbf{h}}_{n,0}\left(t\right)=\sqrt{{\rho }_0{d}_{n,0}^{-{\beta }_{n,0}}\left(t\right)}{\tilde{\mathbf{h}}}_{n,0}\left(t\right),n\in \mathcal{N},\forall t\in \mathcal{T},$$

(2)

respectively, where ${\rho }_0$ denotes the channel gain at the reference distance, and ${\beta }_{m,n}$ and ${\beta }_{n,0}$ denote the path loss exponent. The Rician fading parameters are denoted as ${\tilde{\mathbf{h}}}_{m,n}\left(t\right)=\sqrt{{\xi }_{m,n}/(1+{\xi }_{m,n})}{\mathbf{h}}_{m,n}^{\mathrm{LoS}}\left(t\right)+\sqrt{1/(1+{\xi }_{m,n})}{\mathbf{h}}_{m,n}^{\mathrm{NLoS}}\left(t\right)$ and ${\tilde{\mathbf{h}}}_{n,0}\left(t\right)=\sqrt{{\xi }_{n,0}/(1+{\xi }_{n,0})}{\mathbf{h}}_{n,0}^{\mathrm{LoS}}\left(t\right)$$+\sqrt{1/(1+{\xi }_{n,0})}{\mathbf{h}}_{n,0}^{\mathrm{NLoS}}\left(t\right)$ [32], where ${\xi }_{m,n}$ and ${\xi }_{n,0}$ denote the Rician factors. ${\mathbf{h}}_{m,n}^{\mathrm{LoS}}\left(t\right)$ and ${\mathbf{h}}_{n,0}^{\mathrm{LoS}}\left(t\right)$ are the LoS components. ${\mathbf{h}}_{m,n}^{\mathrm{NLoS}}\left(t\right)\sim \mathcal{CN}(0,{\mathbf{I}}_{K_n})$ and ${\mathbf{h}}_{n,0}^{\mathrm{NLoS}}\left(t\right)\sim \mathcal{CN}(0,{\mathbf{I}}_{K_n})$ are the Non-LoS (NLoS) components [18].

The LoS links ${\mathbf{h}}_{m,n}^{\mathrm{LoS}}\left(t\right)$ and ${\mathbf{h}}_{n,0}^{\mathrm{LoS}}\left(t\right)$ at time slot t are [29]

$$\begin{array}{c}\hfill {\mathbf{h}}_{m,n}^{\mathrm{LoS}}\left(t\right)={[1,e^{-\mathrm{j}{\displaystyle \frac{2\mathsf{\pi }d}{\lambda }}{\widehat{\varpi }}_{m,n}\left(t\right)},\dots ,e^{-\mathrm{j}{\displaystyle \frac{2(K_{n,x}-1)\mathsf{\pi }d}{\lambda }}{\widehat{\varpi }}_{m,n}\left(t\right)}]}^{†}\otimes {[1,e^{-\mathrm{j}{\displaystyle \frac{2\mathsf{\pi }d}{\lambda }}{\check{\varpi }}_{m,n}\left(t\right)},\dots ,e^{-\mathrm{j}{\displaystyle \frac{2(K_{n,y}-1)\mathsf{\pi }d}{\lambda }}{\check{\varpi }}_{m,n}\left(t\right)}]}^{†},\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T},\end{array}$$

(3)

and

$$\begin{array}{c}\hfill {\mathbf{h}}_{n,0}^{\mathrm{LoS}}\left(t\right)={[1,e^{-\mathrm{j}{\displaystyle \frac{2\mathsf{\pi }d}{\lambda }}{\widehat{\varrho }}_{n,0}\left(t\right)},\dots ,e^{-\mathrm{j}{\displaystyle \frac{2(K_{n,x}-1)\mathsf{\pi }d}{\lambda }}{\widehat{\varrho }}_{n,0}\left(t\right)}]}^{†}\otimes {[1,e^{-\mathrm{j}{\displaystyle \frac{2\mathsf{\pi }d}{\lambda }}{\check{\varrho }}_{n,0}\left(t\right)},\dots ,e^{-\mathrm{j}{\displaystyle \frac{2(K_{n,y}-1)\mathsf{\pi }d}{\lambda }}{\check{\varrho }}_{n,0}\left(t\right)}]}^{†},\forall n\in \mathcal{N},\forall t\in \mathcal{T},\end{array}$$

(4)

respectively, where $\lambda$ denotes the carrier wavelength, d denotes the separation distance, ${\widehat{\varpi }}_{m,n}\left(t\right)=sin{\varphi }_{m,n}\left(t\right)cos{\gamma }_{m,n}\left(t\right)$, ${\check{\varpi }}_{m,n}\left(t\right)=sin{\varphi }_{m,n}\left(t\right)sin{\gamma }_{m,n}\left(t\right)$, ${\widehat{\varrho }}_{n,0}\left(t\right)=sin{\varphi }_{n,0}\left(t\right)cos{\gamma }_{n,0}\left(t\right)$, and ${\check{\varrho }}_{n,0}\left(t\right)=sin{\varphi }_{n,0}\left(t\right)sin{\gamma }_{n,0}\left(t\right)$. The vertical and horizontal Angle of Arrivals (AoAs) from the USV m to the UIRS n are denoted by ${\varphi }_{m,n}\left(t\right)$ and ${\gamma }_{m,n}\left(t\right)$, respectively. The vertical and horizontal Angle of Departures (AoDs) from the UIRS n to the shore BS are denoted by ${\varphi }_{n,0}\left(t\right)$ and ${\gamma }_{n,0}\left(t\right)$, respectively. We have $sin{\varphi }_{m,n}\left(t\right)={\displaystyle \frac{|z_n\left(t\right)|}{d_{m,n}\left(t\right)}}$, $sin{\gamma }_{m,n}\left(t\right)={\displaystyle \frac{|x_n\left(t\right)-x_m|}{\sqrt{{(x_n\left(t\right)-x_m)}^2+{(y_n\left(t\right)-y_m)}^2}}}$, $cos{\gamma }_{m,n}\left(t\right)={\displaystyle \frac{|y_n\left(t\right)-y_m|}{\sqrt{{(x_n\left(t\right)-x_m)}^2+{(y_n\left(t\right)-y_m)}^2}}}$, $sin{\varphi }_{n,0}\left(t\right)={\displaystyle \frac{|z_n\left(t\right)-z_0|}{d_{n,0}\left(t\right)}}$, $sin{\gamma }_{n,0}\left(t\right)={\displaystyle \frac{|x_n\left(t\right)-x_0|}{\sqrt{{(x_n\left(t\right)-x_0)}^2+{(y_n\left(t\right)-y_0)}^2}}}$, and $cos{\gamma }_{n,0}\left(t\right)={\displaystyle \frac{|y_n\left(t\right)-y_0|}{\sqrt{{(x_n\left(t\right)-x_0)}^2+{(y_n\left(t\right)-y_0)}^2}}}$.

Adopting the two-ray signal propagation model for marine channels [1], the channel gain $h_{m,0}$ between the USV m and the shore BS is

$$h_{m,0}={\left({\displaystyle \frac{\lambda }{4\mathsf{\pi }{d}_{m,0}}}\right)}^2{\left[2sin\left({\displaystyle \frac{2\mathsf{\pi }{H}_m{H}_0}{\lambda d_{m,0}}}\right)\right]}^2,\forall m\in \mathcal{M},$$

(5)

where $H_m$ denotes the antenna height of the USV m, and $H_0$ denotes the antenna height of the shore BS.

After reflection and adjustment by the UIRS elements, the cascaded channel gain $h_{m,n,0}\left(t\right)\in {\mathbb{C}}^{1\times 1}$ from the USV m to the BS via the UIRS n at time slot t is

$$\begin{array}{cc}\hfill h_{m,n,0}\left(t\right)=& h_{m,0}+{\left({\mathbf{h}}_{n,0}\left(t\right)\right)}^{\mathrm{H}}{\Theta }_n\left(t\right){\mathbf{h}}_{m,n}\left(t\right),\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T}.\hfill \end{array}$$

(6)

We introduce a binary variable ${\alpha }_{m,n}\left(t\right)\in \{0,1\},\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T}$ to represent the association relationship between the USV m and the UIRS n at time slot t. Specifically, ${\alpha }_{m,n}\left(t\right)=1$ indicates that the USV m is associated with the UIRS n, and ${\alpha }_{m,n}\left(t\right)=0$, otherwise. Therefore, the USVs are associated with the same UIRS to form a NOMA cluster for computation offloading, which can be denoted as ${\mathcal{M}}_n\left(t\right)=\left\{m\right|{\alpha }_{m,n}\left(t\right)=1,\forall m\in \mathcal{M}$, $\forall n\in \mathcal{N},\forall t\in \mathcal{T}\}$.

We assume that the perfect successive interference cancellation is performed in the NOMA protocol [33]. The cascaded channel gain at the shore BS serving a group of USVs via the UIRS n at time slot t are sorted in descending order as follows:

$$\begin{array}{cc}\hfill & h_{1,n,0}\left(t\right)\ge \dots \ge h_{m,n,0}\left(t\right)\ge h_{j,n,0}\left(t\right)\ge \dots \ge h_{M_n,n,0}\left(t\right),\forall m,j\in {\mathcal{M}}_n,\forall n\in \mathcal{N},\forall t\in \mathcal{T},\hfill \end{array}$$

(7)

that is, the signal received from the USV with the highest channel gain would be decoded first, while other USVs with lower channel gains would be considered as the co-channel interference. Thus, the data rate of the USV m at time slot t is

$$R_m\left(t\right)=W{log}_2\left(1+\sum _{n=1}^N{\displaystyle \frac{{\alpha }_{m,n}\left(t\right)p_m\left(t\right){\left|h_{m,n,0}\left(t\right)\right|}^2}{{\sum }_{j=m+1}^{M_n\left(t\right)}{p}_j\left(t\right){\left|h_{j,n,0}\left(t\right)\right|}^2+{\sigma }^2}}\right),\forall m\in {\mathcal{M}}_n\left(t\right),\forall t\in \mathcal{T},$$

(8)

where W is the channel bandwidth, ${\sigma }^2$ is the noise power, and $p_m\left(t\right)$ is the transmission power of the USV m.

2.3. Task Execution Model

In this paper, each USV adopts the partial offloading method. The computation task data of each USV are composed of local computing by itself and edge computing by the edge server. Therefore, the computation rate of the USV is the sum of the local computing rate and the offloading data rate.

(1) Local computing: The local computation bits of the USV m at time slot t are

$$B_m^{\mathrm{loc}}\left(t\right)={\displaystyle \frac{\tau f_m\left(t\right)}{C_m}},\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(9)

where $C_m$ denotes the number of CPU cycles required to finish one bit of data at the USV m, and $f_m\left(t\right)$ denotes the computation resources of the USV m. The USV m has a maximum computation resource $f_m^{\max }$ constraint, satisfying

$$0\le f_m\left(t\right)\le f_m^{\max },\forall m\in \mathcal{M},\forall t\in \mathcal{T}.$$

(10)

(2) Edge computing: In edge computing, the USV m offloads partial workloads to the shore BS. Considering the shore BS has a strong computation capability, it can process offloading workloads from USVs. Hence, the offloading bits of the USV m at time slot t are

$$B_m^{\mathrm{off}}\left(t\right)=\tau R_m\left(t\right),\forall m\in \mathcal{M},\forall t\in \mathcal{T}.$$

(11)

2.4. Energy Consumption Model

The system energy consumption includes the following: (1) the local computation energy consumption of each USV $E_m^{\mathrm{loc}}$; (2) the offloading energy consumption of each USV $E_m^{\mathrm{off}}$; (3) the offloading energy consumption of the reflection elements on each UIRS $E_n^{\mathrm{ref}}$; and (4) the flight energy consumption of each UIRS $E_n^{\mathrm{fly}}$. Note that the computation energy consumption of the shore BS is negligible due to the sufficient energy supply.

The local computation energy consumption of the mth USV at time slot t is

$$E_m^{\mathrm{loc}}\left(t\right)={\kappa }_m{B}_m^{\mathrm{loc}}\left(t\right)C_m{\left(f_m\left(t\right)\right)}^2,\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(12)

where ${\kappa }_m$ is the effective switched capacitance of the USV m.

The offloading energy consumption of the mth USV at time slot t is

$$E_m^{\mathrm{off}}\left(t\right)=\tau \sum _{n=1}^N{\alpha }_{m,n}\left(t\right)p_m\left(t\right),\forall m\in \mathcal{M},\forall t\in \mathcal{T}.$$

(13)

The energy consumption of the reflection elements on the nth UIRS at time slot t is [34]

$$E_n^{\mathrm{ref}}\left(t\right)=\tau \sum _{k=1}^{K_n}{p}_{n,k}\left(t\right),\forall n\in \mathcal{N},\forall t\in \mathcal{T},$$

(14)

where $p_{n,k}\left(t\right)$ is the power of the kth reflection element on the nth UIRS.

The flight energy consumption of the UIRS n at time slot t is given as [35]

$$\begin{array}{cc}\hfill E_n^{\mathrm{fly}}\left(t\right)=& \tau (P_0\left(1+{\displaystyle \frac{3{\left(v_n^{\mathrm{hor}}\left(t\right)\right)}^2}{U_{\mathrm{tip}}^2}}\right)+P_1{\left(\sqrt{(1+{\displaystyle \frac{{\left(v_n^{\mathrm{hor}}\left(t\right)\right)}^4}{4{\left(v^{\prime }\right)}^4}}}-{\displaystyle \frac{{\left(v_n^{\mathrm{hor}}\left(t\right)\right)}^2}{2{\left(v^{\prime }\right)}^2}}\right)}^{{\displaystyle \frac{1}{2}}}\hfill \\ \hfill & +P_2{v}_n^{\mathrm{ver}}\left(t\right)+{\displaystyle \frac{1}{2}}{d}_0\rho S_0{A}_0{\left(v_n^{\mathrm{hor}}\left(t\right)\right)}^3),\forall n\in \mathcal{N},\forall t\in \mathcal{T},\hfill \end{array}$$

(15)

where $U_{\mathrm{tip}}$, $d_0$, $\rho$, $S_0$, $A_0$, and $v^{\prime }$ denote the tip velocity of the rotor blade, fuselage drag ratio, air density, rotor solidity, rotor disc area, and average induced speed of the rotor, respectively. $P_0$, $P_1$, and $P_2$ denote the blade profile power, induced power, and vertical propulsion power, respectively. $v_n^{\mathrm{hor}}\left(t\right)={\displaystyle \frac{\sqrt{{(x_n(t+1)-x_n\left(t\right))}^2+{(y_n(t+1)-y_n\left(t\right))}^2}}{\tau }}\le v_{\max }^{\mathrm{hor}}$ is the horizontal velocity for the nth UIRS, where $v_{\max }^{\mathrm{hor}}$ is the maximum horizontal velocity, and $v_n^{\mathrm{ver}}\left(t\right)={\displaystyle \frac{|z_n(t+1)-z_n\left(t\right)|}{\tau }}\le v_{\max }^{\mathrm{ver}}$ is the vertical velocity for the nth UIRS, where $v_{\max }^{\mathrm{ver}}$ is the maximum vertical velocity.

2.5. Problem Formulation

In this paper, we define energy efficiency as the ratio of the total computation bits to the system’s energy consumption, which is expressed as

$$\delta ={\displaystyle \frac{\widehat{B}}{\widehat{E}}}={\displaystyle \frac{{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{m=1}^M}\left(B_m^{\mathrm{loc}}\left(t\right)+B_m^{\mathrm{off}}\left(t\right)\right)}{{\displaystyle \sum _{t=1}^T}\left({\displaystyle \sum _{m=1}^M}\left(E_m^{\mathrm{loc}}\left(t\right)+E_m^{\mathrm{off}}\left(t\right)\right)+{\displaystyle \sum _{n=1}^N}\left(E_n^{\mathrm{ref}}\left(t\right)+\omega E_n^{\mathrm{fly}}\left(t\right)\right)\right)}},$$

(16)

where $\omega$ is the weighting element, $\widehat{B}$ is the total computation bits, and $\widehat{E}$ is the system’s energy consumption.

For fulfilling the demand for specified computation bits of each USV at time slot t, we have

$$B_m\left(t\right)=B_m^{\mathrm{loc}}\left(t\right)+B_m^{\mathrm{off}}\left(t\right)\ge B_m^{\min },\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(17)

where $B_m^{\min }$ is the minimum task size requirement of the USV m at each time slot.

The association relationship between the USV m and the UIRS n at each time slot satisfies ${\alpha }_{m,n}\left(t\right)\in \{0,1\}$. Furthermore, each USV is assumed to associate with one UIRS at each time slot, satisfying

$$\sum _{n=1}^N{\alpha }_{m,n}\left(t\right)=1,\forall m\in \mathcal{M},\forall t\in \mathcal{T}.$$

(18)

Based on the above system model, we formulate an optimization problem to maximize the system’s energy efficiency. In the considered problem, the association relationships between USVs and UIRSs $\mathit{\alpha }=\{{\alpha }_{m,n}\left(t\right),\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T}\}$, computation resource allocation of USVs $\mathbf{f}=\{f_m\left(t\right),\forall m\in \mathcal{M},\forall t\in \mathcal{T}\}$, multi-UIRS phase shifts $\mathit{\phi }=\{{\phi }_{k_{n,x},k_{n,y}}\left(t\right),\forall k_{n,x}\in {\mathcal{K}}_{n,x},\forall k_{n,y}\in {\mathcal{K}}_{n,y},\forall n\in \mathcal{N},\forall t\in \mathcal{T}\}$, and multi-UIRS trajectories $\mathbf{L}=\{{\mathbf{L}}_n\left(t\right),\forall n\in \mathcal{N},\forall t\in \mathcal{T}\}$ are jointly optimized, subject to the computation data requirements of the USVs. Therefore, the EEM optimization problem is mathematically formulated as follows:

$$\left(\mathbf{EEM}\right):\underset{\mathit{\alpha },\mathbf{f},\mathit{\phi },\mathbf{L}}{max}\delta$$

(19)

$$\mathrm{s}.\mathrm{t}.B_m\left(t\right)\ge B_m^{\min },\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(20)

$${\alpha }_{m,n}\left(t\right)\in \{0,1\},\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T},$$

(21)

$$\sum _{n=1}^N{\alpha }_{m,n}\left(t\right)=1,\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(22)

$$0\le {\phi }_{k_{n,x},k_{n,y}}\left(t\right)\le 2\mathsf{\pi },\forall k_{n,x}\in {\mathcal{K}}_{n,x},\forall k_{n,y}\in {\mathcal{K}}_{n,y},\forall n\in \mathcal{N},\forall t\in \mathcal{T},$$

(23)

$$0\le f_m\left(t\right)\le f_m^{\max },\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(24)

$$v_n^{\mathrm{hor}}\left(t\right)\le v_{n,\max }^{\mathrm{hor}},\forall n\in \mathcal{N},\forall t\in \mathcal{T},$$

(25)

$$v_n^{\mathrm{ver}}\left(t\right)\le v_{n,\max }^{\mathrm{ver}},\forall n\in \mathcal{N},\forall t\in \mathcal{T}.$$

(26)

Constraint (20) is the computation data requirement of each USV. Constraints (21) and (22) are the association limitations. Constraint (23) denotes the range of phase shifts. Constraint (24) limits the allocated computation resources of each USV. Constraints (25) and (26) are the mobility limitations.

The EEM is a MINLP problem with coupled variables, which motivates us to develop an integrated convex optimization and MATD3 for solving this challenge.

3. Proposed CO-MATD3 Algorithm

In this section, the CO-MATD3 algorithm is proposed to solve the challenge of the formulated EEM problem. Specifically, the EEM problem is decomposed into two layers, i.e., the EEM-Inner problem of the association relationships $\mathit{\alpha }$, and the EEM-Outer problem of computation resources $\mathbf{f}$, multi-UIRS phase shifts $\mathit{\phi }$, and multi-UIRS trajectories $\mathbf{L}$. The convex optimization is applied to tackle the EEM-Inner problem to obtain the feasible solutions of $\mathit{\alpha }$. Then, the optimized $\mathbf{f}$, $\mathit{\phi }$, and $\mathbf{L}$ can be obtained by the MATD3 algorithm by substituting the obtained feasible $\mathit{\alpha }$ into the EEM-Outer problem.

3.1. EEM-Inner Problem

Due to the non-convex with fractional objective function, we convert it to a parametric formula by using the Dinkelbach method, which is rewritten as follows:

$$\underset{\mathit{\alpha },\mathbf{f},\mathit{\phi },\mathbf{L}}{max}\widehat{B}-\delta \widehat{E}.$$

(27)

When $max\{\widehat{B}-{\delta }^{*}\widehat{E}\}=0$, the objective function has the optimal solution, and ${\delta }^{*}$ is the optimal energy efficiency.

The EEM-Inner problem aims to optimize the association relationships $\mathit{\alpha }$ under given computation resources $\mathbf{f}$, the UIRS phase shifts $\mathit{\phi }$, and the positions of UIRSs $\mathbf{L}$. Therefore, the EEM-Inner problem is reformulated as follows:

$$\mathbf{\left(}\mathbf{EEM}\mathbf{\text{-}}\mathbf{Inner}\mathbf{\text{-}}\mathbf{1}\mathbf{\right)}:\underset{\mathit{\alpha }}{max}\widehat{B}-\delta \widehat{E}$$

(28)

$$\mathrm{s}.\mathrm{t}.\left(20\right),\left(21\right),\left(22\right).$$

(29)

To solve the binary constraint (21), we relax ${\alpha }_{m,n}\left(t\right)$ as a continuous variable, which is expressed as

$$0\le {\alpha }_{m,n}\left(t\right)\le 1,\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T}.$$

(30)

Thus, the EEM-Inner-1 problem can be converted to

$$\mathbf{\left(}\mathbf{EEM}\mathbf{\text{-}}\mathbf{Inner}\mathbf{\text{-}}\mathbf{2}\mathbf{\right)}:\underset{\mathit{\alpha }}{max}\widehat{B}-\delta \widehat{E}$$

(31)

$$\mathrm{s}.\mathrm{t}.0\le {\alpha }_{m,n}\left(t\right)\le 1,\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T},$$

(32)

$$\sum _{n=1}^N{\alpha }_{m,n}\left(t\right)=1,\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(33)

$$\tau R_m\left(t\right)\ge B_m^{\min }-B_m^{\mathrm{loc}}\left(t\right),\forall m\in \mathcal{M},\forall t\in \mathcal{T}.$$

(34)

Note that the association relationships and decoding order of the NOMA cluster are coupled; the EEM-Inner-2 problem cannot be directly solved due to the co-channel interference. To solve it, we apply an interference expansion method. In this context, the USV m would not adopt the decoding order during the transmission, which means that the USV m with the poorest channel condition receives the interference from other USVs; we can rewrite the data rate $R_m^{\prime }\left(t\right)$ of the USV m as

$$\begin{array}{cc}\hfill & R_m^{\prime }\left(t\right)=\sum _{n=1}^N{\alpha }_{m,n}\left(t\right)W{log}_2\left(1+{\displaystyle \frac{p_m\left(t\right){\left|h_{m,n,0}\left(t\right)\right|}^2}{{\displaystyle \sum _{j\in \mathcal{M}/\left\{m\right\}}}{p}_j\left(t\right){\left|h_{j,n,0}\left(t\right)\right|}^2+{\sigma }^2}}\right),\forall m\in \mathcal{M},\forall t\in \mathcal{T},\hfill \end{array}$$

(35)

where we can see

$$\begin{array}{c}\hfill \sum _{j\in \mathcal{M}/\left\{m\right\}}{p}_j\left(t\right){\left|h_{j,n,0}\left(t\right)\right|}^2\ge \sum _{j\in {\mathcal{M}}_n/\left\{m\right\}}{p}_j\left(t\right){\left|h_{j,n,0}\left(t\right)\right|}^2,\end{array}$$

(36)

and then

$$R_m^{\prime }\left(t\right)\le R_m\left(t\right),\forall m\in \mathcal{M},\forall t\in \mathcal{T}.$$

(37)

Therefore, the EEM-Inner-2 problem can be reformulated as

$$\mathbf{\left(}\mathbf{EEM}\mathbf{\text{-}}\mathbf{Inner}\mathbf{\text{-}}\mathbf{3}\mathbf{\right)}:\underset{\mathit{\alpha }}{max}\widehat{B}-\delta \widehat{E}$$

(38)

$$\mathrm{s}.\mathrm{t}.0\le {\alpha }_{m,n}\left(t\right)\le 1,\forall m\in \mathcal{M},\forall n\in \mathcal{N},\forall t\in \mathcal{T},$$

(39)

$$\sum _{n=1}^N{\alpha }_{m,n}\left(t\right)=1,\forall m\in \mathcal{M},\forall t\in \mathcal{T},$$

(40)

$$\tau R_m^{\prime }\left(t\right)\ge B_m^{\min }-B_m^{\mathrm{loc}}\left(t\right),\forall m\in \mathcal{M},\forall t\in \mathcal{T}.$$

(41)

The EEM-Inner-3 problem is convex, and we can use the optimization tool, e.g., CVXPY, to solve it.

3.2. EEM-Outer Problem

With the optimized value of the association relationships $\mathit{\alpha }$, by solving the EEM-Inner problem, we then optimize the computation resources $\mathbf{f}$, UIRS phase shifts $\mathit{\phi }$, and positions of the UIRSs $\mathbf{L}$ in the outer layer. Therefore, the EEM-Outer problem is reformulated as

$$\mathbf{\left(}\mathbf{EEM}\mathbf{\text{-}}\mathbf{Outer}\mathbf{\right)}:\underset{\mathbf{f},\mathit{\phi },\mathbf{L}}{max}\delta$$

(42)

$$\mathrm{s}.\mathrm{t}.\left(20\right),\left(23\right),\left(24\right),\left(25\right),\left(26\right).$$

(43)

We adopt the distributed decision-making at the UIRSs, and apply a policy-based and model-free multi-agent deep reinforcement learning algorithm, e.g., MATD3, to solve the multi-UIRS cooperation problem. In the marine vehicle system, each UIRS acts as an agent to send its state and action information to interact with the environment. In the MATD3-based EEM-Outer algorithm, we reformulate the EEM-Outer problem as the Markov Decision Process (MDP). The state space, action space, and reward function of the MDP are defined as follows:

(1) State space: In time slot t, the set of state space of N agents is $\mathbf{s}\left(t\right)=\left\{s_n\left(t\right)\right|n\in \mathcal{N}\}$. The state space of agent n is $s_n\left(t\right)=\{{\mathbf{h}}_n\left(t\right),{\mathbf{L}}_n\left(t\right)\}$. Here, ${\mathbf{h}}_n\left(t\right)=\left\{h_{m,n,0}\left(t\right)\right|m\in \mathcal{M}\}$ is the cascaded channel gain of the USV m, and ${\mathbf{L}}_n\left(t\right)$ is the location of the UIRS n.

(2) Action space: In time slot t, the set of action space of N agents is given as $\mathbf{a}\left(t\right)=\left\{a_n\left(t\right)\right|n\in \mathcal{N}\}$. The action space of agent n is $a_n\left(t\right)=\{v_n^{\mathrm{ver}}\left(t\right),v_n^{\mathrm{hor}}\left(t\right),{\vartheta }_n\left(t\right),{\mathbf{f}}_n\left(t\right),{\mathit{\phi }}_n\left(t\right)\}$. Here, $v_n^{\mathrm{ver}}\left(t\right)$ is the vertical flight speed of the UIRS n, $v_n^{\mathrm{hor}}\left(t\right)$ is the horizontal flight speed of the UIRS n, ${\vartheta }_n\left(t\right)\in [0,2\mathsf{\pi })$ is the horizontal flight direction of the UIRS n, ${\mathbf{f}}_n\left(t\right)=\left\{f_m\left(t\right)\right|m\in {\mathcal{M}}_n\}$ is the computation resources of the USVs, and ${\mathit{\phi }}_n\left(t\right)=\{{\phi }_{1,1}\left(t\right),\dots ,{\phi }_{k_{n,x},k_{n,y}}\left(t\right),\dots ,{\phi }_{K_{n,x},K_{n,y}}\left(t\right)\}$ is the nth UIRS phase shifts.

(3) Reward function: In time slot t, the set of the reward function of N agents can be expressed as $\mathbf{r}\left(t\right)=\left\{r_n\left(t\right)\right|n\in \mathcal{N}\}$. According to the problem formulation in Section 2, the objective of the EEM problem is to maximize energy efficiency with the constraint of computation data requirements. Therefore, the reward function $r_n\left(t\right)$ is denoted by

$$r_n\left(t\right)={\delta }_n\left(t\right)+\Lambda ,$$

(44)

where ${\delta }_n\left(t\right)={\sum }_{m=1}^M{\alpha }_{m,n}\left(t\right)\left[\left(B_m^{\mathrm{loc}}\left(t\right)+B_m^{\mathrm{off}}\left(t\right)\right)/\left(E_m^{\mathrm{loc}}\left(t\right)+E_m^{\mathrm{off}}\left(t\right)+E_n^{\mathrm{ref}}\left(t\right)+\omega E_n^{\mathrm{fly}}\left(t\right)\right)\right]$ is energy efficiency, and $\Lambda <0$ is the penalty if the constraint (20) is not satisfied, respectively.

The framework for the MATD3-based EEM-Outer algorithm consists of the environment entity, one actor network, two critic networks, and the experience replay buffer, as shown in Figure 2. The actor network is used to select the action based on the current state, and two critic networks are trained to evaluate the action policy. Two independent critic networks are used in the actor–critic framework by choosing the minimum value of two critic networks as the target Q-value to avoid the overestimation.

In the MATD3-based EEM-Outer algorithm, we take the agent n as an example to illustrate how to learn the action selection policy. Each agent has one actor and two critic networks. Let ${\theta }_n^{\mu }$ and ${\theta }_n^{{\mu }^{\prime }}$ denote the parameters of the online and the target actor networks, as well as ${\theta }_n^{{\nu }_1},{\theta }_n^{{\nu }_2}$ and ${\theta }_n^{{\nu }_1^{\prime }},{\theta }_n^{{\nu }_2^{\prime }}$ denoting the parameters of two online and two target critic networks. In time slot t, the agent n makes an action policy to select the action $a_n\left(t\right)$ based on its state $s_n\left(t\right)$, denoted as $a_n\left(t\right)={\mu }_n\left(s_n\left(t\right)\right|{\theta }_n^{\mu })$. Based on the Bellman equation, the Q-value function is denoted as $Q_{n,1}(s_n\left(t\right),a_n\left(t\right))=\mathbb{E}[r_n\left(t\right)+\zeta Q_{n,1}(s_n(t+1),a_n(t+1))]$.

(1) Critic network training and update: The online critic network is updated by minimizing the loss function as follows:

$$\begin{array}{c}\hfill L\left({\theta }_n^{{\nu }_j}\right)=\mathbb{E}\left[{\left(Q_{n,j}(s_n\left(t\right),a_n\left(t\right)|{\theta }_n^{{\nu }_j})-y_n\left(t\right)\right)}^2\right],j=1,2.\end{array}$$

(45)

In order to explore effective actions and achieve the accurate target value, a clipped random noise is added to choose the action as

$$a_n^{\prime }\left(t\right)=\mathrm{clip}({\mu }_n^{\prime }\left(s_n\left(t\right)\right|{\theta }_n^{{\mu }^{\prime }})+\mathrm{clip}(\mathcal{N}(0,ϵ),-d,d),a_{\min },a_{\max }),$$

(46)

where $\mathcal{N}(0,ϵ)$ is the normal distribution, and $\mathrm{clip}(·)$ is a clipping function to limit the selected action between the upper bound of values $a_{\max }$ and the lower bound of values $a_{\min }$. Then, the target value $y_n\left(t\right)$ produced by the target critic networks is calculated by

$$y_n\left(t\right)=r_n\left(t\right)+\zeta \underset{j=1,2}{min}{Q}_{n,j}^{\prime }\left(s_n(t+1),a_n^{\prime }(t+1)|{\theta }_n^{{\nu }_j^{\prime }}\right),$$

(47)

where $\zeta \in [0,1)$ denotes the discount factor, and $Q_{n,j}^{\prime }$ is the Q-value function obtained by the target critic network.

With the Adam optimizer, the parameters of the critic networks ${\theta }_n^{{\nu }_1},{\theta }_n^{{\nu }_2}$ are updated based on the policy gradient method as follows:

$$\begin{array}{c}\hfill {▽}_{{\theta }_n^{{\nu }_j}}L\left({\theta }_n^{{\nu }_j}\right)=\mathbb{E}[(Q_{n,j}(s_n\left(t\right),a_n\left(t\right)|{\theta }_n^{{\nu }_j})-y_n\left(t\right))\times {▽}_{{\theta }_n^{{\nu }_j}}{Q}_{n,j}(s_n\left(t\right),a_n\left(t\right)|{\theta }_n^{{\nu }_j})],j=1,2,\end{array}$$

(48)

$${\theta }_n^{{\nu }_j}={\theta }_n^{{\nu }_j}+{\epsilon }_{n,j}^{\mathrm{cri}}{▽}_{{\theta }_n^{{\nu }_j}}L\left({\theta }_n^{{\nu }_j}\right),j=1,2,$$

(49)

where ${▽}_{{\theta }_n^{{\nu }_j}}·$ denotes the gradient vector with the parameter ${\theta }_n^{{\nu }_j}$, and ${\epsilon }_{n,j}^{\mathrm{cri}}$ is the learning rate of the critic network.

(2) Actor network training and update: The parameter of the actor network is updated by using the deterministic policy gradient method as follows:

$$\begin{array}{c}\hfill {▽}_{{\theta }_n^{\mu }}J=\mathbb{E}[{▽}_{a_n\left(t\right)}{Q}_{n,1}(s_n\left(t\right),a_n\left(t\right)|{\theta }_n^{{\nu }_1}){|}_{a_n\left(t\right)={\mu }_n\left(s_n\left(t\right)\right|{\theta }_n^{\mu })}\times {▽}_{{\theta }_n^{\mu }}{\mu }_n\left(s_n\left(t\right)\right|{\theta }_n^{\mu })],\end{array}$$

(50)

$$\begin{array}{c}\hfill {\theta }_n^{\mu }={\theta }_n^{\mu }+{\epsilon }_n^{\mathrm{act}}{▽}_{{\theta }_n^{\mu }}J,\end{array}$$

(51)

where ${\epsilon }_n^{\mathrm{act}}$ denotes the learning rate of the actor network.

After updating the parameters of the online actor and critic networks, the parameters of the target actor and critic networks can be softly updated as

$${\theta }_n^{{\mu }^{\prime }}\leftarrow {\iota }_n^{\mathrm{act}}{\theta }_n^{\mu }+(1-{\iota }_n^{\mathrm{act}}){\theta }_n^{{\mu }^{\prime }},$$

(52)

$${\theta }_n^{{\nu }_j^{\prime }}\leftarrow {\iota }_{n,j}^{\mathrm{cri}}{\theta }_n^{{\nu }_j}+(1-{\iota }_{n,j}^{\mathrm{cri}}){\theta }_n^{{\nu }_j^{\prime }},j=1,2,$$

(53)

where ${\iota }_n^{\mathrm{act}}\in [0,1]$ is the soft update coefficient of the actor network, and ${\iota }_{n,j}^{\mathrm{cri}}\in [0,1]$ is the soft update coefficient of the critic network.

The process of the proposed MATD3-based EEM-Outer algorithm is summarized as follows. At the beginning of each episode, the state ${\mathbf{s}}_0$ is initiated. In time slot t, the action $a_n\left(t\right)$ of the nth agent is generated by the online network with the current state $s_n\left(t\right)$. After the actions are executed, the agent n obtains the corresponding reward $r_n\left(t\right)$ and transitions to the next state $s_n(t+1)$. Then, the agent n stores the transition $(s_n\left(t\right),a_n\left(t\right),r_n\left(t\right),s_n(t+1))$ into the experience replay buffer ${\mathbb{R}}_n$. In the training process, sample the mini-batch ${\mathbb{R}}_{n,\min }$ from ${\mathbb{R}}_n$. Based on the sampled mini-batch, the network parameters are updated by performing the policy gradient methods. In addition, the online actor network and target networks are updated every d step due to the delayed policy updates adopted to provide stable Q-value estimates [36].

The proposed CO-MATD3 algorithm is summarized in Algorithm 1.

Algorithm 1 CO-MATD3 algorithm for solving EEM problem.

Input:  The number of agents N, maximum number of episodes $\Gamma$, the number of time slots T, updating step d, mini-batch ${\mathbb{R}}_{n,\min }$, learning rate of the actor network ${\epsilon }_n^{\mathrm{act}}$, learning rate of critic networks ${\epsilon }_{n,1}^{\mathrm{cri}}, {\epsilon }_{n,2}^{\mathrm{cri}}$, reward discount factor $\zeta$, and soft update coefficients ${\iota }_n^{\mathrm{act}}, {\iota }_{n,1}^{\mathrm{cri}}, {\iota }_{n,2}^{\mathrm{cri}}$. Output:  Association relationships $\mathit{\alpha }$, computation resources $\mathbf{f}$, UIRS phase shifts $\mathit{\phi }$, and positions of UIRSs $\mathbf{L}$.   1: for agent $n\in \{1,\dots ,N\}$ do   2:       Initialize network parameters ${\theta }_n^{\mu }, {\theta }_n^{{\nu }_1}, {\theta }_n^{{\nu }_2}, {\theta }_n^{{\mu }^{\prime }}, {\theta }_n^{{\nu }_1^{\prime }}, {\theta }_n^{{\nu }_2^{\prime }}$;   3:       Initialize the experience replay buffer ${\mathbb{R}}_n$;   4: end for   5: for episode $e\in \{1,2,\dots ,\Gamma \}$ do   6:       Initialize the state $\mathbf{s}\left(0\right)$;   7:       for time slot $t\in \{1,2,\dots ,T\}$ do   8:             Given $\mathbf{s}(t-1)$, obtain $\mathit{\alpha }\left(t\right)$ by solving (38);   9:             for agent $n\in \{1,\dots ,N\}$ do 10:                   Select the action $a_n\left(t\right)={\mu }_n\left(s_n\left(t\right)\right|{\theta }_n^{\mu })$; 11:                   Obtain reward $r_n\left(t\right)$ and transition to next state $s_n(t+1)$; 12:                   Store transition $(s_n\left(t\right),a_n\left(t\right),r_n\left(t\right),s_n(t+1))$ into ${\mathbb{R}}_n$; 13:                   Randomly sample the mini-batch ${\mathbb{R}}_{n,\min }$ from ${\mathbb{R}}_n$; 14:                   Select the target action $a_n^{\prime }\left(t\right)$ by solving Equation (46); 15:                   Update critic networks by solving Equations (48) and (49); 16:                   if t $mod$ d == 0 then 17:                         Update the actor network by using Equations (50) and (51); 18:                         Update parameters of target networks by solving Equations (52) and (53); 19:                   end if 20:            end for 21:        end for 22: end for 23: Return $\mathit{\alpha }$, $\mathbf{f}$, $\mathit{\phi }$, and $\mathbf{L}$.

4. Numerical Results

In this section, we develop a simulator in Python to evaluate the performance of the proposed CO-MATD3 algorithm. The deep reinforcement learning algorithm based on MATD3 is implemented in PyTorch. In our simulations, we consider that 15 USVs are randomly distributed in an ocean area of 3000 m × 3000 m. For the rotary-wing UAV, we set $U_{\mathrm{tip}}=120$, $d_0=0.6$, $\rho =1.225$, $S_0=0.05$, $A_0=0.503$, $v^{\prime }=4.3$, $P_0=79.856$, $P_1=88.628$, and $P_2=11.46$ [35,37,38]. The simulation parameters are listed in

Table 1. Simulation parameters.

Parameter	Value
The number of reflecting elements on the UIRS n, $K_n$	10
Time slot length, $\tau$	1 s
Channel bandwidth, W	1 MHz
Channel gain at the reference distance, ${\rho }_0$	$-60$ dB
Noise power, ${\sigma }^2$	$-114$ dBm
The Rician factor, ${\xi }_{m,n},{\xi }_{n,0}$	$31.3$ [6]
The antenna height of the USV m, $H_m$	5 m
The antenna height of the shore BS, $H_0$	35 m
Transmission power of the USV m, $p_m$	1 W
The maximum computation resources of the USV m, $f_m^{\max }$	1 GHz
The CPU cycles required to process one bit of USV m, $C_m^{\mathrm{loc}}$	1000 cycles/bit
The effective switched capacitance of the USV m, ${\kappa }_m$	${10}^{-27}$
The power of each reflecting element on the UIRS n, $p_{n,k}$	$0.001$ W [34]

.

We implement four benchmarks for performance comparison as follows:

• Convex Optimization and Single-Agent TD3 (CO-SATD3). In this scheme, the SATD3 algorithm aims to optimize the EEM-Outer problem, while the EEM-Inner problem is optimized by Section 3.1. For the SATD3 algorithm, the shore BS acts as the agent to centrally manage the state and action information of USVs and UIRSs.
• Random phase shifts scheme. In this scheme, the phase shifts of the reflection elements are randomly selected in the constraint range.
• Without IRS scheme. This scheme involves a multi-UAV-assisted marine vehicle system without an IRS, in which each UAV acts as a decode-and-forward relay node to achieve data transmission from USVs to the shore BS.
• Full offloading to BS scheme. In this scheme, the computation tasks of USVs are fully offloaded to the shore BS for edge computing, while the USVs cannot process computation tasks locally.

Figure 3a and Figure 3b show the three UIRS trajectories in 3D and 2D obtained by the proposed CO-MATD3, respectively. It can be observed that three UIRSs increase the flight height, and the flight height of the UIRS 3 is slightly higher than the others, as shown in Figure 3a. This is because the high flight altitude can increase the LoS link probability and reduce path loss to improve the channel quality, especially when the UIRS 3 is far away from the BS. After being trained, the UIRSs have learned to collaborate with each other to provide wireless communication services for USVs. In addition, the UIRSs have the capability to identify areas with a concentration of USVs and move gradually to conserve energy consumption while improving the computation bits. It is worth noting that the initial locations of UIRSs are default locations, and the trajectories of UIRSs are not periodic because the deployment locations of UIRSs need to accommodate the number of UIRSs, distribution of USVs, and offloading workloads of USVs.

Figure 4 illustrates the impact of different learning rates on accumulated rewards obtained by our proposed CO-MATD3 when deep neural networks have three hidden layers with $(512,256,128)$ neurons. Both the actor network’s learning rate ${\epsilon }_n^{\mathrm{act}}$ and critic network’s learning rate ${\epsilon }_n^{\mathrm{cri}}$ are set to $0.01$, $0.001$, $0.0001$, $0.00001$, and $0.000001$. The results show that when the learning rate is $0.0001$, the accumulated reward is higher than other learning rates.

The convergence performance comparison between CO-MATD3 and CO-SATD3 is shown in Figure 5 when $N=3$ and $M=15$. It can be seen that the proposed CO-MATD3 algorithm requires about 3000 episodes to converge properly, while the CO-SATD3 algorithm requires about 4000 episodes to converge. In addition, the final accumulated reward of the CO-MATD3 is slightly higher than that of the CO-SATD3. The reason is that the RMTD scheme based on the CO-MATD3 algorithm can stimulate multiple agents to achieve a maximum reward, due to the fact that each agent shares its experience and learns other agents’ policies based on distributed training to maximize the accumulated reward. However, the neural network of the CO-SATD3 requires the concatenation state of the center agent to construct the global state information, which makes the optimization problem solved by the CO-SATD3 more challenging. Such an observation shows that the multi-agent deep reinforcement learning algorithm is an efficient and valuable solution to achieve faster convergence and better performance.

Figure 6 presents the energy efficiency performance of the proposed CO-MATD3 with that of the four benchmarks versus different numbers of UIRSs when $M=15$. It can be observed that in the optimization schemes with the UIRS assistance (i.e., CO-MATD3, CO-SATD3, random phase shifts, and full offloading to BS), as the number of UIRSs increases, the energy efficiency increases at first. It is worth emphasizing that the case of multiple UIRSs outperforms that of the single UIRS in terms of energy efficiency. Then, when the number of UIRSs exceeds three, the energy efficiency performance begins to decline. The reason is that as the number of UIRSs increases, the transmission rate improves, but the energy consumption of the UIRSs correspondingly increases significantly. In this context, the growth in energy consumption outweighs the increase in total computation bits, leading to a decline in computational efficiency. These findings emphasize the importance of an optimal number of UIRSs in achieving energy efficiency performance. Moreover, we can see that the superiority of the UIRS assistance scheme over the without IRS scheme, which can provide more effective communication and computation services. Note that the number of UIRSs represents the number of UAVs employed in the without IRS scheme. Notably, the benchmarks of random phase shifts and full offloading are worse than our proposed CO-MATD3. This is because poor channel conditions and unreasonable computation offloading inevitably waste communication and computation resources, resulting in low energy efficiency. The above results indicate that the assistance of multiple UIRSs, offloading design, and phase shift optimization have degrees of impact on energy efficiency.

Figure 7 presents the energy efficiency performance of the proposed CO-MATD3 with that of the four benchmarks versus different numbers of USVs when $N=3$. We can see that the proposed CO-MATD3 outperforms the CO-SATD3, random phase shifts, without IRS, and full offloading to BS, which demonstrates the effectiveness of our proposed CO-MATD3 in the optimization of resource management and trajectory optimization. According to the results from Figure 7, the performance of the IRS assistance schemes (i.e., CO-MATD3, CO-SATD3, random phase shifts, and full offloading to BS) is always better than the without IRS scheme, which indicates the employment of IRS can achieve better energy efficiency. In addition, the energy efficiency gap between the proposed CO-MATD3 and the full offloading scheme increases as the number of USVs increases, which demonstrates the advantages of collaborative local computation and computation offloading. Compared with the CO-SATD3, our proposed MATD3 can obtain better energy efficiency, especially for $N>15$ USVs. The reason is that the CO-SATD3 algorithm with central training is much more complex due to the global state construction. Therefore, our proposed CO-MATD3 can better optimize multi-dimensional resources.

Figure 8 studies the impact of the maximum computation resources on the performance of the proposed CO-MATD3 with that of the four benchmarks when $N=3$ and $M=15$. In Figure 8a, the graph reveals a trend in which energy efficiency improves when the computation resources increase from $0.2$ GHz to 1 GHz, and the increase rate becomes much slower. Intuitively, the high computation resources of the USV can improve the local computation rate, which increases the total computation data, as shown in Figure 8b. However, the USV increases the computation resources to improve the local computation rate; it will undoubtedly consume more local computation energy consumption. In this context, although the maximum computation resources of the USV increase, the energy efficiency will not be notably varied when the computation resources are optimized to a certain value. To sum up, the optimization of the computation resources of the USV can effectively increase energy efficiency and total computation data.

Figure 9 studies the impact of the transmission power on the performance of the proposed CO-MATD3 with that of the four benchmarks when $N=3$ and $M=15$. It is evident that the performance of all schemes is enhanced with the increased transmission power of the USV. Especially, we can see that the performance of the proposed CO-MATD3 outperforms benchmarks. As shown in Figure 9a, energy efficiency maintains an upward trend as the transmission power of USV increases. In Figure 9b, as the transmission power of USV increases, the total computation data increases, which can consume large energy consumption simultaneously. At this time, the increase in computation data is more than the consumed energy, and as a result, energy efficiency improves. However, the trend of improving energy efficiency is decreasing. This finding shows when the transmission power of the USV reaches the threshold, energy efficiency will not be notably increased. The reason is that our proposed scheme focuses on energy efficiency maximization rather than computation data maximization.

In this work, energy efficiency is not only affected by the UIRSs’ flight energy consumption but is also related to the communication and computation energy consumption, as well as the total computation bits, which is a complex optimization problem. According to Equation (15), the flight energy consumption is affected by the flying velocity and the length of the slot time. As mentioned in [39], the flight energy consumption reaches its minimum when the flying velocity is approximately $10.2$ m/s during each equal length of slot time. In this work, we aim to improve computation efficiency by increasing computation bits while reducing energy consumption. Therefore, the UIRS is set to move in a straight path at a constant velocity of $10.2$ m/s, which corresponds to the minimum flight energy consumption. This flight scheme is used as a baseline for comparison. Otherwise, we also choose the constant velocity scheme and random flight scheme as a comparison to analyze the UIRS mobility (including flight velocity and angle) impact on energy efficiency.

To evaluate the effectiveness of the UIRS mobility optimization, we use the five different UIRS mobility schemes, where “10 m/s”, “20 m/s”, and “30 m/s” indicate the UIRSs moving at a constant velocity with an optimized angle, “random” indicates the UIRSs moving at arbitrary velocities and angles, and “$10.2$ m/s” indicates the UIRSs moving in a straight flight at a constant velocity. We compare the impact of UIRS mobility on energy efficiency in Figure 10. It can be seen that our proposed scheme with an optimized velocity of the UIRS is better than the other five schemes in terms of energy efficiency. The reason is that our proposed scheme optimizes the velocities and angles of UIRSs according to the requirements of computation offloading to arrive at the optimal positions and obtain the satisfied communication links at each time slot, which ensures the high total computation bits and does not consume too much flight energy consumption. These simulation results demonstrate that the proposed solution can dynamically adjust the velocities and angles of UIRSs to achieve a better trade-off between computation bits and energy consumption.

5. Conclusions

In this paper, we have proposed a resource management and trajectory design scheme for multi-UIRS-assisted marine vehicle systems. Firstly, an energy efficiency maximization problem has been formulated by jointly optimizing the association relationships, computation resources, multi-UIRS phase shifts, and multi-UIRS trajectories. Then, the CO-MATD3 algorithm has been proposed to solve the challenging optimization problem. Specifically, the Dinkelbach method and relaxation method are applied to optimize the discrete variables in the inner layer, and learning-based MATD3 is employed to optimize the continuous variables in the outer layer. Finally, the numerical results have shown that the proposed CO-MATD3 can achieve outstanding overall performance in terms of training convergence and energy efficiency.