Efficient Asynchronous Federated Learning for AUV Swarm

Abstract

The development of automatic underwater vehicles (AUVs) has brought about unprecedented profits and opportunities. In order to discover the hidden valuable data detected by an AUV swarm, it is necessary to aggregate the data detected by AUV swarm to generate a powerful machine learning model. Traditional centralized machine learning generates a large number of data exchanges and faces problems of enormous training data, large-scale models, and communication. In underwater environments, radio waves are strongly absorbed, and acoustic communication is the only feasible technology. Unlike electromagnetic wave communication on land, the bandwidth of underwater acoustic communication is extremely limited, with the transmission rate being only $1/{10}^5$ of the electromagnetic wave. Therefore, traditional centralized machine learning cannot support underwater AUV swarm training. In recent years, federated learning could only interact with model parameters without interacting with data, which greatly reduced communication costs. Therefore, this paper introduces federated learning into the collaboration of an AUV swarm. In order to further reduce the constraints of underwater scarce communication resources on federated learning and alleviate the straggler effect, in this work, we designed an asynchronous federated learning method. Finally, we constructed the optimization problem of minimizing the weighted sum of delay and energy consumption, relying on jointly optimizing the AUV CPU frequency and signal transmission power. In order to solve this complex optimization problem of high-dimensional non-convex time series accumulation, we transformed the problem into a Markov decision process (MDP) and use the proximal policy optimization 2 (PPO2) algorithm to solve this problem. The simulation results demonstrate the effectiveness and superiority of our method.

1. Introduction

An automatic underwater vehicle (AUV) is a kind of submarine robot that can carry out ocean sampling activities independently; it is widely used in underwater research and the marine industry. For example, AUVs have been used to sample coastal frontiers, monitor coastal areas, measure thermocline turbulence, obtain interdisciplinary data, conduct fishery research, and install submarine cables under the frozen sea [1,2]. However, due to the limited capacity of a single AUV, processing more complex tasks by AUV swarm has become a hot research topic. In recent years, underwater research based on the AUV group has become more popular [3,4]. The advantages of the AUV swarm cooperation over a single AUV can be summarized into two points. On the one hand, an AUV swarm can obtain more data than a single AUV, such as underwater hydrological characteristic modeling, on the other hand, an AUV swarm can perform tasks that cannot be completed by a single AUV, such as rounding-up targets, formation cruising, collaborative positioning, etc. In the above two aspects, it is often critical to establish a powerful machine learning model to mine the values behind the observed data, or to improve the effectiveness of the AUV’s own actions. Specifically, for the sake of exploring the hidden valuable information of the data detected by the AUV swarm, it is necessary to aggregate the data detected by the AUV swarm to generate a powerful machine learning model, such as forming the hydrological characteristics model of the water area by data fitting. Moreover, in order to improve the movement of the AUV, it is a requisite to train the AUV to obtain a strong reinforcement learning model to achieve efficient autonomous action of the AUV. In order to support the above two goals, traditional centralized machine learning needs to interact with a large number of data, which will require a lot of communication resources. However, unlike on land, underwater electromagnetic wave communications with high bandwidths cannot be used; they often rely on underwater acoustic communication for data transmission. The bandwidth of underwater acoustic communication is very limited, and the transmission rate is only $1/{10}^5$ of the electromagnetic wave. Therefore, traditional centralized machine learning cannot support underwater AUV swarm training and the production of large models. Federated learning, a new machine learning framework, was first proposed by Google in 2016 [5]. Each data holder uses his/her own data to train a model, and the models interact with each other. Finally, a global model is obtained through model aggregation. Unlike traditional centralized machine learning, federated learning can only interact model parameters without interacting with data, which can greatly reduce communication overhead and reduce dependence on communication resources, and is suitable for the environment where underwater communication resources are scarce. Therefore, this paper introduces federated learning into the collaboration of the AUV swarm. Furthermore, considering that the interaction model parameters are much less than the interactions with the original data, there is still a large number of data, and the traditional federated learning method (i.e., FEDAVG proposed by Google [5]) is synchronized, which is prone to the emergence of the straggler effect, especially in unreliable underwater information transmission and large time delays. Therefore, this paper designs an asynchronous federated learning method to further reduce data transmission and reduce the straggler effect, which will play an important role in the harsh underwater environment. Furthermore, considering the limited energy of the AUV and the time requirement of model training, we constructed an optimization problem of minimizing the weighted sum of delay and energy consumption, relying on jointly optimizing AUV CPU frequency and signal transmission power. In order to solve this complex optimization problem of high-dimensional non-convex time series accumulation, we built the problem into an MDP and used the PPO2 algorithm to solve this problem. The main contributions are as follows:

• To the authors’ knowledge, this paper is the first to introduce federated learning into the AUV swarm. Federated learning can help AUV swarm train large-scale machine learning models in an environment where underwater communication resources are scarce.
• In order to further reduce the constraints of underwater scarce communication resources on federated learning, we designed an asynchronous federated learning method, which can effectively alleviate the straggler effect and further reduce data interaction.
• We constructed the optimization problem of minimum delay and energy consumption, by jointly optimizing AUV CPU frequency and signal transmission power.
• In order to solve the optimization problem efficiently, we converted it into an MDP and proposed the PPO2 algorithm to solve this problem. The simulation results verify the effectiveness of the proposed algorithm.

The content and organizational structure of this article are as follows:

• In the third section, a federated learning model based on the AUV swarm is established. First, we established a federal learning model. Second, the communication system model was established. Then the control model was established, the nodes participating in this round of upload were selected locally, and redundant nodes were adaptively skipped. Then a delay model was established to calculate the time and total time of each phase of federated learning. Then the energy consumption model was established as the energy constraint. Finally, the optimization problem is listed.
• In the fourth section, the PPO2 algorithm is used to solve the optimization problem.
• The fifth section is the experimental part. By changing the relevant parameters, we can observe the communication times, model accuracy, and model The simulation results show the convergence of the PPO2 algorithm.
• The last section of this paper is the summary, which summarizes the main points and shortcomings of this work.

2. Related Work

The bottleneck limiting the performance of the AUV swarm mainly lies in the difficulties of underwater communications [6]. When AUV groups are used for federated learning tasks, as the scale of task models and AUV groups expand, the pressure on the self-organizing data interaction network increases, which requires designing algorithms to save communication resources as much as possible. However, reducing communication resources excessively may lead to compromised federated learning performance and confusion about the AUV formations, which creates a paradox. How to make a trade-off between resource consumption and model performance is worth studying at present.

AUV swarm has the advantages of rapid deployment, controllable action, flexible networks, and other advantages because of its formation, dynamic, mobility, and other characteristics. In recent years, the research into underwater exploration systems based on multiple AUVs or AUV groups has become popular, especially in navigation or path planning [7], collaborative data collection [8], target hunting [9], and so on. For example, in the offshore oil and gas industry, AUVs equipped with underwater environment monitoring sensors cooperate to effectively detect the boundary of oil and gas producing areas. Cui et al. [10] proposed an adaptive path planning algorithm based on the random tree star algorithm to estimate the scalar field underwater for the AUV group. Noguchi et al. [11] put forward a dynamic task allocation and path planning algorithm for the AUV group, which gives full play to the advantages of the AUV group. Huang et al. [12] used the AUV group equipped with scanning imaging sonar, and proposed using the binary Bayesian filter to process the input signal. Then, a path-planning strategy based on the artificial potential field was proposed for real-time target tracking. Cao et al. [13] used the artificial potential field theory to model the underwater environment information and introduce the potential field function into the dispersion, similarity, and difference of the potential field function. On this basis, an intelligent path planning model that can achieve accurate AUV swarm collaborative target search is proposed.

In the federated learning aggregation algorithm, each participant needs to completely send local parameters to the server for model aggregation. These model parameters tend to have a lot of data, which brings huge communication overhead. In [14], the author protects the privacy of the federated learning architecture through covert communication technology, minimizes the federated learning delay by optimizing the interference power, signal power, and training accuracy, and uses an alternating descent algorithm to solve the optimization problem. In [15], the author uses federated learning to build a digital twin model based on the digital twin edge network architecture, uses an asynchronous model update scheme, and performs device selection locally, minimizing channel allocation, and CPU frequency as variables. The federated learning delay uses the artificial intelligence method DNN (deep neural network) model to dynamically solve the optimization problem. In [16], the author selects some clients to send their local model parameters to the server, avoids the training loss, and convergence time being affected by a reasonable local selection strategy, and establishes an optimization problem to optimize the convergence time and training loss, using ANN (artificial neural network) to solve optimization problems. In [17], the author defines a new performance evaluation criterion, i.e., learning efficiency, which is discussed in scenarios where the device is equipped with CPU and GPU, respectively, e.g., channel resource allocation and bandwidth allocation as optimization variables were used to maximize learning efficiency via the Lagrange multiplier method, KKT (Karush–Kuhn–Tucker) conditions. A two-dimensional search algorithm was used to solve the optimization problem and detect and skip some redundant communication rounds locally with adaptive communication rules; a comparable convergence speed was used to the original SGD.

In [18], the authors proposed a communication-saving method for distributed machine learning, which is a gradient-based selection method that reduces redundant communication rounds. In [19], the author introduced the above method [18] into federated learning, based on the traditional SGD (stochastic gradient descent) method. In [20], the author delves into the basis of the above work, and used the gradient sparse method to select redundant nodes locally to reduce the communication amount. The work of this paper continues on this basis. So far, there is no work to introduce federal learning into AUV swarm.

3. System Model

This paper considers the following scenario: a fixed-form swarm consisting of a leader AUV L and a set $\mathcal{M}$ of the M follower AUV sailing at the same depths with constant speed, collecting seabed data, cooperating, and using federated learning to perform various machine learning tasks, such as target recognition, path planning, etc. Each AUV maintains the same distance from other AUVs and navigators, collecting data with their own independent sensors, and training local models. The local model parameters are then sent to the navigator using the uplink. The leader AUV uses the information uploaded by the follower to aggregate into global model parameters and sends it to the follower using the downlink. The follower proceeds with a new round of local training using the received global model parameters. In order to keep AUVs in formation, AUV L and AUV m also need to transfer position information and speed information to each other. In each round, AUV m uploads its current speed and position data to AUV L. AUV L calculates and broadcasts the subsequent speed and direction from the received information.

3.1. Federated Learning Model

Assuming that the input data collected by the follower AUV $m\in \mathcal{M}$ is ${\mathcal{X}}_m$, the output data are ${\mathcal{Y}}_m$, and the local model parameter is ${\mathit{w}}_m$, then the data set of the AUV m can be expressed as ${\mathcal{D}}_m=\{({\mathit{x}}_{m,1},{\mathit{y}}_{m,1}),({\mathit{x}}_{m,2},{\mathit{y}}_{m,2}),\dots ,({\mathit{x}}_{m,N_m},{\mathit{y}}_{m,N_m})\}$, where $N_m$ is the number of the data that AUV m owns, the loss function on its data set ${\mathcal{D}}_m$ can be expressed as average of sample loss functions:

$$F_m\left({\mathit{w}}_m\right)=\frac{1}{N_m}\sum _{i=1}^{N_m}f\left({\mathit{w}}_m;{\mathit{x}}_{m,i},{\mathit{y}}_{m,i}\right),\forall m\in \mathcal{M}$$

(1)

Assuming that the global model parameter is $\mathit{w}$, for an AUV $m\in \mathcal{M}$ containing ${\mathcal{N}}_m$ data, the corresponding local loss function is weighted, averaged, and defined as the global loss function:

$$F\left(\mathit{w}\right)\triangleq \sum _{m=1}^M\frac{N_m{F}_m\left(\mathit{w}\right)}{N}=\frac{1}{N}\sum _{m=1}^M\sum _{i=1}^{N_m}f\left({\mathit{w}}_m;{\mathit{x}}_{m,i},{\mathit{y}}_{m,i}\right),$$

(2)

The purpose of federated learning is to find a parametric model that minimizes the global loss function, i.e., the optimal parametric model can be expressed as

$${\mathit{w}}^{*}=\arg \min F\left(\mathit{w}\right),$$

(3)

where $\mathit{w}={\mathit{w}}_1=\cdots ={\mathit{w}}_M$,

3.2. Communication Model

In the underwater environment, electromagnetic waves and optical signals attenuate quickly and travel over short distances. Therefore, the most commonly used underwater communication is underwater acoustic communication. The attenuation of the water signal with frequency f at distance D can be given by

$$A(D,f)=l^{k}a{\left(f\right)}^D$$

(4)

where k represents the spreading factor, and $a\left(f\right)$ is the absorption coefficient, which can be expressed by the following empirical formula [21]:

$$10loga\left(f\right)=\frac{0.11f^2}{1+f^2}+\frac{44f^2}{4100+f^2}+2.75·{10}^{-4}{f}^2+0.003.$$

(5)

In the marine environment, noise is mainly divided into turbulence noise, shipping noise, wave noise, and thermal noise, respectively. According to [22], the power spectral density (p.s.d) of four main types of noise in dB re $\mathsf{\mu }$Pa per Hz on the communication frequency f can be given by:

$$10log{N}_{\vartheta }\left(f\right)=17-30logf$$

(6)

$$10log{N}_s\left(f\right)=40+20\left(s-\frac{1}{2}\right)+26logf-60log(f+0.03)$$

(7)

$$10log{N}_w\left(f\right)=50+7.5w^{\frac{1}{2}}+20logf-40log(f+0.4)$$

(8)

$$10log{N}_{th}\left(f\right)=-15+20logf$$

(9)

Moreover, $s\in [0,1]$ is the shipping activity factor, while w represents the wind velocity ($\mathrm{m}/\mathrm{s}$). Hence, the combined noise $N\left(f\right)$ can be represented as

$$N\left(f\right)=N_{\vartheta }\left(f\right)+N_s\left(f\right)+N_w\left(f\right)+N_{th}\left(f\right)$$

(10)

Therefore, the normalized SNR of a signal with unit transmitted power and bandwidth can be represented as

$$\gamma (D,f)=\frac{1}{A(D,f)N\left(f\right)}.$$

(11)

The uplink data rate between AUV m and leader AUV L is given by

$$R_m^{\mathrm{U}}=B_m^{\mathrm{U}}{log}_2(1+\frac{p_m\gamma (D,f)}{B_m^{\mathrm{U}}}),$$

(12)

where $B_m^{\mathrm{U}}$ is the allocated uplink bandwidth of the AUV m, while $p_m\in \left(0,p_{max}\right)$ is the transmitting power of the AUV m.

In the iteration, the leader AUV broadcasts the information to the followers and, hence, the downlink data rate between AUV m and leader AUV L can be represented as

$$R_m^{\mathrm{D}}=B^{\mathrm{D}}{log}_2(1+\frac{p_L\gamma (D,f)}{B^{\mathrm{D}}})$$

(13)

where $B^D$ is the downlink bandwidth, $p_L\in \left(0,p_{max}\right)$ denotes the transmitting power of leader AUV.

3.3. Control Model

Due to the high cost of underwater communication, most gradient interactions in the distributed SGD are redundant. Therefore, this paper introduces the concept of lazy nodes, allowing each follower drone to perform self-detection locally so that some nodes skip some rounds of communication. The definition of a lazy node in this paper refers to the work of Chen et al. [19].

$$\frac{{∥{\nabla }_{{\mathcal{M}}_N}^{t-1}∥}^2}{M_N}⩽\frac{{∥{\nabla }_{\mathcal{M}}^{t-1}∥}^2}{M}$$

(14)

Among them, these lazy nodes constitute a set ${\mathcal{M}}_N$ of size $M_N$. In this paper, the gradient descent algorithm is used to optimize the global model, where $\gamma$ represents the learning rate

$$\mathit{w}\left(t\right)=\mathit{w}(t-1)-\gamma {\nabla }_{\mathcal{M}}^{t-1}$$

(15)

$${\parallel {\nabla }_{\mathcal{M}}^{t-1}\parallel }^2⩽\frac{M_N}{{\gamma }^{2}M}{\parallel \mathit{w}\left(t\right)-\mathit{w}(t-1)\parallel }^2,$$

(16)

Since the global model tends to converge, the following approximation is used

$$\mathit{w}\left(t\right)-\mathit{w}(t-1)\approx \mathit{w}(t-1)-\mathit{w}(t-2)$$

(17)

According to the mean inequality, we have

$${\parallel {\nabla }_{{\mathcal{M}}_{\mathrm{N}}^{t-1}}\parallel }^2={\parallel \sum _{m\in {\mathcal{M}}_{\mathrm{N}}}\frac{N_m{\nabla }_m^{t-1}}{N}\parallel }^2⩽\frac{M_{\mathrm{N}}}{N^2}\sum _{m\in {\mathcal{M}}_{\mathrm{N}}}{∥N_m{\nabla }_m^{t-1}∥}^2$$

(18)

Let $M_N=\beta M$, Equation (14) can be deduced as Equation (18)

$${\parallel N_m{\nabla }_m^{t-1}\parallel }^2⩽\frac{N^2}{{\gamma }^2{M}^2\beta }{\parallel \mathit{w}(t-1)-\mathit{w}(t-2)\parallel }^2.$$

(19)

At one round t, AUV m checks locally whether Equation (18) is satisfied. If satisfied, skip this round of uploading, otherwise participate in this round of uploading. Considering the extreme case, if all nodes in a certain round t satisfy Equation (18), then AUV L cannot receive the model information uploaded by the follower AUV. At this time, AUV L randomly selects some follower drones to participate in the upload after a certain time interval. In this paper, we randomly select 1 AUV m to participate in uploading in extreme cases.

In order to prevent AUVs from not communicating for a long time due to the setting of lazy nodes at certain times, we require that AUVs communicate at least once within $\tau$ round. Each AUV locally records whether it uploads itself in each round of uploading

$${\mathrm{\Lambda }}_m\left(\tau \right)=\{{\lambda }_m(t-\tau ),{\lambda }_m(t+1-\tau ),\dots ,{\lambda }_m(t-1)\}$$

(20)

where ${\lambda }_m\left(t\right)\in \{0,1\}$ indicates whether AUV m participates in the t round of gradient upload after controlling the model, ${\lambda }_m\left(t\right)$ = 1 indicates that it participates in this round of the gradient upload, otherwise not.

At one round t, AUV m checks locally whether Equation (20) is all of 0. If so, AUV m must be uploaded in this round.

3.4. Latency Model

3.4.1. Local Parameter Calculating Latency

Let $f_m\in \left(f_{\min },f_{\max }\right)$ and $f_L\in \left(f_{\min },f_{\max }\right)$ denote the CPU frequency of the AUV m and leader AUV L, respectively, where $f_{\min }$ and $f_{\max }$ represent the minimum and the maximum CPU frequency, respectively.

The local computation latency on each AUV m at the t-th slot can be calculated as

$$T_m^{\mathrm{LC}}\left(t\right)=\frac{N_m{c}_m}{f_m\left(t\right)},$$

(21)

where $c_m$ is the required CPU cycles for training one sample data by backpropagation algorithm [17] at AUV m.

3.4.2. Uploading Latency

Assume that the local parameter has a counterpart gradient consists of $\alpha$ elements, and each of them has an average quantitative bit number denoted by $\zeta$. The size of the current position information and speed information is $\psi$. Hence, the total data size of each local parameter is $\left|{\mathit{w}}_m\right|=\alpha \zeta$ [17], uploading latency is given by

$$T_m^{\mathrm{LU}}\left(t\right)=\frac{\left|{\mathit{w}}_m\right|{\lambda }_m\left(t\right)}{R_m^{\mathrm{U}}}+\frac{\psi {\lambda }_m\left(t\right)}{R_m^{\mathrm{U}}}$$

(22)

3.4.3. Global Parameter Aggregating Latency

At the leader AUV L, the global parameter aggregating latency is calculated by

$$T_L^{\mathrm{GA}}\left(t\right)=\frac{c_0{\sum }_{m=1}^M\left|{\mathit{w}}_m\right|}{f_L\left(t\right)},$$

(23)

where $c_0$ is the computational complexity [15] to aggregate the newly updated parameters from follower AUVs, while $\left|{\mathit{w}}_m\right|$ is the data size of the local parameter ${\mathit{w}}_m$.

3.4.4. Global Parameter Updating Latency

The local parameter updating latency is given by

$$T_L^{\mathrm{GU}}\left(t\right)=\frac{c_L^{\prime }}{f_L\left(t\right)},$$

(24)

where $c_L^{\prime }$ is the computational complexity for performing global parameter updating [17].

3.4.5. Downloading Latency

The downloading latency can be calculated as

$$T_L^{\mathrm{GD}}\left(t\right)=\frac{\left|\mathit{w}\right|}{R_m^{\mathrm{D}}}+\frac{\psi }{R_m^{\mathrm{D}}},$$

(25)

where $\left|\mathit{w}\right|$ is the data size of the global parameter, which is basically similar to the local parameter $\left|{\mathit{w}}_m\right|$.

3.4.6. Total Latency

Regardless of extreme cases, there are always devices uploaded in a round, a complete federated learning delay is

$$T\left(t\right)=\underset{m\in \mathcal{M}}{max}\left\{T_m^{\mathrm{LC}}+T_m^{\mathrm{LU}}\right\}+\underset{m\in \mathcal{M}}{max}\left\{T_L^{\mathrm{GA}}+T_L^{\mathrm{GU}}+T_L^{\mathrm{GD}}\right\}.$$

(26)

Consider the extreme case where there are no devices uploaded within the round. According to the settings of the control model, when the AUV L does not receive the gradient upload after the time interval $T^{\prime }$, a device is randomly selected for uploading. Assuming that the selected device is $m_0$, then the federated learning delay for this round is

$$T\left(t\right)=T_{m_0}^{\mathrm{LC}}+T_{m_0}^{\mathrm{LU}}+\underset{m\in \mathcal{M}}{max}\{T_L^{\mathrm{GA}}+T_L^{\mathrm{GU}}+T_L^{\mathrm{GD}}\}+T^{\prime }.$$

(27)

3.5. Energy Consumption Model

3.5.1. Energy Consumption of Follower AUV

The computational energy consumption of the follower AUV is mainly composed of a local parameter calculating and local parameter updating, which can be given by

$$E_m^{\mathrm{Cp}}\left(t\right)=k{f}_m^{\sigma }\left(T_m^{\mathrm{LC}}\left(t\right)\right),$$

(28)

The communication energy consumption is mainly the local parameter uploading, and it can be calculated by

$$E_m^{\mathrm{C}}\left(t\right)=p_m{T}_m^{\mathrm{LU}}\left(t\right),$$

(29)

3.5.2. Energy Consumption on Leader AUV

The computational energy consumption of the leader AUV is mainly caused by a global parameter aggregating, which is given as

$$E_L^{\mathrm{Cp}}\left(t\right)=k{f}_L^{\sigma }(T_L^{\mathrm{GA}}\left(t\right)+T_L^{\mathrm{GU}}\left(t\right)),$$

(30)

Similarly, the communication energy consumption of leader AUV is mainly due to the global parameter downloading, which can be calculated by

$$E_m^{\mathrm{C}}\left(t\right)=p_L{T}_L^{\mathrm{GD}}\left(t\right),$$

(31)

3.5.3. Total Energy Consumption

$$E\left(t\right)=\Phi (E_m^{\mathrm{Cp}}\left(t\right)+E_L^{\mathrm{Cp}}\left(t\right))+\chi (E_m^{\mathrm{C}}\left(t\right)+E_m^{\mathrm{C}}\left(t\right))$$

(32)

3.6. Problem Formulation

Our goal is to minimize the total cost by optimizing the variables $p_m,f_m,p_L,f_L,\beta$, which are defined as follows.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{P}1:& \underset{p_m,f_m,p_L,f_L,\beta }{min}\mathrm{Cost}=\sum _{t=1}^{N_t}(T\left(t\right)+E\left(t\right))\hfill \end{array}\end{array}$$

(33)

$$0⩽p_m⩽p_{max},m\in \mathcal{M},$$

(34)

$$f_{min}⩽f_m⩽f_{max},m\in \mathcal{M},$$

(35)

$$0⩽p_L⩽p_{max},$$

(36)

$$f_{min}⩽f_L⩽f_{max},$$

(37)

$$0⩽\beta ⩽1,$$

(38)

$$E_m^{\mathrm{Cp}}+E_m^{\mathrm{C}}⩽E_m^{\mathrm{thd}},m\in \mathcal{M},$$

(39)

$$E_L^{\mathrm{Cp}}+E_L^{\mathrm{C}}⩽E_L^{\mathrm{thd}},$$

(40)

Among them, $N_t$ is the total number of iterations. Equation (33) is the optimization objective, and the total time of each round is minimized by the optimizing power $p_m,p_L$ and CPU frequency $f_m,f_L$, Equations (34) and (35) are the power constraints, and CPU frequency constraints of the AUV m, respectively, Equations (36) and (37) are the power constraints and CPU frequency constraints of AUV L, respectively, Equation (38) is the lazy node scale factor, and Equations (39) and (40) are the energy constraints of AUV m and AUV L, respectively.

4. Algorithm Design

In this section, we consider PPO to deal with $\mathcal{P}1$, since it is non-convex and has high dimensionality.

4.1. Modeling of Deep Reinforcement Learning Environment

An MDP consists of the state space, action space, state transition matrix function, reward function, and discount factor. Thus the optimization problem $\mathcal{P}1$ can be transformed into the following:

State space: For each time slot t, we use $p_m(t-1),f_m(t-1),p_L(t-1),f_L(t-1),\beta (t-1)$ at the time slot $(t-1)$ to describe the state space.

According to the above, the network state of the agent at time slot t in this paper can be expressed by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill s\left(t\right)=\left(p_m(t-1),f_m(t-1),p_L(t-1),f_L(t-1),\beta (t-1)\right)\end{array}\end{array}$$

(41)

the network state of the agent is given by $S_t=\left\{s\left(t\right)\right\}$

action space: at each the time slot t, $a\left(t\right)$ is composed of the following parts—formally, the action space at time slot t is denoted by: $p_m\left(t\right),f_m\left(t\right),p_L\left(t\right),f_L\left(t\right),\beta \left(t\right)$

$$\begin{array}{c}\hfill \begin{array}{c}\hfill a\left(t\right)=\left(p_m\left(t\right),f_m\left(t\right),p_L\left(t\right),f_L\left(t\right),\beta \left(t\right)\right)\end{array}\end{array}$$

(42)

and the action of the agent is given by $a_t=\left\{a\left(t\right)\right\}$.

State transition function: transition probability of the agent at time slot t can be denoted as $P_T\left(s_{t+1}\mid s_t,a_t\right)$.

Policy: Let $\pi$ denote the policy function, which is based on the observed state to make decisions and control the action of the agent $\pi (a\mid s)=P(a\mid s)$.

Reward function: the reword function in this paper is designed for optimizing $p_m,f_m,p_L,f_L,\beta$; the presented reword function of the agent at one time slot t is expressed as

$$\begin{array}{c}\hfill r_t(s_t,a_t)=-(T\left(t\right)+E\left(t\right))\end{array}$$

(43)

Maximizing the cumulative reward obtained by the sequence $T^{\prime }$ involves the sum of the rewards obtained at each stage, called $R_n\left(T^{\prime }\right)$. Therefore, the expected reward can be obtained as follows with policy $\pi$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R_n\left(T^{\prime }\right)=\sum _{\tau =0}^{\infty }{\xi }_{\tau }{r}_n\left(s_{n+\tau },a_{n+\tau }\right)\end{array}\end{array}$$

(44)

where ${\xi }_{\tau }$ denotes the discounter factor.

4.2. Proximal Policy Optimization Algorithm

The proximal policy optimization (PPO) is the off-policy model-free reinforcement learning algorithm. OpenAI uses it as the current baseline algorithm, which uses a new class of objective functions and updates parameters in small batches with multiple training steps. To better describe reward and prevent overfitting, we used advantage ${\widehat{A}}_n$ instead $R_n\left(T^{\prime }\right)$ to evaluate actions

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\widehat{A}}_n=R_n\left(T^{\prime }\right)-V_{\varphi }\left(s_n\right)\end{array}\end{array}$$

(45)

where $V_{\varphi }\left(s_n\right)$ can be calculated by a value network.

Moreover, our expectation is to update the actor’s policy $\pi$ to maximize the expected reward. So, we need to use the gradient boosting method to update the network parameters $\theta$. The gradient solution process is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathbb{G}=E\left[{\nabla }_{\theta }log{\pi }_{\theta }\left(a_t\mid s_t\right){\widehat{A}}_t\right]\end{array}\end{array}$$

(46)

PPO2 was inspired by the same question as TRPO: using only the current data to improve the policy as much as possible, without causing a sudden decline in the performance of the policy. Differently from TRPO attempting to solve the problem with a complex second-order approach, PPO2 uses a first-order approach that uses a few other tricks to make the new policy approximate the old one. The essence of the PPO algorithm is to introduce a ratio coefficient to indirectly describe the difference between the new strategy and the old strategy, denoted by ${\mathrm{r}}_t\left(\theta \right)=\frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{{\theta }_k}\left(a_t\mid s_t\right)}$; the loss function is defined as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}& {\theta }_{k+1}=arg\underset{\theta }{max}\frac{1}{\left|{\mathcal{D}}_k\right|T}\sum _{\tau \in {\mathcal{D}}_k}\sum _{t=0}^T\hfill \\ & min\left(\frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{{\theta }_k}\left(a_t\mid s_t\right)}{A}^{{\pi }_{{\theta }_k}}\left(s_t,a_t\right),g\left(ϵ,A^{{\pi }_{{\theta }_k}}\left(s_t,a_t\right)\right)\right)\hfill \end{array}\end{array}$$

(47)

where $ϵ$ is a (small) hyperparameter that roughly limits the range of variation of ${\mathrm{r}}_t\left(\theta \right)$. In PPO2, we use the following formula to update the value network strategy

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\varphi }_{k+1}=arg\underset{\varphi }{min}\frac{1}{\left|{\mathcal{D}}_k\right|T}\sum _{\tau \in {\mathcal{D}}_k}\sum _{t=0}^T{(V_{\varphi }\left(s_t\right)-{\widehat{R}}_t)}^2\end{array}\end{array}$$

(48)

The algorithm used in this paper is summarized in Algorithm 1.

Algorithm 1 Proximal policy optimization clip.

Input: initial policy parameters ${\theta }_0$, initial value function parameters ${\varphi }_0$ for$k=1,2,\dots$do    Run policy ${\pi }_{\theta }$ for K timesteps, collecting $s_n,a_n,r_n$    Compute return ${\widehat{R}}_n$ according to Equation (44)    Compute advantages ${\widehat{A}}_n$ according to Equation (45)    Update the policy by maximizing the PPO clip objective according to Equation (47)    Fit value function according to Equation (48) end for

5. Simulation Results

This chapter uses the TensorFlow framework in Python to build a federated learning model, using a MNIST (Modified National Institute of Standards and Technology Database) consisting of 42,000 digital images, and retaining 10% of the data as a test set for the global model. At the same time, a three-layer MLP (multilayer perceptron) neural network was created as a model for the classification task. The number of neurons in each layer was 200, 200, and 10, respectively. We used this model to perform a classification task and classify the resulting accuracy and loss function. This section gives a maximum number of convergence rounds of 1000. Assuming M = 9, the follower AUVs are 20 m apart from each other and form a formation to form a formation and fly at a constant speed in a certain direction, and the leader AUV is 20 m in front of the formation and the center. The rest of the simulation parameters are shown in

Table 1. Values of main parameters.

Parameter	Value	Parameter	Value
k	$1.25\times {10}^{-26}$	$\gamma$	0.01
$\delta$	3	M	20
$B_m^{\mathrm{U}}$	10 kHz	f	30 kHz
$B^{\mathrm{D}}$	10 kHz	s	0.5
$N_m$	4224	$\tau$	5
$c_m$	10,000	$f_{\max }$	0.4 GHz
$c_0$	50	$f_{\min }$	0.2 GHz
$c_L^{\prime }$	50	$E_m^{\mathrm{thd}}$	0.07 W
$|{\mathit{w}}_m|$	1594*64 bit	$E_L^{\mathrm{thd}}$	0.8 W
$\left|\mathit{w}\right|$	1594*64 bit	$\Phi$	1
$p_{\max }$	0.2 W	$\chi$	1

.

5.1. The Performance of Each Index in the Gradient Compression Test

In Figure 1, the comparison of communication times among the different schemes versus $\beta$, We can observe that as $\beta$ increases, the follower AUVs that do not participate in communication within the round decrease more, and the corresponding total number of communications increased. From the analysis of Equation (19), the decrease of $\beta$ leads to the increase of the right side of the inequality, resulting in more nodes satisfying Equation (19), the AUV skipping communication rounds increase, and the communication decreases. The experimental results are in good agreement with the theoretical analysis. It can be concluded from Figure 1 and Figure 2 that the federated learning after the control model still converges. However, as it decreases, the number of communications decreases. Although the communication resources are saved, the global model is affected due to the lack of gradients of some follower AUVs in certain rounds, resulting in a decrease in the performance of federated learning. Specifically, the performance is reduced in accuracy, and the experimental results are consistent with the theoretical analysis.

In Figure 3, we show the total time comparison of different schemes versus $\beta$. We can observe that as $\beta$ increases, the total time reduces. This is because with $\beta$ increasing, the communications gradually increase, and the model can converge faster with more communications, resulting in a shorter total time.

5.2. The Performance Analysis of the Scheme Proposed in This Paper

To show that the scheme proposed in this paper has the best effect on reducing the training cost, we compare it with other offloading schemes.

• Scheme 1: the scheme proposed in this paper.
• Scheme 2: asynchronous federated learning with dynamically optimized $\beta$
• Scheme 3: asynchronous federated learning with fixed $\beta$.
• Scheme 4: asynchronous federated learning with LAG.
• Scheme 5: traditional asynchronous federated learning.

The two sub-graphs in Figure 4 describe the communication times and corresponding accuracy rates of different experimental schemes with different follower AUV numbers. Accuracy will increase. At the same time, a larger communication number corresponds to a higher accuracy rate. When the communication number time is reduced by 710, which is only 21% of the traditional asynchronous federated learning, the accuracy rate is only reduced by 0.03%. This shows that we greatly reduced the communication number, but the accuracy basically did not decrease, indicating that our proposed scheme has good results.

In Figure 5, we show the accuracies of different control models versus the number of follower AUVs. Compared with the work of [19], the control model proposed in this paper increases the accuracy. This shows that the improved control model in this paper is effective.

In Figure 6, we show the cost comparison of different schemes versus the number of follower AUVs. With the increase in the follower AUV, the cost gradually increases. As can be seen from the figure, the scheme proposed in this paper has a smaller cost and can save more resources.

5.3. The Performance Analysis of the PPO2 Algorithm

In Figure 7, we compare the performance of state-of-the-art algorithms in coping with problem $\mathcal{P}1$, including GA, PSO, AC, DDPG, and our employed PPO2 algorithm. We can observe that PPO2 has a smaller cost under the same number of iterations, which shows that this algorithm is better than other algorithms.

6. Conclusions

In order to reduce the constraints of underwater scarce communication resources on AUV swarm machine learning, we designed an asynchronous federated learning method. By constructing the optimization problem of minimizing the weighted sum of delay and energy consumption, the AUV CPU frequency and signal transmission power were jointly optimized. In order to solve this complex optimization problem of high-dimensional non-convex time series accumulation, we transformed the problem into an MDP and used the PPO2 algorithm to solve this problem. Finally, we carried out some experiments to verify the effectiveness of the proposed scheme.