Deep-Reinforcement-Learning-Based Active Disturbance Rejection Control for Lateral Path Following of Parafoil System

Abstract

The path-following control of the parafoil system is essential for executing missions, such as accurate homing and delivery. In this paper, the lateral path-following control of the parafoil system is studied. First, considering the relative motion between the parafoil canopy and the payload, an eight-degree-of-freedom (DOF) model of the parafoil system is constructed. Then, a guidance law containing the position deviation and heading angle deviation is proposed. Moreover, a linear active disturbance rejection controller (LADRC) is designed based on the guidance law to allow the parafoil system to track the desired path under internal unmodeled dynamics or external environmental disturbances. For the adaptive tuning of the controller parameters, a deep Q-network (DQN) is applied to the LADRC-based path-following control system, and the controller parameters can be adjusted in real time according to the system’s states. Finally, the effectiveness of the proposed method is applied to a parafoil system following circular and straight paths in an environment with wind disturbances. The simulation results show that the proposed method is an effective means to realize the lateral path-following control of the parafoil system, and it can also promote the development of intelligent controllers.

1. Introduction

A parafoil system is a unique aircraft consisting of a parafoil canopy, connecting rope, and payload. With its excellent gliding ability and payload characteristics, the parafoil system is currently widely used in aerospace, military, and civil fields. However, the parafoil system applies a flexible canopy to provide the lift force, so it has complicated dynamic characteristics and strong nonlinearity [1]. During flight, the parafoil can be affected by unpredictable wind disturbances. Therefore, determining how to overcome these disturbances and accurately control the parafoil system to follow the desired path is the key to completing missions.

Limited by actual flight tests, which require considerable preparation work and are time-consuming and expensive, a mathematical model is a prerequisite for analyzing the motion characteristics of the parafoil system. To the best of our knowledge, according to the division of the DOFs, the current existing modeling methods include longitudinal four-DOF [2], six-DOF [3,4], eight-DOF [5,6], and nine-DOF [7] models. These models were all obtained through force analysis of the parafoil system. In this paper, considering the relative pitch and yaw motions between the parafoil canopy and the payload, we constructed an eight-DOF model based on a six-DOF model [5].

There have been some research results for the path-following control of the parafoil system. For example, Tao et al. [8] applied a generalized predictive control (GPC)-based method for a parafoil system to follow the designed path for a better control effect, where the guidance law was based on a combination of tracking errors. Zhao et al. [9] introduced a model-free adaptive control method based on iterative feedback tuning (IFT-MFAC) method for parafoil systems, where only the input/output data are needed during the construction. Guo et al. [10] proposed an improved adaptive path-following guidance law to reduce the effect of time-varying wind disturbances, where the convergence rate was improved by replacing the function. Li et al. [11] designed three proportional–integral–derivative (PID) controllers for the lateral motion, longitudinal motion, and velocity on the basis of the motion characteristics of the parafoil system, which overcame the limitations of the traditional guidance-based tracking strategy. As for actual airdrop scenarios, the PID controller still occupies a dominant position, but the PID controller cannot achieve high tracking accuracy, especially under the disturbances of a complex environment. Tao et al. [12,13] used linear active disturbance rejection control (LADRC) to realize the accurate trajectory tracking of the parafoil system. LADRC is currently the most widely used control strategy in practice besides PID; however, the adjustment of the LADRC parameters remains a challenging problem to be studied.

Active disturbance rejection control (ADRC) [14] was first proposed by Han [15], which combines the state observer in modern theory with the error-based ideas in PID. Specifically, ADRC uses an extended state observer (ESO) to observe the unknown disturbances in the system and uses a state error feedback (SEF) control law to eliminate the disturbance. With model-free characteristics and good control effects, ADRC has attracted the attention of many scholars. Gao [16] developed LADRC through the linearization of the ESO and SEF, which significantly promoted the theoretical and engineering application research of ADRC. In terms of theory, Chen [17] and Wang [18] provided proof of the stability of LADRC. LADRC has demonstrated its control advantages in applications such as power system load frequency control (LFC) [19], heading angle control [20], path-following control [21,22], and an electromechanical servo system [23]. For example, Li et al. [24] proposed a guidance law based on a ship’s nonlinear combination of lateral error and heading angle error, and LADRC was used to estimate and eliminate the disturbances. However, this guidance law could only realize path tracking in the y-direction. Inspired by this result, this paper proposes a new guidance law.

Although the number of parameters that need to be adjusted in the LADRC controller has significantly been reduced compared to that in the ADRC, parameter optimization is still a non-negligible part of the controller design process. In most cases, researchers will manually adjust the controller parameters, which makes it challenging to achieve the system’s optimal performance. Therefore, various optimization algorithms have emerged continually. The robustness of heuristic algorithms that can optimize a set of fixed parameters, such as the particle swarm optimization (PSO) algorithm [25] and genetic algorithm (GA) [26], is somewhat limited. For algorithms that can achieve adaptive optimization, such as neural networks [27] and fuzzy control [28], these algorithms have difficulty achieving better results when encountering unknown emergencies.

Deep reinforcement learning (DRL) is a combination of reinforcement learning (RL) and neural networks that uses the computing power of neural networks and encompasses the decision-making ability of RL. With the intelligent characteristics of not relying on models and being able to make decisions autonomously, similar to the human brain, DRL is favored by many scholars. The deep Q-network (DQN) [29] is one of the most classical algorithms in DRL, and it overcomes the shortcomings of the Q table in Q-learning of RL whereby it is difficult to express all states. In other words, the DQN can handle systems with continuous states. The application of DQN in path-following control is mainly to make direct decisions in terms of the control variable. For example, Zhao et al. [30] used DQN to determine the rudder angle and propeller speed during the path tracking of an unmanned surface vessel (USV), which completely separated decision-making and control theory. At present, there are few studies on the application of RL in parafoil systems. Therefore, to promote the development of intelligent controllers in parafoil systems, this paper uses the DQN algorithm to optimize the parameters of the LADRC controller.

Inspired by the previous research, we proposed a DQN-optimized LADRC method for lateral path-following control of the parafoil system in this paper. The main contributions of this paper are summarized as follows:

• A new guidance law for the lateral path-following control of the parafoil system is proposed.
• Based on the guidance law, an LADRC controller is designed to overcome the influence of unknown disturbances during parafoil flight.
• The DQN algorithm of RL is applied to obtain the real-time parameters of the controller based on the parafoil flight states.

The rest of the paper is organized as follows. The eight-DOF parafoil system model is established in Section 2. Section 3 introduces the guidance law and provides the corresponding design process of the LADRC-based path-following controller. Section 4 presents the design process of the DQN-optimized LADRC. The simulation results are given in Section 5, and Section 6 concludes this paper.

2. Dynamic Model of Parafoil System

In the six-DOF parafoil model, the parafoil canopy and payload are regarded as rigid connections, and the relative motion between the two is ignored. In order to describe the motion characteristics of the parafoil more accurately, an eight-DOF dynamic model is established in this paper. The slew, sway, yaw, heave, pitch, and roll motions of the parafoil canopy, and the relative pitch and relative yaw motion between the parafoil canopy and the payload are considered. In this way, we can better observe the attitude of the parafoil system during the movement. Three coordinate systems are established to facilitate parafoil system modeling: the ground coordinate system $O_d{x}_d{y}_d{z}_d$, the parafoil coordinate system $O_s{x}_s{y}_s{z}_s$, and the payload coordinate system $O_w{x}_w{y}_w{z}_w$, as shown in Figure 1.

Furthermore, since the actual model is very complex, in the process of building the model in this paper, the following three assumptions are adopted, and certain simplifications are made:

• The mass center of the parafoil canopy coincides with the aerodynamic pressure center but does not coincide with the gravity center.
• The lift of the payload is ignored and only the aerodynamic drag is considered.
• The elastic deformation of the connecting ropes is ignored.

2.1. Motion Equation of Payload

In this paper, the payload is regarded as a rigid body subjected to aerodynamic forces, gravity, and the pulling force of the connecting rope. The force and moment balance equations of the payload are expressed as

$$\left\{\begin{array}{c}\frac{\partial P_w}{\partial t}+W_w\times P_w=F_w^{aero}+F_w^G+F_w^t\hfill \\ \frac{\partial H_w}{\partial t}+W_w\times H_w=M_w^{aero}+M_w^f+M_w^t\hfill \end{array}\right.,$$

(1)

where P and H represent the momentum and angular momentum, defined below. The superscripts $aero$, G, t, and f represent the aerodynamic force, gravity, connecting rope tension, and friction, respectively. $V_w={\left[u_w,v_w,w_w\right]}^T$ denotes the velocity vector, and $W_w={\left[p_w,q_w,r_w\right]}^T$ denotes the angular velocity vector. The momentum and angular momentum are defined, respectively, as follows:

$$\left\{\begin{array}{c}{P}_w=m_w{V}_w\hfill \\ H_w=J_w{W}_w\hfill \end{array}\right..$$

(2)

The tension and tension moment of the connecting rope are expressed as

$$\left\{\begin{array}{c}{F}_w^t=F_1+F_2\hfill \\ M_w^t=\left[L_{w1}{L}_{w2}\right]{\left[F_{w1}^T{F}_{w2}^T\right]}^T\hfill \end{array}\right.,$$

(3)

where $F_1$ and $F_2$ represent the tension at points $c_1$ and $c_2$, respectively. $L_{w1}=l_2\left[\begin{array}{ccc}0& 1& 0\\ -1& 0& 0\\ 0& 0& 0\end{array}\right]$, $L_{w2}=\frac{l_1}{2}\left[\begin{array}{ccc}0& 0& 1\\ 0& 0& 0\\ -1& 0& 0\end{array}\right]$, and $l_1$ refers to the distance between the connection points $c_1$ and $c_2$. $l_2$ is the distance between the $O_w$ of the payload coordinate system and the midpoint $c_0$ of the two connecting points.

2.2. Motion Equation of Parafoil Canopy

When the canopy is fully inflated and unfolded, the force on the canopy includes aerodynamic forces, gravity, and the pulling force of the connecting rope. The equations of the conservation of linear and angular momentum of the parafoil canopy are expressed, respectively, as

$$\left\{\begin{array}{c}\frac{\partial P_s}{\partial t}+W_s\times P_s=F_s^{aero}+F_s^G+F_s^t\hfill \\ \frac{\partial H_s}{\partial t}+W_s\times H_s+V_s\times P_s=M_s^{aero}+M_s^G+M_s^f+M_s^t\hfill \end{array}\right..$$

(4)

Similarly, the velocity vector $V_s={\left[u_s,v_s,w_s\right]}^T$ and angular velocity vector $W_s={\left[p_s,q_s,r_s\right]}^T$ are defined. It should be pointed out that the movement of the parafoil canopy in the air can be approximately regarded as the movement in an ideal fluid. Hence, the influence of the additional mass on the system needs to be considered. Therefore, the linear momentum and angular momentum of the parafoil canopy considering the additional mass are described as

$$\left[\begin{array}{c}{P}_s\\ H_s\end{array}\right]=\left[A_a+A_r\right]\left[\begin{array}{c}{V}_s\\ W_s\end{array}\right],$$

(5)

where $A_a$ denotes the inertia matrix of the additional mass, and $A_r$ represents the inertia matrix of the canopy’s real mass.

Similarly, in the parafoil coordinate system, there is an expression of the tension of the connecting rope relative to the moment of the parasol center of mass:

$$M_s^t=\left[\begin{array}{cc}{L}_{s1}{T}_{w-s}& L_{s2}{T}_{w-s}\end{array}\right]{\left[\begin{array}{cc}{F}_{w1}^T& F_{w2}^T\end{array}\right]}^T,$$

(6)

where $T_{w-s}$ is defined below. $L_{s1}=l_3\left[\begin{array}{ccc}0& 1& 0\\ -1& 0& 0\\ 0& 0& 0\end{array}\right]$, $L_{s2}=\frac{l_1}{2}\left[\begin{array}{ccc}0& 0& -cos{\psi }_r\\ 0& 0& -sin{\psi }_r\\ cos{\psi }_r& sin{\psi }_r& 0\end{array}\right]$, and $l_3$ represents the distance between the $O_s$ of the payload coordinate system and $c_0$.

2.3. Constraints of Velocity and Angular Velocity

As shown in Figure 1, the canopy and the payload are connected by ropes at two connection points, $c_1$ and $c_2$, and then the midpoint $c_0$ of the two points can be regarded as the total connection point. At this point, there is the following velocity constraint:

$$V_w+W_w\times L_{w-c}=V_s+W_s\times L_{s-c},$$

(7)

where $L_{w-c}$ and $L_{s-c}$ represent the distance from the payload centroid to $c_0$ and the distance from the parafoil canopy centroid to $c_0$, respectively. Differentiation of Equation (7) can yield the following formula:

$$\begin{array}{c}{\dot{V}}_s-L_{s-c}\times {\dot{W}}_s-T_{w-s}{\dot{V}}_w+T_{w-s}{L}_{w-c}\times {\dot{W}}_w\hfill \\ =T_{w-s}{W}_w\times \left(V_w+W_w\times L_{w-c}\right)-W_s\times \left(V_s+W_s\times L_{s-c}\right),\hfill \end{array}$$

(8)

where $T_{w-s}$ is the conversion matrix from the payload coordinate system to the canopy coordinate system. $T_{w-s}$ is expressed as follows:

$$T_{w-s}=\left[\begin{array}{ccc}cos{\theta }_{r}cos{\psi }_r& -sin{\psi }_r& sin{\theta }_{r}cos{\psi }_r\\ cos{\theta }_{r}sin{\psi }_r& cos{\psi }_r& sin{\theta }_{r}sin{\psi }_r\\ -sin{\theta }_r& 0& cos{\theta }_r\end{array}\right],$$

(9)

where ${\theta }_r$ and ${\psi }_r$ denote the relative pitch angle and relative yaw angle, respectively.

As for the relative rotation between the payload and the canopy, the following angular velocity constraint is satisfied:

$$W_w=W_s+{\tau }_s+{\kappa }_w,$$

(10)

where ${\tau }_s={\left[0,0,{\dot{\psi }}_r\right]}^T$, and ${\kappa }_w={\left[0,{\dot{\theta }}_r,0\right]}^T$. The derivative of Equation (10) is as follows:

$$\begin{array}{c}{T}_{w-s}{\dot{W}}_w-{\dot{W}}_s-T_{w-s}{\dot{\kappa }}_w-{\dot{\tau }}_s=\left(W_s-T_{w-s}{W}_w\right)\times W_s-\left(T_{w-s}{\kappa }_w\right)\times {\tau }_s\hfill \end{array}.$$

(11)

Define the state variable as $x_s={\left[V_w^T,W_w^T,V_s^T,W_s^T,{\dot{\psi }}_r,{\dot{\theta }}_r\right]}^T$. Then, by combining Equations (1)–(4), (8) and (11), the eight-DOF dynamic model of the parafoil can be expressed as follows:

$${\dot{x}}_s={\left({\left[D_1^T,D_2^T,D_3^T,D_4^T\right]}^T\right)}^{-1}{\left[E_1^T,E_2^T,E_3^T,E_4^T\right]}^T.$$

(12)

Refer to [4] for a more detailed modeling process.

By denoting $sin\theta$ as $s\theta$ and $cos\theta$ as $c\theta$, the following equation can be obtained:

$$\left[\begin{array}{c}\dot{x}\\ \dot{y}\\ \dot{z}\end{array}\right]={\left[\begin{array}{ccc}\mathrm{c}\theta \mathrm{c}\psi & \mathrm{c}\theta \mathrm{s}\psi & -\mathrm{s}\theta \\ \mathrm{s}\varphi \mathrm{s}\theta \mathrm{c}\psi -\mathrm{c}\varphi \mathrm{s}\psi & \mathrm{s}\varphi \mathrm{s}\theta \mathrm{s}\psi +\mathrm{c}\varphi \mathrm{c}\psi & \mathrm{s}\varphi \mathrm{c}\theta \\ \mathrm{c}\varphi \mathrm{s}\theta \mathrm{c}\psi +\mathrm{s}\varphi \mathrm{s}\psi & \mathrm{c}\varphi \mathrm{s}\theta \mathrm{s}\psi -\mathrm{s}\varphi \mathrm{c}\psi & \mathrm{c}\varphi \mathrm{c}\theta \end{array}\right]}^T{V}_s,$$

(13)

where $\varphi$, $\theta$, and $\psi$ are three Euler angles, i.e., the roll angle, pitch angle, and yaw angle, respectively. Based on Equations (12) and (13), the position of the parafoil can be obtained.

In this paper, the parafoil system is described by the dynamic and kinematic parameters. The dynamic parameters of the parafoil system are the parameters that describe the forces and moments in Equations (1) and (4), which are mainly used to express the relationship between the force of the parafoil system and its motion. The kinematic parameters of the parafoil system are the parameters describing the speed, Euler angles, and acceleration of the parafoil system in Equation (13), which reflects the variation of the position of the parafoil system with time. It is worth mentioning that during the movement of the parafoil, unmodeled dynamics and disturbances in the external environment will cause unpredictable effects. Therefore, the mismatch between theory and reality challenges the controller design.

3. Linear Active Disturbance Rejection Controller (LADRC)-Based Path-Following Control

As mentioned above, the direction of the parafoil system is manipulated by control ropes connecting two sides of the canopy. Specifically, the left control rope is pulled downward to realize a left turn and the right control rope is pulled down to realize a right turn. That is, by controlling the control ropes on both sides, the position and heading angle of the parafoil system are changed simultaneously to track the desired path. Overall, the control command produced by the control system will result in the increase in the force of one of the ropes. Moreover, in the actual control process, the control signal is usually transmitted to the rope in the form of a differential or pulse (1/0), which also brings difficulty to the control system.

3.1. Guidance Law

To achieve convergence of the parafoil position deviation and heading angle deviation to zero simultaneously, according to a guidance law designed in Reference [24], a new guidance law is established as follows:

$$g=g_{0}tanh\left(g_1·\Delta d\right)+\psi -{\psi }_d,$$

(14)

where $g_0$ and $g_1$ are adjustable parameters ($g_0>0,g_1>0$). $\psi$ and ${\psi }_d$ represent the true value and the planned value of the parafoil’s heading angle (yaw angle), respectively. $\Delta d$ is the position tracking error, as shown in Figure 2, expressed as follows:

$$\left\{\begin{array}{c}\Delta d=\frac{\widehat{x}\Delta y-\widehat{y}\Delta x}{\sqrt{\Delta x^2+\Delta y^2}}\hfill \\ {\psi }_d=arctan\left(\Delta y/\Delta x\right)\hfill \end{array}\right..$$

(15)

As shown in Figure 2, the red point $\left(x\right(t),y(t\left)\right)$ represents the current position of the parafoil, and the blue points $\left(x_r\left(i-1\right),y_r\left(i-1\right)\right)$ and $\left(x_r\left(i\right),y_r\left(i\right)\right)$ are the two adjacent points in the desired path. Define

$$\left\{\begin{array}{c}\Delta x=x_r\left(i\right)-x_r\left(i-1\right)\hfill \\ \Delta y=y_r\left(i\right)-y_r\left(i-1\right)\hfill \\ \widehat{x}=x_r\left(i\right)-x\left(t\right)\hfill \\ \widehat{y}=y_r\left(i\right)-y\left(t\right)\hfill \end{array}\right..$$

(16)

The idea of this article is to directly control the flight direction by controlling the tension of the connecting rope so that $\Delta d$ can be 0. From Equation (14), it can be seen that the path tracking error and heading angle tracking error are nonlinearly combined, and we hope that both $\Delta d$ and $(\psi -{\psi }_d)$ can be stabilized to 0, that is, $g=0$. There exists the following theorem.

Theorem 1. 

There exists a $g_0$ such that $\Delta d$ converges to 0 in the case of $g=0$.

Proof of Theorem 1. 

We construct the Lyapunov function as $V=\frac{1}{2}\Delta d^2$. Furthermore, we can obtain

$$\dot{V}=\Delta d·\Delta \dot{d}.$$

(17)

By combining Equations (15) and (16), we can obtain $\Delta d$ in the following form:

$$\Delta d=\left(x_r\left(i\right)-x\left(t\right)\right)sin{\psi }_d-\left(y_r\left(i\right)-y\left(t\right)\right)cos{\psi }_d,$$

(18)

with the derivative

$$\Delta \dot{d}=-\dot{x}\left(t\right)sin{\psi }_d+\dot{y}\left(t\right)cos{\psi }_d.$$

(19)

By ignoring the influence of the pitch angle and the roll angle, Equation (13) can be simplified to obtain the following expressions:

$$\left\{\begin{array}{c}\dot{x}=u_{s}cos\psi -v_{s}sin\psi \hfill \\ \dot{y}=u_{s}sin\psi +v_{s}cos\psi \hfill \end{array}\right..$$

(20)

Then, Equation (19) can be rearranged as

$$\begin{array}{cc}\hfill \Delta \dot{d}& =-\left(u_{s}cos\psi -v_{s}sin\psi \right)sin{\psi }_d+\left(u_{s}sin\psi +v_{s}cos\psi \right)cos{\psi }_d\hfill \\ \hfill & =\sqrt{u_s^2+v_s^2}sin\left(\beta +\psi -{\psi }_d\right),\hfill \end{array}$$

(21)

where $\beta =arctan\left(u_s/v_s\right)$. During the forward flight of the parafoil, it can be approximated that $u_s\gg v_s$, that is, $\beta \approx 0$. Then, Equation (17) has the following expression:

$$\dot{V}=\Delta d·\sqrt{u_s^2+v_s^2}sin\left(\psi -{\psi }_d\right).$$

(22)

In the case of $g=0$, by substituting Equation (14) into Equation (22), we have

$$\dot{V}=\Delta d·\sqrt{u_s^2+v_s^2}sin\left(-g_{0}tanh\left(g_1·\Delta d\right)\right).$$

(23)

There are the following three situations:

(a)

$\Delta d=0$. In this case, $\dot{V}=0$.

(b)

$\Delta d>0$. In this case, $-g_0<-g_{0}tanh\left(g_1·\Delta d\right)<0$. Then, $0<g_0<\pi$, and thus, $\dot{V}<0$.

(c)

$\Delta d<0$. In this case, $0<-g_{0}tanh\left(g_1·\Delta d\right)<g_0$. Then, $0<g_0<\pi$, and thus, $\dot{V}<0$.

   □

3.2. Design of LADRC

The design of the LADRC does not require the model of the system to be known, but it requires the order information of the controlled plant. The guidance law in Equation (14) can essentially be understood as controlling the position deviation through the heading angle. Generally, there is the following relationship between the Euler angles and the angular velocities:

$$\left[\begin{array}{c}\dot{\varphi }\\ \dot{\theta }\\ \dot{\psi }\end{array}\right]=\left[\begin{array}{ccc}1& sin\varphi tan\theta & cos\varphi tan\theta \\ 0& cos\varphi & -sin\varphi \\ 0& sin\varphi /cos\theta & cos\varphi /cos\theta \end{array}\right]\left[\begin{array}{c}{p}_s\\ q_s\\ r_s\end{array}\right].$$

(24)

Further derivation yields the following:

$$\begin{array}{c}\ddot{\psi }=\frac{sin\varphi }{cos\theta }\dot{q}+\frac{cos\varphi }{cos\theta }\dot{r}+\frac{sin\theta sin2\varphi }{{cos}^2\theta }{q}^2-\frac{sin\theta sin2\varphi }{{cos}^2\theta }{r}^2\\ +\frac{cos\varphi }{cos\theta }pq-\frac{sin\varphi }{cos\theta }pr+\frac{2sin\theta cos2\varphi }{{cos}^2\theta }qr.\end{array}$$

(25)

By combining Equations (12) and (25), we can obtain the second-order relationship between $\psi$ and the control variable u:

$$\ddot{\psi }=f_1\left(·\right)+f_2\left(u\right),$$

(26)

where $f_1\left(·\right)$ represents the disturbance, including the internal state and external disturbance information, and $f_2\left(u\right)$ is an expression of the control variable. u represents the deflection of the left and right trailing edges of the canopy.

Since g and $\psi$ have the same order, we can rearrange Equation (26) as

$$\begin{array}{c}\ddot{g}=f\left(·\right)+f_2\left(u\right)-b_{0}u+b_{0}u=f+b_{0}u\end{array},$$

(27)

where f can be regarded as the total disturbances of the parafoil system containing unmodeled dynamics inside the system and disturbances in the external environment, and $b_0$ is an adjustable parameter.

Then, the states can be defined as $x_1=g$, $x_2=\dot{g}$, and $x_3=f$. The linear ESO (LESO) can be expressed as

$$\left\{\begin{array}{c}\dot{\widehat{x}}=A\widehat{x}+Bu+L\left(\dot{\widehat{y}}-\widehat{y}\right)\hfill \\ \widehat{y}=C\widehat{x}\hfill \end{array}\right.,$$

(28)

where $A=\left[\begin{array}{ccc}0& 1& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right]$, $B=\left[\begin{array}{c}0\\ b_0\\ 0\end{array}\right]$, $L=\left[\begin{array}{c}{\beta }_{01}\\ {\beta }_{02}\\ {\beta }_{03}\end{array}\right]$, $C={\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]}^T$, and $\widehat{x}=\left[\begin{array}{c}{\widehat{x}}_1\\ {\widehat{x}}_2\\ {\widehat{x}}_3\end{array}\right]$. In addition, ${\beta }_{01}$, ${\beta }_{02}$, and ${\beta }_{03}$ are observer gains, which are the prerequisite for realizing the observed state $\widehat{x}$ to estimate the true value of the state x. Usually, the pole configuration method is used to configure the observer gain at the pole $-{\omega }_o$, which is arranged as

$$\begin{array}{cc}\hfill \left|sI-\left(A-LC\right)\right|& =s^3+{\beta }_{01}{s}^2+{\beta }_{02}s+{\beta }_{03}\hfill \\ \hfill & ={\left(s+{\omega }_o\right)}^3.\hfill \end{array}$$

(29)

Thus, ${\beta }_{01}=3{\omega }_o$, ${\beta }_{02}=3{\omega }_o^2$, and ${\beta }_{03}={\omega }_o^3$. In this way, by adjusting the parameter ${\omega }_o$, the total system disturbance f can be estimated.

Under the premise of ${\widehat{x}}_3\approx f$, define u in Equation (27) as

$$u=\frac{u_0-{\widehat{z}}_3}{b_0}.$$

(30)

Then, Equation (31) can be obtained:

$$\ddot{g}\approx u_0,$$

(31)

where $u_0$ follows the proportional–derivative (PD) controller:

$$u_0=k_p\left(g_d-{\widehat{x}}_1\right)-k_d{\widehat{x}}_2.$$

(32)

Since the target value of g is 0, $g_d$ is 0. By using the pole placement method in the same way, we obtain $k_p=2{\omega }_c$ and $k_d={\omega }_c^2$.

As seen from the above process, the parameters that need to be adjusted are $g_0$, $g_1$, ${\omega }_o$, ${\omega }_c$, and $b_0$. Here, $g_1$ keeps the heading angle and $\Delta d$ at the same order of magnitude, and $g_0$ can limit the maximum heading angle of the system. Therefore, these two parameters can be adjusted manually. In addition, Ref. [31] shows that $b_0$ in the LADRC is not an essential factor for suppressing interference. Therefore, the parameters to be adjusted are ${\omega }_o$ and ${\omega }_c$.

4. Deep Q-Network (DQN)-Optimized LADRC

We know that RL has the advantages of not relying on models and learning independently, which has shown excellent results in solving sequential decision-making problems. This paper uses the DQN in RL to optimize ${\omega }_o$ and ${\omega }_c$ for the LADRC.

4.1. Basics of DQN

If the entire parafoil system containing the LADRC-based path-following controller is regarded as the environment, then an agent can be designed in RL, similar to the human brain. A series of decisions can be completed through the continuous interaction between the environment and the agent. Usually, the process of RL is described by the Markov decision process (MDP): $\left(S,A,P,R\right)$, where S and A are the environment state space and the agent’s action space, respectively. P represents the probability of the state transition. During the agent training process, the training target comes from the reward function R. On this basis, the cumulative reward $R_c$ can be obtained for each episode:

$$R_c=\sum _{t=0}^{\infty }{\gamma }^t{R}_{t+1},$$

(33)

where $\gamma$ is the discount factor. It can reflect the importance of the reward value at a future moment. In Q-learning, the training goal of the agent is the evaluated value Q corresponding to the state–action pair:

$$Q\left(s,a\right)=E\left[R_c\left|S_t=s,A_t=a\right.\right].$$

(34)

Equation (34) shows the expected value of the cumulative reward when the state is s, and the action is selected as a at time t. In other words, when the agent is sufficiently trained, and when the current system state is s, the action value a corresponding to the maximum Q value can be selected according to the Q table.

The dimensionality of the state space limits the Q-learning, and training is difficult when the state space is large. Therefore, a DQN is generated based on Q-learning, which uses a neural network to represent the Q table. That is, $f_{\mu }\left(s,a\right)\approx Q\left(s,a\right)$, where f is the output of the neural network with weight $\mu$. It should be pointed out that the input of the neural network is the system state, and the output is the $f_{\mu }\left(s,a\right)$ corresponding to the actions. The structure diagram of the DQN is shown in Figure 3.

As shown in Figure 3, there are two neural networks in total, namely, the Q-network and the target $\widehat{Q}$-network. The data for neural network training comes from the replay buffer. The weight of the Q-network is $\mu$, which is updated by the loss function

$$J={\mathbb{E}}_{\left(s,a,R,s^{\prime }\right)\sim \mathcal{D}}\left[{\left(R+\gamma \underset{a^{\prime }}{max}{f}_{{\mu }^{\prime }}\left(s^{\prime },a^{\prime }\right)-f_{\mu }\left(s,a\right)\right)}^2\right],$$

(35)

where $s^{\prime }$ and $a^{\prime }$ correspond to the state s and action a of the next moment, respectively. ${\mu }^{\prime }$ is the weight of the target network, and it is updated periodically. In other words, the weight of the target network does not need to be updated through training, while the weight of the Q-network can be updated according to the gradient descent method, shown as follows:

$$\stackrel{⌢}{\mu }=\mu +\alpha [R+\gamma \underset{a^{\prime }}{\max }{f}_{{\mu }^{\prime }}(s^{\prime },a^{\prime })-f_{\mu }(s,a)]\nabla f_{\mu }(s,a),$$

(36)

where $\alpha$ is the learning rate. In addition, Equation (35) can be understood as the cost function in DQN, which is to minimize the error between the actual $Q(s,a):R+\gamma \underset{a^{\prime }}{max}{f}_{{\mu }^{\prime }}(s^{\prime },a^{\prime })$ value and the estimated $Q(s,a):f_{\mu }\left(s,a\right)$. Then, the optimal parameters can be selected by the estimated Q that is close to the true Q value.

4.2. Design of Agent Based on Parafoil System

According to the above description, we must first define the system’s state and provide an action space. For the parafoil system, since we expect the parafoil system to fly along the desired path, $\Delta d$ and $\Delta \dot{d}$ define the state of the environment. In other words, at time t, the system will generate two state values:

$$\left\{\begin{array}{c}{s}_{01}\left(t\right)=\Delta d,\hfill \\ s_{02}\left(t\right)=\Delta \dot{d}\hfill \end{array}\right..$$

(37)

As for the action space, the parameters that need to be adjusted are ${\omega }_o$ and ${\omega }_c$, so the action space is obtained by permutation and combination based on these two parameters. The action space of ${\omega }_o$ and ${\omega }_c$ is expressed as

$$\left\{\begin{array}{c}{\omega }_o\in \left[{\omega }_{omin},{\omega }_{omax}\right]\hfill \\ {\omega }_c\in \left[{\omega }_{cmin},{\omega }_{cmin}\right]\hfill \end{array}\right.,$$

(38)

where the sampling interval is h for the two parameters. This means that ${\omega }_o$ and ${\omega }_c$ are $N_{{\omega }_o}=\frac{{\omega }_{omax}-{\omega }_{omin}}{h}+1$ and $N_{{\omega }_c}=\frac{{\omega }_{cmax}-{\omega }_{cmin}}{h}+1$, respectively. Therefore, the dimension of the action space is $N_{{\omega }_o}\ast N_{{\omega }_c}$.

The reward function is a direct factor that affects the agent’s training. In order to enable the parafoil system to track the desired path better, we hope that when $\Delta d$ is relatively small, the reward function can be significant. Therefore, the reward function in this paper is designed as

$$R=-10sign\left(\left|s_1\right|-2\right)-\left|s_2\right|.$$

(39)

The reward function means that when $\Delta d$ is less than 2, the agent can obtain a reward of 10. Otherwise, it can obtain a reward of −10. The smaller $\Delta \dot{d}$ is, the greater the reward is. The training process of the DQN is described in Algorithm 1. The schematic diagram of the parameter optimization based on the DQN is shown in Figure 4. It should be pointed out that the process of optimizing the LADRC controller parameters by the DQN algorithm includes two parts: offline training, shown in Algorithm 1, and online parameter acquisition. During offline training, a Q-network is obtained, and the training time is related to the simulation sample step and the number of iterations. The smaller the simulation sample step is or the larger the number of iterations is, the longer the training time will be. Moreover, by inputting the system state at the current moment into the trained Q-network, the optimal action values can be obtained to realize adaptive controller parameters, which is the online parameter selection process.

Algorithm 1: DQN algorithm.

1: Initialize the replay buffer; Initialize the Q-network with random weights $\mu$; Initialize the target $\widehat{Q}$-network with weights ${\mu }^{\prime }=\mu$.   2: for episode = 1:M do   3:     Initialize the states $s_1=s_{01},s_{02}$;   4:     for t = 1:T do   5:     Select a random action $a_t={\omega }_o,{\omega }_c$ with the probability $\epsilon$; otherwise, select $a_t=arg\underset{a}{max}{f}_{\mu }\left(s_t,a\right)$;   6:     Execute action $a_t$ in the LADRC and observe the reward $R_t$ and $s_{t+1}$;   7:     Set $s_{t+1}=s_t$;   8:     Store $(s_t,a_t,R_t,s_{t+1})$ in the replay buffer $\mathcal{D}$;   9:     Randomly extract m sets of data $(s_j,a_j,R_j,s_{j+1})$ from the replay buffer $\mathcal{D}$; 10:     Set $y_j=\left\{\begin{array}{c}{R}_j,\mathrm{if}\mathrm{episode}\mathrm{terminates}\mathrm{at}\mathrm{step}\mathrm{j}+1\hfill \\ R_j+\gamma \underset{a^{\prime }}{max}{f}_{{\mu }^{\prime }}\left(s_{j+1},a_{j+1}\right),\mathrm{otherwise}\end{array}\right.$ 11:     Perform a gradient descent step according to Equation (36); 12:     Every $T_n$ steps, reset ${\mu }^{\prime }=\mu$. 13:     end for 14: end for

5. Simulation and Analysis

5.1. Environment Setting

In this section, the effectiveness of the proposed DQN-LADRC method is verified by simulations. The parafoil system’s parameters are shown in

Table 1. Physical parameters of the parafoil system.

Parameter Description	Value
Wing span	4.5 m
Mean aerodynamic chord	1.3 m
Mass of parafoil	1.7 kg
Mass of payload	20 kg
Wing area	6.5 ${\mathrm{m}}^2$
Rope length	3 m

. In addition, the parameters involved in the DQN training process are shown in

Table 2. Parameter settings of DQN.

Parameter Description	Value
Simulation sample step	0.02 s
Number of iterations	300
Learning rate of Q-network	${10}^{-4}$
Discount factor	0.99

, where the Q-network is a fully connected neural network with two hidden layers. The network’s inputs are the state values shown in Equation (37), and the outputs are the parameters to be optimized. Both hidden layers have 500 neurons. In addition, the learning rate value was fixed during training, and its value was small enough to ensure convergence of the networks.

Furthermore, the range limits of the parameter tuning in this paper were artificially selected to ensure that the simulation could proceed smoothly within the selected range. The change of the parameter range size, on the one hand, could cause the system to directly stop the training when the “bad” parameters were explored; on the other hand, if the parameter range was too large, it would greatly increase the training time unnecessarily.

It should also be noted that, since the state observer will have extreme values in the transient response, to avoid adverse effects on equipment caused by extreme values, the control quantity generated by the motor connecting the control rope of the dynamic parafoil is limited: $-1\le u\le 1$ [32].

5.2. Performance Study of DQN-LADRC under Wind Disturbances

5.2.1. Circular Path Following

In order to verify the effectiveness of the proposed method, the DNQ-LADRC-based lateral path-following control of the parafoil system was conducted under wind disturbances. The desired path was set as a circle with a center of (0 m, 0 m) and a radius of 200 m. In the initial state, the three coordinates coincided, and the initial velocity and angular velocity of the parafoil were $V_s={\left[14.9,0,2.1\right]}^T$ m/s and $W_s={\left[0,0,0\right]}^T$ rad/s, respectively. Furthermore, the parafoil’s initial plane position and initial heading angle were (0 m, 150 m) and $0^{\circ }$, respectively. In the simulation environment, the average wind speed of 2 m/s along the y-axis square of the ground coordinate system was added at 50 s, and the duration was 20 s. $g_0$, $g_1$, and $b_0$ were selected as 0.08, 0.2, and 0.2, respectively. This paper compares the control effect of the proposed method with the traditional LADRC controller under a wind disturbance of the same size and direction, in which the parameters of the LADRC were selected from two sets of boundary values of the action space, as follows:

$$\left\{\begin{array}{cc}{\omega }_o\in \left[1.5,2.7\right],{\omega }_c\in \left[0.4,1\right],\hfill & \mathrm{For}\mathrm{DQN}\text{-}\mathrm{LADRC},h=0.01\hfill \\ {\omega }_o=2.7,{\omega }_c=1,\hfill & \mathrm{For}\mathrm{LADRC}1\hfill \\ {\omega }_o=1.5,{\omega }_c=0.4,\hfill & \mathrm{For}\mathrm{LADRC}2\hfill \end{array}\right..$$

(40)

The abovementioned trained agent was used to obtain the parameters of ${\omega }_o$ and ${\omega }_c$, and the simulation results are shown in Figure 5, Figure 6, Figure 7 and Figure 8.

Figure 5 shows the intuitive effect of the parafoil tracking of the desired path with and without wind disturbances, respectively. We can see that both the DQN-LADRC and the traditional LADRC could overcome disturbances and achieve circular path tracking. Specifically, the proposed method achieved better results than the LADRC with the fixed parameters in terms of the control variables, as shown in Figure 5b, and the tracking errors, as shown in Figure 6. It can be found that when the smaller controller parameters were taken, the overshoot of the tracking error obtained by the LADRC was more significant. For more extensive controller parameters, although the overshoot became smaller, the system shock could not be ignored, and such drastic changes in the flap deflection could not be realized in practice. The Euler angles in Figure a show the flight attitude of the parafoil during path tracking. From Figure b, we can observe that the agent had significantly different action values under the influence of wind disturbances. This is also the core idea of the proposed method: to adjust the controller parameters in real time according to the system state. Figure 8 shows the aerodynamic forces and moments of parafoil canopy and payload in three coordinate axes.

5.2.2. Straight Path Following

To prove that the above results were not accidental, the straight-path-following control for the parafoil system was also studied. The desired path consisted of three line segments, and the initial position of the parafoil was (0, 80 m). With the initial heading angle of 0${}^{\circ }$ and $b_0$ of 0.2, the simulation results by the DQN-LADRC are shown in Figure 9, Figure 10, Figure 11 and Figure 12. In order to reflect the anti-interference performance of the proposed method, a south wind of 5 m/s was added at 150 s and continued until the end of the simulation.

Figure 9 shows the tracking effect of the straight path. It can be seen that the system continuously adjusted the flight direction by controlling the rope after being affected by the wind. It can be observed from Figure 10 that the parafoil could fly according to the planned heading angle, and its attitude angles could finally be stabilized. That is, the parafoil could smoothly follow the designed path. The path tracking error is shown in Figure 11a, where the path tracking deviation was 0 at steady state. In addition, the parafoil had large tracking deviations during sharp turns. The adaptive controller parameters are given in Figure 11b, where ${\omega }_o\in [0.5,1.5]$ and ${\omega }_c\in [0.2,0.5]$, with sampling intervals of 0.01. To demonstrate the proposed method’s effectiveness, the LADRC controller’s control effect with fixed parameters of ${\omega }_o=1.5$ and ${\omega }_c=0.5$ was added to the results. The aerodynamic forces and moments on the canopy and payload are shown in Figure 12 in three coordinate axes. It can be seen that in the z-axis direction there are aerodynamic effects. This is because in the eight-DOF model in this paper, we default that the parafoil is free-falling in the longitudinal direction. Even so, the proposed method can enable the parafoil to track a given path on the plane.

By comparing the tracking effect of the circular path and the straight path, it can be seen that the tracking effect of the straight path was better. This was because the guidance law is based on the idea of tracking a straight line. In circular path tracking, a circle is approximated as an infinite number of straight line segments.

6. Conclusions

This paper studied the lateral path-following control of the parafoil system. First, an eight-DOF dynamic model of the parafoil system was constructed, considering the relative motion between the parafoil canopy and the payload. Then, using the error conversion between the parafoil system’s position information and the desired path, a guidance law was proposed, which essentially converts the path following into the control of the parafoil system’s heading angle. Furthermore, a second-order LADRC was designed based on the guidance law, and the DQN was applied to optimize the controller parameters adaptively. Finally, the proposed DNQ-LADRC was applied to control the parafoil system to follow a circular path under environments with and without disturbances and a straight path under wind disturbances. The simulation results demonstrated that the influence of wind disturbances on the parafoil could not be ignored. The proposed method could overcome the effect of wind disturbances and realize the tracking control of straight or circular paths. Compared with the traditional LADRC controller, the proposed method has certain advantages in terms of the settling time and overshoot.

In future work, we will simultaneously consider implementing lateral and longitudinal path tracking control of the parafoil system and applying the proposed method to actual experiments for verification. In addition, the DQN can realize the optimization of controller parameters, but it also requires human experience to divide the action space. The agent training of the DQN will be difficult when the action space is too large. Therefore, one of our goals is to explore a fully intelligent reinforcement learning algorithm. In this study, only the disturbances caused by uniform wind were considered, but in the actual flight process, the parafoil system will be affected by various complex wind fields. Thus, path-following control of the parafoil based on wind field identification is what we hope to achieve.