Application of Deep Reinforcement Learning in Reconfiguration Control of Aircraft Anti-Skid Braking System

Abstract

The aircraft anti-skid braking system (AABS) plays an important role in aircraft taking off, taxiing, and safe landing. In addition to the disturbances from the complex runway environment, potential component faults, such as actuators faults, can also reduce the safety and reliability of AABS. To meet the increasing performance requirements of AABS under fault and disturbance conditions, a novel reconfiguration controller based on linear active disturbance rejection control combined with deep reinforcement learning was proposed in this paper. The proposed controller treated component faults, external perturbations, and measurement noise as the total disturbances. The twin delayed deep deterministic policy gradient algorithm (TD3) was introduced to realize the parameter self-adjustments of both the extended state observer and the state error feedback law. The action space, state space, reward function, and network structure for the algorithm training were properly designed, so that the total disturbances could be estimated and compensated for more accurately. The simulation results validated the environmental adaptability and robustness of the proposed reconfiguration controller.

1. Introduction

The aircraft anti-skid braking system (AABS) is an essential airborne utilities system to ensure the safe and smooth landing of aircraft [1]. With the development of aircraft towards high speed and large tonnage, the performance requirements of AABS are increasing. Moreover, AABS is a complex system with strong nonlinearity, strong coupling, and time-varying parameters, and is sensitive to the runway environment [2]. These characteristics make AABS controller design an interesting and challenging topic.

The most widely used control method in practice is PID + PBM, which is a speed differential control law. However, it suffers from low-speed slipping and underutilization of ground bonding forces, making it difficult to meet high performance requirements. To this end, researchers have proposed many advanced control methods to improve the AABS performance, such as mixed slip deceleration PID control [3], model predictive control [4], extremum-seeking control [5], sliding mode control [6], reinforcement Q-learning control [7], and so on. Zhang et al. [8] proposed a feedback linearization controller with a prescribed performance function to ensure the transient and steady-state braking performance. Qiu et al. [9] combined backstepping dynamic surface control with an asymmetric barrier Lyapunov function to obtain a robust tracking response in the presence of disturbance and runway surface transitions. Mirzaei et al. [10] developed a fuzzy braking controller optimized by a genetic algorithm and introduced an error-based global optimization approach for fast convergence near the optimum point. The above-mentioned works provide an in-depth study on AABS control; however, the adverse effects caused by typical component faults such as actuator faults are neglected. Since most AABS are designed based on hydraulic control systems, the long hydraulic pipes create an enormous risk of air mixing with oil, and internal leakage. Without regular maintenance, it is easy to cause functional degradation or even failure, which raises many security concerns [11,12]. How to ensure the stability and the acceptable braking performance of AABS after actuator faults becomes a key issue.

In order to actually improve the safety and reliability of AABS, the fault probability can be reduced by reliability design and redundant technology on the one hand [13]. However, due to the production factors (cost/weight/technological level), the redundancy of aircraft components is so limited that the system reliability is hard to increase. On the other hand, fault-tolerant control (FTC) technology can be introduced into the AABS controller design, which is the future development direction of AABS and the key technology that needs urgent attention [14]. Reconfiguration control is a popular branch of FTC that has been widely used in many safety-critical systems, especially in aerospace engineering [15,16]. The essence of reconfiguration control is to consider the possible faults of the plant in the controller design process. When component faults occur, the fault system information is used to reconfigure the controller structure or parameters automatically [17]. In this way, the adverse effects caused by faults can be restrained or eliminated, thus realizing an asymptotically stable and acceptable performance of the closed-loop system. A number of common reconfiguration control methods can be classified as follows: adaptive control [18,19], multi-model switching control [20], sliding mode control [21], fuzzy control [22], other robust control [23], etc. In addition, the characteristics of AABS increase the difficulty of accurate modeling, and many nonlinear reconfiguration control methods are complex and relatively hard to apply in engineering. Therefore, it is crucial to design a reconfiguration controller with a clear structure, and which is model-independent, strong fault-perturbation resistant, and easy to implement.

Han retained the essence of PID control and proposed an active disturbance rejection control (ADRC) technique that requires low model accuracy and shows good control performance [24]. ADRC can estimate disturbances in internal and external systems and compensate for them [25]. Furthermore, ADRC has been widely used in FTC system design because of its obvious advantages in solving control problems of nonlinear models with uncertainty and strong disturbances [26,27,28]. Although the structure is not difficult to implement with modern digital computer technology, ADRC needs to tune a bunch of parameters which makes it hard to use in practice [29]. To overcome the difficulty, Gao proposed linear active disturbance rejection control (LADRC), which is based on linear extended state observer (LESO) and linear state error feedback (LSEF) [30,31]. The bandwidth tuning method greatly reduced the number of LADRC parameters. LADRC has been applied to solve various control problems [32,33,34].

However, it is well known that a controller with fixed parameters may not be able to maintain the acceptable (rated or degraded) performance of a fault system. For this reason, some advanced algorithms with parameter adaptive capabilities have been introduced by researchers that further improve the robustness and environmental adaptability of ADRC, such as neural networks [35,36], fuzzy logic [37,38], and the sliding mode [39,40]. With the development of artificial intelligence techniques, reinforcement learning has been applied to control science and engineering [41,42], and good results have been achieved. Yuan et al. proposed a novel online control algorithm for a thickener which is based on reinforcement learning [43]. Pang et al. studied the infinite-horizon adaptive optimal control of continuous-time linear periodic systems, using reinforcement learning techniques [44]. A Q-learning-based adaptive method for ADRC parameters was proposed by Chen et al. and has been applied to the ship course control [45].

Motivated by the above observations, in this paper, a reconfiguration control scheme via LADRC combined with deep reinforcement learning was developed for AABS which is subject to various fault perturbations. The proposed reconfiguration control method is a remarkable control strategy compared to previous methods for three reasons:

(1) AABS is extended with a new state variable, which is the sum of all unknown dynamics and disturbances not noticed in the fault-free system description. This state variable can be estimated using LESO. It indirectly simplifies the AABS modeling;

(2) Artificial intelligence technology is introduced and combined with the traditional control method to solve special control problems. By combining LADRC with the deep reinforcement learning TD3 algorithm, the selection of controller parameters is equivalent to the choice of agent actions. The parameter adaptive capabilities of LESO and LSEF are endowed through the continuous interaction between the agent and the environment, which not only eliminates the tedious manual tuning of the parameters, but also results in more accurate estimation and compensation for the adverse effects of fault perturbations;

(3) It is a data-driven robust control strategy that does not require any additional fault detection or identification (FDI) module, while the controller parameters are adaptive. Therefore, the proposed method corresponds to a novel combination of active reconfiguration control and FDI-free reconfiguration control, which makes it an interesting solution under unknown fault conditions.

The paper is organized as follows. Section 2 describes AABS dynamics with an actuator fault factor. The reconfiguration controller is presented in Section 3. The simulation results are presented to demonstrate the merits of the proposed method in Section 4, and conclusions are drawn in Section 5.

2. AABS Modeling

The AABS mainly consists of the following components: aircraft fuselage, landing gear, wheels, a hydraulic servo system, a braking device, and an anti-skid braking controller. The subsystems are strongly coupled and exhibit strong nonlinearity and complexity.

Based on the actual process and objective facts of anti-skid braking, the following reasonable assumptions can be made [46]:

(1)

The aircraft fuselage is regarded as a rigid body with concentrated mass;

(2)

The gyroscopic moment generated by the engine rotor is not considered during the aircraft braking process;

(3)

The crosswind effect is ignored;

(4)

Only the longitudinal deformation of the tire is taken into account and the deformation of the ground is ignored;

(5)

All wheels are the same and controlled synchronously.

2.1. Aircraft Fuselage Dynamics

The force diagram of the aircraft fuselage is shown in Figure 1 and the specific parameters described in the diagram are shown in

Table 1. Parameters of aircraft fuselage dynamics.

Name	Description	Value
$H$	Center of gravity height
$y$	Center of gravity height variation
$V$	Aircraft speed
$T_0$	Engine force
$F_x$	Aerodynamic drag
$F_y$	Aerodynamic lift
$F_s$	Parachute drag
$f_1$	Braking friction force between main wheel and ground
$f_2$	Braking friction force between front wheel and ground
$N_1$	Main wheel support force
$N_2$	Front wheel support force
$m$	Mass of the aircraft	1761 kg
$g$	Gravitational acceleration	9.8 m/s2
$h_t$	Distance between engine force line and center of gravity	0.1 m
$h_s$	Distance between parachute drag line and center of gravity	0.67 m
$a$	Distance between main wheel and center of gravity	1.076 m
$b$	Distance between front wheel and center of gravity	6.727 m
$I$	Fuselage inertia	4000 kg·s2·m
$S$	Wing aera	50.88 m2
$S_s$	Parachute area	20 m2
$C_x$	Aerodynamic drag coefficient	0.1027
$C_y$	Aerodynamic lift coefficient	0.6
$C_{xs}$	Parachute drag coefficient	0.75
$T_0^{\prime }$	Intimal engine force	426 kg
$K_v$	Velocity coefficient of engine	1 kg·s/m
$\rho$	Air density	4000 kg·s2/m4

.

The aircraft force and torque equilibrium equations are:

$$\left\{\begin{array}{l}m\dot{V}+F_x+F_s+f_1+f_2-T_0=0\\ F_y+N_1+N_2-mg=0\\ N_{2}b+F_s{h}_s-N_{1}a-T_0{h}_t-f_{1}H-f_{2}H=0\end{array}\right.$$

(1)

According to the influence of aerodynamic characteristics, we can obtain [46]:

$$\left\{\begin{array}{l}{T}_0=T_0^{\prime }+K_{v}V\\ F_x=\frac{1}{2}\rho C_{x}S{V}^2\\ F_y=\frac{1}{2}\rho C_{y}S{V}^2\\ F_s=\frac{1}{2}\rho C_{xs}{S}_s{V}^2\\ f_1={\mu }_1{N}_1\\ f_2={\mu }_2{N}_2\end{array}\right.$$

(2)

2.2. Landing Gear Dynamics

The main function of the landing gear is to support and buffer the aircraft, thus improving the longitudinal and vertical forces. In addition to the wheel and braking device, the struts, buffers, and torque arm are also the main components of the landing gear. In this paper, it is assumed that the stiffness of the torque arm is large enough, and the torsional freedom of the wheel with respect to the strut and the buffer is ignored, so the torque arm is not considered.

The buffer can be reasonably simplified as a mass-spring-damping system [46], and the force acting on the aircraft fuselage by the buffer can be described as:

$$\left\{\begin{array}{l}{N}_1=K_1{X}_1+C_1{\dot{X}}_1^2\\ N_2=K_2{X}_2+C_2{\dot{X}}_2^2\end{array}\right.$$

(3)

$$\left\{\begin{array}{l}{X}_1=a+y\\ X_2=-b+y\end{array}\right.$$

(4)

whose parameters are shown in

Table 2. Parameters of the buffer.

Name	Description	Value
$X_1$	Main buffer compression
$X_2$	Front buffer compression
$K_1$	Main buffer stiffness coefficient	42,529
$K_2$	Front buffer stiffness coefficient	2500
$C_1$	Main buffer damping coefficient	800
$C_2$	Front buffer damping coefficient	800

.

Due to the non-rigid connection between the landing gear and the aircraft fuselage, horizontal and angular displacements are generated under the action of braking forces. However, the struts are cantilever beams, and their angular displacements are very small and negligible. Therefore, the lateral stiffness model can be expressed by the following equivalent second-order equation:

$$\left\{\begin{array}{l}{d}_a=\frac{-\frac{f_1}{K_0}}{\frac{1}{W_n^2}{s}^2+\frac{2\xi }{W_n}s+1}\\ d_V=\frac{d}{dt}(d_a)\end{array}\right.$$

(5)

whose parameters are shown in

Table 3. Parameters of the landing gear lateral stiffness model.

Name	Description	Value
$d_a$	Navigation vibration displacement	Please see Equation (5)
$d_V$	Navigation vibration speed	Please see Equation (5)
$K_0$	Dynamic stiffness coefficient	536,000
$\xi$	Dynamic stiffness coefficient	0.2
$W_n$	Equivalent model natural frequency	60 Hz

.

2.3. Wheel Dynamics

The force diagram of the main wheel brake is shown in Figure 2.

It can be seen that during the taxiing, the main wheel is subjected to a combined effect of the braking torque $M_s$ and the ground friction torque $M_j$. Due to the effect of the lateral stiffness, there is a longitudinal axle velocity $V_{zx}$ along the fuselage, which is superimposed by the aircraft velocity $V$ and the navigation vibration velocity $d_V$. The dynamics equation of the main wheel is [46]:

$$\left\{\begin{array}{l}\dot{\omega }=\frac{M_j-M_s}{J}+\frac{V_{zx}}{R_g}\\ V_w=\omega R_g\\ V_{zx}=V+d_V\\ R_g=R-N{k}_{\sigma }\\ M_j=\mu N{R}_{g}n\end{array}\right.$$

(6)

whose parameters are shown in

Table 4. Parameters of the main wheel.

Name	Description	Value
$\omega$	Main wheel angular velocity
$\dot{\omega }$	Main wheel angular acceleration
$V_w$	Main wheel line speed
$R_g$	Main wheel rolling radius
$N$	Radical load
$J$	Main wheel inertia	1.855 kg·s2·m
$R$	Wheel free radius	0.4 m
$k_{\sigma }$	Tire compression coefficient	1.07 × 10−5 m/kg
$n$	Equivalent model natural frequency	4

.

During the braking, the tires are subjected to the braking torque that keeps the aircraft speed always greater than the wheel speed, that is $V>V_w$. Thus, the slip ratio $\lambda$ is defined to represent the slip motion ratio of the wheels relative to the runway. For the main wheel, using $V_{zx}$ instead of $V$ to calculate $\lambda$ can avoid false brake release due to landing gear deformation, thus effectively reducing the landing gear walk situation [46]. The following equation is used to calculate the slip rate in this paper:

$$\lambda =\frac{V_{zx}-V_w}{V_{zx}}$$

(7)

The tire–runway combination coefficient is related to many factors, including real-time runway conditions, aircraft speed, slip rate, and so on. A simple empirical formula called ‘magic formula’ developed by Pacejka [47] is widely used to calculate and can be expressed as follows:

$$\mu (\lambda ,{\tau }_j)={\tau }_1\sin ({\tau }_2\mathrm{arctg}({\tau }_3\lambda ))$$

(8)

where ${\tau }_j(j=1,2,3)$, ${\tau }_1$, ${\tau }_2$, ${\tau }_3$ are peak factor, stiffness factor, and curve shape factor, respectively.

Table 5. Parameters of the runway status.

Runway Status	${\mathit{\tau }}_{\mathbf{1}}$	${\mathit{\tau }}_{\mathbf{2}}$	${\mathit{\tau }}_{\mathbf{3}}$
Dry runway	0.85	1.5344	14.5
Wet runway	0.40	2.0	8.2
Snow runway	0.28	2.0875	10

lists the specific parameters for several different runway statuses [48].

2.4. Hydraulic Servo System and Braking Device Modeling

Due to the complex structure of the hydraulic servo system, in this paper, some simplifications have been made so that only electro-hydraulic servo valves and pipes are considered. Their transfer functions are given as follows:

$$\left\{\begin{array}{l}M(s)=\frac{K_{sv}}{\frac{s^2}{{\omega }_{sv}^2}+\frac{2{\xi }_{sv}s}{{\omega }_{sv}}+1}\\ L(s)=\frac{K_p}{T_{p}s+1}\end{array}\right.$$

(9)

whose parameters are shown in

Table 6. Parameters of the hydraulic servo system.

Name	Description	Value
$K_{sv}$	Servo valve gain	1
${\omega }_{sv}$	Servo valve natural frequency	17.7074 rad/s
${\xi }_{sv}$	Servo valve damping ratio	0.36
$K_p$	Main wheel rolling radius	1
$T_p$	Pipe gain	0.01
${\mu }_{mc}$	Friction coefficient of brake material	0.23
$N_{mc}$	Number of friction surfaces	4
$R_{mc}$	Effective brake friction radius	0.142 m

.

It should be noted that the anti-skid braking controller should realize both braking control and anti-skid control. To this end, there is an approximately linear relationship between the brake pressure $P$ and the control current $I_c$, which can be described as follows:

$$P=-I_{c}M(s)L(s)+P_0$$

(10)

where $P_0=1\times {10}^7 \mathrm{Pa}$.

The braking device serves to convert the brake pressure into brake torque, which is calculated as follows:

$$M_s={\mu }_{mc}{N}_{mc}P{R}_{mc}$$

(11)

whose parameters are shown in Table 6.

The hydraulic servo system, as the actuator of AABS, is inevitably subject to some potential faults. Problems such as hydraulic oil mixing with air, internal leakage, and vibration seriously affect the efficiency of the hydraulic servo system [49]. Therefore, in this paper, the loss of efficiency (LOE) is introduced to represent a typical AABS actuator fault, which is characterized by a decrease in the actuator gain from its nominal value [26]. In the case of an actuator LOE fault, the brake pressure generated by the hydraulic servo system deviates from the commanded output expected by the controller. In other words, one instead has:

$$P_{fault}=k_{LOE}P$$

(12)

where $P_{fault}$ represents the actuator actual output, and $k_{LOE}\in (0,1]$ refers the LOE fault factor.

Remark 1.

$n\%$LOE is equivalent to the LOE fault gain $k_{LOE}=1-n/100$, $k_{LOE}=1$indicates that the actuator is fault-free.

Remark 2.

Note that if the components do not always have the same characteristics as those of fault-free, it is necessary to establish the fault model. This not only provides an accurate model for the next reconfiguration on controller design, but also ensures that the adverse effects caused by fault perturbation can be effectively observed and compensated for.

Thus, Equation (11) can be rewritten as follows:

$${M_s}^{\prime }={\mu }_{mc}{N}_{mc}{P}_{fault}{R}_{mc}$$

(13)

where ${M_s}^{\prime }$ is the actual brake torque.

Remark 3.

As can be seen from the entire modeling process described above, AABS is nonlinear and highly coupled. The actuator fault leads to a sudden jump in the model parameters with greater internal perturbation compared to the fault-free case. Meanwhile, external disturbances such as the runway environment cannot be ignored.

3. Reconfiguration Controller Design

3.1. Problem Description

Despite the aircraft having three degrees of freedom, only longitudinal taxiing is focused on in AABS. In this paper, AABS adopted the slip speed control type [48], that is, the braked wheel speed $V_{\omega }$ was used as the reference input, and the aircraft speed $V$ was dynamically adjusted by the AABS controller to achieve anti-skid braking. According to Section 2, the AABS longitudinal dynamics model can be rewritten as follows:

$$\ddot{V}=f(V,\dot{V},{\varpi }_{out},{\varpi }_f)+b_{v}u$$

(14)

where $f(\cdot )$ is the controlled plant dynamics, ${\varpi }_{out}$ represents the external disturbance, ${\varpi }_f$ is an uncertain term including component faults, $b_v$ is the control gain, and $u$ is the system input.

Let $x_1=V$, $x_2=\dot{V}$. Set $f(V,\dot{V},{\varpi }_{out},{\varpi }_f)$ as the system generalized total perturbation and extend it to a new system state variable, i.e., $x_3=f(V,\dot{V},{\varpi }_{out},{\varpi }_f)$. Then the state equation of System (14) can be obtained:

$$\left\{\begin{array}{l}{\dot{x}}_1=x_2\\ {\dot{x}}_2=x_3+b_{v}u\\ {\dot{x}}_3=h(V,\dot{V},{\varpi }_{out},{\varpi }_f)\end{array}\right.$$

(15)

where $x_1,x_2,x_3$ are system state variables, and $h(V,\dot{V},{\varpi }_{out},{\varpi }_f)=\dot{f}(V,\dot{V},{\varpi }_{out},{\varpi }_f)$.

Assumption 1.

Both the system generalized total perturbation $f(V,\dot{V},{\varpi }_{out},{\varpi }_f)$and its differential $h(V,\dot{V},{\varpi }_{out},{\varpi }_f)$are bounded, i.e., $\left\{\begin{array}{l}|f(V,\dot{V},{\varpi }_{out},{\varpi }_f)|\le {\sigma }_1\\ |h(V,\dot{V},{\varpi }_{out},{\varpi }_f)|\le {\sigma }_2\end{array}\right.$, where ${\sigma }_1,{\sigma }_2$are two positive numbers.

For System (14), affected by the total perturbation, a LADRC reconfiguration controller was designed next to restrain or eliminate the adverse effects, thus realizing the asymptotic stability and acceptable performance of the closed-loop system.

3.2. LADRC Controller Design

The control schematic of the LADRC is shown in Figure 3.

Firstly, the following tracking differentiator (TD) was designed:

$$\left\{\begin{array}{l}e\left(k\right)=v_1\left(k\right)-v_r\left(k\right)\\ \mathrm{fh}=\mathrm{fhan}\left(e\left(k\right),v_2\left(k\right),r,h\right)\\ v_1\left(k+1\right)=v_1\left(k\right)+h{v}_2\left(k\right)\\ v_2\left(k+1\right)=v_2\left(k\right)+h\mathrm{fh}\end{array}\right.$$

(16)

where $v_r$ is the desired input, $v_1$ is the transition process of $v_r$, $v_2$ is the derivative of $v_1$, and $r$ and $h$ are adjusted accordingly as filter coefficients. The function $\mathrm{fhan}\left(\cdot \right)$ is defined as follows:

$$\mathrm{fhan}\left(x_1,x_2,r,h\right)=-\left\{\begin{array}{l}r\mathrm{sgn}\left(a\right),\left|a\right|>d_0\\ r\frac{a}{d},\left|a\right|\le d_0\end{array}\right.$$

(17)

We established the following form, LESO:

$$\left\{\begin{array}{l}{\dot{z}}_1=z_2-{\beta }_1(z_1-v_1)\\ {\dot{z}}_2=z_3-{\beta }_2(z_1-v_1)+b_{v}u\\ {\dot{z}}_3=-{\beta }_3(z_1-v_1)\end{array}\right.$$

(18)

Selecting the suitable observer gains $({\beta }_1,{\beta }_2,{\beta }_3)$, LESO then enabled real-time observation of the variables in System (14) [50], i.e., $z_1\to v_1$, $z_2\to v_2$, $z_3\to f(V,\dot{V},{\varpi }_{out},{\varpi }_f)$.

Set

$$u=\frac{u_0-z_3}{b_v}$$

(19)

When $z_3$ can estimate $f(V,\dot{V},{\varpi }_{out},{\varpi }_f)$ without error, let LSEF be:

$$\left\{\begin{array}{l}{e}_1=v_1-z_1\\ e_2=v_2-z_2\\ u_0=k_1{e}_1+k_2{e}_2\end{array}\right.$$

(20)

then the system (15) can be simplified to a double integral series structure:

$$\ddot{V}=(f(V,\dot{V},{\varpi }_{out},{\varpi }_f)-z_3)+u_0\approx u_0$$

(21)

Further, the bandwidth method [50] was used and we could obtain:

$$\left\{\begin{array}{l}{\beta }_1=3{\omega }_o\\ {\beta }_2=3{\omega }_o^2\\ {\beta }_3={\omega }_o^3\end{array}\right.$$

(22)

where ${\omega }_o$ is the observer bandwidth. The larger ${\omega }_o$ is, the smaller LESO observation errors are. However, the sensitivity of the system to noise may be increased, so the ${\omega }_o$ selection requires comprehensive consideration.

Similarly, according to the parameterization method and engineering experience [32], the LSEF parameters can be chosen as:

$$\left\{\begin{array}{l}{k}_1={\omega }_c^2\\ k_2=2\xi {\omega }_c\end{array}\right.$$

(23)

where ${\omega }_c$ is the controller bandwidth, $\xi$ is the damping ratio, and in this paper $\xi =1$. Therefore, the parameter tuning problem of LADRC controller was simplified to the observer bandwidth ${\omega }_o$ and controller bandwidth ${\omega }_c$ configuration.

3.3. TD3 Algorithm

TD3 algorithm is an offline RL algorithm based on DDPG proposed in 2015 [51]. This approach adopted a similar method implemented in Double-DQN [52] to reduce the overestimation in function approximation, delaying the update frequency in the actor–network, and adding noises to target the actor–network to release the sensitivity and instability in DDPG. The structure of TD3 is shown in Figure 4.

Updating the parameters of critic networks by minimizing loss:

$$L=N^{-1}\sum {\left(y-Q_{{\theta }_i}\left(s,a\right)\right)}^2$$

(24)

where $s$ is the current state, $a$ is the current action, and $Q_{{\theta }_i}\left(s,a\right)$ stands for the parameterized state-action value function $Q$ with parameter ${\theta }_i$.

$$y=r+\gamma \underset{i=1,2}{\min }{Q}_{{\theta }_i^{\prime }}\left(s^{\prime },\tilde{a}\right)$$

(25)

is the target value of the function $Q_{\theta }\left(s,a\right)$, $\gamma \in \left[0,1\right]$ is the discount factor, and the target action is defined as:

$$\tilde{a}={\pi }_{{\varphi }^{\prime }}\left(s\right)+{ϵ}^{\prime }$$

(26)

where noise ${ϵ}^{\prime }$ follows a clipped normal distribution clip $\left[\mathcal{N}\left(0,\sigma \right),-c,c\right],c>0$. This implies that ${ϵ}^{\prime }$ is a random variable with $\mathcal{N}\left(0,\sigma \right)$ and belongs to the interval $\left[-c,c\right]$.

The inputs of the actor network are both $Q_{\theta }\left(s,a\right)$ from the critic network and the minibatch form the memory, and the output is the action given by:

$$a_t={\pi }_{\varphi }\left(s_t\right)+ϵ$$

(27)

where $\varphi$ is the parameter of the actor network, and ${\pi }_{\varphi }$ is the output form the actor network, which is a deterministic and continuous value. Noise $ϵ$ follows the normal distribution $\mathcal{N}\left(0,\sigma \right)$, and is added for exploration.

Updating the parameters of the actor–network based on deterministic gradient strategy:

$${\nabla }_{\varphi }J\left(\varphi \right)=N^{-1}\sum {\nabla }_a{\left.Q_{{\theta }_1}\left(s,a\right)\right|}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)$$

(28)

TD3 updates the actor–network and all three target networks every d steps periodically in order to avoid a too fast convergence. The parameters of the critic target networks and the actor–target network are updated according to:

$$\left\{\begin{array}{l}{\theta }_i^{\prime }\leftarrow \tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }\\ {\varphi }^{\prime }\leftarrow \tau \varphi +\left(1-\tau \right){\varphi }^{\prime }\end{array}\right.$$

(29)

The pseudocode of the proposed approach is given in Algorithm 1.

Algorithm 1. TD3

1 Initialize critic networks $Q_{{\theta }_1}$$,Q_{{\theta }_2}$ and actor network ${\pi }_{\varphi }$ with random parameters ${\theta }_1$$,{\theta }_2,\varphi$; 2 Initialize target networks $Q_{{\theta }_1^{\prime }}$, $Q_{{\theta }_2^{\prime }}$ with ${\theta }_1^{\prime }\leftarrow {\theta }_1$$,{\theta }_2^{\prime }\leftarrow {\theta }_2$, and target actor network ${\pi }_{{\varphi }^{\prime }}$ with ${\varphi }^{\prime }\leftarrow \varphi$; 3 Initialize replay buffer $ℛ$; 4 For every episode: 5 Initialize state $s$; 6 Repeat; 7 Select action with exploration noise $a\sim \pi \left(a\right)+\mathcal{N}\left(0,\sigma \right)$; 8 Observe reward r and new state $s^{\prime }$; 9 Store transition tuple $\left(s,a,r,s^{\prime }\right)$ in $\mathcal{R}$; 10 Sample mini-batch of $N$ transitions $\left(s,a,r,s^{\prime }\right)$ from $\mathcal{R}$; 11 Attain $\tilde{a}\leftarrow {\pi }_{{\varphi }^{\prime }}\left(s\right)+ϵ$$,\mathrm{where}ϵ\sim clip\left(\mathcal{N}\left(0,\sigma \right),-c,c\right)$; 12 Update critics ${\theta }_i\leftarrow {\min }_{{\theta }_i}{N}^{-1}\sum {\left(y-Q_{{\theta }_i}\left(s,a\right)\right)}^2$; 13 Every d steps: 14 Update $\varphi$ by the deterministic policy gradient: 15 ${\nabla }_{\varphi }J\left(\varphi \right)=N^{-1}\sum {\nabla }_a{\left.Q_{{\theta }_1}\left(s,a\right)\right|}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)$; 16 Update target network: 17 ${\theta }_i^{\prime }\leftarrow \tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$ 18 ${\varphi }^{\prime }\leftarrow \tau \varphi +\left(1-\tau \right){\varphi }^{\prime }$; 19 $s\leftarrow s^{\prime }$; 20 Until$s$ reaches terminal state $s_T$.

3.4. TD3-LADRC Reconfiguration Controller Design

Lack of environment adaptability, poor control performance, and weak robustness are the main shortcomings of parameter-fixed controllers [36]. When a fault occurs, it may not be possible to maintain the acceptable (rated or degraded) performance of the damaged system. Motivated by the above analysis, a reconfiguration controller called TD3-LADRC is proposed in this paper, and its control schematic is shown in Figure 5.

The deep reinforcement learning algorithm TD3 is introduced to realize the LADRC parameters adaption. The details of each part have been described above. The selection of control parameters is treated as the agent’s action $a_t$, and the response result of the control system $s_t$ is considered as the state, i.e., as follows:

$$\left\{\begin{array}{l}{a}_t={\left[{\omega }_o,{\omega }_c\right]}^{\mathrm{T}}\\ s_t=s_{obs}={\left[e,\dot{e},V,\dot{V}\right]}^{\mathrm{T}}\end{array}\right.$$

(30)

where $e=V-V_w$, and $s_{obs}$ is the agent observations vector.

The range of each controller parameter is selected as follows:

$$\left\{\begin{array}{l}{\omega }_c\in \left[0,4\right]\\ {\omega }_o\in \left[100,200\right]\end{array}\right.$$

(31)

The reward function plays a crucial role in the reinforcement learning algorithm. The appropriateness of the reward function design directly affects the training effect of the reinforcement learning, which in turn affects the effectiveness of the whole reconfiguration controller. According to the working characteristics of AABS, the following reward function is selected after several attempts to ensure stable and smooth braking:

$$r_t=1\left(-6\le \dot{V}\le -4\right)+0\left(\dot{V}>-4\parallel \dot{V}<-6\right)-100\left(V<2\parallel e>20\right)$$

(32)

The stop conditions for each training episode are as follows, and one of the three will do:

(1)

The aircraft speed $V<2$;

(2)

The error between main wheel speed and aircraft speed $e>20$;

(3)

Simulation time $t>20 \mathrm{s}$.

Remark 4.

TD3, TD, LESO, and LSEF together constitute the TD3-LADRC controller. Compared to normal LADRC, TD3-LADRC realizes the parameter adaption that makes the controller reconfigurable. The robustness and immunity are greatly improved. It can effectively compensate the adverse effects caused by the total perturbations including faults.

3.5. TD3-LESO Estimation Capability Analysis

In order to prove the stability of the whole closed-loop system, the convergence of TD3-LESO is first analyzed in conjunction with Assumption 1 [53]. Let the estimation errors of TD3-LESO be ${\tilde{x}}_i=x_i-z_i,i=1,2,3$, and the estimation error equation of the observer can be obtained as:

$$\left\{\begin{array}{l}{\dot{\tilde{x}}}_1={\tilde{x}}_2-3{\omega }_o{\tilde{x}}_1\\ {\dot{\tilde{x}}}_2={\tilde{x}}_3-3{\omega }_o^2{\tilde{x}}_1\\ {\dot{\tilde{x}}}_3=h(V,\dot{V},{\varpi }_{out},{\varpi }_f)-{\omega }_o^3{\tilde{x}}_1\end{array}\right.$$

(33)

Let ${\epsilon }_i=\frac{{\tilde{x}}_i}{{\omega }_o^{i-1}},i=1,2,3$, then Equation (33) can be rewritten as:

$$\dot{\epsilon }={\omega }_o{A}_3\epsilon +B\frac{h(V,\dot{V},{\varpi }_{out},{\varpi }_f)}{{\omega }_o^2}$$

(34)

where $A_3=\left[\begin{array}{ccc}-3& 1& 0\\ -3& 0& 1\\ -1& 0& 0\end{array}\right]$, $B={[\begin{array}{ccc}0& 0& 1\end{array}]}^{\mathrm{T}}$.

Based on Assumption 1 and Theorem 2 in Reference [54], the following theorem can be obtained:

Theorem 1.

Under the condition that $h(V,\dot{V},{\varpi }_{out},{\varpi }_f)$is bounded, the TD3-LESO estimation errors are bounded and their upper bound decrease monotonically with the increase of the observer bandwidth ${\omega }_o$.

The proof is given in the Appendix A. Thus, it is clear that there are three positive numbers ${\upsilon }_i,i=1,2,3$, such that the state estimation error $\left|{\tilde{x}}_i\right|\le {\upsilon }_i$ holds, i.e., the TD3-LESO estimation errors are bounded, which can effectively estimate the states of the controlled plant and the total perturbation.

3.6. Stability Analysis of Closed-loop System

The closed-loop system consisted of the control laws (19) and (20), and the controlled object (21) is:

$$\ddot{V}=f-z_3+k_1{e}_1+k_2{e}_2$$

(35)

If we defined the tracking errors as ${\epsilon }_i=v_i-x_i,i=1,2$, then we could attain:

$$\left\{\begin{array}{l}{\dot{\epsilon }}_1={\dot{r}}_1-{\dot{x}}_1=r_2-x_2={\tilde{e}}_2\\ {\dot{\epsilon }}_2={\dot{r}}_2-{\dot{x}}_2=r_3-\ddot{V}\\ =-k_1{\epsilon }_1-k_1{\tilde{x}}_1-k_2{\epsilon }_2-k_2{\tilde{x}}_2-{\tilde{x}}_3\end{array}\right.$$

(36)

Let $\epsilon ={\left[{\epsilon }_1,{\epsilon }_2\right]}^T,\tilde{x}={\left[{\tilde{x}}_1,{\tilde{x}}_2,{\tilde{x}}_3\right]}^T$, then:

$$\dot{\epsilon }(t)=A_{\epsilon }\epsilon (t)+A_{\tilde{x}}\tilde{x}(t)$$

(37)

where $A_e=\left[\begin{array}{cc}0& 1\\ -k_1& -k_2\end{array}\right]$, $A_{\tilde{x}}=\left[\begin{array}{ccc}0& 0& 0\\ -k_1& -k_2& -1\end{array}\right]$.

By solving Equation (37):

$$\epsilon (t)=e^{A_{\epsilon }t}\epsilon (0)+{\displaystyle {\int }_0^t{e}^{A_{\epsilon }(t-\tau )}}{A}_{\tilde{x}}\tilde{x}(\tau )d\tau$$

(38)

Combining Assumption 1, Theorem 1, Theorem 3, and Theorem 4 in the literature [54], the following theorem was proposed to analyze the stability of the closed-loop system:

Theorem 2.

Under the condition that the TD3-LESO estimation errors are bounded, there exists a controller bandwidth ${\omega }_c$, such that the tracking error of the closed-loop system is bounded. Thus, for a bounded input, the output of the closed-loop system is bounded, i.e., the closed-loop system is BIBO-stable.

See the Appendix A for proof.

4. Simulation Results

In order to verify the reconfiguration capability and disturbance rejection capabilities of the proposed method, the corresponding simulations are carried out in this section and compared with conventional PID + PBM and LADRC.

The initial states of the aircraft are set as follows:

(1)

The initial speed of aircraft landing $V\left(0\right)=72 \mathrm{m}/\mathrm{s}$;

(2)

The initial height of the center of gravity $Hh=2.178 \mathrm{m}$.

To prevent deep wheel slippage as well as tire blowout, the wheel speed was kept following the aircraft speed quickly at first, and the brake pressure was applied only after 1.5 s. The anti-skid brake control was considered to be over when $V$ was less than 2 m/s.

In the experiment, both the critic networks and the actor networks were realized by a fully connected neural network with three hidden layers. The number of neurons in the hidden layer was (50,25,25). The activation function of the hidden layer was selected as the ReLU function, and the activation function of the output layer of the actor network was selected as the tanh function. In addition, the parameters of the actor network and the critic network were tuned by an Adam optimizer. The remaining parameters of TD3-LADRC are shown in

Table 7. Parameters of TD3-LADRC.

Name	Value
Control gain $b_v$	2
$TD-r$	0.001
$TD-h$	1
Discount factor $\gamma$	0.99
Actor learning rate	0.0001
Critic learning rate	0.001
Target update rate $\tau$	0.001

.

Remark 5.

It is noted that the braking time t and braking distance x are selected as the criteria for braking efficiency, and the system stability is observed by slip rate $\lambda$.

The model simulation was carried out in MATLAB 2022a, and the TD3 algorithm was realized through the reinforcement learning toolbox. The simulation time was 20 s, the sampling time was 0.001 s. The training stopped when the average reward reached 12,000. The training took about 6 h to complete. The learning curves of the reward obtained by the agent for each interaction with the environment during the training process are shown in Figure 6.

It can be seen that at the beginning of the training, the agent was in the exploration phase and the reward obtained was relatively low. Later, the reward gradually increased, and after 40 episodes, the reward was steadily maintained at a high level and the algorithm gradually converges.

4.1. Case 1: Fault-Free and External Disturbance-Free in Dry Runway Condition

The simulation results of the dynamic braking process for different control schemes are shown in Figure 7 and Figure 8 and

Table 8. AABS performance index.

Performance Index	PID + PBM	LADRC	TD3-LADRC
Braking time (s)	20.48	16.73	14.79
Braking distance (m)	811.9	595.46	571.18

.

As can be seen from Figure 7, PID + PBM leads to numerous skids during braking, which may cause serious loss to the tires. In contrast, LADRC and TD3-LADRC not only skid less frequently, but also have shorter braking time and braking distance. Moreover, the control effect of TD3-LADRC is better than LADRC. Figure 8 shows that TD3-LADRC can dynamically tune the controller parameters to accurately observe and compensate for the total disturbances, and thus improve the AABS performance.

Remark 6.

During the braking process, it is observed that in some instants${\omega }_c=0$. It may not affect the stability of the whole system. On the one hand, the value of ${\omega }_c$does not change the fact that $A_{\epsilon }$ is Hurwitz (see Proof of Theorem 2 for details). On the other hand, ${\omega }_c$ is constantly changed by the agent through a continuous interaction with the environment, and in these instants the agent considers ${\omega }_c=0$ as optimal, i.e., no anti-skid braking control leads to better braking results.

4.2. Case 2: Actuator LOE Fault in Dry Runway Condition

The fault considered here assumed a 20% actuator LOE at 5 s and escalated to 40% LOE at 10 s. The simulation results are shown in Figure 9 and Figure 10 and

Table 9. AABS performance index.

Performance Index	PID + PBM	LADRC	TD3-LADRC
Braking time (s)	23.48	-	17.70
Braking distance (m)	838.46	-	618.12

.

As can be seen in Figure 9, PID + PBM continuously performed a large braking and releasing operation under the combined effect of fault and disturbance. This makes braking much less efficient and risks dragging and flat tires. In addition, LADRC cannot brake the aircraft to a stop which is not allowed in practice. Figure 9c shows that there is a high frequency of wheel slip in the low-speed phase of the aircraft. In contrast, TD3-LADRC retains the experience gained from the agent’s prior training and continuously adjusts the controller parameters online based on the plant states, which ultimately allows the aircraft to brake smoothly. From Figure 10a, it can be seen that the total fault perturbations are estimated fast and accurately based on the adaptive LESO. Overall, TD3-LADRC not only improves the robustness and immunity of the controller in fault-perturbed conditions, but also greatly significantly improves the safety and reliability of AABS.

4.3. Case 3: Actuator LOE Fault in Mixed Runway Condition

The mixed runway structure is as follows: dry runway in the interval of 0–10 s, wet runway in the interval of 10–20 s, and snow runway after 20 s. The fault considered here assumed a 10% actuator LOE at 10 s. The simulation results are shown in Figure 11 and Figure 12 and

Table 10. AABS performance index.

Performance Index	PID + PBM	LADRC	TD3-LADRC
Braking time (s)	49.14	29.82	24.19
Braking distance (m)	1228.71	739.99	672.03

.

The deterioration of the runway conditions has resulted in a very poor tire–ground bond. It can be seen from Figure 11 that both braking time and braking distance have increased compared to the dry runway. Figure 12 shows that TD3-LADRC is still able to achieve controller parameters adaption, accurately observe the total fault perturbations, and effectively compensate for the adverse effects. The whole reconfiguration control system adapts well to runway changes. The environmental adaptability of AABS is improved.

5. Conclusions

A linear active disturbance rejection reconfiguration control scheme based on deep reinforcement learning was proposed to meet the higher performance requirements of AABS under fault-perturbed conditions. According to the composition structure and working principle, AABS mathematical model with an actuator fault factor is established. A TD3-LADRC reconfiguration controller was developed, and the parameters of LSEF and LESO were adjusted online using the TD3 algorithm. The simulation results under different conditions verified that the designed controller can effectively improve the anti-skid braking performance even under faults and perturbations, as well as different runway environments. It successfully strengthened the robustness, immunity, and environmental adaptability of the AABS, thereby improving the safety and reliability of the aircraft. However, TD3-LADRC is so complex that its control effectiveness was verified only by simulations in this paper. The combined effect caused by various uncertainties in practical applications on the robustness of the controller cannot be completely considered. Therefore, in future work, an aircraft braking hardware-in-loop experimental platform is necessary to build, consisting of the host PC, the target CPU, the anti-skid braking controller, the actuators, and the aircraft wheel. The host PC and the target CPU are the software simulation part, while the other four parts are the hardware part.