Learning-Based Optimal Control for Multiple Fixed-Wing UAVs with Prescribed Performance

Abstract

Fixed-wing unmanned aerial vehicle formation control confronts the dual challenges of achieving optimal performance amidst complex nonlinear dynamics and ensuring flight safety by constraining tracking errors. Existing reinforcement learning methods, though effective for optimal control, often overlook these critical safety constraints, which constitutes a serious shortcoming in safety-critical swarm operations where unmanned aerial vehicles confront highly nonlinear dynamics, unknown disturbances, and limited model knowledge, and these practical necessity drives us to synergistically integrate two previously separated techniques. To address this issue, this paper proposes a safe optimal control framework that synergistically integrates prescribed performance control with an actor–critic RL scheme. Specifically, a simplified actor–critic architecture is developed to derive a near-optimal controller for the coupled position–attitude dynamics without requiring an accurate model, thereby enhancing energy efficiency. Concurrently, the prescribed performance control is employed to transform the constrained formation error dynamics into an unconstrained system, guaranteeing that safety distances are strictly maintained. Lyapunov-based analysis proves that all signals in the closed-loop system are semi-globally uniformly ultimately bounded and that formation errors never violate the predefined performance boundaries.

1. Introduction

In recent years, unmanned aerial vehicle (UAV) technology has found extensive application in military reconnaissance, environmental monitoring, and logistics transportation. In particular, fixed-wing UAVs have become critical platforms for cooperative multi-UAV missions due to their long endurance, extended range, and high maneuverability [1,2,3]. Operating multiple fixed-wing UAVs in formation can effectively enhance mission efficiency and system redundancy. However, this simultaneously imposes stricter demands on the robustness and safety of control systems [4,5,6]. On one hand, the dynamics of fixed-wing UAVs are characterized by strong nonlinearity, model uncertainties, and external wind disturbances. In practical fixed-wing UAV systems, structural vibration, damping characteristics, and material properties may also affect control quality [7,8,9]. Lightweight UAV structures are sensitive to vibration, which can degrade sensor accuracy, disturb attitude responses, and reduce formation tracking performance. Recent studies on vibration-aware UAV structures, lightweight composite materials, and sustainable damping materials, such as cork-based composites, indicate that structural damping and material selection are important factors for improving UAV flight stability and control reliability. On the other hand, the necessity to maintain safe inter-UAV distances for collision avoidance requires that control designs explicitly account for constraints on position tracking errors.

For UAV formation control, the investigation of optimal control strategies holds significant theoretical and engineering value. Fixed-wing UAV systems are inherently complex, characterized by strong nonlinearity, multivariable coupling, and model uncertainties. Consequently, formation flight requires not only precise shape maintenance but also the optimization of control performance under energy constraints [10,11,12]. Traditional methods such as the linear quadratic regulator or linear quadratic gaussian struggle to directly address nonlinear dynamics and external disturbances. Furthermore, optimal control problems for nonlinear systems often reduce to solving the Hamilton–Jacobi–Bellman (HJB) equation, which is analytically intractable in practical engineering [13,14,15]. In recent years, reinforcement learning (RL) methods have been increasingly applied to UAV systems due to their potential in solving optimal control problems [16,17,18]. By adopting an actor–critic architecture, RL leverages the function approximation capabilities of neural networks to approximate the HJB equation without relying on an accurate model. This approach yields near-optimal control strategies, which offers a viable pathway for solving optimal control problems in complex nonlinear systems [19,20,21]. For example, the authors in [20] designed a simplified optimal control scheme through constructing the positive definite function with a simple form. In [21], the authors presented an optimized backstepping control scheme for nonlinear strict-feedback systems with unknown dynamic by using the simplified actor–critic architecture. Several recent studies have further advanced robust cooperative control [22] and learning-based swarm control [23]. Nevertheless, they still do not simultaneously address optimality and hard safety constraints. However, most of the aforementioned RL-based methods neglect safety constraints during flight, making them difficult to directly apply to safety-critical formation missions.

Furthermore, safety constraints constitute a critical issue that must be addressed during formation flight [24,25,26]. To avoid collisions, UAVs must maintain safe inter-agent distances, necessitating that position tracking errors be strictly confined within predefined bounds. Prescribed performance control (PPC) addresses this issue by transforming constrained tracking errors into an unconstrained system via predefined performance functions, thereby theoretically guaranteeing that errors satisfy boundary conditions at all times [27,28,29,30]. A prescribed-time PPC method is proposed to ensure docking errors converge to a user-defined domain in [31]. The authors in [32] designed a reconfigurable performance function to solve the control singularity problem of UAV systems under actuator saturation conditions. While this method has been validated in single-UAV flight control, effectively integrating prescribed performance constraints with an RL-based optimal control framework for multi-UAV cooperative formations while simultaneously accounting for the strong nonlinearity of fixed-wing UAVs remains a challenging and active area of research.

Although robust and adaptive control methods have been widely used for UAV formation systems, their primary objective is usually to ensure stability and robustness under uncertainties. For safety-critical fixed-wing UAV swarms, however, the controller must satisfy two requirements simultaneously: strict transient/steady-state bounds on formation errors for collision avoidance, and near-optimal control performance under coupled nonlinear position–attitude dynamics, unknown disturbances, and inaccurate model parameters. Robust/adaptive designs may guarantee boundedness, but they do not explicitly minimize an infinite-horizon performance index and may lead to conservative control actions. On the other hand, RL-based optimal control can approximate the HJB solution and improve control efficiency without an accurate model, but conventional RL schemes do not inherently prevent formation errors from violating safety boundaries during learning and transient maneuvers. PPC is therefore introduced to impose prescribed error constraints, and the actor–critic RL scheme is designed on the transformed unconstrained error dynamics to achieve near-optimal control. This complementary integration is the main motivation of the proposed framework.

Encouraged by the above analysis, this paper aims to combine PPC with RL-based optimal control to develop a safe optimal control framework for fixed-wing UAV formations. This framework is designed to achieve near-optimal formation tracking performance while guaranteeing flight safety. The main contributions are summarized as follows:

• A simplified actor–critic RL scheme is developed to handle the coupled position–attitude dynamics of fixed-wing UAV. By employing neural networks to approximate the solution to the HJB equation, the proposed method realizes approximate optimal control without relying on an accurate model, which reduces computational complexity and improving control energy efficiency.
• The prescribed performance function is introduced to impose predefined boundary constraints on formation errors, which ensures that safe inter-UAV distances are rigorously maintained throughout the formation flight and that addresses the limitation of existing RL control methods which typically neglect safety constraints.
• By using the Lyapunov theory, it is proven that all error signals in the closed-loop system are SGUUB, and that the formation errors remain strictly within the prescribed performance boundaries.

The remainder of this paper is organized as follows. Section 2 presents the UAV dynamics and formation error formulation. Section 3 describes the prescribed performance transformation and the actor–critic RL framework, including the optimal control law, adaptive law, and online learning mechanism. Section 4 provides stability analysis proving SGUUB and prescribed performance bounds. Section 5 shows simulation results, including setup, comparisons, sensitivity, robustness tests, and practical deployment discussions. Appendix A includes the introduction to neural network.

2. Preliminaries

This section introduces the dynamic models of fixed-wing unmanned aerial vehicles and virtual leaders, defines formation tracking errors, and clarifies the control objectives of this paper.

2.1. Dynamics of Fixed-Wing Unmanned Aerial Vehicle

For a multi-UAV system with N UAVs, let $\mathbb{N}:=\{1,2,\dots ,N\}$ represent the set of indices. The motion dynamics of the i-th UAV is formulated as

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{I}}_{ix}=v_{i}cos{\psi }_{i}cos{\theta }_i+l_{ix}\hfill \\ {\dot{I}}_{iy}=v_{i}sin{\psi }_{i}cos{\theta }_i+l_{iy}\hfill \\ {\dot{I}}_{iz}=v_{i}sin{\theta }_i+l_{iz}\hfill \end{array}\right.\end{array}$$

(1)

where $(I_{ix},I_{iy},I_{iz})$ denotes the position of the i-th UAV in an inertia coordinate frame, while $v_i,{\psi }_i,{\theta }_i$ are velocity, course angle, and pitch angle, respectively. $I_{ix},I_{iy},I_{iz}$ denote bounded disturbances in $x,y,z$ axes, respectively.

Considering the autopilots onboard, the time response dynamics of $v_i,{\psi }_i,{\theta }_i$ are modeled as follows [33,34]:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{v}}_i=(v_i^{*}-v_i)({\eta }_v+\Delta {\eta }_v)\hfill \\ {\dot{\psi }}_i=({\psi }_i^{*}-{\psi }_i)({\eta }_{\psi }+\Delta {\eta }_{\psi })\hfill \\ {\dot{\theta }}_i=({\theta }_i^{*}-{\theta }_i)({\eta }_{\theta }+\Delta {\eta }_{\theta })\hfill \end{array}\right.\end{array}$$

(2)

where $v_i^{*},{\psi }_i^{*}$ and ${\theta }_i^{*}$ denote the command velocity, course angle and pitch angle, respectively. ${\eta }_v,{\eta }_{\psi }$ and ${\eta }_{\theta }$ are the positive time constants associated with velocity-hold loop, course angle-hold loop and pitch angle-hold loop, respectively. Additionally, $\Delta {\eta }_v,\Delta {\eta }_{\psi },\Delta {\eta }_{\theta }$ are bounded parameter uncertainties arising from the autopilot dynamics.

The virtual leader is modeled as

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\dot{r}}_x=v_{r}sin{\theta }_{r}cos{\psi }_r\hfill \\ {\dot{r}}_y=v_{r}cos{\theta }_{r}sin{\psi }_r\hfill \\ {\dot{r}}_z=v_{r}sin{\theta }_r\hfill \end{array}\right.\end{array}$$

(3)

where $(r_x,r_y,r_z)$ represents the position of virtual leader. $v_r$ is the velocity, ${\psi }_r$ denotes course angle, and ${\theta }_r$ is pitch angle of virtual leader.

2.2. Problem Statement

Based on (1) and (3), the relative position and state for UAVs is described by

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathit{z}}_i-{\mathit{z}}_r=\Delta {\mathit{z}}_{ir}\hfill \\ {\mathit{\varphi }}_i-{\mathit{\varphi }}_r=\Delta {\mathit{\varphi }}_{ir}\hfill \end{array}\right.\end{array}$$

(4)

where ${\mathit{z}}_i={[I_{ix},I_{iy},I_{iz}]}^T$, ${\mathit{z}}_r={[r_x,r_y,r_z]}^T$ denote the position vectors of the ith UAV and virtual leader, respectively. $\Delta z_{ir}\in {\mathbb{R}}^3$ represents their relative position. Similarly, ${\mathit{\varphi }}_i={[v_i,{\psi }_i,{\theta }_i]}^T$, ${\mathit{\varphi }}_r={[v_r,{\psi }_r,{\theta }_r]}^T$ stand for the flight state vectors of the ith UAV and virtual leader, with $\Delta {\mathit{\varphi }}_{ir}={[\Delta v_{ir},\Delta {\psi }_{ir},\Delta {\theta }_{ir}]}^T$ defining the relative state vector.

Define the position tracking error of the ith UAV as

$$\begin{array}{c}\hfill {\mathit{\omega }}_{iz}=\Delta {\mathit{z}}_{ir}-{\mathit{p}}_{ir}={\left[\begin{array}{c}{\omega }_{iz}^x,{\omega }_{iz}^y,{\omega }_{iz}^z\end{array}\right]}^T\end{array}$$

(5)

where ${\mathit{z}}_{ir}={[z_{ir}^x,z_{ir}^y,z_{ir}^z]}^T$ is the expected relative position of the ith UAV with respect to virtual leader. ${\mathit{z}}_{ir}$, which is specified by the task-driven formation configuration.

Remark 1.

For close UAV formation, practical formation tracking can be achieved by keeping the position error ${\mathit{\omega }}_{zi}$ within a predefined range, where parameters ${\mathit{\alpha }}_{zi}$ and ${\mathit{\alpha }}_{zj}$ are chosen to ensure collision avoidance. The relative position between two UAVs is given by $\Delta {\mathit{z}}_{ij}={\mathit{z}}_i-{\mathit{z}}_j=({\mathit{e}}_{zi}+{\mathit{p}}_{ir})-({\mathit{\omega }}_{zj}+{\mathit{p}}_{jr}).$

Then, the error system of each UAV is derived as

$$\begin{array}{c}\hfill {\dot{\mathit{\xi }}}_i={\mathit{F}}_i+{\mathit{G}}_i{\mathit{U}}_i+{\mathit{D}}_i\end{array}$$

(6)

where

$${\mathit{\xi }}_i={\left[\begin{array}{c}{\omega }_{iz}^x,{\omega }_{iz}^y,{\omega }_{iz}^z,\Delta v_{ir},\Delta {\psi }_{ir},\Delta {\theta }_{ir}\end{array}\right]}^T,$$

$${\mathit{F}}_i=\left[\begin{array}{c}{v}_{i}cos{\psi }_{i}sin{\theta }_i-v_{r}cos{\psi }_{r}sin{\theta }_r-{\dot{q}}_{ir}^x\\ v_{i}sin{\psi }_{i}cos{\theta }_i-v_{r}sin{\psi }_{r}cos{\theta }_r-{\dot{q}}_{ir}^y\\ v_{i}sin{\theta }_i-v_{r}sin{\theta }_r-{\dot{q}}_{ir}^z\\ -{\dot{v}}_r-{\eta }_v{v}_{i}x\\ -{\dot{\psi }}_r-{\eta }_{\psi }{\psi }_i\\ -{\dot{\theta }}_r-{\eta }_{\theta }{\theta }_i\end{array}\right],$$

$${\mathit{G}}_i=\left[\begin{array}{cccccc}0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 1& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ {\eta }_v& 0& 0& 0& 0& 0\\ 0& {\eta }_{\psi }& 0& 0& 0& 0\\ 0& 0& {\eta }_{\theta }& 0& 0& 0\end{array}\right],$$

$${\mathit{U}}_i={[v_i^{*},{\psi }_i^{*},{\theta }_i^{*}]}^T,$$

$${\mathit{D}}_i={[l_{ix},l_{iy},l_{iz},l_{iv},l_{i\psi },l_{i\theta }]}^T.$$

2.3. Prescribed Performance Control

To ensure flight safety, the tracking error ${\mathit{\xi }}_i$ is designed to be limited within a specified boundary, i.e.,

$$-{\zeta }_i<{\omega }_{iz}^k<{\zeta }_i,-{\zeta }_i<\Delta {\phi }_{ir}<{\zeta }_i$$

where $k=x,y,z$ and $\phi =v,\psi ,\theta$.

Define the following performance function as

$${\zeta }_i=({\mu }_{i0}-{\mu }_{i\infty })e^{-\rho t}+{\mu }_{i\infty },$$

where $\rho ,{\mu }_{i0},{\mu }_{i\infty }$ are positive constants. ${\zeta }_{i0}={\zeta }_i\left(0\right)$ with $-{\zeta }_i\left(0\right)<{\omega }_{iz}^k\left(0\right)<{\zeta }_i\left(0\right)$ and $-{\zeta }_i\left(0\right)<\Delta {\phi }_{ir}\left(0\right)<{\zeta }_i\left(0\right)$.

As the design of $\Delta {\phi }_{ir}$ follows the same logic as ${\omega }_{iz}^k$, the details are not repeated here. Instead, only the case for ${\omega }_{iz}^k$ is illustrated. Construct the error transformation as follows [35]:

$$\begin{array}{c}\hfill e_i^k=ln\left({\displaystyle \frac{{\zeta }_i+{\omega }_{iz}^k}{{\zeta }_i-{\omega }_{iz}^k}}\right),t\ge 0\end{array}$$

(7)

Take the derivative of (7), one gets

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{e}_i^k}{dt}}& ={\displaystyle \frac{\partial e_i^k}{\partial {\zeta }_i}}{\dot{\zeta }}_i+{\displaystyle \frac{\partial e_i^k}{\partial {\omega }_{iz}^k}}{\dot{\omega }}_{iz}^k\hfill \\ \hfill & =-{\displaystyle \frac{2{\omega }_{iz}^k}{{\zeta }_i^2-{\omega }_{iz}^{k2}}}{\dot{\zeta }}_i+{\displaystyle \frac{2{\zeta }_i}{{\zeta }_i^2-{\omega }_{iz}^{k2}}}{\dot{\omega }}_{iz}^k\hfill \\ \hfill & ={\Gamma }_i^k+v_i^k{\dot{\omega }}_{iz}^k\hfill \end{array}$$

(8)

where ${\Gamma }_i^k={\displaystyle \frac{-2{\omega }_{iz}^k}{{\zeta }_i^2-{\omega }_{iz}^{k2}}}{\dot{\zeta }}_i$, and $v_i^k={\displaystyle \frac{2{\zeta }_i}{{\zeta }_i^2-{\omega }_{iz}^{k2}}}$.

Then, the error transformation dynamics is given by

$$\begin{array}{c}\hfill {\dot{\mathit{e}}}_i={\mathbf{\Gamma }}_i+{\mathit{v}}_i{\dot{\mathit{\xi }}}_{iz}\end{array}$$

(9)

Taking (6) in (9), it yields

$${\dot{\mathit{e}}}_i={\mathit{F}}_{vi}+{\mathit{G}}_{vi}{\mathit{U}}_i+{\mathit{D}}_{vi}$$

(10)

where ${\mathit{F}}_{vi}={\mathit{v}}_i{\mathit{F}}_i,{\mathit{G}}_{vi}={\mathit{v}}_i{\mathit{G}}_i,$ and ${\mathit{D}}_{vi}={\mathit{v}}_i{\mathit{D}}_i+{\mathbf{\Gamma }}_i.$

Objectives: The focus of this work is to devise an optimal formation control protocol for multiple fixed-wing UAVs system (1) such that

(1)

The formation tracking error of the fixed-wing UAVs remains strictly within the prescribed performance boundary.

(2)

All closed-loop signals of the fixed-wing UAVs are SGUUB.

To fulfill these objectives, the controller design relies on the following definition and assumption as prerequisites.

Definition 1 ([20]).

(Semiglobally Uniformly Ultimately Bounded (SGUUB)) The solution $x\left(t\right)\in {\mathbb{R}}^n$ of nonlinear system $\dot{x}\left(t\right)=f(x,t)$ is said to be SGUUB, if the inequality $\parallel x\left(t\right)\parallel \le \rho$ holds for $t>T(\rho ,x(0\left)\right)$, where ρ and $T(\rho ,x(0\left)\right)$ are constants, and Ω is a compact set.

Lemma 1 ([36,37]).

Design a a positive continuous function $L\left(t\right)\in \mathbb{R}$ with the initial value $L\left(0\right)$. If it satisfies

$$\dot{L}\left(t\right)\le -\nu L\left(t\right)+\sigma ,$$

where constants ν and σ are positive, then the following relationship holds:

$$L\le e^{-\nu t}L\left(0\right)+{\displaystyle \frac{\sigma }{\nu }}(1-e^{-\nu t}).$$

3. Optimal Controller Design and Stability Analysis

This section reviews the construction of the adaptive optimal controller and discusses the system stability.

3.1. Optimal Controller Design

To enforce safe distance constraints, we map the constrained tracking error to an unconstrained variable via PPC. This allows later optimal control design to be performed on the unconstrained dynamics.

Define the performance index function as

$$J_i\left({\mathit{e}}_i\right)={\int }_t^{\infty }{V}_i({\mathit{e}}_i,{\mathit{U}}_i)ds$$

(11)

where $V_i({\mathit{e}}_i,{\mathit{U}}_i)={\mathit{e}}_i^T{\mathit{e}}_i+{\mathit{U}}_i^T{\mathit{U}}_i$ denotes the cost function.

The optimal performance index function is prescribed as

$$\begin{array}{cc}\hfill J_i^{*}\left({\mathit{e}}_i\right)& ={\int }_t^{\infty }{V}_i({\mathit{e}}_i,{\mathit{U}}_i^{*})ds\hfill \\ \hfill & =\underset{{\mathit{U}}_i\in \mathrm{\Psi }\left({\mathrm{\Omega }}_U\right)}{min}\left({\int }_t^{\infty }{V}_i({\mathit{e}}_i,{\mathit{U}}_i)ds\right)\hfill \end{array}$$

(12)

where ${\mathrm{\Omega }}_U\in {\mathbb{R}}^3$ is the compact set, and ${\mathit{U}}_i^{*}$ is the optimal position control signal of UAVs.

Based on (12), the HJB equation can be formulated as

$$\begin{array}{cc}\hfill H_1\left({\mathit{e}}_i,{\mathit{U}}_i^{*},{\displaystyle \frac{d{J}_i^{*}}{d{\mathit{e}}_i}}\right)& ={\mathit{e}}_i^T{\mathit{e}}_i+{\mathit{U}}_i^{*T}{\mathit{U}}_i^{*}+{\displaystyle \frac{d{J}_i^{*}}{d{\mathit{e}}_i}}{\dot{\mathit{e}}}_i\hfill \end{array}$$

(13)

Solving the equation $(\partial H_1/\partial {\mathit{U}}_i^{*})=0$ yields the optimal UAV position control ${\mathit{U}}_i^{*}$ as

$${\mathit{U}}_i^{*}=-{\displaystyle \frac{1}{2}}{\displaystyle \frac{d{J}_i^{*}}{d{\mathit{e}}_i}}{\mathit{G}}_{vi}.$$

(14)

Decomposing the term $(d{J}_1^{*}/d{\mathit{e}}_i)$ yields two parts as follows:

$${\displaystyle \frac{d{J}_i^{*}}{d{\mathit{e}}_i}}=2\left(c_i{\mathit{e}}_i+{\displaystyle \frac{{\mathit{e}}_i{\widehat{\mathrm{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\right)+J_i^0\left({\mathit{e}}_i\right)$$

(15)

in which $c_i>0$ is the design parameter and $J_i^0\left({\mathit{e}}_i\right)=-2\left(c_i{\mathit{e}}_i+{\displaystyle \frac{{\mathit{e}}_i{\widehat{\mathrm{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\right)+d{J}_i^{*}/d{\mathit{e}}_i$.

Then, the optimal control ${\mathit{U}}_i^{*}$ is given by

$${{\mathit{U}}_i}^{*}=-{\mathit{G}}_{vi}^{-1}\left(c_i{\mathit{e}}_i+{\displaystyle \frac{{\mathit{e}}_i{\widehat{\mathrm{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\right)-{\displaystyle \frac{{\mathit{G}}_{vi}}{2}}{J}_i^0\left({\mathit{e}}_i\right)$$

(16)

The unknown term $J_1^0\left(e_{{\xi }_2}\right)$ is continuous, so it can be expressed by NN in the following form:

$$J_1^0\left({\mathit{e}}_i\right)={\mathit{\chi }}_{ai}^T{\mathit{\varphi }}_i\left({\mathit{e}}_i\right)+{\mathit{\delta }}_i\left({\mathit{e}}_i\right)$$

(17)

where ${\mathit{\chi }}_{ai}$, ${\mathit{\varphi }}_i$, and ${\mathit{\delta }}_i$ are the ideal weight, the basis function vector and the approximation error, respectively. And the details of NN is given in Appendix A.

Instead of solving the HJB equation analytically, we use an actor--critic RL architecture to approximate the optimal value function and policy online, using only measured signals.

Inserting (15) into (16) and (17) results in

$${\displaystyle \frac{d{J}_i^{*}}{d{\mathit{e}}_i}}=2\left(c_i{\mathit{e}}_i+{\displaystyle \frac{{\mathit{e}}_i{\widehat{\mathrm{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\right)+({\mathit{\chi }}_{ai}^T{\mathit{\varphi }}_i+{\mathit{\delta }}_i)$$

(18)

$${{\mathit{U}}_i}^{*}=-{\mathit{G}}_{vi}^{-1}\left(c_i{\mathit{e}}_i+{\displaystyle \frac{{\mathit{e}}_i{\widehat{\mathrm{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\right)-{\displaystyle \frac{{\mathit{G}}_{vi}}{2}}({\mathit{\chi }}_{ai}^T{\mathit{\varphi }}_i+{\mathit{\delta }}_i)$$

(19)

Since the ideal weight ${\mathit{\chi }}_{ai}$ is an unknown constant matrix, it makes the optimal UAV position control unavailable. For deriving the valid optimized control, the RL is performed by constructing the following both critic and actor.

Given that the ideal weight matrix ${\mathit{\chi }}_{ai}$ is unknown, the optimal control input (19) is not directly realizable. To address this, we employ an RL algorithm that utilizes critic and actor structures to approximate the optimal control.

The critic used for performance evaluation is given by

$${\displaystyle \frac{d{\widehat{J}}_i^{*}}{d{\mathit{e}}_i}}=2\left(c_i{\mathit{e}}_i+{\displaystyle \frac{{\mathit{e}}_i{\widehat{\mathrm{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\right)+{\widehat{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i$$

(20)

where $d{\widehat{J}}_i^{*}/d{\mathit{e}}_i$ is the estimation of $(d{J}_i^{*}/d{\mathit{e}}_i)$. And the critic NN weight ${\widehat{\mathit{\chi }}}_{ci}$ is updated by the following law:

$${\dot{\widehat{\mathit{\chi }}}}_{ci}=-k_{ci}{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}$$

(21)

where $k_{ci}$ is a positive parameter.

The actor used for performing the control action is given by

$${\widehat{{\mathit{U}}_i}}^{*}=-{\mathit{G}}_{vi}^{-1}\left(c_i{\mathit{e}}_i+{\displaystyle \frac{{\mathit{e}}_i{\widehat{\mathrm{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\right)-{\displaystyle \frac{{\mathit{G}}_{vi}}{2}}{\widehat{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i$$

(22)

where the actor NN weight ${\widehat{\mathit{\chi }}}_{ci}$ is updated by the following law:

$${\dot{\widehat{\mathit{\chi }}}}_{ai}=-{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T\left(k_{ai}\left({\widehat{\mathit{\chi }}}_{ai}-{\widehat{\mathit{\chi }}}_{ci}\right)+k_{ci}{\widehat{\mathit{\chi }}}_{ci}\right)$$

(23)

where $k_{ai}$ is a positive parameter.

Parameters $k_{ci}$ and $k_{ai}$ are tuned to fulfill the requirements listed below:

$$k_{ai}>{\displaystyle \frac{1}{2}},k_{ai}>k_{ci}>{\displaystyle \frac{1}{2}}{k}_{ai}.$$

(24)

Remark 2.

Formation flight of fixed-wing UAVs imposes high computational demands. Traditional optimal control methods struggle with nonlinear dynamics and the intractable HJB equation, leading to heavy online computation. By adopting simplified optimal control strategies, such as using elementary positive definite functions or reduced actor--critic structures, the computational burden can be significantly reduced. This enables real-time implementation without compromising formation performance.

Remark 3.

Online learning mechanism of actor–critic networks is displayed as follows: (i) The critic network takes the transformed error ${ϵ}_i$ as input and outputs the approximate value function ${\widehat{V}}_i={\widehat{\chi }}_{ci}^{\top }{\varphi }_i\left({ϵ}_i\right)$, with weight update rule ${\dot{\widehat{\chi }}}_{ci}=-k_{ci}{\varphi }_i{e}_i^{\top }{G}_{vi}$ derived from minimizing the Bellman residual. (ii) The actor network outputs the approximate optimal control ${\widehat{U}}_i=-{\displaystyle \frac{1}{2}}{R}^{-1}{G}_{vi}^{\top }{\widehat{\chi }}_{ai}^{\top }{\varphi }_i$, and its weights are updated as ${\dot{\widehat{\chi }}}_{ai}=-k_{ai}{\varphi }_i({\widehat{\chi }}_{ai}-{\widehat{\chi }}_{ci})$, forcing the actor to synchronize with the critic and thereby approach the optimal policy. (iii) The learning runs online: at each sampling instant, the RBF basis functions are computed from the current error $e_i$, and both networks update their weights simultaneously without offline training or experience replay.

3.2. Stability Analysis

This section serves to confirm the validity of the presented optimal control approach.

Theorem 1.

Consider the fixed-wing UAV system with prescribed performance described by Equation (1). If the assumption in Lemma 1 holds, then under the joint action of the designed controller (22), critic NN (21), and actor NN (23), it can be guaranteed that:

(1) 

The formation error can converge to a neighborhood around zero.

(2) 

All signals of the closed-loop system are semi-globally uniformly ultimately bounded.

Proof. 

To demonstrate the stability of the system, the Lyapunov function is constructed as

$$\begin{array}{cc}\hfill L_i\left(t\right)=& {\displaystyle \frac{1}{2}}{\mathit{e}}_i^T{\mathit{e}}_i+{\displaystyle \frac{1}{2}}Tr\left\{{\tilde{\mathit{\chi }}}_{ci}^T{\tilde{\mathit{\chi }}}_{ci}\right\}+{\displaystyle \frac{1}{2}}Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\tilde{\mathit{\chi }}}_{ai}\right\}+{\displaystyle \frac{1}{2}}Tr\left\{{\tilde{\mathbf{\Psi }}}_i^T{\tilde{\mathbf{\Psi }}}_i\right\}\hfill \end{array}$$

(25)

where ${\tilde{\mathit{\chi }}}_{ci}={\widehat{\mathit{\chi }}}_{ci}-{\mathit{\chi }}_{ci}$ and ${\tilde{\mathit{\chi }}}_{ai}={\widehat{\mathit{\chi }}}_{ai}-{\mathit{\chi }}_{ai}$ denote the critic and actor NN weight errors, respectively.

The time derivative of $L_i$ is calculated as

$$\begin{array}{cc}\hfill {\dot{L}}_i=& {\mathit{e}}_i^T({\mathit{D}}_i+{\mathit{G}}_{vi}{\mathit{U}}_i)-k_{c_i}Tr\left\{{\tilde{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}\right\}\hfill \\ \hfill & -Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T\right\}\left(k_{ai}\left({\widehat{\mathit{\chi }}}_{ai}-{\widehat{\mathit{\chi }}}_{ci}\right)+k_{ci}{\widehat{\mathit{\chi }}}_{ci}\right)+{\displaystyle \frac{1}{2}}Tr\left\{{\tilde{\mathbf{\Psi }}}_i^T{\dot{\widehat{\mathbf{\Psi }}}}_i\right\}\hfill \end{array}$$

(26)

where ${\mathit{D}}_i={\mathit{F}}_{vi}+{\mathit{D}}_{vi}$.

Due to the unknown value of ${\mathit{D}}_i$, we use NN to obtain the following approximate form:

$$\begin{array}{c}\hfill {\mathit{D}}_i={\mathit{W}}_i{\mathit{\varphi }}_i+{\mathit{\delta }}_i\end{array}$$

(27)

According to Young’s inequality, it yields that

$$\begin{array}{c}\hfill {\mathit{e}}_i^T{\mathit{D}}_i\le {\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i{\mathbf{\Psi }}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}+{\displaystyle \frac{a_i^2}{2}}+{\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i}{2}}+{\displaystyle \frac{{\overline{\delta }}_i^2}{2}}\end{array}$$

(28)

where ${\mathbf{\Psi }}_i=\left|\right|{\mathit{W}}_i{\left|\right|}^2$.

According to (22), one has

$$\begin{array}{cc}\hfill {\dot{L}}_i\le & -c_i{\mathit{e}}_i^T{\mathit{e}}_i-{\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i{\widehat{\mathbf{\Psi }}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}-{\displaystyle \frac{G_i}{2}}{\mathit{e}}_i^T{\widehat{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i+{\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i{\mathbf{\Psi }}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}-k_{c_i}Tr\left\{{\tilde{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}\right\}\hfill \\ \hfill & -Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T\right\}\left(k_{ai}\left({\widehat{\mathit{\chi }}}_{ai}-{\widehat{\mathit{\chi }}}_{ci}\right)+k_{ci}{\widehat{\mathit{\chi }}}_{ci}\right)+{\displaystyle \frac{1}{2}}Tr\left\{{\tilde{\mathbf{\Psi }}}_i^T{\dot{\widehat{\mathbf{\Psi }}}}_i\right\}+{\displaystyle \frac{a_i^2}{2}}+{\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i}{2}}+{\displaystyle \frac{{\overline{\delta }}_i^2}{2}}\hfill \end{array}$$

(29)

where $\left|\right|{\mathit{G}}_{vi}{\left|\right|}^2=G_i$.

To handle the unknown bound of NN approximation errors, we introduce an adaptive parameter ${\widehat{\mathbf{\Psi }}}_i$ that estimates the squared ideal weight norm, reducing computational load. Design the adaptive law as

$$\begin{array}{c}\hfill {\dot{\widehat{\mathbf{\Psi }}}}_i=-b_i{\widehat{\mathbf{\Psi }}}_i+{\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i\left|\right|{\mathit{\varphi }}_i{\left|\right|}^2}{2a_i}}\end{array}$$

(30)

where ${\widehat{\mathbf{\Psi }}}_i-{\mathbf{\Psi }}_i={\tilde{\mathbf{\Psi }}}_i$.

Using the relations of ${\tilde{\mathit{\chi }}}_{ai}$ and ${\tilde{\mathit{\chi }}}_{ci}$, one obtains

$$\begin{array}{cc}\hfill -{\displaystyle \frac{G_i}{2}}{\mathit{e}}_i^T{\widehat{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i& \le {\displaystyle \frac{G_i}{4}}{\mathit{e}}_i^T{\mathit{e}}_i+{\displaystyle \frac{G_i}{4}}Tr\left\{{\widehat{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ai}\right\}\hfill \\ \hfill Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ai}\right\}& ={\displaystyle \frac{1}{2}}Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\tilde{\mathit{\chi }}}_{ai}\right\}+{\displaystyle \frac{1}{2}}Tr\left\{{\widehat{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ai}\right\}-{\displaystyle \frac{1}{2}}Tr\left\{{\mathit{\chi }}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\mathit{\chi }}_{ai}\right\}\hfill \\ \hfill Tr\left\{{\tilde{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}\right\}& ={\displaystyle \frac{1}{2}}Tr\left\{{\tilde{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\tilde{\mathit{\chi }}}_{ci}\right\}+{\displaystyle \frac{1}{2}}Tr\left\{{\widehat{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}\right\}-{\displaystyle \frac{1}{2}}Tr\left\{{\mathit{\chi }}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\mathit{\chi }}_{ci}\right\}\hfill \end{array}$$

In view of condition (24) and Young’s inequality, one gets

$$\begin{array}{cc}\hfill & (k_{ai}-k_{ci})Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}\right\}\hfill \\ \hfill & \le {\displaystyle \frac{k_{ai}-k_{ci}}{2}}Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\tilde{\mathit{\chi }}}_{ai}\right\}+{\displaystyle \frac{k_{ai}-k_{ci}}{2}}Tr\left\{{\widehat{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}\right\}\hfill \end{array}$$

(31)

Based on the above inequalities, it yields

$$\begin{array}{cc}\hfill {\dot{L}}_i\le & -(2c_i-{\displaystyle \frac{G_i}{2}}-1){\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i}{2}}-b_i{\displaystyle \frac{{\tilde{\mathbf{\Psi }}}_i{\tilde{\mathbf{\Psi }}}_i}{2}}+{\displaystyle \frac{a_i^2}{2}}+{\displaystyle \frac{{\overline{\delta }}_i^2}{2}}\hfill \\ \hfill & -{\displaystyle \frac{k_{c_i}}{2}}Tr\left\{{\tilde{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\tilde{\mathit{\chi }}}_{ci}\right\}-{\displaystyle \frac{k_{c_i}}{2}}Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\tilde{\mathit{\chi }}}_{ai}\right\}\hfill \\ \hfill & +{\displaystyle \frac{k_{c_i}}{2}}Tr\left\{{\mathit{\chi }}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\mathit{\chi }}_{ci}\right\}+{\displaystyle \frac{k_{c_i}}{2}}Tr\left\{{\mathit{\chi }}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\mathit{\chi }}_{ai}\right\}\hfill \\ \hfill & -({\displaystyle \frac{k_{a_i}}{2}}-{\displaystyle \frac{G_i}{4}})Tr\left\{{\widehat{\mathit{\chi }}}_{ai}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ai}\right\}\hfill \\ \hfill & -(k_{ci}-{\displaystyle \frac{k_{a_i}}{2}})Tr\left\{{\widehat{\mathit{\chi }}}_{ci}^T{\mathit{\varphi }}_i{\mathit{\varphi }}_i^T{\widehat{\mathit{\chi }}}_{ci}\right\}\hfill \end{array}$$

(32)

Further, the inequality (32) can be written as

$$\begin{array}{cc}\hfill {\dot{L}}_i\le & -(2c_i-{\displaystyle \frac{G_i}{2}}-1){\displaystyle \frac{{\mathit{e}}_i^T{\mathit{e}}_i}{2}}-b_i{\displaystyle \frac{{\tilde{\mathbf{\Psi }}}_i{\tilde{\mathbf{\Psi }}}_i}{2}}+C_i\hfill \\ \hfill & -{\displaystyle \frac{k_{c_i}}{2}}{\lambda }_{{\varphi }_i}^{min}Tr\left\{{\tilde{\mathit{\chi }}}_{ci}^T{\tilde{\mathit{\chi }}}_{ci}\right\}-{\displaystyle \frac{k_{c_i}}{2}}{\lambda }_{{\varphi }_1}^{min}Tr\left\{{\tilde{\mathit{\chi }}}_{ai}^T{\tilde{\mathit{\chi }}}_{ai}\right\}\hfill \end{array}$$

(33)

where ${\lambda }_{{\varphi }_1}^{min}$ is the minimal eigenvalue of ${\mathit{\varphi }}_i{\mathit{\varphi }}_i^T$, $C_i\left(t\right)={\displaystyle \frac{k_{c_i}}{2}}\left|\right|{\mathit{\chi }}_{ci}^T{\mathit{\varphi }}_i{\left|\right|}^2+{\displaystyle \frac{k_{c_i}}{2}}\left|\right|{\mathit{\chi }}_{ai}^T{\mathit{\varphi }}_i{\left|\right|}^2+{\displaystyle \frac{a_i^2}{2}}+{\displaystyle \frac{{\overline{\delta }}_i^2}{2}}$.

Terms of $C_i\left(t\right)$ are bounded, which implies the existence of a constant ${\sigma }_i$ satisfying $|C_i\left(t\right)|\le {\sigma }_i$.

Defining ${\nu }_i=min\{2c_i-{\displaystyle \frac{G_i}{2}}-1,b_i,k_{c_i}{\lambda }_{{\varphi }_i}^{min}\}$, we have

$${\dot{L}}_i\left(t\right)\le -{\nu }_i{L}_i\left(t\right)+{\sigma }_i.$$

(34)

Applying Lemma 1 to (34) yields

$$L\left(t\right)\le e^{-\nu t}L\left(0\right)+{\displaystyle \frac{\sigma }{\nu }}\left(1-e^{-\nu t}\right).$$

(35)

Inequality (35) implies that all error signals are semi-globally uniformly ultimately bounded, and the tracking errors can achieve ideal accuracy through adjustment parameters. The proof is completed.□

To clarify the novelty of our work, we have proposed the

Table 1. Summary of existing control approaches.

Method Category	Main Advantage	Main Limitation	Difference of This Work
Robust/adaptive UAV control	Handles uncertainties and disturbances	Usually does not optimize long-term control cost	Introduces actor–critic RL to approximate optimal control
Conventional actor–critic UAV control	Learns near-optimal policy under limited model knowledge	Safety constraints are not explicitly guaranteed	Uses PPC to enforce formation error bounds
PPC-based UAV control	Guarantees transient/steady-state error constraints	Often lacks optimality or energy-efficiency design	Combines PPC with HJB-based actor-critic learning
Existing RL-PPC control	Combines learning and performance constraints	Often for single UAV, simplified dynamics, or nonlinear systems	Focuses on UAV formation with coupled position–attitude dynamics
Proposed method	Safe and near-optimal formation control	Current validation is simulation-based	Ensures SGUUB stability and defined boundary satisfaction

to clarify the novelty of our work. In particular, the added table summarizes the main differences among robust/adaptive UAV control, conventional actor–critic UAV control, PPC-based UAV control, existing RL-PPC control, and the proposed method. It compares these methods in terms of their main advantages, limitations, and relevance to the present work, thereby making the novelty of our PPC-based actor–critic framework more explicit. These revisions explicitly highlight the methodological advances of the proposed framework beyond existing RL and PPC control methods.

4. Simulation Results

In this section, a simulation experiment is designed and implemented to verify the effectiveness of the proposed optimal method in multi-UAV formation control.

Simulation Environment: The hardware-in-the-loopexperimental platform, supported by Beijing Links Co., Ltd., is established to validate the proposed learning-based safety control strategy for UAV formation tracking. The platform comprises three core functional units: a development computer running MATLAB R2026b, Tacview v1.9.5, and RTSimPlus 2024 for 3D visualization and real-time simulation management; a flight controller that executes the proposed control strategy and transmits control commands to the real-time simulator via the serial port; and a real-time simulator that models the multi-UAV dynamics, communicates with the development computer over Ethernet, and exchanges state information (including position, velocity, and attitude) with the flight controller through the same serial port. Operating at a time step of 0.01 s, the testbed accounts for the actual computational capability of the flight controller and introduces realistic perturbations into the closed-loop UAV system, thereby providing a practical environment for experimental validation.

Figure 1 shows the communication topology used in the simulation, which is generated according to the predefined information-exchange structure among one virtual leader and four follower UAVs. The nodes represent UAV agents, and the arrows indicate the direction of information flow.

4.1. Parameter Settings

In UAV formation tasks, the initial positions of the virtual leader and four followers are set to create a dispersed spatial configuration, which tests the convergence capability of the formation control scheme. Specifically, the virtual leader starts at ${\mathit{z}}_r\left(0\right)={[30,0,500]}^{\mathrm{T}}$ m, while the followers are initially placed at ${\mathit{z}}_1\left(0\right)={[5,5,590]}^{\mathrm{T}}$ m, ${\mathit{z}}_2\left(0\right)={[-35,-30,600]}^{\mathrm{T}}$ m, ${\mathit{z}}_3\left(0\right)={[-23,25,594]}^{\mathrm{T}}$ m, and ${\mathit{z}}_4\left(0\right)={[-54,-25,600]}^{\mathrm{T}}$ m. All UAVs share the same initial velocity of 8 m/s, and initial course and pitch angles of $0^{\circ }$ and $0^{\circ }$, respectively. The autopilot constants are chosen as ${\eta }_v=1$, ${\eta }_{\psi }=10$, and ${\eta }_{\theta }=10$. The prescribed performance function is designed as ${\zeta }_i\left(t\right)=(4-0.2)e^{-0.5t}+0.2$, $i=1,\dots ,4$, where the initial bound 4 ensures a large enough safe region, the steady-state bound $0.2$ guarantees ultimate tracking precision, and the exponential rate $0.5$ dictates the minimum convergence speed. The desired relative positions of the four followers with respect to the virtual leader are selected as ${\mathit{p}}_{1r}={[-20,0,0]}^{\mathrm{T}}$ m, ${\mathit{p}}_{2r}={[-48,-28,0]}^{\mathrm{T}}$ m, ${\mathit{p}}_{3r}={[-50,32,0]}^{\mathrm{T}}$ m, and ${\mathit{p}}_{4r}={[-80,-32,0]}^{\mathrm{T}}$ m.

Both the critic and actor neural networks are designed with seven nodes, a moderate size that balances approximation capability and computational load. The initial weight vectors for the critic networks are set as ${\widehat{\mathit{\varphi }}}_1\left(0\right)={[0.8,0.8,0.8,0.7,0.7,0.7,0.7]}^{\mathrm{T}}$, ${\widehat{\mathit{\varphi }}}_2\left(0\right)={[0.7,0.7,0.7,0.7,0.7,0.75,0.75]}^{\mathrm{T}}$, ${\widehat{\mathit{\varphi }}}_3\left(0\right)={\left[0.6\right]}_{7\times 1}$, and ${\widehat{\mathit{\varphi }}}_4\left(0\right)={\left[0.55\right]}_{7\times 1}$. The initial actor weight vectors are chosen as ${\widehat{\mathbf{\Psi }}}_1\left(0\right)={\left[0.45\right]}_{7\times 1}$, ${\widehat{\mathbf{\Psi }}}_2\left(0\right)={\left[0.6\right]}_{7\times 1}$, ${\widehat{\mathbf{\Psi }}}_3\left(0\right)={\left[0.7\right]}_{7\times 1}$, and ${\widehat{\mathbf{\Psi }}}_4\left(0\right)={\left[0.85\right]}_{7\times 1}$. These values are intentionally diversified across followers to examine the robustness of the learning scheme against different initialization conditions.

4.2. Simulation Verification

The effectiveness of the proposed optimal formation control strategy for fixed-wing UAVs is clearly demonstrated in Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7. Specifically, Figure 2 is produced using Tacview based on the simulated three-dimensional UAV trajectories. It illustrates the spatial formation evolution of the virtual leader and four followers. Figure 2a shows the formation configuration during the maneuver, while Figure 2b shows the formation-keeping scenario after convergence. The represented quantities are UAV spatial positions, calibrated in meters according to the simulation settings. Figure 3 illustrates the formation tracking errors along the x, y, and z axes, where each error remains within the prescribed performance bounds. Furthermore, Figure 3d presents the error performance on z axes without considering PPC, where significantly poorer tracking results are observed. Further insights into the control performance are provided in Figure 4, which plots the errors in velocity, course angle, and pitch angle. Additionally, Figure 4d depicts the error outcome on pitch angle in the absence of PPC, revealing markedly inferior tracking performance.

Figure 5 shows the evolution of actor and critic weights, respectively. All weights converge to small steady-state values, which demonstrates stable learning and successful approximation of the optimal control policy. Finally, Figure 6 shows the evolution of the cost functions $V_1$, $V_2$, and $V_3$, offering a quantitative assessment of the optimization objective. The decreasing trends of these functions indicate that the proposed control scheme successfully minimizes the predefined infinite-horizon costs while respecting safety and performance constraints. Figure 7 presents the evolution of the adaptive laws, indicating that the system signals converge to near-zero steady states.

4.3. Sensitivity Analysis of Learning Parameters

To analyze the influence of the learning gains, a sensitivity study is performed for the actor neural network weights.

Table 2. Sensitivity analysis of actor network learning parameters.

Case	${\mathit{k}}_{\mathit{ci}}$	${\mathit{k}}_{\mathit{ai}}$	Convergence Time	Learning Oscillation	Stability
Actor network	0.8	1	11.7	Low	√
	0.4	1	17.0	Low	√
	1.2	1	8.8	Medium	√
	0.8	1.2	13.0	Medium	√

summarizes the sensitivity analysis results of the actor-network learning parameters. Compared with the nominal case, reducing $k_{ci}$ or $k_{ai}$ leads to smoother learning trajectories but slower convergence, whereas increasing $k_{ci}$ or $k_{ai}$ improves the convergence speed at the cost of increased learning oscillations. It is worth noting that all tested cases remain stable, which verifies the robustness of the proposed learning mechanism with respect to moderate parameter variations. Therefore, the learning gains should be selected by considering both convergence speed and closed-loop smoothness. In this work, the nominal parameters $k_{ci}=0.8$ and $k_{ai}=1.0$ are adopted because they provide a satisfactory balance between fast convergence and low oscillation.

To further investigate the influence of learning gains on the online learning process, a sensitivity analysis is conducted for the actor neural network weights. The nominal learning gains are selected as $k_{c0}=0.8$ and $k_{a0}=1.0$. Based on this nominal setting, different combinations of $k_{ci}$ and $k_{ai}$ are tested while keeping the other simulation parameters unchanged.

Figure 8 illustrates the actor neural network weight trajectories under different learning gain settings. It can be observed that all actor weights remain bounded and eventually converge to a small neighborhood around zero, indicating that the learning process is stable for all tested parameter combinations. Under the nominal setting $k_{ci}=0.8$ and $k_{ai}=1.0$, the actor weights exhibit smooth convergence with relatively low oscillation. When $k_{ci}$ is reduced to $0.4$, the convergence becomes slower, and the weights require a longer time to approach the steady-state region. In contrast, increasing $k_{ci}$ to $1.2$ or increasing $k_{ai}$ to $1.2$ accelerates the convergence of the actor weights, but mild fluctuations and small overshoots appear during the transient stage. These results indicate that larger learning gains can improve the learning speed, while excessively large gains may introduce oscillatory behavior in the weight update process.

4.4. Ablation Study on PPC and RL Components

To further verify the individual contributions of the prescribed performance control and reinforcement learning components, an ablation study is conducted in this subsection. In addition to the proposed complete PPC-RL configuration, two alternative configurations are considered. In the first case, the PPC mechanism is removed while the RL-based optimal learning structure is retained, denoted as the No-PPC configuration. In the second case, the RL component is removed while the PPC mechanism is retained, denoted as the No-RL configuration.

All three configurations are tested under the same initial conditions, desired formation geometry, and simulation parameters, allowing a clear comparison of their tracking performance. The mean square error (MSE), root mean square error (RMSE), and mean absolute error (MAE) are adopted as quantitative performance indices, which are defined as

$$\begin{array}{cc}\hfill \mathrm{MSE}& ={\displaystyle \frac{1}{n}}{\displaystyle \sum _{k=1}^n}{\mathit{e}}_i^2,\hfill \\ \hfill \mathrm{RMSE}& =\sqrt{{\displaystyle \frac{1}{n}}\sum _{k=1}^n{\mathit{e}}_i^2},\hfill \\ \hfill \mathrm{MAE}& ={\displaystyle \frac{1}{n}}{\displaystyle \sum _{k=1}^n}\left|{\mathit{e}}_i\right|.\hfill \end{array}$$

The overall comparison results are listed in

Table 3. Quantitative comparison of the proposed controller and ablation cases.

Data Type	MSE	RMSE	MAE
Proposed PPC-RL	0.000999	0.031613	0.009885
Without PPC	0.030153	0.173645	0.129873
Without RL	0.002762	0.052556	0.028103

.

Table 3 shows that the proposed PPC-RL controller achieves the smallest MSE, RMSE, and MAE. Removing PPC significantly degrades tracking performance (RMSE from 0.031613 to 0.173645, MAE from 0.009885 to 0.129873), confirming that PPC is essential for enforcing prescribed error bounds and transient performance. Removing RL also reduces accuracy (RMSE to 0.052556, MAE to 0.028103), verifying that RL enhances optimality. Thus, PPC and RL play complementary roles: PPC guarantees prescribed performance and safety, while RL improves optimal control, and their integration yields the best overall performance.

Figure 9 compares the tracking errors of the PPC-RL, without PPC, and without RL. The proposed method achieves the fastest convergence and smallest steady-state error. The control scheme without PPC shows larger oscillations and poorer transient performance, while the control scheme without RL performs better than No-PPC but still worse than PPC-RL. These results confirm the necessity of integrating both PPC and RL in the framework.

Discussion: Although the proposed framework shows promising simulation results, several practical issues should be further considered before real fixed-wing UAV deployment. First, the current design assumes reliable communication under a fixed topology, while real UAV swarms may experience delays, packet loss, bandwidth limits, and link interruptions. Future work should therefore consider event-triggered communication, delay compensation, and switching topologies. Second, the control commands, including velocity, course angle, and pitch angle, are constrained by actuator saturation, rate limits, and flight envelope restrictions in real UAVs. Thus, input constraints and safety filters should be incorporated. Third, although the simplified actor–critic structure reduces computational burden, its real-time performance on embedded processors should be further tested, especially for large-scale formations. Finally, sensor noise, GPS drift, and estimation delays may affect error transformation and online learning, so robust filtering, sensor fusion, hardware-in-the-loop tests, and flight experiments will be considered in future work.

5. Conclusions

This study has presented formation optimal control scheme for fixed-wing UAV systems by proposing a safe optimal control framework that bridges the gap between performance optimization and safety assurance. By embedding PPC within a simplified actor–critic RL architecture, the proposed method achieved near-optimal control for coupled position–attitude dynamics without reliance on accurate models, thereby significantly enhancing energy efficiency. Simultaneously, the PPC mechanism ensures that all inter-UAV safety distances were strictly preserved. Furthermore, the Lyapunov theory analysis confirmed that all closed-loop signals remain SGUUB and that formation errors consistently stay within predefined boundaries.

Despite promising results, several limitations remain. First, the framework assumes a fixed communication topology, limiting its use in large-scale or dynamic swarms. Second, actor–critic learning is sensitive to initial weights and lacks provable global convergence. Third, structural vibration and material-dependent damping are not explicitly considered. Fourth, only simulation validation has been conducted. Implementation challenges include careful tuning of prescribed performance parameters and the computational load of online neural-network learning on onboard processors. Future work will focus on: (i) scalability to larger formations with dynamic topologies; (ii) resilience against cyberattacks, communication outages, and vibration-induced disturbances; (iii) extension to vibration-aware fixed-wing UAV models; and (iv) experimental flight tests for real-world validation.