Hierarchical Adaptive PID Tuning for Agile Flight: A Safety-Constrained Reinforcement Learning Approach

Abstract

Multirotor unmanned aerial vehicles (UAVs) suffer from significant control performance degradation during aggressive maneuvers, primarily due to aerodynamic nonlinearities and coupling effects. Conventional fixed-gain PID controllers struggle to simultaneously satisfy performance and robustness requirements across the wide flight envelope. To address this challenge, this paper presents a novel hierarchical safety-constrained reinforcement learning (RL) framework for adaptive PID tuning: the inner loop employs fixed gains, the outer loop leverages proximal policy optimization (PPO) for online adaptive gain scheduling, and linear matrix inequality (LMI) constraints delineate robust parameter boundaries for the adaptive exploration. Importantly, the LMI feasibility strictly guarantees theoretical stability for the fixed inner-loop parameters at the linearization vertices within a linear parameter-varying (LPV) framework. Concurrently, the online outer-loop RL stage is protected by safety boundaries and a Lagrangian penalty mechanism acting as an effective engineering safeguard rather than a rigorous global stability proof. Comprehensive high-fidelity simulation benchmarks demonstrate that, compared with a baseline fixed-gain PID controller, the proposed framework reduces overshoot by 18.5% in high-speed step responses and improves the overall mean RMSE by 15.0% across 100 randomized mixed-trajectory trials (with improvements of up to 40.9% in highly dynamic scenarios), yielding consistent gains in trajectory tracking accuracy and disturbance rejection despite uncertain model variations. By seamlessly blending control-theoretic hard constraints with RL-based soft-parameter tuning, the proposed architecture offers a safe and highly adaptive solution for large-envelope flight control, demonstrating strong engineering relevance.

1. Introduction

Precise control is fundamental to the stable flight and mission execution of UAVs, representing a key enabling technology across aerospace and robotics domains. Quadrotors, as representative underactuated systems, require stable and robust flight control as a prerequisite for autonomous operation. The proportional–integral–derivative (PID) controller remains the most widely adopted control scheme for quadrotor UAVs owing to its structural simplicity, ease of implementation, and well-established stability properties [1,2,3].

The control effectiveness of PID controllers on quadrotor platforms is critically dependent on the accuracy of parameter tuning. Quadrotors are strongly nonlinear, tightly coupled, six-degree-of-freedom underactuated systems: nonlinearities remain mild under low-dynamic conditions such as hovering, but become substantially more pronounced during high-speed aggressive flight as aerodynamic drag (proportional to the square of velocity) and inertial-coupling effects intensify. The resulting dynamics exhibit characteristic linear parameter-varying (LPV) behavior [4,5], rendering fixed-gain PID controllers incapable of simultaneously meeting performance and robustness requirements across the full flight envelope.

Conventional tuning approaches suffer from three fundamental limitations [6,7]. First, trial-and-error methods are time-consuming and rely heavily on expert experience; although metaheuristic search algorithms such as differential evolution [8] can partially automate the process, coupled multi-loop tuning remains intractable. Second, rule-based methods (e.g., adaptive control [9] and active disturbance-rejection control [10]) offer limited coverage and struggle to accommodate the diverse nonlinear operating conditions spanning the full envelope. Third, model-based methods (e.g., backstepping/sliding mode [11] and model predictive control [12]) require accurate global models, yet the strong nonlinearities of real systems make high-fidelity model acquisition prohibitively expensive. Moreover, the tuning process itself entails flight safety risks, as inappropriate parameters may lead to vehicle loss.

Beyond multirotor UAVs, the demand for robust responses to dynamic environments and unmodeled perturbations represents a universal challenge across modern automation domains. For instance, in autonomous driving and active suspension systems, dynamic motion-planning frameworks must adapt rapidly to parameter variations to ensure safety [13]. Similarly, maintaining system stability against external disruptions is equally critical in networked systems facing disturbances and actuator attacks [14]. Inspired by these cross-domain demands for safety and adaptive capability, reinforcement learning (RL)-based adaptive tuning has emerged as an active research area in recent years [15,16].

Existing RL-PID studies can be categorized into three classes. First, direct RL control (RL directly outputs control commands)—Koch et al. [17] and Wang et al. [18] applied PPO to quadrotor attitude/velocity control, but generally lacked explicit safety constraints. Second, RL gain prediction (RL outputs and PID gains)—Sönmez et al. [19] employed action clipping to restrict the exploration range and Wang et al. [18] and Alrubyli et al. [20] applied PPO and Q-learning, respectively, for online PID tuning. Third, RL-PID hybrid (PID structure preserved and RL optimizes gain scheduling)—Dogru et al. [21] applied RL to autonomous PID tuning in chemical processes; Ping et al. [22] extended this approach to fixed-wing aircraft; Xue et al. [23] improved the PPO clipping objective to enhance sample efficiency; and Zhai et al. [24] and Wang et al. [25] explored deep RL for automatic PID parameter tuning. While these studies have advanced performance optimization, they share three common shortcomings: the absence of theoretically grounded safe-parameter-domain constraints [26,27], relying instead on reward penalties or simple clipping; a lack of targeted designs addressing the LPV-parameter-variation characteristics inherent to large-envelope flight; and few studies have systematically exploited the dynamic symmetry of the platform for cross-channel policy transfer.

Building upon the ongoing analysis, this paper develops an RL-based PID-tuning methodology focused on outer-loop nonlinearity compensation. The core design rationale stems from the frequency-domain separation characteristics of quadrotor dynamics: the inner loops (attitude/angular rate) are dominated by rigid-body dynamics with relatively mild nonlinearities and high bandwidth, making them suitable for robust fixed-parameter control, whereas the outer loops (position/velocity) experience significant nonlinear effects from aerodynamic drag (proportional to velocity squared) and inertial coupling during aggressive maneuvers, constituting the region where adaptive parameter scheduling yields the greatest benefit.

A systematic comparison with representative related approaches is presented in

Table 1. Systematic comparison with representative related methods.

Method	Safety Mechanism	Control Architecture	Policy Transfer	Distinction of This Work
Wang et al. [18]	No explicit constraints	RL outputs attitude commands	None	LMI safety domain for outer loop
Sönmez et al. [19]	Action clipping	RL predicts PD gains	None	LMI polytopic constraints; integral terms & transfer
Xue et al. [23]	Penalty reward	PPO outputs control commands	None	Fixed inner-loop PID; LMI safety domain
Saeed et al. [29]	LMI-LPV robust control	Fixed LPV gain scheduling	None	RL online adaptation within LMI domain
This work	LMI + Lagrangian PPO	Inner fixed + outer RL	Long.→Lat. transfer	—

. The key innovation lies in the joint design of three complementary mechanisms: (1) LPV polytopic modeling combined with common Lyapunov LMI constraints [28], which construct a parameterized safety domain across the entire operating envelope, compressing the RL exploration space; (2) dual-timescale Lagrangian-constrained PPO, where the policy network (fast timescale, ${\alpha }_{\theta }={10}^{-4}$) and Lagrange multiplier (slow timescale, ${\alpha }_{\lambda }={10}^{-5}$) are updated in a decoupled manner; and (3) symmetry-based policy transfer, leveraging $I_{xx}\approx I_{yy}$ to directly map the longitudinal policy to the lateral channel. The joint integration of these three mechanisms is absent in existing work.

The main contributions are summarized as follows:

• Hierarchical adaptive control architecture: An “outer-loop adaptive, inner-loop fixed” strategy is adopted, reducing the full-system adaptive tuning problem to low-dimensional parameter optimization of the pitch-channel position loop (P) and velocity loop (PID), thereby lowering the learning complexity while precisely compensating for the dominant dynamic nonlinearities.
• Symmetry-based policy transfer: Exploiting the strong symmetry between the pitch and roll channels inherent to the X-configuration quadrotor, the longitudinal policy is directly mapped to the lateral channel, reducing training costs and enhancing engineering practicality.
• Safety constraints with online fine-tuning: An LMI-derived safe-parameter domain is combined with an online fine-tuning mechanism in a high-fidelity simulation environment, providing bounded parameter guarantees for RL exploration within the LPV linearization framework while enhancing policy adaptability to real-world environments.

Paper organization. The remainder of this paper is structured as follows. Section 2 details the problem formulation, LPV modeling, and the derivation of LMI safety bounds, alongside the RL framework formulation. Section 3 presents the progressive curriculum training configurations and high-fidelity experimental benchmark results. Section 4 offers an in-depth analysis of the adaptive mechanisms, extended baseline comparisons, hyperparameter sensitivity, and corresponding system limitations. Finally, Section 5 concludes the paper with broader engineering perspectives.

2. Hierarchical Adaptive PID Tuning Framework

To address the dual challenges of aerodynamic nonlinearity and model uncertainty across the wide flight envelope, a hierarchical “inner-loop fixed, outer-loop adaptive” control architecture is adopted: an LPV model captures the parameter-varying dynamics, LMI constraints delineate the safe exploration domain, and a safety-constrained PPO algorithm tunes the outer-loop gains online.

2.1. LMI-Based Safe-Parameter-Domain Construction

Taking the longitudinal channel of an X-configuration quadrotor as the modeling target, a dynamics model incorporating aerodynamic nonlinearity ($C_d·u·\left|u\right|$) and geometric nonlinearity ($sin\theta$ terms) is constructed (Equation (1)). Under high-speed flight and aggressive maneuvering, these nonlinearities cause substantial variation in open-loop gain, rendering fixed-gain PID controllers unable to simultaneously satisfy performance and robustness requirements.

$$\left\{\begin{array}{c}\dot{x}=ucos\theta \hfill \\ m\dot{u}=-mgsin\theta -C_{d}u\left|u\right|+F_{T}cos\theta \hfill \\ \dot{\theta }=q\hfill \\ I_{yy}\dot{q}={\tau }_{\mathrm{pitch}}\hfill \end{array}\right.$$

(1)

where x denotes the forward position, u the forward velocity, $\theta$ the pitch angle, q the pitch rate, m the mass, g the gravitational acceleration, $C_d$ the fuselage-drag coefficient, $F_T$ the total thrust, $I_{yy}$ the pitch-axis moment of inertia, and ${\tau }_{\mathrm{pitch}}$ the pitch-control torque.

Using the forward velocity u as the scheduling parameter, the nonlinear system is transformed into an LPV state-space representation (Equation (2)) [4,5]. Jacobian linearization is performed at four characteristic velocity operating points $\{0,5,10,15\}$ m/s, yielding vertex subsystems $({\mathbf{A}}_k,{\mathbf{B}}_k)$, which are assembled into a standard LMI feasibility framework through polytopic convex combination (Equations (3) and (4)) [28]. The selection of these four vertices stems from a practical trade-off between interpolation accuracy and LMI feasibility region size: a 5 m/s interval provides sufficient grid density to accurately capture the mild nonlinearities in the drag coefficient and pitch dynamics up to the aggressive envelope limit of 15 m/s, ensuring that the intermediate interpolated dynamics remain closely bounded by the convex hull. Conversely, introducing an excessive number of vertices would overly shrink the intersection of the LMI constraints, resulting in a highly conservative safe-parameter domain that stifles outer-loop RL exploration capability.

$$\dot{\mathbf{x}}=\mathbf{A}\left(u\left(t\right)\right)\mathbf{x}+\mathbf{B}\left(u\left(t\right)\right)v,v={\tau }_{\mathrm{pitch}}$$

(2)

$$\dot{\mathbf{x}}={\mathbf{A}}_k\mathbf{x}+{\mathbf{B}}_{k}v,k=1,\dots ,4$$

(3)

$$\mathbf{A}\left(u\left(t\right)\right)={\displaystyle \sum _{k=1}^4}{\mu }_k{\mathbf{A}}_k,\mathbf{B}\left(u\left(t\right)\right)={\displaystyle \sum _{k=1}^4}{\mu }_k{\mathbf{B}}_k,{\mu }_k\ge 0,\sum {\mu }_k=1$$

(4)

A four-level cascaded control structure is adopted, with the control law defined in Equations (5)–(8):

$$u_{\mathrm{ref}}=K_{p,x}(x_{\mathrm{ref}}-x)$$

(5)

$${\theta }_{\mathrm{ref}}=K_{p,u}{e}_u+K_{i,u}\int e_{u}dt+K_{d,u}{\dot{e}}_u$$

(6)

$$q_{\mathrm{ref}}=K_{p,\theta }({\theta }_{\mathrm{ref}}-\theta )$$

(7)

$${\tau }_{\mathrm{pitch}}=K_{p,q}{e}_q+K_{i,q}\int e_{q}dt+K_{d,q}{\dot{e}}_q$$

(8)

Introducing the integral error as augmented states ${\mathbf{X}}_a={[e_x,u,\theta ,q,{\xi }_u,{\xi }_q]}^T$, the cascaded PID is equivalently reconstructed as structured state feedback, yielding the augmented closed-loop system:

$${\dot{\mathbf{X}}}_a=\left({\mathbf{A}}_{a,k}-{\mathbf{B}}_{a,k}{\mathbf{K}}_{\mathrm{total}}\right){\mathbf{X}}_a+{\mathbf{E}}_{a,k}r,k=1,\dots ,4$$

(9)

where ${\mathbf{A}}_{a,k}$, ${\mathbf{B}}_{a,k}$, and ${\mathbf{E}}_{a,k}$ are the augmented system matrix, input matrix, and reference matrix at vertex k, respectively; ${\mathbf{K}}_{\mathrm{total}}$ encodes all PID gains; $r\in \mathbb{R}$ is the external reference input (velocity reference $u_{\mathrm{ref}}$, in m/s), and ${\mathbf{E}}_{a,k}$ is the corresponding reference signal distribution vector.

The parameter determination proceeds in two steps:

Step 1—Inner-loop parameter fixation. The LMI constraint set is jointly solved across all four vertices, encompassing quadratic stability, D-stability region ($\alpha$-stability and conic sector) constraints, and $H_{\infty }$ performance bounds (complete derivation in Appendix B, Equations (A1)–(A5)), yielding the fixed inner-loop parameter set ${\Theta }_{\mathrm{in}}^{*}=\{K_{p,\theta }^{*},K_{p,q}^{*},K_{i,q}^{*},K_{d,q}^{*}\}$, which remains constant throughout subsequent RL training.

Step 2—Outer-loop safety boundary computation. Substituting ${\Theta }_{\mathrm{in}}^{*}$ into the augmented system, the LMI feasible region for the outer-loop parameters $\{K_{p,x},K_{p,u},K_{i,u},K_{d,u}\}$ is solved as the intersection across all vertices, yielding the safe convex hull boundary $[{\mathbf{K}}_{min},{\mathbf{K}}_{max}]$. This boundary serves as the physical constraint on the RL action space, with the midpoint of each parameter interval used to initialize the RL agent.

The safe domain construction simultaneously compresses the RL search space and provides bounded parameter-domain constraints within the LPV linearization framework, thereby improving sampling efficiency. It should be noted that the LMI feasibility guarantee applies strictly to the closed-loop stability of the fixed inner-loop parameters at the four linearization vertices. During outer-loop RL online tuning, closed-loop stability is jointly maintained by safety boundary constraints (hard-boundary backstop) and the Lagrangian penalty mechanism (soft-boundary guidance), constituting an engineering safeguard rather than a rigorous global mathematical proof.

2.2. Safety-Constrained Dual-Timescale Reinforcement Learning

The safety-constrained dual-timescale RL framework is established upon LMI-fixed inner-loop parameters providing a stable foundation, PPO performing online adaptive adjustment of outer-loop parameters, with the framework incorporating both longitudinal training and lateral transfer mechanisms (see Figure 1).

As depicted in Figure 1, the entire closed-loop control and training process is formulated into five primary processing steps across the Agent and Environment. Step 1: Policy Execution (Agent). Based on the current state $s_t$, the Actor network computes the raw action ${\mathit{a}}_t$ to adjust the outer-loop parameters. Step 2: Action Processing (Env). The action ${\mathit{a}}_t$ first undergoes direct affine mapping based on the prescribed safety constraints to produce an intermediate gain ${\mathbf{K}}_{\mathrm{target}}$. To prevent aggressive gain fluctuation, an EMA smoothing filter is applied to yield the final deployed gain ${\mathbf{K}}_t$. Step 3: Actuation and Dynamics (Env). The smoothed gain ${\mathbf{K}}_t$ is transmitted to the cascaded PID controller, functioning alongside the reference trajectory to drive the quadcopter mixer and physical components. Step 4: Reward Evaluation (Env). The system states $s_t$ resulting from the performed action are assessed to calculate the immediate composite return $R_t$ and safety cost $C_t$. Step 5: Policy and Multiplier Update (Agent). The calculated rewards and costs flow back to the Agent. At the step-level, the Critic network estimates the value function $V_{\pi }(x,\lambda )$ to guide the Actor and at the episode-level, the expected cumulative safety cost $J_C\left({\pi }_{\theta }\right)$ directs the dynamic adjustment of the Lagrange multiplier $\lambda$, ensuring robust constraint adherence.

2.2.1. MDP Formulation

The optimization objective is to maximize the discounted cumulative return subject to safety constraints (Equation (10)):

$$\underset{\theta }{max}{\mathbb{E}}_{\tau }\left[\sum _t{\gamma }^t{R}_t\right],\mathrm{s}.\mathrm{t}.\mathbb{E}\left[C({\mathbf{s}}_t,{\mathbf{a}}_t)\right]\le \xi$$

(10)

State space $\mathcal{S}$: ${\mathbf{s}}_t\in {\mathbb{R}}^{12}$, comprising longitudinal kinematic states (four dimensions: $x,u,\theta ,q$), tracking errors (two dimensions: position error and velocity error), current normalized gains (four dimensions, reflecting the historical tuning trajectory), and attitude dynamic information (two dimensions: $\dot{\theta },\dot{q}$). The inclusion of current normalized gains in the state vector enables the agent to perceive “how far the current parameters lie from the boundary”, thereby facilitating proactive convergence when approaching the LMI constraint boundary, rather than relying on external hard clipping to passively trigger corrections.

Action space $\mathcal{A}$: ${\mathbf{a}}_t={[K_{p,x},K_{p,u},K_{i,u},K_{d,u}]}^T\in {[-1,1]}^4$, mapped to physical gains through an affine transformation (Equation (11)) that naturally aligns with the LMI safety boundary $[{\mathbf{K}}_{min},{\mathbf{K}}_{max}]$:

$${\mathbf{K}}_{\mathrm{target}}={\displaystyle \frac{{\mathbf{a}}_t+1}{2}}\left({\mathbf{K}}_{\max }-{\mathbf{K}}_{\min }\right)+{\mathbf{K}}_{\min }$$

(11)

Reward function $R_t$ (Equation (12)); the constraint cost $C_t$ is defined in Equation (13):

$$R_t=r_t-\lambda C_t,r_t=-clip\left(\sum _i{w}_i{e}_i^2,0,50\right)$$

(12)

$$C_t=\sum _i{C}_{\mathrm{gain},i}+C_{\mathrm{vel}}+C_{\mathrm{att}}+C_{\mathrm{rate}},C_{\mathrm{gain},i}=max\left(0,|{\mathbf{a}}_{t,i}|-m_{\mathrm{gain}}\right)$$

(13)

The reward-truncation upper bound of 50 prevents a small number of extreme trajectories from dominating gradient updates. The constraint cost $C_t$ simultaneously monitors four categories of safety violations: gain boundary exceedance ($C_{\mathrm{gain},i}$), velocity limit violation ($C_{\mathrm{vel}}$), excessive attitude ($C_{\mathrm{att}}$), and angular rate exceedance ($C_{\mathrm{rate}}$), covering the primary modes of UAV loss of control.

Value function and advantage estimation: Standard MDP value function $V^{\pi }\left(s_t\right)$, action-value function $Q^{\pi }(s_t,a_t)$, and generalized advantage estimation (GAE, ${\lambda }_{\mathrm{GAE}}=0.95$) are employed; complete definitions are provided in Appendix C (Equations (A6) and (A7)). The discount factor $\gamma =0.99$ endows the agent with an effective planning horizon of approximately 2 s at a 20 ms control period, sufficient to encompass a complete velocity step-response transient.

2.2.2. Constraint-Aware PPO Algorithm

Algorithm-selection rationale. PPO is selected over SAC or DDPG [17] based on two engineering considerations. First, the PPO clipping surrogate objective inherently limits the magnitude of single-step policy updates—let ${\rho }_t\left(\theta \right)={\pi }_{\theta }\left({\mathbf{a}}_t\right|{\mathbf{s}}_t)/{\pi }_{{\theta }_{\mathrm{old}}}\left({\mathbf{a}}_t\right|{\mathbf{s}}_t)$ denote the probability ratio; the clipping surrogate objective is:

$$L^{\mathrm{CLIP}}\left(\theta \right)={\mathbb{E}}_t\left[min\left({\rho }_t\left(\theta \right){\widehat{\mathcal{A}}}_t,clip({\rho }_t\left(\theta \right),1-\epsilon ,1+\epsilon ){\widehat{\mathcal{A}}}_t\right)\right]$$

(14)

where $\epsilon =0.2$ and ${\widehat{\mathcal{A}}}_t$ is the generalized advantage estimate. This truncation mechanism is highly compatible with the requirement to prevent catastrophic forgetting during Stage 6 online fine-tuning. Second, in the offline stages (Sub-stages 1–5) with 32 parallel environments, PPO’s on-policy sampling eliminates the need for an experience replay buffer, significantly reducing memory footprint and hyperparameter sensitivity. The total loss function integrates the clipping surrogate objective, value function error, and policy entropy:

$$L\left(\theta \right)=L^{\mathrm{CLIP}}\left(\theta \right)-c_1{L}_{\mathrm{VF}}\left(\theta \right)+c_2{H}_{{\pi }_{\theta }}$$

(15)

where $c_1=0.5$ (value function coefficient), $c_2=0.01$ (entropy coefficient), and $L_{\mathrm{VF}}\left(\theta \right)=\frac{1}{2}{\mathbb{E}}_t\left[{(V_{\theta }\left({\mathbf{s}}_t\right)-V_{\mathrm{target},t})}^2\right]$.

Motivation for Lagrangian constraints. Embedding fixed-weight penalty terms directly into the reward function requires extensive manual tuning, and the optimal weights vary significantly across training stages. Lagrangian relaxation [26] is adopted to transform the constrained optimization objective (Equation (10)) into an unconstrained composite objective:

$$\mathcal{L}(\theta ,\lambda )=J_R\left({\pi }_{\theta }\right)-\lambda \left(J_C\left({\pi }_{\theta }\right)-\xi \right)$$

(16)

where $J_R\left({\pi }_{\theta }\right)$ is the expected cumulative performance reward, $J_C\left({\pi }_{\theta }\right)$ is the expected cumulative safety cost, and $\xi =0.03$ is the violation tolerance threshold. The multiplier $\lambda$ automatically tracks the degree of constraint violation at runtime: it increases to intensify penalties when gains exceed boundaries or flight states are violated, and decreases to restore exploration freedom once constraints are satisfied. This mechanism converts the safety–performance trade-off into an online optimization problem in dual space, eliminating the need for manually setting penalty weights for each training stage.

Design rationale for dual-timescale separation. Taking the gradient of the Lagrangian objective (Equation (16)) with respect to $\theta$ yields the policy update direction; subgradient ascent on $\lambda$ yields the multiplier update. The two are decoupled via different learning rates to achieve timescale separation:

$${\theta }_{t+1}={\theta }_t+{\alpha }_{\theta }{\nabla }_{\theta }\mathcal{L}({\theta }_t,{\lambda }_t),{\lambda }_{t+1}={\mathcal{P}}^{+}\left[{\lambda }_t+{\alpha }_{\lambda }\left(J_C\left({\pi }_{{\theta }_t}\right)-\xi \right)\right]$$

(17)

where ${\mathcal{P}}^{+}$ is a projection operator ensuring $\lambda \ge 0$; the policy network updates rapidly at ${\alpha }_{\theta }={10}^{-4}$, while the Lagrange multiplier adjusts slowly at ${\alpha }_{\lambda }={10}^{-5}$ (${\alpha }_{\lambda }\ll {\alpha }_{\theta }$). The order-of-magnitude difference between the two learning rates ensures that the multiplier undergoes significant adjustment only after the policy has sufficiently converged, analogous to the “fast–slow system separation” in singular perturbation theory: if the multiplier updates too rapidly, the policy is forced to change direction before it has responded to the constraint signal, leading to oscillation or even divergence. The LMI constraint-ablation study (see Section 3.2.1) corroborates the necessity of this mechanism from a complementary perspective: disabling the gain-constraint penalty causes the policy to consistently fail to converge to low constraint-violation levels throughout training.

Structural basis for policy transfer. The X-configuration quadrotor satisfies $I_{xx}\approx I_{yy}$ ($\Delta I/I<4\%$) and the fuselage-drag coefficient $C_d$ is approximately isotropic, endowing the longitudinal and lateral channels with approximately isomorphic dynamical structures in state space. Consequently, the converged longitudinal policy ${\pi }_{\mathrm{long}}^{*}$ can be directly mapped to the lateral channel, reducing lateral-channel training costs to zero while maintaining performance degradation within acceptable bounds (validated in Section 3.2.2).

2.3. Overall Methodology and Agent-Training Framework

The proposed hierarchical safe-RL method is structured into four interconnected steps that sequentially transition from theoretical modeling to practical deployment (see Figure 2).

As illustrated in Figure 2, the consecutive steps closely interact to systematically elevate control performance. Step I (LPV Modeling) decouples the quadcopter dynamics and synthesizes an augmented multi-cell linear parameter-varying (LPV) model covering the flight envelope. This mathematical structure is directly fed into Step II (LMI Safety Domain), where Lyapunov functions are constructed to solve the safe outer-loop parameter boundaries and optimal initial PID gains. These theoretical bounds provide a physically meaningful and constraint-guaranteed search space for the reinforcement learning agent, fundamentally accelerating subsequent training efforts.

Offline multi-stage curriculum training (Step III, Sub-stages 1–5): A progressive curriculum design gradually escalates task difficulty—Stages 1–4 employ fixed simulation parameters, progressing from constant-velocity tracking to chirp-based aggressive maneuvers with wind disturbances and Stage 5 introduces full-domain randomization to enhance generalization. All five offline stages employ 32-environment parallel PPO, warm-started from the best checkpoint of the preceding stage. The interconnection from Step II to Step III restricts the neural network’s exploration within the derived rigorous physical bounds. Consequently, the initial offline stages (Stages 1–4) successfully yield a basic maneuvering policy that achieves rapid convergence on simplified dynamics, thereby establishing a fundamental tracking capability. The subsequent domain randomization phase (Stage 5) produces an intermediate robust policy (${\pi }_5^{*}$) resilient to parameter variations, bridging the gap to real-world deployment by significantly reducing sim-to-real disparities.

Online fine-tuning and policy transfer (Step IV, Sub-stage 6): The Stage 5 converged policy ${\pi }_5^{*}$ is transferred to a Gazebo M480 model integrated with the PX4 autopilot stack for online fine-tuning, employing a conservative learning rate ($\alpha ={10}^{-5}$) and a tightened PPO clipping range ($\epsilon =0.1$) to prevent catastrophic forgetting. The interconnection from Step III to Step IV passes this highly generalized but moderately conservative policy (${\pi }_5^{*}$) into high-fidelity environments. This final fine-tuning step yields the optimal deployable policy, actively compensating for unmodeled aerodynamics and actuator delays. Cumulatively, the results from each step—from theoretical bounding to progressive simulation and final sim-to-real transfer [30]—contribute to the overarching performance improvement by safely decomposing an otherwise intractable end-to-end safe RL exploration into sequential, physically informed optimization tasks. The dual-timescale parameter update still follows the rule described in Equation (17). After longitudinal policy convergence, the policy is directly reused for the lateral channel via dynamic symmetry ($I_{xx}\approx I_{yy}$), without retraining.

3. Results

Our results are presented in the following order: LMI safe domain construction, offline five-stage curriculum training and online fine-tuning results, and performance evaluation across four comparative experiments.

3.1. Results of LMI-Based Safe-Parameter-Domain Construction

3.1.1. UAV Physical Parameters

An X-configuration quadrotor is employed; the complete physical parameters are listed in Appendix D,

Table A2. Simulation-UAV physical parameters.

Category	Parameter	Symbol	Value	Unit
Basic	Total mass	m	1.4	kg
	Gravitational accel.	g	9.8	m/s2
Inertia	Roll axis	$I_{xx}$	0.0211	kg·m2
	Pitch axis	$I_{yy}$	0.0219	kg·m2
	Yaw axis	$I_{zz}$	0.0366	kg·m2
Geometry	Arm length	l	0.241	m
Aerodynamics	Thrust coeff.	$C_t$	$1.105\times {10}^{-5}$	N/(rad/s)2
	Torque coeff.	$C_m$	$1.779\times {10}^{-7}$	N·m/(rad/s)2
	Drag coeff.	$C_d$	0.073	N/(m/s)2
	Damping torque coeff.	$C_{dm}$	0.0055	N·m/(rad/s)2
Motor	Response time const.	$T_m$	0.02	s

(mass 1.4 kg, arm length 0.241 m, $I_{xx}=0.0211$ kg·m², $I_{yy}=0.0219$ kg·m², $C_d=0.073$ N/(m/s)², motor time constant $T_m=0.02$ s).

3.1.2. Longitudinal LPV Model and Polytopic Construction

Within the operating envelope $u\in [0,15]$ m/s, linearization is performed at four characteristic points, yielding vertex systems $({\mathbf{A}}_k,{\mathbf{B}}_k),k=1,\dots ,4$. The augmented state ${\mathbf{X}}_a$ comprises position error, velocity, attitude angle, angular rate, and integral states of the velocity and angular rate loops.

3.1.3. LMI Constraint Design and Solution Results

The LMI constraint parameters for each control loop are listed in Appendix D,

Table A3. LMI-constraint parameters for each control loop.

Control Loop	$\mathit{\alpha }$-Stability	Conic Angle $\mathit{\phi }$	Phase Margin	Gain Margin	Bandwidth
Angular rate loop	≥2.0	≤$50°$	≥$60°$	≥10 dB	[10, 30] rad/s
Attitude loop	≥2.0	≤$50°$	≥$60°$	≥10 dB	[5, 15] rad/s
Velocity loop	≥1.0	≤$60°$	≥$50°$	≥8 dB	[1, 5] rad/s
Position loop	≥0.8	≤$65°$	≥$45°$	≥6 dB	[0.5, 2] rad/s

.

Step 1—Inner-loop parameter fixation. A high-bandwidth, well-damped configuration is selected: attitude loop $K_{p,\theta }=8.0$ and angular rate loop $K_{p,q}=1.5$, $K_{i,q}=12.0$, and $K_{d,q}=0.04$. The closed-loop poles at all four velocity vertices fall within the $D_{\alpha =2.0,\phi =50°}$ sector, with a phase margin $\ge 60°$ and a gain margin $\ge 10$ dB (see Figure 3).

Step 2—Outer-loop safety boundary. Substituting ${\mathsf{\Theta }}_{\mathrm{in}}^{*}$ into the augmented system, the outer-loop LMI feasible region is solved; the results are presented in

Table 2. Longitudinal outer-loop parameter LMI safety domain.

Parameter	Lower Bound ${\mathit{K}}_{min}$	Upper Bound ${\mathit{K}}_{max}$	Initial Value (Midpoint)
$K_{p,x}$	0.2	4.5	2.4
$K_{p,u}$	1.0	10.0	5.5
$K_{i,u}$	3.0	25.0	14.0
$K_{d,u}$	0.05	0.55	0.3

(see Figure 4).

Leveraging the X-configuration structural symmetry ($I_{xx}\approx I_{yy}$), the lateral channel adopts the same parameter boundaries as the longitudinal channel. The yaw and altitude channels employ fixed parameters obtained from single-point hovering LMI solutions: altitude outer loop $K_{p,z}=1.0$; altitude inner loop $K_{p,\dot{z}}=4.85$, $K_{i,\dot{z}}=2.85$, and $K_{d,\dot{z}}=1.0$; yaw outer loop $K_{p,\psi }=1.5$; and yaw inner loop $K_{p,r}=1.35$, $K_{i,r}=0.0$, and $K_{d,r}=0.5$.

3.2. PPO-Based Adaptive Controller Parameter Training

3.2.1. Offline Five-Stage Curriculum Training (Sub-Stages 1–5)

The RL agent exclusively adjusts the longitudinal outer-loop parameters: state ${\mathbf{s}}_t\in {\mathbb{R}}^{12}$ and action ${\mathbf{a}}_t\in {[-1,1]}^4$, affine-mapped to the safety boundary $[{\mathbf{K}}_{min},{\mathbf{K}}_{max}]$ of Table 2. The five-stage offline training configuration is detailed in

Table 3. Five-stage offline curriculum training configuration.

Stage	Reference Signal	Environment/Randomization	Training Objective	Steps
Stage 1	Constant velocity $u_x=5.0$ m/s	No wind, widest constraints	Baseline tracking	1.0 M
Stage 2	Sinusoidal, 8 m/s, 0.3 Hz	No wind	Periodic dynamic tracking	2.0 M
Stage 3	Chirp, 0.1–0.5 Hz, 8–0 m/s	No wind	Wideband frequency response	3.0 M
Stage 4	Chirp (same as Stage 3)	Mild wind 0–2 m/s	Tracking under disturbance	4.0 M
Stage 5	Mixed chirp/sinusoidal/step	Full-domain rand., wind 0–2.5 m/s	Generalization	5.0 M

. The penalty weight $W_{\mathrm{gain}}=5.0$ and the gain soft-constraint margin $m_{\mathrm{gain}}=0.70$ are maintained consistently across all stages (hard-constraint margin $m_{\mathrm{hard}}=0.95$).

The Lagrange multiplier ${\lambda }_j$ is updated according to the slow-timescale rule of Equation (17) (${\alpha }_{\lambda }={10}^{-5}$, violation tolerance threshold $\xi =0.03$).

Core PPO hyperparameters are listed in Appendix D,

Table A4. PPO-hyperparameter configuration.

Category	Parameter	Value
Algorithm	${\epsilon }_{\mathrm{clip}}$	0.2
	$\gamma$	0.99
	${\lambda }_{\mathrm{GAE}}$	0.95
Network training	${\alpha }_{\theta }$	${10}^{-4}$ (linear decay)
	${\alpha }_{\lambda }$	${10}^{-5}$
	${\delta }_{\mathrm{grad}}$ (gradient clipping)	0.5
	$N_{\mathrm{epoch}}$	5
	$N_{\mathrm{batch}}$	512
Loss weights	$c_1$ (vf_coef)	0.5
	$c_2$ (ent_coef)	0.01
Sampling	$N_{\mathrm{step}}$	1024
	$N_{\mathrm{env}}$ (parallel envs)	32 (Stages 1–5)/1 (Stage 6)

Training platform: AMD Ryzen-9 5950X/NVIDIA RTX 4060/32 GB DDR4, Ubuntu 22.04 LTS. Total training: approximately $1.68\times {10}^7$ timesteps (offline Stages 1–5: 6.5 M; online Stage 6: approximately 6.0 M).

(${\epsilon }_{\mathrm{clip}}=0.2$, $\gamma =0.99$, ${\lambda }_{\mathrm{GAE}}=0.95$, $N_{\mathrm{env}}=32$, $N_{\mathrm{step}}=1024$).

Training platform: AMD Ryzen-9 5950X/NVIDIA RTX 4060/32 GB DDR4, Ubuntu 22.04 LTS. The total offline training comprises approximately $1.5\times {10}^7$ timesteps (Stages 1–5 combined).

Figure 5 illustrates the reward-curve evolution across the five offline training stages. The magnitude of negative transfer at each stage transition progressively diminishes with advancing stages, indicating that the progressive curriculum design effectively reduces the cost of cross-task knowledge transfer.

Quantitative performance summaries at the conclusion of the five offline stages are presented in

Table 4. Performance summary at the conclusion of each offline training stage.

Stage	Environment	Final Reward	Mean Velocity Error
Stage 1	Simplified simulation	≈−570	0.41 m/s
Stage 2	Simplified simulation	≈−4750	1.03 m/s
Stage 3	Simplified simulation	≈−2870	1.24 m/s
Stage 4	Simplified simulation	≈−2940	1.24 m/s
Stage 5	Simplified + domain rand.	≈−5460	1.47 m/s

.

LMI constraint-ablation study. To verify the necessity of the LMI safety domain constraints, the RL action constraints are degraded to simple symmetric clipping ($K\in [K_{\mathrm{mid}}\pm \Delta K]$, where $\Delta K$ is comparable to the LMI boundary width), the gain-constraint penalty weight $W_{\mathrm{gain}}$ is set to zero, and all other hyperparameters are held constant. Comparative training is conducted on the Stage 5 (domain randomization) task. As shown in

Table 5. LMI constraint-ablation study statistics (Stage 5, last 1000 episodes).

Metric	With LMI	Without LMI	Difference
Constraint violations/ep	$13.8\pm 5.6$	$48.4\pm 2.5$	$-71.4\%$
Episode Reward	$-6185\pm 2610$	$-4783\pm 1052$	— †

^† The reward function compositions differ between the two variants (with LMI including constraint penalty terms); hence, episode reward is not directly comparable.

and Figure 6, the LMI-constrained variant achieves a 71.4% reduction in constraint violations (gain-boundary exceedances per episode): $13.8\pm 5.6$ vs. $48.4\pm 2.5$. The policy without LMI constraints, lacking penalty guidance, consistently fails to converge to low constraint-violation levels throughout training, validating that the LMI constraint mechanism is a necessary condition for achieving control within safe boundaries.

3.2.2. Gazebo Online Fine-Tuning (Stage 6) and Lateral Transfer

(1) Stage 6—High-Fidelity Online Fine-Tuning

Policy ${\pi }_5^{*}$ is transferred to a Gazebo M480 model integrated with the PX4 autopilot stack for online fine-tuning. Realistic sensor noise is introduced (${\sigma }_v=0.10$ m/s, ${\sigma }_p=0.05$ m, and ${\sigma }_{\theta }=0.02$ rad), with the system operating at 50 Hz via ROS2 DDS and a communication latency of 3–5 ms. Fine-tuning proceeds for a total of $6.0\times {10}^6$ timesteps; the smoothed episode reward rises steadily from approximately $-5576$ to $-5171$, reflecting gradual adaptation to fixed-bias plant parameters and sensor noise (see Figure 7). The limited absolute improvement ($\Delta \approx 400$) indicates that Stage 5 full-domain randomization had already brought the policy close to the Stage 6 performance ceiling, so Stage 6 fine-tuning primarily consolidates robustness rather than relearning the task. Stage 6 employs single-environment serial sampling in Gazebo ($N_{\mathrm{env}}=1$), resulting in an effective data throughput far lower than that of the offline 32-environment parallel stages (e.g., Stage 5 effective throughput: approximately $5.0\mathrm{M}\times 32=160\mathrm{M}$). The policy, nonetheless, converges rapidly for three reasons: (1) Stage 5 full-domain randomization has sufficiently enhanced generalization, and online fine-tuning only needs to eliminate motor dynamic residuals rather than relearn the task; (2) the conservative learning rate ($\alpha ={10}^{-5}$) prevents catastrophic forgetting; and (3) the initial policy ${\pi }_5^{*}$ already possesses strong tracking capability, significantly narrowing the fine-tuning search space.

(2) Lateral Channel Policy Transfer

Based on the dynamic symmetry of the X-configuration quadrotor ($I_{xx}\approx I_{yy}$, approximately isotropic $C_d$), the converged longitudinal policy ${\pi }_{\mathrm{long}}^{*}$ is directly mapped to the lateral channel without retraining. The test conditions are $u_y=8sin\left(0.2\pi t\right)$ m/s superimposed with 0–5 m/s random wind, comprising $n=100$ independent trials (50 trials each at 3 m/s and 5 m/s wind speeds); the results are shown in Figure 8.

The lateral-channel RMSE exceeds the longitudinal value by only 16.3%, which is below the commonly adopted 20% transfer feasibility threshold, validating the viability of the “train longitudinal–reuse lateral” approach.

3.3. Experimental Results and Analysis

The following four experiments compare RL-PID against an engineering-tuned fixed-gain PID ($K_{p,x}=0.7$, $K_{p,u}=3.0$, $K_{i,u}=3.0$, $K_{d,u}=0.3$).

The three experiments in Section 3.3.1, Section 3.3.2 and Section 3.3.3 are each conducted with $n=10$ independent repetitions in the high-fidelity simulation environment, each run under 1–3 m/s random wind disturbance (uniformly random wind direction and seed fixed at 42 for reproducibility), with statistics reported as mean ± standard deviation ($\overline{x}\pm s$, $n=10$). Improvement is computed as the ratio of means: $\Delta =({\overline{x}}_{\mathrm{PID}}-{\overline{x}}_{\mathrm{RL}})/{\overline{x}}_{\mathrm{PID}}\times 100\%$.

3.3.1. High-Speed Step Response

Experimental setup. The UAV starts from hover and receives a 13 m/s step velocity command sustained for 10 s, with a pitch angle constraint $\left|\theta \right|\le 60°$ (see Figure 9). The detailed quantitative results are summarized in

Table 6. Step-response performance comparison ($n=10$, random wind 0–3 m/s, and mean ± std). Bold values indicate the best performance.

Metric	RL-PID	Fixed-PID	Improvement
Rise time $t_r$ (s)	$\mathbf{1.451}\pm \mathbf{0.604}$	$1.837\pm 0.869$	$+21.0\%$
Settling time $t_s$ (s)	$\mathbf{3.982}\pm \mathbf{2.918}$	$8.042\pm 2.054$	$+50.5\%$
Overshoot $M_p$ (%)	$\mathbf{1.692}\pm \mathbf{1.178}$	$2.077\pm 0.977$	$+18.5\%$
Steady-state error $e_{ss}$ (m/s)	$\mathbf{0.175}\pm \mathbf{0.240}$	$0.471\pm 0.462$	$+62.8\%$
RMSE (m/s)	$\mathbf{3.309}\pm \mathbf{0.206}$	$3.423\pm 0.311$	$+3.3\%$
MAE (m/s)	$\mathbf{1.363}\pm \mathbf{0.310}$	$1.646\pm 0.441$	$+17.1\%$

.

3.3.2. Emergency-Braking Response

Experimental setup. The UAV cruises at $u=13$ m/s; at $t=3$ s, a step command to 0 m/s is issued, which is sustained for 10 s (see Figure 10). The quantitative results are summarized in

Table 7. Emergency-braking-performance comparison ($n=10$, random wind 0–3 m/s, and mean ± std). Bold values indicate the best performance.

Metric	RL-PID	Fixed-PID	Improvement
Braking time $t_{\mathrm{brake}}$ (s)	$\mathbf{1.194}\pm \mathbf{0.141}$	$1.319\pm 0.091$	$+9.5\%$
Braking distance $d_{\mathrm{brake}}$ (m)	$\mathbf{7.773}\pm \mathbf{1.319}$	$8.428\pm 1.287$	$+7.8\%$
Max. pitch angle ${\theta }_{max}$ (°)	$59.418\pm 0.688$	$\mathbf{58.001}\pm \mathbf{2.132}$	$-2.4\%$

.

3.3.3. Frequency-Domain Sweep Test

Experimental setup. Two test configurations using chirp signals: Test A (constant-amplitude sweep): amplitude ±8 m/s and frequency linearly increasing from 0.1 Hz to 1.0 Hz over 20 s. Test B (constant-frequency amplitude sweep): frequency fixed at 0.5 Hz and amplitude linearly increasing from 3 m/s to 13 m/s over 20 s. The quantitative results are summarized in

Table 8. Frequency-sweep-test performance comparison ($n=10$, random wind 0–3 m/s, and mean ± std).

Test Type	Metric	RL-PID	Fixed-PID	Improvement
Const.-amplitude	RMSE (m/s)	$\mathbf{3.781}\pm \mathbf{0.137}$	$4.086\pm 0.059$	$+7.5\%$
	MAE (m/s)	$\mathbf{2.947}\pm \mathbf{0.130}$	$3.218\pm 0.064$	$+8.4\%$
Const.-frequency	RMSE (m/s)	$\mathbf{3.873}\pm \mathbf{0.118}$	$4.429\pm 0.060$	$+12.6\%$
	MAE (m/s)	$\mathbf{3.066}\pm \mathbf{0.062}$	$3.453\pm 0.029$	$+11.2\%$

, and the corresponding response curves are shown in Figure 11.

3.3.4. Comprehensive Statistical Performance Test

Experimental setup. A total of 100 randomized mixed-trajectory trials (see Figure 12) covering four trajectory types—constant velocity, sinusoidal, step, and chirp—were conducted under 0–5 m/s random wind disturbance. Sample sizes per category were weighted by flight mission frequency: step 30 trials, constant velocity 24 trials, sinusoidal 24 trials, and chirp 22 trials, totaling 100 trials. Initial velocity, wind speed, and wind direction were uniformly randomly sampled within each category. All reported improvements were computed relative to the fixed PID baseline as $\Delta =({\mathrm{RMSE}}_{\mathrm{PID}}-{\mathrm{RMSE}}_{\mathrm{RL}})/{\mathrm{RMSE}}_{\mathrm{PID}}\times 100\%$.

4. Discussion

4.1. Adaptive-Mechanism Analysis

By synthesizing the gain-variation curves across the four comparative experiments (Figure 9d, Figure 10d and Figure 11c,f), three characteristic adaptive scheduling behaviors can be identified:

• Error-driven regulation. When the tracking error is large, $K_p$ and $K_i$ increase to accelerate convergence; as the error diminishes, $K_i$ decreases to suppress steady-state oscillation. In the step-response experiment (Section 3.3.1), $K_{i,u}$ actively rises during the transient phase ($t<2$ s) and returns to a lower value at steady state, ultimately reducing steady-state error by 62.8% while simultaneously decreasing overshoot by 18.5%.
• State-synchronous regulation. Under periodic reference signals, gain variations exhibit correlation with both the amplitude and frequency of the reference signal. In the frequency sweep test (Section 3.3.3), $K_{p,u}$ and $K_{i,u}$ increase correspondingly during high-amplitude segments and decrease during low-amplitude segments, with the gain-adjustment rhythm synchronized to the reference signal period. This indicates that the policy exploits the dynamic characteristics of the reference signal as an implicit scheduling variable, enabling RL-PID to significantly outperform the fixed-gain scheme on chirp trajectories (improvement of 40.9%) and sinusoidal trajectories (improvement of 18.8%).
• Constraint-aware regulation. When system states approach the $60°$ pitch-angle-constraint boundary, the policy proactively reduces gains to mitigate the risk of constraint violation. In the emergency-braking experiment (Section 3.3.2), the RL-PID mean maximum pitch angle ($59.4\pm 0.7°$) is slightly higher than that of the fixed PID ($58.0\pm 2.1°$); neither violates the $60°$ hard constraint, while RL-PID achieves a 9.5% reduction in braking time through more aggressive gain scheduling without any constraint violations.

These three behaviors demonstrate that the control law learned by the RL policy is not a fixed-gain mapping, but rather a comprehensive response to the current error state, reference signal dynamics, and system safety margins.

4.2. Comprehensive Performance Discussion

Two consistent conclusions emerge from

Table 9. Mixed-trajectory test overall statistics ($n=100$).

Metric	RL-PID	Fixed PID	Improvement
Velocity RMSE (m/s)	$1.90\pm 1.70$	$2.23\pm 1.69$	15.0%
Velocity MAE (m/s)	$1.21\pm 1.21$	$1.52\pm 1.26$	20.5%
Max. pitch angle (°)	$53.9\pm 11.1$	$52.3\pm 12.3$	$-3.1\%$

,

Table 10. Per-trajectory RMSE comparison (mean ± std, unit: m/s).

Trajectory Type	n	RL-PID	Fixed PID	Improvement
Constant velocity	24	$1.65\pm 0.90$	$1.72\pm 0.91$	$+4.1\%$
Sinusoidal	24	$1.75\pm 1.98$	$2.16\pm 2.00$	$+18.8\%$
Step	30	$2.80\pm 2.13$	$2.97\pm 2.22$	$+5.8\%$
Chirp	22	$1.11\pm 0.52$	$1.87\pm 0.46$	$+40.9\%$

and

Table 11. Summary of core performance improvements across four experiments (improvement $=(\mathrm{PID}-\mathrm{RL})/\mathrm{PID}\times 100\%$).

Experiment	Core Metric	RL-PID	Fixed PID	Improvement
Step response	Overshoot (%)	$1.69\pm 1.18$	$2.08\pm 0.98$	18.5%
	Steady-state error (m/s)	$0.18\pm 0.24$	$0.47\pm 0.46$	62.8%
	MAE (m/s)	$1.36\pm 0.31$	$1.65\pm 0.44$	17.1%
Emergency braking	Braking time (s)	$1.19\pm 0.14$	$1.32\pm 0.09$	9.5%
	Braking distance (m)	$7.77\pm 1.32$	$8.43\pm 1.29$	7.8%
Sweep (const.-amp.)	RMSE (m/s)	$3.78\pm 0.14$	$4.09\pm 0.06$	7.5%
Sweep (const.-freq.)	RMSE (m/s)	$3.87\pm 0.12$	$4.43\pm 0.06$	12.6%
Mixed ($n=100$)	Velocity RMSE (m/s)	$1.90\pm 1.70$	$2.23\pm 1.69$	15.0%
	Chirp RMSE (m/s)	$1.11\pm 0.52$	$1.87\pm 0.46$	40.9%

:

Performance improvement is positively correlated with task-dynamic complexity. For steady-state tracking (constant velocity trajectories and 4.1% improvement) and large-amplitude step responses (RMSE improvement of 3.3%), the fixed PID already performs reasonably well, leaving limited marginal benefit from RL-PID. However, for time-varying dynamic signals (chirp, sinusoidal, and sweep), the advantage of RL-PID expands significantly with increasing task-dynamic complexity.

Safety constraints are effectively maintained throughout performance enhancement. In the mixed-trajectory test, the RL-PID maximum pitch angle ($53.9°\pm 11.1°$) is slightly higher than that of the fixed PID ($52.3°\pm 12.3°$), yet none of the 100 trials exceeded the $60°$ hard constraint (see Table 9). The slightly higher mean maximum pitch angle of RL-PID is attributable to its more aggressive gain scheduling—actively increasing $K_{p,u}$ during high-dynamic segments to accelerate tracking, thereby driving pitch angles closer to the constraint boundary—while the Lagrange multiplier mechanism suppresses boundary exceedance risk in a timely manner when the system approaches constraints. This observation is consistent with the “constraint-aware regulation” analysis in Section 4.1: RL-PID trades a limited pitch-angle margin for faster dynamic response while maintaining overall compliance with hard constraints. The joint action of the LMI safety domain and Lagrangian-constrained PPO forms a dual-layer protection mechanism of “soft-boundary guidance + hard-boundary backstop.”

Variance analysis of the mixed-trajectory test. The coefficient of variation (CV ≈ 89%) of the RL-PID velocity RMSE in Table 9 originates from trajectory-type heterogeneity rather than sporadic controller instability. The per-category statistics in Table 10 corroborate this assessment: the inherent task-difficulty gap between chirp-trajectory RL-PID RMSE (1.11 m/s) and step-trajectory RMSE (2.80 m/s) is approximately 2.5-fold, naturally inflating the pooled variance when mixed. The fixed-PID pooled standard deviation (±1.69) is virtually identical to that of RL-PID (±1.70), further supporting this conclusion.

4.3. Extended Baselines, Runtime Trade-Offs, and Training Sensitivity

Positioning against extended baselines. Since fixed-gain PID and the LMI-ablated variant do not cover the full control landscape, it is useful to position the proposed method against Gain-Scheduled PID, LPV/LMI fixed-gain scheduling, MPC, and ADRC. Conceptually, RL-PID separates three roles that are often coupled: robust feasibility, online performance adaptation, and computational deployment. The LMI region provides an explicit safe-search domain, avoiding unconstrained gain exploration; the RL policy then exploits the remaining physical margin to reduce the conservatism of worst-case LPV/LMI tuning under time-varying references. Compared with MPC, this design shifts the optimization burden from online receding-horizon solving to offline policy training, leaving only a lightweight neural-network evaluation during flight. Compared with ADRC, the search space is bounded by explicit physical and stability constraints rather than relying solely on feedback-error-driven disturbance rejection. Thus, the observed gain does not come from RL alone or from the LMI box alone, but from their division of labor: LMI defines admissible behavior, while RL selects task-dependent gains within that admissible set.

Computation–performance trade-off analysis. Figure 13 provides a quantitative view of this positioning by comparing Fixed PID, Gain-Scheduled PID, Linear MPC, and RL-PID across three benchmark scenarios. The x-axis reports per-step computation time in $\mathsf{\mu }$s, and the y-axis reports velocity RMSE; error bars denote the standard deviation over $n=10$ trials. The results show that Linear MPC achieves the lowest RMSE (≈0.81–0.93 m/s), but at a much higher computational cost (276 $\mathsf{\mu }$s/step), whereas Fixed PID is cheapest (≈3.4 $\mathsf{\mu }$s/step) but loses accuracy in dynamic trajectories such as Chirp Sweep. RL-PID lies between these extremes: its runtime cost is close to Gain-Scheduled PID (approximately 9–10 $\mathsf{\mu }$s/step), while its accuracy approaches MPC in the Chirp Sweep and Mixed Maneuver scenarios. In the easier step-response case, all methods converge to a similar RMSE (≈0.76 m/s), so computation becomes the dominant differentiator. Overall, RL-PID occupies a favorable region of the Pareto front, retaining PID-class computational efficiency while recovering much of the adaptive performance usually associated with more expensive online optimization.

Training sensitivity and robustness of the comparison. The above trade-off is meaningful only if the RL-PID policy is not a fragile outcome of hyperparameter tuning. In practice, stable learning requires the training dynamics to follow the same hierarchy as the controller design: the actor should first learn useful tracking behavior, while the Lagrange multiplier gradually enforces constraint discipline. We therefore use separated timescales (${\alpha }_{\theta }={10}^{-4}$ for the policy and ${\alpha }_{\lambda }={10}^{-5}$ for the multiplier). When ${\alpha }_{\lambda }$ was increased toward ${\alpha }_{\theta }$, small transient violations were penalized too early, causing oscillatory policies and loss of tracking ability; when it was too small, constraint enforcement became delayed during long transients. Similarly, overly tight violation margins or oversized penalty weights pushed the policy toward nearly static gains inside the LMI domain, recreating the conservatism that RL adaptation is intended to reduce. Initializing near the LMI midpoint and using soft margins therefore implements a practical “soft guidance before hard backstop” mechanism, supporting the stability of the Pareto comparison rather than merely improving one tuned run.

4.4. Limitations and Future Work

• Real-World Applicability and Sim-to-Real Gaps. Fundamentally, safety guarantees in this study theoretically hold under bounded-LPV mathematical derivations, but remain highly sensitive to significant model mismatch or unmodeled aerodynamic perturbations. Unmodeled actuator latency, aggressive sensor noise, or structural asymmetry ($\Delta I/I>4\%$) could breach the current strict offline bounds. While the proposed method is validated in a high-fidelity Gazebo environment incorporating standard actuator dynamics, the present contribution is inherently an analytical and simulation-based engineering study rather than a full hardware-level deployment. Future iterations must account for unmodeled noise scaling and explicitly tackle the hardware deployment sim-to-real chasm using domain adaptation or fine-tuning.
• Position–velocity trade-off. The proposed method prioritizes velocity tracking performance; position error may become non-negligible during highly dynamic tests. Future work may introduce multi-objective reward functions that explicitly balance velocity-tracking accuracy and position drift.
• Extended baseline comparisons. The analytical defense surrounding MPC, LPV, and ADRC benchmarks lays the foundation for future empirical validation. Implementing these exhaustive parallel baselines on identical test-bed environments constitutes an inevitable metric for definitively isolating the improvements yielded, specifically by RL, against other sophisticated controllers.

5. Conclusions

This paper develops a novel hierarchical adaptive-PID-control method integrating proximal policy optimization with LMI-based constraints to alleviate multirotor-control degradation during large-envelope flight. Utilizing an LPV formulation, LMI feasibility strictly dictates inner-loop stability and enforces constrained ranges for outer-loop online adjustments. Consequently, within high-fidelity simulation environments under demanding benchmarks, the proposed framework substantially outpaces fixed-gain formulations, demonstrating a 15% RMSE reduction overall and up to 40.9% performance elevation under severe chirp dynamics. Crucially, the mathematical safety guarantee solely protects the fixed inner loop at defined vertices, while the online RL parameter adaptation acts as a highly resilient engineering safeguard via Lagrangian boundaries rather than providing absolute global systemic stability out of simulation. Moving forward, validating robustness against acute structural asymmetry and unmodeled actuator delays, and conducting comparative live-flight studies against advanced nonlinear pipelines (such as Model Predictive Control) remain pivotal research priorities for transitioning this resilient framework into a fully deployed mechanism.