Safe UAV Control Against Wind Disturbances via Demonstration-Guided Reinforcement Learning

Highlights

What are the main findings?

• A hybrid controller combining PPO (RL), a PID expert (DGRL), and a real-time CBF safety filter achieves safer and more accurate hovering under gusts in both AirSim and indoor CoDrone tests.
• The hybrid design converges faster and more stably while incurring fewer safety violations during training and deployment.

What is the implication of the main finding?

• Demonstration-guided safe RL enables deployable, safety-aware UAV control in disturbed, uncertain environments without sacrificing steady-state precision.
• The framework provides a practical Sim-to-Real pathway for small UAVs, supporting precision tasks where robustness and formal safety constraints are critical.

Abstract

Unmanned Aerial Vehicle (UAV) operating in complex environments require guaranteed safety mechanisms while maintaining high performance. This study addresses the challenge of ensuring strict flight safety during policy execution by implementing a Control Barrier Function (CBF) as a real-time action filter, thereby providing a rigorous, formal guarantee. The methodology integrates the primary Proximal Policy Optimization (PPO) algorithm with a Demonstration-Guided Reinforcement Learning (DGRL), which leverages Proportional–Integral–Derivative (PID) expert trajectories to significantly accelerate learning convergence and enhance sample efficiency. Comprehensive results confirm the efficacy of the hybrid architecture, demonstrating a significant reduction in constraint violations and proving the framework’s ability to substantially accelerate training compared to PPO. In conclusion, the proposed methodology successfully unifies formal safety guarantees with efficient, adaptive reinforcement learning, making it highly suitable for safety-critical autonomous systems.

1. Introduction

Unmanned aerial vehicle (UAV) has been widely applied in tasks such as environmental monitoring, precision agriculture [1], and search and rescue missions [2]. Among these applications, precise hovering and stable control are fundamental capabilities. Real-world environments are inherently dynamic and unpredictable. However, factors such as sudden wind gusts, sensor noise, and system latency interact in complex ways, making it highly challenging to achieve safe, robust, and stable autonomous control. Traditional control methods, such as Proportional–Integral–Derivative (PID) controllers, are commonly employed in UAV flight systems due to their simplicity and fast response.

Nevertheless, these traditional methods have inherent limitations when facing complex, dynamic environments, to overcome the limitations of traditional control strategies. Reinforcement learning (RL) has emerged in recent years as a promising solution in robotics and UAV applications [3]. Among various RL algorithms, Proximal Policy Optimization (PPO) has demonstrated effectiveness in nonlinear continuous control tasks due to its stability and sample efficiency [4]. For instance, Bodi Ma et al. [5] applied RL to UAV tracking control under wind disturbances. Jiying Wu et al. [6] proposed the Watcher-Actor-Critic (WAC) framework, which improves the stability and efficiency of policy learning through a supervised RL mechanism and was successfully implemented on a physical UAV platform. However, applying RL methods directly to physical UAV faces challenges. To address these challenges, researchers have explored integrating RL with complementary control techniques, such as Demonstration-Guided Reinforcement Learning (DGRL) [7]. Specifically, Gon Kim et al. [8] proposed the SRAIL framework, which combines adversarial imitation learning with a two-stage automatic PID tuning process. This approach enables the use of imperfect demonstration data to establish an initial control policy, which is subsequently refined via RL. As a result, even in the absence of high-quality demonstration data, the RL agent can still perform stable and safe exploration. Such approaches significantly shorten training times and prevent policies from converging to unsafe states. Unlike traditional Imitation Learning (IL), which relies on a supervised imitation loss to merely replicate expert actions, our approach transforms the expert behavior into an intrinsic, differentiable reward signal and explicitly integrates it into the PPO reward function. This mechanism allows the agent to efficiently leverage expert guidance for accelerated convergence while fully retaining the exploratory capability of RL. Furthermore, Deploying RL policies from simulation to real UAV systems (Sim-to-Real) [9] remains a major research challenge. Simulation environments cannot fully capture re-al-world physical complexities such as sensor errors, wind disturbances, or hardware delays, which often cause policies to fail. This discrepancy is widely known as the Sim-to-Real Gap. To mitigate this issue, several approaches have been proposed, including Domain Randomization [10], Online Fine-Tuning [11], and Safe Transfer Learning [12]. Given the high risks associated with real-world deployment, Safe Transfer Learning has emerged as a particularly important research direction. Within this domain, the use of safety filters has proven especially critical. For example, Ugurlu et al. [13] investigated combining Model Predictive Control (MPC) with Deep Reinforcement Learning (DRL), where the MPC module functions as a safety filter. This approach leverages the strict stability and feasibility guarantees of MPC to manage physical uncertainties, such as modeling errors and delays, thereby ensuring that DRL policies can be deployed in real environments with greater safety and reliability. Specifically, safety mechanisms like the Control Barrier Function (CBF) [14]. provide a mathematical framework for enforcing safety constraints in real-time. CBF acts as a safety filter that evaluates and corrects potentially unsafe actions before execution, providing formal guarantees that the system state remains within a predefined safe set [15]. Owing to this capability, CBF have emerged as one of the key methodologies in Safe RL [16].

Safe RL techniques, such as those relying solely on CBFs, can enforce safety constraints during execution. This provides a fast and safe initialization pathway for the policy. Despite the potential of these methods, achieving safe and robust autonomous control still faces significant hurdles. Traditional PID methods with fixed, single-gain settings struggle with poorly modeled dynamics and complex disturbances, leading to performance degradation and compromised stability [17,18]. RL suffers from poor sample efficiency and hazardous exploration. Classical Imitation Learning (IL) [19] approaches are fundamentally limited by the performance ceiling of the expert and lack the ability to effectively handle unknown disturbances. Finally, while CBFs provide safety guarantees, methods relying solely on them still require the underlying policy to engage in substantial, risky exploration near safety boundaries in order to converge. For safety-critical systems like UAVs, this poses a prohibitive risk. Given these complementary strengths and unresolved challenges, a unified framework that can efficiently integrate these complementary approaches to realize safe and sample-efficient hover control for UAVs remains absent.

Crucially, existing Safe RL methods often rely on computationally intensive Quadratic Programming (QP) solvers for CBF enforcement, rendering them impractical for lightweight, resource-constrained platforms like the CoDrone [20] due to severe latency constraints. This gap necessitates a method that is simultaneously safe, sample efficient, and robust without demanding high computational power.

Therefore, in order to establish a robust control strategy, this study proposes a hybrid control architecture that integrates PPO-based RL, a PID expert policy [21], and a CBF safety filter. The framework is designed to jointly address policy learning efficiency, control stability, and flight safety.

Our main contributions are summarized as follows:

• Unified hybrid-DGRL framework: We propose a three-layer control architecture that integrates PPO-based adaptive control, a calibrated PID expert for DGRL, and a CBF module for formally grounded, real-time safety enforcement.
• Fundamentally novel DGRL and accelerated safe training: We introduce a DGRL mechanism that converts expert behavior into an intrinsic reward signal for PPO optimization. This design fundamentally differs from conventional imitation learning by avoiding approaches that rely on an explicit imitation loss, ensuring safe policy initialization and significantly accelerating convergence.
• Sim-to-real validation on a resource-constrained UAV platform: We deploy and validate the proposed hybrid-DGRL policy on a lightweight, resource-constrained CoDrone platform. Experiments under controlled airflow disturbance demonstrate that our method achieves superior hover stability, training efficiency, and safety guarantees compared with PID and PPO baseline controllers.
• Computationally feasible safety and non-interference: We detail the implementation of a simplified CBF-inspired Analytical Clipping Filter which, unlike resource-intensive QP solvers, provides effective safety bounds in real-time. This guarantees the UAV state remains within a predefined safe set during exploration while minimizing processing time and policy distortion.

2. Methods

The control architecture comprises three modules, as shown in Figure 1. These modules include a PPO policy (gray) that produces a nominal control action from real-time state estimates, a CBF safety layer (bule) that checks the feasibility of that action at every control cycle and minimally adjusts it to satisfy predefined altitude safety bounds, and a PID expert (green) that supplies demonstration data during training.

The signal flow is as follows. Sensor measurements are processed by a state estimator and sent to the PPO policy to generate a nominal action. That action is then filtered by the CBF to produce the final command applied to the vehicle. In parallel, the PID controller generates state–action pairs during training to support DGRL. The remainder of this chapter presents, in order, the policy learning module, the CBF safety filter, and the expert PID controller.

2.1. PPO-Based RL

To avoid the high risk and cost of RL from scratch on real hardware, we adopt a simulate then deploy workflow. The policy is first learned in a virtual environment and then transferred to a physical platform. Within this workflow, the policy learning module is built on PPO, which produces a nominal control action from state observations. We additionally introduce a CBF as a real-time safety filter that evaluates the feasibility of the nominal action. Only CBF filtered commands are ultimately issued to the vehicle. This design preserves PPO’s adaptability and performance potential while providing formal safety guarantees, enabling efficient learning in simulation. RL excels at sequential decision making under uncertainty and at balancing exploration with exploitation, and is therefore regarded as an effective approach for highly dynamic and complex control problems [22,23]. RL serves as the overarching methodological framework in our design, whereas PPO is the specific algorithm used to instantiate this framework in practice.

2.1.1. State Space and Action Space

For state design, four core elements are used as inputs to the PPO policy network. The state vector $s_t$ encompasses the core environmental states required for vertical control, as well as expert guidance signals:

$$s_t=[h_{error},v_{PPO},a_{PPO},{action}_{PID}].$$

(1)

The state features utilized include $h_{error}$, which is defined as the difference between the current altitude and the target altitude, $v_{PPO}$, which captures the dynamic trend and helps to prevent overshoot at high approach speeds [24], $a_{PPO}$, which conveys exogenous forces such as wind and enables anticipatory corrections, and ${action}_{PID}$, which is the output of the PID expert controller serving as a demonstration signal that guides PPO toward more stable and precise behavior. The first three elements describe the environment state and support prediction and adjustment, whereas the fourth leverages the stability and effectiveness of PID in UAV control [25,26,27] as the basis for demonstration-driven learning.

In this study, the agent’s action space is a one-dimensional continuous variable representing the vertical thrust command [22]. The action is normalized to the range [−1.0, 1.0]. Positive values indicate an increase in vertical thrust that drives ascent, whereas negative values indicate a decrease in thrust that drives descent. This design allows PPO to modulate thrust precisely and achieve accurate hovering control.

2.1.2. Reward Function Design

The environment reward is based on the distance between the UAV and the target altitude. Following the use of exponential factors for weighting velocity and acceleration terms in prior work [28], we apply a similar exponential factor to the distance, referred to as the Distance Decay Factor $e^{\left(-{\displaystyle \frac{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}}}\right)}$ (also called the Altitude Decay Factor), which is expressed as follows:

$$R_{env}=100e^{\left(-{\displaystyle \frac{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}}}\right)},$$

(2)

here, scale denotes the target value, and distance is the error between the current altitude and the target. This reward provides a dense learning signal that encourages the agent to approach the target progressively.

2.1.3. Policy Update Mechanism

The core mechanism for updating the PPO policy involves maximizing a specially designed surrogate loss function, which integrates a clipping technique to restrict the magnitude of policy updates. The objective function is defined as:

$$L^{CILP}\left(\theta \right)={\mathbb{E}}_t[\mathrm{m}\mathrm{i}\mathrm{n}(r_t(\theta ){\hat{A}}_t, clip(r_t\left(\theta \right),1-ϵ,1+ϵ){\hat{A}}_t)].$$

(3)

The components are defined as follows:

r_t(θ) denotes the probability ratio between the new policy ${\pi }_{\theta }$ and the old policy ${\pi }_{{\theta }_{old}}$ at time step $\mathrm{t}:$

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{old}}\left(a_t\right|s_t)}},$$

(4)

here, $s_t$ represents the state vector as defined in this study (detailed in Section 2.1.1), and $a_t$ is the action taken by the agent.

${\hat{A}}_t$ represents the advantage estimator. To balance the bias and variance of the estimation, this study employs Generalized Advantage Estimation (GAE), whose finite-horizon calculation based on the trajectory length $T$ is formulated as follows:

$${\hat{A}}_t=\sum _{t=0}^{T-t-1}{\left(\gamma \lambda \right)}^l{\delta }_{t+l}.$$

(5)

where $\gamma$ is the discount factor and $\mathit{\lambda }$ is the GAE smoothing parameter. This estimation relies on the Temporal Difference (TD) error ${\delta }_t$, defined as:

$${\delta }_t=R_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right),$$

(6)

here, $R_t$ denotes the immediate reward obtained at time step $t$, and $V\left(s\right)$ is the state-value function.

ϵ is a predefined hyperparameter that specifies the clipping range, ensuring the policy does not deviate excessively from the previous iteration.

In the implementation, the agent utilizes a Rollout Buffer Size ($n\_steps$) to collect samples before performing the policy update. The optimization process employs a learning rate ($\alpha$), and both the policy network and the value network are constructed according to a specific network architecture.

A complete list of all hyperparameters and their specific values used for the PPO training in this study can be found in Section 3.

PPO’s clipping mechanism constrains policy updates and prevents divergence [29], yielding superior convergence and robustness on continuous-control tasks compared with TRPO and A2C [4,30,31]. These properties make PPO particularly suitable for the hovering control problem considered here.

2.2. CBF Safety Filter

In deploying RL on small UAV, even a well-trained policy in simulation may issue unsafe actions under previously unseen real-world conditions. Because RL lacks built-in safety guarantees, we incorporate a real-time safety filter for the PPO agent to prevent accidents during physical deployment. Accordingly, we place a CBF mechanism at the output of the PPO policy.

Due to the demanding requirements for low end-to-end latency and high control frequency necessary for stable real-time operation of the resource-constrained CoDrone platform, this mechanism is implemented as a simplified, CBF-inspired Analytical Clipping Filter, and not as a complete QP solver. This choice prioritizes the minimization of processing time within the control loop.

The CBF safety filter is designed to prevent excessive descent that could lead to ground impact and to block ascent beyond a predefined ceiling. Therefore, the safe set is defined as:

$$S=\left\{h|h_{min}\le h\le h_{max}\right\},$$

(7)

here, h denotes the UAV’s instantaneous altitude, while $h_{min}$ and $h_{max}$ represent the minimum and maximum safety altitudes, respectively. To ensure that the UAV’s state remains within this safe set, we use the distances to the lower and upper boundaries, $h_{low}$ and $h_{high}$, to compute the safe velocity bounds.

$$h_{low}={h-h}_{min} ,{ h}_{high}=h_{max}-h.$$

(8)

The core principle of this Dynamic Constraint Mechanism is to ensure that the nominal action does not result in an immediate violation of the safety set [32]. This mechanism dynamically computes a safe velocity range based purely on the UAV’s distance from the safety boundaries and constrains the agent’s nominal action within this range. The safe velocity range is characterized by a downward velocity limit $v_{lower}$ and an upward velocity limit $v_{upper}$, which are calculated as follows:

$$v_{lower}\left(h\right)=\{\begin{array}{c}0.1, if {\displaystyle \frac{h_{low}}{h_{CBF}}}<0.1,\\ -{\displaystyle \frac{h_{low}^2}{h_{CBF}}}, otherwise,\end{array} v_{upper}\left(h\right)=\{\begin{array}{c}-0.1, if {\displaystyle \frac{h_{high}}{h_{CBF}}}<0.1,\\ {\displaystyle \frac{h_{high}^2}{h_{CBF}}}, otherwise,\end{array}$$

(9)

here, $h_{CBF}=h_{max}-h_{min}$ denotes the total altitude range, with $h_{low}={h-h}_{min} ,{ h}_{high}=h_{max}-h$.

This nonlinear relationship ensures that when the UAV is far from the boundaries, the constraint is loose, allowing for a wider range of control outputs and granting the system a large degree of control freedom. The filter minimally intervenes with the nominal action from PPO, allowing the vehicle to execute high-speed maneuvers. Conversely, when the UAV approaches the boundary, the constraint tightens rapidly, strictly limiting the UAV’s velocity toward the boundary and forcing the system to decelerate or move away from the boundary. Moreover, to guarantee immediate and stable corrections when the UAV is extremely close to a boundary, additional conditional checks are introduced. The positive and negative signs of the corrective term indicate the upward and downward directions, respectively.

Finally, the agent’s action is constrained by these dynamically computed safe velocity limits, yielding the final safe action clip($v_{lower}$, $v_{upper}$). The quadratic dependence yields loose constraints in the interior (larger allowable range) and rapidly tightening limits near $h_{min}$ or $h_{max}$. The hard caps ±0.1 (in normalized units) enforce immediate repulsion from the boundary by forbidding further motion toward it.

Through this mechanism, even if the PPO agent’s nominal output is potentially unsafe, the CBF filter applies real-time corrections prior to execution, ensuring that the UAV consistently operates within the defined safety boundaries.

2.3. DGRL

Early in training, especially when experience is limited, the RL agent’s exploratory behavior can produce unstable or even hazardous actions. This prolongs convergence and increases the risk of real-world deployment. Training a policy from scratch on a complex task is inefficient, which motivates the use of reliable demonstrations to accelerate learning.

To ensure methodological clarity, we explicitly distinguish and define two terms in this study. The term “demonstration” specifically refers to the actions generated by the PID expert in response to the system state, which serve as guidance signals for DGRL. In contrast, the term “experiment” refers exclusively to the process of evaluating the performance and safety of the fully trained final policy and all baseline controllers.

2.3.1. Expert Demonstrator and PID Tunings

Because PID controllers are widely used in UAV control [25,26,27] and valued for their stability and reliability, we adopt the Ziegler–Nichols (ZN) oscillation method [33] to automatically tune PID gains and provide a robust expert. ZN is an empirical procedure that does not require an accurate mathematical model yet yields practically useful PID parameters, making it suitable for nonlinear and uncertain aerial systems. The tuned PID serves both as an expert demonstrator for RL and as a baseline controller for comparison. Using PID outputs as demonstration signals helps PPO converge faster, mitigates large oscillations during early exploration, and improves overall training efficiency.

The ZN tuning procedure is conducted as follows. First, the integral gain ($K_i$) and derivative gain ($K_d$) of the PID controller are set to zero, leaving the controller to operate in proportional-only mode. This step allows for observation of the system’s response characteristics under proportional control.

Next, the proportional gain ($K_p$) is gradually increased while monitoring the UAV’s altitude response. The value at which the system transitions from a stable (decaying oscillation) response to sustained, constant-amplitude oscillations is identified as the critical gain ($K_{pu}$). While the system exhibits sustained oscillations, the time interval between two consecutive peaks (or troughs) is measured, this interval is referred to as the ultimate period ($T_u$).

Finally, using the obtained values of $K_{pu}$ and $T_u$ the final PID parameters are computed according to the classical formulas provided in the ZN tuning table,

$$K_p=0.6·K_{pu},K_i={\displaystyle \frac{1.2·K_{pu}}{T_u}},K_d={\displaystyle \frac{0.075·K_{pu}·T_u}{1}}.$$

(10)

The ZN-tuned PID controller plays a pivotal role in the RL training process. During the simulation phase, at each time step, the controller generates corresponding state-action pairs, which are used as expert demonstrations in DGRL.

2.3.2. Demonstration-Guided Reward Shaping

To further enable the PPO agent to effectively imitate the expert’s behavior, this study designs an Imitation Reward $R_{\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}$, which is defined as follows:

$$R_{\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}=-100e^{\left(-{\displaystyle \frac{∆\mathrm{Z}}{Z_{target}}}\right)}{{(action}_{PPO}-{action}_{PID})}^2,$$

(11)

where ${action}_{PPO}$ and ${action}_{PID}$ denote the actions of the PPO agent and the PID controller, respectively. The exponential term $e^{\left(-{\displaystyle \frac{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}}}\right)}$ serves as a decay factor, adjusting the penalty intensity according to the distance from the target. In this expression, $Z_{target}$ represents the target value, and $∆\mathrm{Z}$ is the absolute error between the real-time altitude and the target altitude. When the distance is large, the penalty is attenuated, encouraging the agent to maintain exploration. Conversely, as the UAV approaches the target, the penalty gradually increases, reinforcing the PPO agent’s imitation of the expert behavior. Consequently, the overall reward $R$ is defined as:

$${R=R_{env}+R}_{\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}.$$

(12)

This design is motivated by the desire to capture the precision and stability of a well-tuned PID controller while leveraging exploration to mitigate the inherent overshoot phenomenon of PID control [27], thereby achieving precise hovering. Specifically, we adopt the concept of balanced rewards from the Maximum Reciprocal Reward Method [28], using the distance decay factor to balance the Environmental Reward and the Imitation Reward. This approach promotes exploration when the UAV is far from the target, addressing the PID’s inherent overshoot, while strengthening precise control near the target to ensure hovering stability. Through this approach, the agent not only inherits the stable characteristics of the PID controller but also surpasses its limitations, demonstrating superior hovering performance.

Crucially, the PID expert in this study primarily serves as the demonstration provider. Its purpose is not to provide optimal control but rather to provide the UAV with “reasonable basic behavior” during no-wind disturbances. During training, the PPO agent refines the behavior induced by the demonstrations through the reward signal. Thus, even if the PID expert is not perfectly tuned, the RL policy can still acquire more robust control behavior through exploration and policy updates. Therefore, while demonstration quality affects early-stage guidance, it does not limit the final performance that PPO can achieve.

The DGRL architecture exhibits strong robustness to noise and imperfections in the demonstration data. This resilience stems from the nature of the expert data and the design of the learning mechanism. First, the demonstration data is synchronously generated in real-time during the RL training process, not pre-recorded offline. Since the PID expert operates within a simulated environment containing sensor and system noise, the demonstration data inherently carries practical, non-ideal noise characteristics, ensuring the PPO agent learns to handle real-world variations. Second, the PPO mechanism does not rely on rigid Behavioral Cloning. It learns a generalized, smooth policy through multiple mini batches. Crucially, the reward function’s reliance on the high-quality environmental reward allows the agent to correct suboptimal actions caused by noise in the imitation reward, thereby preventing the policy from overfitting to noisy data points. Finally, the CBF Analytical Clipping Filter (Section 2.2) acts as the ultimate execution-layer safeguard, preventing any potentially hazardous actions generated by the policy from being executed.

The policy is capable of learning decisions that surpass expert performance when conflicts arise between $R_{env}$ and $R_{\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}$. The agent can select an action that slightly deviates from the expert’s behavior but yields a higher cumulative long-term return while maintaining a favorable $R_{env}$, thereby overcoming the performance ceiling often imposed by the strict behavioral constraints of traditional Imitation Learning (IL).

3. Experimental Setup

This chapter details the specific experimental configurations and implementation parameters used for both the simulation training and the physical validation tests. All experimental results are presented in the subsequent Section 4.

3.1. Simulator and Real-World Platforms

To rigorously evaluate the proposed controller, we conduct experiments in both a high-fidelity simulation environment and a physical real-world testbed. The primary control objective in both domains is to maintain a stable hover at a target altitude of 100 cm.

Policy training is conducted in Microsoft AirSim v1.8.1 [34] using the default QuadX multirotor configuration, with key parameters including a total vehicle mass of $m=1.0 \mathrm{kg}$ and an arm length of $L=0.2275 \mathrm{m}$. We deliberately use this default model configuration to avoid reliance on finely calibrated physical parameters, a strategy intended to prevent policies from overfitting the simulator mechanics and failing to generalize to future real-world data.

AirSim is a high-fidelity, open-source simulator built on Unreal Engine that provides physically realistic environments and accurate aerodynamic models for a variety of aerial platforms, including quadrotor UAVs. Its robust physics engine and photorealistic rendering make it a valuable tool for bridging the simulation-to-reality (sim-to-real) gap, offering a safe, scalable, and reproducible testbed.

The simulation dynamics are governed by 6-Degree-of-Freedom (6-DOF) rigid-body dynamic equations, which calculate the translation and rotation of the UAV under the influence of propeller thrust and gravity. The lift generated by each propeller is modeled as proportional to the square of the angular velocity, while control moments for roll, pitch, and yaw are generated by adjusting the differential rotational speeds of the four rotors. To balance computational efficiency with realism, the simulator employs a simplified aerodynamic model that primarily accounts for fundamental gravity and thrust forces. Specifically, the dynamics model incorporates quadratic drag proportional to the square of the flight velocity magnitude. This drag formulation is consistent with the rigid-body physics modeling approach utilized by the official AirSim documentation.

In contrast to the simulation setup, the indoor validation is performed on a lightweight CoDrone platform. Despite its compact form factor, with dimensions of $130 \mathrm{mm}\times 130 \mathrm{mm}$ and a total weight of 37 g, the CoDrone is governed by the same fundamental 6-Degree-of-Freedom (6-DOF) rigid-body dynamics as the simulation model. To ensure consistency between the two domains, the control period ($∆t$) is uniformly set to 1 s for both the simulation training and the hardware implementation.

The experimental focus of this study centers on the robust control of vertical dynamics (Z-axis control). In all simulations and physical experiments, we simplified the learning space into a single-dimensional pure altitude control problem. Specifically, in the AirSim simulator, the policy only outputs the throttle command for the Z-axis (altitude). The X and Y axis velocities are constrained to zero during the control step, achieved by using the officially provided moveByVelocityAsync API with fixed zero values for the horizontal components. In the physical CoDrone experiments, the flight environment is constrained to minimize horizontal drift, focusing the controller’s task on compensating for vertical disturbances. Throughout all experiments, altitude safety bounds are enforced as [$h_{min}$, $h_{max}$] = [30, 200] cm via a closed-loop CBF safety layer to prevent boundary violations.

To balance generalization and comparability, we separate the training and evaluation settings while applying identical safety bounds in both phases. During the training phase, the focus is placed on maximizing model adaptability to unknown situations through parameter randomization. Specifically, the initial altitude for each flight is randomly set, sampled uniformly from 50 to 180 cm. The external disturbances we apply to the controller include two types. A sustained vertical wind, with a constant velocity randomly sampled from −100 to 100 cm/s and held throughout the entire flight. And transient vertical gusts, with a strong force randomly sampled from −300 to 300 cm/s, each lasting 3 s and triggered at random intervals of 5 to 15 s.

When conducting evaluation tests in simulated and actual environments, emphasis is placed on robustness and reproducibility. Therefore, to ensure reproducibility, the controller is rigorously tested in key scenarios. Conditions are fixed to ensure repeatability and to stress the controller in a critical scenario. The initial altitude is set to a state near 65 cm (sampled from U (60,70) cm) to jointly test ascent and stabilization.

To evaluate the robustness and stability of the proposed control algorithm under persistent external wind disturbances, a fixed wind interference experimental setup was established in an indoor environment. The experimental platform comprises a vertical metal stand with a small electric fan securely mounted via an iron rod at a center height of 150 cm, as shown in Figure 2. Additional details of the experimental setup are provided in the Supplementary Materials. This fixed configuration ensures that the position and angle of the wind source remain consistent across all trials, effectively avoiding the airflow instability and angle variations typically associated with handheld fans. Furthermore, the fan’s velocity was precisely measured and calibrated using an anemometer beforehand to ensure the physical disturbance settings were consistent with the corresponding parameters used in the simulation.

The UAV performs autonomous flight, navigating to the designated position directly beneath the wind field. Once the UAV stabilizes at a target altitude of 100 cm, maintaining a 50 cm vertical separation from the disturbance source. By maintaining fixed heights for both the fan and the UAV, this setup guarantees the consistency and repeatability of the disturbance conditions, allowing for a rigorous assessment of the system’s stability performance in a controlled environment.

3.2. Baselines and Implementation Details

The proposed hybrid control framework is designed to deliver end-to-end validation from simulation training to physical deployment, striking a balanced trade-off among learning efficiency, safety assurance, and practical deploy ability. It provides a concrete and reproducible solution for autonomous hovering of small UAV in uncertain environments. To assess its effectiveness, we compare against three representative baselines. These baselines include a classical PID controller tuned via the ZN method, serving as a robust control benchmark, a PPO policy, representing a learning-based baseline, and a PPO model augmented with a CBF safety filter, used to isolate the contribution of a formal safety mechanism under the same learning framework. All methods are implemented and evaluated under identical simulation settings and experimental procedures, with matched control frequency and safety bounds, to ensure fair comparison and reproducibility.

3.2.1. PPO Training Parameters

This study uses AirSim as the simulation platform for RL. All RL models are implemented and trained with the Stable-Baselines3 library [35], an open-source PyTorch-based toolkit that provides reliable implementations of widely validated RL algorithms. We adopt the PPO algorithm within SB3 to train the agent. All RL models are trained in AirSim for a total of 1,000,000 timesteps. To ensure statistical robustness, the simulation evaluations are repeated with N = 5 distinct random seeds, and the results are reported with 95% confidence intervals. The key PPO hyperparameters are summarized in

Table 1. Proximal Policy Optimization (PPO) Hyperparameter.

	Symbol	Value
Rollout Buffe Size	$n\_steps$	$512$
Learning Rate	$\alpha$	$3\times {10}^{-4}$
Clip Epsilon	$ϵ$	0.2
Discount Factor	$\gamma$	0.99
GAE	$\lambda$	0.95
Policy/Value Network Architecture	-	MLP: 64,64
Batch Size/Updates	Batch size	64

.

3.2.2. PID Parameters

As both a performance baseline and the source of expert demonstrations, the PID gains are tuned via the ZN procedure, yielding $K_p$ = 0.00984, $K_i$= 0.00384 and $K_d$ = 0.00631.

3.3. Performance Benchmarks and Strategy Nomenclature

In the subsequent quantitative evaluation (Chapter 4), we rigorously compare the performance and safety metrics of four distinct control strategies. To streamline the discussion and improve the readability of the results, we introduce the following simplified nomenclature for each tested strategy, replacing the verbose acronym combinations, as shown in

Table 2. Strategy Nomenclature.

Simplified Policy Naming	Full Configuration	Role in Evaluation
Hybrid-DGRL	PPO, PID, CBF	The framework proposed in this study integrates RL performance optimization, PID-based demonstrations, and formal safety guarantees.
PPO-Safety	PPO, CBF	The standard PPO policy, where actions are executed through a CBF safety filter.
PPO	PPO	The standard PPO baseline policy.
PID	PID (ZN-Tuned)	The classical control baseline.

. The definitions are strictly adhered to throughout the discussion.

4. Results and Discussion

This chapter provides a systematic evaluation of the proposed Hybrid-DGRL in both simulation and real-world Flight. We begin with a safety-aware hovering under gusts, conducting matched AirSim and indoor experiments with aligned initial conditions and disturbance profiles to assess sim-to-real consistency. We then describe a unified evaluation protocol and metrics, including convergence steps, steady-state root Mean Square Error (RMSE), standard deviation (SD) of Position Jitter, boundary violations during the training phase, Constraint Violation Rate (CVR) and time-to-recover (TTR), along with three baselines (PID, PPO, PPO-Safety). However, the boundary violations and CVR for the evaluation phase are not specifically discussed, as all controllers successfully stayed within safety boundaries and thus yielded incomparable results. An ablation study follows to quantify the marginal contributions of the CBF safety layer and the PID-based demonstrations. Simulation results are reported as mean trajectories with 95% confidence intervals, while real-world tests present representative flight traces and aggregated statistics. The two sets of results mutually corroborate one another.

4.1. Primary Validation: Safety-Aware Hovering Under Gusts

This section focuses on the primary validation of the proposed Hybrid-DGRL framework, specifically centering on safety-aware hovering under gusts in both simulation and real-world settings. We compared three baselines (PPO, PPO-Safety, PID) with the Hybrid-DGRL across six metrics, including convergence steps, steady-state RMSE, SD of Position Jitter, boundary violations, CVR, and TTR. We aligned the test scenarios between AirSim and indoor experiments under identical initial conditions and gust magnitudes to assess behavioral consistency and Sim-to-Real generalization. Finally, we provided flight trajectories for key scenarios together with aggregated statistics to complement the quantitative results.

Because confidence bands in time-series plots can become overly wide with small sample sizes, simulation results are shown as mean curves with 95% confidence intervals (N = 5), whereas real-world tests are illustrated with a single representative trajectory for the time history, and all trials are summarized in tables reporting RMSE, jitter standard deviation, TTR, and boundary-violation counts.

4.1.1. Flight Evaluation and Discussion in the Simulation Environment

We first examine flight trajectories (Z-axis altitude over time) in the simulation environment under identical initial conditions and gust profiles for four controllers. The focus is a qualitative comparison, highlighting differences in takeoff transients, steady-state hovering behavior, and post-disturbance recovery back to the 100 cm setpoint.

Figure 3 provides a comprehensive comparison of the Z-axis altitude trajectory tracking performance among four control strategies, the PID, PPO, PPO-Safety incorporating safety constraints, and the Hybrid-DGRL, under simulation environment conditions. The experimental setup involved applying a single, instantaneous wind gust disturbance between approximately t = 30 s and t = 33 s. This disturbance lasted for 3 s with a downward wind speed of approximately 380 cm/s, designed to examine the Disturbance Rejection and Recovery Capability of each controller.

In the phase prior to t = 30 s, the PID strategy exhibits typical undesirable transient behavior, with its trajectory showing significant overshoot and sustained ringing during the initial convergence. The minimum altitude briefly dropped below 75 cm, illustrating the limitations of classical PID controllers when dealing with nonlinear systems. In contrast, the three RL-based methods all demonstrate superior transient response, converging to the 100 cm target altitude more smoothly and rapidly. Among them, the proposed Hybrid-DGRL shows the smoothest and fastest convergence, indicating that the PID component, acting as a Demonstration Source, effectively improves the quality of the RL policy’s steady-state behavior.

During the gust disturbance period between approximately t = 30 s and t = 33 s, all controllers experienced a downward offset, but the PID strategy showed the largest displacement and the longest recovery time. The offset experienced by the PPO and PPO + CBF strategies was markedly smaller, demonstrating the superiority of RL policies in making adaptive decisions under complex environmental disturbances.

In summary, the Hybrid-DGRL achieves an optimal comprehensive balance across transient convergence speed, steady-state tracking accuracy, and disturbance rejection robustness by integrating the adaptive decision making of PPO with the high-quality demonstrations of PID.

As shown in Figure 4, the PPO agent rapidly elevates the UAV’s Z-axis altitude from approximately 65 cm to the 100 cm target altitude within the first few seconds of flight. This smooth ascent and strong initial tracking performance demonstrate the PPO algorithm’s ability to learn an efficient control policy. Subsequently, the UAV’s Z-axis oscillates around the target with only minor deviations, indicating that the PPO policy has successfully acquired a stable hovering strategy. Upon the onset of the wind gust at 30 s, the altitude briefly dips. Nevertheless, the policy quickly recovers and restores the UAV near the target altitude once the disturbance subsides, illustrating that the RL framework has endowed the agent with notable robustness and recovery capability.

The Z-axis altitude exhibits dense, high-frequency, small-amplitude fluctuations in Figure 5. Although the trajectory remains close to the 100 cm target for most of the time, the CBF’s real-time projection suppresses high-risk outputs, allowing the policy to focus on reward-driven accurate tracking. At the same time, small deviations trigger more frequent, low-magnitude corrections, which increases local control activity. Under gust disturbances, the system maintains boundary protection and returns to the target band rapidly.

Figure 6 illustrates the canonical behavior of a well-tuned PID controller. The UAV ascends rapidly from the initial altitude but exhibits overshoot and oscillations. After reaching the setpoint, the altitude trajectory continues to oscillate and then gradually settles. When subjected to wind disturbances, the controller tends to overreact, inducing additional oscillations. This indicates that the PID needs time to dissipate the accumulated error and reestablish stability. After stabilizing near 100 cm, the subsequent gust produces a noticeable short-term overreaction due to the lack of an adaptive mechanism. Both the recovery time and the residual oscillation amplitude are larger than those observed by the learning-based methods. Nevertheless, once disturbances subside, the steady-state performance is strong.

Overall, the figure reflects the classical trade-offs of PID control. These trade-offs are predictable start-up overshoot and ringing, followed by high precision and reliability under relatively stationary conditions.

Figure 7 illustrates the advantages of the Hybrid-DGRL. The UAV reaches the 100 cm target altitude within approximately 5 s while avoiding the initial overshoot of a PID controller. This demonstrates the effective collaboration between PPO and PID. In steady-state, the altitude remains close to the target for an extended period. Compared to the steady-state error [36] of PPO and the higher-frequency oscillations in the CBF method, the hybrid strategy exhibits smaller-amplitude, smoother micro-adjustments.

4.1.2. Indoor Real-World Flight Evaluation and Results Analysis

To assess feasibility and disturbance rejection in real conditions, we conducted indoor flight tests on a CoDrone platform. A downward wind disturbance of approximately 380 cm/s, matching the simulator setting, was applied for 3–5 s after the system had reached steady hovering. Evaluation focused on the Z-axis altitude response relative to the 100 cm target, including transient behavior and recovery quality.

Figure 8 presents a representative experimental trajectory that illustrates how PPO and Hybrid-DGRL behaved under a strong gust applied at about 27–33 s. The PID controller exhibited classic transient behavior with pronounced overshoot during takeoff (peak near 175 cm) followed by sizable oscillations, hence we applied the wind disturbance after the system stabilized at approximately 75 s. Under disturbance it showed the largest instantaneous deviation and the longest restabilization time. In contrast, both PPO and the proposed Hybrid-DGRL rose to the 100 cm setpoint more smoothly and more quickly, suppressing the unfavorable transients typically observed with ZN-tuned PID.

During the actual gust, the Hybrid-DGRL a smaller and well-controlled altitude deviation and returned to the vicinity of the setpoint rapidly. Although the time to enter and remain within the target band was slightly longer than that of PPO which may marginally affect the aggregate RMSE and standard deviation, the trajectory was visibly smoother with lower oscillation amplitude, indicating superior stability.

The PPO controller in Figure 9 ascends smoothly from approximately 65 cm during the initial phase, reaching the target around t ≈ 10 s without noticeable overshoot or large oscillations. When a strong wind disturbance is applied approximately 30 s, the altitude deviates significantly, dropping to a minimum of approximately 78 cm. However, the policy quickly initiates recovery, returning to the target setpoint by about t = 40 s, demonstrating the policy’s self-correction capability and steady-state robustness against sudden perturbations.

The PID controller in Figure 10 exhibits typical classic control behavior. The UAV rises rapidly at startup, showing a pronounced overshoot (peak about 175 cm), followed by sustained oscillations during convergence. The altitude briefly settles near the target before 65 s, but when a strong wind disturbance is applied approximately 75 s, the altitude drops abruptly to a minimum of approximately 87 cm. The controller compensates and returns to near 100 cm by approximately $t=100 \mathrm{s}$. Overall, while PID maintains good steady-state accuracy under quiescent conditions, its transient robustness and overshoot suppression are markedly inferior to the learning-based methods.

The Hybrid-DGRL controller in Figure 11 exhibits the fastest and smoothest takeoff response, reaching 100 cm within a short time while effectively avoiding initial overshoot. When a strong gust is applied at approximately 27 s, the altitude experiences only a minimal, transient deviation, dropping to approximately 83 cm at its lowest point. Post-disturbance oscillations remain small, with the altitude quickly sustained within the band of 96 cm and 104 cm for extended periods. This synergy enables low-amplitude oscillations and reliable boundary protection under real disturbances.

4.2. Training Efficiency and Stability (Convergence Curves)

This section evaluated training efficiency and stability using convergence curves. The Hybrid-DGRL model exhibited a faster and more stable decay of policy stochasticity and required fewer samples to reach a low-variance regime, while maintaining marked safety constraints. These results indicate that the PID demonstration effectively improved early policy quality and reduced cold-start instability. Furthermore, the CBF layer successfully constrained unsafe exploration without materially degrading learning progress.

Overall, the hybrid design achieved the best balance between sample efficiency, stability, and safety during training.

The resulting controller models from the training presented in Figure 12 are employed for evaluation. The horizontal axis denotes training steps, and the vertical axis reports the SD of the policy output. Policy SD serves as a key indicator of the transition from exploration to exploitation. A lower SD implies a narrower action distribution, more concentrated decisions, and a stronger tendency to execute actions deemed optimal by the agent.

In terms of trends, Hybrid-DGRL shows the fastest reduction and has nearly converged at around 200,000 steps. This suggests that incorporating a PID-based imitation signal enables the agent to acquire a low-variance, stable control strategy early in training. By contrast, PPO and PPO-Safety exhibit slower reductions in SD, indicating a higher training cost to reach performance comparable to the hybrid framework. In particular, PPO-Safety reduces SD more slowly at the beginning, likely because real-time suppression of unsafe actions by the CBF constrains early exploration. However, after a certain amount of training, it can still converge effectively, but the number of steps required is higher than the hybrid strategy.

Overall, combining CBF-based safety constraints with PID demonstrations simultaneously restricts hazardous exploration and mitigates cold-start instability, yielding faster and more stable learning. In other words, Hybrid-DGRL not only enters the exploitation phase earlier but also attains higher-quality convergence given the same training budget. These results support the practical value of demonstration-driven safe RL during pre-deployment training.

4.3. Ablation Study and Steady-State Performance

This section presents an ablation analysis that systematically compares three methods, PPO, PPO-Safety, and Hybrid-DGRL. To establish the overall baselines, we first summarize results from both the simulation environment and the physical UAV platform in

Table 3. Simulation Environment Ablation Study: Root Mean Square Error (RMSE), SD of Position Jitter, Time-To-Recover (TTR), Number of Safety Boundary Violations, Constraint Violation Rate (CVR), and Convergence Steps.

	PPO	PPO-Safety	PID	Hybrid-DGRL
RMSE (cm)	5.3707 ± 0.3689	5.7902 ± 0.9679	3.2774 ± 0.4187	4.7660 ± 1.2634
SD of Position Jitter (cm)	4.3671 ± 0.6893	5.5565 ± 1.1896	3.2687 ± 0.3956	4.3713 ± 1.0213
TTR(s)	4.1749 ± 3.5683	4.0892 ± 3.8723	6.2767 ± 3.9148	6.0031 ± 4.7395
Number of Safety Boundary Violations	$5.0510\times {10}^3$	$9.3500\times {10}^2$	N/A	$6.75\times {10}^2$
CVR (%)	0.50501	0.0935	N/A	0.0675
Convergence steps	$3.8298\times {10}^5$	$4.0499\times {10}^5$	N/A	$1.9661\times {10}^5$

and

Table 4. Three Metrics of Altitude Control Performance for Each Method in the Real-World Environment.

	PPO	PID	Hybrid-DGRL
RMSE (cm)	7.7982	$8.1079$	5.0104
SD of Position Jitter (cm)	4.1062	$8.0605$	3.4011
TTR(s)	4.5	N/A	2.34

. For simulation training, averages are reported over available episodes.

The results are then organized into three subsections. Section 4.3.1 reports simulation ablations, quantifying the marginal contributions of the safety layer and demonstration signals. Section 4.3.2 reports real-world ablations on the UAV, examining whether the same trends persist under physical gusts. And Section 4.3.3 provides a cross-environment synthesis that highlights consistencies and discrepancies and offers engineering interpretations.

Metric definitions are as follows. RMSE and SD are computed only from samples collected after the system first enters and remains within the ±5 cm target band, thereby focusing on steady-state precision and jitter during hovering. TTR is defined as the time required, after the gust is applied, for the UAV to return to and remain within the ±5 cm band, quantifying transient recovery quality. Boundary violations count events in which altitude leaves the predefined safe set, while CVR represents the percentage of total training steps in which such violations occurred. Convergence steps denote the number of training steps needed for the policy’s action standard deviation to fall below 0.05. This design cleanly separates steady-state versus transient behavior, safety versus learning efficiency, and enables a comprehensive comparison of long-horizon stability, post-disturbance recovery, safety, and training cost.

4.3.1. Ablation Study on Simulation Results

This subsection all evaluation metrics are reported as the mean and 95% confidence interval (CI) over multiple trials, with the training-related metrics (Convergence Steps and Boundary Violations, and CVR) specifically presented as mean values. The aggregated results are summarized in Table 3.

As shown in Table 3, steady-state performance reveals clear trade-offs among the controllers. The PID baseline attains the highest steady-state accuracy with RMSE 3.2774 ± 0.44187 cm and jitter SD 3.2687 ± 0.3956 cm, which indicates an inherent advantage for maintaining precision during relatively quiescent hovering.

In contrast, RL methods yield higher errors. PPO and PPO-Safety reach RMSE 5.3707 ± 0.3689 cm and 5.7902 ± 0.9679 cm, with jitter SD 4.3671 ± 0.6893 cm and 5.5565 ± 1.1896 cm. The similarity of these results suggests that adding CBF provides formal safety assurance without materially degrading steady-state tracking accuracy.

The proposed Hybrid-DGRL integrates the strengths of both paradigms. It achieves RMSE 4.7660 ± 1.2634 cm and jitter SD 4.3713 ± 1.0213 cm. Although slightly less accurate than the PID baseline, it clearly outperforms the RL methods, indicating that the learned policy and PID demonstrations complement each other in suppressing oscillations and improving stability.

For transient response, PPO and PPO-Safety recover fastest from gusts. Their time to recover is 4.1749 ± 3.5683 s and 4.0892 ± 3.8723 s. PID is slower at 6.2767 ± 3.9148 s, and the Hybrid-DGRL records 6.0031 ± 4.7395 s. The somewhat longer recovery of the hybrid arises because PID and CBF together enforce a more conservative return trajectory that prioritizes smooth and safe re-entry into the ±5 cm band rather than the shortest possible time.

Safety and learning efficiency also differ. In simulation training, in terms of safety, the introduction of the CBF yielded a substantial improvement. PPO exhibited $5.0510\times {10}^3$ (0.50501%) safety boundary violations during training, highlighting the risks associated with unconstrained exploration, which can hinder learning of correct and safe control strategies. In contrast, the number of out-of-bounds events for and decreased dramatically to $9.3500\times {10}^2$ (≈81.5% reduction) and $6.75\times {10}^2$ (≈86.6% reduction), respectively. This demonstrates that the CBF effectively filters unsafe actions in real-time, reducing high-risk sampling, while the PID demonstrations provide stable behavioral guidance, further accelerating the reduction in boundary-crossing incidents caused by wind disturbances. and terms of learning efficiency the hybrid requires the fewest convergence steps at $1.9661\times {10}^5$ compared with $3.8298\times {10}^5$ for PPO and $4.0499\times {10}^5$ for PPO-Safety. This indicates that the demonstration signal shortens the cold start and accelerates convergence.

Overall, when steady-state accuracy, disturbance recovery, safety violations, and training efficiency are considered together, the Hybrid-DGRL offers the most balanced and deployable option. It approaches the accuracy of PID, carries formal safety guarantees through CBF, and retains the robustness and generalization benefits of learning-based control, which makes it well suited to safety-critical operation under wind disturbances.

4.3.2. Ablation Study on Real-World UAV Results

In the indoor experimental setting, we followed the same testing protocol as in simulation and began collecting statistics only after the system first entered and maintained the 100 ± 5 cm target band. For comparability with the simulation results, this section reports a quantitative comparison using three metrics, with aggregated results summarized in Table 4. The design focuses on assessing deployability and robustness under real gust disturbances and sensor noise.

As shown in Table 4, Hybrid-DGRL delivers the strongest steady-state performance. Its altitude-control RMSE is 5.0104 cm, which is clearly lower than PPO at 7.7982 cm and far below the PID baseline at $8.1079$ cm. This indicates that the hybrid controller effectively reduces steady-state error under real disturbances.

For stability, the Hybrid-DGRL attains a position-jitter SD of 3.4011 cm, roughly on par with PPO at 4.1062 cm and markedly better than PID at $8.0605$ cm. Relative to PPO, the hybrid reduces jitter by about 0.7 cm. This small margin is likely due to fine-scale variations in the indoor wind field, differences in hardware thrust output and sensor noise, or boundary-near adjustments introduced by the safety layer.

Regarding transient recovery time (TTR), the Hybrid-DGRL achieved the fastest return to the setpoint, registering a TTR of 2.34 s. In contrast, the PPO strategy recorded a significantly longer TTR of 4.5 s, and the PID controller failed to re-establish steady-state operations within the measured duration (TTR: N/A). The superior and rapid recovery demonstrated by the hybrid framework is attributed to the synergistic effect of its components, where the PID element provides instantaneous and precise local actuation, and the CBF ensures that the rapid response remains within safe boundaries, thereby enabling the PPO policy to quickly execute an optimal recovery trajectory.

Overall, the hybrid approach successfully combines the adaptability of PPO with the stable demonstrations of PID, while CBF provides formal safety protection. Under real gusts and sensor noise, it reduces steady-state error and suppresses jitter, yielding the best overall deployability. In comparison, PID is more sensitive to unmodeled disturbances and therefore shows the largest steady-state error and jitter, while the hybrid model performs best in all metrics.

4.3.3. Cross-Environment Synthesis

By combining the ablation results from simulation (Table 3) and real-world flights (Table 4), we evaluated the sim-to-real transferability of the Hybrid-DGRL and the marginal contribution of each module. Two consistent findings emerged across both environments. First, the hybrid method delivered a stable advantage in steady-state accuracy. In the real platform, the hybrid controller achieved an RMSE of 5.0104 cm, which outperformed PPO at 7.7982 cm. In simulation, the hybrid RMSE was 4.7660 cm and again surpassed PPO at 5.3707 cm. These results indicate that the PID demonstration signal helped PPO refine steady-state tracking, so the hybrid achieved better nominal accuracy across environments.

Second, CBF and PID played complementary roles in safety and learning efficiency. In simulation, removing CBF, which turns the hybrid into PPO, increased safety-boundary violations from $9.3500\times {10}^2$ to $5.0510\times {10}^3$, an increase of about 440.2%, showing that CBF’s formal safety constraints effectively suppressed hazardous exploration. Removing PID, which turns the hybrid into PPO-Safety, increased the steps to convergence from $1.9661\times {10}^5$ to $4.0499\times {10}^5$, a delay of about 106.5%, showing that demonstrations shortened cold-start learning and improved sample efficiency. These two indicators could not be measured directly on hardware, yet the simulation evidence strongly supports the necessity of both modules.

Two notable discrepancies between simulation and reality are worth discussing. The first concerns the TTR metric. In simulation, the hybrid required 6.0031 s, which was longer than PPO at 4.1749 s. However, on the real platform, the results were inverted, PPO recovered in 4.5 s, which was significantly slower than the hybrid, which recovered in just 2.34 s. This inversion of performance highlights the superior generalization capability of the hybrid framework in complex real-world environments. The rapid recovery demonstrated by the hybrid arises because the Hybrid-DGRL framework successfully leverages the precise and rapid local corrections from the PID component. This synergy, combined with the CBF safety layer, enables the hybrid to achieve both minimal recovery time and guaranteed safety, making it significantly more favorable for deployment in complex physical systems than PPO.

The second discrepancy concerns the PID baseline. In simulation, PID achieved the lowest steady-state RMSE at 3.2774 cm and outperformed the hybrid. In the real environment, the hybrid achieved 5.0104 cm and clearly outperformed PID at $1.9730\times {10}^1$ cm. The gap is explained by unmodeled disturbances and hardware effects in the real setting that exceed the idealized assumptions in simulation. Stand-alone PID was sensitive to these factors and lost accuracy. The PPO component within the hybrid learned adaptive compensation for complex disturbances, which yielded much better steady-state accuracy than PID on the real platform and highlighted the robustness benefit of learning-based control.

Overall, the Hybrid-DGRL framework combined the robustness of RL, the sample efficiency of demonstrations, and the formal safety guarantees of CBF. Achieving the fastest TTR among all methods, it delivered the best balance of steady-state accuracy and safety on hardware and demonstrated strong sim-to-real transferability and deployment value.

4.4. Discussion

Hybrid-DGRL Across both simulation and real UAV systems experiments, the hybrid consistently outperformed the baselines in training efficiency, safety, control accuracy, and robustness, the relative ranking and qualitative trends are aligned between the two settings. Minor numerical deviations do not alter these conclusions and can be attributed to common sources of the Sim-to-Real gap, including sensor noise and latency, actuator nonlinearities and saturation, mismatches between fan-generated flows and the simulated wind model, battery voltage sag and inertial variations, and lower or variable control-loop rates and communication delays on hardware. The significant performance gains arise from the complementary roles of the three components.

During training, Hybrid-DGRL exhibits the fastest decay in policy variance, reaching a comparably low-variance with roughly half the steps required by PPO and PPO-Safety. Two factors drive this advantage, a PID-based imitation signal that provides a stable behavioral anchor during cold start, thereby accelerating convergence toward high-quality policies, and real-time action projection by the CBF, which suppresses hazardous exploration, substantially reduces safety violations, and improves overall learning safety.

In flight tests, the hybrid avoids the initial overshoot and ringing that are typical of PID control and maintains low RMSE and low jitter under sudden gusts, combining rapid recovery with smooth steady-state behavior. By comparison, PPO shows good recovery but somewhat higher steady-state jitter, whereas PID achieves strong steady-state accuracy in quiescent conditions yet is more sensitive to unmodeled dynamics and complex disturbances, resulting in inferior transient response and overshoot suppression.

5. Conclusions

This study presented and validated a deployable hybrid control framework that integrates PPO-, CBF-, and PID-based imitation to enable precise UAV hovering stability under controlled airflow disturbances.

Overall, the Hybrid-DGRL framework strikes a favorable balance between performance and deployability by leveraging the complementary strengths of adaptive learning (PPO), precise local adjustment (PID demonstrations), and formal safety enforcement (CBF). While the current validation is confined to quadrotor platforms and the corresponding flight conditions, this hybrid framework demonstrates strong potential for generalization and scalability to other safety-critical systems, provided the platform dynamics and safety constraints can be appropriately formulated for the CBF module. The framework successfully preserves consistent relative trends from simulation to UAV systems, effectively narrowing the Sim-to-Real gap and improving reliability and reproducibility under the controlled experimental conditions.

In summary, we successfully developed and validated this Hybrid-DGRL control framework. Physical deployment on the CoDrone platform demonstrated that the method provides superior stability, efficiency, and certified safety over traditional baselines, offering a robust and reproducible solution for autonomous hovering.

Future work will focus on three directions. We will first further narrow the sim-to-real gap by incorporating more realistic sensor noise, latency, and wind dynamics in simulation, combined with lightweight on-hardware fine-tuning to accelerate adaptation.

Second, the task scope will be expanded from the current single-axis altitude control to full 3D state control, including planar (X-Y) positioning and yaw control. Achieving highly reliable, robust, and precise positioning in the planar domain is a critical objective. This extension necessitates developing a more general CBF formulation capable of handling the coupled, nonlinear dynamics of lateral motion (e.g., pitch/roll constraints) while preserving a lightweight, real-time safety layer compatible with the computational limits of small platforms. Furthermore, we will investigate the application of this framework to safety-critical tasks such as obstacle avoidance and multi-UAV coordination.

Third, we will strengthen the generality and competitiveness of our approach. We plan to conduct a systematic investigation of hyperparameter choices, specifically quantifying the relationship between demonstration noise intensity and policy performance. Additionally, we will perform ablation studies and outdoor trials across a broader set of benchmarks and, critically, integrate and conduct comprehensive comparison evaluations against state-of-the-art Safe RL methodologies.