Lightweight Obstacle Avoidance for Fixed-Wing UAVs Using Entropy-Aware PPO

Abstract

Obstacle avoidance during high-speed, low-altitude flight remains a significant challenge for unmanned aerial vehicles (UAVs), particularly in unfamiliar environments where prior maps and heavy onboard sensors are unavailable. To address this, we present an entropy-aware deep reinforcement learning framework that enables fixed-wing UAVs to navigate safely using only monocular onboard cameras. Our system features a lightweight, single-frame depth estimation module optimized for real-time execution on edge computing platforms, followed by a reinforcement learning controller equipped with a novel reward function that balances goal-reaching performance with path smoothness under fixed-wing dynamic constraints. To enhance policy optimization, we incorporate high-quality experiences from the replay buffer into the gradient computation, introducing a soft imitation mechanism that encourages the agent to align its behavior with previously successful actions. To further balance exploration and exploitation, we integrate an adaptive entropy regularization mechanism into the Proximal Policy Optimization (PPO) algorithm. This module dynamically adjusts policy entropy during training, leading to improved stability, faster convergence, and better generalization to unseen scenarios. Extensive software-in-the-loop (SITL) and hardware-in-the-loop (HITL) experiments demonstrate that our approach outperforms baseline methods in obstacle avoidance success rate and path quality, while remaining lightweight and deployable on resource-constrained aerial platforms.

1. Introduction

Fixed-wing Unmanned Aerial Vehicles (UAVs) have emerged as key platforms with the development of the Low-Altitude Economy (LAE) [1] and the rise of Urban Air Mobility (UAM) [2]. Due to the extended endurance and high-speed characteristics of fixed-wing UAVs compared to rotary UAVs [3], they are especially preferred in applications such as long-distance cargo delivery [4], wide-area inspection [5], and emergency operations [6]. When a UAV is flying at low altitudes, autonomously avoiding potential obstacles, such as human-made facilities and terrains, becomes a key capability to guarantee its flight safety. Achieving low-altitude obstacle avoidance on a fixed-wing UAV is especially challenging due to its higher speed and inescapably large turning radius compared to other aerial platforms, as shown in Figure 1. It requires not only extended environment perception capabilities to realize early collision warning but also more flexible avoidance path planning subject to more rigorous dynamic constraints, to generate feasible avoidance trajectories.

Classic path planning methods predominantly rely on sampling [7], or optimization techniques [8], which necessitate comprehensive environmental perception, or rely on high-precision prior maps. The high-speed flight property of fixed-wing UAVs often makes it impractical to produce comprehensive maps or collect accurate environmental data of unknown scenarios [9]. Specifically, some scholars utilize costly sensors, such as LiDAR [10], or RADAR [11] to enhance environmental perception. However, these approaches not only increase system complexity and cost but also limit their applicability in scenarios with limited onboard resources. Thus, a lightweight navigation framework that removes the dependency on complete environment information or expensive sensors is imperative.

The visual sensor is a most widely used sensor in robot applications with its unparalleled low Size, Weight, and Power consumption (SWaP) and high sensory resolution. Compared to classic intelligent heuristic algorithms [12,13,14], DRL demonstrates unique strengths in tackling the obstacle avoidance problem with instantaneous visual information. More specifically, the success of DRL in gaming applications [15] has spurred interest in exploring its potential for visual avoidance [16], a domain where DRL further exemplifies its strength as an end-to-end learning framework [17,18]. However, strategic problems still need to be resolved for the integration of DRL into the fixed-wing UAV obstacle problem. Studies [19,20,21] have employed reward functions based on distance metrics to produce favorable results. However, these reward functions only consider target arrival and fail to account for path smoothness, which is particularly crucial for flight platforms such as fixed-wing UAVs. When considering the coupling optimization of visual information and obstacle avoidance, some studies use multi-frame depth maps [22] as the input of deep reinforcement learning collision avoidance, which undoubtedly increases the systematic latency and jeopardizes real-time performance, especially on edge computing platforms, which is a critical index for the high-speed fixed-Wing UAV. Besides the specific strategic problem, a persistent common challenge in utilizing DRL algorithms in robot applications is the imbalance between exploration and exploitation during training. Excessive focus on exploration can hinder algorithm convergence, while insufficient exploration may overlook superior solutions [23].

Keen to address the above challenges in using DRL to achieve fixed-wing obstacle avoidance, we propose a lightweight deep reinforcement learning framework that utilizes single-frame images captured by low-cost vision sensors as input and exploits the advantages of Proximal Policy Optimization (PPO) [24] to effectively address obstacle avoidance tasks, offering the following contributions:

• Considering the flight stability and dynamic constraints of fixed-wing UAVs, we propose an optimization framework formulated as an entropy-aware PPO learning model, which incorporates a reward function balancing target approach and path maintenance to ensure smooth and efficient collision avoidance flight trajectories.
• We introduce a strategy updating mechanism based on entropy-aware adjustment to address the challenge of local optimization caused by PPO’s reliance on historical data during training. This mechanism ensures that our algorithm identifies obstacle-avoidance strategies with higher success rates.
• We demonstrate that the proposed framework outperforms other methods in obstacle avoidance efficiency and flight path smoothness through software-in-the-loop and hardware-in-the-loop experiments, and we confirm the feasibility of running the algorithm on edge devices.

The remainder of this paper is organized as follows. Section 2 reviews related works, and Section 3 presents the problem definition and its mathematical formulation. Section 4 introduces the entropy-aware PPO. Section 5 discusses the computational experiments and their results. Section 6 concludes the paper.

2. Related Work

2.1. Fixed-Wing UAV Collision Avoidance

In contrast to obstacle avoidance algorithms for quadcopters, those for fixed-wing UAVs must account for complex dynamic constraints. For example, a fixed-wing UAV usually has narrower cruise velocity bounds, making it unable to change its velocity abruptly or hover in place like a quadrotor UAV.

Classic obstacle avoidance algorithms, such as Dijkstra [25] and A-star [26], are commonly used in static obstacle environments. However, these methods encounter significant challenges with local minima and often generate trajectories lacking smoothness, particularly in environments with closely spaced obstacles or narrow passages. Another class of algorithms, such as artificial potential field methods [27], RRT [28], and VFH [29], is more suitable for dynamic obstacle environments. These methods do not account for the dynamic constraints of fixed-wing UAVs, which often necessitate extensive post-processing to smooth the generated paths. To mitigate the reliance on post-processing, many researchers have adopted Dubins curves [30] for fixed-wing UAV path planning. Dubins curves employ a combination of straight-line and circular arc segments to generate paths that precisely satisfy the kinematic constraints of fixed-wing UAVs. In addition to Dubins-based planners, several recent works have explored the use of meta-heuristic optimization algorithms that incorporate kinematic constraints, such as maximum curvature, directly into their cost functions. These methods provide greater flexibility and solution diversity in complex and constrained environments, particularly when smooth path generation is crucial for fixed-wing UAVs. For example, ref. [31] proposed a constrained differential evolution algorithm tailored for UAV path planning in disaster scenarios, effectively embedding obstacle and curvature constraints into the planning process. Ref. [32] introduced a multi-strategy fusion differential evolution approach to enhance path feasibility in complex terrains. More recently, ref. [33] developed DE3D-NURBS, a 3D planner that integrates differential evolution with non-uniform rational B-splines (NURBS), enabling curvature-constrained and obstacle-aware trajectories in cluttered 3D spaces. These methods offer robust alternatives to traditional planners by accommodating both environmental complexity and vehicle-specific motion constraints.

Different from the above, which puts the obstacle avoidance idea in the planning layer, there are also a large number of studies that consider the obstacle avoidance module in the control layer, for example, approaches based on Model Predictive Control (MPC) [34] primarily use optimization theory or continuously update waypoints or routes to prevent collision. Nevertheless, this approach necessitates the creation of highly detailed models of aircraft, with the modeling process for aircraft with varying dynamics being inherently distinct. Consequently, the potential for generalization is limited.

The aforementioned methods face significant challenges when sensors provide only partial or incomplete environmental and obstacle information. To address this problem, numerous studies have focused on leveraging learning-based algorithms to solve obstacle avoidance under partially observed or unknown environmental conditions.

2.2. DRL for Visual Navigation

Deep reinforcement learning, a prominent subfield of machine learning, provides a unique advantage in facilitating interactive and adaptive learning within complex and uncertain environments. This capability makes it a preferred approach among researchers aiming to tackle intricate problems that traditional algorithms struggle to address effectively. From the initial deployment of table storage to address discrete state and action spaces, such as SARSA [35] and Q-learning [36], these techniques can effectively address problems with reduced complexity. Subsequently, the concept of value approximation in neural networks led to the development of Deep Q-learning (DQN) [37], Double DQN [38], and Dueling DQN [39]. These algorithms overcome the limitations of previous approaches, effectively enabling the handling of continuous state spaces. Dueling DQN, on the other hand, incorporates an advantageous function to assess the quality of an action within the dual network structure. In contrast to the aforementioned algorithms, which are based on the iterative updating of Bellman value functions, DDPG [40], SAC [41], TRPO [42], and PPO [24] are based on the theory of gradient descent. Among these options, PPO stands out for its stability, usability, and efficiency. The introduction of a clipping loss function serves to enhance the stability of the training process, limiting the magnitude of each policy update and thereby avoiding the potential for drastic policy changes. Compared to TRPO, PPO simplifies the training process by avoiding complex constrained optimization calculations while keeping policy updates within a safe range. In addition to excelling in benchmark tasks, PPO proves effective in addressing complex real-world problems. However, one limitation of PPO is its high data requirement for training and its heavy reliance on historical data, which may lead to excessive dependence on prior experience if early-stage learning data is insufficient.

The combination of DRL and visual information aims to optimize the navigation efficiency and performance of RL by improving the feature extraction and fusion of information from the perception side. Some researchers have utilized RGB images [43], area-segmented images [16], and depth images [22] to guide robots in optimizing navigation and obstacle avoidance strategies. Many works typically rely on comprehensive maps or collect accurate local maps; however, the high-speed flying nature of fixed-wing UAVs often makes it difficult to obtain comprehensive maps or gather accurate data in unknown environments where environmental information is impractical to acquire [16,44,45,46]. Other scholars have enhanced the generalization capabilities of DRL by incorporating human knowledge as prior information, enabling navigation in novel environments with sparse rewards [47]. While a substantial quantity of data can be augmented with human knowledge through supervised learning, the acquisition of strategic learning is frequently constrained by the methodologies employed for label generation. Conversely, the acquisition of human knowledge necessitates a considerable investment of effort.

In summary, integrating deep reinforcement learning with visual information presents a powerful approach to solving robot navigation and obstacle avoidance challenges. However, existing DRL-based navigation and obstacle avoidance algorithms often suffer from an imbalance between learning and utilization, resulting in prolonged convergence times, low efficiency, and suboptimal obstacle avoidance performance. To address this, our approach incorporates a self-regulating entropy mechanism to enhance reinforcement learning performance. Combined with a backpropagation reward mechanism, this approach significantly improves navigation efficiency in unknown obstacle environments for fixed-wing UAVs.

3. Methodology

3.1. Problem Formulation

The monocular vision-based obstacle avoidance problem can be modeled as a Markov Decision Process (MDP), which is characterized by a tuple. $\{\mathcal{S},\mathcal{A},\mathcal{P},r\}$. At time step t, the fixed-wing UAV collects environmental state variables $s_t$ using its camera. To ensure realistic flight behavior, the UAV’s forward velocity is constrained within $[v_{\min },v_{\max }]$, where $v_{\min }>$ 20 m/s, reflecting the physical limitations of fixed-wing platforms. The policy is trained under these bounds, preventing hovering or zero-velocity behavior that is infeasible in practice. Based on the state $s_t\in \mathcal{S}$, the UAV selects an action $a_t$ from the action space $\mathcal{A}$. The action $a_t\in \mathcal{A}$ interacts with the environment, generating a reward signal $r_t$ and resulting in a transition to the next state $s_{t+1}$. Given a forward speed $v_t$ and angular velocity ${\omega }_t$, this constraint translates to $\left|{\omega }_t\right|\le {\displaystyle \frac{v_t}{r_{\min }}}$. This effectively limits the maximum allowable heading change per time step and ensures that the action commands result in curvature-continuous, dynamically feasible trajectories. The objective of the algorithm is to find a policy that maximizes the cumulative reward ${\sum }_{t=0}^{\infty }{\gamma }^t·r_t$ by selecting actions $a_t$ that yield the highest expected return at any given time step t.

3.1.1. State Space

The state space encompasses the environmental data collected by the camera and the information regarding the target. This can be represented as

$$\mathcal{S}=\left\{{\mathcal{S}}^{\mathrm{env}},{\mathcal{S}}^{\mathrm{tar}}\right\}$$

(1)

where ${\mathcal{S}}^{\mathrm{env}}$ represents the environment captured by the camera, while ${\mathcal{S}}^{\mathrm{tar}}$ denotes the features related to the target. ${\mathcal{S}}^{\mathrm{env}}$ refers to a latent representation obtained from the encoder of an autoencoder network, designed to reduce redundant and adversarial information. In this context, the RGB image captured by the front-view monocamera ${\mathbf{i}}_{\mathrm{RGB}}$ is processed to extract depth information, as illustrated below

$${\mathcal{D}}_t={\Gamma }_{\mathrm{depth}}({\mathbf{I}}_{\mathrm{RGB}},{\theta }_{\mathrm{depth}}),$$

(2)

where ${\mathcal{D}}_t\subset \mathcal{D}\in {\mathbb{R}}^{H\times W}$ denotes a depth map with dimensions H (height) and W (width), and ${\Gamma }_{\mathrm{depth}}$ is the depth estimation model with parameter ${\theta }_{\mathrm{depth}}$ at time step t.

The latent representation is subsequently derived through the convolutional encoding of the current generated depth map. This process, at a given time step t, can be expressed as follows

$${\mathbf{f}}_t={\Gamma }_{\mathrm{enc}}({\mathcal{D}}_t,{\theta }_{\mathrm{e}}),$$

(3)

where ${\mathbf{f}}_t\subseteq \mathbf{f}\in {\mathbb{R}}^K$ denotes the latent variable of size K, while ${\Gamma }_{\mathrm{enc}}$ represents the encoding function parameterized by ${\theta }_{\mathrm{e}}$. Accordingly, ${\mathcal{S}}^{\mathrm{env}}$ is derived as follows

$${\mathcal{S}}^{\mathrm{env}}=\left[\mathbf{f}\right],$$

(4)

${\mathcal{S}}^{\mathrm{target}}$ represents a local goal, which can be expressed as follows

$${\mathcal{S}}^{\mathrm{target}}=[d,\alpha ],$$

(5)

where d and $\alpha$ represent the normalized relative distance and angle to the goal position, respectively. In this context, we consider a 2-dimensional coordinate system, where d is computed as follows

$$d={\displaystyle \frac{\parallel {\mathit{p}}_{\mathrm{target}},{\mathit{p}}_{\mathrm{ego}}{\parallel }_2}{d_{\max }}},$$

(6)

Here, $\parallel {\mathit{p}}_{\mathrm{target}}-{\mathit{p}}_{\mathrm{ego}}{\parallel }_2$ denotes the Euclidean distance between the UAV’s current position and the target. The term $d_{\max }$ represents the maximum allowable travel distance in the mission area, acting both as a safety constraint and a normalization factor. By dividing by $d_{\max }$, we ensure that the distance measure $d\in [0,1]$, which stabilizes reward scaling and improves training convergence. $\alpha$ is in radians and is calculated by

$$\alpha =arctan2\left({\displaystyle \frac{{\mathit{p}}_{\mathrm{target},y}-{\mathit{p}}_{\mathrm{ego},y}}{{\mathit{p}}_{\mathrm{target},x}-{\mathit{p}}_{\mathrm{ego},x}}}\right)/\pi ,$$

(7)

where x and y correspond to the longitudinal and lateral axes of the coordinate system, respectively.

3.1.2. Action Space

To adapt to the flight characteristics of fixed-wing UAVs, the action space is composed of waypoints in various directions within the body-fixed coordinate system under a constant altitude system, as well as the continuation of the action from the previous time step. This can be formulated as

$$\mathcal{A}=\left\{\begin{array}{cc}{\mathit{w}}_{t-1}\hfill & \mathrm{if}\mathrm{continue}\mathrm{last}\mathrm{action}\hfill \\ {\mathit{w}}_t\hfill & \mathrm{otherwise}\hfill \end{array},\right.$$

(8)

where ${\mathit{w}}_t$ represents the choosing waypoint and can be calculated as

$$(x_t^{body},y_t^{body})\stackrel{\Delta }{=}\lambda ·(cos\left({\Delta }_{\mathrm{yaw}}\right),sin\left({\Delta }_{\mathrm{yaw}}\right)),$$

(9)

where ${\Delta }_{\mathrm{yaw}}\in \left\{0,\pm {\displaystyle \frac{\pi }{6}},\pm {\displaystyle \frac{\pi }{4}},\pm {\displaystyle \frac{\pi }{3}}\right\}$ represents the discrete desired change in yaw angle magnitude and $\lambda$ represents the Euclidean distance between the calculated waypoint and the current position.

Remark 1.

In this study, the UAV control input is limited to the yaw angle, as the obstacle avoidance and navigation policy operate in a 2D horizontal plane. The UAV maintains a constant altitude throughout the mission, which is regulated by a separate altitude-hold controller. This design choice is informed by real-world deployment constraints of fixed-wing UAV autopilot systems, particularly PX4 and ArduPilot, both of which typically perform heading changes in discrete steps due to bounded turn rates and control limitations inherent in their autopilot design.

3.1.3. Reward Function

The design of the reward function remains one of the most significant challenges in DRL algorithms. A primary limitation of RL is that reward functions are typically hand-crafted and tailored to specific domains. There has been quite a bit of research on replay buffer, and most of the work provides a way to automatically obtain cost functions from expert demonstrations. However, these approaches are often computationally intensive, and the optimization required to identify a reward function that accurately represents expert trajectories is inherently complex. This paper focuses on designing a denser reward function to enhance the obstacle avoidance strategy, aiming not only to achieve high success rates in avoiding obstacles but also to enable smoother paths. Using the “knowledge” stored in the replay buffer for reward function design means that the agent’s reward is shaped not only by the immediate outcome of its current action but also by its contrast or divergence from past experiences. This encourages information gain, exploration of novel knowledge, and avoidance of redundant experiences or repetitive behaviors. Behavioral guidance toward aligning with more optimal past experiences. In the process of obstacle avoidance, this paper introduces a reward function that incorporates an inference mechanism to ensure robust learning under conditions of general applicability and rapid convergence. Specifically, we use the following four sub-reward terms: target-reaching reward $r_{\mathrm{target}}$, collision penalty $r_{\mathrm{collision}}$, distance incentive $r_{\mathrm{dis}}$, and trajectory-following reward $r_{\mathrm{track}}$. Each component is associated with a positive scalar weight $C_i$, where $\forall i\in \{1,2,3,4\}$, $C_i$ represents the weight of the i-th reward module.

When the drone reaches its designated target, it immediately receives a reward $r_{\mathrm{target}}$ defined as

$$r_{\mathrm{target}}=\left\{\begin{array}{cc}{C}_1& \mathrm{if}\mathrm{reaches}\mathrm{target}\\ 0& \mathrm{otherwise}\end{array}\right..$$

(10)

If the drone experiences a collision, it incurs a negative reward $r_{\mathrm{collision}}$ as a penalty defined as

$$r_{\mathrm{collision}}=\left\{\begin{array}{cc}-C_2& \mathrm{if}\mathrm{collision}\mathrm{happens}\\ 0& \mathrm{otherwise}\end{array}\right..$$

(11)

The drone should approach the target as quickly as possible, making it necessary to encourage the drone to be closer to the target at time t than at time $t-1$. The corresponding reward $r_{\mathrm{dis}}$ is defined as

$$r_{\mathrm{dis}}=\Delta d·C_3,$$

(12)

$$\Delta d=d_{t-1}-d_t,$$

(13)

where $d_t$ represents the relative distance between the drone and the target point at time t.

To encourage the UAV to move consistently toward the goal, we introduce a directional alignment reward based on the cosine similarity between the agent’s instantaneous motion vector and the direct vector pointing from its current position to the target. This reward does not rely on a pre-planned global path but instead promotes locally goal-directed behavior at each timestep. Therefore, we designed a reward function $r_{\mathrm{track}}$ that encourages the drone to follow the planned path while learning to interpret depth information to avoid obstacles. The corresponding reward $r_{\mathrm{track}}$ is defined as

$$r_{\mathrm{track}}=\delta ·C_4,$$

(14)

$$\delta ={\displaystyle \frac{{\mathit{p}}_{\mathrm{target}}·{\mathit{p}}_{\mathrm{ego}}}{{∥{\mathit{p}}_{\mathrm{target}}∥}_2·{∥{\mathit{p}}_{\mathrm{ego}}∥}_2}},$$

(15)

where $\forall i\in \left\{1,\dots ,4\right\},C_i>0$ represents the weight of each reward module. ${\mathit{p}}_{\mathrm{ego}}$ denotes the UAV’s current velocity vector, and ${\mathit{p}}_{\mathrm{target}}$ denotes the vector from the UAV’s current position to the target. A larger $\delta$ indicates that the UAV is better aligned with the target direction, thereby contributing to smoother and more efficient navigation. This formulation is particularly suitable for partially observable environments where global trajectories are unavailable.

To encourage the strategy to “move closer to high-quality experience” rather than merely maintaining the entropy of the action, we introduce the distribution of high-value experience in the replay buffer.

$$\begin{array}{cc}& r_{\mathrm{buffer}}\left(s\right)=-D_{\mathrm{KL}}\left({\pi }_E(·|s)\parallel {\pi }_{\theta }(·|s)\right)\\ & =-\sum _a{\pi }_E\left(a\right|s)log{\displaystyle \frac{{\pi }_E\left(a\right|s)}{{\pi }_{\theta }\left(a\right|s)}}\end{array}$$

(16)

where ${\pi }_E\left(a\right|s)\propto exp\left({\displaystyle \frac{r(s,a)}{\tau }}\right)$. $\tau$ is a temperature parameter controlling the sharpness of the softmax distribution over rewards with $\tau$ denoting a temperature parameter that controls the stochasticity of the expert policy. A smaller $\tau$ results in a more deterministic policy favoring high-reward actions, while a larger $\tau$ induces a flatter, more exploratory distribution. The reward is taken as the maximum weighted value over all experiences in the buffer. We construct a buffer with a priority queue structure, specifically to retain high-value and high-quality experiences.

The overall reward function is constructed by combining the four aforementioned sub-terms as follows,

$$r=r_{\mathrm{target}}+r_{\mathrm{collision}}+r_{\mathrm{dis}}+r_{\mathrm{track}}+r_{\mathrm{buffer}}.$$

(17)

Remark 2.

The smoothness of the generated path is promoted not through explicit analytical constraints, but through training under physically realistic dynamics and reward components that discourage abrupt directional changes. Quantitative curvature analysis confirms the emergence of smooth and feasible paths appropriate for fixed-wing flight.

4. Lightweight Obstacle Avoidance Using Entropy-Aware PPO

In this section, we design a novel entropy-aware PPO-based lightweight model to solve the fixed-wing UAV obstacle avoidance problem. The framework contains a lightweight backbone, an efficient strategy selection mechanism, and a new optimization objective function.

4.1. Overview

The overall framework is illustrated in Figure 2. A depth map is first generated from a monocular RGB image using ZoeDepth [48]. The resulting depth is encoded by a lightweight backbone [49] to extract visual features, which are concatenated with goal features and input into the policy network to predict actions. The critic network simultaneously estimates state values. An adaptive entropy module regulates the exploration-exploitation balance during training, while the replay buffer is continuously updated based on a tailored reward function to facilitate stable policy optimization. Our method is motivated by previous studies that examined the encoding of depth maps from multiple consecutive frames as state variables in reinforcement learning models for navigation and obstacle avoidance tasks. However, the storage and encoding of multiple frames of depth maps lead to high memory consumption and degrade the real-time performance of the system, rendering this approach unsuitable for fast-moving fixed-wing UAVs equipped with edge computing devices.

To address the challenge of increased memory consumption caused by stacking multiple depth maps and to alleviate potential generalization issues that arise in depth inference, we incorporated a fine-tuned monocular depth estimation model proposed by [48], which is proven to be reliable across a wide range of environments. By fine-tuning this depth model for our specific application, we are able to generate reliable enough depth maps for the following deep reinforcement learning module and at the same time reduce the computational burden of processing multiple depth frames. Additionally, one of our primary objectives was to ensure that the proposed architecture sustained computational efficiency, particularly when deployed on edge devices with limited processing capabilities. To this end, we integrated [49], a model specifically designed for efficient feature extraction, as part of our system architecture. Specifically, it improves feature extraction by performing element-wise multiplication between two linear transformation features, an operation inherently optimized for execution on Neural Processing Unit (NPU) architectures. NPUs are specifically optimized for matrix operations and parallel processing [50], making them particularly suitable for operations involving intensive linear algebra computations. By leveraging the compatibility between the feature fusion mechanism and NPU hardware, we achieved both high performance and low power consumption, which are essential for edge computing environments.

4.2. Strategy Selection Mechanism

In this section, we undertake a detailed examination of the factors that must be taken into account when selecting strategy mechanisms from two distinct perspectives.

4.2.1. Balance Exploration and Exploitation

The challenge of reinforcement learning lies in striking a balance between exploration and exploitation. An excess of exploration can result in situations where the algorithm fails to converge or converges slowly, whereas an excess of exploitation can lead to the disadvantage of local optimality. In the traditional PPO algorithm framework, it uses an importance sampling mechanism to train the model. More importantly, its assumed that the distributions of the training and learning models are consistent. However, the approach can lead to over-dependence of the data of the learning model on the merits of the training data when the data trained by the intelligences are not picked for good strategies, making the learning success rate decrease. To solve the problem, we design a new strategy selection mechanism.

4.2.2. Lowering Sensitivity to Prior Knowledge

When viewed through the lens of prior knowledge, the efficacy of conventional PPO algorithms, along with other deep reinforcement learning techniques, is markedly influenced by the data accumulated in previous iterations. In an effort to mitigate this reliance on prior knowledge and drawing inspiration from maximum entropy methods, we have devised policy mechanisms that are not only robust and stable but also exhibit rapid convergence through the use of self-tuning.

Definition 1.

The strategy entropy in the Markov process affects the balance between exploration and utilization, where for each state and action, the constraint is given as follows.

$$H\left(a\left|s\right.\right)=e^{{\displaystyle \frac{{\displaystyle \sum _t}{\gamma }^t{r}_t}{r_e}}}\left(-\sum _a{\pi }_{\theta }(s,a)log\pi (s,a)\right),$$

(18)

where $r_e$ denotes an expected value of reward. When the aforementioned entropy H is higher, there is a greater propensity for utilization. Conversely, when entropy is lower, there is a greater propensity for exploration.

Remark 3.

The previously mentioned strategy of entropy allows us to effectively address the challenge of balancing exploration and exploitation. However, in light of the necessity for simplified implementation and reduced computational complexity in engineering, there is a clear need for the development of more sophisticated entropy operators.

To make it learn under conditions that increase its success rate, we design a more generalized strategy entropy mechanism.

$$H\left(a\left|s\right.\right)={\displaystyle \frac{{\displaystyle \sum _t}{M}_s}{Batch}}\left(-\sum _a{\pi }_{\theta }(s,a)log{\pi }_{\theta }(s,a)\right),$$

(19)

where $M_s$ is the total number of successes in a $Batch$, and $Batch$ represents a set of data samples used when updating a policy or value function.

Lemma 1.

$H\left(\pi \right(s,a\left)\right)$ is η-smooth, equipped with the Taylor’s theorem, such that

$$\begin{array}{c}{∥\nabla H\left({\pi }_{\theta }(s,a)\right)-\nabla H\left({\pi }_{\theta }(s^{\prime },a)\right)∥}_{\infty }\hfill \\ \le \eta {∥{\pi }_{\theta }(s,a)-{\pi }_{\theta }(s^{\prime },a)∥}_{\infty },\hfill \end{array}$$

(20)

where η is a coefficient.

Theorem 1.

For any $k<N,k\in \mathbb{N}$, the entropy can be used as the sample mean as follows,

$$\begin{array}{c}\begin{array}{c}H\left({\pi }_{\theta }(s,a)\right)={\displaystyle \frac{1}{T}}{\displaystyle \sum _{i=0}^{T-1}}[\left(T-1\right)V^m{C}_k\hfill \\ {{∥{\pi }_{\theta }(s,a)-{\pi }_{\theta }(s^{\prime },a)∥}^m]}^{\lambda },\hfill \end{array}\hfill \end{array}$$

(21)

where $C_k={\left[{\displaystyle \frac{k!}{\left(k+1-\left[\lambda \right]\right)!}}\right]}^{\lambda }$ and $V^m={\pi }^{{\displaystyle \frac{m}{2}}}/({\displaystyle \frac{m}{2}}+1)!$ is the volume of the unit ball $\mathcal{B}(0,1)$ in ${\mathbb{R}}^m$. Furthermore, it holds that

$$\begin{array}{c}\underset{N\to \infty }{lim}{H}_N^k\left({\pi }_{\theta }(s,a)\right)=H\left({\pi }_{\theta }^{\mathcal{L}}(s,a)\right),\hfill \end{array}$$

(22)

where ${\pi }^{\mathcal{L}}(s,a)$ denotes the Lebesgue measure.

Proof. 

Equipped with Lemma 1, let ${\pi }^{\ast }=arg\underset{\pi \in \Pi }{max}\mathcal{H}\left({\pi }_{\theta }\left(s\right)\right)$

$$\begin{array}{c}H\left({\pi }_{\theta }^{\ast }(s,a)\right)-H\left({\pi }_{\theta }(s,a)\right)\hfill \\ \le \kappa exp(-T\eta )+2\beta \sigma +\zeta .\hfill \end{array}$$

(23)

Note that $H\left(s\right)$ is finite if ${\pi }_{\theta }\left(s\right)$ is of bounded support. Indeed, consider the imposed smoothing on $H\left({\pi }_{\theta }(s,a)\right)$, and we have

$$\begin{array}{c}H\left({\pi }_{\theta }(s,a)\right)\ge \underset{{\pi }_{\theta }\in \Pi }{max}H\left({\pi }_{\theta }(s,a)\right)-\sigma ,\hfill \end{array}$$

(24)

and

$$\begin{array}{c}{∥\nabla H\left({\pi }_{\theta }(s,a)\right)∥}_{\infty }\le \kappa =e^{{\displaystyle \frac{{\displaystyle \sum _t}{\gamma }^t{r}_t}{Exp\left(R\right)}}},\hfill \end{array}$$

(25)

where $T\ge 10\zeta \kappa log10\zeta$.

Hence, $H\left({\pi }_{\theta }(s,a)\right)$ tends to be the support of ${\pi }_{\theta }^{\mathcal{L}}(s,a)$. This concludes the proof.    □

4.3. Entropy-Aware PPO Method

Based on the preceding analysis of entropy and its role in reinforcement learning, we developed an optimized Proximal Policy Optimization (PPO) framework that incorporates entropy-aware regulation. By dynamically adjusting the entropy coefficient during training, the proposed method effectively balances the trade-off between exploration and exploitation. This adaptive mechanism enhances policy robustness, improves convergence stability, and ensures safer decision-making, which is particularly critical for lightweight obstacle avoidance in fixed-wing UAVs. The loss function of the traditional PPO algorithm is mainly based on the advantage function. To improve the universality of the algorithm, an advantage function based on the reward function in Equation (17) is designed to make the algorithm a closed loop.

$$\begin{array}{c}{A}_{\theta }\left(s_t,a_t\right)\stackrel{\Delta }{=}{Q}_{\theta }\left(s_t,a_t\right)-V_{\theta }\left(s_t\right)\hfill \\ ={\mathbb{E}}_{s_t,a_t}\left({\displaystyle \sum _l}{r}_{t+l}\right)-{\mathbb{E}}_{s_t}\left({\displaystyle \sum _l}{r}_{t+l}\right),\hfill \end{array}$$

(26)

where θ is the vector of policy parameters before the update, and $Q_{\theta }\left(s_t,a_t\right)$ and $V_{\theta }\left(s_t\right)$ are the inferring action-value function and inferring value function, which can be obtained with the reward function in Equation  (17). Therefore, $A_{\theta }\left(s_t,a_t\right)$ is an inferring advantage function at time step t.

In the PPO algorithm, the importance sampling mechanism is used to control the updating range of the policy while optimizing the policy so as to avoid the problem of drastic changes caused by excessive updating. However, during the training process, we usually use some old strategies that have been trained to collect samples, rather than using the latest strategies that are currently available. This leads to the problem of mismatch between the sample and the current policy, which is called “policy offset”. In order to ensure the exploration of better solutions when the distribution gap between the two datasets is large, we no longer assume that the distribution of the old and new strategies is similar, and we encourage the exploration of new strategies when the distribution difference between the old and new strategies is large. Therefore, this paper explores and utilizes data considering the distribution of old and new strategies.

$$\begin{array}{c}\begin{array}{c}{\mathbb{E}}_{(s_t,a_t)\sim {\pi }_{\theta }}\left[A_{\theta }\left(s_t,a_t\right)\nabla log{\pi }_{\theta }\left(a_t^n\left|s_t^n\right.\right)\right]\hfill \\ ={\mathbb{E}}_{(s_t,a_t)\sim {\pi }_{{\theta }^{\prime }}}[{\displaystyle \frac{{\pi }_{\theta }\left(a_t\left|s_t\right.\right)}{{\pi }_{{\theta }^{\prime }}\left(a_t\left|s_t\right.\right)}}{\displaystyle \frac{{\pi }_{\theta }\left(s_t\right)}{{\pi }_{{\theta }^{\prime }}\left(s_t\right)}}{A}_{\theta }\left(s_t,a_t\right)\hfill \\ \nabla log{\pi }_{\theta }\left(a_t^n\left|s_t^n\right.\right)],\hfill \end{array}\hfill \end{array}$$

(27)

where ${\theta }^{\prime }$ represents the vector of policy parameters after the update, ${\pi }_{\theta }$ and ${\pi }_{{\theta }^{\prime }}$ are the old and new strategies, respectively.

In traditional PPO implementations, the process of training is influenced by a fixed hyperparameter which determines the exploration magnitude. In this paper, a new PPO method with an inferring reward mechanism and adaptive entropy is introduced, which incorporates a dynamic scaling of the entropy coefficient based on the recent return obtained by the agent. Based on the above discussion, the final loss function can be written in the following form,

$$\begin{array}{c}{L}_{\theta }^{CLIP}={\mathbb{E}}_{(s_t,a_t)\sim {\pi }_{\theta }^{\prime }}[\min \left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\left|s_t\right.\right)}{{\pi }_{{\theta }^{\prime }}\left(a_t\left|s_t\right.\right)}}{\displaystyle \frac{{\pi }_{\theta }\left(s_t\right)}{{\pi }_{{\theta }^{\prime }}\left(s_t\right)}}{A}_{\theta }(s_t,a_t),\right.\hfill \\ \hfill \left.clip\left({\displaystyle \frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }^{\prime }}(a_t\mid s_t)}},1-\epsilon ,1+\epsilon \right)A_{\theta }(s_t,a_t)\right)],\end{array}$$

(28)

$$\begin{array}{c}{L}_{\theta }^{VF}={\left(V_{\theta }\left(s_t\right)-V_t^{\mathrm{target}}\right)}^2,\hfill \end{array}$$

(29)

$$\begin{array}{c}{L}_{\theta }^{ENT}={\mathbb{E}}_{(s_t,a_t)\sim {\pi }_{\theta }}\left[H\left({\pi }_{\theta }\left(a_t\right|s_t)\right)\right],\hfill \end{array}$$

(30)

$$\begin{array}{c}{L}^{EA}=L^{CLIP}-w_1{L}^{VF}+w_2{L}^{ENT},\hfill \end{array}$$

(31)

$${\mathcal{L}}_{\mathrm{total}}=L^{EA}+\lambda ·{\mathbb{E}}_{s\sim \mathcal{B}}\left[\sum _a{\pi }_E\left(a\right|s)log{\pi }_{\theta }\left(a\right|s)\right]$$

(32)

where $\mathcal{B}$ is the replay buffer, i.e., a set of collected experiences; $(s,a)$ represents a sampled experience from the buffer consisting of the state and action. $w_1$ weights the policy entropy term, which encourages exploration by penalizing low-entropy (i.e., overly deterministic) policies; $w_2$ weights the extrinsic reward term, which guides the agent toward task-specific goals such as obstacle avoidance and target reaching.

$${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{total}}={\nabla }_{\theta }{L}^{EA}+\lambda \sum _{s\in \mathcal{B}}\sum _a{\pi }_E\left(a\right|s){\nabla }_{\theta }log{\pi }_{\theta }\left(a\right|s).$$

(33)

It can be seen from the gradient that the guiding strategy approaches the frequent actions in the buffer. When the strategy and the replay buffer are very close, the gradient approaches 0 and will not interfere with the normal optimization of the PPO algorithm. When the current strategy moves away from the buffer, it will push the strategy parameter θ towards the buffer, that is, it will imitate the actions that frequently occur in experience. When the initial tendency of the strategy is to explore, buffer experience can accelerate convergence and avoid getting trapped in random or inefficient behaviors. The pseudo-code of the algorithm can be found in Algorithm 1.

Algorithm 1 Entropy-aware PPO for UAV obstacle avoidance

Initialization: Initialize actor network ${\pi }_{\theta }$, critic network $V_{\varphi }$.

Initialize high-quality experience buffer $B_{\mathrm{high\_quality}}$.

Define environment and parameters: max_episodes, max_timesteps.

Define hyperparameters: learning rate α, discount factor γ, PPO clip ϵ, weights $w_1,w_2$, guidance coefficient λ.

Load pre-trained lightweight depth model ${\Gamma }_{\mathrm{depth}}$ and encoder ${\Gamma }_{\mathrm{enc}}$.

for episode = 1 to max_episodes do

Reset environment and UAV position $p_{\mathrm{ego}}$; clear episode buffer $B_{\mathrm{episode}}$.

for timestep t = 1 to max_timesteps do

Capture RGB image $I_{\mathrm{RGB}}$; compute depth map $D_t={\Gamma }_{\mathrm{depth}}\left(I_{\mathrm{RGB}}\right)$.

Encode visual features $f_t={\Gamma }_{\mathrm{enc}}\left(D_t\right)$.

Get relative distance d and angle α to the target; form state $s_t=[f_t,d,\alpha ]$.

Get action probability and value: $a_t\sim {\pi }_{\theta }\left(s_t\right)$, $V_{\varphi }\left(s_t\right)$.

Execute $a_t$; observe $s_{t+1}$, collision status $c_{\mathrm{status}}$, goal status $g_{\mathrm{status}}$.

Compute rewards:

$r_{\mathrm{target}}=C_1$ if target reached, else 0;

$r_{\mathrm{collision}}=C_2$ if collision, else 0;

$r_{\mathrm{distance}}=C_3·(d_{t-1}-d_t)$;

$r_{\mathrm{track}}=C_4·cos\left(\mathrm{angle}(p_{\mathrm{ego}},p_{\mathrm{target}})\right)$

Total reward: $r_{\mathrm{total}}=r_{\mathrm{target}}+r_{\mathrm{collision}}+r_{\mathrm{distance}}+r_{\mathrm{track}}$

Store $\{s_t,a_t,r_{\mathrm{total}},log{\pi }_{\theta }\left(a_t\right|s_t)\}$ in $B_{\mathrm{episode}}$

if $r_{\mathrm{total}}$ is high or successful transition then

Add transition to $B_{\mathrm{high\_quality}}$

end if

if collision, target reached, or timestep limit then

break

end if

end for

for $k=1$ to $K_{\mathrm{epochs}}$ do

Compute advantage estimates $A_t$ using GAE

Compute adaptive entropy: $$H\left(a\right|s)={\displaystyle \frac{M_s}{B}}·\left(-\sum {\pi }_{\theta }\left(a\right|s)log{\pi }_{\theta }\left(a\right|s)\right)$$

PPO clipped loss: $$L^{\mathrm{CLIP}}=\mathbb{E}\left[\min ({\rho }_t{A}_t,\mathrm{clip}({\rho }_t,1-ϵ,1+ϵ)A_t)\right]$$

Value function loss: $L^{\mathrm{VF}}={(V_{\varphi }\left(s_t\right)-V_{\mathrm{target}})}^2$

Entropy loss: $L^{\mathrm{ENT}}=\mathbb{E}\left[H\left(a\right|s)\right]$

Replay buffer imitation loss: $L^{\mathrm{BUFFER}}=D_{\mathrm{KL}}({\pi }_E\parallel {\pi }_{\theta })$

Total loss: $${\mathcal{L}}_{\mathrm{total}}=L^{\mathrm{CLIP}}-w_1{L}^{\mathrm{VF}}+w_2{L}^{\mathrm{ENT}}-\lambda L^{\mathrm{BUFFER}}$$

Update θ, ϕ; using Adam on ${\mathcal{L}}_{\mathrm{total}}$

Clear $B_{\mathrm{episode}}$

end for

end for

Output: Trained actor policy ${\pi }_{\theta }$

5. Experimental Validation

To evaluate its effectiveness, three experimental setups are designed in this section. First, we design an ablation experiment to separately demonstrate the impact of the designed reward function and the update mechanism. Second, we demonstrate the superiority of the proposed method through comparisons with other deep reinforcement learning algorithms. Finally, a hardware-in-the-loop simulation experiment is conducted to verify the deployment capability of the proposed framework on edge devices, while also comparing it with classic sample-based methods.

5.1. Training Settings

We conduct the training on a machine equipped with an Intel Xeon E5-2678 V3 CPU and two NVIDIA RTX 3090 GPU. A high-fidelity simulator, AirSim (v 1.8.1) [51] build on Unreal Engine (v UE4.27), is employed to build the different environments and provide data including RGB images captured by its camera and fixed-wing UAV’s position. The fixed-wing UAV’s dynamics model is provided by JSBSim (v 1.2.0) [52], an open source platform widely regarded for its high accuracy in modeling aerodynamics and flight physics. The specific model used in this work is the Skywalker X8 which is from Skywalker Technology Co., Ltd., Shenzhen, China, a popular choice for its stability and versatility in various flight scenarios. The neural network models are established using the PyTorch (v 3.6) framework.

We conduct the training of the proposed method with the parameters shown in

Table 1. Common parameter settings for PPO and proposed algorithm.

Parameter	Value
Air Speed (m/s)	30
Depth Map Size ($H,W$)	224, 224
Reward Term Weight ($C_1,-C_2,C_3,C_4$)	30, −30, 0.5, 1.0
Flying Distance Cap $d_{max}$ (m)	1300
Learning Rate	0.0003
Gamma ($\gamma$)	0.95
Clip Range ($ϵ$)	0.3
K Epochs	2
Batch Size	2048
Value Loss Coefficient ($w_1$)	0.5
Entropy Loss Coefficient ($w_2$)	0.1
Max Timesteps Per Episode	60
Max episodes	3000
State Dimension	256
Action Dimension	8

within a 1000 m by 600 m rectangular urban environment constructed using UE. In the experiments, target points are randomly selected from three predefined flight paths. The parameters listed in Table 1 were selected based on a combination of UAV platform constraints, established practices in the reinforcement learning literature, and empirical tuning. Specifically, the air speed and maximum flight distance ($d_{\max }$) were set to reflect the operational characteristics of high-speed fixed-wing UAVs commonly reported in previous studies [1,53]. The input depth map resolution ($224\times 224$) is consistent with widely used visual encoders, providing sufficient spatial information while maintaining computational efficiency. ($C_1,C_2,C_3,C_4$) were initially determined through heuristic design and then refined via ablation studies. These weights were tuned to ensure a balanced influence of goal-reaching incentives, collision penalties, heading smoothness, and directional alignment. The PPO hyperparameters, including learning rate, discount factor (γ), clip range (ϵ), batch size, and epoch number, were initialized using standard values reported in prior works [24]. A light grid search was conducted to improve convergence speed and policy stability in our environment. Coefficients for value loss ($w_1$) and entropy regularization ($w_2$) were selected to balance policy exploration and value fitting. Overall, this combination of physically realistic parameters and principled tuning contributes to the stable and effective training of the proposed navigation policy.

As shown in Figure 3, varying numbers of obstacles are distributed along the three flight paths. The variation in obstacle density across the routes simulates the challenge of avoidance faced by fixed-wing UAVs in environments with different levels of obstacle density. Image data are collected using a simulated camera provided by AirSim, generating color images with a resolution of 480 × 640 for the depth estimation module.

5.2. Ablation Studies

In this section, we study the importance of various design modules in our framework.

5.2.1. Inferred Reward Function

To evaluate the contribution of the reward structure to trajectory quality and overall performance, we conduct both visual and quantitative analyses of the designed reward modules. Beyond the commonly used distance-based term $r_{\mathrm{target}}$, $r_{\mathrm{collision}}$ and $r_{\mathrm{dis}}$, our framework introduces two additional components: the trajectory alignment reward $r_{\mathrm{track}}$ and the backtracking penalty $r_{\mathrm{buffer}}$, which are critical for ensuring directional stability and forward motion efficiency, respectively.

Table 2. Ablation study of $r_{track}$, $r_{buffer}$, and entropy-aware loss. ✓ indicates the module is used; – indicates not used. Bold values indicate the best performance in each column.

Scheme	${\mathit{r}}_{\mathbf{track}}$	${\mathit{r}}_{\mathbf{buffer}}$	Entropy-Aware Loss	Collision Rate	Fail Reach	Episode Length
A (baseline)	–	–	–	73%	16%	69.8
B	✓	✓	–	17%	15%	151.4
C	✓	–	–	13%	21%	173.7
D	–	✓	–	9%	11%	130.4
E (complete)	✓	✓	✓	5%	9%	121.6

presents the results of an ablation study where each of these modules is removed in turn. When the $r_{\mathrm{track}}$ term is excluded, the UAV shows a drastic decline in reach rate (from $81\%$ to $21\%$) and a sharp increase in episode length, indicating frequent detours and unstable trajectories. This suggests that without directional guidance, the UAV struggles to maintain course alignment even if it avoids collisions. Similarly, removing the $r_{\mathrm{buffer}}$ term leads to degraded performance, as reflected in both reach rate (dropping to $16\%$) and longer flight durations. This highlights the importance of penalizing regressions or inefficient backtracking during navigation. Due to the large performance gap caused by different reward configurations, plotting the results as curves would result in poor visual comparability. Therefore, we adopt a table format to clearly illustrate the impact of each reward component on overall behavior. Collectively, these results validate the effectiveness of incorporating both $r_{\mathrm{track}}$ and $r_{\mathrm{buffer}}$ into the reward formulation, enabling the UAV to produce more stable, efficient, and dynamically feasible trajectories.

We evaluate the contribution of the designed reward function to the smoothness and stability of flight trajectories. First, we employ only the distance reward function, assigning it a weight of $C_3$ = 1 and referring to this configuration as the distance model. Subsequently, we compare the smoothness of the flight trajectories generated by the distance model and the proposed model along three predefined flight paths. As illustrated in Figure 4b,d,f, the stability that is observed in the proposed model’s trajectories can be attributed to the carefully designed reward function, which balances multiple factors such as obstacle avoidance and path smoothness. In contrast, the trajectories generated by the model using only $r_{\mathrm{dis}}$ exhibit more abrupt directional changes, as highlighted by the jagged red solid lines in Figure 4a,c,e. Such rapid course corrections can impose additional strain on the fixed-wing UAV’s control system, which can lead to potential instability, particularly in complex urban environments. By incorporating a smoothness criterion into the reward structure, the proposed model effectively reduces the necessity for drastic course adjustments, enabling the UAV to follow a more fluid and consistent path.

This improvement in flight stability is critical in real-world scenarios, where maintaining smooth trajectories helps minimize energy consumption and ensures safer navigation through dynamic and uncertain environments. Thus, the results clearly demonstrate the superiority of the proposed reward function in producing smoother and more stable flight paths for fixed-wing UAVs, ultimately enhancing overall flight performance in obstacle-rich environments.

5.2.2. Adaptive Entropy

To evaluate the impact of the proposed Adaptive Entropy on the obstacle avoidance tasks, we design a strategy comparison experiment. We, respectively, train the PPO models with entropy weights of 0.01 and 0.001, and then compared their rewards per episode during training with the proposed method. The rewards obtained during training are shown in Figure 5. The lower weight model shows a gradual improvement in performance, starting with relatively low rewards and steadily increasing as the training progresses, indicates moderate variability, suggesting consistent learning behavior. The higher weight, while starting at a lower initial reward, exhibits a steady increase over time, indicating higher variability in performance, particularly in the earlier stages of training. Our proposed method demonstrates the fastest learning curve, with cumulative rewards increasing rapidly in the early episodes. By the end of the training, it converges at the highest cumulative reward. The shaded area around the orange curve is relatively narrow, indicating low variability and suggesting that the proposed method is more stable and consistent across different episodes.

These results indicate that the proposed method outperforms PPO model with different entrophy weights in terms of cumulative rewards, demonstrating its effectiveness in navigating the fixed-wing UAVs through environments with obstacles. Additionally, the faster convergence of the proposed method shows its potential for quicker policy learning, making it suitable for practical applications where rapid adaptation is critical.

5.3. Policy Comparison

To evaluate the effectiveness of our proposed fixed-wing UAV obstacle avoidance method, we conducted tests in three distinct scenarios named Scene I (City), Scene II (Line-cruising), and Scene III (Valley).

We compare our proposed method with several established reinforcement learning algorithms, including PPO, TRPO, A3C, DQN, and DDPG. All algorithms are tested in the same initial simulation environment, with repeated trials in each scenario to ensure robust results. In each scenario, the agent’s task is to fly from a starting position to a target without colliding with any obstacles. The performance of each algorithm is measured using the task completion rate (Success Rate, %), which represents the percentage of trials in which the agent successfully completed the task. Each test is repeated 100 times per scenario to minimize the impact of randomness on the results.

Table 3. Task completion results of different obstacle avoidance strategies in different scenes. ↓ means that the arrows in the table headers indicate that the values are sorted in descending order. Bold values indicate the best performance in each column.

Success Rate (%, ↓)	Scene I: City	Scene II: Line-Cruising	Scene III: Valley
Proposed	86.0	80.0	74.0
PPO	82.0	76.0	69.0
TRPO	80.0	74.0	68.0
A3C	78.0	72.0	66.0
DQN	77.0	70.0	64.0
DDPG	76.0	68.0	62.0

presents the task completion rates for our proposed method and the other strategies across the three scenarios. The results demonstrate that our proposed approach outperformed the baseline algorithms in all scenarios. Specifically, in the City and Line-cruising scenarios, our method achieved task completion rates of 86.0% and 80.0%, respectively, which are higher than those of the other algorithms. In the more complex Valley scenario, our approach still maintain a strong performance with a 74.0% success rate. In comparison, the PPO algorithm achieves $82.0\%$, $76.0\%$, and $69.0\%$ in the three scenarios, which is slightly lower than our method. Other algorithms such as TRPO, A3C, DQN, and DDPG perform relatively worse, with success rates below $80.0\%$ in all scenarios. Notably, in the Valley scenario, DDPG exhibit the lowest success rate of $62.0\%$.The experimental results indicate that our proposed method is capable of effectively handling different levels of obstacle complexity across various environments, achieving higher task completion rates than existing reinforcement learning strategies. The superior performance of the proposed method, particularly in complex environments, may be the result of its adaptive strategy optimization and efficient exploration mechanism. While the performance of all algorithms is comparable in simpler scenarios, such as Line-cruising, our method shows a significant advantage in more complex scenarios like City and Valley.

In addition to comparing various strategies of deep reinforcement learning, we also compared them with the existing sample-based methods in the same scenario as described in Section 5.3. In Scene I, the fixed-wing UAV in an urban environment, where it have to fly through densely packed buildings and avoid structural obstacles as illustrated in Figure 6a. In Scene II, simulating a power line inspection in mountainous terrain, the fixed-wing UAV’s primary task is to avoid obstacles such as mountainous ridges and power poles while maintaining proximity to power lines as depicted in Figure 6b. In Scene III, the fixed-wing UAV face a desert canyon with dynamic terrain changes, where the algorithm has to account for steep ascents and descents while avoiding natural formations such as cliffs and ridges as shown in Figure 6c. However, it is important to note that our algorithm incorporates a field-of-view (FOV) constraint, which restricts the UAV’s perception and decision-making to only the local visible region. This mimics realistic onboard sensor limitations. As a result, in complex environments such as Scene III, where obstacles may occlude the target or create narrow corridors, the agent may temporarily lose access to the global goal location or choose a safer, reachable subgoal within its visible area. The comparison of flight trajectories is presented in Figure 6, showcasing the performance of both algorithms across the three environments. In Scenario I, the proposed algorithm generates a smoother and more compact trajectory, as shown in Figure 7a. In contrast, the sample-based method tends to select detour regions with fewer obstacles, resulting in a longer path. The corresponding angular velocity curve in Figure 7a indicates that, although the proposed method exhibits higher initial control effort, it converges more quickly to a stable state, demonstrating superior adaptability in cluttered environments. In Scenario II, where obstacles are sparsely distributed, the sample-based method produces a trajectory that is closer to the straight-line reference path, as depicted in Figure 7b. In Scenario III, although the trajectory generated by the sample-based method appears smoother in Figure 7c, its maneuverability is fundamentally constrained by local sampling under field-of-view (FOV) limitations. As a result, it fails to execute the sharp turns required to navigate through the terrain. The angular velocity curve in Figure 7c clearly reveals abrupt control variations in the sample-based method. In contrast, the proposed algorithm maintains consistent maneuverability and successfully identifies feasible turning paths despite the FOV constraints.

The corresponding quantitative results are summarized in

Table 4. Quantitative comparison across three representative scenes. Bold values indicate the best performance in each column.

Indicator	Scene I	Scene II	Scene III
Success Rate (Proposed)	83%	80%	74%
Success Rate (Baseline)	72%	86%	–
Avg. Path Length (m) (Proposed)	952.3	1980.2	1727.8
Avg. Path Length (m) (Base)	1080.6	1806.3	–

. In Scene I, the proposed method achieves a higher success rate and a shorter average path length compared to the baseline, while maintaining similar angular smoothness, indicating better obstacle-aware planning in urban environments. In Scene II, where obstacles are more sparse and globally observable, the sample-based method performs competitively in terms of success rate and path efficiency. However, in Scene III that is characterized by complex terrain and frequent occlusions the sample-based method fails to reach the target reliably, as reflected by the missing entries (“–”) in Table 4. This failure is primarily due to the following two critical factors considered in our design: the dynamics constraints of fixed-wing UAVs and the field-of-view (FOV) limitation imposed during planning. Since the sample-based method relies on sampling only within the currently visible region and lacks global reasoning, it is prone to deviating from the goal and selecting unreachable paths, as evident from the blue trajectory in Figure 6c, which veers away from the canyon passage. In contrast, the proposed DRL-based algorithm effectively incorporates visibility and motion constraints, enabling robust navigation even in such challenging scenarios.

5.4. Hardware-in-the-Loop Simulation

A hardware-in-the-loop simulation experiment is conducted to demonstrate the deployability of the proposed algorithm. Additionally, a comparison with sample-based algorithms [54] is made to validate the performance of the proposed approach. The simulation platform consists of a computer equipped with an Intel i5-13600KF CPU and an NVIDIA RTX 4070Ti SUPER GPU, acting as the primary simulation unit. The onboard edge computing platform, the OrangePi 5B, equipped with a Rockchip RK3588s processor and a Neural Processing Unit (NPU), is used to execute real-time inference of the trained model. The model is initially trained and converted to the RKNN format using the RKNN-Toolkit2 (v2.1.0) and deployed via the Python (v3.10) API. The experimental platform is shown in Figure 8 and validation scenes are the same as the aforementioned Policy Comparison tests.

The quantitative analysis of the results is presented in Figure 9, where the X and Y coordinate distributions of the flight trajectories are plotted for each scene. The proposed algorithm’s path (solid line) is compared with the expected trend (dashed line) across all three environments. As shown in Figure 9a, the deviations between the proposed path and the reference straight-line path were minimal, indicating that the learned policy closely follows the intended direction in less challenging environments. In Scene II, the proposed method produced a noticeably smoother path with fewer oscillations, especially in the Y coordinate distribution, demonstrating its superiority in densely cluttered environments, as depicted in Figure 9b. This trend contrasts sharply with the results shown in Figure 9c, where the greatest divergence between the two methods is evident. The proposed algorithm’s ability to execute sharp turns and navigate narrow spaces enabled it to successfully complete the flight path, in contrast to the sample-based method, which struggled in this scenario.

Overall, the experimental results show that the proposed algorithm outperforms the sample-based method in more complex environments (Scene II and Scene III), particularly in terms of path smoothness and path length. While the sample-based method performs better in simpler environments with sparse obstacles (Scene I), the proposed algorithm demonstrates greater adaptability and robustness in challenging, real-world scenarios. These findings suggest that the proposed algorithm’s ability to make quick decisions in real-time, demonstrating its potential for deployment in real-world edge-based fast moving fixed-wing UAV systems where obstacle avoidance is critical.

6. Conclusions

In this paper, we present a lightweight DRL framework that leverages inferred single-frame depth maps as input and employs a compact network architecture to address the obstacle avoidance challenges of high-speed fixed-wing UAVs. Our framework incorporates an inferring reward function to account for the stability and dynamic constraints of fixed-wing UAVs, along with an adaptive entropy-based strategy update mechanism to balance exploration and exploitation during training. The proposed method is evaluated in various scenarios through hardware-in-the-loop simulations and compared with other reinforcement learning algorithms. Experimental results demonstrate that our framework significantly outperforms baseline methods in terms of obstacle avoidance effectiveness and path smoothness.

Despite these promising results, the current approach has certain limitations. The use of inferred depth maps may impact obstacle detection accuracy, particularly for small or fast-approaching obstacles. In future work, we plan to validate the algorithm on a real VTOL fixed-wing UAV platform to assess its performance in real-world environments. Furthermore, we aim to improve the framework’s adaptability and robustness by incorporating multi-agent cooperation for distributed obstacle avoidance, applying meta-learning techniques to enable rapid adaptation to unseen scenarios, and integrating uncertainty-aware decision-making to handle partial observability in complex environments. We also intend to embed energy-efficient navigation strategies to improve operational endurance and explore sim-to-real transfer methods to enhance the reliability of deployment in practical UAV missions.