Integrating Visual Perception with Conservative Enhanced Bio-Inspired Optimization for Safe UAV Trajectory Planning

Abstract

Unmanned Aerial Vehicle (UAV) trajectory planning in complex three-dimensional environments with threats remains a challenging optimization problem requiring efficient algorithms and threat detection capabilities. This study proposes the Conservative Enhanced Dwarf Mongoose Optimization Algorithm (CEDMOA), which introduces four key innovations to the original DMOA: hybrid population initialization, adaptive vocalization parameters, elite-guided learning strategy, and intelligent restart mechanisms. This work proposed the integration of CEDMOA with a novel vision-based threat detection system using YOLO object detection technology, enabling the identification and incorporation of threats into the optimization process. CEDMOA was comprehensively evaluated on the CEC2022 benchmark test suite, demonstrating superior performance compared to other state-of-the-art algorithms in solution quality and convergence stability. The results show the approach successfully generates an optimal collision-free flight trajectory in complex environments in UAV trajectory planning with both static and dynamic threats. Combining metaheuristic optimization with computer vision technology provides a robust framework for autonomous navigation that adapts to changing threat conditions. Experimental results validate the effectiveness of both the enhanced algorithm and the vision-based threat integration approach for practical UAV operations.

1. Introduction

Unmanned Aerial Vehicles (UAVs) have become integral to a wide array of applications across various sectors due to their versatility, cost-effectiveness, and ability to operate in challenging environments. In military operations, UAVs are employed for reconnaissance, surveillance, and targeted strikes, providing intelligence without risking human lives [1,2]. In civilian life, UAVs facilitate agricultural monitoring by assessing crop health through aerial imagery [3], support disaster relief efforts by delivering supplies to inaccessible areas, enable efficient logistics and package delivery in e-commerce, and contribute to environmental monitoring by tracking wildlife or pollution levels [4]. As technology advances, UAVs are increasingly equipped with advanced sensors, AI-driven autonomy, and communication systems, allowing them to perform complex tasks such as swarm coordination and adaptive navigation in dynamic settings [5,6]. However, these capabilities are constrained by multiple factors, including the need for obstacle avoidance and stringent requirements. Consequently, trajectory optimization planning emerges as a pivotal technology, aiming to generate optimal paths that minimize flight time, ensure collision-free routes, and comply with operational constraints, thereby enhancing overall mission efficiency and safety in both static and dynamic environments.

1.1. ResearchGap of the Current Situation

Trajectory planning methods for UAVs can be categorized into several approaches. Sampling-based algorithms, such as Rapidly-exploring Random Trees (RRTs) [7] and Probabilistic Roadmaps (PRMs) [8], rely on random sampling to explore the configuration space and construct feasible paths, making them suitable for high-dimensional problems. Potential Field Methods create virtual forces to attract the UAV toward the goal while repelling it from obstacles, offering simplicity and applicability. Graph search algorithms, including A* and Dijkstra’s [9], discretize the environment into grids or graphs to find shortest paths, providing optimality in structured spaces. Additionally, mathematical optimization techniques like Mixed-Integer Linear Programming (MILP) or Nonlinear Programming (NLP) formulate the problem as an optimization task with constraints on kinematics and dynamics. While these methods perform well in static, known environments by generating reliable paths, they suffer from notable drawbacks: sampling-based approaches often result in high computational complexity, non-smooth trajectories, and probabilistic completeness rather than optimality; potential field methods are prone to local minima traps, especially in cluttered or narrow passages, leading to oscillatory behavior or failure to reach the goal; graph search algorithms scale poorly in large or high-resolution environments due to exponential memory and time requirements; and mathematical optimization struggles with non-convex problems, requiring convex approximations that may sacrifice accuracy, while also lacking robustness in dynamic scenarios where replanning is frequent and must be computationally efficient [10,11]. In recent years, reinforcement learning has been widely applied in the trajectory planning of UAVs, mainly focusing on specific demand navigation tasks [12,13].

To address the challenges of local optima entrapment, high computational demands, and poor adaptability in complex, non-convex environments posed by traditional methods, researchers have increasingly turned to metaheuristic algorithms, which draw inspiration from natural phenomena to perform global searches and approximate optimal solutions efficiently [10,14]. These include Genetic Algorithms (GA) [15] that mimic evolutionary processes through selection, crossover, and mutation; Particle Swarm Optimization (PSO) inspired by bird flocking for collective intelligence-based search [16]; Ant Colony Optimization (ACO) [17] simulating ant pheromone trails for pathfinding; and other variants like Simulated Annealing (SA) [18] or Grey Wolf Optimizer (GWO) [19,20]. Over the past few years, numerous studies have explored the integration of metaheuristic algorithms into UAV trajectory optimization [21]. Recent work has proposed hybrid frameworks combining metaheuristics with machine learning for autonomous navigation in disaster zones, optimizing trajectories for minimal time and risk. Additionally, studies have developed ACO-enhanced methods for formation flying in UAV swarms [1,22], demonstrating better scalability in large-scale environments. Wang et al. proposed a strategy optimization algorithm integrating adaptive conduction heat search, quadratic interpolation, and elite population genetics [23]. Meanwhile, Tenniche et al. employed a compact Water Cycle Algorithm to tackle constraints in indoor environments [24]. Other algorithms employed in multi-UAV systems include the Improved Bat Algorithm and the Late Acceptance Hill Climbing-Based Algorithm [25,26]. These investigations underscore the potential of metaheuristics in achieving near-optimal solutions for multi-objective problems like energy minimization, coverage maximization, and real-time adaptation [27].

1.2. Problem Statement and Objective

In the existing framework of intelligent optimization algorithms for UAV planning, visual information has not been fully taken into account as an element of the cost function. Therefore, our problem lies in developing a comprehensive optimization planning framework that considers the visual cost, especially to establish a UAV trajectory planning method considering the actual situation and visual threats by remolding an optimization method.

1.3. Main Contribution

This study has carried out research work in the following areas:

Novel YOLO-integrated threat perception system for UAV planning. This work introduces visual threat detection capabilities into three-dimensional UAV trajectory optimization by seamlessly integrating YOLO object detection with path planning algorithms. The system processes live imagery to identify threats in the environment and converts detection results into soft constraint penalties through a Gaussian decay function, enabling unified modeling of both static terrain obstacles and visual threats.

Conservative Enhanced Dwarf Mongoose Optimization Algorithm (CEDMOA). Building upon the original DMOA framework [28], CEDMOA incorporates conservative enhancements while preserving the biological inspiration of mongoose social behavior. The algorithm introduces hybrid initialization combining random exploration with opposition-based learning, adaptive parameter control for smooth transition from exploration to exploitation, elite-guided learning from the best solutions during later iterations, and intelligent restart mechanisms to prevent premature convergence. These enhancements collectively improve convergence speed and solution quality for complex three-dimensional UAV trajectory optimization problems.

Intelligent UAV 3D trajectory planning framework. The framework seamlessly combines object detection capabilities with enhanced metaheuristic optimization to address complex trajectory planning challenges. The system unifies static terrain constraints and visual threats into the optimization model. Through intelligent algorithm design and adaptive parameter control, the framework achieves good performance in solution quality and convergence speed. Extensive experimental evaluations demonstrate significant improvements over traditional methods in terms of path safety, optimization efficiency, and environmental adaptability. The proposed approach successfully handles complex three-dimensional environments with both predefined obstacles and unknown threats. This work provides a robust solution for autonomous UAV navigation in challenging real-world scenarios. The integrated framework is a serviceable advancement in intelligent UAV planning.

2. Materials and Methods

2.1. Model of UAV

The mathematical representation of UAV trajectories plays a crucial role in determining both the efficiency and effectiveness of trajectory planning algorithms. This study uses the spherical coordinate parametrization framework that provides a more compact, physically meaningful, and computationally efficient representation of three-dimensional UAV trajectories. The choice of coordinate system fundamentally impacts the optimization landscape. Spherical coordinates offer distinct advantages over Cartesian representations: they naturally encode the incremental nature of aircraft motion, provide intuitive parameters that correspond directly to flight characteristics, and enable the incorporation of physical constraints through parameter bounds. This parametrization approach transforms the trajectory planning problem from a high-dimensional Cartesian optimization into a more structured spherical parameter space, leading to improved convergence properties and reduced computational complexity.

The complete trajectory is represented as follows:

$$P=\{Q,Q_1,Q_2,\dots ,Q_n,Q\}$$

(1)

The proposed trajectory representation discretizes the path into n intermediate waypoints between the start and goal positions. Each waypoint i is parameterized in spherical coordinates as follows:

$$P_i={\left(r_i,{\psi }_i,{\varphi }_i\right)}^T\hspace{1em}\mathrm{i}=\mathrm{1,2},\dots ,n$$

(2)

where: $r_i$ is the Radial distance (step size) ${\mathsf{\psi }}_{\mathrm{i}}$ is the Elevation angle, ${\mathsf{\varphi }}_{\mathrm{i}}$ is the Azimuth angle. $\mathrm{n}$ is the number of intermediate waypoints.

The spherical parameter space is carefully designed to ensure physical realizability and computational efficiency. The radial distance is bounded to prevent excessive step sizes:

$$0\le r_i\le r_{max}={\displaystyle \frac{2|p_{end}-p_{start}|}{n}}$$

(3)

This upper bound ensures that the maximum step size is proportional to the direct distance between start and goal, scaled by the number of waypoints. The elevation angle is constrained to limit vertical maneuverability: $-{\displaystyle \frac{\pi }{4}}\le {\psi }_i\le {\displaystyle \frac{\pi }{4}}$.

This constraint prevents excessively steep climbs or descents, maintaining realistic flight profiles. The azimuth angle is bounded relative to the direct heading:

$${\varphi }_0-{\displaystyle \frac{\pi }{4}}\le {\varphi }_i\le {\varphi }_0+{\displaystyle \frac{\pi }{4}}$$

(4)

where ${\mathsf{\varphi }}_0=\arctan 2\left({\mathrm{y}}_{\mathrm{f}}-{\mathrm{y}}_{\mathrm{s}},{\mathrm{x}}_{\mathrm{f}}-{\mathrm{x}}_{\mathrm{s}}\right)$, represents the direct bearing from start to goal. This constraint encourages trajectories that maintain a general progression toward the target while allowing reasonable deviations for obstacle avoidance.

Spherical coordinates offer several distinct advantages: they naturally constrain the search space to physically realizable trajectories, reduce the dimensionality of the optimization problem, and provide intuitive parameters that directly correspond to aircraft motion characteristics such as heading changes and climb rates. The constraints incorporate multiple operational limitations of UAVs to generate a reliable flight trajectory, and then convert it into the Cartesian coordinate system:

1. Transformation from starting point

$$\left\{\begin{array}{c}{x}_1=x_s+r_{1}cos\left({\psi }_1\right)sin\left({\varphi }_1\right)\\ y_1=y_s+r_{1}cos\left({\psi }_1\right)cos\left({\varphi }_1\right)\\ z_1=z_s+r_{1}sin\left({\psi }_1\right)\end{array}\right.$$

(5)

2. Incremental coordinate calculation:

$$\left\{\begin{array}{c}{x}_i=x_{i-1}+r_i\mathit{cos}\left({\psi }_i\right)\mathit{sin}\left({\varphi }_i\right)\\ y_i=y_{i-1}+r_i\mathit{cos}\left({\psi }_i\right)\mathit{cos}\left({\varphi }_i\right)\\ z_i=z_{i-1}+r_i\mathit{sin}\left({\psi }_i\right)\end{array}\right.$$

(6)

Converted Cartesian coordinates must satisfy spatial boundary constraints: ${\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\le {\mathrm{x}}_{\mathrm{i}}\le {\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}},{\mathrm{y}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\le {\mathrm{y}}_{\mathrm{i}}\le {\mathrm{y}}_{\mathrm{m}\mathrm{a}\mathrm{x}},{\mathrm{z}}_{\mathrm{m}\mathrm{i}\mathrm{n}}\le {\mathrm{z}}_{\mathrm{i}}\le {\mathrm{z}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$.

Boundary clipping operations include the following:

$$\begin{gathered} x_i=\mathit{max}\left(x_{min},\mathit{min}\left(x_i,x_{max}\right)\right) \\ {y}_i=\mathit{max}\left(y_{min},\mathit{min}\left(y_i,y_{max}\right)\right) \\ {z}_i=\mathit{max}\left(z_{min},\mathit{min}\left(z_i,z_{max}\right)\right) \end{gathered}$$

2.2. Trajectory Constraints

The fundamental challenge in UAV trajectory planning lies in the need to balance multiple competing objectives—path length minimization, threat avoidance, altitude compliance, and maneuverability constraints—while ensuring the generated trajectory remains feasible for the aircraft’s physical capabilities. The choice of coordinate system significantly impacts both the efficiency of the optimization process and the quality of the resulting solutions.

2.2.1. Overall Objective Function

The optimization objective function of UAV path planning is defined as follows:

$$J_{total}=\sum _{i=1}^5{w}_i\cdot J_i$$

(7)

where $w_i$ are weight coefficients for each sub-objective.

$$J_{total}=w_1{J}_1+w_2{J}_2+w_3{J}_3+w_4{J}_4+w_5{J}_5$$

(8)

where $J_1$, $J_2$, $J_3$, $J_4$, $J_5$ are the constraints, $w_i$ is the weight coefficients for each sub-objective.

This weighting strategy achieves a balance in multi-objective optimization, prioritizing energy efficiency and smoothness while ensuring safety.

2.2.2. Path Length Constraint

The path length cost function is defined as the sum of three-dimensional Euclidean distances of all path segments:

$$J_1=\sum _{i=1}^{N-1}|p_{i+1}-p_i{|}_2$$

(9)

where ${\mathrm{p}}_{\mathrm{i}}={\left[{\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{a}\mathrm{b}\mathrm{s},\mathrm{i}}\right]}^{\mathrm{T}}$, 3D coordinates of the i waypoint; ${\mathrm{z}}_{\mathrm{a}\mathrm{b}\mathrm{s},\mathrm{i}}={\mathrm{z}}_{\mathrm{r}\mathrm{e}\mathrm{l},\mathrm{i}}+\mathrm{H}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}}\right)$, the absolute altitude (relative to sea level), $\mathrm{H}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}}\right)$ are terrain elevation function; N is the total number of waypoints.

Detailed formula for three-dimensional distance calculation:

$$J_1=\sum _{i=1}^{N-1}\sqrt{{\left(x_{i+1}-x_i\right)}^2+{\left(y_{i+1}-y_i\right)}^2+{\left(z_{abs,i+1}-z_{abs,i}\right)}^2}$$

(10)

2.2.3. Threat Avoidance Constraint

Threat avoidance is a critical safety element in UAV trajectory planning. In both military and civilian applications, UAVs must effectively avoid various threat zones (such as radar coverage areas, air defense systems, and restricted zones) while maintaining mission efficiency. The model settings for the threat zone are shown in

Table 1. Hierarchical threat management zones.

Zones	Range Formula	Calculation Rules
Core Threat Zone	$d<R_j+d_{drone}$	Absolute no-fly zone/Entry results in infinite penalty
Aircraft Safety Zone	$R_j+d_{drone}\le d<R_j+d_{drone}+S_m$	Safety zone considering UAV physical dimensions/Hard constraint boundary
Buffer Safety Zone	$R_j+d_{drone}+S_m\le d<R_j+d_{drone}+{2S}_m$	Soft constraint zone providing additional safety margin/Progressive penalty function

, hierarchical threat management mechanism presented in this section achieves fine-grained threat handling through multi-layer safety zones, ensuring flight safety without excessive avoidance that would compromise length efficiency.

Each threat ${\mathrm{T}}_{\mathrm{k}}$ is defined as a quadruple:

$$T_j=\left(x_j,y_j,z_j,R_j\right)$$

(11)

where $\left(x_j,y_j,z_j\right)$ is threat center coordinates, $R_j$ is the basic threat radius, and the definition of three concentric safety zones for hierarchical threat management:

For the shortest distance from the path segment $P_i{P}_{i+1}$ to the threat center $T_j$:

$$d_{ij}=\underset{t\in \left[\mathrm{0,1}\right]}{\mathit{min}}|\left(P_i+t\left(P_{i+1}-P_i\right)\right)-T_j{|}_2$$

(12)

The path length cost function minimizes the total three-dimensional Euclidean distance:

$$J_2=\sum _{j=1}^M\sum _{i=0}^{n-1}{K}_j\cdot \mathit{max}[0,{\left(R_j+R_u+S_m-d_{ij}\right)}^2]$$

(13)

where $K_j$ is the threat penalty gain coefficient.

2.2.4. Altitude Constraint

Consistent altitude maintenance preserves the designed control margins required for precise trajectory tracking, thereby enabling accurate mission execution. Simultaneously, it guarantees statutory terrain clearance, eliminating the primary hazard of controlled flight into surface obstacles. Optimal altitude balancing energy consumption and safety.

As shown in the Figure 1a, the ideal flight altitude is as follows:

$$z_{ideal}={\displaystyle \frac{z_{max}+z_{min}}{2}}$$

(14)

Minimum flight altitude ${\mathrm{z}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$, maximum flight altitude ${\mathrm{z}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$. Ideal flight altitude ${\mathrm{z}}_{\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{l}}$ represents the optimal altitude balancing energy consumption and safety.

Altitude constraint cost function:

$$J_3=\sum _{i=0}^n\left[\mathit{max}\left(0,z_{\min }-\left(z_i-z\left(x_i,y_i\right)\right)\right)+\mathit{max}\left(0,\left(z_i-z\left(x_i,y_i\right)\right)-z_{\max }\right)\right]$$

(15)

Actual flight altitude is relative to ground level:

$$z_{rel,i}=z_{abs,i}-H\left(x_i,y_i\right)$$

(16)

Ground collision detection: ${\mathrm{z}}_{\mathrm{r}\mathrm{e}\mathrm{l},\mathrm{i}}\ge 0,\forall \mathrm{i}\in \{\mathrm{1,2},\dots ,\mathrm{n}\}$.

Figure 1b illustrates the other physical and control restrictions that the UAV must adhere to during its flight, including speed, acceleration, pitch angle, roll angle, and turning radius limits. The flight path is in a curved shape. It usually makes turns or performs non-linear movements in an environment with obstacles. The red arrows in the figure indicate the body coordinate system. In this study, we focused on the flight planning of the path itself, treating it as a point mass, and set the contact radius with obstacles to simulate the size of the aircraft body.

2.2.5. Maneuverability Constraint

Maneuverability constraints ensure the generated trajectory adheres to the aircraft’s physical capabilities. The turn angle constraint prevents excessive horizontal turns, while the climb angle constraint avoids exceeding vertical maneuver limits, making the optimized trajectory executable in real flight.

For three consecutive waypoints, define the horizontal projection vectors as follows: $\overrightarrow{{\mathrm{v}}_{\mathrm{i}}}=\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},0\right),\overrightarrow{{\mathrm{v}}_{\mathrm{i}+1}}=\left({\mathrm{x}}_{\mathrm{i}+1},{\mathrm{y}}_{\mathrm{i}+1},0\right)$. The turn angle constraint ensures that the UAV maintains a feasible turning radius throughout the trajectory. The turning angle α between consecutive path segments is calculated as follows:

$${\alpha }_i=\mathit{arccos}\left({\displaystyle \frac{\overrightarrow{v_{i+1}}\cdot \overrightarrow{v_i}}{\left|\overrightarrow{v_{i+1}}\right|\left|\overrightarrow{v_i}\right|}}\right)$$

(17)

The turn angle cost function is defined as follows:

$$J_{\mathrm{turn}}=\sum _{i=1}^{n-1}{\mathit{max}\left(0,{\alpha }_i-{\alpha }_{\max }\right)}^2$$

(18)

Similarly, the climb angle constraint addresses the vertical maneuverability limitations of the UAV. The climb angle β is calculated as follows:

$${\beta }_i=\mathit{arctan}\left({\displaystyle \frac{z_{i+1}-z_i}{|\overrightarrow{v_{i+1}}-\overrightarrow{v_i}|}}\right)$$

(19)

The climb angle cost function is formulated as follows:

$$J_{\mathrm{climb}}=\sum _{i=0}^{n-1}{\mathit{max}\left(0,\left|{\beta }_i\right|-{\beta }_{\max }\right)}^2$$

(20)

These maneuverability constraints work in conjunction with the previously defined path length, threat avoidance, and altitude constraints to produce trajectories that are not only theoretically optimal but also practically executable. The quadratic penalty formulation ensures continuous and differentiable cost functions that are well-suited for gradient-based optimization methods, while the threshold-based approach provides clear boundaries for feasible aircraft maneuvers.

The total maneuverability cost function is defined as follows:

$$J_4=J_{\mathrm{turn}}+J_{\mathrm{climb}}$$

(21)

2.2.6. YOLO Soft Constraint

In the actual operation of UAVs, it is also necessary to have feedback and references regarding the current flight status to obtain a threat response strategy. In this study, YOLO is not used for traditional object recognition, but as a core component for threat detection and obstacle recognition. Threat set obtained from YOLO detection:

$$\mathcal{D}=\{\left(c_j,w_j,{\sigma }_j\right):j=\mathrm{1,2},\dots ,N_{YOLO}\}$$

(22)

where: ${\mathrm{c}}_{\mathrm{j}}={\left[{\mathrm{x}}_{\mathrm{c},\mathrm{j}},{\mathrm{y}}_{\mathrm{c},\mathrm{j}}\right]}^{\mathrm{T}}$, Detected threat center coordinates, ${\mathrm{w}}_{\mathrm{j}}$ is the threat weight (based on detection confidence). The image coordinates are transformed to world coordinates using camera calibration parameters and geometric projection models, accounting for UAV position, orientation, and camera mounting configuration.

The YOLO-detected threats are incorporated into the trajectory optimization through a Gaussian soft constraint model that creates continuous influence fields around each detected object. The threat cost function is formulated as follows:

$$J_5=\sum _{i=1}^N\sum _{j=1}^{N_{YOLO}}{w}_j\cdot \mathit{exp}\left(-{\displaystyle \frac{|p_{i,2D}-c_j{|}^2}{2{\sigma }_j^2}}\right)$$

(23)

where $p_{i,2D}={\left[x_i,y_i\right]}^T$, the 2D projection of waypoint, $|p_{i,2D}-c_j{|}^2={\left(x_i-x_{c,j}\right)}^2+{\left(y_i-y_{c,j}\right)}^2$, and the square of the planar distance. This formulation creates smooth, differentiable cost surfaces that guide the optimization algorithm away from detected threats while avoiding the discontinuities associated with hard constraint approaches. The YOLO soft constraints $J_5$ use obstacle detection from a bird’s-eye view to dissuade the UAV from detouring around obstacles through the form of a cost function, rather than flipping over the obstacles in an extreme situation. The values obtained from this view do not affect the actual three-dimensional path planning.

The threat weight incorporates both detection confidence and object type significance:

$$w_j=g_{label}\cdot \mathit{max}\left(0,s_j\right)$$

(24)

where ${\mathrm{s}}_{\mathrm{j}}$ is the YOLO detection confidence score, ${\mathrm{g}}_{\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}}$ is the Gain coefficient based on label type. The Gaussian scale parameter ${\mathsf{\sigma }}_{\mathrm{j}}$ determines the spatial extent of each threat’s influence and is adaptively computed based on the detected object characteristics:

$${\sigma }_j=\mathit{max}\left({10}^{-3},r_j\cdot {\sigma }_{scale}\right)$$

(25)

In practical implementation, penalty function method is used to handle constraints:

$$\widetilde{J_{total}}\left(s\right)=J_{total}\left(s\right)+\sum _k{M}_k\cdot {\mathit{max}\left(0,g_k\left(s\right)\right)}^2$$

(26)

where: ${\mathrm{g}}_{\mathrm{k}}\left(\mathrm{s}\right)$ is the inequality constraint function; $M_k$ is the Penalty coefficient.

In this section, we present Conservative Enhanced Dwarf Mongoose Optimization (CEDMOA)—a significantly refined metaheuristic optimizer inspired by the cooperative hunting and social communication behaviors of dingo packs. Building upon the foundational DMOA framework, CEDMOA integrates five key algorithmic enhancements grounded in biological realism, theoretical optimization principles, and practical engineering robustness. These innovations collectively address persistent challenges in population-based optimization: premature convergence, poor exploration–exploitation balance, sensitivity to initialization, and inadequate handling of boundary constraints. Below, we detail each component with theoretical justification, mathematical formulation, and implementation insights.

2.3. Dwarf Mongoose Optimization

The Dwarf Mongoose Optimization Algorithm (DMOA) [28] is a metaheuristic optimization method inspired by the cooperative foraging and social behaviors of the dwarf mongoose (Helogale parvula), a highly social mammal. DMOA models the interactions among three distinct roles within a mongoose group: the Alpha Female, Scouts, and Babysitters, to achieve an effective balance between exploration and exploitation in complex search spaces.

The crux of the entire issue is considering the minimization problem, min f(x).

2.3.1. Algorithm Initialization

The DMOA algorithm first initializes the population with size nPop, which includes nAlphaGroup Alpha individuals and nBabysitter babysitter individuals:

$$X_i=X_{min}+rand\times \left(X_{max}-X_{min}\right)$$

(27)

where $X_i$ represents the position of the i-th individual; ${\mathrm{X}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and ${\mathrm{X}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$ represent the lower and upper bounds of the search space, respectively.

2.3.2. Alpha Group Foraging Behavior

Individuals in the Alpha group select the Alpha female through roulette wheel selection for foraging. The fitness function is defined as follows:

$$F_i=\mathit{exp}\left(-{\displaystyle \frac{Cos{t}_i}{MeanCost}}\right)$$

(28)

The selection probability is as follows:

$$P_i={\displaystyle \frac{F_i}{{\sum }_{j=1}^{nAlphaGroup}{F}_j}}$$

(29)

The position update formula for Alpha female is as follows:

$$X_{new}=X_i+\varphi \times \left(X_i-X_k\right)$$

(30)

where $\mathsf{\varphi }={\displaystyle \frac{\mathrm{p}\mathrm{e}\mathrm{e}\mathrm{p}}{2}}\times \mathrm{U}(-\mathrm{1,1})$ is the vocalization coefficient, peep = 2, represents uniform random numbers in the interval [−1,1], and k is a randomly selected individual index not equal to i.

2.3.3. Scouting Behavior

Scout individuals use the same position update mechanism but introduce a sleeping mould parameter:

$$s{m}_i={\displaystyle \frac{Cos{t}_{new}-Cos{t}_i}{\mathit{max}\left(Cos{t}_{new},Cos{t}_i\right)}}$$

(31)

The next position update for scout individuals is as follows:

$$X_{next}=\left\{\begin{array}{c}{X}_i- CF·\varphi ·rand·\left(X_i-M\right) {\tau }_{new}> \tau \\ X_i+ CF·\varphi ·rand·\left(X_i-M\right) {\tau }_{new}\le \tau \end{array}\right.$$

(32)

where $CF={(1-{\displaystyle \frac{t}{MaxIt}})}^{\frac{2t}{MaxIt}}$ is the convergence factor, $M={\displaystyle \frac{X_i\times s{m}_i}{X_i}}$, and ${\tau }_{new}=mean\left(sm\right)$.

2.3.4. Babysitting Behavior

When the abandonment counter C(i) of an individual reaches the threshold L, babysitter individuals reinitialize their positions:

$$X_i=X_{min}+rand\times \left(X_{max}-X_{min}\right),\mathrm{if} C\left(i\right)\ge L$$

(33)

where $\mathrm{L}=\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\left(0.6\times \mathrm{n}\mathrm{V}\mathrm{a}\mathrm{r}\times \mathrm{n}\mathrm{B}\mathrm{a}\mathrm{b}\mathrm{y}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{r}\right)$.

2.4. Conservative Enhanced Dwarf Mongoose Optimization

2.4.1. Dual-Learning Enhanced Population Initialization

To overcome the limitations of purely random initialization—such as clustered initial solutions and slow convergence—we introduce a hybrid initialization strategy that synergistically combines stochastic sampling with Opposition-Based Learning (OBL) [29]. This approach leverages the principle that for any candidate solution x, its opposite point x^o = L + U − x (where L and U denote lower and upper bounds) often provides complementary information about the search landscape. By allocating 90% of the initial population via uniform random sampling and reserving 10% for opposition-based counterparts (aligned with the golden-section-inspired ratio), CEDMOA ensures both genetic diversity and accelerated convergence from the outset.

2.4.2. Adaptive Vocalization Coefficient (Communication Fading)

Inspired by the natural decay of vocal intensity in foraging dwarf mongooses [30], CEDMOA introduces a time-adaptive “peep” coefficient:

$$\mathrm{peep}\left(\mathrm{t}\right)={\mathrm{peep}}_{\mathrm{base}}\left(1-0.2\cdot {\displaystyle \frac{\mathrm{t}}{{\mathrm{T}}_{\mathrm{m}\mathrm{a}\mathrm{x}}}}\right)$$

(34)

This coefficient linearly decays from 2.0 to 1.6, smoothly reducing the magnitude of search perturbations over time and enabling a natural transition from global exploration to local exploitation.

This dual-initialization scheme enhances population diversity, reduces cold-start latency, and promotes more uniform coverage of the feasible search space—critical for high-dimensional and multimodal optimization problems common in unmanned system trajectory planning and sensor deployment.

2.4.3. Elite-Guided Strategy

During the Alpha group foraging process, the improved algorithm learns from the current best solution with a 10% probability:

$$X_{new}=\left\{\begin{array}{cc}{X}_i+ \varphi ·\left(X_{best}-X_i\right)\hfill & if rand< 0.1 and t >{\displaystyle \frac{MaxIt}{3}}\hfill \\ X_i+ \varphi ·\left(X_i-X_k\right)\hfill & otherwise\hfill \end{array}\right.$$

(35)

This strategy is only activated in the later stage of the algorithm ($t >{\displaystyle \frac{MaxIt}{3}}$) to avoid premature convergence.

2.4.4. Improved Boundary Handling Mechanism

The original algorithm lacks an effective boundary handling mechanism, while the improved algorithm adopts a boundary constraint strategy:

$$X_i=\mathit{max}\left(\mathit{min}\left(X_i,X_{max}\right),X_{min}\right)$$

(36)

This ensures that all individual positions are within the feasible domain, improving the stability of the algorithm.

2.4.5. Intelligent Restart Mechanism

The improved algorithm optimizes the babysitting behavior by adopting an intelligent restart strategy:

$$X_i=\left\{\begin{array}{cc}{X}_{\mathrm{best}}+ \sigma ·N\left(\mathrm{0,1}\right)\hfill & if rand< 0.5 and t >{\displaystyle \frac{MaxIt}{4}}\hfill \\ X_{\min }+ rand·\left(X_{\max }-X_{\min }\right)\hfill & otherwise\hfill \end{array}\right.$$

(37)

where $({\mathrm{X}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathrm{X}}_{\mathrm{m}\mathrm{i}\mathrm{n}})\times 0.05$ is the noise scale, and $\mathrm{N}\left(\mathrm{0,1}\right)$ represents the standard normal distribution. This mechanism restarts near the optimal solution with a 50% probability in the middle and later stages of the algorithm, balancing exploration and exploitation capabilities.

2.4.6. Algorithm Complexity Analysis

The time complexity of the CEDMOA algorithm is as follows:

$$T\left(n\right)=O\left(MaxIt\times nPop\times nVar\times T_{obj}\right)$$

(38)

where ${\mathrm{T}}_{\mathrm{o}\mathrm{b}\mathrm{j}}$ is the computational complexity of the objective function. Compared to the original DMOA algorithm, CEDMOA only adds constant-level computational overhead, mainly from boundary checking and conditional judgments for intelligent restart.

These improvement strategies work synergistically to significantly enhance the convergence accuracy and search efficiency of the algorithm while maintaining the core advantages and simplicity of the original DMOA algorithm.

2.5. Algorithm Process

2.5.1. Initialization Phase

The algorithm begins with a hybrid population initialization strategy that combines random generation with opposition-based learning. Specifically, ninety percent of the initial population is generated using conventional uniform random distribution within the search space bounds, while the remaining ten percent employs opposition-based learning to create individuals at opposite positions in the search space. This dual initialization approach enhances population diversity and increases the probability of discovering promising search regions from the outset.

2.5.2. Adaptive Vocalization Mechanism

A key enhancement in CEDMOA is the introduction of an adaptive vocalization parameter that dynamically adjusts throughout the optimization process. The vocalization coefficient starts at its maximum value and gradually decreases as iterations progress, ensuring strong exploration capabilities during early iterations while transitioning to exploitation as the algorithm approaches convergence. This adaptive mechanism provides a better balance between global exploration and local exploitation compared to the fixed parameter in the original DMOA.

2.5.3. Enhanced Alpha Group Foraging

The Alpha group foraging phase incorporates an elite-guided strategy that selectively leverages information from the current best solution. The fitness-based selection mechanism employs exponential transformation to calculate individual fitness values, followed by roulette wheel selection to choose Alpha females for foraging activities.

The position update strategy implements a conditional branching mechanism where individuals have a small probability of learning directly from the global best solution, but only during the latter two-thirds of the optimization process. This selective elite guidance ensures that the algorithm maintains exploration capabilities in early stages while gradually incorporating exploitation of the best-found solutions. When elite guidance is not applied, the algorithm follows the original DMOA foraging behavior based on interactions between randomly selected individuals.

2.5.4. Boundary Handling and Greedy Selection

All newly generated positions undergo strict boundary constraint enforcement to ensure feasibility within the defined search space. The algorithm then applies greedy selection criteria, accepting improved solutions while maintaining abandonment counters for individuals that fail to improve over successive iterations.

2.5.5. Scout Behavior and Sleeping Mould

The scouting phase maintains the original DMOA mechanism for position updates while calculating sleeping mould parameters that quantify the relative improvement or deterioration of individual solutions. These parameters influence subsequent position adjustments through a convergence factor that adapts based on the current iteration progress, providing dynamic control over the exploration-exploitation balance.

2.5.6. Intelligent Restart Process

The babysitting behavior incorporates an intelligent restart strategy that activates when individual abandonment counters exceed a predefined threshold. Upon triggering, the algorithm employs a probabilistic restart approach where half of the restarted individuals are repositioned near the current best solution using small Gaussian perturbations, while the other half undergo complete random reinitialization within the search bounds. This dual restart strategy balances intensive local search around promising regions with continued global exploration.

2.5.7. Convergence and Termination

The algorithm iteratively updates the global best solution and increments the iteration counter until the maximum number of iterations is reached. Throughout this process, the convergence factor continuously modulates the influence of various algorithmic components, ensuring an appropriate balance between exploration and exploitation phases.

2.5.8. Computational Efficiency

CEDMOA maintains computational efficiency comparable to the original DMOA, with algorithmic enhancements introducing only minimal computational overhead. The improvements primarily involve conditional branching logic and boundary checking operations, making CEDMOA practically viable for real-world optimization problems while significantly improving solution quality. The detailed algorithm pseudocode and flowchart can be found in Algorithm 1 and Figure 2, respectively.

Algorithm 1 Pseudo code of CEDMOA

Start: 1: Set parameters N, Tmax, Xmin, Xmax, D, f; 2: Initialize nBabysitter = 3, nAlphaGroup = N − nBabysitter; 3: Set L = round(0.6 × D × nBabysitter), peepbase = 2; 4: For i = 1:floor(0.9 × nAlphaGroup) do 5: Xi = Xmin + rand × (Xmax − Xmin); 6: End for 7: For i = floor(0.9 × nAlphaGroup) + 1:nAlphaGroup do 8: ${\mathrm{X}}_{\mathrm{i}}^{\mathrm{O}\mathrm{B}\mathrm{L}}={\mathrm{X}}_{\mathrm{m}\mathrm{i}\mathrm{n}}+{\mathrm{X}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathrm{X}}_{\mathrm{i}}^{\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}}$; 9: End for 10: Evaluate fitness values and determine the best Xbest; 11: While (t ≤ Tmax) do 12:$\mathrm{p}\mathrm{e}\mathrm{e}\mathrm{p}=\mathrm{p}\mathrm{e}\mathrm{e}{\mathrm{p}}_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}\times \left(1-0.2\times \mathrm{t}/{\mathrm{T}}_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)$; 13: Calculate fitness Fi and selection probability Pi; 14: For m = 1: nAlphaGroup do 15: Select Alpha female i using roulette wheel selection; 16: If rand() < 0.1 and t > Tmax/3 then 17: Xnew = Xi + φ × (Xbest − Xi); 18: Else 19: Xnew = Xi + φ × (Xi − Xk); 20: End If 21: Xnew = max(min(Xnew, Xmax), Xmin); 22: If f(Xnew) ≤ f(Xi) then 23: Xi = Xnew; 24: Else 25: Ci = Ci + 1; 26: End If 27: End for 28: For i = 1: nScout do 29: Update scout position and calculate smi; 30: End for 31: For i = 1: nBabysitter do 32: If Ci ≥ L then 33: If rand() < 0.5 and t > Tmax/4 then 34: Xi = Xbest + σ × N(0,1); 35: Else 36: Xi = Xmin + rand × (Xmax − Xmin); 37: End If 38: Ci = 0; 39: End If 40: End for 41: Update CF and next mongoose positions; 42: Update the best Xbest solution so far; 43: t = t + 1; 44: End while 45: Return Xbest; Stop

2.5.9. Camera Coordinate Transformation

In the visual perception system, it is necessary to convert the target position of the image coordinates output by the YOLO detector into the position in the world coordinate system. This study adopts a simplified geometric projection model, and the core conversion is as follows:

The coordinates of the detection box in the image coordinate system are $\left(x_{img},y_{img},w_{img},h_{img}\right)$, $\left(x_{img},y_{img}\right)$, $w_{img},h_{img}$ represent the width and height of the detection box, respectively.

Calculation of the center point of the detection box:

$$\begin{gathered} x_c=x_{img}+{\displaystyle \frac{w_{img}}{2}} \\ {y}_c=y_{img}+{\displaystyle \frac{h_{img}}{2}} \end{gathered}$$

(39)

Linear mapping from image coordinates to world coordinates:

$$\begin{gathered} x_w={\displaystyle \frac{x_c}{W_{img}}}\times MAPSIZ{E}_X \\ {y}_w={\displaystyle \frac{y_c}{H_{img}}}\times MAPSIZ{E}_Y \end{gathered}$$

(40)

$W_{img}$ and $H_{img}$ represent the width and height of the image. $MAPSIZ{E}_X$ and $MAPSIZ{E}_Y$ represent the size range of the world map. $\left(x_w,y_w\right)$ represent the transformed world coordinates.

The influence radius of the threat area is estimated based on the bounding box dimensions as follows:

$$r=r_{default}+\alpha \cdot \mathrm{m}\mathrm{a}\mathrm{x}(w_{img},h_{img})\cdot s_{pixel}\cdot s_{radius}$$

(41)

where $r_{default}$ is the default threat radius, $\mathsf{\alpha }=0.5$ is a scaling coefficient, $s_{pixel}$ is the pixel-to-map scaling factor, and $s_{radius}$ is a radius scaling coefficient.

To validate the effectiveness of the proposed algorithm, a synthetic image generation method based on elevation data is designed. This approach constructs realistic terrain representations and overlays threat information, path planning elements, and simulated noise to mimic real-world UAV-captured imagery.

The raw elevation matrix $H$ is first normalized:

$$H_{norm}={\displaystyle \frac{H-\mathrm{m}\mathrm{i}\mathrm{n}\left(H\right)}{\mathrm{m}\mathrm{a}\mathrm{x}\left(H\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(H\right)}}$$

(42)

A pseudo color mapping is then applied to convert the normalized elevation data into an RGB image:

$$I_{base}=f_{color}\left(H_{norm}\right)$$

(43)

where $f_{color}$ is the Parula colormap function, mapping grayscale values to a 256-color RGB space.

3. Evaluation of Algorithm

To compare the effectiveness of the newly improved CEDMOA with some other selected mainstream optimization algorithms, we use some functions from CEC2022 (

Table 2. Benchmark functions from CEC2022.

Function	Num	Dim	Search Scope
${\displaystyle f_1\left(x\right)=\sum _{i=1}^D{x}_i^2+{\left(\sum _{i=1}^{D}0.5i{x}_i\right)}^2+{\left(\sum _{i=1}^{D}0.5i{x}_i\right)}^4}$	F1	10~20	[−100,100]
${\displaystyle f_2\left(x\right)=\sum _{i=1}^{D-1}\left[100{\left(x_{i+1}-x_i^2\right)}^2+{\left(x_i-1\right)}^2\right]}$	F2	10~20	[−100,100]
${\displaystyle f_3\left(x\right)={\left({\displaystyle \frac{1}{D-1}}\sum _{i=1}^{D-1}\sqrt{s_i}\left[{\mathit{sin}}^2\left(50s_i^{0.2}\right)+1\right]\right)}^2}$	F3	10~20	[−100,100]
${\displaystyle f_4\left(x\right)=\sum _{i=1}^D\left[y_i^2-10\mathit{cos}\left(2\pi y_i\right)+10\right]}$	F4	10~20	[−100,100]
${\displaystyle f_5\left(x\right)={\mathit{sin}}^2\left(\pi w_1\right)+\sum _{i=1}^{D-1}{\left(w_i-1\right)}^2\left[1+10{\mathit{sin}}^2\left(\pi w_i+1\right)\right]+{\left(w_D-1\right)}^2\left[1+{\mathit{sin}}^2\left(2\pi w_D\right)\right]}$	F5	10~20	[−100,100]
${\displaystyle f_6\left(x\right)=418.9829D-\sum _{i=1}^D{x}_i\mathit{sin}\left(\sqrt{\left|x_i\right|}\right)}$	F6	10~20	[−100,100]
${\displaystyle f_7\left(x\right)=\mathit{min}\left\{\sum _{i=1}^D{\left(x_i-{\mu }_0\right)}^2,dD+s\sum _{i=1}^D{\left(x_i-{\mu }_1\right)}^2\right\}+10\sum _{i=1}^D\left[1-\mathit{cos}\left(2\pi \left(x_i-{\mu }_0\right)\right)\right]}$	F7	10~20	[−100,100]
${\displaystyle f_8\left(x\right)=\sum _{i=1}^{D}g\left(f_{Rosenbrock}\left(x_i,x_{i+1}\right)\right)}$	F8	10~20	[−100,100]
${\displaystyle f_9\left(x\right)=\sum _{i=1}^D\left[x_i^2-10\mathit{cos}\left(2\pi x_i\right)+10\right]}$	F9	10~20	[−100,100]

) for practical verification and computational efficiency trade-offs. The comparison methods adopted include the existing IGWO, GWO, GJO, POA, Chimp, SCA, and DMOA [31,32,33,34]. The whole verification was finished on a Windows 11 system with an Intel Ultra 7 255HX CPU. The population size for the operation is set to 50, and the maximum number of iterations is 500. Analysis of the algorithm’s effectiveness from five aspects, including Overall Ranking, Success Rate, Efficiency, Stability, and the iterative trend graph.

The convergence curves for functions F1–F9 (Figure 3) demonstrate CEDMOA’s superior optimization capability across diverse problem landscapes. In unimodal functions (e.g., F1 and F2), CEDMOA exhibits rapid initial convergence, achieving fitness values below 10—4 within 50 iterations—a pace significantly faster than competitors like GWO and SCA. For multimodal and hybrid functions (e.g., F4, F6, and F9), CEDMOA maintains a consistent downward trajectory without premature convergence, ultimately reaching the lowest final fitness values. Notably, in F6, CEDMOA’s solution quality surpasses other algorithms by one to two orders of magnitude, while its smooth convergence curve indicates robust exploration-exploitation balance. This consistent outperformance across all nine test functions underscores CEDMOA’s enhanced global search ability and precision in high-dimensional optimization.

The statistical comparison charts (Figure 4) further validate CEDMOA’s efficacy through four critical metrics. First, the overall ranking of 4a shows CEDMOA achieving the lowest average rank, decisively outperforming all competitors, which ranks 3.0–7.5, and confirming its top-tier performance. Second, the success rate chart of 4b reveals CEDMOA’s 100% reliability across independent runs, matching or exceeding other algorithms. Third, the efficiency of 4c illustrates a favorable time-quality trade-off: CEDMOA requires only marginally more execution time than the fastest algorithms, GWO, but delivers substantially higher solution quality. Finally, the stability of 4d highlights CEDMOA’s minimal solution variability, reflecting consistent convergence behavior. Collectively, these metrics establish CEDMOA as highly reliable, efficient, and stable under varying conditions.

Both the convergence curves and statistical metrics converge to a unified conclusion: CEDMOA’s proposed enhancements yield demonstrable improvements over state-of-the-art algorithms. Its ability to consistently achieve faster convergence, higher solution precision, and greater robustness across all CEC2022 test functions—while maintaining computational efficiency and stability—validates the effectiveness of the modified strategies. This comprehensive evidence confirms that CEDMOA is a viable and competitive optimizer for complex real-world applications, with its improvements providing tangible benefits in both theoretical and practical contexts.

4. Results of Trajectory Planning Simulation

4.1. Simulation Trajectory Parameter Setting

In this study, YOLO is not used for traditional object recognition but as a core component for real-time threat detection and obstacle recognition. In order to separately observe the effects with and without the visual cost function, the UAV trajectory simulation is carried out in two groups. One group does not consider the visual cost function, while the other group does. This paper designs three terrain scenarios with six flight missions. Each scenario includes two start-end point layouts. Performance comparisons are also conducted between the existing seven optimization methods and CEDMOA.

The experimental evaluation is conducted on a three-dimensional UAV path planning scenario using a 1045 × 879 pixel terrain map derived from ChrismasTerrain.tif elevation data, with the operational environment spanning coordinates from (1,1) to (1045,879) in the horizontal plane and altitude constraints between 100–200 m. The UAV mission is configured with a 10-node trajectory path through spherical coordinate representation with radial, azimuthal, and polar angle constraints. Eight cylindrical threat zones with radii ranging from 70 to 80 m are strategically positioned throughout the environment to simulate realistic obstacle avoidance scenarios, with a safety margin of 3.5 m and a penalty coefficient of 1.0 applied for threat proximity calculations. The comparative evaluation encompasses eight state-of-the-art metaheuristic algorithms, including iGWO, GWO, GJO, Chimp, POA, SCA, DMOA, and the proposed CEDMOA, with each algorithm executed for 300 iterations using a population size of 50 individuals across 30 independent runs to ensure statistical significance. For the visual threat detection component, YOLOv8 with tiny-YOLO-COCO pre-trained weights is integrated with a confidence threshold of 0.4, processing synthetic aerial imagery generated at 0.8 pixel-to-map scale ratio with Gaussian decay functions modeling threat influence zones, while the multi-objective cost function incorporates weighted coefficients of 12, 0.7, 2.5, 3, and 2 for path length, obstacle avoidance, altitude deviation, trajectory smoothness, and YOLO threat penalties respectively. On this basis, a series of comparisons is made for the optimal value, standard deviation, mean, median, and worst value of the calculated cost.

Table 3. Simulate planning and set conditions.

Simulation	Obstacles	Way Points	Target	Max Iterations
Verification 1 & 7	A4	10	(100,100,50) → (900,700,150)	300
Verification 2 & 8	A4	10	(500,500,50) → (900,700,150)	300
Verification 3 & 9	B6	10	(100,100,50) → (850,600,150)	300
Verification 4 & 10	B6	10	(500,500,50) → (900,700,150)	300
Verification 5 & 11	C8	10	(100,50,100) → (950,800,100)	300
Verification 6 & 12	C8	10	(500,500,50) → (900,700,150)	300

and

Table 4. Obstacle setting for verification.

Configuration	Threat ID	Center (x,y)	Radius R	Height z
A4	1	(300,250)	80	100
	2	(700,500)	80	150
	3	(550,300)	70	150
	4	(400,500)	70	150
B6	1	(250,350)	80	150
	2	(700,550)	70	105
	3	(600,350)	80	150
	4	(400,500)	70	150
	5	(550,750)	70	150
	6	(400,200)	80	150
C8	1	(200,200)	60	100
	2	(400,190)	70	125
	3	(750,300)	80	150
	4	(400,650)	80	150
	5	(500,350)	80	150
	6	(300,400)	80	150
	7	(600,700)	70	100
	8	(650,500)	80	130

show all specific configuration parameters and validation targets, including obstacle placements, path endpoints, and iteration counts. The specific obstacle setting conditions are clearly indicated in Table 4. The obtained results are, respectively, shown in Figure 5, the actual trajectory diagram of the UAV.

4.2. Without Visual Cost Function

Figure 5 illustrates the 3D trajectory planning results for all verification scenarios, where CEDMOA consistently generates smooth and feasible results compared to competing algorithms. The experimental results demonstrate CEDMOA’s superior performance in UAV trajectory planning across all verification scenarios. In Verification 1 (Figure 5a–c), CEDMOA generated a smooth trajectory with a length of approximately 480m, significantly outperforming GJO, which failed to produce a feasible solution in this constrained environment. It is prone to falling into a state of no solution in a highly constrained environment, a limitation that CEDMOA successfully overcomes. The convergence analysis (Figure 6) reveals CEDMOA’s accelerated optimization capability, achieving a cost function value of 1.43 × 10⁴ by iteration 100 in Verification 3, while DMOA remained at a high level. The improvement in solution quality is consistent across all scenarios. The statistical analysis (Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12) further confirms CEDMOA’s dominance. Notably, CEDMOA maintains relatively excellent objective function values and also exhibits relatively outstanding standard deviation stability.

The box plot analysis in Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12 quantifies CEDMOA’s comprehensive advantages among various algorithms; the optimal path length is maintained while achieving a competitive cost function. This balance between computational efficiency and solution quality makes CEDMOA particularly suitable for UAV where both factors are critical. The consistent outperformance across all six verification scenarios confirms CEDMOA’s robustness in handling diverse environmental constraints and its superiority over existing optimization techniques for UAV trajectory planning.

4.3. With Visual Cost Function

The YOLO-frames consist of six synthetic visual images generated from terrain and threat data to simulate time-series visual observations during UAV flight operations. These images incorporate colorized terrain maps, red-highlighted threat zones, and start/end point markers, with each frame containing slight random noise and threat size variations to emulate realistic visual condition changes. The system feeds these six frames into the YOLOv8 detector for threat identification, then statistically fuses the multi-frame detection results and converts them into map-coordinate soft constraint threat costs, which are ultimately integrated into the multi-objective optimization function for visual perception-based threat avoidance in planning. The results from verification 7 to verification 12 are obtained under the visual cost function, and the views of threats detected by vision are provided in Figure 13. The red circle in the picture represents the range of obstacles detected visually, while the other colors represent terrain information.

The experimental results with integrated YOLO-based visual threat detection demonstrate CEDMOA’s robust performance in complex trajectory planning scenarios. In Figure 14, which presents the 3D trajectory visualization for Verification 7–12, CEDMOA consistently generates smooth and feasible paths that effectively navigate around visually detected threats while maintaining appropriate safety margins. The side and top views clearly illustrate how CEDMOA dynamically adjusts trajectories to avoid threat zones without excessive detours, preserving path efficiency and smoothness.

The convergence curves in Figure 15 reveal CEDMOA’s superior optimization capability under visual constraints. Across all verification scenarios (7–12), CEDMOA has significant advantages in terms of optimal value and path length. This is particularly evident in Verification 10 in Figure 15d, where CEDMOA demonstrates rapid convergence and maintains the low cost function throughout the optimization process. The statistical comparisons in Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21 provide quantitative evidence of CEDMOA’s consistent superiority. In Verification 10 (Figure 19), CEDMOA achieves the lowest, best, mean, and worst cost values among all algorithms, with a notably narrow distribution in the box plot indicating an exceptional solution with flexibility of the length and the stability of the calculation degree reflected by the standard deviation. Similarly, in Verification 12 (Figure 21), CEDMOA maintains the most favorable cost distribution while demonstrating competitive execution times. The data for verification 1–12 are all provided in Appendix B.

When comparing with the non-visual scenarios, two important observations emerge. First, the introduction of visual threat constraints increases the overall complexity of the optimization problem, as evidenced by the generally higher cost function values across all algorithms. Second, CEDMOA’s relative performance advantage becomes even more pronounced in the visual scenarios compared to the non-visual cases, suggesting enhanced capability to process and respond to visual information while maintaining trajectory optimality.

The results confirm that CEDMOA not only effectively handles the additional complexity introduced by visual threat detection but actually demonstrates greater relative performance improvement compared to other algorithms in the presence of visual constraints, making it particularly suitable for real-world UAV applications where visual perception plays a critical role in navigation safety.

4.4. Performance Analysis of CEDMOA

The experimental results demonstrate that the proposed CEDMOA algorithm consistently achieves superior performance across both conventional path planning scenarios and visual threat-integrated environments. In the baseline configuration without visual cost components, CEDMOA exhibits the most favorable convergence characteristics among all evaluated algorithms and the highest solution stability as measured by standard deviation metrics. When the YOLO-based visual threat detection system is integrated through the J5 cost component, CEDMOA maintains its competitive advantage while successfully incorporating threat avoidance capabilities into the optimization process. The algorithm demonstrates remarkable adaptability to the enhanced multi-objective formulation, effectively balancing the additional visual threat penalties with traditional path planning objectives, including trajectory length, obstacle avoidance, altitude optimization, and smoothness constraints. Notably, the integration of visual cost functions does not compromise CEDMOA’s optimization efficiency, as the algorithm continues to outperform benchmark methods in terms of both solution quality and computational convergence. This consistent superior performance across different problem formulations validates the robustness of the proposed conservative enhancement strategies and confirms CEDMOA’s effectiveness as a comprehensive solution for intelligent UAV applications.

5. Conclusions and Discussion

5.1. Conclusions

This study proposes a novel approach by integrating YOLO-based visual detection errors as a constraint term in the optimization framework. Our research presents a comprehensive intelligent UAV three-dimensional path planning framework that successfully integrates visual threat perception with advanced metaheuristic optimization algorithms. The developed system addresses critical challenges in autonomous UAV navigation by combining real-time object detection with the novel CEDMOA algorithm for multi-objective trajectory optimization.

The proposed framework demonstrates significant achievements in handling both static terrain obstacles and visual threats within a unified optimization model. Extensive experimental evaluations validate the effectiveness of the approach, with CEDMOA consistently outperforming state-of-the-art algorithms in terms of solution quality, convergence speed, and stability. The visual threat detection system successfully processes imagery and converts detection results into actionable path planning constraints.

5.2. Discussion

The practical significance of this work lies in its ability to provide robust, efficient, and adaptable solutions for complex three-dimensional path planning challenges in complex environments. The system’s unified framework makes it highly suitable for real-world UAV applications where both predefined obstacles and unpredictable threats must be considered. This research contributes to the advancement of UAV technology by bridging the gap between visual perception and intelligent path optimization, providing a feasible way for future developments in autonomous navigation systems.

The integration of visual threats via YOLO-based detection profoundly influences the final UAV trajectory by introducing a dynamic soft constraint that imposes Gaussian-decay penalties around real-time detected objects, effectively guiding the CEDMOA optimizer to deviate from potentially hazardous areas not accounted for in static threat models. Unlike baseline approaches that rely solely on predefined static obstacles and threats, the proposed method dynamically reshapes the trajectory in response to unknown or moving threats such as airborne obstacles or ground-based hazards identified in live imagery, resulting in smoother, more adaptive paths that maintain greater separation distances while preserving overall efficiency in path length and maneuverability. This visual perception layer significantly enhances trajectory safety and robustness in complex, real-world environments with evolving conditions, as evidenced by experimental validations showing collision-free navigation amid both static terrain and dynamic visual threats, underscoring the method’s superior value for practical autonomous UAV operations where traditional static-only planning would fail or produce overly conservative routes.

5.3. Limitations and Future Work

The verification was conducted through software simulation in Matlab R2025a, and the work presented in this paper can fully demonstrate the feasibility of the application. Future research will focus on considering more practical movement limitations of UAVs, as well as the combination of actual properties and algorithms, in order to achieve more practical and applicable results.