Research on Path Planning Methods and Characteristics of Urban Unmanned Aerial Vehicles Under Noise Constraints

Highlights

What are the main findings?

• Under noise threshold constraints, UAVs tend to increase their altitude above ground level (AGL) to maintain noise compliance, and this altitude adjustment exhibits a clear stepwise upward pattern. As the UAV source noise level increases or the noise threshold becomes more stringent, the required AGL correspondingly rises and may conflict with the prescribed maximum AGL altitude, thereby creating a trade-off between noise compliance and altitude limitations.
• Adjusting the noise threshold can significantly influence the feasibility of UAV trajectory planning under noise constraints. For a given urban scenario with specified local noise limits, the maximum allowable UAV source noise level can be further identified. This offers a quantitative basis for evaluating the noise-compliant operational feasibility of UAVs with different noise characteristics in the target area.

What are the implications of the main findings?

• A tiered regulatory framework can be established by jointly considering UAV noise characteristics and requirements adapted to the local needs, thereby enabling differentiated AGL flight altitude, depending on UAV noise source strength.
• Under specific built-environment scenarios and local noise limits, UAV entry limits and operational restriction policies can be established, thereby enabling the matching between UAV noise classes and local noise limits prior to mission dispatch and ensuring noise compliance.

Abstract

This study proposes TNAP-DDQN, a deep reinforcement learning method for urban low-altitude UAV path planning under residential noise threshold constraints. With time cost and safety risk as the optimization objectives, operational constraints such as collision risk and maximum AGL altitude are incorporated to achieve coordinated optimization of noise compliance, operational safety, and efficiency. To mitigate action space contraction and training instability induced by multiple constraints, a Noise-Degradation-Mask-based Action Bias Network (NDM-ABN) is introduced at the action selection layer. A three-tier degradation scheme prevents empty candidate sets, while bias-based decision making is applied to approximately tied actions to stabilize the policy. Moreover, multi-step prioritized experience replay (PER) improves sample efficiency and long-horizon return modeling, and potential-based reward shaping (PBRS) transforms sparse constraint signals into auxiliary rewards. Simulation results indicate that: (1) NDM-ABN is the key module for stabilizing the noise-exposure process by suppressing high-noise actions; (2) the required AGL is related to the UAV source noise level and local noise limits, implying the need for differentiated AGL altitude classes; and (3) the maximum admissible UAV source noise level increases as the threshold is relaxed. The proposed method provides quantitative guidance for noise-entry and AGL altitude regulation, while future work will incorporate additional metrics (e.g., A-weighted equivalent sound level) to better capture noise fluctuations and short-term peaks.

1. Introduction

With the rapid development of unmanned aerial vehicle (UAV) technologies, UAVs have been increasingly deployed in civil and commercial applications, particularly in parcel delivery, agricultural monitoring, and environmental protection. However, as UAV operations in urban environments continue to expand, noise has gradually become a major public concern. Therefore, how to effectively mitigate noise-related impacts within UAV path planning has become an urgent problem.

To address various operational challenges in UAV missions, researchers have developed a wide spectrum of path planning algorithms over the past decade. However, as systematically documented in recent comprehensive surveys, these methodological advances primarily focus on conventional optimization objectives, leaving acoustic noise constraints largely unaddressed. Jin et al. [1] proposed a new quadrotor cooperative source-seeking algorithm aimed at solving the source localization problem in environments affected by disturbances and communication constraints. The algorithm combines formation control and gradient-free optimization, guiding a group of quadrotors towards the optimal point of a scalar field without relying on gradient estimation. Experimental and simulation results validated the effectiveness of this method, demonstrating that the quadrotors successfully reach the target source position while maintaining formation, ensuring robustness and stability.

Nevertheless, as systematically documented in recent comprehensive surveys, these methodological advances primarily focus on conventional optimization objectives, leaving acoustic noise constraints largely unaddressed. Meng et al. [2] traced the evolution from classical graph-based algorithms to modern AI techniques, emphasizing the shift toward energy-aware planning and multi-UAV cooperative strategies. Complementing this, Sheltami et al. [3] further mapped these algorithmic paradigms—approximately 30% classical, 29% meta-heuristic, 18% AI-based, and 23% hybrid—onto diverse application domains.

Nevertheless, most existing studies on UAV path planning primarily optimize time cost or safety risk and generally overlook noise-related impacts. Jin et al. [4] explored self-triggered distributed control for fixed-wing UAV formations under velocity and overload constraints. The study proposed a UAV kinematic model incorporating wind disturbances and designed an inner–outer loop control structure. By utilizing a self-triggered sampling mechanism, communication overhead was reduced, and the control method ensured that UAVs remained within desired speed and overload ranges while maintaining formation. Stability of the system was proven using the small-gain theorem, and experimental results demonstrated the method’s effectiveness in addressing various constraint conditions. Regarding conventional path-planning methods, Zammit et al. [5] compared the performance of the A* and RRT* algorithms in three-dimensional environments, highlighting the advantages of A* in generating shorter paths and achieving high computational efficiency. Brown et al. [6] proposed a trajectory generation method that combines quintic polynomials and multi-objective particle swarm optimization for high-altitude long-endurance UAV maritime radar surveillance tasks. This method optimizes fuel consumption, detection probability, and revisit time, achieving globally optimal trajectory planning.

Subsequently, Cheriet et al. [7] conducted an in-depth analysis of A*, RRT*, and PSO for path-planning efficiency in urban 3D scenarios; through experimental evaluations under different flight conditions, their work provides references for selecting appropriate path-planning algorithms. However, although these methods perform well in terms of path efficiency, they do not explicitly consider noise constraints into path planning. To enhance operational safety, Primatesta et al. [8] proposed a risk-aware path-planning method that incorporates risk factors into route selection to reduce potential hazards, and Zhang et al. [9] further developed a ground-risk-map-based planning approach by jointly optimizing ground risk and flight cost, emphasizing the need to consider multiple risk factors in urban environments. Feng et al. [10] introduced a risk-assessment framework for low-altitude airspace path planning by integrating multi-dimensional risk quantification, aiming to improve both flight safety and efficiency. To enhance operational safety, Pang et al. [11] proposed a risk-based UAV traffic network model that classifies and quantifies risks in urban airspace, thereby enabling a comprehensive evaluation framework for path planning. Pol Mestres et al. [12] introduced a motion planning algorithm that integrates Control Lyapunov Functions (CLFs) and Control Barrier Functions (CBFs) with Rapidly Exploring Random Trees (RRT). The algorithm generates safe and dynamically feasible paths, ensuring collision-free navigation that can be executed with CLF-CBF controllers. The proposed method was shown to efficiently generate executable paths, avoiding complex boundary value problem computations, and was validated through simulation and hardware experiments, demonstrating faster execution times compared to traditional methods. In addition, Song et al. [13] incorporated an artificial potential field into an improved A* algorithm to dynamically adjust the trajectory, improve path smoothness, and support safe UAV flight in complex urban environments. Niknejad et al. [14] proposed Da SP-RRT, which constructs data-driven invariant sets to guarantee safety while optimizing trajectory costs, addressing the safety–performance trade-off in unknown dynamic environments.

Additionally, to address frequent path conflicts and excessive computational overhead in large-scale multi-UAV cooperative operations within urban low-altitude airspace, Lu et al. [15] proposed a conflict-free path-planning method that integrates three-dimensional jump point search (3D-JPS) with an incremental update mechanism. However, these studies primarily focus on homogeneous UAV swarms in urban environments. Zhang et al. [16] proposed a distributed time-varying optimization algorithm that guides multiple UAVs in dynamic target tracking while ensuring collision avoidance through a fixed-time consistency protocol. This approach addresses the challenges of global optimization and fast convergence, offering a new perspective for path planning in multi-UAV missions. Wang et al. [17] proposed a PSE-D model-based cooperative path planning framework for UAV-USV systems in anti-submarine missions, decomposing the problem into waypoint determination and trajectory generation phases to address the heterogeneous planning requirements between aerial and surface platforms.

With the rise of deep reinforcement learning (DRL), DRL-based path-planning approaches have also attracted substantial attention. Zhou [18] proposed a DRL-based path-planning algorithm that significantly improves efficiency and safety by optimizing the reward design. Xie et al. [19] employed DRL to address the complexity of dynamic environments, enhancing the real-time performance and flexibility of path planning. Han et al. [20] proposed a PSO-DQN method that strengthens the learning capability of the Q-network, thereby improving planning efficiency. Thanh Thi Nguyen et al. [21] reviewed the application of DRL to path planning in autonomous systems and highlighted both challenges and recent advances, indicating that DRL methods have stronger potential than traditional approaches. Xu Zhenyang et al. [22] proposed the TCP-DQN algorithm, which substantially improves planning efficiency and safety performance for low-altitude vehicles in complex environments via goal-oriented curriculum learning and a prioritized replay strategy. In application domains such as logistics delivery and environmental monitoring, Deng et al. [23] proposed a path-planning method that combines an improved beetle antennae search (BAS) with simulated annealing (SA); by dynamically adjusting the search step size and antenna sensing length, the method improves search accuracy and real-time capability while ensuring flight safety and obstacle avoidance. Hu et al. [24] proposed a hybrid path-planning method that integrates an improved ant colony optimization algorithm (IACO) with rapidly exploring random trees (RRT), focusing on logistics delivery in dense urban-building environments to improve search efficiency and path accuracy. Cao et al. [25] further developed an IACO–RRT hybrid method to enhance planning accuracy and strengthen applicability in complex urban environments.

Although the above studies have made progress in terms of safety, efficiency, and obstacle avoidance, their optimization models generally do not sufficiently account for in-flight noise constraints. As UAV operations in urban environments continue to expand, the noise-related impacts of UAV operations on urban environments and residents have increasingly attracted academic attention. Christian et al. [26] reported, through a comparative study, that UAV noise is more disturbing than road traffic noise under comparable sound pressure levels. Bulusu et al. [27] investigated the noise-related impacts as sociated with large-scale UAV operations in low-altitude urban airspace via simulation, indicating that UAV noise may induce pronounced perceptual impacts on urban residents. Whelchel et al. [28] examined the noise generated by multirotor UAVs under different operating modes and found that flight mode and receiver location significantly affect noise perception. Torija et al. [29] further investigated the influence of UAV hovering on urban soundscapes and discussed the interaction between UAV noise and other environmental noise sources in cities, such as road traffic noise. Torija et al. [30] suggested that, although the sound pressure level of UAV noise is lower than that of conventional aircraft noise, its distinctive timbre and frequency characteristics may still impose potential community impacts. In addition, Schaffer et al. [31] provided a systematic review and noted that the psychoacoustic effects of UAV noise are more complex than those of traditional transportation noise. Hui et al. [32] showed that UAV noise during hovering and forward flight is strongly associated with cognitive disturbance and human emotional responses, particularly in relatively quiet environments. Collectively, these studies demonstrate that UAV noise can induce non-negligible noise-related impacts in urban settings.

Against this backdrop, a growing body of work has begun to incorporate noise-related impacts into path planning. Bian et al. [33] evaluated the environmental impacts of UAV noise in urban airspace via virtual flight simulations, highlighting the critical role of flight trajectories in noise propagation. To effectively reduce noise pollution, Tan et al. [34] proposed a low-noise UAV trajectory planning method that optimizes flight routes based on a noise assessment platform to mitigate environmental impacts during operations. In the same year, Tan et al. [35] further investigated UAV facility siting to maximize logistics efficiency while reducing noise impacts. Tan et al. [36] also proposed, through virtual flight simulations, a low-noise path-planning method aimed at reducing UAV noise pollution via rational route design. In addition, Zhang et al. [37] proposed a multi-objective optimization approach that jointly considers flight cost, noise pollution, and safety risk to optimize low-altitude urban logistics networks. Finally, Chen et al. [38] presented a real-time trajectory planning method that integrates noise control and safety risk.

Despite the growing body of research on low-noise path-planning methods, two important limitations remain. First, local noise limits have not been explicitly incorporated into path-planning constraints, making it difficult for the resulting trajectories to satisfy practical regulatory requirements in noise-sensitive areas. Second, most existing studies do not jointly address the trade-off between safety risk and time cost. To address these limitations, this study investigates UAV path planning in a simplified urban environment constructed from realistic urban building geometries under explicit local noise-limit constraints, with the objective of minimizing safety risk and flight time.

The main contributions of this study are threefold. First, local noise limits are explicitly incorporated into the path-planning framework, thereby improving the practical relevance of the planned trajectories under urban regulatory requirements. Second, this study performs path planning with safety risk and flight time as optimization objectives, ensuring local noise compliance while effectively coordinating noise compliance with other objectives. Third, the proposed TNAP-DDQN algorithm integrates the Noise Degradation Mask-based Action Bias Network (NDM-ABN), a three-level degradation mechanism, and a bias-based decision strategy to enhance action feasibility and policy stability under multiple constraints, while a hybrid experience replay mechanism is introduced to improve sample efficiency and long-term reward learning.

The remainder of this paper is organized as follows. Section 2 introduces the problem formulation and related constraints. Section 3 presents the proposed method. Section 4 reports the experimental setup and results analysis. Section 5 concludes the paper and discusses future research directions.

2. Materials and Methods

Three-dimensional UAV path planning in low-altitude urban airspace is subject to multiple operational constraints and multi-objective trade-offs in practical deployment. Rules such as the maximum AGL altitude, safety separation, and local noise limits jointly shrink the feasible flight domain and the available action set, thereby increasing the difficulty of 3D path search and decision learning. To clarify, all quantities related to altitude in this study are defined as the height above ground level (AGL).

Accordingly, this study formulates the problem as a Markov decision process (MDP) within a reinforcement learning framework. Under this formulation, the state space, action space, and reward function constitute the core elements of the training interaction environment. Specifically, the state space explicitly encodes key information related to the UAV, the environment, and relevant operational constraints; the action space defines a discrete set of three-dimensional maneuvers; and the reward function provides step-wise feedback and maps operational constraints into learnable optimization signals. In UAV path-planning models, noise can be incorporated in different ways. For example, it may be imposed as a strict feasibility constraint, introduced as an explicit cost term in the overall objective function, or incorporated as a penalty term to guide learning. These different treatments correspond to different modeling priorities and may lead to different optimization behaviors. In this study, local noise limits are first treated as explicit constraints to ensure that the generated trajectories satisfy clear noise-compliance requirements in residential environments. At the same time, a noise penalty is further introduced into the reward design to provide additional guidance for learning within the feasible region. This treatment is adopted because the objective of this study is not to directly trade off noise compliance against other objectives, but rather to carry out path planning under mandatory noise constraints while still improving efficiency and safety. Furthermore, building on the multi-objective cost modeling, this study introduces potential-based reward shaping to transform time cost and safety risk into more continuous process rewards, thereby strengthening the guidance during learning. The following sections will further describe the above modeling components and design details.

2.1. Problem Modeling and Optimization Objectives

In this study, the optimization problem is clearly defined as generating a feasible and optimal three-dimensional flight path for UAVs used in logistics delivery in low-altitude urban environments under local noise limits, with the objective of minimizing time cost and safety risk. To achieve this, several constraints are considered, including noise thresholds, collision risk, maximum AGL (Above Ground Level), and safety distance. After completing the problem modeling, the core objective of this study is to utilize the proposed TNAP-DDQN algorithm to solve this specific constrained optimization problem, ensuring the optimal path is found in complex urban environments while balancing flight efficiency and safety. As shown in Figure 1, the optimization framework includes state-action space modeling, definition of the objective function and constraints, and the application of a deep reinforcement learning (DRL) algorithm to solve the problem.

2.2. State–Action Space Modeling

This study employs a three-dimensional grid-based representation to model static obstacles in low-altitude urban environments, as illustrated in Figure 2. Specifically, a cubic airspace of size $a\times b\times c$ is first defined, and a Cartesian coordinate system is established with the origin $O$. Considering both building heights and computational resource constraints, the grid resolution $\mathsf{\Delta }$ is determined, and the airspace is uniformly discretized into $n$ grid cells. Each grid cell is classified as either a free cell or an obstacle cell. To this end, an indicator variable ${\phi }_i$ is introduced to denote the $i$th grid cell located at coordinates $\left(x_i,y_i,z_i\right)$: if the cell contains a static obstacle, then ${\phi }_i=0$, indicating an obstacle cell; otherwise, ${\phi }_i=1$, indicating a free cell.

The UAV action space $A$ is defined as the set of all feasible movement commands to adjacent grid cells, comprising 26 discrete actions in total (as shown in Figure 3). In the figure, the current grid cell (purple) and all reachable neighboring cells (gray) are used to provide an intuitive illustration of the action space.

2.3. Multi-Objective Modeling and Potential-Based Reward Shaping

In this study, trajectory planning is formulated with flight time and safety risk as the core optimization objectives. Flight time directly determines mission execution efficiency, whereas safety risk reflects the potential threat posed by the UAV to the ground. These two objectives are often mutually constraining: risk avoidance typically increases flight time, while pursuing the shortest path may lead to elevated risk. By jointly optimizing both objectives, this study aims to identify an effective balance between efficiency and safety. The total flight time is calculated as follows:

$$C_{\mathrm{time}}=\min \sum _n^{i=1}{\displaystyle \frac{\sqrt{{\left(x_{i+1}-x_i\right)}^2+{\left(y_{i+1}-y_i\right)}^2+{\left(z_{i+1}-z_i\right)}^2}}{v_i^{i+1}}}$$

(1)

$$v_i^{i+1}=\left\{\begin{array}{ccc}{v}_{\mathrm{h}}& z_{i+1}=z_i& \\ v_{\mathrm{u}}& z_{i+1}>z_i& i\in [1,n]\\ v_{\mathrm{d}}& z_{i+1}<z_i& \end{array}\right.$$

(2)

The variable $v_i^{i+1}$ represents the UAV’s speed when transitioning from the $i$th grid to the $i+1$th grid. The speed parameter is assumed to remain constant within each individual grid-cell transition and only takes different values according to the motion state. Specifically, $v_{\mathrm{h}}$, $v_{\mathrm{u}}$, and $v_{\mathrm{d}}$ represent the UAV’s speed during level flight, climb, and descent, respectively.

The safety-risk cost adopted in this study refers to the cost associated with pedestrian and ground-vehicle casualties caused by a UAV crash, and its unit is expressed as the number of fatalities per flight hour. This cost is denoted by $C_{safe}(x_i,y_i,z_i)$ and consists of two components: the pedestrian fatality risk $c_{safe\_p}(x_i,y_i,z_i)$ and the ground-vehicle occupant fatality risk $c_{safe\_v}(x_i,y_i,z_i)$ [38,39]:

$$C_{safe}(x_i,y_i,z_i)=c_{safe\_p}(x_i,y_i,z_i)+c_{safe\_v}(x_i,y_i,z_i)$$

(3)

Here, $C_{safe}(x_i,y_i,z_i)$ denotes the safety-risk cost associated with the $i$-th grid cell at the 3D location $(x_i,y_i,z_i)$. Specifically, $c_{safe\_p}(x_i,y_i,z_i)$ and $c_{safe\_v}(x_i,y_i,z_i)$ represent the safety-risk costs for ground pedestrians and vehicle occupants, respectively. The safety risk faced by ground pedestrians arises from a continuous process, i.e., UAV loss of control, collision with a pedestrian, and the subsequent injury. The corresponding risk cost is calculated as follows [39]:

$$c_{safe\_p}(x_i,y_i,z_i)=P_{\mathrm{UAV}}\times N_{\mathrm{hit}}^{\mathrm{p}}(x_i,y_i,z_i)\times R_{\mathrm{p}}(x_i,y_i,z_i)$$

(4)

$$N_{\mathrm{hit}}^{\mathrm{p}}(x_i,y_i,z_i)=S_{\text{\_}\mathrm{hit}}\times {\sigma }_{\mathrm{p}}(x_i,y_i)$$

(5)

$$R_p(x_i,y_i,z_i)={\displaystyle \frac{1}{1+\sqrt{\frac{\alpha }{\beta }}{\left(\frac{\beta }{E_{imp}}\right)}^{\frac{1}{4S_c(x_i,y_i,z_i)}}}}$$

(6)

$$v=\sqrt{{\displaystyle \frac{2mg}{R_1{S}_{\text{\_}\mathrm{hit}}\rho }}\left(1-{\mathrm{e}}^{-{\displaystyle \frac{h{R}_1{S}_{\text{\_}\mathrm{hit}}\rho }{m}}}\right)}$$

(7)

Equations (4)–(7) quantify the ground-pedestrian safety-risk cost associated with a UAV crash originating from the $i$th grid cell. Following the risk-modeling framework in [39], the pedestrian risk cost is expressed as a function of the UAV system failure rate $P_{\mathrm{UAV}}$, the number of affected pedestrians $N_{\mathrm{hit}}^{\mathrm{p}}(x_i,y_i,z_i)$, and the corresponding fatality rate $R_{\mathrm{p}}(x_i,y_i,z_i)$, where the latter depends on both impact kinetic energy and the ground-shielding effect. The impact kinetic energy E_imp of the falling UAV is known as $E_{imp}={\displaystyle \frac{1}{2}}m{v}^2$. In this formulation, the number of affected pedestrians is determined by the ground impact area and the local population density, while the fatality model further incorporates the shielding coefficient $S_{\mathrm{c}}$, lethality energy-threshold parameters $\alpha$ and $\beta$, UAV mass m, ground-impact velocity $v$, and UAV AGL $h$. The model formulation and the associated parameter settings adopted in this study are both based on [39]. Specifically, $S_{\mathrm{c}}$ is set to 0.5, $\alpha$ and $\beta$ are set to ${10}^6\mathrm{J}$ and $100\mathrm{J}$ respectively, the aerodynamic drag coefficient $R_1$ is set to 0.3, and the air density $\rho$ is set to $1.225\mathrm{kg}/{\mathrm{m}}^3$. Similarly, the risk-cost model for vehicle occupants is also formulated following the framework in [39], as detailed below:

$$c_{safe\_v}(x_i,y_i,z_i)=P_{\mathrm{UAV}}\times N_{\mathrm{hit}}^{\mathrm{v}}(x_i,y_i,z_i)\times R_{\mathrm{v}}(x_i,y_i,z_i)$$

(8)

$$N_{\mathrm{hit}}^{\mathrm{v}}(x_i,y_i,z_i)=S_{\text{\_}\mathrm{hit}}\times {\sigma }_{\mathrm{v}}(x_i,y_i)$$

(9)

Here, Equation (8) computes the safety-risk cost for in-vehicle occupants resulting from a UAV crash originating from the $i$-th grid cell and impacting vehicles. This risk cost is still formulated as the product of the UAV system failure rate $P_{\mathrm{UAV}}$, the number of affected vehicles $N_{\mathrm{hit}}^{\mathrm{v}}(x_i,y_i,z_i)$, and the occupant fatality rate $R_{\mathrm{v}}(x_i,y_i,z_i)$. The number of affected vehicles is calculated by Equation (9), which depends on the impact coverage area $S_{\text{\_}\mathrm{hit}}$ and the regional vehicle density ${\sigma }_{\mathrm{v}}(x_i,y_i)$. The occupant fatality rate $R_{\mathrm{v}}(x_i,y_i,z_i)$ is derived from the ratio of “annual average traffic fatalities” to “annual average traffic accidents”.

To alleviate the sparse-reward issue and provide continuous optimization guidance, this study adopts a potential-based reward shaping approach. Specifically, potential functions are constructed for the two core optimization objectives, i.e., flight time and safety risk, to quantify the objective-related cost at any intermediate state $S_t$. After normalization and fusion, these potentials form the total potential function. In this study, the flight time cost and safety risk cost are first normalized and then combined without introducing an additional subjective weighting coefficient. This treatment is adopted because the main focus of this work is the design of a noise-constrained path planning method, rather than the calibration of value preferences between efficiency and safety. In practice, different weighting settings may reflect varying operational priorities. To avoid introducing an additional source of subjective bias, this study employs a normalized and unweighted fusion of the two cost terms. This mechanism transforms sparse and delayed terminal returns into dense gradient signals that are directly coupled with step-wise decisions, thereby significantly improving learning efficiency and policy stability. The following sections first present the mathematical formulations of the two optimization objectives and then detail the corresponding cost-type potential function designs:

$$\mathsf{\Phi }(S_t)={\tilde{C}}_{time}(S_t)+{\tilde{C}}_{safe}(S_t)$$

(10)

Here, ${\tilde{C}}_{time}(S_t)$ and ${\tilde{C}}_{safe}(S_t)$ denote the normalized values of the corresponding metrics. The time-cost term is defined based on an estimate of the remaining flight time to the target. This estimate is obtained by computing the three-dimensional Euclidean distance between the current state and the target and dividing it by the UAV average flight speed $v_{\arg }$. The formulation is given as follows:

$$\hat{T}(S_t)={\displaystyle \frac{D_{curr\_goal}}{v_{\arg }}}$$

(11)

The normalized time-cost term ${\tilde{C}}_{\mathrm{time}}(S_t)$ is given by the following expression:

$${\tilde{C}}_{\mathrm{time}}(S_t)={\displaystyle \frac{\hat{T}(S_t)}{1.2\times T_{\max }}}$$

(12)

Here, the normalization factor $T_{\max }$ is computed using the maximum possible flight distance in the 3D map (estimated by the diagonal length) and the UAV’s minimum climb speed, and a moderate margin is further introduced to cover detours in practical flight. This design ensures that the time-cost term remains on a stable numerical scale.

The safety-risk cost term is jointly determined by the UAV AGL and a shielding correction. Its value is computed using the safety-risk model in Equation (3), where the indicator variable $\theta$ is used to denote whether a building exists directly beneath the UAV: if no building is present beneath the UAV, then $\theta =1$; otherwise, $\theta =0.25$ [40].

$$C_{safe}(S_t)=\theta \times C_{safe}(x_i,y_i,z_i)$$

(13)

The normalized safety-risk cost is given by the following equation:

$${\tilde{C}}_{safe}(S_t)={\displaystyle \frac{C_{safe}(S_t)}{C_{safe}(S_t,R_{\max })}}$$

(14)

Here, $C_{safe}(S_t,R_{\max })$ is the reference value computed by Equation (4) at the maximum allowable AGL altitude of 120 m, which is used to normalize the risk cost onto a stable numerical scale. Based on this, the potential-based shaped reward function adopted in this study is formulated as follows:

$$F\left(S_t,S_{t+1}\right)=\mathsf{\Phi }(S_t)-\mathsf{\Phi }(S_{t+1})$$

(15)

2.4. Reward Function Design

To meet practical requirements for urban UAV path planning and to ensure that the UAV can accomplish missions efficiently and safely in complex environments, the reward function is composed of five terms: (i) a baseline reward, which defines task progression and termination conditions; (ii) a collision penalty, which penalizes collision events; (iii) an altitude penalty, which constrains the AGL flight altitude; (iv) a goal-progress reward, which improves sample efficiency and accelerates convergence; and (v) a noise penalty, which accounts for noise-exposure control while optimizing path efficiency and safety.

(i)

Baseline reward

The baseline reward mechanism is constructed as follows: a high reward is given when the agent reaches the goal state, indicating task completion; otherwise, a negative reward is assigned to encourage the UAV to reach the goal as quickly as possible. The specific formulation is given by the following equation:

$$R_{base}=\left\{\begin{array}{c}{R}_{goal}\hspace{1em}\hspace{1em}ifstate=goalstate\\ BR\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}otherwise\end{array}\right.$$

(16)

Here, $R_{goal}=100$ represents the goal-reaching reward coefficient, and $BR=-0.1$ denotes the baseline reward.

(ii)

Collision penalty

The collision penalty function is used to impose a negative reward for behaviors that bring the UAV close to obstacles. Its formulation is as follows:

$$R_{obstacle}=\left\{\begin{array}{c}\begin{array}{c}-5,\hspace{1em}\hspace{1em}\hspace{1em}if{D}_{obstacle}\le T{H}_1\\ -3,\hspace{1em}\hspace{1em}ifT{H}_1<D_{obstacle}\le T{H}_2\end{array}\\ \begin{array}{c}-1,\hspace{1em}\hspace{1em}ifT{H}_2<D_{obstacle}\le T{H}_3\\ 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}if{D}_{obstacle}>T{H}_3\end{array}\end{array}\right.$$

(17)

In this equation, $D_{obstacle}$ denotes the Euclidean distance from the current state to the nearest obstacle, and $T{H}_1$, $T{H}_2$, and $T{H}_3$ are distance thresholds used for risk grading, with values set to 1, 2, and 3 grid cells, respectively.

(iii)

AGL penalty

The AGL penalty function constrains the UAV to fly within the desired AGL range. Its specific formulation is as follows:

$$R_{height}=\left\{\begin{array}{c}-2if z<IMH\\ 0if IMH\le z\le MHC\\ -4if z>MHC\end{array}\right.$$

(18)

In this equation, $z$ represents the current AGL, $IMH=50 \mathrm{m}$ is the minimum desired AGL [37], and $IMC=120 \mathrm{m}$ is the maximum allowable AGL.

(iv)

Goal-progress reward

The goal-progress reward is calculated based on the change in Euclidean distance between the UAV and the target. Specifically, the reward value increases as the distance difference decreases from the previous step to the current step. The specific formulation is given by the following equation:

$$R_{goal}=4\times \left(\left(D_{prev\_goal}-D_{curr\_goal}\right)/D_{step\_len}\right)$$

(19)

In this equation, $D_{prev\_goal}$ and $D_{prev\_goal}$ represent the Euclidean distances from the previous state and the current state to the target, respectively, and $D_{step\_len}$ denotes the actual displacement length between two adjacent steps. $D_{step\_len}$ is used to normalize the distance change, thereby preventing bias in the reward magnitude caused by different action step sizes.

(v)

Noise penalty

Considering that noise is a significant constraint for low-altitude urban flight, this study constructs the noise penalty function based on the noise propagation attenuation model [38,41]. Its formulation is as follows:

$$L_p(r)=L_s\left(r_0\right)-10{\log }_{10}(r^2/r_0^2)$$

(20)

$$P_{noise}=-0.5\times \left(L_p(r)-L_T\right)$$

(21)

$$R_{noise}=\max \left(P_{noise},-10\right)$$

(22)

In this formulation, Equation (20) is derived from the noise propagation attenuation model according to Ref. [41] (Equation (21) chapter 1). In this model, $L_p(r)$ represents the sound pressure level at the receiver, $L_s$ denotes the reference sound pressure level of the UAV noise source, and $r$ is the propagation distance between the receiver and the noise source. $r_0$ is the reference distance, set to 1 m in this study, which is the sound pressure level. To better illustrate the relationship between the UAV, the noise source, and the receiver, a diagram (Figure 4) is provided.

According to the measurement data reported by relevant literature [42], the actual tested UAV was the Prioria Hex, with a takeoff mass of 7.3 kg, and the measured sound pressure level at the receiver was $L_p(r)=65 \mathrm{dB}$ when the propagation distance from the noise source to the receiver was $r=15 \mathrm{m}$. Based on the attenuation relationship in Equation (20), this measured value is equivalently converted to the reference sound pressure level of the UAV noise source at $r_0=1 \mathrm{m}$, which is approximately $L_s(r_0)=89 \mathrm{dB}$, and used as the noise source for the calculations.

Theoretically, the size of the sound source energy should be represented by sound power. However, since there is a one-to-one correspondence between sound pressure level and sound power level ($L_w=L_p\left(r\right)+20\lg \left(r\right)+11$), this study will use the sound pressure level at 1 m, $L_s(r_0=1\mathrm{m})$, to discuss the different sound source energy levels. Additionally, it is assumed that the noise source is uniformly distributed and omnidirectional.

$L_T$ denotes the local noise limit. According to the daytime limit for Class 1 areas specified in

Table 1. Performance Changes with Progressive Module Addition.

Module Added	Flight Time (/s)	Safety Risk Value	Average Sound Pressure Level (/dB)	Maximum SPL at the Nearest Building Façade (/dB)
DDQN (Baseline)	251.40	2.4118 × 10−4	60.85	69
+ Noise Penalty	222.08	3.0051 × 10−4	60.17	69
+ NDM	217.79	2.0237 × 10−4	56.45	51.39
+ ABN	235.00	2.1900 × 10−4	52.62	59.45
+ MSL	228.86	2.2067 × 10−4	52.11	59
+ PER	195.37	2.3564 × 10−4	52.66	54.38

, “Equivalent Sound Level Limits for Environmental Noise in Urban Areas” [43], $L_T$ was set to 55 dB. Here, if no obstacles are present, $r=120 m$ is used when there is no obstruction; if obstacles exist, $r$ is the Euclidean distance from the sound source to the nearest obstacle. Equations (21) and (22) are used for the noise penalty, where a penalty is imposed if the noise exceeds the local noise limit, whereas no penalty is applied when it remains below the local noise limit. To prevent the penalty from becoming too large and causing training instability, which would negatively impact path planning, the penalty term is capped at −10. Finally, the environmental baseline reward is summed with the multi-objective potential-based shaping terms described earlier to form the total reward function used for training. The specific formulation is as follows:

$$R_{total}=R_{base}+R_{obstacle}+R_{height}+R_{goal}+R_{noise}+F\left(S_t,S_{t+1}\right)$$

(23)

3. Algorithm

3.1. Improved DDQN Algorithm Based on Noise Constraints

To address the challenges of high-dimensional state spaces, sparse rewards, and decision oscillations caused by approximately tied action values in three-dimensional urban low-altitude UAV path planning, this study proposes the TNAP-DDQN algorithm (Figure 5), based on the Double DQN framework [44]. TNAP stands for Threshold-Noise-Aware Path Planning. The method adopts a staged training strategy to improve convergence stability: the action-bias mechanism is not activated during the warm-up phase, while it is gradually introduced in the stable phase, alongside adjusted update settings and a low-frequency soft target-network update.

To explicitly incorporate noise constraints into the decision process and reduce oscillatory action switching, a Noise-Degradation-Mask-Based Action Bias Network (NDM-ABN) is introduced at the action-selection layer. This mechanism rapidly evaluates the noise compliance of candidate actions and masks actions that violate local noise limits. Compared with existing constrained reinforcement learning methods, the innovation of NDM-ABN lies in its hierarchical degradation mechanism, which represents a novel approach to action masking. The three-tier degradation strategy gradually relaxes the noise constraint’s strictness, avoiding action space collapse under overly strict noise constraints while ensuring training continuity.

Moreover, NDM-ABN introduces bias decisions for actions with the same value, enhancing action separability and stabilizing policy decisions. This innovation effectively addresses the issue of selecting actions with identical values, a challenge commonly encountered by standard exploration strategies like epsilon-greedy, and improves learning stability and exploration efficiency through refined bias adjustments.

Finally, to improve sample efficiency and the accuracy of long-term reward estimation, the algorithm introduces a Hybrid Experience Replay Mechanism, which samples at a preset ratio of 70:30. In total, 70% of the samples are drawn from the standard buffer, which stores single-step transition experiences, ensuring learning stability. The remaining 30% are drawn from the prioritized multi-step experience buffer, which stores experience trajectories with accumulated multi-step rewards. These samples are prioritized according to the absolute value of the temporal difference error (TD-error), allowing the algorithm to efficiently utilize high-information samples and accelerate the propagation of long-term rewards. This approach effectively balances local update stability and the effective dissemination of long-term rewards.

3.2. Noise-Aware Action Filtering and Bias-Based Decision Mechanism

In three-dimensional grid-based path planning, where actions must simultaneously satisfy both geometric feasibility and noise constraints, this study performs a two-stage filtering process on the 26 discrete actions at each decision step. First, actions that lead to out-of-bounds or collision scenarios are eliminated. Then, using a precomputed building distance transformation table, the sound pressure levels corresponding to the next states of each action are quickly estimated, and actions that exceed the local noise limit are masked. This process confines the search space within the feasible set under dual constraints. The method adopts a staged training strategy to improve convergence stability. During the warm-up phase, the action-bias mechanism is not activated so as to avoid amplifying value-estimation errors before the value network becomes sufficiently stable. In the stable phase, the action-bias mechanism is gradually introduced, together with adjusted update settings and a low-frequency soft target-network update, to enhance training controllability and convergence stability.

To explicitly incorporate noise constraints into the decision process, a noise-aware action-filtering mechanism is first introduced at the action-selection layer. Candidate actions that violate local noise limits are directly masked out. To avoid action-space collapse under overly strict noise thresholds, a hierarchical degradation mechanism is further employed. When the number of feasible actions falls below a predefined lower bound, a more relaxed feasibility-protection threshold is activated; if the feasible set remains insufficient, the filtering process is further degraded to retain only geometrically feasible actions. Through this design, decision continuity can be preserved under complex urban constraints and highly restrictive noise conditions.

After action filtering, approximate ties may still occur among the Q-values of the remaining candidate actions, which may lead to random switching and trajectory oscillation during policy execution. To alleviate this issue, the proposed Noise-Degradation-Mask-Based Action Bias Network (NDM-ABN) is introduced as a tie-breaking module. Specifically, the bias term is applied only within the approximately tied candidate set, whereas the original greedy selection is retained when no such tie exists. Therefore, this module serves as a local refinement of action selection rather than a global reordering of all candidate actions. Since it operates only at the decision layer and does not alter the underlying Bellman update, the fundamental value-learning structure of Double DQN remains unchanged.

The value of masked actions is defined as follows:

$${\tilde{Q}}_{main}(s_t,a_t)=\left\{\begin{array}{cc}{Q}_{main}(s_t,a_t),& m_t(a)=1\\ -\infty ,& m_t(a)=0\end{array}\right.$$

(24)

Here, ${\tilde{Q}}_{main}(s_t,a_t)$ represents the value estimation of the output action $a_t$ by the main network at state $s_t$. $m_t(a)$ is used to differentiate between valid and invalid actions. However, under the combined influence of multiple constraints, the candidate actions retained after mask-based filtering often exhibit very similar value estimates, resulting in an approximate tie phenomenon. In such cases, if the standard greedy selection strategy is still applied directly, even small numerical fluctuations may cause frequent switching between actions at adjacent decision steps, thereby leading to local path jitter or even oscillations in the overall trajectory. To address this issue, the NDM-ABN is introduced into the decision-making process for approximate-tie actions. First, the candidate action set corresponding to approximate ties is defined as follows:

$$C_t=\left\{a\mid m_t(a)=1,\left|{\tilde{Q}}_{main}(s_t,a_t)-{\tilde{Q}}_{\max }\right|<\epsilon \right\}$$

(25)

where ${\tilde{Q}}_{main}(s_t,a_t)$ denotes the maximum value among the currently feasible actions, and $\epsilon$ represents the threshold used to determine approximate ties. In this study, $\epsilon$ is set to 0.0001. Accordingly, when only one action in the candidate set is significantly superior, the algorithm still follows the original greedy selection strategy directly. Only when multiple actions fall into an approximate-tie state does the bias network participate in the decision scoring process, thereby avoiding unnecessary interference with non-tie samples.

To explicitly incorporate noise constraints into action-level decision making, the predicted next state of each candidate action is further evaluated in terms of noise, which is formulated as follows:

$$L(a_t)=L({s^{\prime }}_t)=L_p(r;L_w,\mathrm{env}),r=d({s^{\prime }}_t,d({s^{\prime }}_t))$$

(26)

Here, $d({s^{\prime }}_t)$ represents the building receiver point closest to the UAV’s next state, $d({s^{\prime }}_t,d({s^{\prime }}_t))$ denotes the propagation distance between the UAV and this receiver point, $\mathrm{env}$ represents the urban environment, and $L(a_t)$ indicates the sound pressure level at the receiver point. To avoid all actions being masked due to strict noise thresholds during takeoff or landing, the algorithm introduces a buffer zone mechanism: if the next state is within the buffer radius of the starting or ending point, the threshold constraint is temporarily relaxed; otherwise, strict threshold assessment is applied. The specific computation is as follows:

$$\tau ({s^{\prime }}_t)=\left\{\begin{array}{cc}{L}_n,& d_{\mathrm{start}}({s^{\prime }}_t)\le R_{b}or{d}_{\mathrm{goal}}({s^{\prime }}_t)\le R_b\\ L_T,& \mathrm{otherwise}\end{array}\right.$$

(27)

Here, $L_T$ is the noise threshold, $L_n=65 \mathrm{dB}$ is the buffer threshold, and $R_b=5$ is the buffer radius defined based on unit grid cells. The action filtering mechanism first checks whether the next state is within the buffer zone. It then determines the validity of the action based on the corresponding threshold.

To prevent situations where no valid actions are available due to overly strict noise constraints in obstacle-dense areas, this study introduces a minimum valid action count constraint $N_{\min }=1$ and a feasibility guarantee threshold $FPT=60 \mathrm{dB}$. If $C_t^{\left(h\right)}\ge N_{\min }$ is met, subsequent decisions are made directly within the feasible region $C_t^{\left(h\right)}$; otherwise, the feasibility guarantee threshold is activated to ensure policy continuity. The final output is the set of valid actions selected based on the feasibility guarantee threshold. Accordingly, the final decision candidate set is defined as follows:

$${C^{\prime }}_t=\left\{\begin{array}{ccc}{C}_t^{(h)},& \left|C_t^{(h)}\right|\ge N_{\min }& \\ C_t^{(s)},& \left|C_t^{(h)}\right|<N_{\min }\mathrm{and}\left|C_t^{(s)}\right|\ge N_{\min }& \\ C_t^{(0)},& \left|C_t^{(0)}\right|<N_{\min }& \end{array}\right.$$

(28)

where $C_t^{(h)}$, $C_t^{(s)}$, and $C_t^{(0)}$ denote the action sets satisfying the strict noise threshold, the feasibility-guarantee threshold, and the geometric feasibility condition only, respectively, and $N_{\mathrm{m}\mathrm{i}\mathrm{n}}$ represents the minimum number of valid actions. Through this hierarchical degradation mechanism, the algorithm can effectively prevent the action set from becoming completely empty in complex urban environments.

For the approximate-tie actions within the final candidate set $C_t^{\mathrm{\prime }}$, this study further introduces the action-bias term generated by the NDM-ABN to locally adjust the main-network value, and the decision score is constructed as

$$Q^{dec}(s_t,a_t)={\tilde{Q}}_{main}(s_t,a_t)+\alpha \cdot {\hat{a}}_i$$

(29)

Here, $\alpha =0.02$ is the scaling factor used to adjust the impact of the bias on the ${\tilde{Q}}_{main}(s_t,a_t)$ value. ${\hat{a}}_t$ is the bias value generated by the action attention network.

In addition to the decision stage, the bias network also needs to learn how to generate effective bias terms during training. To this end, the temporal-difference target is adopted in the training stage to construct the supervisory signal:

$$Q_{\mathrm{target}}(s_t,a_t)=R_t+\gamma \cdot Q_{\mathrm{target}}({s^{\prime }}_t,{a^{\prime }}_t)$$

(30)

where $R_t$ denotes the immediate reward, $\gamma$ is the discount factor, and $Q_{\mathrm{target}}({s^{\prime }}_t,{a^{\prime }}_t)$ represents the value estimate of the next state–action pair given by the target network. Based on this target value, the parameters of the bias network are updated through error backpropagation, enabling the output bias term to gradually acquire discriminative capability in terms of long-term return.

3.3. Prioritized Multi-Step Experience Replay Mechanism

To enhance the sample efficiency and stability of Double DQN in three-dimensional path planning, this study introduces the Prioritized n-Step Replay Mechanism (see Figure 6). This mechanism accelerates the propagation of long-term feedback through (n)-step accumulated rewards and improves the utilization of high TD-error samples by incorporating Prioritized Experience Replay (PER). Additionally, Importance Sampling (IS) weights are used to correct the bias introduced by non-uniform sampling, further improving training performance. It should be noted that, although PER can significantly improve sample efficiency by replaying informative transitions more frequently, it also changes the sampling distribution from uniform replay to a priority-dependent distribution. As a result, the sampled mini-batch no longer strictly follows the original experience distribution, which may introduce estimation bias into value updates and, in principle, affect the stability and convergence behavior of policy learning if no correction is applied.

This study adopts the following (n)-step accumulated reward definition to accelerate the propagation of long-term reward signals:

$$R_t^{(n)}=\sum _{n-1}^{k=0}{\gamma }^{k}clip(r_{t+k})\hspace{1em}\hspace{1em}{r}_{t+k}\in \left[-R,R\right]$$

(31)

Here, $\gamma =0.95$ is the discount factor, which is set in this study; $clip(r_{t+k})$ denotes the reward clipping, with the reward clipping threshold set to 50, used to suppress extreme rewards and stabilize training. The $n$-step TD target is selected by the main network for action, and its value is evaluated by the target network. It is defined as follows:

$$y_t^{(n)}=\left\{\begin{array}{c}{R}_t^{(n)},\mathrm{if}{s}_{t+n}\mathrm{is}\mathrm{terminal}\\ R_t^{(n)}+{\gamma }^n\cdot Q_{\mathrm{target}}(s_{t+n},\arg \underset{a}{\max }{Q}_{\mathrm{mian}}(s_{t+n},a)),\mathrm{otherwise}\end{array}\right.$$

(32)

Here, $Q_{\mathrm{target}}$ is the target network, $Q_{\mathrm{main}}$ is the main network, and $s_{t+n}$ is the state after $n$ steps. $\arg \underset{a}{\max }{Q}_{\mathrm{main}}(s_{t+n},a)$ is used to select the optimal action at the current state, which is then evaluated by the target network to mitigate Q-value overestimation. The TD error used for prioritized replay is defined as

$${\delta }_n=y_t^{(n)}-Q_{\mathrm{main}}(s_t,a_t)$$

(33)

To improve the utilization of high-value samples, Prioritized Experience Replay (PER) assigns sampling probabilities based on the Temporal Difference (TD) error, giving samples with larger errors higher sampling priority. The priority $p_i$ in PER is defined as follows:

$$p_i={(|{\delta }_n|+ϵ)}^{\alpha }$$

(34)

In Prioritized Experience Replay (PER), the sampling probability of a sample is determined by its TD error. Here, the constant $ϵ=0.01$ is used to prevent zero priority, and the exponent $\alpha =0.6$ is used to adjust the level of prioritization. A larger $\alpha$ leads to a stronger deviation from uniform sampling, which increases replay efficiency for informative samples but may also aggravate sampling bias. Therefore, the prioritization mechanism improves learning efficiency at the cost of introducing a distribution mismatch between replayed samples and the original behavior distribution. Based on the above priority, the sampling probability $P(i)$ of a sample is given by

$$P(i)={\displaystyle \frac{p_i}{{\displaystyle \sum _{i=1}^N}{p}_i}}$$

(35)

Here, $N$ represents the total number of samples in the experience pool. To correct the estimation bias introduced by non-uniform sampling in Prioritized Experience Replay (PER), this study employs Importance Sampling (IS) for weighted correction. Specifically, for a sample $i$, its Importance Sampling weight $w_i$ is defined as follows:

$$\begin{array}{c}\beta =\min \left(1.0,\beta +{\beta }_{\mathrm{increment}}\right)\\ w_i={\left({\displaystyle \frac{1}{N\cdot P(i)}}\right)}^{\beta }\end{array}$$

(36)

Here, $\beta =0.4$ is the bias correction exponent, and ${\beta }_{\mathrm{increment}}=0.01$ is the bias correction increment. The importance sampling weight is used to reduce the estimation bias introduced by non-uniform sampling. Specifically, when $\beta$ gradually approaches 1 during training, the update becomes closer to an unbiased estimate, thereby alleviating the potential adverse effect of PER-induced sampling bias on convergence in the later training stage. In this way, the proposed framework seeks to balance the sample-efficiency advantage of PER and the training-stability requirement of Double DQN. To prevent excessively large weights from causing gradient instability, the weights are normalized. The specific formula is as follows:

$${\tilde{w}}_i={\displaystyle \frac{w_i}{\max w_j}}$$

(37)

Here, $w_i$ is the computed IS weight, ${\tilde{w}}_i$ is the normalized IS weight, and $\max w_j$ is the maximum weight among all samples. The normalization step is further used to avoid excessively large correction terms, which could otherwise amplify gradient fluctuations and reduce training stability. Therefore, the IS weighting and normalization together serve as the main bias-compensation mechanism in the PER framework adopted in this study. Additionally, to optimize the balance between exploration and exploitation, this study employs an adaptive (n)-step mechanism: the step size is dynamically adjusted based on the real-time distance between the UAV and the target. When the distance is large, the step size is increased to accelerate exploration, and as the UAV approaches the target, the step size is decreased to facilitate fine control. The specific adjustment formula is as follows:

$$n_{\mathrm{step}}=max\left(n_{step\_\min },n_{step\_\min }+\mathrm{norm}\mathrm{distance}\times \left(n_{step\_\max }-n_{step\_\min }\right)\right)$$

(38)

Here, $n_{step\_\min }=2$ and $n_{step\_\max }=3$ represent the minimum and maximum step sizes, respectively, and $\mathrm{norm}\mathrm{distance}$ is the normalized distance from the current state to the target. This formula allows the step size to be adaptively adjusted based on the task progress (distance to the target).

4. Experiments and Evaluation

4.1. Simulation Environment Setup

Key training parameters are summarized in Table 1. The experimental environment was established based on a typical urban area in Chengdu to represent the operational context and noise-related impacts of low-altitude urban flight. Geographic information was obtained from Baidu Maps, and a three-dimensional static environment model was constructed using a grid-based method, with a grid resolution of 10 m [45] and an airspace altitude range of 0–150 m. In this study, the ground surface within the selected Chengdu study area is assumed to be flat, i.e., the ground elevation relative to mean sea level is treated as identical throughout the entire area. The distribution of urban buildings and the constructed static environment model are presented in Figure 7. The key training parameters and UAV system parameters are provided in Appendix B (

Table A1. Relevant Parameter Settings.

Parameter	Value
Learning Rate	0.001
Discount Factor	0.95
Exploration Rate	0.9
Exploration Decay Factor	0.995
Minimum Exploration Rate	0.01

and

Table A2. UAV Parameter Settings.

Parameter	Value
Maximum Flight Altitude (/m)	120
UAV Empty Weight (/kg)	7.3
$S_{\mathrm{hit}}/{\mathrm{m}}^2$	0.0188
$v_{\mathrm{h}}/(\mathrm{m}·{\mathrm{s}}^{-1})$	18
$v_{\mathrm{d}}/(\mathrm{m}·{\mathrm{s}}^{-1})$	3
$v_{\mathrm{u}}/(\mathrm{m}·{\mathrm{s}}^{-1})$	5
$S_c$	0.5
$P_{\mathrm{UAV}}$	3.42 × 10−4

), where the UAV configuration was determined based on relevant literature [42,46] and a specific DJI model.

In this study, obstacles are not modeled as convex hulls. Compared with convex-hull-based modeling, the grid representation is more convenient for obstacle description and collision checking, whereas convex hull methods require additional geometric preprocessing and increase computational complexity in the three-dimensional environment. Therefore, a grid-based obstacle modeling method is adopted in this study to balance modeling practicality and training efficiency.

Based on the constructed safety risk model, the safety risk in a low-altitude environment is calculated. The road network density and vehicle ownership data for Chengdu’s administrative roads are obtained from relevant reports such as the “2025 China Urban Road Network Density and Operational Status Monitoring Report.” From this, the vehicle density per unit grid in the road network is estimated to be 0.622 vehicles/m². The population of Chengdu is obtained from sources such as the Chengdu Statistics Bureau website, where the total population is 15.98 million and the total area is 14,335 square kilometers, resulting in a population density of 0.027875 people/m² per unit grid area. Additionally, based on data from the National Bureau of Statistics, the annual average number of traffic fatalities and the annual number of traffic accidents are used to calculate the average number of fatalities per accident, which is approximately 0.228 fatalities per accident.

4.2. Algorithm Comparison Experiment

To systematically assess the impact of different improvement modules on UAV path planning performance, this study employs a stepwise module-comparison design. Starting with the basic Double DQN model, while keeping the environment, network structure, and hyperparameters consistent, the Noise Penalty, NDM, ABN, MSL, and PER modules are sequentially introduced. By constructing progressively enhanced comparison models, the influence of each module on policy learning can be clearly identified. Table 1 shows the performance changes in the path after each module is added, and Figure 7 presents the time-series curve of the sound pressure level at the nearest receiver points along the path for each model.

As shown in Table 1, the results reflect a progressive module-addition process, in which each module is introduced on top of the previous configuration. The baseline DDQN shows the worst noise performance, with the highest average SPL of 60.85 dB and the highest maximum SPL at the nearest building façade of 69 dB. Here, the maximum façade SPL is evaluated only during the steady-flight phase, excluding takeoff and landing. After adding the noise penalty, the average SPL decreases only slightly to 60.17 dB, while the maximum façade SPL remains 69 dB, indicating that the reward-level noise penalty alone has limited effect. After introducing NDM, the average SPL, safety risk, and maximum façade SPL all decrease noticeably, showing that this module can effectively suppress high-noise actions.

With the further addition of ABN and MSL, the average SPL remains at a relatively low level of about 52 dB, while flight time and safety are further improved. Among them, MSL achieves the lowest average SPL of 52.11 dB. After incorporating PER, the flight time is further reduced to 195.37 s, which is the best among all configurations, while both the average SPL and maximum façade SPL remain low. Overall, the final enhanced model achieves a good balance among flight efficiency, noise control, and safety.

Figure 8 further reflects the differences in noise compliance among the ablation models. The baseline DDQN remains above the 55 dB threshold over most of the trajectory and exhibits several pronounced peaks, indicating poor noise control. After adding the noise penalty, some high-noise segments are reduced, but repeated threshold exceedances are still observed in the middle section of the trajectory, suggesting that reward-level penalization alone is insufficient to achieve stable noise control.

After introducing NDM, the SPL curve decreases significantly and becomes smoother, with most segments remaining below the threshold, indicating that this module can effectively improve noise-compliance stability. With the further addition of ABN and MSL, the curve becomes more stable overall, showing that these modules further enhance the continuity and consistency of low-noise decisions. After incorporating PER, although slight local fluctuations appear in the final stage, the overall performance is still clearly better than that of the baseline and the noise-penalty-only model.

It should also be noted that, because different modules generate different final trajectories, the SPL data are no longer updated once the path reaches the target point in the later stage. As a result, some areas in the figure show missing SPL data.

Figure 9 compares the path-planning behaviors of different ablation models in an urban environment. As noise-related modules are gradually introduced, clear changes can be observed in both the horizontal trajectory and altitude distribution. Figure 9a shows that, after noise constraints are introduced, the models tend to avoid dense building areas in the horizontal plane. Figure 9b shows that the models generally begin climbing earlier and maintain higher flight altitudes to reduce noise exposure near buildings.

For the baseline DDQN, the path is relatively direct, with only slight deviations near building boundaries, and the flight altitude is mainly maintained at 50–60 m, indicating that it mainly relies on local obstacle avoidance. After introducing the Noise Penalty, the cruising altitude increases to about 60–70 m, suggesting that the model begins to reduce noise exposure by increasing altitude.

With the further addition of ABN, NDM, and MSL, the preference for low-noise regions becomes more evident, and the cruising altitude increases overall. After incorporating PER, the horizontal path becomes smoother and the altitude remains at about 90–100 m, showing better continuity and coordination.

Overall, with the progressive addition of noise-related modules, the path-planning strategy gradually evolves from low-altitude direct obstacle avoidance to a noise-constrained planning mode that combines horizontal detours and altitude compensation, thereby achieving a better balance among noise control, trajectory continuity, and operational efficiency.

Figure 10 shows the evolution of the total reward per episode during training. Overall, all models exhibit a rapid reward increase in the early stage and then gradually become stable, indicating that all models can learn feasible policies. However, their convergence behaviors differ clearly. The baseline DDQN and the Noise Penalty model show relatively large fluctuations in the early and middle stages, indicating weaker training stability.

After introducing NDM, the reward curve becomes noticeably smoother and the model enters the stable stage earlier, showing that this module can effectively reduce training oscillations. With the further addition of ABN, the reward evolution becomes more stable, indicating that it helps improve action-selection consistency and training stability. After incorporating MSL, the reward evolution becomes more continuous, suggesting that multi-step return modeling enhances long-term reward estimation. Finally, after adding PER, the model shows the most stable behavior in the late training stage, indicating that prioritized experience replay further improves training stability and sample utilization efficiency.

Based on the analysis of each module’s contribution in the ablation experiment, this study further conducts an algorithm comparison experiment. Figure 11 presents the path comparison results of TNAP-DDQN, B-APF-DQN [47], S-JPS [48], and Dueling-DQN [49], while

Table 2. Algorithm Comparison Results.

Algorithm	Flight Time (/s)	Safety Risk Value	Average Sound Pressure Level (/dB)	Maximum SPL at the Nearest Building Façade (/dB)
TNAP-DDQN	195.37	2.3564 × 10−4	52.66	54.38
B-APF-DQN	171.91	2.6059 × 10−4	59.04	69
S-JPS	172.31	2.9053 × 10−4	58.85	62.97
Dueling-DQN	163.31	2.4950 × 10−4	56.85	69

summarizes the performance of each algorithm.

TNAP-DDQN has a flight time of 195.37 s, which is 13.64%, 13.39%, and 19.63% higher than that of B-APF-DQN, S-JPS, and Dueling-DQN, respectively, indicating that it adopts a more conservative trajectory strategy under explicit noise constraints and therefore incurs additional time cost.

However, this time cost leads to better noise-control performance. The average SPL of TNAP-DDQN is 52.66 dB, which is 6.38 dB, 6.19 dB, and 4.19 dB lower than that of B-APF-DQN, S-JPS, and Dueling-DQN, respectively. Its maximum SPL at the nearest building façade is also the lowest, at 54.38 dB. Here, the maximum façade SPL is evaluated only during the steady-flight phase, excluding takeoff and landing. These results indicate that TNAP-DDQN can more effectively reduce noise exposure during flight.

In terms of safety, the safety-risk value of TNAP-DDQN is 2.3564 × 10⁻⁴, which is 9.57%, 18.89%, and 5.56% lower than that of B-APF-DQN, S-JPS, and Dueling-DQN, respectively. Overall, TNAP-DDQN shows better noise-control performance and lower safety risk than the compared algorithms. Although the differences in computation time are small, clear differences still exist in flight time, noise, and safety metrics.

Figure 12 shows the time-series results of the sound pressure level (SPL) at the nearest receiver points along the path for each algorithm, highlighting the differences in noise exposure across the planned trajectories. Overall, the SPL curve of TNAP-DDQN is the lowest and most stable. For most of the route, its SPL remains clearly below the 55 dB threshold, indicating that the proposed method can more effectively incorporate noise constraints into path planning and maintain a stable low-noise state along the trajectory.

In contrast, the SPL curves of B-APF-DQN, S-JPS, and Dueling-DQN are generally above the threshold for most of the cruise phase, with only a few intervals close to the threshold. These algorithms are less effective in maintaining stable noise compliance and show more pronounced noise peaks in certain intervals.

All algorithms exhibit an instantaneous SPL increase during the takeoff and climb phases, caused by the transient effect of the changing distance between the UAV and the receiver, rather than a reflection of noise control during the cruise phase. The advantage of TNAP-DDQN is that it maintains lower and more stable SPLs during the cruise phase, avoiding prolonged periods above the threshold seen in other algorithms.

It should be noted that due to the different paths generated by each algorithm, some algorithms do not have SPL data in the later stages of the path. This reflects performance differences across algorithms.

4.3. Comparison and Analysis of Paths with Different Noise Constraints

To further investigate the impact of noise constraints on urban low-altitude UAV path planning and to analyze the required AGL under different UAV source noise levels, comparative experiments were conducted by adjusting the local noise limits and the source noise intensity while keeping the environmental parameters unchanged. Based on the preceding analysis, the UAV noise source was modeled as a point source in the subsequent experiments, and a reference sound pressure level of approximately 89 dB at a reference distance of 1 m was adopted as the baseline. On this basis, to compare the influence of different source noise levels on path-planning results, the UAV source noise level was discretized at 3 dB intervals, ranging from 65 dB to 95 dB. These discrete values were used as the parameter $L_s\left(r_0\right)$ in Equation (20) to compute the received sound level $L_p(r)$ at an arbitrary distance $r$.

The local noise limits were defined according to the standards for acoustic environment functional zones [43], including the Class 0 ($L_T=50 \mathrm{dB}$) and Class 1 ($L_T=55 \mathrm{dB}$) limits, while the Class 2 limit ($FPT=60 \mathrm{dB}$) was used as the feasibility protection threshold. In this study, these limits correspond to the daytime limits specified in the acoustic-environment functional-zone standard [43].

(i)

Noise threshold $L_T=55 \mathrm{dB}$

Appendix B (

Table A3. Comparison of Results Under Different Noise Sources ($L_T=55 \mathrm{dB}$).

Pressure Level at 1 m Along the Source (Ls (r0 = 1)/dB)	Flight Time (/s)	Safety Risk Value	Average Sound Pressure Level (/dB)
95	217.15	2.3453 × 10−4	55.36
92	199.70	2.2510 × 10−4	53.34
89	195.37	2.3564 × 10−4	52.66
86	182.22	2.0348 × 10−4	51.68
83	174.38	2.1900 × 10−4	51.10
80	169.05	2.1900 × 10−4	48.49
77	165.37	2.0126 × 10−4	47.34
74	164.02	2.6502 × 10−4	46.45
71	163.61	2.6668 × 10−4	43.55
68	162.31	2.6003 × 10−4	40.52
65	162.61	2.9663 × 10−4	38.09

) summarizes the path-optimization results under different UAV sound pressure levels for a fixed noise threshold of $L_T=55 \mathrm{dB}$. The table reports the corresponding flight time, safety-risk value, and average sound pressure level at the nearest receiver points for each UAV sound pressure level, thereby providing a quantitative basis for analyzing how the UAV sound pressure level affects trajectory performance under the same threshold constraint.

As shown in Table A3, as the UAV source noise level gradually decreases from 95 dB to 65 dB, the average sound pressure level correspondingly declines from 55.35 dB to 38.09 dB, while the flight time is also reduced overall from 217.15 s to 162.61 s. This indicates that, under higher UAV source noise levels, the path-planning strategy tends to adopt more conservative flight maneuvers in order to satisfy the local noise limits, thereby resulting in a higher time cost. In contrast, when the UAV source noise level is lower, the feasible space under the local noise limits is relatively expanded, and the corresponding constraint cost is reduced accordingly.

It should also be noted that the Safety Risk Value does not exhibit a strictly monotonic trend. This is mainly because, under different UAV source noise levels, the optimization process may produce different flight paths, and the proportions of building-shielded areas and unshielded ground areas traversed by these paths are not necessarily the same. Since ground risk is influenced not only by the altitude above ground level (AGL) but also closely related to the shielding conditions of the underlying area, the Safety Risk Value shows some fluctuations rather than a simple monotonic relationship with the UAV source noise level. Nevertheless, from the overall results, these fluctuations are relatively small, indicating that the safety-risk levels remain generally within a similar range under different source noise conditions.

Figure 13 illustrates the UAV trajectory characteristics under different UAV source noise levels from both the top-view and front-view perspectives. In the top view (Figure 13a), under the same local-noise-limit constraint, the overall horizontal direction of the trajectories remains consistent, while noticeable differences can still be observed in the detailed spatial layouts, especially in the middle and later segments of the routes. This indicates that, under noise constraints, different UAV source noise levels affect the spatial organization of the trajectories, leading to different path configurations while preserving a broadly similar start-to-goal direction.

In the front view (Figure 13b), the AGL profiles exhibit a clear layered platform structure, and the cruising AGL generally increases with the UAV source noise level. Specifically, lower UAV source noise levels (65–77 dB) mainly correspond to relatively low AGL platforms of about 50–60 m. A UAV source noise level of 83 dB corresponds to a cruise platform of about 60–70 m, while 86 dB further increases to about 80–90 m. At 89 dB, the cruise AGL rises to about 90–100 m, whereas 92 dB and 95 dB require significantly higher platforms of approximately 120 m and 140 m, respectively. These results indicate that, as the source becomes louder, the feasible low-altitude corridor is progressively compressed, and the planning strategy therefore becomes more reliant on earlier climbing, maintaining a higher AGL, and sustaining longer high-altitude cruise segments in order to enlarge the propagation distance margin and preserve noise compliance.

Figure 14 shows that, as the UAV source noise level increases, the SPL curves at the nearest receiver points shift upward and progressively approach the 55 dB threshold, indicating a gradual reduction in the available noise-feasibility margin. Under low UAV source noise levels (65–77 dB), the curves remain stably below the threshold over most of the trajectory, whereas under moderate UAV source noise levels (80–86 dB), several key segments become critically close to the threshold. For 89–92 dB, the curves remain near the threshold in multiple segments, suggesting a markedly narrowed noise-feasibility margin; by contrast, 95 dB is the only case that is clearly clustered around the threshold and more prone to local exceedances. In all cases, transient SPL rises are observed during the initial climb and final descent, mainly due to rapid changes in the UAV–receiver distance during these phases. Combined with Figure 13b, these results indicate that the path-planning strategy increasingly relies on a higher AGL to maintain noise compliance.

It is worth noting that some areas in the graph are missing SPL data, as different algorithms produce results based on different SPL calculation methods. As a result, no further noise changes occur in the subsequent steps, leading to missing SPL data in certain areas of the graph. This phenomenon reflects the impact of different SPL calculation methods on the algorithm’s results.

(ii)

Noise threshold $L_T=50 \mathrm{dB}$

Compared to the more relaxed noise threshold $L_T=55 \mathrm{dB}$, the detailed path-optimization results under different UAV sound pressure levels for the stricter threshold are provided in Appendix B (

Table A4. Comparison of Results Under Different Noise Sources ($L_T=50 \mathrm{dB}$).

Pressure Level at 1 m Along the Source (Ls (r0 = 1)/dB)	Flight Time (/s)	Safety Risk Value	Average Sound Pressure Level (/dB)
95	223.68	2.1401 × 10−4	54.30
92	212.73	2.4285 × 10−4	53.17
89	203.74	2.4673 × 10−4	50.91
86	190.14	2.4118 × 10−4	49.06
83	187.70	2.0514 × 10−4	47.89
80	183.20	2.1734 × 10−4	45.87
77	177.86	2.1734 × 10−4	45.13
74	171.85	2.0958 × 10−4	42.76
71	164.44	2.0736 × 10−4	42.02
68	162.61	2.6003 × 10−4	41.36
65	162.31	2.9663 × 10−4	38.13

). The table reports the corresponding flight time, safety-risk value, and average sound pressure level at the nearest receiver points for each UAV sound pressure level, thereby providing a quantitative basis for evaluating how a tighter noise threshold influences trajectory performance.

Similar to the results under the 55 dB threshold, Table A4 shows that, under the stricter threshold of $L_T=50 \mathrm{dB}$, both the average sound pressure level and the flight time generally decrease as the source-noise intensity is reduced. This indicates that tighter noise constraints impose a higher time cost under strong source-noise conditions, whereas lower UAV source noise levels provide a relatively larger noise-feasible margin. The safety-risk value does not vary monotonically, but instead reflects the combined effect of AGL adjustment and the building-shielding conditions beneath the UAV trajectory.

Figure 15 further compares the trajectories under different UAV source noise levels. In the top view (Figure 15a), all cases generally maintain a similar start-to-goal direction, whereas the detailed path layouts vary with the UAV source noise level, indicating that stronger noise constraints require a greater sacrifice of geometric directness. In the front view (Figure 15b), the AGL profiles show a clear stepwise increase with increasing UAV sound pressure level: 65–74 dB mainly corresponds to 50–60 m, 77 dB to 69–70 m, 80 dB to 70–80 m, 83 dB to 80–90 m, 86 dB to 90–100 m, 89 dB to 110–120 m, 92 dB to 130–140 m, and 95 dB to about 150 m. These results indicate that, under the local noise limits, the planning strategy increasingly relies on earlier climbing and sustained high-AGL cruise segments to enlarge the propagation-distance margin and maintain noise compliance.

Figure 16 shows the time-series characteristics of the sound pressure level (SPL) at the nearest receiver points under stricter local noise limits, revealing the impact of different UAV source noise levels on path planning. Overall, none of the curves exceed the feasibility protection threshold (FPT); however, as the noise threshold tightens, the SPL curves gradually approach the threshold line, indicating that the noise feasibility margin is decreasing.

A more detailed analysis shows that, at lower UAV source noise levels (65–74 dB), the SPL curves remain below the 50 dB threshold for most of the flight segments, with minimal fluctuations, indicating that there is still a sufficient noise feasibility margin. When the UAV source noise level increases to 77–86 dB, the curves approach the threshold more frequently, suggesting that the trajectory has entered a critically feasible state. As the UAV source noise level further increases to 89–95 dB, threshold contact and local exceedances become more frequent, reflecting a significant reduction in the noise feasibility margin. This indicates that, at high UAV source noise levels, the path-planning strategy can no longer maintain stable noise compliance throughout the full trajectory.

Some areas in the graph are missing SPL data due to the use of different SPL calculation methods by different algorithms, causing noise changes to stop in subsequent steps, resulting in missing SPL data in certain areas.

(i)

Sensitivity Analysis of Noise Thresholds and Noise Source Levels

Based on the path optimization results under different UAV sound pressure levels and noise threshold conditions, Figure 17, Figure 18 and Figure 19 summarize the variations in required AGL, flight time, and noise exposure.

Figure 17 shows that the required AGL increases stepwise with UAV sound pressure level. The values shown here represent the highest cruise AGL adopted during the stable-flight phase rather than a constant altitude throughout the entire trajectory. Under relatively low UAV sound pressure levels (65–74 dB), the AGL requirements under the 50 dB and 55 dB thresholds are similar, remaining around 50–60 m. As the UAV sound pressure level increases, the stricter 50 dB threshold requires earlier and larger altitude increases than the 55 dB threshold, indicating a stronger reliance on altitude compensation.

Figure 18 shows that, under both threshold settings, the total flight time generally increases with UAV sound pressure level, and this increase is more pronounced under the stricter 50 dB threshold. Under the 50 dB threshold, the flight time rises from about 162.31 s at 65 dB to 223.68 s at 95 dB, whereas under the 55 dB threshold it increases from about 162.61 s to 217.15 s over the same range. The difference between the two cases is relatively small at lower UAV sound pressure levels but becomes increasingly evident as the UAV sound pressure level increases. This indicates that a stricter threshold leads to stronger AGL compensation and, consequently, a higher time cost.

Figure 19 shows that, under both threshold settings, the average SPL at the nearest receiver point increases monotonically with UAV sound pressure level. Under the 50 dB threshold, the average SPL remains below the limit from 65 dB to 86 dB but exceeds the threshold from 89 dB onward. Under the 55 dB threshold, the average SPL remains below the limit up to 92 dB and only slightly exceeds it at 95 dB. These results indicate that the stricter 50 dB threshold causes the trajectories to approach the noise boundary earlier, whereas the 55 dB threshold retains a relatively larger noise-feasibility margin over a wider range of UAV sound pressure levels. It should also be noted that the average SPL reflects only the overall noise level and does not guarantee full-process compliance with the noise constraint during the flight, excluding the takeoff and landing phases. Even when the average value remains below the 50 dB or 55 dB threshold, local or short-term exceedances may still occur in certain critical flight segments. Therefore, further analysis in combination with the maximum SPL at the nearest building façade is still required.

5. Conclusions

This study proposes the TNAP-DDQN method for urban low-altitude UAV path planning under residential noise threshold constraints. The method incorporates multiple operational constraints, including flight time, collision risk, and maximum AGL altitude, to achieve coordinated optimization of noise compliance, safety, and efficiency. To address action-space compression, tied action values, and training instability under multiple constraints, a Noise Degradation Mask-based Action Bias Network (NDM-ABN) is introduced at the action-selection stage. In addition, multi-step prioritized experience replay (PER) and potential-based reward shaping are used to improve sample efficiency and long-term reward learning.

The main conclusions are as follows: First, the ablation and comparison results show that NDM-ABN is the key module in TNAP-DDQN. Compared to using only a noise penalty in the reward function, NDM-ABN directly suppresses high-noise actions during action selection, thereby improving the stability of noise control. Second, the required AGL (Above Ground Level) is significantly affected by the UAV sound pressure level and local noise limits; higher UAV sound pressure levels or stricter noise limits both correspond to higher AGL requirements. This study further provides the corresponding AGL requirements under different UAV sound pressure levels and noise threshold conditions, which can serve as a direct reference for differentiated AGL altitude management of UAVs with different noise characteristics. Third, the results show that stricter noise thresholds significantly reduce the acceptable range of UAV source levels. Under the combined constraints of local noise thresholds and the maximum allowable AGL altitude of 120 m, the maximum acceptable UAV sound pressure level in the scenario considered in this study is 86 dB for the 50 dB noise threshold and 92 dB for the 55 dB noise threshold.

This study provides quantitative guidance for UAV noise-entry management by clarifying the relationship among UAV source noise level, local noise limits, and required AGL. However, the equivalent sound pressure level used in this study cannot fully describe short-term fluctuations and peak variations in noise signals.

Moreover, the simplified point-source assumption, geometric attenuation model, and regular-cuboid obstacle representation improve computational efficiency. However, these simplifications may not fully capture the complexity of acoustic propagation and obstacle shapes in real urban environments, leading to potential deviations between the results and actual conditions. Future work will therefore consider more comprehensive acoustic indicators, higher-precision aerodynamic noise simulation, and more flexible obstacle representation methods such as convex hulls or polyhedral geometric modeling, and field or laboratory measurements for model calibration and validation.