An Improved Deep Q-Learning Approach for Navigation of an Autonomous UAV Agent in 3D Obstacle-Cluttered Environment

Abstract

The performance of the UAVs while executing various mission profiles greatly depends on the selection of planning algorithms. Reinforcement learning (RL) algorithms can effectively be utilized for robot path planning. Due to random action selection in case of action ties, the traditional Q-learning algorithm and its other variants face the issues of slow convergence and suboptimal path planning in high-dimensional navigational environments. To solve these problems, we propose an improved deep Q-network (DQN), incorporating an efficient tie-breaking mechanism, prioritized experience replay (PER), and L2-regularization. The adopted tie-breaking mechanism improves the action selection and ultimately helps in generating an optimal trajectory for the UAV in a 3D cluttered environment. To improve the convergence speed of the traditional Q-algorithm, prioritized experience replay is used, which learns from experiences with high temporal difference (TD) error and avoids uniform sampling of stored transitions during training. This also allows the prioritization of high-reward experiences (e.g., reaching a goal), which helps the agent to rediscover these valuable states and improve learning. Moreover, L2-regularization is adopted that encourages smaller weights for more stable and smoother Q-values to reduce the erratic action selections and promote smoother UAV flight paths. Finally, the performance of the proposed method is presented and thoroughly compared against the traditional DQN, demonstrating its superior effectiveness.

1. Introduction

In recent years, unmanned aerial vehicles (UAVs) have widely been adopted in various real-life scenarios to execute various missions successfully with path planning being the key factor behind their success and the safe flight [1,2,3,4,5]. For autonomous flights, path planning is indispensable for navigating the UAV to various targets to accomplish various missions without human intervention. Path-planning algorithms can be categorized into two main streams: classical or conventional and intelligent algorithms. Classical algorithms include Dijkstra, A-star, dynamic A, rapidly-exploring random trees (RRTs), probabilistic roadmaps (PRMs), potential fields, and genetic algorithms [6,7,8], while the intelligent algorithms are based on reinforcement learning and neural networks predictors [9,10,11].

Conventional path-planning algorithms for UAVs struggle to address the suboptimality in 3D robot environments and face the issue of local optima. To solve such problems, researchers have recently adopted algorithms from reinforcement learning with deep learning, rendering improved and dynamic path-planning methods for mobile robots and UAVs [9,12,13]. In a recent study [14], the authors have proposed a Q-learning-based RL approach for UAV path planning in both static and dynamic environments. By formulating the obstacle assessment model and path-planning task as a Markov decision process (MDP), they define a structured state space, action space, and reward function to optimize trajectory generation. To enhance exploration efficiency, the authors integrate a heuristic-guided ε-greedy policy, which strategically biases action selection toward promising regions of the state space. This modification significantly improves both learning convergence and policy effectiveness compared to random exploration. Empirical results demonstrate that their method outperforms conventional path-planning techniques, including A*, RRT, and standard DQN, in terms of computational efficiency and path optimality. A related study [15] also employs a Q-learning-based RL approach for UAV path planning in environments with both static and dynamic obstacles. The method introduces a distance-based prioritization policy, leveraging the Euclidean distance between the start and goal points to guide action selection. While the study accounts for dynamic obstacles, its policy learning is limited to single-goal scenarios. Similarly, obstacle avoidance and path planning for a UAV have been done using the Q-learning algorithm in [16]. Authors in [17] address the single-goal path-planning problem for UAVs by proposing a deep reinforcement learning (DRL) framework that integrates deep neural networks (DNNs) with an actor-critic architecture. This approach enables the UAV agent to learn an optimal action policy for autonomous trajectory generation. The actor-critic-based DRL framework enhances the agent’s capability to handle high-dimensional state spaces and continuous action domains, leading to more efficient behavior optimization in complex environments. The same has been discussed in a survey study on actor-critic-based reinforcement learning in [18]. In [19], the authors have used DNNs and Q-learning to realize a deep Q-network to learn a policy for UAV path planning. This study incorporates artificial potential fields with the traditional DQN to present an improved variant of the algorithm. Similarly, authors in [20,21,22,23] have discussed DRL-based methods for path planning of various kinds of UAVs. Policy gradient (PG) is another DRL algorithm that is used commonly for various sequential decision-making problems. In [24], authors have presented a two-stage RL framework for multi-UAV path planning using the PG algorithm. Similarly, the study in [25] discusses a deep deterministic policy gradient (DDPG)-based control framework for learning and autonomous decision-making capability for UAVs. The reviewed literature indicates that reinforcement learning methods are increasingly being adopted by researchers to address path-planning challenges for aerial agents.

Traditional Q-learning is a widely adopted reinforcement learning algorithm due to its inherent advantages of algorithmic simplicity and its ability to adapt effectively to dynamic environments [26,27]. However, it faces numerous shortcomings and related problems resulting in suboptimal path planning in high-dimensional robot environments. Figure 1 illustrates the fundamental RL-based UAV path-planning framework. The agent interacts with the UAV environment by selecting actions based on its current policy, while receiving feedback in the form of rewards and updated states. This interaction loop enables the agent to iteratively improve its policy for optimal trajectory planning in complex 3D environments. The key challenges in existing approaches include inefficient action selection, simplistic reward structures, and slow convergence rates [28]. These limitations often result in the generation of random and suboptimal UAV paths in 3D obstacle-cluttered environments [29]. The baseline DQN relies solely on current experiences to update the Q-values, which frequently leads to slow learning and poor policy performance due to data inefficiency and the presence of highly correlated training samples. The DQN with experience replay (DQN-ER) addresses this issue by storing past transitions and sampling them randomly, thereby reducing temporal correlation and stabilizing the training process [30]. This technique enhances policy robustness, particularly during the early exploration phase. However, uniform sampling in experience replay treats all transitions equally, which diminishes the influence of rare but valuable experiences. DQN with reward shaping (DQN-RS) seeks to overcome this by designing structured reward signals that guide the agent’s behavior, even when the goal state is not immediately achieved [31]. Despite its potential, improper reward shaping can lead to unintended learning outcomes. Overall, a persistent limitation across DQN and its variants is their tendency to produce random action selections, which, even when occasionally optimal, often result in suboptimal UAV trajectories.

To solve these problems, this paper proposes an improved deep Q-learning algorithm leveraging the use of deep neural networks. The proposed deep Q-network (DQN-Proposed) algorithm combines multiple enhancements to address different limitations of traditional DQN. The improved action selection by efficiently breaking the ties between equally valued Q-actions renders an optimal path and ultimately the optimal UAV trajectory with minimum length. This modified tie-breaking mechanism ensures consistent behavior when Q-values are similar and prevents random oscillations in policy due to numerical ties in action selection. This mechanism is not deterministic in early training stages due to the stochastic nature of exploration and thus does not cause the agent to prematurely converge to suboptimal policies. Importantly, the agent still relies on Q-value estimates to guide exploration and learning. The tie-breaking rule does not override the learned policy but only aids in disambiguating equally valued actions. In practice, this reduces policy jitters and leads to smoother convergence, without introducing additional risk of being trapped in local optima.

To improve the convergence and stability, we have incorporated the prioritized experience reply and L2-regularization. In traditional experience replay, transitions (state, action, reward, next state) are stored and sampled uniformly during training. Experiences with higher TD errors (i.e., surprising or poorly learned ones) are more likely to be revisited and learned from. Based on temporal difference error, PER introduces a priority score for each transition and focuses on the most informative transitions. Hard-to-learn examples are revisited more frequently, which accelerates the overall learning. This also allows the agent to learn from fewer, high-impact experiences rather than wasting time on well-predicted and low-impact ones. Moreover, rare and high-reward experiences (e.g., reaching a goal) are prioritized, which helps the agent to rediscover these valuable states and improve learning. For better stability, PER allows the use of high-error samples that correct the Q-network more effectively which ultimately prevent it from getting stuck in suboptimal policies.

Path planning in three-dimensional (3D) environments involves many state-action pairs that cause poor generalization. In the proposed Q-learning approach, we have incorporated L2-regularization to prevent the network from memorizing specific paths and to generalize to unseen obstacles and routes. This also encourages smaller weights that lead to more stable and smoother Q-values to reduce the erratic action selections (e.g., sudden direction changes) and encourages smoother UAV flight paths. In complex 3D grids, overfitting to early exploration data can trap the UAV in suboptimal paths. L2 keeps the model flexible enough to adapt to better policies later. Together, these techniques create a more stable, sample-efficient, and goal-directed learning framework that outperforms all other DQN variants in both training and test environments.

The key contributions of this paper are as follows:

1.

An improved Q-learning algorithm based on deep neural networks is proposed for autonomous UAV navigation in complex 3D obstacle-cluttered environments.

2.

A dynamic action-value adjustment mechanism is introduced to enhance action selection, reduce randomness, and promote the generation of more optimal UAV trajectories.

3.

Prioritized experience replay is employed to accelerate convergence by sampling transitions with high temporal difference error thus avoiding the inefficiencies of uniform sampling.

4.

The replay buffer is designed to emphasize high-reward experiences (e.g., goal-reaching states), which allows the agent to revisit and learn more effectively from valuable states.

5.

L2-regularization is integrated into the learning process to constrain weight magnitudes that lead to smoother Q-value updates, reduced erratic behavior, and more stable UAV flight paths.

The remainder of this paper is as follows: Section 2 provides background information related to UAV model and reinforcement learning. Section 3 presents problem formulation and the proposed Q-learning approach to solve the navigation problem of a UAV in a 3D environment. Section 4 elaborates the experimental setting and results followed by a detailed discussion on the results obtained. Finally, Section 5 concludes this study.

2. Background

This section presents a concise yet comprehensive overview of the theoretical foundations essential for understanding the proposed UAV navigation framework. It covers key concepts in quadrotor dynamics, the principles of deep reinforcement learning, and the Q-learning algorithm with emphasis on their relevance to autonomous path planning in complex 3D environments.

2.1. Mathematical Model of a Quadrotor UAV

This section presents a set of well-established equations of a quadrotor model from [32]. This model is used for trajectory generation in Section 4 and acts as an agent in an MDP environment.

The translational states of the quadrotor UAV can be written as follows:

$$\ddot{x}=\left(\mathrm{c}\mathrm{o}\mathrm{s}\psi \mathrm{s}\mathrm{i}\mathrm{n}\theta +\mathrm{c}\mathrm{o}\mathrm{s}\theta \mathrm{s}\mathrm{i}\mathrm{n}\varphi \mathrm{s}\mathrm{i}\mathrm{n}\psi \right){\displaystyle \frac{1}{m}}{U}_1,$$

(1)

$$\ddot{y}=\left(\mathrm{s}\mathrm{i}\mathrm{n}\psi \mathrm{s}\mathrm{i}\mathrm{n}\theta -\mathrm{c}\mathrm{o}\mathrm{s}\psi \mathrm{c}\mathrm{o}\mathrm{s}\theta \mathrm{s}\mathrm{i}\mathrm{n}\varphi \right){\displaystyle \frac{1}{m}}{U}_1,$$

(2)

$$\ddot{z}=-g+\left(\mathrm{c}\mathrm{o}\mathrm{s}\varphi \mathrm{c}\mathrm{o}\mathrm{s}\theta \right){\displaystyle \frac{1}{m}}{U}_1,$$

(3)

where $U_1$ is the total thrust vector produced by the four rotors and controls the vertical motion, $m$ is the mass of the UAV, $g \mathrm{i}\mathrm{s} \mathrm{g}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}$, and $\ddot{x}$, $\ddot{y}$, and $\ddot{z}$ are the second order derivatives of (acceleration terms) of translation states $x, y, z$ of quadrotor UAV.

The rotational states of the quadrotor model can be written as follows:

$$\ddot{\varphi }= \dot{\theta }\dot{\psi }\left({\displaystyle \frac{I_y-I_z}{I_x}}\right)-{\displaystyle \frac{I_r}{I_x}}\dot{\theta }{\mathrm{\Omega }}_r+{\displaystyle \frac{l}{I_x}}{U}_2,$$

(4)

$$\ddot{\theta }=\dot{\varphi }\dot{\psi }\left({\displaystyle \frac{I_z-I_x}{I_y}}\right)+{\displaystyle \frac{I_r}{I_y}}\dot{\varphi }{\mathrm{\Omega }}_r+{\displaystyle \frac{l}{I_y}}{U}_3,$$

(5)

$$\ddot{\psi }=\dot{\varphi }\dot{\theta }\left({\displaystyle \frac{I_x-I_y}{I_z}}\right)+{\displaystyle \frac{1}{I_z}}{U}_4,$$

(6)

where $U_2, U_3$, and $U_4$ are torques around the roll ($\varphi$), pitch ($\theta$), and yaw axes ($\psi$), respectively, which control the UAV’s orientation and rotational motion. The terms $I_x, I_y,$ and $I_z$ denote the inertia tensor, $I_r$ is the rotor inertia, and ${\mathrm{\Omega }}_r$ is the relative speed of cross-coupled rotor pairs, $\ddot{\varphi , }\ddot{\theta }, \ddot{\psi }$ are the acceleration terms, and $\dot{\varphi }, \dot{\theta }, \dot{\psi }$ are the velocity terms of rotational states $\varphi , \theta , \psi$ of the quadrotor, respectively.

2.2. Deep Reinforcement Learning

Reinforcement learning enables an agent to interact with an environment to maximize cumulative rewards. Deep reinforcement learning combines RL with deep learning to handle complex decision-making tasks in high-dimensional environments. Unlike classical RL, which uses Q-tables, DRL employs deep neural networks to approximate Q-values and policies directly from raw inputs. This allows agents to generalize better and learn more effectively in large state and action spaces. Core components of DRL include the agent, which selects actions based on a policy; the environment, which responds with rewards and state updates; and value functions, e.g., $V\left(s\right)$ and $Q(s,a)$, which estimate expected long-term rewards. DRL has driven substantial progress in diverse domains such as robotics, autonomous systems, and strategic game playing.

Q-learning is a fundamental model-free reinforcement learning algorithm that allows an agent to learn how to make optimal decisions in an environment by estimating the Q-function. The Q-function represents the expected cumulative reward for taking a specific action in a given state and then following the optimal policy thereafter. Unlike other methods, Q-learning does not require a prior model of the environment which confers that it does not need to know how the environment works in advance. This makes it highly versatile and applicable to a wide range of problems, from game playing to autonomous decision-making in robotics. In the Q-learning algorithm, the Q-table stores the estimated Q-values for each state-action pair. Initially, these values are set to zero or small random numbers. The agent interacts with the environment by observing the current state, selecting an action (often using an $\epsilon$-greedy strategy to balance exploration and exploitation), and receiving a reward while transitioning to a new state. The Q-value for the chosen state-action pair is then updated using the Bellman equation, which incorporates the immediate reward and the discounted maximum Q-value of the next state. This process repeats over many episodes and gradually refines the Q-values until they converge to their optimal values. Finally, this enables the agent to learn the best actions for each state.

The Q-function is updated using the Bellman equation [33], which expresses the relationship between the current Q-value and the future Q-values. The Q-learning update rule can be described as follows:

$$Q\left(s,a\right)=Q\left(s,a\right)+\alpha \left[r+\gamma \underset{a^{\prime }}{\max }Q\left(s^{\prime },a^{\prime }\right)- Q\left(s,a\right)\right],$$

(7)

where $\alpha$ is the learning rate, $r$ is the immediate reward, $\gamma$ is the discount factor (how much future rewards are valued), $\underset{a^{\prime }}{\max }Q\left(s^{\prime },a^{\prime }\right)$ is the maximum Q-value for the next state $s^{\prime }$.

Q-learning formulation can be described using the Q-value, which represents the expected cumulative reward starting from state $s$, taking action $a$, and following the optimal policy thereafter.

The optimal policy can be extracted as follows:

$$\pi (s)=\underset{a}{argmax }Q\left(s,a\right)$$

(8)

To balance the exploration, actions can be selected using the following $\epsilon$-greedy strategy:

$$\pi (s)= \left\{\begin{array}{cc}random action& with probability \epsilon \\ \underset{a}{argmax }Q\left(s,a\right)& with probability 1-\epsilon \end{array} ,\right.$$

(9)

where $\epsilon$ balances exploration and exploitation.

3. Proposed DQN Framework

3.1. Problem Formulation

Path-planning problem for an aerial agent in a 3D environment can be formulated as a finite episodic MDP assuming that the terms $\mathcal{S},\mathcal{A}\left(s\right),$ and $\mathcal{R}$ are finite and $s\in \mathcal{S}$, $r\in \mathcal{R}$, and $a\in \mathcal{A}\left(s\right)$.

For 3D path planning, an arbitrary finite navigation space can be defined as follows:

$$\mathcal{S}=\left\{\left(x,y, z\right)|x\in \left\{\mathrm{1,2},3,\dots x_n\right\}, y\in \left\{\mathrm{1,2},3,\dots y_n\right\},z\in \left\{\mathrm{1,2},3,\dots z_n\right\}\right\},$$

(10)

where $x, y,$ and $z$ are states and $(x_n\times y_n\times z_n)$ defines the total number of MDP states.

The navigation environment is modeled as a discrete 3D grid indexed by row $i$, column $j$, and altitude $k$, where each cell represents a unique spatial state. The action space comprises six deterministic motions corresponding to unit transitions along the primary axes: forward $(i-1, j, k)$, backward $(i+1, j, k)$, left $(i, j-1, k)$, right $(i, j+1, k)$, upward $(i, j, k+1)$, and downward $(i, j, k-1)$. These actions enable the UAV agent to explore the environment in all three spatial dimensions which form the basis for state transitions within the Markov decision process framework.

The corresponding finite action space is

$$\mathcal{A}=\left\{\left(i+1, j, k\right), \left(i-1, j, k\right), \left(i, j+1, k\right), \left(i, j-1, k\right), \left(i, j, k+1\right), \left(i, j, k-1\right)\right\},$$

(11)

where $(i, j, k)$ are the index terms of $x, y,$ and $z$, respectively.

The reward function is formulated as follows:

$$\mathcal{R}\left(s, a, s^{\prime }\right)= \left\{\begin{array}{cc}100& for goal state (x_g,y_g,z_g)\\ -1& for other states \left(x,y, z\right)\\ -100& for obstacle states \left(x_{obs},y_{obs}, z_{obs}\right) \end{array} ,\right.$$

(12)

where $(x_g,y_g,z_g)$ represent the terminal or the goal state of the navigation task, and $\left(x_{obs},y_{obs}, z_{obs}\right)$ denote the obstacles in the occupancy grid map. The reward values in Equation (12) were chosen to provide clear guidance to the agent during learning: a strong positive reward (+100) for reaching the goal to reinforce success, a strong penalty (−100) for hitting obstacles to discourage collisions, and a small negative reward (−1) for all other states to encourage shorter, more efficient paths.

The simulator generates an occupancy grid map using environmental image data as input. Various kinds of grids, such as squares, rectangles, triangles, and trapezoids can be used to discretize the environment in the occupancy map. In our designed simulator, square boxes are used to discretize the environment and to generate the grid map. The size of the grid cell determines the accuracy, safety, and computational time of the algorithm and therefore this parameter needs to be chosen carefully depending on the real-time dimensions of the environment.

3.2. Proposed DQN Approach

For a given state $s$ and all possible actions $a\in \mathcal{A}\left(s\right)$, we can write the Q-value function for the main DQN as follows:

$$Q\left(s,a;\theta \right)=f_{\theta }\left(s\right).$$

(13)

The term $f_{\theta }\left(s\right)$ represents the output of deep neural network for input state $s=\left(x,y, z\right)$, which is encoded into a feature vector. The output is a vector of Q-values for all possible actions $a\in \mathcal{A}\left(s\right)$ given a state $s$. The term $f_{\theta }\left(s\right)$ also represents the neural network function approximation for Q-value learning.

We can formulate $Q\left(s,a;\theta \right)$ as follows:

$$h^{\left(0\right)}=s,$$

(14)

where $h^{\left(0\right)}$ is input layer of DNN.

For hidden layers $l=1, 2,\dots , n-1$, we can write

$$h^{\left(l\right)}=\sigma \left(W^{\left(l\right)}{h}^{\left(l-1\right)}+b^{\left(l\right)}\right),$$

(15)

where $W^{\left(l\right)}\in {\mathbb{R}}^{h^{\left(l\right)}\times h^{(l-1)}}$, and $b^{\left(l\right)}\in$ ${\mathbb{R}}^{h^{\left(l\right)}}$ for each layer $l$ and $\sigma$ is activation function which is ReLU in our case.

The output layer of DNN is defined as follows:

$$Q\left(s,a;\theta \right)\mathrm{ }=W^{\left(n\right)}{h}^{\left(n-1\right)}+b^{\left(n\right)},$$

(16)

where $\theta =\left\{W^{\left(1\right)}, { b}^{\left(1\right)},\dots , W^{\left(n\right)}, b^{\left(n\right)}\right\}$ represents the learnable network parameters and output $Q\left(s,a;\theta \right)$ is a vector $\left[Q\left(s,a_1\right), Q\left(s,a_2\right), \dots , Q\left(s,a_n\right)\right]$ of size $n$ and $n=\left|\mathcal{A}\left(s\right)\right|$, while $\left|\mathcal{A}\left(s\right)\right|$ represents the number of possible actions.

The DNN is trained to minimize the mean squared error (MSE) between the predicted Q-values $Q\left(s,a;\theta \right)$ and the target Q-values $y$.

The loss function can be written as follows:

$$L\left(\theta \right)= {\mathbb{E}}_{(s, a, r,s^{\prime })~\mathcal{D}}\left[{\left(y_i-Q\left(s_i,a_i;\theta \right)\right)}^2\right].$$

(17)

We can define the target Q-value function $y_i$ as follows:

$$y_i=r_i+ \gamma \underset{a^{\prime }}{\max }Q\left(s_i^{\prime },a^{\prime };{\theta }^{-}\right),$$

(18)

where ${\theta }^{-}$ represents the parameters of target DQN which is a copy of the main DQN that is updated periodically ${\theta }^{-}\leftarrow \theta$. The term $\mathcal{D}$ represents prioritized experience replay buffer which stores past experiences $\left(s,a,r,s^{\prime }\right)$.

To minimize the loss function $L\left(\theta \right)$, the parameters $\theta$ of the DQN are updated using the gradient descent algorithm and the gradient update rule is as follows:

$$\theta \leftarrow \theta -\alpha {\nabla }_{\theta }L\left(\theta \right),$$

(19)

where $\alpha$ is learning rate and ${\nabla }_{\theta }L\left(\theta \right)$ is the gradient of the loss with respect to $\theta$.

The policy $\pi \left(s\right)$ is derived using an $\epsilon$-greedy strategy:

$$\pi \left(s\right)= \left\{\begin{array}{cc}random action& with probability \epsilon \\ \underset{a}{argmax }Q\left(s,a;\theta \right)& with probability 1-\epsilon \end{array}\right.$$

(20)

The variable decay rate can be set as follows:

$${\epsilon }_t={\epsilon }_{min}+\left({\epsilon }_o-{\epsilon }_{min}\right)\times e^{-{\lambda }_{decay }t},$$

(21)

where ${\epsilon }_o$ is initial exploration rate, ${\epsilon }_{min}$ is minimum exploration rate, and ${\lambda }_{decay }$ is decay rate that controls how fast ${\epsilon }_t$ drops.

The goal is to learn the optimal Q-function $Q^{*}\left(s,a;\theta \right)$ that maximizes the cumulative discounted reward:

$$Q^{*}\left(s,a;\theta \right)= \underset{\pi }{\max } \mathbb{E}\left[{\sum }_{t=0}^{\infty }{\gamma }^{t}R(s_t,{\mathrm{a}}_t,{\mathrm{s}}_{t+1})\right]$$

(22)

To implement the prioritized experience replay, each experience $\left(s,a,r,s^{\prime }\right)$ in the replay buffer $\mathcal{D}$ is assigned a priority $p_i$ which determines its likelihood of being sampled. Priority is typically based on the temporal-difference error ${\delta }_i$.

$$p_i=\left|{\delta }_i\right|+ϵ$$

(23)

The temporal-difference error ${\delta }_i$ is

$${\delta }_i=y_i-Q\left(s_i,a_i;\theta \right),$$

(24)

where ${\delta }_i$ denotes the TD error for the $i$-th experience, and $ϵ$ is small positive constant to ensure all experiences have a non-zero chance of being sampled.

The probability $P_i$ of sampling experience $i$ is proportional to its priority as follows:

$$P_i={\displaystyle \frac{p_i^{{\alpha }_{PER}}}{\sum _k{p}_k^{{\alpha }_{PER}}}} ,$$

(25)

where $\alpha$ is a hyperparameter with range $0\le \alpha \le 1$ and controls the tradeoff between greedy prioritization (${\alpha }_{PER}=1$) and uniform sampling ($\alpha =0$). $\sum _k{p}_k^{{\alpha }_{PER}}$ normalizes the probability.

To correct the bias introduced by prioritized sampling, each sampled experience is weighted by the following importance sampling weight [33]:

$${\mathcal{W}}_i={\left({\displaystyle \frac{1}{N}}·{\displaystyle \frac{1}{P_i}}\right)}^{\beta },$$

(26)

where $N$ is the size of the replay buffer, $\beta$ is a hyperparameter that controls the degree of bias correction with range $0\le \beta \le 1$.

From (17) and (26), we can now modify the loss function to account for prioritization as follows:

$$L_{PER}\left(\theta \right)= {\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathcal{W}}_i{\left(y_i-Q\left(s_i,a_i;\theta \right)\right)}^2$$

(27)

To prevent overfitting, we penalize the large weights in DNN using the weight decay approach also known as L-2 regularization as follows:

$$L_{L2}\left(\theta \right)=\lambda {\sum }_{l=1}^n\left({‖W_l‖}_2^2+{‖b_l‖}_2^2\right) ,$$

(28)

where $\lambda$ is the regularization coefficient that controls the penalty strength, $n$ shows the total number of DNN layers, and ${‖ .‖}_2^2$ is squared L2-norm of all weight vectors of DNN.

The modified loss function with L2-regularization becomes

$$L\left(\theta \right)=L_{PER}\left(\theta \right)+L_{L2}\left(\theta \right),$$

(29)

$$L\left(\theta \right)={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathcal{W}}_i{\left(y_i-Q\left(s_i,a_i;\theta \right)\right)}^2+\lambda {\sum }_{l=1}^n\left({‖W_l‖}_2^2+{‖b_l‖}_2^2\right).$$

(30)

Let ${‖\theta ‖}_2^2={\sum }_{l=1}^n\left({‖W_l‖}_2^2+{‖b_l‖}_2^2\right)$, for gradient descent the weight update now includes the regularization gradient as follows:

$$\theta \leftarrow \theta -\alpha \left({\nabla }_{\theta }L\left(\theta \right)-2\lambda \theta \right).$$

(31)

The added term $2\lambda \theta$ encourages weights to stay smaller and prevents the overfitting of DNN.

Due to limited action space, there may exist more than one optimal policy, each generating similar Manhattan cost. However, during trajectory generation these policies may generate different path lengths. In order to render an optimal policy that ultimately reduces the path length, we have proposed a modified Q-value $Q\left(s,a;\theta \right)$ selection (action- selection) mechanism as given subsequently.

It is quite possible to have multiple actions that have near-identical Q-values during the greedy policy phase. Let $\mathcal{A}$ denote the action space, $Q(s,a;\theta )$ represent the Q-value for action $a$ in state $s$, and $\tau$ be a small positive threshold for tie-breaking. The set of near-optimal actions ${\mathcal{A}}_{best}$ is defined as follows:

$${\mathcal{A}}_{best}\leftarrow \left\{a \in \mathcal{A}|Q\left(s,a;\theta \right)\ge \underset{a}{max }Q\left(s,a;\theta \right)-\tau \right\}$$

(32)

From these candidates ${\mathcal{A}}_{best}$, valid actions ${\mathcal{A}}_{valid}$ are filtered to exclude those leading to obstacles or out-of-bounds states:

$${\mathcal{A}}_{valid}\leftarrow \left\{a \in {\mathcal{A}}_{best}|s^{\prime } \mathrm{i}\mathrm{s}\mathrm{ }\mathrm{a}\mathrm{ }\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}\mathrm{ }\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{e}\right\}.$$

(33)

The final action $a^{*}$ is selected by minimizing the Euclidean distance to the $s_g$:

$$a^{*}{\leftarrow argmin}_{a \in {\mathcal{A}}_{valid}}{‖s^{\prime }-s_g‖}_2,$$

(34)

where ${‖ .‖}_2$ is Euclidean distance.

$$a^{*}=\left\{\begin{array}{cc}{argmin}_{a \in {\mathcal{A}}_{valid}}{‖s^{\prime }-s_g‖}_2\mathrm{ }& if \left|{\mathcal{A}}_{valid}\right|>1\\ {argmax}_{a \in \mathcal{A}\mathcal{ }\mathcal{ }}Q\left(s,a;\theta \right) & otherwise\end{array}\right.$$

(35)

This rule integrates with the ε-greedy policy $\pi \left(s\right)$ as follows:

$$\pi \left(s\right)= \left\{\begin{array}{cc}random action from \mathcal{A}\mathrm{ }& with probability \epsilon \\ a^{*}& with probability 1-\epsilon \end{array}\right.$$

(36)

The exploration rate $\epsilon$ follows the Equation (21) and decays over time to balance the exploration and exploitation.

For transitions where tie-breaking occurred, the TD target becomes

$$y_i=r_i+\gamma Q\left(s_{i+1}, \pi \left(s_{i+1}\right);{\theta }^{-}\right),$$

(37)

where $\pi \left(s_{i+1}\right)$ uses the same tie-breaking logic during target computation.

Similarly, the tie-breaking-based policy affects the priority calculation of PER as follows:

$${\delta }_t\leftarrow \left|r_t+ \gamma Q\left(s_{t+1}, \pi \left(s_{t+1}\right);{\theta }^{-}\right)-Q\left(s_t,a_t;\theta \right)\right|.$$

(38)

Based on these theoretical developments, we propose a modified DQN approach in Algorithm 1. This algorithm has been used to conduct different experimental tests in our designed RL simulator. Figure 2 shows the schematic of the proposed DQN approach that has been adopted in this work.

Algorithm 1 Proposed DQN with modified tie-breaking based policy, PER, and L2 regularization
1:	Input:
2:	Environmental states $\mathcal{S}=\left(x,y,z\right),$ actions $\mathcal{A}=\left\{a_1,\dots ,a_6\right\}$$, s_g=\left(x_g,y_g,z_g\right)$
3:	Reward function $R\left(s,a,s^{\prime }\right)$:        $R\left(s,a,s^{\prime }\right)=\left\{\begin{array}{c}+100\\ -100\\ -1\end{array}\begin{array}{c}{ \mathrm{i}\mathrm{f} s}^{\prime }=s_{goal}\\ { \mathrm{i}\mathrm{f} s}^{\prime } \mathrm{i}\mathrm{s} \mathrm{o}\mathrm{b}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{c}\mathrm{l}\mathrm{e}/\mathrm{o}\mathrm{u}\mathrm{t} \mathrm{o}\mathrm{f} \mathrm{b}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\\ \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\right.$
4:	Hyperparameters: $\alpha$$, \epsilon$$, \gamma$$, ϵ$$, {\alpha }_{PER}$$, \beta$$, {\epsilon }_{min}$, ${\epsilon }_o$, $\lambda$, ${\lambda }_{decay },$$\tau$.
5:	Output: Trained Q-network $Q\left(s,a;\theta \right)$
6:	procedure Initialize
7:	Main Q-network $Q\left(s,a;\theta \right)$, target network $Q\left(s,a;{\theta }^{-}\right)$ with ${\theta }^{-}\leftarrow \theta$
8:	Prioritized replay buffer $\mathcal{D}$ with capacity $N$
9:	$t\leftarrow 0$
10:	end procedure
11:	for episode $i=1$ to $M$ do
12:	Reset environment to $s_0$
13:	while $s_t$ is not terminal do
14:	Action Selection (Policy):
15:	if $rand\left(\right)< \epsilon$ then
16:	$a_t\leftarrow$ random action from $\mathcal{A}$
17:	else
18:	$q_{max}\leftarrow \underset{a}{max }Q\left(s,a;\theta \right)$
19:	${\mathcal{A}}_{best}\leftarrow \left\{a \in \mathcal{A}|Q\left(s,a;\theta \right)\ge q_{max}-\tau \right\}$
20:	${\mathcal{A}}_{valid}\leftarrow \left\{a \in {\mathcal{A}}_{best}|s^{\prime } \mathrm{i}\mathrm{s} \mathrm{a} \mathrm{f}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e} \mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{e}\right\}$
21:	if $\left|{\mathcal{A}}_{valid}\right|>1$ then
22:	$a_t{\leftarrow argmin}_{a \in {\mathcal{A}}_{valid}}{‖s^{\prime }-s_g‖}_2$
23:	else
24:	$a_t{\leftarrow argmax}_{a }Q\left(s,a;\theta \right)$
25:	end if
26:	end if
27:	Execute $a_t$, observe $r_t$$, s_{t+1}$
28:	Store Transition:
29:	${\delta }_t\leftarrow \left\{\begin{array}{c}|r_t+\gamma \underset{a^{\prime }}{\mathit{max}}Q\left(s_{t+1},a^{\prime };{\theta }^{-}\right)-Q\left(s_t,a_t;\theta \right)|\\ |r_t+\gamma Q\left(s_{t+1}, \pi \left(s_{t+1}\right);{\theta }^{-}\right)-Q\left(s_t,a_t;\theta \right)|\end{array}\right.\begin{array}{c} without tie\\ for tie\end{array}$
30:	${\left(p_t\right)}^{{\alpha }_{PER}}\leftarrow {\left({\delta }_t+ϵ\right)}^{{\alpha }_{PER}}$
31:	Store $\left({s_t, a_t,r}_t, s_{t+1}, p_t\right)$ $\mathrm{in} \mathcal{D}$
32:	Sample Mini-Batch:
33:	Sample K transitions $\left({s_i, a_i,r}_i, s_{i+1}, p_i\right)$ with $P\left(i\right)\propto$${\left(p_i\right)}^{{\alpha }_{PER}}$
34:	Compute importance sampling weights ${\mathcal{W}}_i\leftarrow {\left({\displaystyle \frac{1}{N}}.{\displaystyle \frac{1}{P_i}}\right)}^{\beta }$
35:	Compute Loss with PER and L2:
36:	for each transition $i$ if $s_{i+1}$ is non-terminal do
37:	$y_i\leftarrow \left\{\begin{array}{c}{r}_i+\gamma \underset{a^{\prime }}{\mathit{max}}Q\left(s_{i+1},a^{\prime };{\theta }^{-}\right)\\ r_i+\gamma Q\left(s_{i+1}, \pi \left(s_{i+1}\right);{\theta }^{-}\right)\end{array}\right.\begin{array}{c} without tie\\ for tie\end{array}$
38:	end for
39:	$L\left(\theta \right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathcal{W}}_i{\left(y_i-Q\left(s_i,a_i;\theta \right)\right)}^2+\lambda \sum _{l=1}^n\left({‖W_l‖}_2^2+{‖b_l‖}_2^2\right)$
40:	Update Parameters:
41:	Perform gradient descent $\theta \leftarrow \theta -\alpha \left({\nabla }_{\theta }L\left(\theta \right)-2\lambda \theta \right)$
42:	Update target network $\theta \leftarrow {\theta }^{-}$
43:	$\mathrm{Recompute} {\delta }_t$ for sampled transitions and update $p_i$ $\mathrm{in} \mathcal{D}$
44:	Decay exploration rate ${\epsilon }_t\leftarrow {\epsilon }_{min}+({\epsilon }_o-{\epsilon }_{min})\times e^{-{\lambda }_{decay }t}$
45:	Set $t\leftarrow t+1$
46:	end while
47:	end for

3.3. Analysis of DQN Convergence

This section provides a formal derivation of the convergence guarantees of DQN based on classical reinforcement learning theory. The proof structure builds upon the contraction property of the Bellman operator and the Banach Fixed-Point Theorem, while incorporating assumptions relevant to function approximation and DQN-specific mechanisms such as target networks and experience replay.

Definition 1

(Bellman Operator) [33]. Let $Q: \mathcal{S}\times \mathcal{A}\to \mathbb{R}$ be an action-value function over states $\mathcal{S}$ and actions $\mathcal{A}$. The Bellman operator ${\mathcal{T}}^{*}$ is defined as follows: ${\mathcal{T}}^{*}Q(s,a) ={\mathbb{E}}_{s^{\prime }}\left[r+ \gamma \underset{a^{\prime }}{\mathit{max}}Q\left(s^{\prime },a^{\prime }| s, a\right)\right]$, where $\gamma \in \left[0,\left.1\right)\right.$ is the discount factor.

Definition 2

(Banach Fixed-Point Theorem) [34]. If ${\mathcal{T}}^{*}$ is a contraction mapping on a complete metric space $X$, then ${\mathcal{T}}^{*}$ has a unique fixed point $x^{*} \in X$ such that ${\mathcal{T}}^{*}\left(x^{*}\right) = x^{*},$ and for any initial point $x_0 \in X$, the sequence $x_{k+1} ={\mathcal{T}}^{*}\left(x_k\right)$ converges to $x^{*}.$

Lemma 1

(Contraction Mapping). The Bellman optimality operator ${\mathcal{T}}^{*}$ is a $\gamma$-contraction in the sup-norm and for any two Q-functions $Q_1$ and $Q_2$, it satisfies the following:

$${‖{\mathcal{T}}^{*}{Q}_1-{\mathcal{T}}^{*}{Q}_2‖}_{\infty }\le \gamma {‖Q_1-Q_2‖}_{\infty },\forall Q_1, Q_2$$

(39)

Hence, ${\mathcal{T}}^{*}$ is a $\gamma$-contraction under the supremum norm${‖ .‖}_{\infty }$.

Assumption 1.

Let $Q_{\theta }$ be a neural network approximator. The projection operator  $\Psi$ maps any Q-function to the closest representable  $Q_{\theta }$:

$$\mathrm{\Psi }Q=\underset{\theta }{\mathrm{argmin}}‖Q-Q_{\theta } ‖$$

(40)

Assumption 2.

We have bounded function approximation error if there exists $ϵ\ge 0$ such that

$$\underset{Q}{\mathrm{s}\mathrm{u}\mathrm{p}}\parallel \mathrm{\Psi }Q-Q{\parallel }_{\infty }\le ϵ.$$

(41)

Theorem 1

($\Psi {\mathcal{T}}^{*}$ is a contractor). The DQN performs approximate dynamic programming by repeatedly applying  $Q_{k+1}=\Psi {\mathcal{T}}^{*}{Q}_k$. Under Assumptions 1 and 2, the composed operator  $\Psi {\mathcal{T}}^{*}$ satisfies

$$\parallel \mathrm{\Psi }{\mathcal{T}}^{*}{Q}_1-\mathrm{\Psi }{\mathcal{T}}^{*}{Q}_2{\parallel }_{\infty }\le \kappa {‖Q_1-Q_2‖}_{\infty }+ϵ,$$

(42)

where $\kappa \in \left[0,\left.1\right)\right.$ is a contraction-like constant and $ϵ$ is approximation error.

Proof of Theorem 1.

By Lemma 1:

$${‖{\mathcal{T}}^{*}{Q}_1-{\mathcal{T}}^{*}{Q}_2‖}_{\infty }\le \gamma {‖Q_1-Q_2‖}_{\infty }$$

(43)

Now projecting this onto $Q$ via $\mathrm{\Psi }$ and using triangular inequality,

$$\parallel \mathrm{\Psi }{\mathcal{T}}^{*}{Q}_1-\mathrm{\Psi }{\mathcal{T}}^{*}{Q}_2{\parallel }_{\infty }\le \parallel \mathrm{\Psi }{\mathcal{T}}^{*}{Q}_1-{\mathcal{T}}^{*}{Q}_1{\parallel }_{\infty }+{‖{\mathcal{T}}^{*}{Q}_1-{\mathcal{T}}^{*}{Q}_2‖}_{\infty }+\parallel {\mathcal{T}}^{*}{Q}_2-\mathrm{\Psi }{\mathcal{T}}^{*}{Q}_2{\parallel }_{\infty }$$

(44)

Let $ϵ=\underset{Q}{\mathrm{s}\mathrm{u}\mathrm{p}} \parallel \mathrm{\Psi }{\mathcal{T}}^{*}{Q}_1-{\mathcal{T}}^{*}{Q}_1{\parallel }_{\infty }$ be the worst-case approximation error, then

$${‖{\mathrm{\Psi }\mathcal{T}}^{*}{Q}_1-{\mathrm{\Psi }\mathcal{T}}^{*}{Q}_2‖}_{\infty }\le \gamma {‖Q_1-Q_2‖}_{\infty }+2ϵ$$

(45)

If $ϵ$ is small, this resembles a contractor:

$$\parallel \mathrm{\Psi }{\mathcal{T}}^{*}{Q}_1-\mathrm{\Psi }{\mathcal{T}}^{*}{Q}_2{\parallel }_{\infty }\le \kappa {‖Q_1-Q_2‖}_{\infty }+ϵ.$$

(46)

This completes the proof. □

Assumption 3

(Lipschitz Continuity) [35].  Let $Q_1, Q_2\in {\mathbb{R}}^{\left|\mathcal{S}\right|\times \left|\mathcal{A}\right|}$ be two action-value functions and let $\Psi : {\mathbb{R}}^{\left|\mathcal{S}\right|\times \left|\mathcal{A}\right|}\to {\mathbb{R}}^{\left|\mathcal{S}\right|\times \left|\mathcal{A}\right|}$ denote the projection operator onto the function class representable by a neural network, then $\Psi$ is assumed to be $L$-Lipschitz continuous:

$$\parallel \mathrm{\Psi }{Q}_1-\mathrm{\Psi }{Q}_2{\parallel }_{\infty }\le L{‖Q_1-Q_2‖}_{\infty }$$

(47)

where $L$ represents the Lipschitz constant of the projection operator $\mathrm{\Psi }$.

Corollary 1

(Stable DQN Iteration). Under Assumptions 1–3, if $L\gamma <1$, then $\Psi {\mathcal{T}}^{*}$ is a contractor with coefficient $\kappa =L\gamma$.

Proof. 

Combine Theorem 1 with Assumption 3. □

Corollary 2

(Approximate Fixed-Point Convergence). If approximation error is small $ϵ<{\displaystyle \frac{1-\gamma }{2}} \delta$, then DQN converges to neighborhood of $Q^{*}$ where $\delta$ is the desired precision constant which ensures the fixed point is within $\delta$ of $Q^{*}$.

$$\underset{k\to \infty }{\mathit{lim}\mathit{sup}}{‖Q_k-Q^{*}‖}_{\infty }\le {\displaystyle \frac{2ϵ}{1-\gamma }}$$

(48)

Proof. 

From Theorem 1, the iteration $Q_{k+1}=\mathrm{\Psi }{\mathcal{T}}^{*}{Q}_k$ is a contractor mapping with error $2ϵ$. The result follows from Banach Fixed-Point Theory for approximate contractions.

In deep Q-networks,$Q$ is approximated using a neural network $Q(s,a; \theta ).$ Although function approximation introduces non-linearities and potential divergence, using techniques such as target networks and experience replay helps stabilize training. Recent results show that under bounded approximation error and regular updates of target network parameters, DQN can approximate a contraction mapping which leads to empirical convergence. □

3.4. UAV Trajectory Generation

The trajectory generation problem needs to meet multiple task requirements such as flight efficiency, obstacle avoidance, and dynamical feasibility to enable the development of an extensible planner [36]. In our study, the resulting waypoint sequence from RL planner is refined into a dynamically feasible trajectory using the differential flatness property of quadrotor dynamics. This ensures that the final desired trajectory respects the UAV’s motion constraints and smoothness requirements that make it suitable for real-world execution. While our learning-based planner focuses on geometric efficiency during training, the differential flatness-based optimization acts as a post-processing layer to ensure that global feasibility and dynamic consistency are preserved. This two-stage approach allows us to prioritize simplicity and convergence speed during learning, while still achieving practical feasibility in execution.

Following the differential flatness theorem for quadrotor UAV from [37], the selected flat outputs from the quadrotor model (1–6) are $\left[x\left(t\right),y\left(t\right),z\left(t\right),\psi \left(t\right)\right]$. To ensure efficient motion and minimize control effort during UAV flight, it is essential to reduce the higher-order derivatives of position, particularly the snap, which refers to the fourth derivative of translational states $\left[x\left(t\right),y\left(t\right),z\left(t\right)\right]$ with respect to time [38]. Minimizing snap leads to smoother trajectories and reduces the burden on actuators, thereby enhancing energy efficiency and mechanical stability. To achieve this objective, the translational trajectory $\chi \left(t\right)$ is designed to minimize snap throughout the flight duration. Consequently, the optimal desired trajectory $\chi \left(t\right)$ must satisfy specific smoothness constraints and assume the following analytical form:

$$\chi \left(t\right)={argmin}_{\left(x\right(t\left)\right)}{\int }_0^T\mathcal{L}\left(x^{\left(iv\right)},x^{\left(iii\right)},\ddot{x}, \dot{x},x,t\right) .dt,$$

(49)

where $\mathcal{L}\left(x^{\left(iv\right)},x^{\left(iii\right)},\ddot{x},\dot{x},x,t\right).dt$ is equal to ${\left(x^{\left(iv\right)}\right)}^2.dt$ for minimum snap trajectory.

Now, the Euler–Lagrange equation can be solved as follows:

$${\displaystyle \frac{\partial \mathcal{L}}{\partial x}}-{\displaystyle \frac{d}{dt}}\left({\displaystyle \frac{\partial \mathcal{L}}{\partial \dot{x}}}\right)+{\displaystyle \frac{d^2}{{dt}^2}}\left({\displaystyle \frac{\partial \mathcal{L}}{\partial \ddot{x}}}\right)-{\displaystyle \frac{d^3}{{dt}^3}}\left({\displaystyle \frac{\partial \mathcal{L}}{\partial x^{\left(3\right)}}}\right)+{\displaystyle \frac{d^4}{{dt}^4}}\left({\displaystyle \frac{\partial \mathcal{L}}{\partial x^{\left(4\right)}}}\right)=0$$

(50)

This yields the following condition:

$$x^{\left(viii\right)}=0$$

(51)

The solution of (51) produces the optimal trajectory in the following form:

$$\chi \left(t\right)={\lambda }_0+{\lambda }_{1}t+{\lambda }_2{t}^2+\cdots +{\lambda }_m{t}^m, \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} m=7$$

(52)

For selected UAV flat outputs $\left[x\left(t\right),y\left(t\right),z\left(t\right),\psi \left(t\right)\right]$ from (1–6), using (52), a complete trajectory can be formulated over various time slots as follows:

$${P\left(t\right)}_{x\left(t\right),y\left(t\right),z\left(t\right),\psi \left(t\right)}=\left\{\begin{array}{c}\sum _{j=0}^m{\lambda }_{0j}{t}^j t_0\le t<t_1 \\ \sum _{j=0}^m{\lambda }_{1j}{t}^j t_1\le t<t_2 \\ \begin{array}{c}⋮\\ \sum _{j=0}^m{\lambda }_{nj}{t}^j \end{array}{t}_n\le t<t_{n+1} \end{array}\right.,$$

(53)

where $m$ defines the degree of polynomial.

Equation (53) computes the desired optimal UAV trajectory in terms of selected flat outputs of the UAV model. This trajectory inherently satisfies the UAV’s dynamic and motion constraints and serves as the basis for generating the corresponding control commands. The derivation of these commands is usually handled as a classical control problem. The controller that we have implemented in our simulation environment can be found in [39].

4. Experiments and Results

We have conducted various experiments in our simulator designed exclusively for RL-based UAV navigation. The simulator is custom-developed in MATLAB (v2022) and tailored for reinforcement learning-based UAV navigation tasks in 3D grid-based environments. It includes modules for environment generation, obstacle placement, action execution, reward assignment, and state transition management. The simulator allows flexibility in defining environment size, obstacle density, and UAV dynamics which makes it suitable for training and evaluating various RL algorithms. The simulation environment is explicitly designed for quadrotor UAVs where different RL algorithms for UAV path planning can be tested, and based on the differential flatness theorem, the trajectory of any order can be generated. The deep neural networks to accomplish the DQN are trained using the local system resources with NVIDIA RTX-4060 GPU (8 GB dedicated + 32 GB shared memory). We have conducted four experiments: (1) DQN-based path planning, (2) DQN with experience replay (DQN-ER), (3) DQN with reward shaping (DQN-RS), and (4) path planning using the proposed DQN approach (DQN Proposed) which is based on Algorithm 1. For all the simulation tests, we have considered the same environmental conditions, which include the placement of obstacles, rewards, and penalties (MDP setting), exploration decay rate, start and goal positions, DNN architecture and number of hidden layers, weights and biases initialization, and activation functions, etc. The parameters used in the simulation tests are listed in

Table 1. Simulation parameters.

No	Description	Symbol	Value
1	Learning rate	$\alpha$	0.001
2	Discount factor	$\gamma$	0.95
3	Small positive constant to control the experience sampling	$ϵ$	0.01
4	Tradeoff factor between greedy prioritization and uniform sampling	${\alpha }_{PER}$	0.6
5	Bias correction factor	$\beta$	0.4
6	PER batch size	$K$	32
7	Buffer size	$N$	10,000
8	Minimum exploration rate	${\epsilon }_{min}$	0.1
9	Initial exploration rate	${\epsilon }_o$	1
10	L2- regularization coefficient	$\lambda$	0.0001
11	Decay rate	${\lambda }_{decay}$	0.9
12	Threshold for equally valued actions	$\tau$	2

appended below. The hyperparameters listed in Table 1 were initially chosen based on common values reported in related literature for similar UAV navigation tasks. We then performed a limited manual tuning by adjusting key parameters such as learning rate, discount factor, and batch size to observe their impact on training stability and convergence speed. Due to computational constraints, a full grid search or automated hyperparameter optimization was not conducted. However, the selected values consistently produced stable training and good performance across multiple runs.

As previously described, the path-planning problem for an aerial agent operating in a cluttered 3D environment is modeled as a finite episodic Markov decision process. In this formulation, static obstacles are encoded as undesirable or terminal states associated with high negative rewards, effectively penalizing the agent for unsafe navigation. Since the focus of this study is on global path planning, only static obstacles are considered in the environment model. Obstacle information is directly embedded into the state space and incorporated into the environment’s transition dynamics. During training, any attempt by the agent to move into an obstacle cell results in a significant negative reward (e.g., −100), which discourages such actions and helps the agent to avoid the obstacles. Furthermore, the environment restricts transitions into obstacle-occupied states, thereby treating them as invalid actions and enforcing physical feasibility during the learning process.

4.1. Case 1 DQN

Simple Q-learning with deep neural networks is known as DQN and serves as the baseline. This provides fundamental insights into the UAV navigation problem in an MDP-based grid world. Due to its reliance on direct Q-learning without experience replay or reward shaping, this approach exhibits slow convergence and high variance in training rewards. The UAV struggles with sparse rewards which results in suboptimal policies due to the lack of exploration incentives and the inherent instability of neural network-based Q-value approximation. This method is prone to overfitting to recent transitions which leads to erratic policy updates and prolonged training times before achieving stable performance.

Figure 3 shows the MDP environment of the DQN-based UAV agent where the obstacles are bad states and yield penalties if collisions occur. Similarly, we also have start and goal states in this environment and reaching the goal state yields a higher positive reward. Part (a) of Figure 3 shows the optimal policy learned by the DQN agent after the Q-actions converge to their final values. Notably, there could be many other optimal policies in this 3D environment that can render a similar distance cost based on the number of steps. Due to the stochastic nature of MDP, the agent may pick any random but optimal policy from these possible optimal policies. However, all these optimal policies render different UAV trajectories even though some have a much reduced path length due to truncation of redundant waypoints. The linear waypoint trajectory for the quadrotor UAV is shown in part (b) of the same figure. As the quadrotor is a second-order system, therefore, it cannot traverse this trajectory having infinite curvature. The trajectory computed using the differential flatness theorem is followed smoothly by the quadrotor UAV and is shown in part (c). However, this is obvious from the figure that this trajectory is not the most optimal trajectory as there exists almost a direct straight-line trajectory between the start and the goal points (see Section 4.4).

The learning dynamics and performance of the DQN-based UAV navigation system can be comprehensively analyzed through several key training curves with each providing unique insights into different aspects of the algorithm’s behavior. Figure 4 shows curves of various metrics obtained after the implementation of the DQN algorithm for UAV path planning. Part (a) depicts the cumulative reward obtained by the agent and serves as the primary indicator of overall learning progress. The increasing trend demonstrates the agent’s improving ability to maximize cumulative rewards while plateaus suggest policy convergence. Initial high variance reveals instability in training due to higher initial exploration. Part (b) demonstrates the steps per episode curve which measures the path-finding efficiency. The decreasing number of steps indicates the DQN-based UAV agent is learning shorter and more optimal trajectories to the goal. The occasional spikes reflect temporary failures due to exploration or environmental complexity. Part (c) depicts the training loss, and it reveals how well the neural network approximates the true Q-values. An initial high loss is as per expectation due to random weight initialization. The training loss curve follows a declining trend as the network learns gradually. This leads to higher cumulative rewards and reduced variance in performance, as the UAV generalizes better across states. The spikes and oscillations indicate issues like overly aggressive learning rates and the absence of sufficient experience replay.

Meanwhile, the exploration decay rate demonstrated in plot (d) tracks the balance between exploration and exploitation, which shows how the ε-greedy strategy transitions from predominantly random actions to Q-value-driven decisions. The low epsilon value ensures continued exploration even during the late stage of the DQN training. The success rate curve shown in part (e) of Figure 4 quantifies the reliability of the DQN agent by measuring the percentage of episodes where the goal is reached. The increasing trend of the curves indicates that the agent gradually learns better policies and reaches the goal more frequently. However, the success rate of DQN is around 90 percent which can still be improved. Part (f) of Figure 4 depicts the average Q-values curve and provides a window into the agent’s confidence in its actions. The rising values reflect improving policy quality and convergence of the DQN agent to the final and optimal solution in terms of Q-values.

4.2. Case 2 DQN-ER

In contrast, DQN with experience replay utilizes a replay buffer to uniformly sample past transitions which effectively reduces the temporal correlation inherent in sequential decision-making. This enhancement leads to more stable training dynamics and improved convergence speed compared to the baseline DQN. This is because the agent learns from a more diverse and uncorrelated set of experiences. However, uniform sampling may still overlook rare but critical experiences, such as near-collision states and high-reward transition. This may limit the ability of the algorithm to refine its policy efficiently in complex scenarios. Figure 5 shows the MDP-based UAV environment with the DQN-ER algorithm for path planning. The learned random but optimal policy by the DQN-ER agent is shown in part (a). Again, there could be several other optimal policies each having the same distance cost function but may generate different suboptimal UAV trajectories. This scenario again demands the need to enable the UAV agent to learn or select such an optimal policy that could ultimately generate the best and optimal trajectory. Parts (b) and (c) show the linear waypoint trajectory, and the smooth and dynamically feasible trajectory based on differential flatness theory, respectively. The quadrotor UAV efficiently tracks the generated trajectory based on the waypoints received by the DQN-ER planner.

The DQN-ER demonstrates superior performance over the baseline DQN across all key training metrics which reflects its enhanced stability, efficiency, and learning robustness. Figure 6 shows the metric plots obtained using the DQN with the experience replay mechanism. From Figure 5, we have seen there is not any improvement in the UAV final trajectory due to the stochastic nature of the algorithm. However, the metric plots of DQN-ER shown in Figure 6 are better as compared to baseline DQN plots. This quantifies the efficient use of the experience replay mechanism in the traditional DQN approach.

The cumulative reward obtained by the UAV agent using DQN-ER as shown in part (a) has less initial variance and convergence to final rewards is fast, smoother, and has more consistent ascent. This is due to the replay buffer that mitigates the correlation between consecutive updates and allows the agent to learn from a diverse set of past experiences rather than just recent transitions. This leads to higher cumulative rewards and reduced variance in performance, as the UAV generalizes better across states. Similarly, the steps-per-episode curve as shown in plot (b) of the same figure depicts a steeper decline that indicates that the agent discovers more efficient navigation paths earlier in training. Contrarily, the baseline DQN often struggles with erratic step-counts due to overfitting to recent experiences. The randomized sampling smooths out learning and enables the UAV to converge to optimal trajectories more reliably. Similarly, the training is much better due to more consistent gradient updates as shown by the minimal training loss curve depicted in plot (c).

The higher spikes are due to the random sampling of diverse target values from past experiences. The exploration decay rate is kept the same in all experiments. Meanwhile, the success rate curve rises more rapidly and reaches a higher plateau in DQN-ER, as the agent leverages a broader range of experiences to overcome challenging states (e.g., obstacle-dense regions). Finally, the average Q-value curve in DQN-ER shows steadier growth with fewer signs of overestimation as the diversified training batches lead to more accurate value estimates. On the contrary, the baseline DQN is prone to overfitting to noisy Q-value targets due to lacking experience replay which ultimately results in erratic spikes.

4.3. Case 3 DQN-RS

Another variant of the baseline DQN incorporates reward shaping to mitigate the challenge of sparse rewards. By introducing auxiliary feedback signals, reward shaping guides the UAV toward more desirable behaviors and improves policy learning. This technique is expected to significantly accelerate early-stage learning by providing the agent with more frequent and informative reward signals. However, improperly designed reward function introduces bias which may lead the UAV agent to exploit shaped rewards at the expense of the true objective. For instance, excessive penalties for minor deviations may discourage exploration, while overly optimistic shaping could result in locally optimal but globally subpar trajectories. Careful tuning is essential to ensure that the shaped rewards align with the true goal of reaching the target efficiently. It is evident from Figure 7 that the policy learned by the UAV agent using the DQN-RS algorithm is again rendering a suboptimal trajectory for UAV tracking. Although reward shaping has improved the other training metrics as shown in Figure 8, the policy is still random, and the UAV agent needs to learn those policy actions that ultimately could generate the most optimal trajectory.

4.4. DQN-Proposed

The proposed hybrid approach, combining PER, L2 regularization, and incorporating modified tie-breaking mechanisms, outperforms the other variants in both learning efficiency and final policy rendering. PER ensures that high-TD-error transitions that have the most significant learning potential are sampled more frequently. This helps in accelerating convergence in critical states and overcomes the issues of simple ER. L2 regularization alleviates overfitting by penalizing excessive weights and promotes generalization across unseen grid configurations, while the modified tie-breaking strategy prevents the agent from oscillating between equally valued actions. The effective tie-breaking mechanism adopted in the proposed approach renders only that policy that subsequently generates the best and optimal UAV trajectory. Together, these components produce a policy that not only learns faster but also achieves higher cumulative rewards. Moreover, it also navigates the 3D obstacle-cluttered environment with greater reliability than the baseline and intermediate variants. The results demonstrate a smoother and more stable ascent in reward curves. Meanwhile, the UAV consistently avoids local optima and converges to near-optimal paths.

Figure 9 (part a) shows the improved policy learned by the UAV agent using the proposed DQN algorithm. We see the policy arrows are more directed towards the actual goal. From part (b), we can see the waypoint trajectory which has the same cost function as was in all previous cases. However, the trajectory generated from these waypoints is more optimal as compared to other trajectories that were computed previously. The proposed approach effectively reduces the trajectory length and ultimately enhances the endurance of the UAV.

From the training metrics demonstrated in Figure 10, we can see the improved performance of the proposed algorithm. The reward-per-episode curve in part (a) demonstrates a faster initial rise and higher asymptotic performance compared to other variants, as PER ensures critical transitions (e.g., obstacle avoidance or goal achievement) are replayed more frequently. In contrast, the baseline DQN suffers from slow convergence due to sparse rewards, DQN-ER lacks focus on high-impact experiences, and DQN-RS introduces bias which may lead the UAV agent to exploit shaped rewards at the expense of the true objective. The steps-per-episode curve shown in part (b) reveals that our proposed approach achieves optimal paths sooner and with fewer outliers, as PER efficiently propagates knowledge from pivotal moments and the proposed tie-breaking mechanism efficiently picks the best action from the equally valued actions. While DQN-ER and DQN-RS improve over the baseline, they still miss PER’s targeted learning and perform relatively worse. The training loss curve in part (c) shows smoother and more stable convergence in our proposed method. The PER focuses on high-TD-error transitions which reduce variance in updates. Meanwhile, the baseline and DQN-ER exhibit noisier loss curves, and DQN-reward shaping overfits to shaped rewards without PER’s balancing effect.

Critically, the success rate curve of our proposed approach (demonstrated in part (e)) climbs more rapidly and reaches a higher peak of 100 percent after 1500 episodes. All other DQN variants could not achieve 100 percent success rate even after 2000 episodes. This shows the superiority of our proposed approach. Finally, the average Q-value curve in part (f) reflects more accurate, faster, and stable value estimates. PER corrects overestimations through focused replay, and the tie-breaking mechanism accurately selects the policy actions for equally valued actions. In contrast, other variants either suffer from biased estimates (baseline), uniform sampling inefficiencies (DQN-ER), or shaping-induced local optima (DQN-RS).

4.5. Discussion and Comparative Analysis

Figure 11 shows the comparative results of our proposed DQN approach with baseline DQN and all other variants. We can see in part (a) the average reward curve of our proposed DQN approach is faster, smoother, and has less variance as compared to other variants which shows the effectiveness and superiority of our method. Similarly, in part (b) the convergence of steps count curve of our proposed approach outperforms all other methods which again shows the validation of the superiority of our method. In part (c), the training loss curve of our method shows less perturbations due to less variance and is more stable comparatively. Part (d) shows the exploration decay rate which is kept the same for all the experiments. The comparative success rate graphs of all the methods in part (e) again verify that the proposed DQN approach has significant advantages over other methods. Similarly, the average Q-values converge quickly and smoothly in our proposed approach and again verify that our proposed method outperforms all other methods.

Figure 12 demonstrates the top view of all the trajectories that were followed by the UAV, and it is obvious from this view that our proposed method generated the most optimal trajectory which is almost a straighter line from start to goal position. The same has been conferred by Figure 13 where we have calculated the trajectory lengths of all the trajectories followed by the UAV under various simulation experiments. The path obtained by our proposed method generates the shortest UAV trajectory which is about 10.72 grid units.

Collectively, the simulation results demonstrate that our proposed DQN approach consistently outperforms the baseline DQN, DQN-ER, and DQN-RS across all evaluated metrics. By integrating prioritized replay, the agent learns more efficiently from high-impact transitions. The incorporation of L2 regularization in the proposed DQN framework plays a pivotal role in enhancing the stability and generalizability of the learning process. By adding a penalty term proportional to the squared magnitude of the network weights to the loss function, L2 regularization effectively prevents the neural network from overfitting to noisy and sparse reward signals, which is particularly crucial in the UAV navigation task where environmental states and rewards can be highly variable. This regularization ensures that the Q-values remain conservative and robust and mitigates the risk of over-optimistic predictions that could destabilize training. The addition of tie-breaking further refines exploration and prevents convergence to those policies which render sub-optimal UAV trajectories. Together, these enhancements lead to faster convergence, higher success rates, and more stable training dynamics which are evidenced by the superior reward curves, reduced step counts, and smoother loss profiles. These findings validate the robustness of our framework for UAV navigation in complex 3D environments and suggest broader applicability to other reinforcement learning tasks which require precise, adaptive decision-making. A detailed video animation of our presented work can be found as Supplementary Material (see Video S1).

The proposed DQN-based framework, although validated in a simulated 3D cluttered environment, is developed with a strong emphasis on real-world applicability and deployment feasibility. In real-world scenarios, the trained policy can be integrated into physical UAV platforms equipped with onboard sensors such as GPS, IMUs, and depth or LiDAR sensors for environment perception and state estimation. The discrete high-level actions generated by the DQN agent can be translated into low-level control commands using a trajectory tracking or waypoint-following controller. Additionally, key real-world challenges including sensor noise, localization errors, actuator delays, and dynamic environmental changes must be addressed during deployment. To bridge the gap between simulation and physical testing, our future work will focus on hardware-in-the-loop simulations and field experiments to evaluate the robustness and adaptability of the proposed method in uncontrolled and partially observable environments.

5. Conclusions

In this work, an improved deep Q-learning framework has been proposed to address the limitations of traditional DQN-based reinforcement learning in UAV path planning within complex 3D environments. The proposed framework incorporates a modified tie-breaking mechanism to avoid suboptimal local decisions, prioritized experience replay to enhance sample efficiency, and L2 regularization to improve training stability. The improved learned policy promotes optimal flight paths and ultimately generates smoother and optimal UAV trajectory. The results demonstrate that the combined use of these techniques leads to significantly improved navigation performance compared to baseline methods. Specifically, as demonstrated in the results section, the proposed approach has significantly reduced the trajectory length due to its more efficient and informed action-selection mechanism. Moreover, across multiple performance metrics including average reward, step count, training loss, success rate, and average Q-values, the proposed method has consistently outperformed all other DQN variants. These improvements reflect faster convergence, more stable learning dynamics, and enhanced policy effectiveness in navigating complex 3D environments. The proposed framework provides a reliable and scalable solution for autonomous UAV navigation in cluttered spaces and lays the groundwork for further research in real-world deployment and multi-agent coordination scenarios. Future work could also explore dynamic reward shaping and hybrid exploration strategies to further optimize performance. Additionally, we plan to extend this research to real-world UAV experiments. In a physical setup, factors such as sensor noise, hardware limitations, real-time processing delays, and environmental disturbances (e.g., wind or GPS inaccuracies) are expected to influence navigation performance. Addressing these challenges will be essential to successfully transferring the proposed approach from simulation to practical deployment.