Hybrid Machine Learning and Reinforcement Learning Framework for Adaptive UAV Obstacle Avoidance

Abstract

This review explores the integration of machine learning (ML) and reinforcement learning (RL) techniques in enhancing the navigation and obstacle avoidance capabilities of Unmanned Aerial Vehicles (UAVs). Various RL algorithms are assessed for their effectiveness in teaching UAVs autonomous navigation, with a focus on state representation from UAV sensors and real-time environmental interaction. The review identifies the strengths and limitations of current methodologies and highlights gaps in the literature, proposing future research directions to advance UAV technology. Interdisciplinary approaches combining robotics, AI, and aeronautics are suggested to improve UAV performance in complex environments.

1. Introduction

Drones, sometimes referred to as unmanned aerial vehicles, or UAVs, have become more and more widespread in a variety of industries. These include the following: surveillance, inspection of structures, carrying of goods, as well as search and rescue in case of disasters. Commonly known as drones, UAVs are a type of aircraft that has received much attention and been adopted very widely in most of its applications. The demographics of this utilization stretch over solutions as different as aerial imaging, inspection of structures and ridges, deliveries, and search and rescue [1]. Over the past few years, interest has been rising in the literature related to unmanned aerial vehicles (UAVs) [2,3]. UAVs are often referred to as unmanned aircraft, these are aircraft that can fly on their own and do not require onboard pilots [4]. One of the main characteristics of UAVs is their high mobility, ease of deployment and, as a rule, low maintenance requirements [5,6]. Moreover, UAVs can mount different sensors, which make them versatile for a wide range of crucial tasks [7]. UAVs are applicable in a wide range of fields, such as wildfire surveillance [8,9], public surveillance systems [10], searching or tracking objects [11], goods delivery [12], medical aid, search and rescue [13], emergency message relaying [14], and intelligent transportation. However, they lack optimal behavior in complex dynamic environments due to human interventions and restriction in their transmission by radio frequencies [4]. Out of all the infrastructures based on roots, UAVs are regarded to be one of the most auspicious solutions for advancing smart environments [15].

The timeline depicted in Figure 1 illustrates the dramatic rise in scholarly publications concerning UAV obstacle avoidance from 2012 to 2024. Initially, the research was quite limited, with only two publications in 2012, slowly growing to a dozen by 2016. This slow start highlights the early development phases of UAV technologies, where the focus was predominantly on basic flight operations rather than the more intricate capability of autonomously navigating obstacles. As UAV applications began expanding into more complex environments, the need for sophisticated navigation systems became clear, sparking increased research in machine learning and reinforcement learning. These fields of artificial intelligence (AI) have been significant in training UAVs to learn how to navigate and address the presence of obstacles based on data collected by the UAV sensors. Reinforcement learning (RL) has been very useful when it comes to training UAVs to make necessary and appropriate decisions within a timeline in dynamic scenarios. Since 2017, more researchers have addressed the problem of UAVs and, by 2023, 306 publications were published. This growth suggests the improvement in AI and computational resources, as well as the maneuver complexity in unmanned operation, which consists of both stationary and moving obstacles. Thus, the application of modern machine learning and reinforcement learning technologies has made it possible to create intelligent, learning UAV systems, enhancing a company’s capabilities in terms of the level of automation and safety in operations. However, one more fact should be mentioned: the conflict in Ukraine has ramped up the augmentation of many aspects of UAV technology. The need to cater for surveillance and identification, as well as concrete evaluation of impact in conflict regions has continued to revolutionize UAV development for both military and civilian importance. For example, demands like using UAVs in enemy territory, which are counter responsive to threats, such as electromagnetic attack and counter UAV, have led to progress in stealth technology, secure communication, and self-decision-making technologies. The incorporation of machine learning (ML) with RL has not only improved these systems but made them admissible to adversarial or dynamic environments. Current research is concerned with developing UAVs that would have swarm behavior, threat identification, and self-navigation features required in military operations. Such scrutiny and accelerated innovations are precipitated by the functional usage demands precipitated by conflicts, as demonstrated in Ukraine. This spike in research and development is a clear indication of the scientific community’s broader initiative to enhance AI technology in UAVs, ensuring they can withstand the demands of contemporary warfare and take a lead in civilian uses such as in disaster aid and environmental surveillance.

Objective of the Review

Various RL algorithms are employed to teach UAVs how to navigate their surroundings independently. RL is effective in addressing diverse challenges where the UAV essentially learns from experience, mimicking the decision-making processes of a human expert in the field. Through interaction with the environment, the UAV processes information about its current state, selects actions accordingly, and is rewarded for providing comments. State representation for decision-making is facilitated by data gathered from UAV cameras and sensors scanning the environment [16]. The main goals of this review article are varied and encompass giving readers a thorough grasp of the state of the art in UAV navigation and obstacle avoidance research utilizing ML and RL methods. The objectives include exploring and evaluating various RL algorithms used for teaching UAVs to navigate autonomously and comparing the effectiveness of different RL approaches in improving their navigation capabilities. Another key objective is to analyze how ML techniques can enhance UAV performance in complex and dynamic environments and to examine the integration of ML algorithms with UAV systems for real-time obstacle detection and avoidance.

In this review, we intend to summarize the published work in both UAV navigation and obstacle avoidance with an emphasis on identifying current approaches, including specific strengths and weaknesses in the existing literature. Moreover, this review aims to identify gaps in the current literature and areas where future research may be needed. We present the issues, then highlight some potential areas where these complex ML-based techniques can be applied for UAV navigation and suggest future research to further improve the existing architecture of UAV navigation systems. This review intends to give an overview of the current computer vision progress in UAV exploration and suggest new methodologies dependent on a blend of approaches with automatons, AI, and flight related research. The article offers a glimpse into the latest developments and upcoming possibilities in UAV navigation and obstacle avoidance through the application of ML and RL methods.

2. Search Methodology

Literature Search Strategy

We meticulously developed our literature search strategy to cover the broad scope of UAV navigation and obstacle avoidance, illustrated in Figure 2. We began by exploring a broad range of databases, including IEEE Xplore, Google Scholar, PubMed, Science Direct, and the ACM Digital Library, which allowed us to gather a wide variety of interdisciplinary research sources. We selected key search terms, such as “UAV”, “obstacle avoidance”, “reinforcement learning”, “deep reinforcement learning”, “DL”, “ML-Techniques”, “UAV path planning”, “UAV obstacle avoidance”, “AI drone navigation”, “drone collision”, “UAV navigation”, “collision avoidance UAV control”, “drones”, “autonomous flight”, “autonomous drones”, “aerial robots”, “Unmanned Aerial Vehicle”, and “machine learning”, to cover various aspects of UAV technology comprehensively. Initially, we screened each article based on its title and abstract to ensure relevance to our research goals.

In the second phase, illustrated in Figure 3, we honed our strategy by concentrating on the most frequently occurring keywords identified in our initial search. This detailed stage involved examining the frequency and context of these keywords within the articles. This analysis helped us pinpoint major research trends and key methodological approaches in UAV obstacle avoidance. Our two-phase approach not only made the vast amount of literature more manageable but revealed significant research gaps and new advancements in UAV technology. This method provided a strong foundation for our review, ensuring that readers gain a thorough understanding of both the breadth and depth of current research in this field.

3. Machine Learning for Enhanced UAV Obstacle Avoidance

3.1. Overview of Machine Learning

Through the optimization of obstacle avoidance tactics, machine learning is essential in allowing autonomous navigation in the context of UAVs. Alan Turing first proposed the idea of AI in 1950, which aligned with the initial rise of computers [17]. He developed the Turing test to differentiate between machines and humans using a set of questions and answers. If the computer could pass the test, it would be considered to have intelligence comparable to that of humans. After the Turing test, scientists began to explore AI, ultimately resulting in its formal recognition as an academic field in 1956 [18]. Systems can now learn from data and provide data-centric solutions thanks to machine learning. ML is extensively utilized on the internet and in distant settings [19]. The integration of UAVs and machine learning enables functionalities like image categorization and segmentation, drone path mapping, caching, scheduling, and surveillance [20]. ML technology allows UAVs to navigate independently by recognizing objects, modifying flight paths, and managing resources instantly. ML algorithms are also capable of spotting irregularities in data collected by UAV sensors, leading to prompt reactions to security issues and precise trend analysis [21]. The use of ML algorithms for real-time processing of UAV data allows for flexible decision-making and swift responses to dynamic conditions [22].

ML plays a critical role in this field by enabling the analysis and comprehension of a vast amount of data collected by UAV sensors in real time. Traditional control methods often face challenges in dynamic environments where UAVs encounter complex and unpredictable obstacles. Utilizing ML, UAVs can use previous experiences which resulted from different scenarios, improving navigation strategies for the next time, while in most cases the process will be done autonomously. In other words, it can be summarized that an improvement in the obstacle avoidance of a UAV requires ML. As depicted in Figure 4, the concurrent use of supervised, unsupervised, and reinforcement models show that such a framework offers a strong foundation for the practical implementation of intelligent UAV systems, which can learn to take flight without external guidance and adapt their operating parameters to existing environmental conditions. Integration of ML in the obstacle avoidance of the UAV has greatly enhanced the arena by enabling decision-making.

In Figure 4, to improve UAV obstacle avoidance, a variety of ML techniques have been applied, including RL, unsupervised learning (UL), and supervised learning (SL). To delve deeper into these methods, we will start with supervised learning. This method is widely employed in ML and entails training algorithms with labelled datasets, where each data point is associated with a particular outcome. This training aids UAVs in forecasting results or making decisions using new data. The subsequent section will elaborate on the distinct methods and benefits of employing SL in UAV obstacle avoidance.

3.2. Supervised Learning (SL)

SL is the dominant method in ML, where the agent learns from input–output pairs to create a function that maps inputs to outputs. The goal is to label the output based on the input, especially in classification scenarios where the output falls into different categories [23]. As one kind of learning paradigm, supervised learning (SL) has been the basis for enabling unmanned aerial vehicles to learn from labelled datasets, such as sensor readings and flight data, that can enhance its performance in optimal navigation and obstacle avoidance. ANNs can be trained for UAV sensor data to classify the shape, nature of obstacles, or whether it is a static dynamic that could help to reroute the flight path in real time. Deep Learning (DL), a subset of SL, enables UAVs to process massive datasets from realistic environments, making UAVs more efficient in terms of predicting and avoiding obstacles. For UAV applications which rely on fast decision-making, and where autonomous capabilities are required in dynamic environments, the design of such SL techniques is key. In order to provide comprehensive insight into these strategies,

Table 1. Key Supervised Learning Techniques and their Applications in UAV Obstacle Avoidance.

Algorithms	Type	Description	References
LSTM RNN	SL	Improved prediction accuracy based on recent observations	[24]
Neural Network Pipeline (NNP)	SL	High accuracy (2%) and fast processing speed.	[25]
Neural Network	SL	Good real-time performance	[26]
Neural Network	SL	Created a depth map using the CGAN network to prevent more collisions.	[27]
Neural Network	SL	CNN network was used to forecast the depth map.	[28]
Neural Network	SL	Suggested a method that combines the epipolar geometry with the depth neural network.	[29]
Neural Network	SL	Utilized CNN to calculate the image’s optical flow.	[30]
Neural Network	SL	Suggested an ANN-based route planning method.	[31]
Neural Network	SL	Used the G-FCNN algorithm to assess the route.	[32]
Neural Network	SL	Utilizing the Q-learning method and neural network, the 3D collision avoidance problem was resolved.	[33]
Parameter FNN	SL	Attitude control	[34]
RBF NN	SL	Attitude control	[35]
Adaptive FNN	SL	Control of aerobatic maneuvers	[36]
CL	SL	Tracking trajectory	[37]
CL, BKR-CL	SL	Change of controller	[38]
RWNN	SL	Trajectory tracking	[39,40]
ReLU Network	SL	Aerobatic maneuver	[41]
NN	SL	Generalization of dynamics	[42]
SVR	SL	Attitude control	[43]
ELM	SL	Single UAV navigation	[44]
CNN	SL	Getting around a city	[45]
Imitation learning	SL	MAV flying over a woodland	[46]
CNN	SL	Trajectory prediction	[47]
NN, RL	SL	Estimating collisions	[48]
DNN, RL	SL	State estimation	[49]
Regression	SL	Estimation of altitude	[50]
SVM	SL	Mapping a river	[51]
GMM, SVM	SL	Detecting landing sites	[52]
OBIA	SL	Precision agriculture	[53]
Hybrid deep NN	SL	Classification of crops	[54]
DNN	SL	Perception of mountain trails	[55,56]

summarizes the various SL approaches used for UAV obstacle avoidance and describes their fundamental purposes.

In the next section, we will examine the UL techniques crucial for identifying patterns and structures within data without predetermined labels.

3.3. Unsupervised Learning (UL)

UL is the branch of ML that handles inputs without labelled output or mapping function. In turn, the algorithm does not receive the correct answers to the problems with parameters but must independently search for the patterns and dependencies within the data. One area of ML which includes UL is that the system finds its own way of sorting data through pattern recognition; the most well-known of this is clustering, in which the data points are grouped in categories that share a set of features [57]. ML approaches and methods, specifically unsupervised ones, are demanded in improving the navigation for a UAV and its avoidance of obstacles. These methods are helpful because they enable UAVs to learn and act based on complex environments while the labeled dataset is very costly and time-consuming to collect.

Popular methods for working with obstacle data are K-means for data clustering, grouping data points that are similar, and DBSCAN, for identifying the data obstacles by point density. Multilevel obstacle cluster analysis is conducted with the help of hierarchical clustering. Moreover, preprocessing data is performed by PCA and t-SNE to make an effective visual representation of the data. Approaches such as Isolation Forest or Auto encoders for obstacle identification are utilized to find unexpected obstacles due to the changes in the obstacle pattern. These methods are vital so that UAVs do not run into new obstacles that cannot be anticipated from the training data set. Additionally, SOMs reduce the obstacle data into a format that makes spatial distribution analysis more manageable. Other notable algorithms include the GMM, Affinity Propagation, Mean Shift Clustering, Spectral Clustering, and ICA, all contributing to more efficient and effective UAV obstacle avoidance systems [58].

UL does not involve labelled data or specific target values for direct comparison. Instead, algorithms perform tasks such as clustering or pattern recognition on the dataset, extracting significant features and representations. The trained model then makes predictions or classifications based on these learned patterns.

Figure 5 depicts a visual overview of the discussed ML techniques, encompassing various SL, UL, and RL strategies utilized in UAV obstacle avoidance.

3.4. Challenges in Machine Learning Techniques and the Need for Reinforcement Learning

ML is an ever-evolving field that improves machine intelligence, allowing machines to outperform humans in various tasks. However, ML approaches often face challenges, such as overfitting and underfitting, especially in dynamic situations due to their static nature. In robotics, the main challenges revolve around autonomous learning and adaptation, leading to significant research and development efforts. Specifically regarding UAVs, the primary challenge is to achieve autonomy and efficient failure response. Furthermore, ML systems demand extensive data for learning complex tasks, making data collection a laborious and sometimes impractical process.

RL is adept at adapting and continuous learning, making it well-suited for tasks that involve intricate decision-making, such as operating autonomous vehicles. Because of individual feedback, RL can learn from minimal data with interactive experience. In this way, by properly regulating the levels of exploration and exploitation, RL can reveal high-value strategies that ML may not notice, thus providing flexible solutions essential in changeable environments.

4. Integrating Machine Learning and Reinforcement Learning for UAV Obstacle Avoidance

The development of proactive methods that may control multifaceted and ever-changing contexts is vital for the development of UAVs. RL has come out as a technique capable of arriving at decisions in an uncertain environment; it is an powerful constituent in the development of UAVs equipped with an understanding of the environment and the knowledge of how to move around and avoid obstacles. Figure 6 categorizes the use of deep reinforcement learning (DRL) in UAVs and outlines the three major operational categories: path planning, navigation, and control. The areas highlighted above need to be developed to advance UAV obstacle avoidance performance. In DRL, the path planning of going from one point to another is done dynamically by analyzing the changes in the environment and the presence of obstacles. In the navigation domain, DRL compiles sensory inputs to help UAVs steer through their areas of operation. The control is based on the usage of DRL to adjust and maintain the stability of drone flight and to provide proper reactions to interactions with the environment and changes in flight conditions. Incorporation of these DRL applications also increases the level of autonomy and accuracy of UAVs in elemental navigation and their ability to effectively avoid obstacles.

Figure 7 presents RL and ML being used to allow UAVs to naturally and seamlessly fly between urban constructions. Path planning, navigation and control systems are illustrated in the diagram to achieve mobility in complex areas filled with obstacles. Path planning acts as the fundamental component through which the UAV can constantly update its path to bypass hurdles like trees and buildings. Control of the flight of the UAV is necessary, which is possible with the help of sensors located in its body and GPS. The control part of the system allows the UAV to implement navigational choices while keeping the UAV stable and varying in response to the external environment.

This illustration also shows the need for well-developed control systems and precise planning of the paths that UAVs will follow in increasing the level of autonomy of these vehicles. This picture also emphasizes the importance of modern systems of observing the environment in efficient obstacle avoidance. These technologies combine to afford significant autonomy to UAVs and ensure they can easily switch between real time modification of environment and the charting of routes. Such a configuration creates the basis for further studies; thus, in the subsequent sections of this document, especially in Section 4.2, reserved for UAV Navigation, and Section 4.4, dedicated to UAV Control we go into detail about the mechanics of UAV navigation, strategies and methods of UAV control, and ways to improve the accuracy and adaptability of the UAV control in challenging conditions.

For the purposes of path planning, navigation and flight control, various approaches are intensively used. In addition to the increasingly popular AI methods, model-based design is classically used. The built numerical models describe the basic issues of aircraft flight, such as flight mechanics, and aerodynamics [65]; also important in the context of electric drives are power supply and electric drive models, commonly used energy sources, and energy storage [65], as well as, less frequent and more difficult to interpret photovoltaic [66,67] or fuel cells [68]. These models are used directly in navigation, path planning or control, or indirectly together with artificial intelligence techniques.

4.1. Path Planning with Reinforcement Learning

Path planning is one of the most relevant topics as it influences the extent of a vehicle’s autonomy most of the time. It not only pertains to the degree of speculation/autonomy, but has implications for built-in subroutines, directions, capabilities, and resilience. A crucial concept in vehicle navigation, path planning covers trip duration management, obstacle avoidance, and various other challenges. Research often utilizes reinforcement learning to address these tasks, particularly in UAVs operating in uncertain environments where precise mathematical models are unavailable. Hence, the integration of RL and DL enables vehicles to learn their routes autonomously. By employing sophisticated learning algorithms, UAVs can navigate through dynamic environments safely, mitigating the risk of collisions [69]. Indeed, RL algorithms have found extensive applications across various research domains associated with UAV technology. One study suggested integrating DRL with an LSTM network to streamline the learning algorithm’s convergence speed. Another approach, as described in [70], focused on object avoidance and target tracking by incorporating a reward function and penalty actions to attain a smoother trajectory. Further studies employed RL algorithms for precise attitude control [71]. RL has been applied in conjunction with various optimizers, including the grey wolf optimizer (GWO). The GWO is a metaheuristic optimization algorithm with a strong background derived from the leading structure and hunting behavior of grey wolves. Nevertheless, it is defined by an adaptable bionic framework that allows for its context-based usage [72]. There is an algorithm for UAV path planning which possesses the ability to create a proper and efficient path for the UAV. This approach integrates the capabilities of RL and the GWO to improve four essential processes: exploration, geometric convergence, exploitation, and optimal course alteration [73]. The typical strategies of mitigating obstacles while working with large sets of data typically rely on computer science methods like the Simultaneous two methods, which are Structure from Motion (SFM) and Localization and Mapping (SLAM) [74]. An important aspect of the SLAM algorithm is the building of the map of the surroundings of the area in the near vicinity of the UAV and, at the same time, updating of the UAV’s position and route planning, based on information from various sensors. Another method of obstacle detection is to detect an object’s characteristics with the help of a depth camera. This work employs a novel DRL algorithm that combines imitation learning and DRL to train the data [75]. Moreover, in reference [76], a new learning algorithm was proposed to build a network structure with special attention to the IoT devices. This method entailed training a DDQN network alongside other aspects such as the available flight time of the autonomous drone and the positioning and number of sensors to be deployed. Thus, utilizing this network, the UAV can modify its actions as well as make navigation decisions when necessary. SFM then utilizes optical flow sensors to create a depth map as well as structure of the surroundings. Both SFM and SLAM require a planned path for the UAV to navigate. Therefore, the UAV must first stop to survey the area and plan the path it will follow [77]. These techniques are not good for real-time obstacle avoidance applications. Despite attempts to improve SLAM for immediate use, they do not address non-stationary obstacles with unpredictable motion nor identify unfeatured walls commonly present in indoor settings [27]. In the field of obstacle avoidance, the authors of [78] introduced a self-trained UAV capable of navigating around both stationary and moving obstacles in a 3D urban setting. A deep deterministic policy gradient (DDPG)-based learning method was used in the study to provide guidance to the vehicle towards its destination while minimizing the risk of collisions. This algorithm incorporates a reward mechanism aimed at reducing the distance between the UAV and its target, along with penalties for colliding with obstacles. Another approach treats obstacle avoidance as a model-free RL problem, suitable for environments with incomplete prior knowledge. This method employs DRL to train UAVs for indoor navigation in 3D hallway environments, using a diverse dataset of images capturing various environmental conditions. However, despite extensive training, the accuracy of the resulting deep Q-network may be insufficient. Furthermore, this method fails to mimic human learning behavior effectively.

Consequently, researchers have shifted focus towards studying human behavior in obstacle avoidance. Humans rely on memory to store and recall relevant information, guiding decision-making in diverse scenarios. Similarly, UAVs facing partial observability issues require memory-like mechanisms for real-time navigation planning [27]. In another investigation, the fusion of object avoidance to reach the end target via the shortest path and minimum time interval was explored. Flight path learning employed Q-learning algorithms, similar to prior studies, necessitating computational capacity for path planning while evading random obstacles. To ensure calculation speed, a dedicated unit handled computations, with the acquired results then implemented to the UAV. The simulation involved a basic indoor environment with walls, obstacles, start and target positions, each endowed with distinct reward values. Multiple maps of varying sizes and obstacle densities were generated accordingly [35]. The methodology employs a temporal attention deep recurrent Q-network to optimize the expected total of discounted rewards and enable effective object avoidance in novel environments through RL. Q-learning guides action selection by maximizing reward values and ensuring optimal performance based on predefined policies [79]. A study similar to [69] employed an RL model to enhance UAV performance by optimizing a position controller. To determine the next course of action for the future state, this model used the vehicle’s present position and its learning framework. The resulting position was then fed into the position controller, which instructed the propellers controller to generate enough thrust force to move the vehicle to its intended new position. A new methodology has been suggested, exploring different obstacles faced by UAVs. When sharing mission-related data online, it suggests using an interference-aware path planning technique to minimize interference with ground maps and lower wireless transmission delay [80]. The challenge is framed as a dynamic non-cooperative game, where the UAV takes on the role of the user. By employing DRL and utilizing echo state network (ESN) cells, trajectories for UAV navigation are optimized in relation to the ground network and other UAVs. This approach allows users to make decisions based on previous rewards, such as location and transmission power level. DRL is increasingly used in UAV path planning, enabling real-time adjustments to changing environments. Chao et al. (2019) introduced a real-time path planning strategy that considered potential threats from enemies, particularly crucial for anti-terrorism operations where self-preservation is vital in the face of possible radar detection and missile threats [81]. A simulated environment is created, incorporating adversary elements using the STAGE Scenario software tool, which features a comprehensive database including battlefield entities like radar and missiles [82]. A model is developed to assess the situation by utilizing information on the positions of vehicles and enemy units, which consist of radar and surface-to-air missiles. When the UAV is detected within radar range, calculations are performed to determine the distance between the enemy and the vehicle, as well as the maximum killing zone radius of the missile. The vehicle is trained with a dueling double deep Q-network (D3QN) algorithm for course planning. At the start of each training episode, the initial parameters of the network are established, and the UAV is placed in the STAGE Scenario. The situation map is then created, based on the assessment model, indicating the current state. An action is chosen and executed accordingly. Following the action’s completion, fresh data is added to the scenario map and an instant reward value is noted. The most recent version of the data is kept in memory, replacing the previous one [83].

4.2. Reinforcement Learning for UAV Navigation

In recent efforts to achieve optimal control for UAVs, DL algorithms have gained significant attention. An experimental study was conducted using various reinforcement learning techniques, categorizing them into two primary groups: discrete action space and continuous action space. The study begins by comparing different methods for drone navigation, including SL, UL, and RL, while highlighting the pros and cons of each approach. RL involves training the UAV to navigate and avoid obstacles through trial and error, allowing it to learn autonomously once the training environment is set up [84]. Initially, research introduced an approach for model-free DRL using convolutional neural networks [85]. While most UAV research focuses on simulation platforms for training and optimizing algorithms, the study aims to apply RL algorithms on actual hardware to achieve optimal path tracking in unknown environments. It addresses the key challenge of guiding the UAV along an optimal path by determining actions based on its current state (SK) and thrust force (τ). A conventional PID controller is proposed for stable navigation, using learning parameters and gain values as inputs. The controller adjusts the UAV’s next state based on accumulated rewards [86]. The paper progresses by applying RL, deriving algorithms for sequential decision-making in which the agent moves through a visual world in discrete time intervals. These equations explain how actions are chosen to maximize a reward signal in both discrete and continuous action spaces. For discrete action space, the agent uses a policy-based approach, employing a deep Q-network (DQN) to compute state values from high-dimensional data like images. To overcome DQN challenges, a new algorithm called double dueling DQN (D3QN) is introduced, combining a double DQN and a dueling DQN, which shows strong performance in experiments by reducing correlation and improving state quality. The study uses Airtime, a simulation platform to produce a realistic scene, using Unreal Engine visuals. However, the simulation has simple paths for UAVs, with all trees on flat ground [59]. To address this, a new environment with various obstacles was created, using sensors and CNN inputs to find the best path for the drone. To ascertain behavior, several DQN algorithms were used for continuous action space learning: actor-critic with Kronecker-factored trust region (ACKTR), proximal policy optimization (PPO), deep deterministic policy gradients (DDPG), and trust region policy optimization (TRPO). Manual data labelling was replaced with a policy gradient technique combined with U-Net-based segmentation. The results showed D3QN’s superiority in discrete action space and ACKTR’s smooth trajectories in continuous action space. Expert pilots performed better in wooded areas, while ACKTR excelled in an arena setting. The study also explored optimizing drone delivery routes using RL algorithms, defining drone delivery as the path a drone takes to reach its destination while navigating various obstacles [87]. Since drones are used for outdoor applications such as surveillance and delivery, precise navigation cannot rely solely on an international satellite navigation system (GNSS) [60]. Outdoor drone tasks require obstacle avoidance, which can be achieved through artificial intelligence methods [69]. The study examines various DRL algorithms that enable drones to reach targets through optimal decision-making driven by reward values. It introduces conceptual DRL algorithms to address current delivery challenges and suggests innovative architectures to improve existing solutions. In this context, RL represents the decision-maker as the agent and its environment as the surroundings. The agent interacts with the environment, with each action corresponding to a state within the state-space model [48]. The first suggested algorithm is the DQN, which combines the Q-network with neural network techniques [88]. DQN relies heavily on CNN for action-value optimization, including periodic updates to align action values with target values, reducing correlation. Experience replay, inspired by biology, randomizes data to eliminate correlation among states. Double DQN addresses positive bias by separating actions. Experiments were conducted in a virtual environment mimicking reality for safety. The state-space consists of three states, each paired with a neural network: original CNN, 2D JNN, and modified JNN-3D for vertical movement. Training each model took about 40 h, aiming to improve efficiency through checkpoints. Based on the findings, employing checkpoints led to a rise in successful action episodes and a decrease in failed ones. JNN achieved success in fewer episodes than CNN, with higher reward values and faster stabilization [89]. Another evolving use for UAVs is in search and rescue (SAR) operations, requiring UAV navigation through unknown terrains filled with dangers and obstacles [90]. Using UAVs in this field has spared human rescuers from entering hazardous environments and reduced the risks associated with sending humans on such missions [91].

Starting in 2020, studies began addressing the challenge of dynamic obstacles encountered by UAVs during flight, especially in outdoor environments. A study investigated how ML and RL approaches might be used to improve UAV navigation and obstacle avoidance. With an emphasis on state representation from UAV sensors and real-time environmental interaction, the efficacy of various RL algorithms in teaching UAVs autonomous navigation is evaluated. In addition to pointing out gaps in the literature and highlighting the advantages and disadvantages of present approaches, the study suggests future research avenues to develop UAV technology. A taxonomy of these artificial intelligence systems for UAV navigation is provided in Figure 8, which illustrates the various methodologies and their applications. These include learning-based approaches like DRL, asynchronous advantage actor-critic (A3C), partially observable Markov decision process (POMDP), and GWO, as well as optimization-based approaches like differential evolution (DE), genetic algorithm (GA), and particle swarm optimization (PSO).

Table 2. Optimization-based AI approaches.

Algorithms, References	Type	Characteristics	Goals
PSO [92]	MPSO	PSO-generated infeasible paths are transformed into feasible ones through the integration of an error factor.	For the best collision-free 3D UAV route planning, use MPSO.
PSO [93]	DPSO	Discrete steps facilitate propagation, while multiple augmentations are utilized to improve convergence.	IoT and image processing in UAV control
PSO [94]	GBPSO	It assesses the current global path against alternative candidates to identify the most optimal choice.	GBPSO for enhanced 3D UAV routing
ACO [95]	Multi-ACO	It addresses the TSP problem, taking into account both intra-colony and inter-colony pheromone values.	ACO with many colonies for effective UAV path planning
ACO [96]	Double-ACO	It employs GA for generating the initial population.	Double-ant colony speeds UAV planning.
ACO [97]	PFACO	Achieves rapid convergence by utilizing MMAS and APF for enhanced global search capabilities.	PFACO for improved UAV path planning.
GA [98]	GA	The UAVs’ heading angle rate, acceleration, and rising angle rate make up their chromosomes.	Best possible 3D UAV route planning with GIS information.
GA [99]	Improved-GA	It comprises an encoding vector derived from the sequence of UAV yaw angles.	Using evolutionary algorithms to enhance UAV route planning.
GA [100]	HR-MAGA	It employs a hierarchical recursive procedure to establish a more detailed path.	Recursive hierarchical evolutionary algorithms for UAV route planning optimization.
GA [101]	OCGA	Uses OC and TLBO searches to achieve quick convergence.	Optimizing the OCGA algorithm for UAV route planning.
SA [102]	Modified-SA	Selection of the POIs is stochastic	Boost the effectiveness of UAV route planning.
SA [103]	SA-ACO	Hybrid algorithm	Enhance efficiency of underground path planning.
SA [104]	Grid-based SA	SA based on grid maps for 2D UAV navigation	Improve the coverage route planning of UAVs.
DE [105]	ADE	Selective mutation in hybrid algorithm.	In urban settings, maximize the effectiveness of UAV path planning.
DE [106]	CDE	Adaptive mutation limited differential evolution algorithm for catastrophe situational awareness in UAV path planning optimization.	Enhancing UAV route planning for disaster awareness using limited differential evolution.
DE [107]	DEAKP	Prioritize knee solutions over the Pareto front within a constrained multi-objective optimization context.	By optimizing distance, risk metrics, and operational limitations, a knee point-guided differential evolution method may be used to increase the efficiency of UAV path planning in catastrophe scenarios.
PIO [108]	SCPIO	The use of randomization, ergodicity of motion trajectories, strong sensitivity to beginning values, and hierarchical social class.	Creating a multi-UAV route planning model using SCPIO and TSS to improve efficiency and cooperation in challenging conditions.
PIO [109]	AOQPIO	The behavior of the system is highly sensitive to initial conditions, exhibits ergodic motion trajectories, and incorporates elements of randomization.	Adaptive QPIO improves UAV route planning in hazardous situations, outperforming PSO and its variations.
CS [110,111]	Improved-CS	Implement crossover and mutation operators within genetic algorithms and employ Chebyshev collocation points for representing coordinates.	Adaptive QPIO improves UAV route planning for better performance in difficult settings.
A* [112]	Dijkstra	Uses Hamiltonian and Eulerian path model and Dijkstra	Putting forward a unique GPS-free route planning and unmanned vehicle localization technique that makes use of cooperative algorithms.
A* [113]	DEA*	Hybrid DE and A* for urban UAV path planning.	Hybrid DE and A* for UAV collision-free path planning.
A* [114]	A*	Multi-step search with real-time A*	Stealth UAV real-time path planning.
GWO [115]	HSGWO-MSOS	Hybrid algorithm	Creating a hybrid HSGWO-MSOS algorithm to plan the best UAV route.
GWO [116]	2D-GWO	Ensure minimum fuel usage and zero threat	Creating GWO to maximize the effectiveness of UCAV route planning.
GWO [117]	3D-GWO	Localization and obstacle avoidance	Creating GWO to maximize the effectiveness of UCAV route planning.

illustrates optimization-based AI approaches.

4.3. Learning-Based AI Approaches

4.3.1. Reinforcement Learning

RL stands as a potent and extensively applied AI methodology, acquiring knowledge about its surroundings through action and strategy evaluation. At its core, RL comprises an agent and an environment, wherein the agent, employing the Markov decision process (MDP), engages with the environment, discerning optimal actions to pursue [118]. This technique involves an agent learning through interaction with a real or simulated environment. Periodically, the agent makes decisions, observes outcomes, and receives feedback, thereby improving its task performance over time. RL finds applications in autonomous navigation, where robots adjust strategies autonomously to achieve optimal policies. In essence, RL algorithms learn from past actions to solve specific problems.

Table 3. RL Techniques and their Applications in UAV Navigation and Obstacle Avoidance.

Algorithms, References	Type	Characteristics	Goals
RL [60]	Q-Learning	A calibrated reward system, employing e-greedy policy, operates in a coordinate-based state-action space. Q-learning adjusts PID controller parameters for navigation in a 2D environment, with function approximation based on FSR for efficiency.	Indoor navigation
RL [119]	Q-Learning	An adaptive reward system, utilizing e-greedy policy, functions within a goal-oriented state-action space. Navigating a 2D environment, the system autonomously seeks out charging points.	Improving QoE
RL [120]	Q-Learning	Utilizing an RSS-driven state-action space, the system employs a dynamic reward system and e-greedy policy, while also integrating obstacle avoidance.	Indoor SaR
RL [121]	TD-Learning	Employing a discrete state-action space, the system utilizes linear function approximation using tile coding to handle large state-spaces, alongside an e-greedy policy.	UAV-BS navigation
RL [122]	Multi-agent Q-Learning	The system anticipates user movement trajectories within a coordinate-based state-action space, factoring in energy usage with gradual convergence and integrating a dynamic rewarding system.	Data transmission with minimum power
RL [123]	Multi-agent Q-Learning	Real-time decentralized path planning with a model-based reward system, e-greedy policy, and reduced state-action space, all within a 3D environment.	Perform sense and send tasks
RL [124]	Double Q-Learning	An evolving reward system, paired with an e-greedy policy, functions within a goal-centric state-action space in a 2D environment.	Temporary BS support
RL [125]	Q-learning and SARSA	Utilizing reinforcement learning, optimizing UAV energy consumption, enhancing path planning efficiency.	UAV Efficiency Enhancement
RL [126]	Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Deep Q-Networks (DQN)	Obstacle detection and avoidance via diverse reinforcement learning approaches, contrasts DQN, PPO, and SAC methodologies for addressing static and dynamic obstacles, conducts evaluations in a virtual environment using AirSim and Unreal Engine 4, demonstrates SAC’s superior performance, and highlights the effectiveness of off-policy algorithms (DQN and SAC) compared to on-policy counterparts (PPO) for autonomous drone navigation.	Develop a drone system for autonomous obstacle avoidance.
RL [127]	(DNN-MRFT)	RL agent integration, direct policy transfer, DNN-MRFT parameter identification, reward function design, and fusion of high-level velocity actions and low-level control validated through simulations and real-world experiments with a 90% success rate, ensuring originality.	Closing the sim2real gap in UAVs with RL transfer and integrated control.

, “RL Techniques and Their Applications in UAV Navigation and Obstacle Avoidance”, provides an overview of the different RL approaches and how they are used in UAV navigation and obstacle avoidance. In the context of UAV operations, this table offers a thorough summary of several RL algorithms, their traits, and their particular objectives.

4.3.2. Deep Reinforcement Learning (DRL)

To create solutions that become better with practice, DRL integrates ideas from RL with DL. This approach relies on repeated iterations and the assessment of a reward function to determine the most effective behavior for an agent. DRL approaches may be broadly classified into three categories: model-based, policy-based, and value-based. The goal of value-based RL is to maximize a value function for long-term rewards over a sequence of activities. Policy-based RL focuses on finding the policy that optimizes the objective function, with deterministic and stochastic approaches. Model-based RL either provides the agent with a predefined environmental model or requires the agent to learn the model to complete tasks within that environment.

An overview of the several DRL methods for obstacle avoidance and UAV navigation is given in

Table 4. Overview of Deep Reinforcement Learning Techniques for UAV Navigation and Obstacle Avoidance.

Algorithms, References	Type	Characteristics	Goals
DRL [128]	MDP-based Dueling DDQN	Utilizing a discrete action space, action approximation is performed through Artificial Neural Networks (ANNs), employing dual neural network architecture.	Simultaneous UAV navigation and radio mapping
DRL [129]	MDP-based DQN	Employing an RSS-driven state space, alongside an E-greedy policy, the system integrates a dynamic rewarding mechanism.	MIMO-based UAV navigation
DRL [130]	MDP-based DQN	Operating within a discrete action space contingent upon dispatched UAVs, the system incorporates an e-greedy policy and prioritized replay memory.	Urban vehicular connectivity
DRL [131]	MDP-based MADQN	Training centrally with an e-greedy policy, multiple agents share their individual locations, operating within a discrete action space.	AoL-aware WPCN
DRL [132]	MDP-based DQN	Incorporating an exceedingly sparse rewarding system and obstacle avoidance, the system employs LwH to enhance convergence within intricate environments.	UAV navigation in complex environment
DRL [133]	MDP-based DDQN	The system leverages CNN for map-based navigation while taking into account energy consumption.	UAV Coverage Path Planning
DRL [134]	MDP-based DQN	Employing object detection assistance, the system navigates through an object-oriented state space, integrating a binary rewarding system alongside an e-greedy policy.	Collision free UAV navigation
DRL [135]	MDP-based DQN	The system integrates user data freshness, UAV power consumption, and coordinates into the state space for DRL training with stochastic gradient descent.	UAV-BS Navigation
DRL [75]	MDP-based DQN	Implementing TD3fD for policy enhancement, employing a CNN-based DQN, the system includes obstacle avoidance capabilities.	Obstacle free UAV navigation
DRL [136]	POMDP-based DRL	MDP and POMDP are both employed for global and local planning, with policy improvements managed by TRPO.	Indoor navigation
DRL [137]	EDDQN	Employing a CNN-based DQN for map-guided navigation, the system adapts a modified Q-function to ensure effective obstacle avoidance.	Autonomous UAV exploration
DRL [138]	POMDP-based DDQN	CNN-based DDQN, grid-based navigation and searching	UAV-assisted data harvesting
Advantage Actor-Critic (A3C) [139,140]	A3C-based DRL	Prioritized experience replay is applied based on A3C, while multi-agent DDPG handles policy enhancement, alongside a pre-determined UAV navigation pattern.	UAV-assisted MEC
Advantage Actor-Critic (A3C) [141]	Fast-RDPG	Utilizing Fast-RDPG with LSTM-based DRL, the system achieves rapid convergence through an online algorithm.	UAV navigation in complex environment
Advantage Actor-Critic (A3C) [142]	A3C-based DRL	Utilizing A3C with a tailored policy gradient, the system accounts for energy consumption via an e-greedy policy, enabling decentralized multi-UAV navigation.	Distributed multi-UAV navigation
DRL [143]	DAC-SAC	The DAC-SAC algorithm integrates dual experience buffers, a Convolutional Neural Network (CNN), and a self-attention mechanism to improve UAV obstacle avoidance. Simulated trials affirm their efficacy in diverse environments, including scenarios with depth image inputs.	UAV autonomous obstacle avoidance
DRL [61]	D3QN	The trajectory planning approach for UAVs with cellular connectivity employs deep reinforcement learning and integrates connectivity and obstacle data to ensure safe flight, effectively navigating obstacles without AI plagiarism concerns.	Unique trajectory planning approach for cellular-connected UAVs
DRL [144]	FRDDM-DQN	The study presents FRDDM-DQN, an advanced deep-reinforcement-learning algorithm that combines Faster R-CNN with a Data Deposit Mechanism. It improves obstacle extraction, agent training, and adjusts to dynamic scenarios. In experiments, FRDDM-DQN demonstrates autonomous navigation, collision avoidance, and superior performance compared to alternative algorithms.	UAV autonomous navigation and collision avoidance

below, along with a description of each method’s unique objectives.

4.3.3. Deep Learning (DL)

Deep learning, especially through deep neural networks (DNN) in deep reinforcement learning (DRL), is extensively used for UAV navigation based on vision. Recent advancements in tasks such as object identification, localization, image segmentation, and depth perception enable the effective application of DNN methods. Researchers have successfully utilized these techniques to identify roads and streets in important routes and urban areas, aiming to enhance autonomy in self-driving vehicles [139]. DL allows UAVs to interpret large volumes of sensory input in real time, enabling rapid decision-making and adaptive learning. This is achieved by using sophisticated neural network topologies.

Table 5. Overview of DL Techniques for UAV Navigation and Obstacle Avoidance.

Algorithms, References	Type	Characteristics	Goals
DL [145]	CNN	CNN using the IDSIA dataset and ReLU activation	UAV navigation within unstructured trail
DL [146]	CNN	Following trails, recovering from disturbances, and avoiding obstacles.	UAV trail navigation
DL [147]	DNN	IDSIA dataset, trail navigation, and guidance	UAV navigation within forest
DL [148]	DCNN-GA	Model based on Exception and hyper-parameter tweaking based on GA	Indoor UAV navigation
DL [149]	Dynamic path planning IFDS with ANN	Optimization using ESCSO, Enhanced path navigation in intricate settings, real-time processing, reliable obstacle avoidance, improved computational efficiency.	Real-time UAV path planning enhancement

illustrates the several DL approaches used for obstacle avoidance and UAV navigation, each with unique features and objectives.

4.4. Reinforcement Learning for UAV Control

UAV control for specific tasks is complex due to dynamic conditions, energy efficiency needs, and vulnerability to disturbances. Classical control, based on linear dynamics, was insufficient, leading to the adoption of ML, particularly RL. DRL emerged to address the need for higher accuracy and robustness, enabling new control tasks in real time without predefined models. It supports various approaches, from attitude control to managing communication channels among multiple UAVs. DRL integrates well-known model-free methods like DDPG, TRPO, and PPO, each suited to specific applications based on their strengths and weaknesses. Utilizing the PPO algorithm, DRL was implemented for fixed-wing UAV control, addressing nonlinear dynamics and lateral-longitudinal coupling. Despite disturbances and varied conditions, PPO ensured stable flight and achieved the desired speed and angles. With low computation time, it’s suitable for UAV control. The critic network assesses behavior rewards for training, whereas the actor network calculates ideal conduct. The reward function is based on state distance from the desired one. Normalized data inputs were used to reduce computation time. However, successful outcomes were limited to simulations, casting doubt on real-world feasibility [150]. In tackling an issue with attitude control, different algorithms, such as DDGP, TRPO, PPO, and PID, were assessed. The findings leaned towards favoring the PPO algorithm, even though it shares similarities with TRPO. PPO’s benefits entail simplified implementation and adjustment. Simulation outcomes, evaluated using metrics like rising time and stability, indicate suitability for continuous tasks [58]. The hybrid flying system utilizes deep reinforcement learning (DRL), particularly the proximal policy optimization (PPO) algorithm, for immediate learning and avoiding collisions. It presents a versatile control approach suitable for multiple UAVs across diverse dynamics and scenarios. Integral error terms guarantee precise control without extra adjustments. The reward function in DRL evaluates energy efficiency, flight stability, and tracking accuracy. While hybrid UAVs present difficulties, flapping wing UAVs demand precise calculations and ongoing adaptation for efficient flight in varying conditions [151]. The goal of the project is to develop a model-free UAV control system by the use of deep reinforcement learning, especially the deep deterministic policy gradient (DDPG) method. Deformation control is managed through airfoil-mounted steering gears. Actions are chosen based on the current state, with random and deterministic policies guiding exploration and action evaluation. The reward function factors in angular orientation and linear position tracking [152].

The field of reinforcement learning is advancing quickly, with Robust-DDPG demonstrating better performance compared to DDPG in terms of convergence and stability. This improvement incorporates delayed learning, adversarial attack filtering, and mixed exploration to guarantee steady learning, resilient control, and swift convergence. Control parameters involve roll angle and linear speed, with the reward mechanism targeting effective goal attainment and collision avoidance [153]. The system ensures the accurate autonomous landing of a UAV on a designated platform using visual markers and camera input. Deep Q-learning networks (DQNs) are utilized, segmented into marker detection and vertical landing tasks. The adoption of double DQN architecture improves system alignment and complexity handling. Sparse rewards stemming from reliance on Markov decision processes are mitigated through “partitioned buffer replay”, which categorizes experience data to enhance decision-making [62]. Despite the potential shown by deep Q-networks in UAV control, they encounter limitations when tasked with image processing. Employing deep reinforcement learning, the UAV autonomously captures high-quality images of a person’s face while adjusting its height and position. The deep deterministic policy gradient (DDPG) outperforms conventional methods like PID control and 2D face recognition algorithms. The selection of rewards, based on control error influences both learning speed and tracking accuracy. The method demonstrates effectiveness even under discrete control commands, such as those encountered in simulation environments [154]. The DCNNP controls a swarm of five UAVs tasked with formation and defense using MARL. Centralized learning allows a single agent to coordinate behaviors. DCNNP surpasses random policy but exhibits slightly lower performance compared to perfect policy, particularly with fewer UAVs in surveillance tasks [63].

A new algorithm, DRL-EC3, built on an adjusted DDPG and actor-critic policy, seeks to lower energy usage in swarm UAVs while upholding task precision. The findings indicate the superiority of DRL-EC3 over random and greedy policies, particularly as UAV numbers rise [64]. The research emphasizes the significance of UAV channel allocation for effective task communication within swarms. DRL is preferred over Q-learning for its ability to manage large datasets efficiently. This approach significantly improves communication accuracy and speed among UAVs, marking a notable advancement in UAV control research [155].

A policy is an agent’s decision-making function (control strategy), which is a mapping from conditions to actions. There are two sorts of policies: on-policy and off-policy. Proximal policy optimization (PPO), trust region policy optimization (TRPO), and SARSA are examples of on-policy techniques that aim to assess or enhance the policy which is applied while making judgments. By contrast, policies other than the one used to create the data are evaluated or improved using off-policy approaches such as Q-learning, deep Q-network (DQN), and deep deterministic policy gradient (DDPG): see Figure 9. Without the need for a model, DRL provides a flexible framework for resolving these issues. It can also learn from mistakes and forecast an accurate answer based on future observations. RL algorithms are divided into two categories: model-based and model-free. Model-free RL algorithms select the best course of action based on the agent’s trial and error, whereas model-based RL algorithms attempt to select the best course of action based on the learned model of the environment. There are benefits and drawbacks to both model-free and model-based algorithms.

Table 6. Comprehensive Overview of Model-Free RL Algorithms for Enhanced UAV Control.

Algorithm	Description
SARSA	SARSA, which stands for State–Action–Reward–State–Action, is an algorithm used in artificial intelligence (AI) that belongs to the class of model-free and on-policy temporal difference (TD) learning methods. It functions by learning an action-value function.
SARSA Lambda	SARSA Lambda extends SARSA by including eligibility traces, enabling it to learn from more extended rewards. Moreover, SARSA Lambda is an on-policy TD learning algorithm.
Deep Q Network (DQN)	DQN, called deep Q-network, merges Q-learning with deep learning methods and operates on-policy.
Double DQN	Double DQN, a variant of DQN, employs two separate neural networks to estimate the action-value function, mitigating overestimation bias. It follows an off-policy approach.
Noisy DQN	Noisy DQN, a variant of DQN, injects noise into the action-value function during training to foster exploration and prevent the agent from being trapped in local optima. It adheres to an off-policy strategy.
Prioritized Replay DQN	Prioritized Replay DQN, a modification of DQN, prioritizes experience replay based on significance, enhancing the agent’s learning by focusing on crucial experiences. This algorithm operates on an off-policy basis.
Categorical DQN	Categorical DQN, an adaptation of DQN, employs a categorical distribution to model the action-value function, particularly beneficial for environments featuring a multitude of actions. This algorithm operates on an off-policy basis.
Distributed DQN (C51)	The Distributed DQN (C51) modifies DQN by using a quantized distribution for the action-value function, which improves its performance in continuous environments. This algorithm works on an off-policy basis.
Normalized Advantage Functions (NAF)	Normalized Advantage Functions (NAF) is a policy gradient method that employs advantage functions to adjust the policy. This algorithm operates on an off-policy basis.
Continuous DQN	Continuous DQN is an adaptation of the DQN algorithm tailored for environments with continuous action spaces. This algorithm operates on an off-policy basis.
REINFORCE (Vanilla policy gradient)	REINFORCE is a policy gradient method that employs the REINFORCE estimator for policy updates. This algorithm operates on a policy-based approach.
Policy Gradient	A class of algorithms known as Policy Gradient is designed to learn policies by directly optimizing predicted rewards. This strategy uses a policy-based methodology.
TRPO	A trust region is used by the policy gradient algorithm Trust Region Policy Optimization (TRPO) to ensure the security of policy changes. The on-policy methodology is used by this algorithm.
PPO	A policy gradient approach called Proximal Policy Optimization (PPO) uses a clipping goal to limit the size of policy updates. The on-policy methodology is used by this algorithm.
A2C/A3C	Policy gradient methods A2C (Advantage Actor-Critic) and A3C (Asynchronous Advantage Actor-Critic) are made to train several agents at once. These two algorithms follow an on-policy approach.
DDPG	Deep Deterministic Policy Gradient (DDPG) is a policy gradient algorithm tailored for continuous action spaces. It follows an off-policy approach.
TD3	In order to increase stability, TD3, a modified version of DDPG, uses two Q-networks and two target Q-networks in addition to an off-policy methodology.
SAC	A policy gradient algorithm that combines Q-learning and policy gradient approaches is called Soft Actor-Critic (SAC). It employs an off-policy strategy.
ACER	Actor-Critic with Experience Replay (ACER) is a policy gradient algorithm that incorporates experience replay to enhance efficiency. It follows an off-policy approach.
ACKTR	Actor-Critic with Kronecker Product Representation (ACKTR) is a policy gradient algorithm that utilizes a Kronecker product representation to train a factored policy. It follows an off-policy approach.

summarizes model-free RL techniques.

5. Future Directions and Challenges

Significant progress has been made in UAV technology, especially in the areas of autonomous navigation and obstacle avoidance. Still, there are several issues that require more study and improvement. One of the key challenges still facing us is autonomous and dependable navigation in complicated situations. Although UAV obstacle avoidance has greatly improved thanks to current ML and RL techniques, these algorithms perform worse in more complex and unstructured situations. For example, Figure 10 illustrates that RL algorithms perform better in urban settings than in forested and mountainous areas. Aggregated results based on a few cutting-edge studies are presented in Figure 10, Figure 11 and Figure 12 for obviation hindrances and navigating safely by UAV. The data was sourced from a review of papers published in the years from 2012 to 2024. The literature reviewed different methodologies including simulation and real-world experiments, which we summarized to showcase the trends and progression of reinforcement learning applications on UAVs.

This suggests that stronger algorithms that can manage a variety of unforeseen situations are required. Moreover, the success rate of RL algorithms declines with environmental complexity, underscoring the necessity of creating more sophisticated algorithms capable of adjusting to complicated settings, as seen in Figure 11.

UAV operations need real-time, high-accuracy, low-latency decision-making, particularly in areas where GPS is unavailable. To successfully satisfy these requirements, current methods must be improved. The trade-off between accuracy and latency for various RL algorithms shows that although some algorithms manage to strike a decent balance, others still have difficulty optimizing both parameters simultaneously. The latency and accuracy of RL algorithms illustrate Algorithm A’s superior balance, as shown in Figure 12.

In addition, UAVs have the potential to revolutionize precision agriculture and disaster relief, significantly enhancing operational efficiency in both sectors. AI-equipped UAVs can optimize crop monitoring and yield projections in precision agriculture, while also improving disaster management by providing real-time data and support during emergency response operations. The complexity of the environment rises, and this triggers the need for more sophisticated algorithms in learning across different environments, especially for exams in the RL. Another aspect that can increase decision-making speed in dynamic environments is the optimization of the UAV computational structures to accommodate real-time data-intensive computations. That is why we believe that in the context of the increasing popularity of UAVs, it is necessary to consider regulating issues and questions related to safety. The adoption of standard safety measures will lead to a decrease in accident occurrence frequencies and thus effective implementation of UAVs into the national airspaces. There is a need for cooperation between representatives of the industry, governmental bodies, and technological creators to design the required legislation to provide protection for citizens and their data.

An extensive review of the literature must be carried out to determine or assess the effectiveness of UAVs depending on the context of their use. Settling norms regarding response times, failure rates, and accuracy also set standards for the future evolution of technology. These situations mean that quantitative improvements and outcomes analysis are critical to addressing these challenges and progressing UAVs to fulfil the requirements of complex environments.

5.1. Comparative Analysis of Existing Studies

Finally, in other sections of this study, such as Section 4.3.3, we discussed in detail the various types of deep learning algorithms used in UAV obstacle avoidance; while in Section 5.1 we provide an overview of existing studies by comparing different methods to one another like dynamic obstacles and static obstacles avoidance techniques. This section addresses more the surveying and reviewing of previous works rather than the technical depth found in other sections like Section 4.3.3. As shown in

Table 7. Existing surveys related to UAVs obstacle detection and avoidance.

References	Year	Approach	Observations
[156]	2019	Obstacle detection in wooded areas and takeoff dynamics	considerable computational overhead
[165]	2020	using radio frequency signals to detect drones	comparatively high computational cost
[166]	2022	Avoiding collisions with stationary and moving objects within a structure	reduced UAV speed and obstacle detection using LiDAR sensors
[157]	2020	Estimate the movement of pertinent pixels in the robot’s intended path.	When it’s raining or gloomy outside, visual performance will be reduced.
[167]	2021	Obstacle identification and avoidance using vision	seeing just one side at a time and requiring a lot of processing power to see other orientations
[168]	2018	From trajectory tracking to model predictive control in 3D among several moving objects	NMPC is known to be computationally expensive.
[169]	2018	System for avoiding obstacles	High cost of calculation
[25]	2021	From trajectory prediction to vision-based collision avoidance with dynamic objects	High cost of processing hardware and computing
[170]	2021	Vision-based collision avoidance in maneuverable aircraft	Expensive processing costs and inability to identify tiny things
[159]	2023	Avoiding collisions with both moving and stationary objects using vision	High failure rate in complicated environments
[160]	2023	Taking into account the obstacle’s location and velocity, dynamic obstacle avoidance	Restricted to a 2D setting. Assumes UAV can determine position and velocity of every barrier and UAV.
[161]	2023	Avoiding obstacles with a Nano drone	A typical, untested indoor environment with 100% dependability at 0.5 m/s
[158]	2019	Monocular depth prediction in real time and obstacle detection and avoidance using a lightweight probabilistic CNN (pCNN)	The method is shown to be faster and more accurate than the state-of-the-art approaches on the devices such as TX2 and 1050Ti with increased accuracy from sparse depth from visual odometry.
[164]	2017	Trajectory tracking with collision avoidance using a potential function a null-space controller	Confirmed on a Parrot AR. Drone, works well in dynamic scenarios with a low-cost RGB-D sensor platform
[162]	2020	The following paper aims to present a thorough analysis of the current approaches to collision avoidance systems as applied to unmanned vehicles with particular regard to UAVs.	Compared several approaches and types of sensors in order to discuss the problem of avoiding collisions with UAVs.
[163]	2017	Gathered a massive repository of UAV crash incidents for developing a self-supervising navigation strategy	Uses crash data to provide evidence about organization of vehicle motion in crowded spaces

, this portion of the research describes the various methods and findings from previous surveys connected to UAV obstacle identification and avoidance.

6. Conclusions

Integrating ML approaches as well as RL, the autonomous navigation and collision avoidance of UAVs has been a great enhancement. This article highlights a wide range of RL algorithms and applied ML solutions for autonomous UAV navigation. This paper investigates how these algorithms operate in coordination with UAV sensors for environ-mental assessment and real-time decision-making. This has been experimentally successful for allowing UAVs to make informed decisions quickly, through the development of RL algorithms. The usage of SL, UL, and RL types form a strong framework for intelligent UAV system design that can learn in different environmental conditions. This review focuses on how the combination of these approaches has been effective for improving UAV performance, especially in complex and dynamic environments. The results show these RL algorithms have the potential to solve a variety of autonomous navigation problems for UAVs. The fact is that these algorithms have been able to process information from the cameras and sensors of drones and choose from their data an optimal action by which they will be rewarded. If UAV systems are combined with ML methods, the efficiency and endurance of airborne navigation based on object presence can be substantially increased, e.g., gap closing in real-time visual detection to land promptly, so this will definitely improve the problem response time for obstacle avoidance.

The development of UAV depends on the enhancement of various factors to incorporate the multidisciplinary fields of robotics, AI, and Aeronautics. Thus, this assessment has identified shortcomings in the current body of literature and directions for further study to improve the reliability and effectiveness of UAV navigation systems. These two categories of learning will require constant improvement in the future in an effort to produce better and more efficient UAV operations. Therefore, combining ML with RL exhibits its impact in enhancing the UAV technology in the way of autonomous flying and avoiding the movements of objects. The effectiveness of various RL algorithms has been meticulously assessed, and the pros and cons of existing approaches have been acknowledged. For scholars and professionals in the field, this study provides a comprehensive overview of the current state of research in UAV navigation and obstacle avoidance, rendering it an invaluable resource.