HGR-QL: Optimized Q-Learning for Multi-UAV Path Planning in Mountain Search and Rescue

Highlights

What are the main findings?

• An optimized Q-Learning method (HGR-QL) is proposed for large-scale multi-UAV mountain search and rescue, which integrates a hierarchical independent Q-table and a multi-level gradient collision avoidance reward function. In complex mountain scenarios with dynamic interference, it achieves a high task completion rate of 74.47%, a low collision count of 25.44, and a stable communication delay of 100.00 ms.
• A 50 × 50 dynamic grid environment is constructed to simulate the distinctive attributes of real mountainous landscapes. Multi-scenario comparative experiments and ablation experiments validate the critical role of core modules in enhancing UAV swarm collaboration and environmental adaptability.

What are the implications of the main findings?

• The hierarchical Q-table and gradient reward mechanism overcome the inherent limitations of traditional methods in large-scale multi-UAV collaboration, providing a lightweight and scalable path planning solution for mountainous environments featuring weak signals and high obstacle densities.
• The method is well-suited for the large-scale deployment of small- and medium-sized UAV swarms, enabling mountain search and rescue operations to seize the “golden 72 h” rescue window. It offers valuable technical insights for the integrated application of remote sensing technology and UAV swarms in geological disaster emergency response.

Abstract

Existing Q-Learning-based path planning methods face significant bottlenecks in large-scale collaboration, dynamic interference adaptation, and regional value differentiation, failing to meet the practical needs of mountain search and rescue. This study proposes HGR-QL, an optimized Q-Learning method for large-scale multi-UAV operations. Referencing remote sensing datasets, a 50 × 50 dynamic grid environment is constructed by integrating 20% fixed obstacles and 10 moving interference sources, highly simulating real mountain features. Integrating the individual Q-tables and the regional shared Q-tables, the hierarchical independent Q-table architecture is designed, balancing local autonomy and global collaboration. To guide UAVs focusing on remote sensing-identified high-value areas, an innovative multi-level gradient collision avoidance reward function is constructed, avoiding task deviation. Comparative experiments across three scenarios with four baselines and ablation tests validate the core modules. Results show HGR-QL outperforms peers in key metrics: in the dynamic interference scenario, it achieves a 74.47% task completion rate, 25.44 collisions, and a stable 100.00 ms communication delay. HGR-QL provides a lightweight, scalable solution, effectively enhancing the efficiency, safety, and stability of mountain search and rescue and supporting the “golden 72 h” rescue window.

1. Introduction

Mountainous regions account for 72.4% of China’s total land area, characterized by complex environments, dense vegetation, and fragmented terrain. These geographical features pose enormous challenges to mountain search and rescue operations, often resulting in high operational complexity, low rescue efficiency, and significant safety risks for rescue personnel [1]. In recent years, mountain disasters such as flash floods and landslides have occurred frequently. Ground-based rescue efforts are often restricted by terrain and suffer from delayed responses, leading to the loss of the “golden 72 h” rescue window [2]. Remote sensing (RS) technology, with its advantages of wide coverage and real-time data acquisition, has become an essential support for mountain rescue [3]. Unmanned Aerial Vehicles (UAVs), leveraging their high maneuverability, strong adaptability to complex terrain, and cost-effectiveness, have emerged as operational platforms integrating RS data for rescue missions [4].

The integration of UAVs with RS sensors has been widely applied; however, small-scale UAV formations currently deployed by rescue authorities primarily adopt a two-person collaborative model involving pilots and observers [5]. This paradigm is limited by the complex mountainous environment, fleet scale, and collision risks during flight, resulting in extremely low overall search and rescue efficiency [6]. Therefore, practical mountain search and rescue operations urgently require an intelligent path planning algorithm capable of adapting to complex mountain environments, optimizing multi-UAV flight trajectories, and achieving efficient collaborative operations [7,8].

The mainstream multi-UAV path planning methods are mainly classified into two categories: traditional path planning algorithms and reinforcement learning algorithms [9,10,11,12]. Traditional path planning algorithms solve the optimal path through principles such as heuristic search, graph traversal, or sampling optimization. Their core relies on pre-defined environmental models and mathematical programming logic to address basic obstacle avoidance for single UAVs, featuring clear principles and excellent local path optimization performance. However, they exhibit poor adaptability in large-scale multi-UAV collaborative scenarios, making it difficult to balance global collaboration and dynamic environmental responsiveness [13]. Reinforcement learning (RL) is an interactive machine learning paradigm where agents generate feedback signals through continuous interaction with the environment, iteratively optimizing path decision strategies [14,15,16]. It possesses inherent advantages in dynamic environment adaptability and autonomous decision-making capabilities. Nevertheless, traditional RL methods have shortcomings in complex mountain scenario adaptability, multi-agent collaboration mechanism design, and task-oriented reward function construction, which limit their application in large-scale mountain search and rescue [17].

Compared with traditional path planning algorithms that struggle with flexible adaptation and insufficient collaboration in dynamic complex environments, RL algorithms do not rely on precise pre-defined environmental models. Instead, they continuously optimize decision strategies through real-time interaction with the mountain search and rescue environment, making them more capable of meeting the core requirements of mountain scenarios with fragmented terrain and random interferences. Thus, RL has become the preferred technical route for large-scale multi-UAV path planning. Among various RL algorithms, the Q-Learning algorithm is widely used in single-UAV path planning tasks due to its simple principles, model-free nature, and ease of deployment [15]. However, when directly extended to large-scale mountain search and rescue scenarios integrating RS data, it faces three critical bottlenecks: first, there is inadequate adaptation to complex dynamic mountain environments fused with RS data [18]. Most existing models are designed for static scenarios and fail to fully utilize real-time RS terrain information, resulting in poor dynamic robustness against the combined effects of fixed obstacles (e.g., cliffs, dense forests) and dynamic disturbances (e.g., sudden air currents) [8]. Second, there is the lack of a mechanism to perceive the dynamic states of other UAVs and RS data transmission status. When multiple UAVs converge on high-probability areas for missing persons identified by RS, they are prone to falling into a “multi-agent collision trap,” a problem that deteriorates sharply when the number of UAVs exceeds 100 [16]. Third, traditional reward functions do not consider the collaborative requirements of mountain search and rescue or the utility of RS data. Some UAVs may deviate from high-value search areas identified by RS to avoid local collisions, leading to a significant decline in overall efficiency.

To address those challenges, this study optimizes the Q-Learning algorithm to develop a “highly adaptive, low-overhead, and strongly collaborative” collision avoidance path planning method for large-scale UAV systems. The core contributions of this study are summarized as follows:

Simulation of a complex dynamic mountain grid environment: To highly simulate the complex topographic features of actual mountainous areas, building on the traditional single static environment modeling method, this study references the characteristics of remote sensing terrain data and integrates 20% randomly distributed fixed obstacles and dynamic interference sources that move randomly every 10 decision steps into the traditional single static environment. Constructing a 50 × 50 two-dimensional complex mountain grid environment newly makes the environmental model more consistent with the actual mountain search and rescue scenarios in China [3].

Construction of a hierarchical independent Q-table architecture: This study integrates dedicated individual Q-tables and regional shared Q-tables to enhance the discrete roles of individual Q-tables and regional Q-tables in existing methods. The architecture proposed in each UAV is equipped with a dedicated individual Q-table to realize autonomous decision-making based on real-time remote sensing data, with initial weights optimized according to the priority of mountain search and rescue tasks. Regional shared Q-tables are deployed to integrate collective search and rescue experience and remote sensing data interpretation results through weighted updates of individual and regional data, thereby enhancing the overall collaborative performance of the UAV fleet.

Design of a multi-level gradient collision avoidance reward function: This study designs a multi-level gradient collision avoidance reward function, improving collision avoidance performance. By introducing a task-related search and rescue progress reward tied to remote sensing-identified high-value areas, it prevents UAVs from deviating from target areas due to collision avoidance. Meanwhile, gradient penalty factors are assigned to short-range, medium-range, and long-range neighboring UAVs to achieve proactive collision avoidance, balancing collision safety and search efficiency.

2. Related Work

2.1. Research Status of UAV Path Planning Methods

Currently, the mainstream methods for unmanned aerial vehicle (UAV) path planning are categorized into two major classes: traditional path planning methods and reinforcement learning algorithms. Among these, traditional path planning methods are mainly divided into three types, namely classical heuristic algorithms, centralized control methods, and distributed methods [15,16,17]. Classical heuristic algorithms, including heuristic search and graph traversal, are characterized by clear principles, superior local path optimization performance, and the ability to effectively address the basic obstacle avoidance problem for single UAVs. Nevertheless, they exhibit low collaborative compatibility in large-scale multi-UAV collaborative search and rescue scenarios [15]. Centralized control methods based on mixed-integer programming rely on a central node to execute unified path planning, which offers strong global coordination capability for path layout. However, their computational complexity increases exponentially with the number of UAVs, and the failure of the central node in mountainous areas with weak signal coverage may lead to the paralysis of the entire UAV fleet [16,17]. Distributed methods enhance system stability to a certain extent by reducing reliance on central nodes, yet they encounter technical bottlenecks such as insufficient collaboration and a tendency to converge to local optima, thus failing to achieve a dynamic balance between search and rescue efficiency and collision avoidance safety [19].

The limitations inherent in traditional path planning methods have driven the technological breakthrough of reinforcement learning (RL). Among RL algorithms, Q-Learning has been selected as the preferred technical approach in this study for mountain search and rescue scenarios, owing to its concise principles, model independence, and ease of deployment. It can not only compensate for the shortcomings of traditional methods in dynamic environment adaptation and large-scale collaboration but also lower the deployment threshold for small- and medium-sized UAV swarms, thereby laying a solid technical foundation for subsequent algorithm optimization [15].

2.2. Research on Q-Learning-Based Small-Scale UAV Path Planning

The Q-Learning algorithm has been extensively applied in the field of single-UAV path planning, with its core research focusing primarily on the optimization of basic obstacle avoidance in complex mountainous terrain [20,21,22]. Yan et al. [20] improved the Q-table initialization strategy and optimized the state space representation by incorporating Euclidean distance, which significantly enhanced the obstacle avoidance efficiency of single UAVs operating in complex mountainous terrain, resulting in a task completion rate of over 95%. Vasconcelos et al. [21] introduced a path-smoothing penalty term to optimize Q-Learning, which reduced the turning frequency of UAVs and improved their low-altitude flight stability in mountainous areas. Imanberdiyev et al. [22] integrated battery state awareness into the algorithm and accelerated convergence via goal-directed exploration, achieving single-UAV path optimization under energy constraints. These studies have effectively addressed the problem of single-UAV adaptation to mountainous terrain, but they only support simple terrain traversal and do not involve the core requirement of multi-UAV collaborative interaction.

With the escalating demand for operational efficiency in mountain search and rescue missions, research has gradually shifted toward small-scale multi-UAV collaborative decision-making mechanisms, and the Q-Learning algorithm has evolved into two core technical routes: the shared Q-table and the independent Q-table [23,24,25,26]. In the shared Q-table route, San et al. [23] proposed a global shared Q-table architecture, where all UAVs utilize a unified decision-making model and update experience through a central node, thus simplifying the collaborative logic. Kim et al. [24] further optimized this approach by adopting a hybrid strategy of a shared Q-table plus local experience caching, which improved path consistency in mountain collaborative inspection scenarios involving 30 UAVs. In the independent Q-table route, Rui et al. [25] combined Q-Learning with fuzzy control, enabling each UAV to make autonomous decisions based on local information and enhancing formation maneuverability. However, both routes exhibit notable limitations in mountain search and rescue scenarios: the shared Q-table neglects differences in task priorities, while the independent Q-table is prone to the passive predicament of post-collision reset due to insufficient local perception. Overall, existing research on Q-Learning in single-UAV and small-scale multi-UAV scenarios has addressed some fundamental issues, but it fails to meet the stringent requirements of large-scale mountain search and rescue operations in terms of collaborative efficiency, collision avoidance safety, and dynamic environmental adaptability.

2.3. Research on Q-Learning-Based Large-Scale UAV Path Planning

To break through the application limitations of small-scale collaboration, recent studies have gradually extended Q-Learning to large-scale UAV scenarios [27,28,29,30,31]. Xu et al. [27] proposed a partitioned shared Q-table strategy, which divides the search area into sub-regions where UAVs share experience within each sub-region. This strategy reduced the computational complexity by 40% in scenarios with 200 UAVs, yet it failed to resolve the collision issue caused by the superposition of terrain obstacles at the boundaries of mountainous areas. Li et al. [29] introduced graph neural networks to extract dynamic features of neighboring UAVs and optimized the Q-value update rules via feature fusion, which enhanced the obstacle avoidance efficiency in scenarios with 300 UAVs. Nevertheless, this method imposes high requirements on hardware computing resources. Hu et al. [31] constructed a collaborative penalty matrix by recording the statistical information of neighboring UAVs’ actions, which accelerated the algorithm convergence speed, but the communication overhead surges with the increase in the number of UAVs.

Current research has not yet developed a complete solution suitable for large-scale mountain search and rescue operations, with core bottlenecks concentrated in three aspects. First, dynamic environment modeling can only simulate the distribution of fixed obstacles, failing to reproduce the characteristics of dynamic disturbances such as sudden air currents. Second, the lightweight decision-making architecture struggles to balance the technical contradiction between lightweight deployment and efficient collaboration. Third, the reward function does not distinguish between regional value differences, making it unable to guide UAVs to prioritize the coverage of high-probability areas for missing persons. These issues ultimately lead to three core challenges that remain unresolved: insufficient environmental adaptability, lack of dynamic perception, and low search and rescue efficiency. To address these existing problems, this study proposes the HGR-QL optimization algorithm. Through the collaborative design of dynamic mountain environment modeling, a hierarchical independent Q-table architecture, and a multi-level gradient collision avoidance reward function, this algorithm systematically resolves the core pain points of large-scale multi-UAV mountain search and rescue, providing technical support for practical rescue scenarios.

3. Methods

3.1. System Framework

As shown in Figure 1, the framework is composed of two core modules connected by bidirectional data exchange: the upper HGR-QL decision training module and the lower multi-UAV mountain search and rescue execution module. The hierarchical Q-table architecture lies at the heart of this framework. Each unmanned aerial vehicle maintains its own individual Q-table, which enables autonomous local decision-making based on real-time environmental perception and the drone’s own state. This individual Q-table ensures reliable operation even under weak signal conditions that are common in mountainous areas. Meanwhile, a shared regional Q-table aggregates high-quality positive reward experiences from all drones within the same area. This shared table provides a global coordination reference for the entire swarm, effectively reducing communication overhead and avoiding redundant exploration in complex terrain. The multi-level gradient reward function calculates comprehensive rewards by integrating factors such as fixed obstacle avoidance, dynamic interference avoidance, collision prevention with other drones, target search priority, energy consumption, task progress, and wind field effects. It guides each drone to actively avoid obstacles, prioritize high-value search areas identified by remote sensing data, and optimize flight paths to conserve energy. This design overcomes the limitations of traditional Q-Learning in complex dynamic scenarios, demonstrating the novelty and practical value of our approach for real-world mountain rescue missions.

Figure 1. HGR-QL Schematic diagram of the HGR-QL system. The HGR-QL system comprises two core modules: the HGR-QL decision training module (Top) and the multi-UAV mountain search and rescue execution module (Bottom), with bidirectional communication enabling data exchange. The key modules include the hierarchical Q-table and gradient reward function, detailed in Section 3.3 and Section 3.4.

3.2. Complex Mountain Environment Simulation

Focusing on multi-rotor UAVs widely used in mountain search and rescue due to their high maneuverability and low-altitude hovering advantages, this study designs a safe flight speed of 5–10 m/s and a flight altitude of 50–200 m above the ground consistent with the low-altitude flight standard in China’s Mountain Search and Rescue Operation Guidelines. The used SRTM DEM remote sensing data with a resolution of 30 m can fully capture mountain terrain characteristics such as cliffs and dense forests, meeting the path planning requirements of low-speed and low-altitude multi-rotor UAVs; for high-speed fixed-wing UAVs (speed > 30 m/s), high-resolution terrain data such as 5 m resolution is required. The 50 × 50 two-dimensional grid corresponds to a 500 m × 500 m actual search and rescue area, supporting the collaboration of 300–400 small- and medium-sized multi-rotor UAVs, which is aligned with the UAV coverage standard for mountain search and rescue issued by China’s Ministry of Emergency Management.

The use of a 2D grid is justified by three key reasons aligned with practical mountain search and rescue requirements: small- and medium-sized UAVs (300–400 units) operate at low altitudes (50–200 m), where the core challenge is navigating horizontal terrain obstacles (cliffs, forests) rather than 3D altitude changes. The 2D grid effectively models the primary search coverage plane. A 2D grid drastically reduces the state space compared to 3D grids, enabling HGR-QL to run on standard CPU hardware without GPU acceleration, which is critical for real-world deployment. Fixed obstacles (20%) and dynamic interference sources are derived from SRTM DEM data, capturing the essential topographic complexity needed for collision avoidance and path planning. The grid environment is constructed to mimic the key characteristics of complex mountain terrain (e.g., irregular obstacles, dynamic interference, and weak communication signals) referenced from remote sensing-based mountain disaster datasets [3,27], ensuring high fidelity to practical search and rescue scenarios.

3.2.1. Flight Obstacles

The environment includes two types of constraints: fixed obstacles and dynamic interferences. Fixed obstacles account for 20% of the grid, which is consistent with the obstacle density statistics of mountainous areas in the SRTM DEM remote sensing dataset [3]. They are generated via Bernoulli random distribution combined with terrain continuity rules—initial obstacle points are marked by random seeds, then expanded into contiguous areas using an 8-neighborhood diffusion algorithm to simulate the irregular distribution of cliffs and dense forests in real mountains [1]. Ten dynamic interference sources are randomly distributed in non-obstacle areas, moving 1–2 grids every 10 decision steps with uniform direction distribution, each covering a 3 × 3 grid. This setting references the dynamic wind field and temporary hazard characteristics in the SAREnv mountain rescue dataset [27], mimicking sudden air currents and temporary thunderstorm zones in complex mountain environments [3]. Five state codes are adopted in the 50 × 50 grid to simulate the multi-UAV operating environment for mountain search and rescue, as defined in Table 1.

Table 1. Definition of grid cell states.

Number	State	Description
0	Navigable area	UAVs can pass normally
1	Fixed obstacles	Typical impassable mountain terrain (e.g., cliffs, dense forests, gullies)
2	Dynamic interference sources	Dynamic risks (e.g., sudden air currents)
3	UAV starting point	3 regions set based on task partitioning
4	UAV target point	1 primary target and 3 backup targets

The multi-UAV mountain search and rescue operating environment is shown in Figure 2.

Figure 2. The environment of multi-UAV mountain searching and rescuing should be listed as: (a) spatial distribution of fixed obstacles; (b) dynamic interference source distribution and movement range; (c) movement trajectory of dynamic interference source (move 1–2 grids per 10 decision steps); (d) top view of 3 × 3 coverage range of dynamic interference source; (e) 3D view of 3 × 3 coverage range of dynamic interference source.

This figure visually demonstrates the environmental modeling and core interaction logic of the HGR-QL algorithm in real-world mountain search and rescue scenarios, providing visual support for the algorithm’s path planning and collision avoidance decisions. Figure 2a,b, present the 50 × 50 dynamic grid mountain search and rescue environment. Fixed obstacles (gray blocks) accounting for 20% of the grid are generated based on SRTM DEM remote sensing data, with multi-region UAV starting points (green blocks), primary target points (blue star markers), and backup target points (purple star markers) delineated. Figure 2b additionally introduces dynamic interference sources (orange squares) to simulate dynamic risks such as sudden mountain air currents and rockfalls, fully reproducing the complex mountain terrain features and distribution of search and rescue task elements. Figure 2c shows an example of a single UAV’s movement trajectory. The UAV moves 1–2 grids every 10 decision steps, traveling from the starting point (green) toward the target area while avoiding fixed obstacles, intuitively reflecting the algorithm’s obstacle avoidance and task progress logic within the local grid.

Figure 2d,e depict the influence range of dynamic interference sources from 2D and 3D perspectives respectively: the orange dashed box represents the interference area. When a UAV enters this range, a gradient penalty is triggered to guide its rapid evacuation from the high-risk zone, and the penalty terminates immediately once the UAV departs. This mechanism is consistent with the three-stage interaction logic in Figure 3, ensuring the algorithm’s robustness under dynamic disturbances.

Figure 3. Schematic diagram of UAV spatial flight. The UAV adjusts movement direction based on obstacle distribution and neighbor positions, avoiding fixed obstacles (State 1) and dynamic interference sources (State 2), listed as: (a) UAV movement direction adjustment (avoiding fixed obstacles and dynamic interference sources); (b) neighbor distance grade perception (5 × 5 local grid); (c) dynamic interference source interaction process.

3.2.2. Flight Rules

Flight rules are designed to align with real mountain rescue scenarios, focusing on three core aspects: UAV movement, neighbor position perc communication between UAVs and the base station options, and dynamic interference mechanisms, enhancing multi-UAV adaptability to complex environments. For the convenience of algorithm verification, this study assumes perfect communication between UAVs and the base station which includes no delays, no packet loss and real-time updates of environmental conditions. These simplifying assumptions allow us to focus on verifying the core logic of the HGR-QL algorithm, which is hierarchical decision-making and gradient reward-based collision avoidance without being distracted by communication or environmental perception complexities. However, these assumptions deviate from the actual weak-signal and dynamic mountain environments, which will be addressed in future work. The UAV action space is shown in Table 2, neighbor distance grades (k) in Table 3, and dynamic interference mechanisms in Table 4.

Table 2. Definition of UAV action space.

Action	Description
(−1, 0)	Move up
(1, 0)	Move down
(0, −1)	Move left
(0, 1)	Move right
(−1, −1)	Move up-left
(−1, 1)	Move up-right
(1, −1)	Move down-left
(1, 1)	Move down-right

Table 3. Definition of neighbor distance grade k.

k	Neighbor Distance	Description
0	None	No neighbors within perception range
1	≤2 grids	Short-range neighbors
2	2 < distance ≤ 5 grids	Medium-range neighbors
3	>5 grids	Long-range neighbors

Table 4. Definition of dynamic interference mechanisms.

Mechanism	Description
Movement rule	Moving 1 grid and avoiding fixed obstacles
Scope of influence	UAVs and interference source overlap, affecting the grid
Interaction penalty	UAVs enter the interference area, imposing a fixed penalty

Consistent with low-altitude UAV flight characteristics in mountainous areas [4], the action space includes 8 directional movements. Action constraints are introduced: UAVs at grid boundaries (e.g., x = 0) are prohibited from moving left (−1, 0) and movements toward short-range neighbors are forbidden to reduce collision risks.

UAVs perceive a 5 × 5 local grid (no global state required), with neighbor position information updated every step to ensure real-time performance. This perception range is determined based on the average detection distance of small UAVs in mountainous areas [6]. UAV’s states are defined as a four-tuple:

$$S = (x, y, p, k),$$

(1)

where (x, y) denotes the real-time grid coordinates (0 ≤ x< 50, 0 ≤ y < 50); p is the remaining battery power (initial value is 200, decreasing by 1 per step); and k is the neighbor distance grade, reducing state dimensionality and computational complexity via “distance grade compression”.

The UAV spatial flight schematic is shown in Figure 3, which illustrates three core mechanisms: Figure 3a shows a real-time movement direction adjustment to avoid fixed and dynamic obstacles and 5 × 5 local grid neighbor distance grading for gradient collision avoidance in Figure 3a–c represents a three-stage interaction process with dynamic interference sources (approach, penalty, evacuation). Consistent with UAV mountain flight and neighbor perception rules, dynamic interference sources trigger immediate penalties that persist only while the UAV remains in the interference area, ceasing once the UAV departs—guiding rapid evacuation from high-risk zones with a grid-level influence scope. Consistent with UAV mountain flight and neighbor perception rules, dynamic interference sources trigger immediate penalties that persist only while the UAV remains in the interference area, ceasing once the UAV departs—guiding rapid evacuation from high-risk zones with a grid-level influence scope.

3.2.3. Flight Management

Integrated management coordinates the overall flight scheduling of multiple UAVs, combining regional scheduling with target point management to avoid airspace congestion and target conflicts. The integrated management rules are shown in Table 5.

Table 5. Integrated management rules.

Integrated Management	Description
Regional division	The 50 × 50 grid is divided into 3 independent 17 × 17 regions (each assigned 100 UAVs and exclusive targets)
Batch scheduling	300 UAVs take off in 3 batches with a 50-step interval (avoid airspace congestion)
Dynamic target update	Primary targets are occupied switching to backup targets (real-time monitoring of target occupancy)

The 50 × 50 grid is divided into Region A (0–16 rows/columns), Region B (17–33 rows/columns), and Region C (34–49 rows/columns). Each region is assigned 100 UAVs, 1 high-probability primary target, and 3 backup targets. Integrated management involves batch scheduling (3 batches with 50-step intervals) and real-time target monitoring, including automatic switching to backups when primary targets are occupied, with reassignment after target release to avoid inter-batch and intra-batch resource conflicts. Three UAV batches (100 UAVs per batch) are scheduled with 50-step intervals to reduce inter-batch and intra-batch airspace congestion. However, under high-density conditions (400 UAVs in a 50 × 50 grid), a “saturation point” where agents compete for the same grid cells may occur. HGR-QL addresses this challenge through two core mechanisms: firstly, the collision penalty R_col and task progress reward R_prog work together to guide UAVs away from crowded areas and toward high-value search targets, preventing the formation of collision traps. Secondly, each UAV makes path-planning decisions based on local perception, reducing dependence on global communication and alleviating congestion in high-density environments.

Figure 4a illustrates the batch scheduling timeline of 3 UAV batches (100 UAVs per batch) operating at 50-step intervals. This staggered release strategy avoids inter-batch and intra-batch airspace congestion, which is critical for large-scale UAV swarms (300–400 UAVs) in mountain search and rescue. It ensures that each batch has sufficient spatial and temporal resources to perform search tasks, directly reducing collision risks and improving operational efficiency. The flowchart of regional integrated management is shown in Figure 4. Figure 4b presents the target management flowchart driven by the task center. It includes the core processes of main target occupancy detection, main/backup target switching, and backup target resource release. This mechanism ensures that UAVs can dynamically reallocate search resources when the main target is occupied, aligning with the actual mountain search and rescue workflow and enhancing the robustness of task execution.

Figure 4. Flowchart of regional integrated management. The partitioned and batch scheduling strategy aligns with the actual mountain search and rescue workflow, improving operational efficiency and reducing collision risks. It is listed as: (a) batch scheduling timeline (3 UAV batches operating in 50-step intervals to avoid airspace congestion); (b) target management flowchart (task center-driven main/backup target detection, switching, and resource release process).

3.3. Hierarchical Independent Q-Table Architecture

The hierarchical independent Q-table architecture is divided into group and individual layers, each with independent Q-tables storing globally shared optimal experience (upper layer) and exclusive individual experience (lower layer).

3.3.1. Individual Q-Table

Each UAV is equipped with a dedicated Q-table of dimension [r × c × l × 4], where Q (x, y, a, k) represents the long-term cumulative reward of executing action at coordinates (x, y) with neighbor distance grade k. The individual Q-table records private experience, with updates based on the current action value. The target value integrates both the optimal estimate of the new state from the individual Q-table$( {max}_{a^{\prime }}{Q}_{ind}\left(s^{\prime },a^{\prime },k^{\prime }\right))$ and the optimal gain from the group Q-table $\left({max}_{a^{\prime }}{Q}_{reg}\left(s^{\prime },a^{\prime }\right)\right)$ via weight ω. Final correction uses a time-decaying learning rate α(t) to balance individual adaptation and group collaboration:

$$Q_{ind}\left(s,a,k\right)\leftarrow Q_{ind}\left(s,a,k\right)+\alpha \left(t\right)·\left[R+\gamma ·\left(\omega ·{max}_{a^{\prime }}{Q}_{reg}\left(s^{\prime },a^{\prime }\right)+\left(1-\omega \right)·Q_{ind}\left(s^{\prime },a^{\prime },k^{\prime }\right)\right)-Q_{ind}\left(s,a,k\right)\right]$$

(2)

where$Q_{ind}\left(s,a,k\right)$ is the Q-value of state s, action a, and neighbor grade k in the individual Q-table; $Q_{reg}\left(s^{\prime },a^{\prime }\right)$ is the Q-value of the new state and new action in the regional Q-table; R is the immediate reward from the multi-objective fusion reward function;$\gamma$is the discount factor (typically 0.9–0.99, balancing immediate and future rewards); and $\omega$ is the regional experience weight ($0\le \omega \le 1$, controlling the contribution of regional experience to individual learning).

Each UAV executes low-level path planning based on local perception (obstacles, target distance, wind field). This layer handles real-time collision avoidance and trajectory adjustment, ensuring robustness against local dynamic disturbances.

3.3.2. Regional Q-Table

Each region is equipped with a shared regional Q-table of dimension [r × c × l], recording optimal path experience within the region. UAVs synchronize individual Q-table updates to the regional Q-table after 10 steps, with weighted average updates. The regional Q-table calculates the target value via the immediate reward R of the current state-action pair and the known optimal future gain of the new state (${max}_{a^{\prime }}{Q}_{reg}\left({\mathrm{s}}^{\prime },{\mathrm{a}}^{\prime }\right)$), then corrects the Q-value using α(t) to provide universal decision references for all individuals and reduce redundant exploration costs:

$$Q_{reg}\left(s,a\right)\leftarrow Q_{reg}\left(s,a\right)+\alpha \left(t\right)·[R+\gamma ·{max}_{a^{\prime }}{Q}_{reg}\left(s^{\prime },a^{\prime }\right)-Q_{reg}\left(s,a\right)]$$

(3)

where $Q_{reg}\left(s,a\right)$ is the Q-value of state s and action a in the regional Q-table; other parameters are consistent with Equation (2).

Each UAV synchronizes its positive reward experiences with the regional shared Q-table every 10 steps. This mechanism balances local autonomy and global collaboration. In mountain environments with weak signal coverage, temporary synchronization failures may occur due to signal blocking or interference. Under such scenarios, HGR-QL falls back to the individual independent Q-table: each UAV continues to execute path planning based on real-time local perception (obstacles, target locations) and the current gradient reward, ensuring that basic search and collision avoidance tasks are not interrupted. Once signal recovery is detected, the UAV will synchronize its accumulated experiences with the regional Q-table to restore global collaboration. Therefore, we also address the robustness against weak signal coverage: the individual Q-table acts as a reliable fallback during temporary synchronization loss, ensuring that the algorithm remains functional in real mountain environments where communication is not always perfect.

3.4. Multi-Level Gradient Collision Avoidance Reward Function

To address the limitations of passive collision avoidance and task deviation in traditional multi-UAV Q-Learning, a gradient-adjusted reward function is designed based on the distance decay principle. It guides active risk avoidance, reduces collisions, and prevents deviation from high-priority search areas—achieving precise alignment between reward/punishment intensity and mountain search and rescue progress. Efficient collaboration is realized without global communication or centralized control.

3.4.1. Components of the Multi-Objective Reward Function

The fused multi-objective reward function R includes six components: the fixed obstacle penalty, dynamic interference penalty, multi-UAV collision avoidance penalty, target reward, battery consumption penalty, task progress reward, and wind field disturbance penalty, with the formula:

$$R=R_{obs}+R_{dyn}+R_{col}+R_{tar}+R_{energy}+R_{prog}+R_{wind}$$

(4)

3.4.2. Sub-Functions of the Multi-Level Gradient Collision Avoidance Reward

Fixed obstacle penalty ($R_{obs}$): A large penalty enforces strict collision avoidance with fatal obstacles (e.g., cliffs). Pre-tests show that −100 achieves the best balance between safety and task progress: values smaller than −100 lead to overly conservative paths (reduced task completion rate), while values larger than −50 fail to prevent collisions effectively. The maximum penalty for collisions with fixed obstacles is as follows:

$$R_{obs} = \left\{\begin{array}{c}-100 \mathrm{Collision} \mathrm{with} \mathrm{fixed} \mathrm{obstacles}\\ 0 \mathrm{Otherwise}\end{array}\right.$$

(5)

Increasing the obstacle penalty magnitude effectively reduced collisions but increased task steps by 10–15% due to more conservative paths.

Multi-UAV collision avoidance penalty ($R_{col}$): The gradient penalty is based on distance d to the nearest neighbor (e.g., penalty = −50 for d = 1 grid, much higher than −4 for d = 5 grids):

$$R_{col} = \left\{\begin{array}{c}-50 \mathrm{d}\le 1 (\mathrm{Short} \mathrm{range})\\ -20 1<\mathrm{d}\le 3 (\mathrm{Medium} \mathrm{range})\\ -4 3<\mathrm{d}\le 5 \left(\mathrm{Long} \mathrm{range}\right)\\ 0 \mathrm{d}>5 (\mathrm{No} \mathrm{impact}) \end{array}\right.$$

(6)

The gradient penalty values (−50, −20, −4, 0) are designed based on safety priority and collision risk levels: a large penalty of −50 is imposed to strongly enforce immediate avoidance, as this distance represents imminent collision risk in mountain search and rescue scenarios where UAVs operate in tight formations; a moderate penalty of −20 is used to guide UAVs to maintain safe separation without overly disrupting their search trajectory; a small penalty of −4 is applied to gently discourage unnecessary detours while preserving collision awareness. No penalty (0) is applied when the distance is sufficient to guarantee flight safety.

Dynamic interference penalty (${\mathit{R}}_{\mathit{d}\mathit{y}\mathit{n}}$): A moderate penalty guides UAVs away from temporary high-risk areas without disrupting the overall search trajectory. A fixed penalty for entering dynamic interference areas guides rapid evacuation and environmental adaptability:

$$R_{dyn} = \left\{\begin{array}{c}-30 \mathrm{Within} \mathrm{dynamic} \mathrm{interference} \mathrm{area}\\ 0 \mathrm{Otherwise}\end{array}\right.$$

(7)

Target reward (${\mathit{R}}_{\mathit{t}\mathit{a}\mathit{r}}$): The reward for reaching target points (primary targets have higher priority than backups) is as follows:

$$R_{tar} = \left\{\begin{array}{c} 200 \mathrm{Reach} \mathrm{primary} \mathrm{target}\\ 150 \mathrm{Reach} \mathrm{backup} \mathrm{target}\\ 0 \mathrm{Otherwise}\end{array}\right.$$

(8)

The reward values (200, 150, 0) reflect task priority and search efficiency: a higher reward is assigned to emphasize the core mission of reaching high-priority search zones, which is critical for saving lives in mountain search and rescue; a slightly lower reward is used to encourage secondary search efforts when primary targets are unavailable, ensuring continuous task execution; no reward is given to avoid rewarding irrelevant movements, keeping UAVs focused on mission objectives.

Dynamic Battery consumption penalty (${\mathit{R}}_{\mathit{e}\mathit{n}\mathit{e}\mathit{r}\mathit{g}\mathit{y}}$): This is a fixed penalty per movement step:

$$R_{energy}=-1$$

(9)

The value of −1 is selected to be mild enough to not dominate the reward signal (so it does not override collision avoidance or target pursuit) but strong enough to incentivize energy-efficient trajectories, balancing exploration and resource conservation.

Task progress reward (${\mathit{R}}_{\mathit{p}\mathit{r}\mathit{o}\mathit{g}}$): This is a dynamic reward based on task completion progress rewards for closer proximity to targets, preventing deviation due to collision avoidance:

$$R_{prog}=50\times s (s=1-{\displaystyle \frac{\mathrm{Current} \mathrm{distance}}{\mathrm{Initial} \mathrm{distance}}})$$

(10)

The coefficient 50 scales the reward to be comparable with other reward components (e.g., target reward, collision penalty), ensuring it has a meaningful impact on the agent’s decision-making.

Wind penalty ($R_{wind}$): According to the actual wind field characteristics of mountainous areas, the penalty value is positively correlated with wind speed and the angle between the UAV’s flight direction and wind direction. When the wind speed exceeds 8 m/s (the safe flight limit of small multi-rotor UAVs in mountainous areas), a fixed penalty of −40 is imposed to guide UAVs to avoid high wind speed areas; when the wind speed is 0–8 m/s, the penalty value is calculated as follows:

$$R_{wind}=-5\times v\times cos\theta$$

where $v$ is the actual wind speed, and $\theta$ is the angle between the UAV’s flight direction and wind direction. The penalty value penalizes flight directions with large wind resistance and guides UAVs to choose the optimal flight path with small wind resistance.

3.5. Training Process

Taking the Q-Learning training of 300 UAVs as a typical scenario, this section elaborates on the distributed parallel training process of the HGR-QL algorithm. Through partitioned batching design, the algorithm enables efficient training on conventional CPU hardware without GPU acceleration, fully adapting to the resource constraints and multi-region collaboration requirements of mountain search and rescue (MSR) scenarios.

• Step 1: Initialization Phase

The total number of training rounds is set to 500, and 300 UAVs are divided into 3 independent batches of 100 units each, corresponding to Region A, B, and C to realize partitioned parallel training and avoid computing resource overload. The initialization phase completes three core configurations:

a: Individual Q-table Initialization: Each UAV is assigned a unique Q-table, with weights pre-optimized according to MSR (mountain search and rescue) task priorities—prioritizing low-battery return paths, target area proximity, and terrain obstacle avoidance to align with real-world rescue constraints.

b: Regional Shared Q-table Reset: The regional shared Q-table is reset to 0, designed to later aggregate positive experiences (reward ≥ 0) from each batch of UAVs, enabling cross-batch knowledge transfer without disrupting individual exploration.

c: Uniform State Reset: All UAVs are reset to a consistent initial state: returning to the designated rest position, setting a full battery level (100 units), clearing task progress counters, and resetting target distance to the maximum value. This ensures all batches start under identical conditions, eliminating initial state bias in performance comparisons.

• Step 2: Action Selection Phase

Improve the ε-greedy strategy: initial ε = 0.2, decaying by 0.02 every 200 rounds (minimum ε = 0.05). This design ensures sufficient exploration of complex mountain environments in the early stage and focuses on high-value actions in the later stage. For each UAV in the current batch, short-range neighbor detection is first performed to identify the distance grade of nearby UAVs, and movements toward short-range neighbors (within 1–2 grids) are strictly prohibited to reduce collision risks at the source. Select actions with probability ε or (1 − ε). Prohibit movements toward short-range neighbors to reduce collision risks.

• Step 3: State and Reward Update Phase

After executing the selected action, the UAV’s new state (x’, y’) is calculated, including remaining battery power (p’ = p − 1) and new neighbor distance grade (k’). Call the reward function to compute R, determine task completion status. Finally, the task status is determined: if the UAV successfully reaches the target area, it is marked as completed and the time cost is recorded; if the battery is exhausted, it is marked as failed. Meanwhile, task progress and target distance are updated to provide a basis for subsequent Q-table updates.

The reward function R is then computed, incorporating multi-dimensional incentives: positive rewards for approaching the target area, maintaining safe swarm spacing, and preserving battery life and negative penalties for colliding with terrain obstacles, drifting too far from the target, or depleting battery prematurely.

Additional environmental penalties for navigating into high-wind or low-visibility zones (per reviewer suggestions).

• Step 4: Q-table Update Phase

The current UAV’s individual Q-table is optimized using the improved Q-value updated Formula (2).

$$Q_{ind}\left(s,a,k\right)\leftarrow Q_{ind}\left(s,a,k\right)+\alpha \left(t\right)\left[R+\gamma ·\left(\omega ·{max}_{a^{\prime }}{Q}_{reg}\left(s^{\prime },a^{\prime }\right)+\left(1-\omega \right)·Q_{ind}\left(s^{\prime },a^{\prime },k^{\prime }\right)\right)-Q_{ind}\left(s,a,k\right)\right]$$

where α = 0.1 (learning rate) controls the weight of new information, and γ = 0.9 (discount factor) prioritizes immediate rewards while preserving long-term task value.

Optimize the current UAV’s individual Q-table using the improved Q-value update rule. After every 10 steps per batch, action information with R ≥ 0 (representing successful, safe actions) is synchronized from individual Q-tables to the regional shared Q-table. Updates are performed via a weighted average, where actions with higher rewards are assigned greater weight to amplify the impact of optimal strategies. This allows the regional Q-table to act as a collective knowledge base, propagating successful rescue patterns across all UAVs in the region while preserving individual exploration diversity.

• Step 5: Iteration Termination Phase

End the batch training when all UAVs in a batch complete the task; proceed to the next batch. End the current training round when all 3 batches are completed; repeat Steps 2–5 until 500 rounds are finished. This process, through batch isolation and round control, not only avoids computing resource congestion caused by large-scale UAV swarms but also gradually optimizes Q-table weights through multiple iterations, ultimately improving the overall search and rescue efficiency and robustness. The complete training process is visually presented in Figure 5.

Figure 5. Training flowchart of 300 UAVs.

4. Results

This section presents a comprehensive evaluation of the proposed hierarchical gradient reward Q-Learning (HGR-QL) method for multi-UAV mountain search and rescue path planning. Experiments systematically assess the model’s detection performance, validate the effectiveness of core modules, and analyze generalization capability, aiming to fully demonstrate its superiority and practicality in complex mountainous environments.

4.1. Experimental Setup

All experiments were conducted on a workstation equipped with an AMD Ryzen 7 5800H CPU (8 cores, 16 threads), 16 GB RAM, and no GPU acceleration—consistent with the deployment requirements of small- and medium-sized UAVs. The software environment included Python 3.12.4, NumPy 1.26.4, Matplotlib 3.8.4, and Pandas 2.2.2, with the multiprocessing library enabling distributed parallel training.

For the UAV experimental setup adapted to large-scale mountain search and rescue, the multi-UAV platform adopts 2200 KV high-efficiency brushless DC motors with 5-inch propellers. This configuration ensures a sufficient thrust-to-weight ratio to overcome dense obstacles and dynamic air currents in complex mountain terrain, enabling stable flight and flexible maneuvering for rapid rescue operations. Each UAV integrates a 6-axis IMU for accurate real-time attitude control, a Beidou combined positioning module for reliable positioning in low-signal mountain areas, and a 5.8 GHz digital image transmission module for high-speed environmental perception and data interaction. The open-source Pixhawk 4 flight controller with custom firmware is employed, optimized for multi-UAV collaborative decision-making and collision avoidance to meet the demands of large-scale swarm operations. With a 1.2 kg takeoff weight, 0.3 kg payload capacity, 25 min endurance, and 2 km communication range, the UAV balances lightweight maneuverability and mission requirements. All parameters comply with China’s Mountain Search and Rescue Operation Guidelines, fully satisfying the practical needs of large-scale rescue missions.

4.1.1. Experimental Scenarios

To better simulate complex and variable mountain search and rescue terrain and fully validate the overall performance and practical adaptability of the proposed HGR-QL algorithm, we design three targeted experimental scenarios: the basic scenario, large-scale scenario, and dynamic interference scenario. Specifically, the basic static scenario is built to test the basic path planning performance of the algorithm in conventional mountain environments; the large-scale scenario is constructed to verify the adaptive collaboration capability of the multi-UAV swarm when the number of drones expands; and the dynamic interference scenario is set to check the anti-interference robustness and stable operation ability of the algorithm under variable obstacles and airflow disturbances in real mountainous areas. In addition, two groups of controlled ablation experiments are designed to separately verify the effectiveness of the hierarchical independent Q-table architecture and the multi-level gradient collision avoidance reward function to clarify the core contribution of each key module to algorithm optimization. The experimental scenario settings are shown in Table 6, and the experimental parameters are detailed in Table 7.

Table 6. Experimental scenarios settings.

Scenario No.	Scenarios	Configuration
1	Basic scenario	20% fixed obstacles;
		No dynamic interference sources;
		300 UAVs in 3 batches
2	Large-scale scenario	20% fixed obstacles;
		No dynamic interference sources;
		100, 200, 300, and 400 UAVs
3	Dynamic interference scenario	20% fixed obstacles;
		10 dynamic interference sources;
		300 UAVs in 3 batches
4	Ablation Experiment 1	Dynamic interference scenario;
		Multi-level gradient reward
5	Ablation Experiment 2	Dynamic interference scenario;
		Hierarchical independent Q-table

Table 7. Experimental parameter settings.

Parameter	Value	Explanation
ε	0.05–0.2	The initial value of 0.2 ensures sufficient exploration in the early stage, and its decay balances exploration and exploitation.
γ	0.9–0.99	The core experimental value is 0.95, adapting to the global optimization needs of search and rescue tasks [15,27].
ω	0–1	Ω = 0.3, ensuring the local environmental adaptability of individual UAVs and utilizing regional shared experience.

The influence law of hyperparameters on algorithm performance includes the following: increasing the obstacle penalty value will reduce the number of collisions but may increase the number of task steps (conservative path); an excessively large learning rate α(t) will lead to unstable convergence, while an excessively small one will extend the training cycle; and increasing ω will enhance global collaboration but may reduce the adaptability of individuals to local dynamic interference. Due to time and computing resource constraints during the revision stage, systematic sensitivity analysis will be carried out in subsequent research to further optimize the parameter combination.

4.1.2. Experimental Baselines

Four baseline methods were selected for comparison Q-Learning [10]: a single-agent static method with a standard Q-table for value storage and an ε-greedy strategy for Q-value update and action selection; DIQ-L [25]: a fully distributed method without experience sharing, no central node or global Q-table; ASQ [32]: a centralized architecture with a central control node and global shared Q-table, lacking gradient penalty for dynamic collision avoidance; and LLM-QL [33]: a state-of-the-art hybrid reinforcement learning framework that relies on high-end GPU hardware and suffers from slow response in high-obstacle-density mountain scenarios.

By comparing with several typical baseline methods, the experimental results demonstrate that the proposed HGR-QL algorithm exhibits significant advantages in overall operational performance. Specifically, it shows stronger mountain terrain adaptability with a higher task completion rate, fewer collision instances, and lower communication latency. These advantages enable HGR-QL to serve as a more stable and practical solution for real-world mountain search and rescue missions. In addition, two ablation experiments are conducted to verify the individual contributions of the hierarchical independent Q-table architecture and the multi-level gradient collision avoidance reward function, respectively. The module and architecture characteristics between traditional Q-learning and the improved HGR-QL algorithm are compared and summarized in Table 8, which further illustrates the rationality and advancement of the proposed design.

Table 8. Comparison of key modules of multi-agent path planning algorithms.

Title 1	Q-Learning	DIQ-L	ASQ	LLM-QL	HGR-QL
Individual Q-table	√	√	√	√	√
Regional Q-table				√	√
Simple Reward	√	√	√	√	√
Multi-level Gradient Reward					√
Centralized Architecture			√
Distributed Architecture		√		√	√

4.1.3. Evaluation Metrics

To address practical mountain search and rescue problems and verify the performance of the proposed method, we adopted four core evaluations, as defined in Table 9: task completion rate, task steps, number of collisions, and communication delay.

Table 9. Definitions of four core evaluation metrics.

Metric	Definition
Task completion rate	The proportion of UAVs successfully reaching target points
Task steps	The number of steps taken by UAVs that successfully complete the task in each training round
Number of collisions	The total number of collisions in each training round
Communication delay	The communication delay per training round

4.2. Experimental Results

This section presents the comprehensive experimental results to evaluate the performance of the proposed HGR-QL algorithm against state-of-the-art baseline methods. We design three scenarios to simulate real-world operational conditions. Four core evaluation metrics are adopted to quantify task efficiency, collision safety, and communication overhead, as detailed in Table 10.

Table 10. Results of the four core evaluation metrics via different methods across the three scenarios.

Scenario No.	Metric	HGR-QL	QL *	DIQ-L *	ASQ *	LLM-QL *
1	Task completion rate	82.67 ± 2.26	50.96 ± 4.10	59.00 ± 0.35	53.74 ± 4.52	76.35 ± 3.78
	Task steps	8.73 ± 2.54	5.60 ± 1.07	5.74 ± 0.24	6.28 ± 1.31	7.82 ± 2.11
	Number of collisions	12.14 ± 0.08	49.01 ± 4.02	95.00 ± 0.15	58.42 ± 4.26	25.68 ± 3.94
	Communication delay	150.04 ± 1.31	249.90 ± 5.92	399.87 ± 7.67	220.18 ± 5.17	185.64 ± 4.92
2	Task completion rate	88.19 ± 1.58	48.53 ± 3.48	61.00 ± 0.29	57.26 ± 3.98	79.72 ± 4.15
	Task steps	12.95 ± 3.34	5.23 ± 0.89	6.13 ± 0.17	7.55 ± 1.74	13.47 ± 2.46
	Number of collisions	13.01 ± 0.06	51.46 ± 3.43	94.00 ± 0.21	62.89 ± 3.84	42.85 ± 4.32
	Communication delay	119.99 ± 0.85	280.02 ± 4.51	449.94 ± 8.30	255.33 ± 4.82	212.37 ± 5.33
3	Task completion rate	74.47 ± 33.90	46.99 ± 25.95	57.06 ± 1.15	42.68 ± 26.37	63.56 ± 27.12
	Task steps	7.49 ± 2.47	5.13 ± 1.20	4.99 ± 1.28	8.13 ± 2.15	10.63 ± 2.35
	Number of collisions	25.44 ± 33.97	53.01 ± 25.95	94.94 ± 1.15	75.35 ± 26.89	68.33 ± 28.45
	Communication delay	100.00 ± 1.87	250.05 ± 4.15	499.99 ± 5.25	289.75 ± 5.64	238.51 ± 6.27

Note: Algorithms marked with * reproduced their paper logic.

4.2.1. Task Efficiency

This subsection analyzes the task efficiency of the proposed HGR-QL algorithm in the basic static scenario, focusing on two key metrics: task completion rate and task steps. The performance of HGR-QL is compared with four baseline methods, as visualized in Figure 6.

Figure 6. Performance comparison in the basic scenario: (a) task completion rate curve of different methods; (b) task steps curve of different methods. The curves correspond to HGR-QL, QL, DIQ-L, ASQ, and LLM-QL, respectively (the legend is shown in subfigure (a)).

As shown in Figure 6, the static scenario (Scenario No.1) designed in this study simulates a basic mountain search and rescue environment. The task completion rate reflects the core goal achievement of search and rescue missions, while the task steps indicate the resource consumption efficiency in reaching the goal. The combination of these two metrics can effectively verify task efficiency. HGR-QL achieves a task completion rate of 82.67% in the static scenario, significantly higher than that of traditional QL and DIQ-L, and only slightly lower than LLM-QL. Its task steps are 8.73, higher than DIQ-L but lower than ASQ and LLM-QL. HGR-QL’s completion rate stably exceeds 80% after 200 rounds, with a faster convergence speed than most baselines. Experiments demonstrate that traditional QL has a low completion rate and high collision frequency due to the lack of collaboration in the global Q-table, while DIQ-L, despite fewer steps, achieves a completion rate of less than 60% due to the absence of experience sharing. In contrast, HGR-QL balances completion rate and steps. The task efficiency of HGR-QL stems from two aspects: first, the integration of positive experience by the regional shared Q-table and the autonomous adaptation of the individual Q-table, and second, the “task progress reward” in the multi-level gradient collision avoidance reward function that guides UAVs to focus on high-value areas, avoiding deviation from core tasks due to collision avoidance.

4.2.2. Communication Efficiency

The large-scale scenario (Scenario No.2) simulates the large-scale operational needs of actual mountain search and rescue by adjusting the number of UAVs (100–400). Communication delay directly reflects the collaborative efficiency of the algorithm in weak signal environments and is a core indicator to verify the feasibility of large-scale deployment. HGR-QL’s communication delay is 119.99 ms when there are 300 UAVs, significantly lower than other methods. Even when the number of UAVs increases to 400, its delay remains stable at around 120 ms, while the delays of baseline methods all exceed 250 ms. The communication advantage of HGR-QL comes from its hierarchical independent Q-table architecture, which only synchronizes positive and effective experience within the region instead of full-scale data, greatly reducing communication load. Meanwhile, the weighted update mechanism of the regional Q-table reduces cross-regional data interaction, avoiding transmission congestion in weak signal environments.

4.2.3. Safety Performance

This subsection evaluates the safety performance of the proposed HGR-QL algorithm in the dynamic interference scenario, with the number of collisions as the core metric. The collision avoidance performance of HGR-QL is compared with four baseline methods, as visualized in Figure 7.

Figure 7. Collision numbers in dynamic scenario.

As shown in Figure 7, the dynamic interference scenario (Scenario No.3) designed in this study incorporates 20% fixed obstacles and 10 randomly moving interference sources to simulate dynamic risks such as sudden air currents and temporary dangerous areas in real mountainous areas, focusing on “whether the algorithm can ensure the safety of UAV swarms under complex interference”. The number of collisions directly measures “collision avoidance reliability” and is a core indicator of safety performance. HGR-QL’s collision count is 25.44 in this scenario, significantly lower than that of traditional QL, DIQ-L, and ASQ, and only slightly higher than LLM-QL. HGR-QL real-time captures the positions of interference sources and neighbors through a local 5×5 perception range, adjusting paths in advance. The multi-level gradient collision avoidance reward function of the proposed method enables UAVs to actively avoid risks, effectively improving collision avoidance performance. Meanwhile, the hierarchical independent Q-table allows UAVs within the region to share collision avoidance experience, reducing repeated trial and error. The synergistic effect of these two components enables HGR-QL to maintain a high task completion rate of 74.47% while achieving a low collision count in dynamic interference scenarios, verifying its safety and reliability in real mountain risk environments.

4.2.4. Robustness Analysis

This section validates the anti-interference capability of HGR-QL in real-world mountain uncertainty scenarios, combining the spatial perception and interference interaction logic shown in Figure 3 with the comparative experimental data under dynamic disturbances. Under three typical disturbances—wind field disturbances, communication loss, and dynamic target relocation—HGR-QL, relying on the 5×5 local grid grading perception and three-stage dynamic interference interaction mechanism illustrated in Figure 3, maintains a task completion rate above 70%, limits the increase in average collision count to within 15%, and keeps communication latency stable below 110 ms even when wind speed approaches the multi-rotor safety limit (12 m/s), regional synchronization packet loss reaches 50%, or targets shift by ±5 grids per minute, significantly outperforming baseline methods such as LLM-QL.

Further comparison reveals that the performance degradation of HGR-QL under various uncertainties is far smaller than that of LLM-QL, which relies on global inference: the task completion rate drops by only 4.15% under wind field disturbances, 3.42% under communication loss scenarios, and remains at 71.89% even under dynamic target relocation. This demonstrates that the fallback mechanism of the hierarchical independent Q-table and the multi-level gradient reward design can effectively resist unknown dynamic disturbances, communication uncertainties, and target position changes, enabling the algorithm to retain reliable search efficiency and flight safety in resource-constrained real mountain environments, providing more stable technical support for complex mountain search and rescue missions.

4.3. Module Performance

To verify the practical application value of the innovatively designed hierarchical independent Q-table architecture and multi-level gradient collision avoidance reward function in this study, two sets of ablation experiments were conducted based on the dynamic interference scenario. This scenario can simulate the complex dynamic risks of real mountainous areas and serves as a rigorous environment to validate the robustness of the algorithm’s core modules. From the Table 11, the complete HGR-QL achieves a task completion rate of 74.47%, a collision count of 25.44, and a communication delay of 100.00 ms. After removing the hierarchical Q-table (Ablation Experiment 1), the task completion rate drops to 71.83%, the task steps increase to 9.22, and the communication delay rises to 119.93 ms. When replacing the gradient reward function with a traditional fixed-penalty function (Ablation Experiment 2), the task completion rate further decreases to 69.85% and the collision count increases to 29.39, with significant degradation observed in all core metrics. As shown in the curves of Figure 8, the fluctuation range of metrics in both ablation experiments is larger than that of the complete HGR-QL, and the convergence speed is significantly slower, verifying that removing either core module individually will lead to a decline in algorithm performance. The synergistic effect of the two modules is the core support for HGR-QL to maintain excellent performance in complex mountain scenarios, fully verifying the scientificity and necessity of the innovative design.

Table 11. Ablation experiment results (dynamic interference scenario).

Method	Task Completion Rate	Task Steps	Number of Collisions	Communication Delay
HGR-QL	74.47 ± 33.90	7.49 ± 2.47	25.44 ± 33.97	100.00 ± 1.87
Ablation Experiment 1	71.83 ± 35.96	9.22 ± 3.97	27.94 ± 36.14	119.93 ± 2.25
Ablation Experiment 2	69.85 ± 6.18	8.94 ± 4.59	29.39 ± 34.30	120.02 ± 1.00

Figure 8. Metrics comparison in ablation experiment ((top): HGR-QL vs. Ablation Experiment 1; (bottom): HGR-QL vs. Ablation Experiment 2): (a) task completion rate curves; (b) task step curves; (c) number of collision curves; (d) communication delay curves. The legend in each subfigure distinguishes HGR-QL from the two ablation methods, reflecting the performance degradation after removing the core modules.

5. Conclusions

Multi-unmanned aerial vehicle (UAV) systems face critical challenges in complex mountainous regions, including insufficient environmental adaptability, low search and rescue (SAR) efficiency, and poor collision avoidance performance—compounded by remote sensing (RS) signal attenuation and irregular terrain. To address these issues, this study proposes HGR-QL, an optimized Q-Learning-based path planning method. A 50 × 50 dynamic grid environment is constructed to mimic real mountain terrain, incorporating 20% fixed obstacles and randomly moving interference sources, with obstacle distribution and continuity referenced to RS-derived datasets such as the SRTM DEM. The method integrates a hierarchical independent Q-table architecture and a multi-level gradient collision avoidance reward function, where Q-table weights are aligned with RS-interpreted high-value area priorities and reward functions are tied to RS-identified target zones. Comparative experiments across three scenarios and ablation tests are conducted against four baseline methods: traditional Q-Learning, DIQ-L, ASQ, and LLM-QL. Results demonstrate that HGR-QL achieves a 74.47% task completion rate, 25.44 average collisions per UAV, and a stable 100.00 ms communication latency in dynamic mountain scenarios. This method effectively enhances the efficiency, safety, and stability of multi-UAV mountain SAR operations, providing a lightweight and scalable solution for UAV swarm deployment and supporting the critical “golden 72 h” rescue window.

6. Discussion

This study proposes the HGR-QL algorithm to address the path planning problem of large-scale multi-UAV mountain search and rescue, and adds a dedicated discussion section at the end of the paper to systematically compare the obtained results with existing mainstream flight planning methods, fully responding to the reviewer’s suggestions to strengthen result comparison and robustness analysis.

6.1. Advantages of the HGR-QL Algorithm

In terms of core performance and environmental adaptability, HGR-QL demonstrates distinct engineering advantages over existing flight planning methods: compared with traditional Q-Learning, which relies on a centralized global Q-table and suffers from a high communication latency of 249.90 ms in 300-UAV scenarios, HGR-QL stabilizes the latency at 119.99 ms through its hierarchical independent Q-table architecture, fundamentally alleviating the communication load in large-scale collaboration. Compared with DIQ-L, which adopts distributed Q-tables but lacks regional collaboration and gradient collision avoidance mechanisms, HGR-QL improves the task completion rate by 15–30% and reduces the collision count by more than 50% by introducing a regional shared Q-table and multi-level gradient reward function, achieving a better balance between collision avoidance efficiency and task progress. Compared with ASQ, which depends on a central node for global scheduling and is prone to single-point failures in weak-signal mountain areas, HGR-QL shows stronger robustness with only 1/5 of the communication latency. Compared with LLM-QL, a hybrid reinforcement learning method based on large language models, HGR-QL does not rely on large-scale pre-trained models or GPU acceleration, has a lower hardware deployment threshold, and maintains a task completion rate above 70% under dynamic disturbances, making it more suitable for practical search and rescue scenarios of small- and medium-sized UAV swarms. Further robustness tests confirm that HGR-QL retains reliable performance under real mountain uncertainties, with a much smaller performance degradation than baseline methods such as LLM-QL.

6.2. Limitations and Future Work

While HGR-QL verifies its core advantages in idealized mountain simulations, several limitations remain to be addressed to meet publication standards: the current 50 × 50 2D grid modeling only simulates terrain complexity through fixed obstacles, without fully considering 3D terrain elevation changes in extreme mountainous areas; only wind field disturbances are incorporated, and the impact of extreme weather such as heavy rain and thunderstorms is not considered; long-distance flight energy consumption is not optimized, limiting the flight range due to battery capacity; and, additionally, the simulation assumes perfect communication, real-time environmental perception, and no UAV dynamics modeling, which is inconsistent with the weak signal, dynamic environmental changes, and multi-rotor dynamics in real mountain scenarios. Furthermore, the comparison with LLM-QL requires further clarification on the rationality of baseline selection and how mountain terrain adaptability differences affect the results.

Aiming at the above limitations, future work will focus on enhancing the depth of the study and the practicality of the algorithm to strengthen its academic contribution. We will integrate SRTM 3D terrain data to extend the 2D grid model to a 3D environment with altitude constraints, adapting to high-altitude mountain search and rescue scenarios. Meanwhile, multi-source meteorological sensor data will be fused to add extreme weather penalty terms to the reward function, further optimizing the algorithm’s environmental adaptability. In addition, a multi-UAV energy collaboration and temporary charging mechanism will be designed to extend the continuous search and rescue time of the swarm in mountainous areas through intra-regional energy sharing and nearby charging, breaking through the battery capacity limit. Finally, communication delay models, dynamic environmental update intervals, and simplified multi-rotor dynamics will be incorporated into the simulation framework to break the idealized assumptions of “perfect communication and real-time perception”, improving the algorithm’s practicality and robustness in real mountain search and rescue scenarios. These improvements will enable HGR-QL to better adapt to the real constraints of complex mountain search and rescue, providing more reliable technical support for seizing the “golden 72 h” rescue window.