Performance Evaluation of UAV-Coordinated Multi-Scenario Disaster Relief Operations Based on ResNet and Attention Mechanism

Abstract

Utilizing coordinated UAV formations for emergency disaster relief is a key future trend, but traditional evaluation methods have three major drawbacks: high computational complexity, heavy reliance on expert experience, and poor generalization in multi-scenario small-sample settings. To address these issues, this paper first designs a four-level evaluation index system that covers 5 core capabilities and targets 4 typical disaster relief scenarios. Next, it establishes an AHP model that quantifies the performance of 406 UAV formations, thereby providing high-quality labeled data for subsequent research. Furthermore, the paper constructs a ResNet + Atten deep learning network with a hybrid architecture, which improves both the self-learning ability of expert knowledge and the efficiency of multi-scenario evaluation. To solve small-sample overfitting and expert bias, the paper proposes a physically meaningful controllable perturbation data augmentation method: one that works by perturbing 23 UAV performance metrics within a 5–15% range to expand the sample size. Comparative experiments are conducted using three methods, BP neural networks, ResNet, and LSTM, and results show that ResNet + Atten achieves superior performance. Additionally, the data augmentation method effectively enhances the generalization ability of the model. The proposed method provides a reliable method for evaluating the performance of UAV multi-scenario collaborative disaster relief operations.

1. Introduction

In recent years, with the rapid advancement of technologies such as artificial intelligence and networking communications, unmanned aerial vehicles (UAVs) have frequently been deployed in emergency rescue and disaster relief operations in special terrains like high-altitude regions and remote mountainous areas. This is due to their characteristics of rapid response, large-area search capability, vertical delivery, low operational costs, and enhanced operator safety. Particularly in disaster relief scenarios such as forest firefighting, maritime search and rescue, earthquake relief, and monitoring of flash floods and heavy snow, they play an increasingly vital role in personnel search and rescue, material delivery, emergency communications, power line repairs, and disaster monitoring [1,2,3,4,5,6]. Articles [7,8] explore research in areas such as cluster communication and efficient task allocation, focusing on system architecture and task assignment algorithms. Deploying swarm drones for disaster relief allows multiple unmanned aircraft to synergistically leverage their respective strengths to accomplish emergency tasks, significantly enhancing rapid response capabilities and search-and-rescue efficiency. Before the UAV swarm undertakes disaster relief missions, analyzing and evaluating the effectiveness of collaborative disaster relief tasks across different scenarios and UAV formations can provide commanders with insights for optimizing resource allocation, such as the number, model, and payload of UAVs, based on specific conditions. This analysis also offers valuable references for technological innovation and application development in the UAV field.

Current research on UAV effectiveness primarily focuses on the military domain, concentrating on two aspects: the performance evaluation of individual UAVs and the coordinated strike capabilities of UAV swarms [9,10,11]. In contrast, research on evaluating the effectiveness of collaborative unmanned emergency rescue operations in the civilian sector remains limited. Hu Jie, Huang Jichuan, and others developed an evaluation model for UAV swarm combat effectiveness metrics using the Analytic Hierarchy Process (AHP). They validated the effectiveness of this evaluation method through real-world test data from heterogeneous UAV formations [12,13]. Feng Gong et al. achieved a scientific assessment of mission reliability for UAV swarms in complex operational environments by constructing a heterogeneous network model and effective mission loops, and combining dynamic reconfiguration algorithms and Monte Carlo simulation methods [14]. Ruimin Pu et al. proposed a threat assessment framework encompassing seven indicators—including target type, operational status, and friend-or-foe identification—and combined the Analytic Hierarchy Process (AHP) with consistency verification methods to evaluate UAV combat effectiveness and reliability in cooperative air combat [15]. Articles [16,17,18] explore methods to enhance performance through multi-platform coordination, communication coordination mechanisms, and environmental adaptability. Researching the emergency disaster relief capabilities of cooperative UAVs is of great significance for optimizing future UAV emergency response operations. Common efficacy evaluation methods include the ADC efficacy model [19,20] and the AHP hierarchical analysis method [21,22]. Both approaches require extensive expert knowledge and involve complex pairwise comparison matrices. The traditional methods lack self-learning capabilities for processing expert knowledge and exhibit low evaluation efficiency when addressing the complex challenges posed by collaborative and multi-scenario systems in efficacy assessment.

Neural networks and artificial intelligence deep learning have good self-learning, adaptive, and generalization capabilities, with strong nonlinear modeling ability. They can automatically learn features in data and effectively characterize the emergent efficiency and task specificity of complex internal networks within systems [23,24,25]. Therefore, in recent years, scholars have started introducing artificial intelligence methods into the field of efficiency evaluation. Niharika Gupta et al. [26] proposed a Deep Dense Layer Neural Network (DDLNN). By employing a six-layer fully connected architecture with ReLU activation functions and an ADAM optimizer, they achieved high-precision assessment and prediction of diabetes after hyperparameter tuning and K-fold cross-validation. Yang Ping et al. addressed the limitations of traditional tree-based indicator systems in capturing complex equipment interdependencies by constructing a networked indicator system. They pioneered the use of deep learning for performance classification, achieving a 99.3% prediction accuracy with a multilayer neural network trained on 5000 data center samples, thereby offering a novel approach for evaluating complex systems [27]. Yu Xiaolan et al. systematically reviewed the evolutionary application of neural networks in performance evaluation—from fuzzy neural networks and BP neural networks to deep learning—indicating that small-sample deep learning will become a future trend. They identified three major bottlenecks: training data, preprocessing, and parameter optimization [28]. Additionally, other research fields have also incorporated artificial intelligence methods for performance evaluation [29,30,31,32,33].

To address the need for optimizing resource allocation in UAV-coordinated multi-scenario disaster relief operations, this study first designed a multi-scenario-adaptable effectiveness evaluation index system for UAV disaster relief and establishes an Analytic Hierarchy Process (AHP)-based effectiveness assessment model. To enhance the self-learning ability of the evaluation method regarding expert knowledge and improve the efficiency of multi-scenario effectiveness assessment, a ResNet + Attention (ResNet + Atten)-based deep learning evaluation model was further developed. To tackle challenges such as small sample sizes and biases in expert knowledge, this study innovatively adopts a physically driven controllable perturbation-based data augmentation method, which further enhances the generalization ability of the deep learning model and the objectivity of effectiveness prediction.

The ResNet + Atten deep learning network proposed in this study adopt a hybrid architecture that integrates multi-head self-attention with MLP-based residual blocks, which is distinct from traditional convolutional ResNets such as ResNet-50, and ResNet-101. Compared with convolutional ResNet, its key advantages lie in better adaptability to non-grid data, stronger capability to model long-range dependencies, and more flexible feature interactions. To verify the advantages of the proposed methods, comparative analyses are performed against alternative approaches such as BP neural networks, ResNet, LSTM. The results confirm the validity of the proposed methodology in evaluating the performance of UAV-coordinated disaster relief operations and offer valuable insights for improving the generalization capabilities of deep learning networks in various practical performance assessment scenarios.

The key contributions of this work are as follows:

(1)

An indicator system for evaluating the performance of multi-scenario UAV-coordinated disaster relief operations is designed, and an AHP-based performance assessment model is established. By integrating expert knowledge, this model provides fundamental data support for evaluating the effectiveness of UAV-coordinated disaster relief operations across different scenarios.

(2)

A ResNet + Atten deep learning model is developed for evaluating the performance of UAV-coordinated multi-scenario disaster relief operations. This model integrates the feature correlation capture capability of attention mechanisms with the deep training advantages of residual networks, effectively improving the self-learning ability of the effectiveness evaluation method regarding expert knowledge and enhancing the efficiency of multi-scenario effectiveness assessment.

(3)

A physically meaningful controllable perturbation data augmentation method is innovatively proposed. By implementing controllable perturbations on UAV performance metrics, this method further enhances the generalization capability, stability, and prediction objectivity of the ResNet + Atten model. It also addresses problems such as overfitting in small-sample training and biases in expert knowledge, bringing the model more in line with engineering practice.

This paper is structured as follows: Section 2 establishes an evaluation framework for assessing the efficacy of UAVs in typical disaster relief scenarios. Section 3 employs the traditional AHP method to evaluate the collaborative disaster relief efficacy of UAVs across multiple scenarios. Section 4 develops an evaluation model for UAV collaborative multi-scenario disaster relief efficacy based on ResNet + Atten. Section 5 analyses the validity of the ResNet + Atten efficacy evaluation method. Section 6 summarises the entire paper and proposes further research directions.

2. Evaluation Index System for the Performance of UAV-Coordinated Multi-Scenario Disaster Relief Operations

2.1. Typical Application Cases of UAVs in Multi-Scenario Disaster Relief Operations

This paper employs three types of UAVs to establish cooperative disaster relief formations. Type 1 is a vertical takeoff and landing quadcopter [34], equipped with small electro-optical detection payloads and compact rescue payloads to perform small-area search and detection as well as targeted rescue delivery. Type 2 is a gasoline-powered fixed-wing aircraft with horizontal takeoff and landing capabilities [35,36]. It can simultaneously carry large electro-optical detection payloads, emergency networking communication payloads, and heavy rescue payloads. Its primary function is rapid heavy rescue delivery, while also supporting large-area search and detection and networking communication. Type 3 is an electric vertical takeoff and landing fixed-wing UAV [37], which can simultaneously carry electro-optical detection payloads, emergency networking communication payloads, and medium-sized rescue payloads. It primarily focuses on medium-sized targeted rescue delivery while also supporting search and detection, and networking communication.

The performance parameters of the three UAV types are shown in

Table 1. Types and Performance Parameters of Disaster Relief UAVs.

Type	Velocity	Flight Time	Flight Distance	Load Capacity	Detection Range	Resolution
Type 1	100 km/h	4 h	100 km	50 kg	500 m	0.1 m
Type 2	500 km/h	4 h	600 km	500 kg	5 km	0.5 m
Type 3	250 km/h	2 h	300 km	300 kg	2 km	0.2 m

.

In practical applications, deploying larger UAV formations for disaster relief inevitably enhances coordination efficiency but also increases operational costs and subsequent maintenance requirements. Therefore, field operations typically select a specific number of UAVs based on mission-specific conditions. This paper employs three distinct types of UAVs to form cooperative formation, enabling disaster relief missions in specific scenarios. Each formation can be represented by the following mathematical model:

$$\left\{\begin{array}{l}Fun=A_1+A_2+\cdots +A_i+B_1+B_2+\cdots +B_j+C_1+C_2+\cdots +C_k\\ i=1,2,\cdots ,n-2\\ j=1,2,\dots ,n-2\\ k=1,2,\cdots ,n-2\\ i+j+k=n\end{array}\right.$$

(1)

In the above Equation (1), Fun denotes the formation scheme, A represents type 1, B type 2, and C represents type 3. $i$, $j$, and $k$ denote the quantities of the three aircraft types, with the total number being $n$. Among the three types, each must have at least one aircraft, forming a heterogeneous drone formation.

Given one fixed number $n$, the number m of specific drone formation configurations can be calculated using the following formula:

$$m={\displaystyle \frac{(n+1)(n+2)}{2}}-3n$$

(2)

2.2. Evaluation Index System Construction

The effectiveness evaluation metrics for UAV-assisted emergency disaster relief are closely tied to specific application scenarios. Four typical scenarios include earthquake disaster monitoring and rescue, maritime vessel search and rescue, emergency communications in high-altitude disaster zones, and logistics delivery in remote mountainous areas. Among these, the earthquake disaster monitoring and rescue scenario demands the most comprehensive capabilities for UAV collaboration. Therefore, this paper focuses on establishing an effectiveness evaluation metric system for UAV-assisted disaster relief specifically tailored to this scenario.

The application of drones in earthquake disaster monitoring and rescue operations can be divided into four main phases [6,38]:

• Disaster Monitoring and Search Phase: Drones conduct detection and perception of affected personnel, transportation infrastructure damage, power line disruptions, and aftershock activity.
• Command and Decision-Making Phase: Through air–ground sharing of the situational awareness gathered during the first phase, commanders gain comprehensive situational awareness to direct operations and make decisions.
• Cooperative Formation Phase: This involves coordinated path planning, controllable formation operations, and dynamic joining/reorganization of drone formations.
• Rescue Implementation Phase: This includes coordinated drone operations for targeted supply delivery, power line restoration, and evacuation of affected personnel.

Additionally, maintaining uninterrupted communication during earthquake rescue operations is critical. This requires establishing a drone communication network to ensure seamless information flow between the command center, control stations, drones, and disaster zones.

Based on the above analysis, the core capabilities constituting the effectiveness of collaborative UAV disaster relief operations are collaborative search and detection capabilities, collaborative networking and communication capabilities, collaborative planning and decision-making capabilities, collaborative formation flight capabilities, and collaborative rescue delivery capabilities. By analyzing key metrics for these five capabilities in collaborative UAV disaster relief, this study establishes a comprehensive evaluation metric system for UAV collaborative disaster relief effectiveness. Among these, collaborative search and detection capabilities primarily encompass indicators such as sensor detection range, resolution, positioning accuracy, target detection probability, and all-weather capability for electro-optical and radar sensors [39]. Collaborative networking and communication capabilities primarily refer to the communication transmission capacity between UAVs and command centers, as well as rescue subjects. These includes metrics such as effective system capacity, node connectivity, emergency networking coverage, and emergency air–ground communication quality. Collaborative planning and decision-making capabilities primarily reflect the ability of UAVs to assist ground command personnel in decision-making. Given the need for rapid decision-making at disaster sites, collaborative mission planning, collaborative control relationships, and dynamic real-time decision-making capabilities are established as indicators for human–machine collaborative planning capabilities. Cooperative formation capability is evaluated based on a drone’s excellent maneuverability, formation performance, and high survivability during mission execution. Therefore, weather adaptability, decentralization level, formation reconfiguration capability, aircraft availability, and payload diversity serve as metrics for this capability. Cooperative rescue delivery capability forms the foundation for rapid rescue operations by swiftly deploying diverse mission payloads. Core evaluation metrics include flight speed, service ceiling, range, endurance, payload capacity, and payload delivery accuracy.

Cooperative formation capability is evaluated based on a drone’s excellent maneuverability, formation performance, and high survivability during mission execution. Therefore, weather adaptability, decentralization level, formation reconfiguration capability, aircraft availability, and payload diversity serve as metrics for this capability. Cooperative rescue delivery capability forms the foundation for rapid rescue operations by swiftly deploying diverse mission payloads. Core evaluation metrics include flight speed, service ceiling, range, endurance, payload capacity, and payload delivery accuracy.

This paper constructs an evaluation index system for the effectiveness of UAV-associated disaster relief operations based on the hierarchical analysis method for quantitative analysis of non-quantitative events. The system is structured into four distinct levels, the scenario layer, indicator layer, capability layer, and solution performance layer, as illustrated in Figure 1.

3. Evaluation of UAV Collaborative Multi-Scenario Disaster Relief Effectiveness Based on AHP

3.1. Performance Evaluation Model Based on AHP

Based on the evaluation index system for the performance of UAV-coordinated multi-scenario disaster relief outlined in Section 2.2, the mathematical model for the indicators of UAV formation-based coordinated disaster relief using the AHP [40] is expressed as follows:

$$F=(k_1{F}_{SS}+k_2{F}_{TX}+k_3{F}_{JC}+k_4{F}_{BD}+k_5{F}_{JY})$$

(3)

In the Formula (3), $F$ represents the overall effectiveness indicator for UAV collaborative disaster relief operations, $F_{SS}$ represents collaborative search and detection capability, $F_{TX}$ represents collaborative networking and communication capability, $F_{JC}$ represents collaborative planning and decision-making capability, $F_{BD}$ represents collaborative formation flight capability, $F_{JY}$ represents collaborative rescue delivery capability, and $k_i(i=1,2,\cdots ,5)$ represents the weighting coefficients for each indicator.

Based on the performance evaluation indicator system and the characteristics of various UAV performance metrics, the collaborative search and detection assessment model for multiple UAVs is established as follows:

$$F_{SS}=\max (F_{DW}^j)+{\lambda }_2{\displaystyle \sum _{j=1}^n{F}_{FX}^j}+{\lambda }_3{\displaystyle \sum _{j=1}^n{F}_{FW}^j}+{\lambda }_4{\displaystyle \sum _{j=1}^n{F}_{TCNL}^j}+\max (F_{FBL}^j)$$

(4)

In the Formula (4), $F_{DW}$ represents target positioning accuracy, $F_{FX}$ represents target detection probability, $F_{FW}$ represents detection range, $F_{TCNL}$ represents all-weather detection capability, $F_{FBL}$ represents detection resolution, ${\lambda }_2$, ${\lambda }_3$, and ${\lambda }_4$ represent the respective weighting factors for each performance metric, and $n$ represents the number of UAVs.

The evaluation model for cooperative networking communication capabilities among multi-UAVs is as follows:

$$F_{TX}={\lambda }_6{\displaystyle \sum _{j=1}^n{F}_{RL}^j}+{\lambda }_7{\displaystyle \sum _{j=1}^n{F}_{LT}^j}+{\lambda }_8{\displaystyle \sum _{j=1}^n{F}_{ZW}^j}+\max (F_{KD}^j)$$

(5)

In the Formula (5), $F_{RL}$ represents the effective system capacity, $F_{LT}$ represents node connectivity, $F_{ZW}$ represents emergency network coverage, $F_{KD}$ represents air–ground communication quality, ${\lambda }_6, {\lambda }_7, {\lambda }_8$ represent the weighting factors for each performance indicator, and $n$ represents the number of drones.

The evaluation model for collaborative planning and decision-making capabilities among multi-UAVs is defined as follows:

$$F_{JC}={\lambda }_{10}{\displaystyle \sum _{j=1}^n{F}_{RW}^j}+{\lambda }_{11}{\displaystyle \sum _{j=1}^n{F}_{KZ}^j}+{\lambda }_{12}{\displaystyle \sum _{j=1}^n{F}_{DT}^j}$$

(6)

In the Formula (6), $F_{RW}$ represents collaborative mission planning, $F_{KZ}$ represents collaborative control relationships, $F_{KZ}$ represents dynamic real-time decision-making capability, ${\lambda }_{10}, {\lambda }_{11}$ and ${\lambda }_{12}$ represent the respective weighting factors for each capability metric, and $n$ denotes the number of unmanned aerial vehicles.

The performance evaluation model for the coordinated formation flight capability of multi-UAVs is defined as follows:

$$F_{BD}=\max (F_{TH}^j)+{\lambda }_{14}{\displaystyle \sum _{j=1}^n{F}_{ZH}^j}+{\lambda }_{15}{\displaystyle \sum _{j=1}^n{F}_{KY}^j}+{\lambda }_{16}{\displaystyle \sum _{j=1}^n{F}_{QZX}^j}+{\lambda }_{17}{\displaystyle \sum _{j=1}^n{F}_{CG}^j}$$

(7)

In the Formula (7), $F_{TH}$ represents aircraft weather adaptability, $F_{ZH}$ denotes payload diversity, $F_{KY}$ signifies UAV availability, $F_{QZX}$ indicates decentralization level, $F_{CG}$ reflects formation reconfiguration capability, ${\lambda }_{14}, {\lambda }_{15}, {\lambda }_{16}$ and ${\lambda }_{17}$ represent the weighting factor for each performance metric, and M denotes the number of UAVs.

The evaluation model for the collaborative rescue delivery capability metrics of multi-UAVs is defined as follows:

$$\begin{array}{l}{F}_{JY}=\max ({\displaystyle \sum _{j=1}^n{F}_{SD}^j})+\max ({\displaystyle \sum _{j=1}^n{F}_{GD}^j})+{\lambda }_{20}{\displaystyle \sum _{j=1}^n{F}_{HC}^j}+\\ {\lambda }_{21}{\displaystyle \sum _{j=1}^n{F}_{XH}^j}+{\lambda }_{22}{\displaystyle \sum _{j=1}^n{F}_{TSNL}^j}+{\lambda }_{23}{\displaystyle \sum _{j=1}^n{F}_{TSJD}^j}\end{array}$$

(8)

In the Formula (8), $F_{SD}$ represents the maximum flight speed of the UAV, $F_{GD}$ represents the flight altitude range, $F_{HC}$ represents the flight distance, $F_{XH}$ represents the endurance time, $F_{TSNL}$ represents the load delivery capability, $F_{TSJD}$ represents the load delivery accuracy, ${\lambda }_{20}, {\lambda }_{21}, {\lambda }_{22}$ and ${\lambda }_{23}$ represent the respective weighting factors for each indicator, and $n$ represents the number of UAVs.

3.2. Performance Evaluation Based on the AHP

This paper focuses on four typical scenarios: earthquake disaster area monitoring and rescue (Scenario 1), maritime vessel search and rescue (Scenario 2), emergency communications in plateau disaster areas (Scenario 3), and remote mountainous area material delivery (Scenario 4). According to the 1–9 scale in AHP and expert experience, a 5th-order pairwise judgment matrix P is constructed based on five factors influencing the effectiveness indicator. This results in the maximum eigenvalue of matrix P and its corresponding normalized eigenvector. Consequently, the specific weight $k_i(i=1,2,\cdots ,5)$ of each collaborative UAV capability element within the effectiveness indicator is obtained.

$$P=\left[\begin{array}{ccccc}{X}_{11}& X_{12}& X_{13}& X_{14}& X_{15}\\ X_{21}& X_{22}& X_{23}& X_{24}& X_{25}\\ X_{31}& X_{32}& X_{33}& X_{34}& X_{35}\\ X_{41}& X_{42}& X_{43}& X_{44}& X_{45}\\ X_{51}& X_{52}& X_{53}& X_{54}& X_{55}\end{array}\right]$$

(9)

In a pairwise judgment matrix $P$, all diagonal elements are 1, and the product of symmetric elements within the matrix equals 1, that is:

$$\left\{\begin{array}{l}{X}_{ii}=1\\ X_{ij}\times X_{ji}=1\\ i=1,2\cdots ,5\\ j=1,2\cdots ,5\end{array}\right.$$

(10)

The following pairwise judgment matrices $P1, P2, P3, P4, P5$ and $P01, P2, \dots , P23$, etc., exhibit the pattern described in Equation (10). When calculating weights, the eigenvalue method is employed to perform pairwise consistency tests on the judgment matrix. The maximum eigenvalue ${\lambda }_{\max }$ obtained from the judgment matrix is substituted into the following equation:

$$\left\{\begin{array}{l}CI={\displaystyle \frac{{\lambda }_{\max }-m}{m-1}}\\ CR={\displaystyle \frac{CI}{RI}}\end{array}\right.$$

(11)

In the above Equation (11), m denotes the order of the matrix. According to the definition of the consistency ratio, when $CR$ < 0.1, the consistency is regarded as meeting the requirements, and a smaller $CR$ is preferable.

This study employs a scoring system where three UAV experts and two disaster response experts assign weighting coefficients to various capability and performance metrics. Each expert’s score carries equal weight in the overall total.

This paper constructs a pairwise judgment matrix for UAVs based on the four typical scenarios described above: A 5 × 5 matrix P1 for collaborative search and detection capabilities, a 4 × 4 matrix P2 for collaborative networking and communication capabilities, a 3 × 3 matrix P3 for collaborative planning and decision-making capabilities, a 5 × 5 matrix P4 for collaborative formation flight capabilities, and a 6 × 6 matrix P5 for collaborative rescue and delivery capabilities. Subsequently, the maximum eigenvalues of matrices P1 to P5 and their corresponding normalized eigenvectors can be calculated. This yields the weight ${\lambda }_i(i=1,2,\cdots ,23)$ assigned to each specific performance element within the capability metrics.

$$\left\{\begin{array}{l}P1=\left[\begin{array}{ccccc}{A}_{11}& A_{12}& A_{13}& A_{14}& A_{15}\\ A_{21}& A_{22}& A_{23}& A_{24}& A_{25}\\ A_{31}& A_{32}& A_{33}& A_{34}& A_{35}\\ A_{41}& A_{42}& A_{43}& A_{44}& A_{45}\\ A_{51}& A_{52}& A_{53}& A_{54}& A_{55}\end{array}\right]\\ ⋮\\ P5=\left[\begin{array}{cccccc}{E}_{11}& E_{12}& E_{13}& E_{14}& E_{15}& E_{16}\\ E_{21}& E_{22}& E_{23}& E_{24}& E_{25}& E_{26}\\ E_{31}& E_{32}& E_{33}& E_{34}& E_{35}& E_{36}\\ E_{41}& E_{42}& E_{43}& E_{44}& E_{45}& E_{46}\\ E_{51}& E_{52}& E_{53}& E_{54}& E_{55}& E_{56}\\ E_{61}& E_{62}& E_{63}& E_{64}& E_{65}& E_{66}\end{array}\right]\end{array}\right.$$

(12)

Taking Scenario 1 as an example,

Table 2. Weighting Coefficients for UAV Collaboration Capabilities and Performance in Scenario 1.

Collaborative Capability	Capability Coefficient	Performance Metrics	Performance Coefficient
Search and Perception	0.3448	Target Positioning Accuracy	0.1111
		Target Detection Probability	0.1111
		Detection Range	0.1111
		All-weather detection	0.3333
		Detection resolution	0.3333
Networking and Communication	0.1724	Available system capacity	0.1667
		Node connectivity	0.1667
		Emergency Network Coverage	0.3333
		Air–Ground Communication Quality	0.3333
Planning and Decision-Making	0.069	Collaborative Task Planning	0.3333
		Cooperative control relationship	0.3333
		Dynamic real-time decision-making	0.3333
Formation Flight	0.069	Weather Adaptability	0.3448
		Load Diversity	0.3448
		Availability	0.1724
		Degree of decentralization	0.0690
		Formation Reconfiguration	0.0690
Cooperative Rescue Delivery	0.3448	Maximum flight speed	0.0455
		Flight Altitude Range	0.0455
		Flight distance	0.2273
		Battery life	0.2273
		Payload Delivery Capability	0.2273
		Payload Delivery Accuracy	0.2273

presents the weighting coefficients for the capabilities and performance metrics of drone-assisted disaster relief in this scenario.

The above analysis focuses on Scenario 1. For the other three typical scenarios, the drone collaboration capability coefficients and performance coefficients have to be redefined based on expert knowledge. For instance, in Scenario 2, within the pairwise judgment matrix P for the five factors under the collaborative capability layer, elements related to collaborative perception and search capabilities exhibit larger values, while the other four capabilities show smaller relative values. At the same time, in the pairwise judgment matrices P4 and P5 for performance metric factors, elements such as aircraft weather adaptability, aircraft range, and endurance time also exhibit larger values. Therefore, for multi-scenario performance evaluation, the Analytic Hierarchy Process increases the complexity of performance assessment.

Establish three-order matrices P01, P02, and P03–P23 for the three UAV types as shown in Formula (13). The maximum eigenvalues of P01, P02, and P03–P23 and their corresponding normalized eigenvectors can be obtained, so that the weight coefficients of each factor for the three UAV types in the index can be derived, as shown in

Table 3. Performance Index Coefficients for Three UAV Types.

Performance Metrics	UAV Type
	Type 1	Type 2	Type 3
Target Positioning Accuracy	0.5455	0.1818	0.2727
Target Detection Probability	0.5000	0.2500	0.2500
Detection Range	0.1250	0.6250	0.2500
All-weather detection	0.1429	0.7143	0.1429
Detection resolution	0.5455	0.1818	0.2727
Available system capacity	0.1667	0.5000	0.3333
Node connectivity	0.3333	0.3333	0.3333
Emergency Network Coverage	0.1250	0.6250	0.2500
Air–Ground Communication Quality	0.5455	0.1818	0.2727
Collaborative Task Planning	0.3333	0.3333	0.3333
Cooperative control relationship	0.4286	0.1429	0.4286
Dynamic real-time decision-making	0.4000	0.2000	0.4000
Weather Adaptability	0.1429	0.7143	0.1429
Load Diversity	0.1111	0.5556	0.3333
Availability	0.3333	0.3333	0.3333
Degree of decentralization	0.3333	0.3333	0.3333
Formation Reconfiguration	0.5455	0.1818	0.2727
Maximum flight speed	0.1111	0.5556	0.3333
Flight Altitude Range	0.1000	0.7000	0.2000
Flight distance	0.1250	0.6250	0.2500
Battery life	0.4000	0.4000	0.2000
Payload Delivery Capability	0.1250	0.6250	0.2500
Payload Delivery Accuracy	0.6522	0.1304	0.2174

.

$$\left\{\begin{array}{l}P01=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right]\\ P02=\left[\begin{array}{ccc}{b}_{11}& b_{12}& b_{13}\\ b_{21}& b_{22}& b_{23}\\ b_{31}& b_{32}& b_{33}\end{array}\right]\\ ⋮\\ P23=\left[\begin{array}{ccc}{x}_{11}& x_{12}& x_{13}\\ x_{21}& x_{22}& x_{23}\\ x_{31}& x_{32}& x_{33}\end{array}\right]\end{array}\right.$$

(13)

Based on the data in Table 2 and Table 3, combined with the mathematical model presented in this section, the effectiveness of different UAV formation schemes in Scenario 1 can be calculated. For the other three typical scenarios—maritime vessel search and rescue (Scenario 2), emergency communications in high-altitude disaster zones (Scenario 3), and logistics delivery in remote mountainous areas (Scenario 4)—the values of elements P and P1–P5 must be dynamically adjusted according to the importance of each capability or performance metric within the specific disaster relief context. For instance, in Scenario 2, increase the weight of the collaborative perception search capability coefficient and endurance time; in Scenario 3, increase the weight of the collaborative networking communication search capability coefficient, weather adaptability, and endurance time; in Scenario 4, increase the weight of the collaborative rescue delivery capability coefficient, aircraft range, payload delivery capability, and payload delivery accuracy. Based on this, recalculate the capability coefficients and performance coefficient values in Table 3 to compute the effectiveness values for UAV collaborative disaster relief under different scenarios using the model described in this section. Figure 2 and Figure 3 present the AHP-based results for 406 UAV formation schemes across the four scenarios.

In summary, employing the AHP for performance evaluation involves extensive, complex, and cumbersome calculations. Moreover, the assessment outcomes are largely dependent on experts’ experience and knowledge levels, and the evaluation process may exhibit deviations from objective circumstances.

4. Performance Evaluation Model Based on ResNet + Attention

4.1. Performance Evaluation Model

Based on the limitations of the traditional AHP method outlined above, this study aims to establish a more objective and generalizable method for evaluating effectiveness than the AHP approach. The approach is to utilize AI’s powerful nonlinear modeling capabilities and data feature extraction abilities. With limited actual measurement samples, it can integrate the strengths of multiple experts while effectively correcting and synthesizing significant biases existing in expert knowledge. AI deep learning possesses strong self-learning, adaptive, and generalization capabilities, enabling it to automatically learn features from data. It can effectively characterize the emergent efficiency and task specificity of complex internal networks within a system [41,42]. Therefore, to enhance the objectivity and generalization capability of predicting the effectiveness of UAV collaboration in multi-scenario disaster relief, this paper constructs a performance evaluation model based on ResNet + Atten deep learning to assess and predict the effectiveness.

The ResNet + Atten model adopts hybrid architecture combining ‘multi-head self-attention’ with MLP-based residual blocks. Compared to traditional convolutional ResNets (such as ResNet-50 and ResNet-101), this hybrid structure has advantages in adapting to non-grid data, modeling long-range dependencies, and enabling flexible feature interactions. The core of traditional convolutional ResNets lies in their convolutional layers, and their design relies on grid-structured data (such as the two-dimensional pixel grid of images). They extract local features through local sliding windows (convolution kernels), and their parameter-sharing mechanism is well-suited for processing data with spatial local correlations. In contrast, the ‘self-attention + MLP residual block’ structure does not depend on grid topologies and is more friendly to non-grid data (such as tabular data, sequence data, irregularly structured data, etc.). Convolutional ResNet serves as a ‘local perceptron’ optimized for local feature extraction in grid-based data (images), while the ‘self-attention + MLP residual block’ acts as a ‘global correlator’ designed for modeling universal feature associations.

The ResNet + Atten drone-assisted disaster relief performance evaluation model is a regression network integrating self-attention mechanisms and residual MLP blocks [43,44], designed to learn mapping relationships from high-dimensional input features and output continuous values. The overall architecture comprises five core modules: data setup, input and embedding, self-attention, stacked residual MLP blocks, and regression output. The specific modules are shown in Figure 4. Key features include that the attention mechanism captures feature dependencies through self-attention, enhancing the weighting of critical performance evaluation features, and effectively handling correlations among high-dimensional features. Residual connections mitigate deep network training challenges by preserving original feature information, accelerating model convergence, and improving stability. The stacked MLP progressively extracts abstract features through multiple fully connected layers and nonlinear activations, adapting to complex regression tasks. Overall, this model combines the ‘feature correlation capture’ of attention mechanisms with the ‘deep training advantages’ of residual networks, making it suitable for learning nonlinear mapping relationships from high-dimensional inputs and performing regression predictions.

The Dataset module consists of data features and formats, training sets, and testing sets. Both training and testing sets include raw data and performance-perturbation-enhanced data. Features primarily encompass four UAV cooperative disaster relief scenarios, 30 UAV formation schemes, and their corresponding efficiency values. Raw data mainly consists of samples formed by UAV formation schemes and efficiency values obtained via the AHP method. Performance-perturbation-enhanced data is generated by augmenting the raw data; the data augmentation method is detailed in Section 4.2.

The Input & Embedding module consists of a feature input layer and a fully connected layer. It receives raw input features and maps them to a high-dimensional embedding space, providing a foundation for subsequent attention mechanisms and feature extraction. The former receives raw 34-dimensional (inputSize) input features (30-dimensional individual drone formation model features + 4-dimensional mission scenario features), elevating the dimension to embedDim through a fully connected layer to enhance the feature representation capability of the UAV formation. In the drone formation performance regression task, its core function is feature homogenization. Through linear transformation, it converts discrete formation features and sparse scene features into high-dimensional vectors with the same distribution, resolving the failure to capture correlations caused by the direct input of heterogeneous features.

The self-attention module consists of a multi-head attention layer and a fully connected layer. It captures dependencies within input features (such as the importance of correlation between features of different dimensions) and enhances the weights of key features. Multi-head attention (with numHeads heads) decomposes embedDim-dimensional features into numHeads embedDim/numHeads-dimensional subspaces for parallel attention computation. The effectiveness of drone formation depends on dynamic interactions between formations and mission features. This module actively focuses on critical combinations by calculating feature correlation weights, simultaneously capturing multi-type dependencies such as ‘formation-formation,’ ‘formation-mission,’ ‘mission-task’, thereby avoiding single-head association blind spots. After the attention layer outputs, a fully connected layer maintains consistent dimensions to prevent feature transmission disruption. Finally, the final output is concatenated into embedDim dimensions to enhance the expressive ability of attention.

Residual MLP Blocks consist of numBlocks residual blocks. A single residual block comprises a fully connected layer 1, a ReLU activation function, a fully connected layer 2, a residual connection (addition layer), and a ReLU activation function. By extracting more complex features through multi-layer nonlinear transformations, while utilizing residual connections to mitigate the vanishing gradient problem in deep networks, training stability is enhanced. Each residual block employs a fully connected layer 1 (fc1) to reduce the embedDim-dimensional features to hiddenSize dimensions, minimizing computational load while introducing nonlinearity. The ReLU activation function boosts nonlinear expressive ability and filters out ineffective features. Fully connected layer 2 (fc2) expands the hiddenSize-dimensional features back to embedDim dimensions, matching the input dimension. The residual connection adds the output of fc2 to the original input (residual), preserving original feature information and preventing feature degradation in deep networks. The final ReLU activation function further enhances nonlinearity. The model stacks numBlocks residual blocks, progressively enhancing the drone collaboration feature abstraction capability.

The Regression Output module consists of a Fully Connected Layer and a Regression Layer. The former compresses the embedDim-dimensional features from the preceding module into a 1-dimensional vector, while the latter defines the regression optimization objective using Mean Squared Error (MSE) as the loss function. As the final unit in the ‘feature-to-performance’ mapping, its core function is dimensionality reduction. Through fully connected layer learning, it transforms high-dimensional features into a 1-dimensional continuous output directly corresponding to the drone formation’s performance value, completing the transition from feature space to target space. Meanwhile, it performs loss calculation and gradient backpropagation. Based on MSE, it quantifies the deviation between predictions and actual performance, optimizing network parameters through backpropagation to guide the model to reduce regression error. The 1D fully connected layer avoids the discretization bias inherent in Softmax classification tasks and adapts to continuous performance value outputs. The MSE loss aligns with the continuous nature of performance values. Its penalty mechanism for samples with large errors reduces prediction errors in drone formation schemes, supporting the recommendation of optimal drone formations in practice.

4.2. Data Augmentation Methods

When using deep learning for performance evaluation with small sample sizes, models are easy to overfit, leading to inadequate generalization capabilities [45]. Meanwhile, experts’ knowledge of scoring performance metrics is biased, and performance indicators fluctuate due to random factors during drone formation tasks. To address these challenges, this paper proposes a physically driven, controllable perturbation data augmentation method [46,47]. By applying controllable random perturbations to UAV performance metrics (the core of data augmentation), it resolves issues of small sample sizes, poor generalization, and potential detachment from reality in multi-scenario collaborative disaster relief performance evaluation. This ensures the trained model adapts better to the demands of real-world UAV formation tasks without compromising data logic. The core principle of this data augmentation approach differs from traditional rotation/cropping techniques used in deep learning image processing tasks. Instead, it generates more similar yet non-repetitive training samples by simulating the random fluctuations of UAV performance metrics observed in real scenarios. The process can be broken down into three steps:

(1) Identify 23 drone performance metrics as disturbance variables. First, 23 fundamental metrics for UAVs of A/B/C-class are exacted from the AHP weight matrix w. These metrics are critical to determining formation effectiveness, including: Detection-related: Target positioning accuracy, detection probability, detection range. Communication-related: Effective system capacity, node connectivity, air–ground communication quality. Maneuverability-related: Maximum flight speed, flight altitude range, endurance time. Mission-related: Payload delivery capability, formation reconfiguration capability.

(2) Design disturbance rules that follow a controllable normal distribution. To prevent disturbances from deviating from realistic physical scenarios (e.g., negative performance values or excessive fluctuations), a truncated normal distribution is employed to design the logic. The key parameters and constraints are listed in

Table 4. Key Parameters and Constraints in Data Augmentation.

Disturbance Segment	Parameter Implementation	Physical Meaning
Disturbance Intensity Control	perturb_ratio = 0.05 (perturbation amplitude 5~15%)	Simulate small fluctuations in drone performance under real-world conditions (such as ±15% deviation caused by equipment aging or environmental interference), rather than extreme failures.
Disturbance Distribution	perturb = randn (1) × perturb_ratio (Normal distribution random factor)	fluctuations comply with the real-world pattern of ‘most values clustering around the mean while a minority deviate’ (e.g., among 30 drones, most performance metrics align closely with the standard value, while a few show slight deviations).
Extreme Value Truncation	perturb = max(min(perturb, 2 × perturb_ratio), −2 × perturb_ratio) (Within ±2σ)	Avoid generating unreasonable extreme samples (such as abrupt improvements of 20% or drops below zero), ensuring that the perturbed data remains consistent with physical common sense.
Non-negativity protection	perturbed_value = max(perturbed_value, 0)	Performance metrics (such as detection range and battery life) cannot be negative; enforce constraints to ensure data validity.

.

(3) Recalculate performance using perturbation indicators to generate augmented samples. Independently add perturbations to each of the 23 indicators for every UAV class (A/B/C), obtain the perturbed_base_perf. For each formation scheme (e.g., 10 A + 10 B + 10 C) and mission scenario (e.g., search and rescue), recalculate the performance value Y using the perturbed performance indicators. The final X_fused (34-dimensional features) and Y (performance values) constitute the augmented dataset.

Generate random numbers from a standard normal distribution. Use MATLAB 2025b to generate random numbers from a normal distribution with a mean of 0 and a standard deviation of 1.

$$e_{i,t}~N(0,1)$$

(14)

Multiply the random number by the perturbation strength $\alpha$ to obtain the initial perturbation factor:

$${\dot{e}}_{i,t}=e_{i,t}\cdot \alpha$$

(15)

Scope Restriction:

$${\widehat{e}}_{i,t}=\left\{\begin{array}{c}-2\alpha ,{\dot{e}}_{i,t}<-2\alpha \\ {\dot{e}}_{i,t},-2\alpha \le {\dot{e}}_{i,t}\le 2\alpha \\ 2\alpha ,{\dot{e}}_{i,t}>2\alpha \end{array}\right.$$

(16)

Apply perturbations to the original values while ensuring non-negativity (performance metrics cannot be negative):

$${\widehat{x}}_{i,t}=\max \left(x_{i,t}\cdot (1+{\widehat{e}}_{i,t}),0\right)$$

(17)

In Equation (17), $x_{i,t}$ denotes the raw value of the $i$-th performance metric ($i$ = 1, 2, …, 23) for the $t$-th drone type ($t$ = 1, 2, 3, corresponding to A/B/C). By executing the above steps sequentially for all 23 metrics and 3 drone types, the final perturbed performance matrix is obtained.

Considering compatibility with drone-assisted disaster relief operations, this paper selects performance perturbation over other augmentation methods. This approach avoids common data augmentation techniques such as randomly scrambling formation characteristics or adding Gaussian noise to input features. Its underlying principle is that augmentation methods must align with mission requirements. Performance metrics (23-dimensional) are continuous physical parameters. Minor perturbations neither alter the formation’s core structure (type ratios) nor simulate real-world fluctuations, which makes them the most suitable data augmentation method for formation effectiveness prediction. Given the nature of performance prediction, this data augmentation method directly addresses the following critical issues in model training:

(1)

Address the issue of insufficient sample size and improve model generalization capability. The original total sample size of 1624 is a small training dataset, which can easily lead to model overfitting. By adding independent perturbations to 23 performance metrics, each original sample can generate multiple augmented samples with slightly varying values. Theoretically, varying the perturbation seed can generate an infinite number of similar samples, which is equivalent to indirectly expanding the sample size. This allows the model to learn the patterns of performance under fluctuations rather than memorizing the original sample verbatim, thereby improving its generalization ability to real-world scenarios.

(2)

Simulate real-world performance fluctuations, making the model more aligned with engineering realities. In actual drone formation missions, performance metrics cannot be entirely fixed. Sensor accuracy varies slightly across different drone batches (e.g., positioning accuracy ±0.5 m). Factors like wind speed and electromagnetic interference cause fluctuations in communication quality and detection range (e.g., detection range reduced by 3%). After prolonged use, drone endurance may decrease by 5–10%. These performance perturbations precisely simulate the above real-world fluctuations. The trained model does not suppose all drones have identical performance but adapts to minor deviations. This prevents prediction errors during actual deployment caused by mismatches between real-world performance and training assumptions.

(3)

Suppress overfitting and enhance model stability. During training with small samples, models tend to overlearn noise in the original data. For instance, if the effectiveness value of a certain formation plan is high because of computational errors, the model may mistakenly conclude that this formation is always more effective. The perturbed samples effectively introduce reasonable noise to the original data, disrupting deterministic correlations. For instance, the effectiveness of a 10A + 10B + 10C formation is no longer a fixed value but fluctuates around a mean. The model must learn the mapping between metric fluctuations and performance variations, thereby reducing reliance on noise in the original samples and lowering overfitting risk.

5. Performance Evaluation and Comparative Analysis

5.1. Performance Evaluation Analysis

The ResNet + Atten deep learning network requires a large volume of sample data for model training. To address the shortage of large-scale practical performance data samples for UAV formation disaster relief operations, this paper employs the traditional AHP hierarchical analysis method to obtain the training dataset required for the ResNet + Atten deep learning network. Scenarios, drone formation schemes, performance data derived from AHP analysis, and expanded sample data generated through drone performance perturbation data augmentation are utilized as training and testing datasets. This enables training of the neural network structure parameters for AI deep learning algorithms, enhancing the objectivity of drone disaster relief performance prediction. Meanwhile, comparative analysis using limited real-world disaster relief data from small-scale drone formations verifies the validity of the model and methodology.

Before training the ResNet + Atten model, various model parameters must be configured. Selecting appropriate parameters significantly impacts the model’s predictive performance. A grid search strategy was employed to determine suitable parameters. The primary parameter settings for the ResNet + Atten model are shown in

Table 5. Key Parameter Settings for the ResNet + Atten Model.

Module	Parameter	Value
Input & Embedding	inputSize	n + 4
	embedDim	256
Self-Attention	numHeads	8
	Query, Key, Value shape	[batch_size, 256]
	Multi-Head Split	[batch_size, 8, 32]
	Query × Key^T shape	[batch_size, 32, 32])
	Softmax shape	[batch_size, 32, 32]
	Concatenation	[batch_size, 256]
Residual MLP Blocks	hiddenSize	64
	numBlocks	2
Regression Output	OutputSize	1

.

A UAV emergency disaster relief formation was established using three mature aircraft types, comprising 30 units: vertical takeoff and landing quadcopters, gasoline-powered horizontal takeoff and landing fixed-wing aircraft, and electric vertical takeoff and landing fixed-wing aircraft. The impact of different aircraft quantities on overall effectiveness is the primary focus of this performance study. During evaluation, 5%, 10%, and 15% perturbed data were generated for each of the four scenarios, resulting in 4 × 406 data points per perturbation condition. All three sets of enhanced perturbation data were used to train the ResNet + Atten deep learning performance evaluation model. After training, Figure 5, Figure 6, Figure 7 and Figure 8 present the predicted performance results for the 30-drone fleet across 406 formation configurations under the four scenarios.

Figure 5, Figure 6, Figure 7 and Figure 8 show the x, y, and z coordinates representing the quantities of UAV models 1, 2, and 3, respectively. Colors ranging from blue to deep red indicate the magnitude of effectiveness values. Figure 5, Figure 6, Figure 7 and Figure 8 reveal the following patterns regarding how the number of different drone models within a fleet affects effectiveness: (1) For earthquake disaster monitoring and rescue scenarios (Scenario 1), Model 3 has the greatest impact on collaborative effectiveness, followed by Model 1, while Model 2 contributes relatively little. (2) For maritime vessel search and rescue scenarios (Scenario 2), Models 3 and 1 have nearly identical effects on collaborative effectiveness, while Model 2 contributes less. (3) For the emergency communication scenario in high-altitude disaster areas (Scenario 3), Model 1 has the greatest impact on collaborative effectiveness, followed by Model 3, while Model 2 contributes the least. (4) For the remote mountainous area logistics delivery scenario (Scenario 4), Models 2 and 3 have essentially the same impact on collaborative effectiveness, while Model 1 contributes the least.

To obtain the cumulative importance of attention-based learning features and enhance model interpretability, we analyzed one quantitative feature and six performance features for each model type. Across the three models, this yielded a total of 21 features, supplemented by four task-specific features, resulting in a combined 25-dimensional feature set. Based on this, we normalized the cumulative importance of the model’s attention features, producing Figure 9. And two conclusions can be drawn from Figure 9:

(1)

Feature importance aligns with aircraft model priority: Among the top four features by cumulative importance, three correspond to Type 3 and Type 1 aircraft. This fully matches the conclusion that “Type 3 has the greatest influence in Scenario 1, followed by Type 1,” demonstrating that the learning model has captured formation configuration logic consistent with physical principles.

(2)

Correspondence with AHP weights: Type 3 and Type 1 represent aircraft types with high importance characteristics. Their performance metrics (e.g., payload delivery capability for Model 3 and detection resolution for Type 1) exhibit weight coefficients ≥ 0.3333 in the AHP method (Table 2). The model’s attention-based cumulative importance provides cross-validation for the weight analysis of traditional methods.

5.2. Efficacy Comparison Analysis

To validate the effectiveness of the ResNet + Atten performance evaluation model for predicting the efficacy of UAV-coordinated disaster relief operations, this paper introduces Four models—BP neural network, ResNet, LSTM network, and ResNet + Atten—were employed to predict the disaster relief effectiveness of drones under different formation configurations. three methods—BP neural network, ResNet, and LSTM—for comparative analysis. BP neural networks and LSTMs serve as baseline models rather than structurally optimal models. Each model was run 10 times for performance regression prediction, calculating the Mean Absolute Error (MAE), Mean Squared Error (MSE), and regression prediction coefficient of determination (R²) metrics. The formulas for MSE, MAE, and R² are as follows:

$$MSE={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}$$

(18)

$$MAE={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n|}{y}_i-{\widehat{y}}_i|$$

(19)

$$R^2={\displaystyle \frac{{\displaystyle \sum _{i=1}^n{({\widehat{y}}_i-\overline{y})}^2}}{{\displaystyle \sum _{i=1}^n{(y_l-\overline{y})}^2}}}$$

(20)

In the above Formulas (18)–(20), $n$ represents the total number of test samples. Smaller values of MSE and MAE indicate better prediction performance, while a larger $R^2$ value signifies superior regression prediction results.

The parameter settings for the three comparison models balance model characteristics and fairness. BP serves as the baseline model with the simplest parameters. LSTM adjusts the number of epochs, batch size, and learning rate to accommodate its temporal nature. ResNet maintains core training parameters consistent with BP, reflecting structural differences solely through the ‘number of residual blocks’ to ensure the reliability of comparative experiments.

The three models were trained on the same dataset and thus share the following fundamental input parameters. Input dimension (inputSize): 34 dimensions (derived from the fused feature vector X_fused, comprising ‘30-dimensional formation features + 4-dimensional mission features’; automatically obtained in the code via inputSize = size (X_fused, 2)). Task type: Regression task (outputs a single-dimensional ‘efficiency value’ for the UAV formation). For clearer comparison,

Table 6. Key Parameter Settings for the Three Comparison Models.

Parameter Dimension	BP	LSTM	ResNet	The Core Reason for the Differences
Hidden Layer/Number of Units	1 layer × 64	1 LSTM layer × 128 + 1 MLP layer × 128	2 residual blocks × 64	LSTM requires temporal modeling and has a large number of units; ResNet requires residual stacking and has a large number of blocks.
Maximum Training Rounds	30	50	30	LSTM has high computational complexity and requires more iterations to converge.
Size of batch	64	32	64	LSTM has many parameters, small batches reduce memory usage
Initial learning rate	1 × 10−3	1 × 10−4	1 × 10−3	LSTMs are prone to oscillation, while small learning rates ensure stable convergence.
Regularization	None	None	None	All are simplified configurations (compared to ResNet + Atten)

summarizes the key parameter differences among the three models.

In the comparative analysis of model effectiveness, K-fold cross-validation is employed. A 10-fold cross-validation is configured, dividing the data into 10 equal parts, with 9 parts used for training and 1 part for validation in each iteration. A fixed random seed (RandomSeed123) was used to ensure consistent fold partitioning across multiple runs, thereby facilitating reproducibility. To prevent data leakage, independent training/validation indices are extracted for each fold, guaranteeing that validation set data never participates in the model training for the current fold. Models are rebuilt for each fold to avoid weight leakage from previous folds. Evaluation metrics are stored for each fold, with the final mean calculated.

Training and testing of the performance evaluation model were conducted across four scenarios: 10 aircraft with 36 formation schemes, 20 aircraft with 171 formation schemes, and 30 aircraft with 406 formation schemes. A total of 4 × 603 = 2452 undisturbed data points were used, employing 10-fold cross-validation. Among these, 90% of the samples were allocated for training, with the remaining 10% reserved for testing. Comparative results of the four evaluation methods are presented in

Table 7. Comparison of Prediction Accuracy Among Four Different Methods.

Method	MAE	MSE	R2
BP	0.3835	0.2548	0.9551
LSTM	0.4790	0.4273	0.9247
ResNet	0.3790	0.2468	0.9565
ResNet + Atten	0.3568	0.2020	0.9674

and Figure 10, Figure 11, Figure 12 and Figure 13.

A comparative analysis of the performance of four methods—BP, ResNet, LSTM, and ResNet + Atten—reveals that the ResNet + Atten deep learning network achieves lower mean MSE and MAE values alongside higher R² values. This indicates that compared to other approaches, the ResNet + Atten method delivers superior prediction outcomes, which further demonstrates the effectiveness of the proposed method in forecasting the disaster relief efficiency of drone collaboration.

5.3. Effectiveness Analysis of Data Augmentation Methods

Considering performance deviations in real-world applications, the drone performance perturbation data augmentation method proposed in this paper simulates performance variations. This approach helps prevent sudden increases in prediction errors during actual deployment caused by mismatches between performance and training assumptions, thereby suppressing overfitting and enhancing model stability. In the performance perturbation testing, unperturbed data, 10% perturbed data, 20% perturbed data, and 30% perturbed data were generated for four scenarios, respectively. Each perturbation scenario comprised 4 × 603 data points. 10-fold cross-validation was employed, with 90% of the data allocated for training and 10% for testing.

Table 8. Prediction Accuracy of Trained Networks with Different Perturbation Data.

Prediction Accuracy	Disturbance Test Data Ratio
	0%	10%	20%	30%
0% disturbance test data	0.9674	0.9492	0.9220	0.8873
10% disturbance test data	0.9717	0.9628	0.9439	0.9167
20% disturbance test data	0.9682	0.9743	0.9692	0.9545
30% disturbance test data	0.8648	0.9155	0.9493	0.9694

presents the regression prediction accuracy results for different perturbation levels in both training and testing datasets. Table 8 indicates that prediction accuracy slightly decreases with increasing perturbation magnitude. However, when both training and testing datasets undergo the same perturbation level, prediction accuracy remains largely unchanged. The ResNet + Atten performance evaluation model demonstrates guaranteed prediction accuracy under varying data biases, exhibiting no convergence issues or significant deviations. This validates the stability of the performance evaluation model.

Table 9. Prediction Accuracy of Performance Regression Under Drone Loss Conditions.

Disturbance Level	MAE	MSE	R2
No-perturbation	0.7744	0.7820	0.6105
5% perturbation	0.6855	0.6285	0.6947
10% perturbation	0.6096	0.5272	0.7514
15% perturbation	0.5443	0.4474	0.7958

presents the regression prediction accuracy metrics under four scenarios, using a 30-drone formation as a typical case study. These metrics are evaluated under the special condition of losing one drone and varying performance perturbations. All data used across the four scenarios included 10% perturbation, totaling 4 × 406 data points. 10-fold cross-validation was employed, with 90% of the data allocated for training and 10% for testing. Table 9 demonstrates that prediction accuracy progressively improves with increasing performance perturbations. This indicates enhanced model generalization and adaptability to real-world scenarios under disturbance. Consequently, the data augmentation method proposed in this paper effectively boosts model generalization capabilities. It enables the model to learn performance patterns under fluctuating conditions, thereby improving its ability to generalize to real-world scenarios.

5.4. Validity Analysis of the Method Based on Measured Data

Currently, real-world scenarios for collaborative disaster relief with UAVs typically involve the coordination of 3 to 5 homogeneous drones. In certain training scenarios aimed at selecting high-performance units, formations of up to 10 UAVs may be deployed. Figure 14 illustrates the flight trajectories in a typical Scenario 2 configuration, depicting a maritime rescue training exercise with 10 coordinated UAVs. The flight paths are satellite-navigated, with the red boundary indicating the maritime search area. Type 1 (green dashed trajectory) performs close-range, localized search; Type 3 (orange dashed trajectory) conducts medium-range search; and Type 2 (blue dashed trajectory) executes long-range, extensive search operations. By leveraging their respective strengths, the three UAV types collaboratively and efficiently complete the search mission.

To further validate the effectiveness of the efficiency evaluation method adopted in this paper, a comparative analysis was conducted based on limited available measured data.

Table 10. Actual Performance Values of 10-Drone Formation in Disaster Relief Operations vs. AHP Efficiency Values.

Quantity of Type 1	Quantity of Type 2	Quantity of Type 3	Scenario 2	Scenario 4	Measured Value	AHP Value
1	1	8	1	0	3.38	3.2662
1	2	7	0	1	4.80	4.5651
1	3	6	1	0	3.37	3.1384
1	4	5	0	1	4.46	4.3047
1	5	4	1	0	3.07	3.0105
1	6	3	0	1	4.12	4.0443
1	7	2	1	0	2.96	2.8827
1	8	1	0	1	3.88	3.7839
2	1	7	1	0	3.28	3.2392
2	2	6	0	1	4.65	4.5737
2	3	5	1	0	3.17	3.1114
2	4	4	0	1	4.41	4.3133
2	5	3	1	0	3.06	2.9836
2	6	2	0	1	4.17	4.0529
2	7	1	1	0	2.96	2.8558
3	1	6	0	1	3.93	3.7925
3	2	5	1	0	3.22	3.1484
3	3	4	0	1	4.58	4.4522
3	4	3	1	0	3.12	3.0205
3	5	2	0	1	4.35	4.1918
3	6	1	1	0	3.01	2.8927
4	1	5	0	1	4.11	3.9313
4	2	4	1	0	3.22	3.1214
4	3	3	0	1	4.64	4.4608
4	4	2	1	0	3.11	2.9936
4	5	1	0	1	4.40	4.2004
5	1	4	1	0	3.27	3.1584
5	2	3	0	1	4.81	4.5997
5	3	2	1	0	3.16	3.0305
5	4	1	0	1	4.57	4.3392
6	1	3	1	0	3.27	3.1314
6	2	2	0	1	4.86	4.6083
6	3	1	1	0	3.16	3.0036
7	1	2	0	1	4.72	4.3479
7	2	1	1	0	3.21	3.0405
8	1	1	0	1	4.99	4.4867

presents the efficiency data obtained from collaborative disaster relief training tests in Scenario 2 and Scenario 4 with different formation configurations of 10 UAVs in total. The formations consist of three UAV types, with a maximum of 8 units per model and a fixed total of 10 UAVs per formation. Table 10 also includes the efficiency data obtained using the traditional classic AHP method. The table shows a positive deviation of 1% to 10% between the measured data and the AHP data, which falls within an acceptable range. This demonstrates the feasibility of the AHP method used in this paper for generating the training set, and also justifies the rationality of the performance perturbation data augmentation approach.

The ResNet + Atten deep learning performance evaluation model was trained using measured data, encompassing 18 formation schemes that yielded 36 sets of measured data across two scenarios. Using a 5-fold cross-validation method, 80% of the data was allocated for training, while 20% was reserved for testing. The results of performance predictions using different methods are presented in

Table 11. Efficiency Regression Prediction with Very Few Sample Measured Data.

Method	MAE	MSE	R2
BP	0.3010	0.1386	0.7168
LSTM	0.3318	0.1567	0.6798
ResNet	0.2789	0.1441	0.7056
ResNet + Atten	0.2894	0.1067	0.7820

. As shown in Table 11, under conditions with very few samples, the accuracy advantage of the performance prediction is not significant.

To further validate the model’s predictive accuracy using measured data, we incorporated efficiency samples calculated from the AHP method into the measured dataset, expanding the dataset size to 72 groups. Of these, using a 5-fold cross-validation method, 80% were used for training and 20% for testing. The prediction results from different methods are shown in

Table 12. Efficiency Regression Prediction Based on Measured Data + AHP Data.

Method	MAE	MSE	R2
BP	0.2083	0.0564	0.8637
LSTM	0.2542	0.0878	0.8049
ResNet	0.1978	0.0612	0.8640
ResNet + Atten	0.0852	0.0113	0.9749

. As shown in Table 12, increasing the size of the data sample improves the accuracy of performance prediction.

The comparative analysis using measured data further demonstrates the generalization capability and adaptability to real-world scenarios of the ResNet + Atten model established in this paper. It also validates the effectiveness of the AHP method for acquiring training data and the feasibility of employing controllable drone performance perturbation for data augmentation. However, it should be emphasized that under conditions of extremely limited sample size, the advantages of this method are not pronounced, which may be inherent to the nature of neural network approaches.

6. Conclusions

This study addresses key challenges in UAV multi-scenario collaborative disaster relief performance evaluation—high computational complexity, heavy reliance on expert experience, and poor generalization in multi-scenario small-sample settings (Non-extremely small sample size)—via three core components, enabling systematic training and accurate prediction of relief effectiveness.

First, an adaptable multi-scenario evaluation index system (four-level: scenario–capability–indicator–scheme) and AHP model are established, covering four scenarios (earthquake rescue, maritime search, plateau communication, mountain logistics) and five core capabilities. With expert-derived weights, the AHP quantifies effectiveness for UAV formations, supplying high-quality labeled data for deep learning.

Second, a ResNet + Atten model (multi-head self-attention + MLP residual blocks) is proposed. Comparisons with BP, ResNet, and LSTM validate its superiority, performing with high prediction accuracy in both AHP data and measured + AHP data conditions.

Third, an innovative physically meaningful perturbation augmentation method targets UAV metrics, applying different perturbations to simulate real-world fluctuations. It resolves overfitting and expert biases, boosting generalization.

Notably, the established evaluation model and ResNet + Atten parameters could be further validated and optimized to better fit practical UAV operations. Future work will adjust core perturbation amplitude of UAV metrics based on actual conditions and verify adaptability in complex compound scenarios. These will help the model better reflect actual UAV collaborative relief performance and support optimal formation deployment.