Task Allocation of Multiple Unmanned Aerial Vehicles Based on Deep Transfer Reinforcement Learning

Abstract

With the development of UAV technology, the task allocation problem of multiple UAVs is remarkable, but most of these existing heuristic methods are easy to fall into the problem of local optimization. In view of this limitation, deep transfer reinforcement learning is applied to the task allocation problem of multiple unmanned aerial vehicles, which provides a new idea about solving this kind of problem. The deep migration reinforcement learning algorithm based on QMIX is designed. The algorithm first compares the target task with the source task in the strategy base to find the task with the highest similarity, and then migrates the network parameters obtained from the source task after training, stored in the strategy base, so as to accelerate the convergence of the QMIX algorithm. Simulation results show that the proposed algorithm is significantly better than the traditional heuristic method of allocation in terms of efficiency and has the same running time.

1. Introduction

With the development of UAV technology, UAV with autonomous ability has been widely used in various fields because of its strong flexibility and high efficiency, especially for obstacle avoidance [1], rescue [2], mapping [3] and other tasks. Due to the limitations of a single UAV, nowadays, UAVs mostly use the cluster method to perform complex tasks, which inevitably involves the task allocation problem. A good task allocation scheme can improve the efficiency and quality of completing tasks, which has attracted the attention of many scholars.

Most of the existing research at home and abroad uses heuristic methods to solve the problem of UAV task assignment. Zhang Ruipeng et al. [4] used the hybrid particle swarm algorithm to solve the problem of UAV task allocation. In order to jump out of the local optimum, they proposed a variable neighborhood search algorithm. By analyzing the hunting process of gray wolf groups, Peng Yalan et al. [5] modeled the group interaction mechanism and cooperative predation behavior of gray wolves, mapped the cooperative predation behavior mechanism of gray wolves to the dynamic task allocation of UAV clusters, and gave the dynamic task allocation process of UAV clusters. Yang Huizhen et al. [6] proposed a new circular competition method based on the dynamic ant colony labor division model. The method specifically solved the conflict problem caused by the dynamic change in the position of the task with time, and well dealt with the dynamic task allocation problem of heterogeneous multi AUV systems. Boyu Qin et al. [7] proposed a distributed group cooperative dynamic task assignment method based on the extended contract network protocol, which targeted and solved the dynamic assignment problem of UAV swarms performing cooperative reconnaissance and attack tasks on multiple targets in complex combat scenarios. Jiang Shiwen [8] used the swarm optimization algorithm based on beetle antennae search (BSO) in the task allocation process. The defect that particle swarm optimization algorithm is easy to “premature” is avoided by introducing the left and right whisker bidirectional optimization mechanism of longicorn search into the particle swarm optimization algorithm.And the global optimal value is then locally optimized while enhancing the global optimization ability. However, the above algorithms that using heuristic methods to solve the task allocation problem are inevitably faced with the problem of falling into the local optimum. At this time, the deep reinforcement learning provides a new idea.

Reinforcement learning (RL) is one of the important research fields of machine learning, which is suitable for solving complex sequential decision problems. It has achieved remarkable results in the fields of robot control, scheduling optimization, and multi-agent collaboration [9]. The basic idea is that the agent adopts a trial-and-error strategy to learn the optimal strategy by interacting with the environment. Deep reinforcement learning has a wide range of applications in UAV formation control [10], navigation [11], and tracking [12].

However, the use of reinforcement learning to solve the task assignment problem is still in its infancy, and there are few related studies. Tang Fengzhu et al. [13] used the deep Q-network (DQN) algorithm based on the value function to solve the problem of low task completion due to task completion time constraints in the scenario of the random delivery of UAV tasks. Pengxing Zhu et al. [14] proposed an improved semi-random Q-learning algorithm. This algorithm increases the probability of obtaining a better task assignment scheme by changing the way Q-learning algorithm selects the next action during random exploration. Cheng Ding et al. [15] proposed a new RL approach to solve the task allocation problem of multi-AUV (autonomous underwater vehicle) in ocean currents, and introduced APAA (automatic policy amendment algorithm) to make AUVs understand how their behavior affects the final result. However, these studies using reinforcement learning methods all face the problem of slow algorithm convergence. Transfer reinforcement learning [16,17], as a paradigm of machine learning, can solve this problem very well.

Transfer learning [18] is a machine learning method whose goal is to transfer knowledge from one domain (source domain) to another domain (target domain) so that the target domain can achieve better learning effects. Usually, the application scenario of transfer learning is that the amount of data in the source domain is sufficient, while the amount of data in the target domain is small. The feature that the transferred task requires less data can be used to solve the problem of slow convergence of reinforcement learning.

In this paper, a deep transfer reinforcement learning algorithm based on QMIX is proposed to solve the multi-UAV task assignment problem involving materials scheduling. The use of the QMIX network cleverly avoids the problem of action space explosion and also avoids the problem that heuristic methods easily fall into local optimal solutions. The introduction of transfer learning solves the problem of long training time for deep reinforcement learning and improves real-time performance. Finally, the effectiveness of the algorithm is verified by simulation experiments.

2. Background

2.1. Reinforcement Learning

Reinforcement learning is mainly composed of agents, environments, states, actions, and rewards. The agent selects and executes an action in a certain state, and the environment gives rewards for this action and updates the state. The agent learns according to the reward given by the environment, and its purpose is to maximize the cumulative reward. The reinforcement learning model diagram is shown in Figure 1.

Generally, reinforcement learning problems can be described by a Markov decision process (MDP). MDP is defined as a quintuple $<S,A,Psa,Rsa,\gamma >$, where S represents the state space, A represents the action space, $Psa$ represents the probability of taking action a and transitioning to state $S^{\prime }$ in a certain state S, $Rsa$ represents the reward value obtained by taking action a in state S, and $\gamma$ represents the discount factor, which is the degree of attenuation of future returns, where $s,s^{\prime }\in S$, $a\in A$. When $psa$, $Rsa$, $\gamma$ are all known, the model-based dynamic programming method can be used to solve it, otherwise, the model-free reinforcement learning method can be used to solve it. Dynamic programming is a method that divides a complex problem into sub-problems, solves the sub-problems, and finally combines the solutions of the sub-problems to solve the original problem.

Reinforcement learning training algorithms can be divided into two categories: value-based function and strategy-based iteration. The algorithm based on value function needs to update the long-term value $V\left(s\right)$ of each state s, and its update function is

$$\begin{array}{c}\hfill V_{i+1}\left(s\right)\leftarrow \underset{a}{max}\sum _{s^{\prime }}(Ts,a,s^{\prime })[R(s,a,s^{\prime }+\gamma V_i\left(s^{\prime }\right)]\end{array}$$

(1)

The algorithm based on the strategy iteration needs to be divided into two steps. First, a policy is assumed, and the iteration makes the policy cost function converge

$$\begin{array}{c}\hfill V_{i+1}^k\left(s\right)\leftarrow \sum _{s^{\prime }}T(s,{\pi }_k\left(s\right),s^{\prime })\left[R\right(s,{\pi }_k\left(s\right),s^{\prime })+\gamma V_i^k\left(s^{\prime }\right)]\end{array}$$

(2)

Then we optimize according to all possible actions for each state,

$$\begin{array}{c}\hfill {\pi }_{k+1}\left(s\right)=arg\underset{a}{max}\sum _{s^{\prime }}T(s,a,s^{\prime })[R(s,a,s^{\prime })+\gamma V^{{\pi }_k}\left(s^{\prime }\right)]\end{array}$$

(3)

Finally, the optimal strategy for each state is obtained.

2.2. QMIX Algorithm

QMIX [19] is one of the classic algorithms in multi-agent reinforcement learning. The essential difference between multi-agent and single-agent settings is that each agent in the multi-agent environment will interact with the environment, causing the environment to change. However, usually the reward value we obtain, r, is the reward obtained by all the agents acting together, so it is impossible to distinguish the individual reward value of the action taken by a single agent.

The most popular solution to this problem is centralized training with decentralized execution. Centralized training is used during the training process, and the global state can be used to ensure the convergence of the algorithm. In the execution process, each agent can only obtain its own current observation, thus distributed execution.

QMIX uses a hybrid network to combine the local value functions of single-agent, and adds global state information assistance in the training and learning process to improve the performance of the algorithm. The structure can be divided into three parts:

• Agent network: each agent has to learn an independent value function $Qa\left({\tau }^a,u_t^a\right)$. The network structure of the agent adopts the DRQN network structure.
• Hybrid network: This part adopts a fully connected network structure, with $Qa\left({\tau }^a,u_t^a\right)$ of each agent as input, and takes the joint action value function $Q_{tot}\left(\tau ,u\right)$ as the output.
• Hyperparameter network: This part is mainly responsible for generating the weights of the hybrid network.

The cost function of QMIX is

$$\begin{array}{c}\hfill L\left(\theta \right)=\sum _{i=1}^b\left[y_i^{tot}-Q_{tot}\left(\tau ,a,s;\theta \right)\right]\end{array}$$

(4)

where b represents the number of samples sampled from empirical memory,

$$\begin{array}{c}\hfill y^{tot}=r+\gamma max\overline{Q}\left({\tau }^{\prime },a^{\prime },s^{\prime };\overline{\theta }\right)\end{array}$$

(5)

where $\overline{Q}\left({\tau }^{\prime },a^{\prime },s^{\prime };\overline{\theta }\right)$ represents the objective network function.

2.3. Transfer Learning

The purpose of transfer learning is to use the existing knowledge to learn new knowledge. Its core problem is to find the similarities between the source domain and the target domain, and transfer through these similarities. There are three key points in transfer learning, that is, what to transfer, how to transfer and when to transfer.

Algorithms of transfer learning can be mainly divided into several categories:

• Instance-based transfer learning: Find data in the source domain that are similar to the target domain, adjust the data, and use them for training in the target domain.
• Feature-based transfer learning method: The source domain and the target domain have some similar features. After feature transformation, the source domain and the target domain have the same data distribution.
• Model-based transfer learning method: The target domain and the source domain share the parameters of the model, that is, the model trained with a large amount of data in the source domain is directly applied to the target domain.
• Relation-based transfer learning method: When the target domain is similar to the source domain, their internal connections are also similar. This method transfers the relationship between data in the source domain to the target domain.

3. Problem Description

The research background of this paper is that natural disasters occur frequently in my country, and the rescue of a large number of trapped people after the disaster is the focus of the current emergency work. The use of drone swarms for the transportation of relief materials is an important research direction. The focus of this paper is the distribution scheme of heterogeneous UAVs to transport the materials carried by them to the rescue area.

3.1. Simulation Environment Modeling

Multiple UAVs form the UAV swarm to be allocated

$$\begin{array}{c}\hfill UAV=\left\{ua{v}_1,ua{v}_2,\dots ,ua{v}_M\right\}\end{array}$$

(6)

where M is the number of drones in the cluster. In order to digitize the indicators of the drone, the parameters of each drone are represented by a collection

$$\begin{array}{c}\hfill \left\{x_{\mathrm{s}},y_s,x_c,y_c,speed,voyage,dis,ret,loadlist\right\}\end{array}$$

(7)

where

$x_s$: the horizontal axis coordinate of the starting point of the drone;

$y_s$: the vertical axis coordinate of the starting point of the drone;

$x_c$: the current horizontal axis coordinate of the drone;

$y_c$: the current vertical axis coordinates of the drone;

$speed$: the average speed of the drone;

$voyage$: The maximum range that the drone can fly in a single flight without supplementation;

$dis$: The current flight range of the drone;

$ret$: Whether the drone is returning, the returning drone will no longer be assigned tasks, the initial value is 0;

$loadlist$: The list of materials carried by the drone, which is an array.

Rescue areas with different urgency and needs form a task set

$$\begin{array}{c}\hfill Target=\left\{targe{t}_1,targe{t}_2,\dots ,target{t}_N\right\}\end{array}$$

(8)

where N is the number of rescue areas. The information of each rescue area is represented by a collection

$$\begin{array}{c}\hfill \left\{x,y,level,loadlist\right\}\end{array}$$

(9)

where

x: the horizontal axis coordinate of the rescue area;

y: the vertical axis coordinate of the rescue area;

$level$: The emergency level of the rescue area;

$loadlist$: The list of materials needed by the rescue area, which is an array.

When the environment is initialized, all the information about the drone swarm and rescue areas needs to be provided. After initialization, the environment will record the initialization information to facilitate the reset of task assignment. The result of the task assignment is entered in the form of $\left[x_1,x_2,\dots ,x_M\right]$, where $x_i\in \left[0,N\right]$. $x_i\in [0,N-1]$ means that the drone i goes to the rescue area $x_i+1$ for material transportation. If the drone can meet the needs of the rescue area, we unload the amount of materials needed in the rescue area from the drone. Otherwise, unload all the materials on the drone to the rescue area, and modify the location of the drone, the amount of materials carried by the drone, the distance flown by the drone, and the amount of materials needed in the rescue area. When $x_i=N$, it means that the drone completes the flight mission and returns, and the parameter $ret=1$ of the drone needs to be set.

3.2. MDP Model of Task Allocation

Using model-free reinforcement learning to solve the multi-UAV task assignment problem requires modeling the problem as a Markov decision process. Modeling requires the definition of state space, observation space, action space, and reward function.

3.2.1. State Space

According to the description of the simulation environment above, it can be found that only the drone cluster and the rescue areas are involved in this environment, so the state space only needs to contain all the information of the drone cluster and the rescue areas. For the sake of simplicity, we flatten the UAV information and the rescue area information. The state space data form is shown in (10), where the expression of $UA{V}_i$ is (11), and $Targe{t}_i$ expression is Formula (12).

$$[UA{V}_1,UA{V}_2,\cdots ,UA{V}_M,Targe{t}_1,\cdots ,Targe{t}_N]$$

(10)

$$\begin{array}{cc}\hfill & [ua{v}_i^{x_s},ua{v}_i^{y_s},ua{v}_i^{x_c},ua{v}_i^{y_c},ua{v}_i^{speed},\hfill \\ \hfill & ua{v}_i^{voyage},ua{v}_i^{dis},ua{v}_i^{ret},ua{v}_i^{loadlis{t}_1},\hfill \\ \hfill & ua{v}_i^{loadlis{t}_2},\dots ,ua{v}_i^{loadlis{t}_{tot}}]\hfill \\ \end{array}$$

(11)

$$\begin{array}{cc}\hfill & [targe{t}_i^x,targe{t}_i^y,targe{t}_i^{level},\hfill \\ \hfill & targe{t}_i^{loadlis{t}_1},targe{t}_i^{loadlis{t}_2},\dots ,targe{t}_i^{loadlis{t}_{\mathit{tot}}}]\hfill \\ \end{array}$$

(12)

3.2.2. Observation Space

Since the problem involved in this paper is the multi-UAVs task assignment, involving multiple agents, the distributed reinforcement learning method is used to solve it. The observation space of each UAV is related to itself but not related to other UAVs. The observation space is not the same as the state space, so it needs to be redefined. The observation space of the i-th UAV is shown in (13).

$$[UA{V}_i,Targe{t}_1,Targe{t}_2,\cdots ,Targe{t}_N]$$

(13)

3.2.3. Action Space

The action space of each drone is related to the current state of the drone. When $UA{V}_i·ret=0$, that is, when the drone is not in the returning state, the action space of the drone is $\left[0,1,\cdots ,j,\cdots N-1\right]$, where action j represents the UAV to perform the task $targe{t}_{i+1}$. When $UA{V}_i·ret=1$, the drone is in the returning state, and the action space of the drone is $\left[N\right]$, where $targe{t}_N$ is a virtual target point, which means that the drone continues to return to home.

3.2.4. Reward Function

The reward function is very important for reinforcement learning because the agent needs to learn the action through the reward value. It can be said that the quality of the reward function determines the convergence of the algorithm and the convergence result. The goal of the task allocation problem solved in this paper is to complete the task allocation under the condition that the total voyage of the UAV is as small as possible. At the same time, priority shall be given to meeting the material needs of the rescue areas with a high degree of urgency. Experiments show that the reward value calculation formula shown in (14) has a good effect, where $offerResource{e}_{ua{v}_i}$ means that $ua{v}_i$ can satisfy $targe{t}_j$ the number of resources required, $needResouc{e}_{targe{t}_j}$ indicates the number of resources required by $targe{t}_j$, $target{t}_j$ represents the drone selected by $ua{v}_i$ to perform the task, and ${\displaystyle \sum _{i=0}^M}dis(ua{v}_i,targe{t}_j)$ represents the Euclidean distance between the drone and the target point.

$$\begin{array}{cc}\hfill Reward& =\sum _{i=0}^M\frac{offerResourc{e}_{ua{v}_i}}{needResouc{e}_{targe{t}_j}}·targe{t}_j·level-\hfill \\ \hfill & \sum _{i=0}^{M}dis(ua{v}_i,targe{t}_j)\hfill \end{array}$$

(14)

4. Task Assignment Algorithm of Deep Transfer Reinforcement Learning Algorithm Based on QMIX

4.1. DQN-Based Task Assignment Method

The multi-UAV dynamic task assignment problem is essentially a NP-hard problem, and it is easy to fall into the local optimal solution when traditional heuristic methods are used to solve this problem. Therefore, the improvement of the algorithm focuses on how to jump out of the local optimum to find the global optimum, and the emergence of deep reinforcement learning provides a new solution. When the UAV makes a decision, it only depends on the state at the current moment, so the dynamic Markov decision process can be used to solve such problems. At present, most of the related research on the UAV task assignment based on reinforcement learning uses the DQN algorithm [20] based on the value function to solve it.

Since DQN is a single-agent reinforcement learning algorithm, in the implementation process, all UAVs need to be treated as a single-agent. The current state space S is the current data of all drones and the current data of all tasks, and the action space A is the combination of actions of all drones. The update algorithm of its Q-value function is

$$\begin{array}{c}\hfill Q(s,a)\leftarrow Q(s,a)+\mu [R+\gamma \underset{a}{max}Q(s^{\prime },a)-Q(s,a)]\end{array}$$

(15)

4.2. Task Allocation Algorithm Based on QMIX

For the task assignment scenario mentioned in this article, if the DQN algorithm is used to solve it, its action space is $M^N$ (M is the number of drones, N is the number of target points); such an action space is obviously unacceptable when M and N are large. To solve this problem, this paper chooses the QMIX algorithm. Like all reinforcement learning methods, the agent in the QMIX algorithm interacts with the environment through a trial-and-error mechanism, optimizes by maximizing cumulative rewards, and finally achieves the optimal strategy. Different from the centralized strategy of DQN algorithm, the QMIX algorithm adopts the structure of centralized training and distributed execution, and its network is divided into the mixing network and agent network. Each UAV has its own agent network, the input of the network is the observation space of the UAV, and the output is the Q value function of the UAV. The input of the mixing network includes the state space, and the output is the Q-value function of the joint action. The upper limit of the action space of each drone is N, and the action space decreases from $M^N$ to N, which solves the problem of action space explosion. The specific implementation steps are in Algorithm 1.

4.3. Deep Transfer Reinforcement Learning Algorithm Based on QMIX

All deep reinforcement learning algorithms have a common problem: poor real-time performance. Faced with the same allocation problem, the time required by the QMIX method is 10 times or even 100 times that of the genetic algorithm. In the face of such a time difference, although the allocation result obtained by the QMIX algorithm is obviously better than the genetic algorithm, its availability is also very low. In response to this problem, this paper introduces transfer learning to solve it. First, a representative set of source tasks that can cover most of the assigned tasks is learned, and the network parameters of its mixing network and agent network are stored in the policy library. Whenever a new task needs to be assigned, the similarity between the task and the task in the policy library is calculated, and the task with the highest similarity is found to transfer network parameters. Combining the transfer learning with the QMIX algorithm, the algorithm proposed in this paper is the deep transfer reinforcement learning algorithm based on QMIX. The algorithm flowchart of the algorithm is shown in Figure 2, and the specific implementation steps are in Algorithm 2.

Algorithm 1 Task allocation algorithm based on QMIX.

Input: UAV swarm information, target point information

Output: Agent Network and Mixing Network network parameters

1:  Initialize network parameters, data storage unit D, capacity M, total number of iteration rounds T, target network parameter update frequency p

2:  while$\mathrm{i}\le \mathrm{T}$do

3:      Simulation environment, UAV cluster information, target point information restoration

4:      while $\mathrm{step}\le {\mathrm{step}}_{limit}$ do

5:          Obtain the current mission state S, the observed value O of each UAV and the feasible action A of each UAV from the environment

6:          Each drone obtains the Q value of each action taken in the current state through eval netwrok

7:          Choose an action based on the resulting Q value

8:          Get reward R from the environment based on the actions of all drones in the current state

9:          Store $S,S_{next}$, the observed value of each UAV O, the action taken by each UAV A, the feasible action of each UAV U, the reward R, and whether the assigned task is completed $terminated$, into the data storage unit D

10:      if $\mathrm{len}\left(\mathrm{D}\right)\ge \mathrm{M}$ then

11:          Randomly sample some data from D

12:          Update the network parameters according to the loss function $$L\left(\theta \right)=\sum _{i=1}^b\left[y_i^{tot}-Q_{tot}\left(\tau ,a,s;\theta \right)\right]$$

13:      if $imodp==0$ then

14:          Update the parameters of the target network

Algorithm 2 Deep transfer reinforcement learning algorithm based on QMIX.

Input: UAV swarm information, target point information, strategy library and its corresponding source task information

Output: Task assignment results

1:  while$\mathrm{i}\le \mathrm{N}$do

2:      Simulation environment, UAV cluster information, target point information restoration

3:      Calculate the sum total of the squared differences between the input information and the information corresponding to the ith source task

4:      if $\mathrm{total}\le \min$ then

5:          $\min =\mathrm{total}$

6:          $\mathrm{flag}=\mathrm{i}$

7:  Copy the network parameters of the i source task to the network of the new task

8:  Execute Algorithm 1, but do not need to initialize the network parameters

5. Algorithm Verification and Analysis

5.1. Experimental Setup

In order to verify the effectiveness of this algorithm, 5 UAVs and 10 target points were randomly generated this time, and allocated for 3 kinds of resources. The data structures of UAVs and target points are shown in Figure 3 and Figure 4. This experiment is coded in Python, and the neural network part of the QMIX algorithm is implemented using the Pytorch deep learning framework. The experimental platform is a computer with Apple M1 pro chip and 16 GB memory.

5.2. Experimental Process

5.2.1. Task Allocation Based on QMIX Algorithm

The parameter settings of the algorithm are shown in

Table 1. QMIX algorithm parameter list.

Name	Value
seed	123
n_epochs	50,000
evaluate_per_epoch	100
batch_size	32
buffer_size	100
update_target_params	200
drqn_hidden_dim	64
qmix_hidden_dim	32

, and the cumulative reward value of the QMIX algorithm is shown in Figure 5. It can be seen from the figure that the cumulative reward value obtained by this algorithm increased rapidly in the first 7700 cycles, and then converged to 2.443. Through analysis, it can be seen that the fluctuations after 5000 cycles are affected by random actions with small probability.

5.2.2. Task Assignment Based on Genetic Algorithm

The cumulative reward value obtained by the heuristic genetic algorithm for the same set of data is shown in Figure 6, and its parameter settings are shown in

Table 2. Genetic algorithm parameter list.

Name	Value
seed	123
num_total	30
iteration	200
max_step	5

. Observing the image, it can be found that in about the first 20 cycles, the cumulative reward value obtained by the algorithm shows an extremely rapid upward trend. In the 20th to 70th cycles, the cumulative reward obtained by the algorithm shows a certain platform period, but eventually rises. It can be judged that it falls into the local optimum. Due to the existence of random actions, it successfully jumps out of the local optimum and gradually moves toward global optimization. From the 70th epoch, the cumulative reward system is no longer increased, and finally converges to 1.589, significantly lower than the qmix algorithm.

5.2.3. Deep Transfer Reinforcement Learning Algorithm Based on QMIX

From the above two experiments, we can find that although the cumulative reward value of the genetic algorithm is much lower than that of the QMIX algorithm, the iteration period required for the QMIX algorithm to converge is 250 times that of the genetic algorithm. Therefore, it is necessary to introduce transfer learning at this time, and find the source task with the smallest similarity difference with the secondary task in the strategy library to perform parameter transfer. After the network parameters are migrated, the comparison of the cumulative reward value between this algorithm and the genetic algorithm is shown in Figure 7, and the parameter settings of this algorithm are shown in

Table 3. Algorithm parameter setting in this paper.

Name	Value
seed	123
n_epochs	200
evaluate_per_epoch	1
batch_size	32
buffer_size	100
update_target_params	200
drqn_hidden_dim	64
qmix_hidden_dim	32

. Observing the image, it can be found that the algorithm drops sharply in the first 20 cycles. After analysis, it is because the new task is different from the source task. Then, it shakes sharply in 20–160 cycles, and finally converges at 2.828. The result is significantly better than the genetic algorithm.

Table 4. Algorithm runtime comparison.

Algorithm	Running Time/s
Genetic algorithm	6
QMIX algorithm	120
Algorithm in this paper	8

shows the comparison of the running time of the genetic algorithm, the algorithm based on QMIX, and the algorithm proposed in this paper. It can be found that the optimal value converged by the algorithm proposed in this paper is obviously better than the genetic algorithm, and its running time is basically the same as that of the genetic algorithm.

5.2.4. Universal Verification

The experiment given above is the allocation effect when the number of UAVs is less than the number of target points. Now experiments are carried out for the number of UAVs equal to the number of target points, and the number of UAVs is greater than the number of target points to verify the universality of the algorithm given in this paper. In this paper, the algorithm is tested on 5, 10, and 20 UAVs and 10, 20, and 30 target points. The experimental results show that the cumulative reward value of the algorithm proposed in this paper is obviously better than that of genetic algorithm, and the time is equivalent to that of genetic algorithm.

Figure 8 and Figure 9 are the task assignment results of 10 UAVs and 10 target points. It can be seen that the optimal value of the final convergence of the algorithm proposed in this paper is obviously better than that of the genetic algorithm.

Figure 10 and Figure 11 show the task assignment results of 10 UAVs and 5 target points. It can be seen that the optimal value of the final convergence of the algorithm proposed in this paper is still significantly better than that of the genetic algorithm.

Through the experimental verification in this section, the algorithm proposed in this paper is effective when the number of drones is less than the number of target points, the number of drones is equal to the number of target points, and the number of drones is greater than the number of target points.

6. Conclusions

Aiming at the task assignment problem of multiple UAVs, this paper proposes a task assignment algorithm based on deep transfer reinforcement learning based on QMIX. The algorithm helps to combine the experience of the source task to the new task and accelerate the convergence speed by migrating the network parameters learned by QMIX in the source task. The experimental results show that the algorithm proposed in this paper is significantly better than the traditional heuristic method in allocation efficiency, and the running time is basically the same. The algorithm provides a new idea for using artificial intelligence methods to solve the task assignment problem.

The future research work is mainly divided into the following aspects: First, improve the similarity calculation method between the new task and the source task group. The existing similarity calculation method is too straightforward, which may lead to the source tasks with high similarity not being identified. Second, consider whether there is a better migration method. The existing migration mode is network parameter migration, which requires the number of UAVs and target points of the migrated task to be consistent with that of the new task. However, QMIX is a distributed decision algorithm, and this migration method does not make good use of its advantages. Third, the current algorithm mainly considers the range constraint, but time is also a very important factor in practical scenarios, which needs to be taken into account in subsequent research.