Multi-UAV Urban Logistics Task Allocation Method Based on MCTS

Abstract

Unmanned aerial vehicles (UAVs) open new methods for efficient and rapid transportation in urban logistics distribution, where task allocation is a significant issue. In urban logistics systems, the energy status of UAVs is a critical factor in ensuring mission fulfillment. While extensive literature addresses the energy consumption of UAVs during tasks, the feasibility of energy replenishment must be addressed, which introduces additional uncertainty to the task allocation. This paper realizes multi-tasking, considering the energy consumption and replenishment of UAVs, to ensure that the tasks can be accomplished while reducing energy consumption. This paper proposes uniform distribution K-means to realize balanced multi-task grouping. Based on the Monte Carlo tree search (MCTS), a task-allocation-oriented MCTS method is proposed, including improving the selection and simulation process of MCTS. The aim was to collaborate with multiple trees for node selection and record historical simulation information to guide subsequent simulations for better results. Finally, the optimality of the proposed method was validated by comparing it with other relevant MCTS methods through several randomized experiments.

1. Introduction

In recent years, unmanned aerial vehicles (UAVs) have made outstanding progress in structure, control systems, and communication systems [1]. With technological advances and cost reductions, the use of UAVs in areas such as mapping [2], agriculture [3], public safety [4], transportation systems [5], health systems [6], and logistics [7] has been gradually increasing, especially in the logistics sector.

Currently, the logistics industry is experiencing rapid growth, with the number of parcels delivered by various companies increasing annually. However, some inherent limitations and challenges are associated with traditional logistics and transportation methods, such as traffic congestion, transportation time, and high labor costs [8,9]. The emergence of UAVs brings excellent opportunities for change in the logistics industry. First, UAVs are not restricted by ground transportation conditions and can carry out rapid cargo distribution, significantly improving logistics efficiency [10]. Moreover, the flexibility of UAVs enables them to enter areas with complicated terrain or inconvenient transportation options, providing a new solution for logistics supply in remote areas and making up for the gaps that traditional logistics cannot cover [11]. Unlike traditional logistics’ dependence on petroleum resources [12], UAVs use clean energy, reducing energy needs [13,14]. The green and low-carbon application mode of UAVs could help to address problems such as environmental pollution and global warming and meet the needs of green and sustainable development [15,16].

Thanks to the high flexibility, low cost, and eco-friendliness of UAVs [17], many companies are actively exploring the introduction of UAVs into the logistics industry. As the world’s largest e-commerce company, Amazon has launched the Prime Air drone delivery service, aiming to deliver parcels to homes within 30 min [18]. As a globally renowned fast-food giant, Domino’s Pizza is a franchisee that has obtained the right to deliver pizzas by drone. Pizza delivery is strictly limited by local regulators. Therefore, with the authorization of the New Zealand government, they completed a test of drone pizza delivery [19]. The German logistics company Deutsche Post DHL has initiated a project called Parcelcopter, which delivers medical supplies by drone. This service is particularly suitable for remote areas and situations where drugs are urgently needed [20]. In addition, companies like Google, Walmart, and Alibaba are also committed to drone delivery research and have conducted a series of delivery tests [21].

Furthermore, to address the issue of UAV flight time being limited by battery capacity, some enterprises and scholars are focusing on the research of UAV docks [22,23]. UAV docks allow UAVs to land, change batteries automatically, and then take off again [24]. Through such a design, UAVs can undertake multiple tasks consecutively, extending the working hours of UAVs and laying the groundwork for UAV applications in logistics.

In the logistics industry, a core issue is allocating multiple tasks to different UAVs effectively. Allocating multiple tasks to multiple UAVs can be seen as a multi-robot task allocation (MRTA) problem. In the past few years, many studies have proposed various methods for MRTA problems, mainly categorized into methods based on linear programming, heuristic-based methods, and auction algorithms.

The task allocation problem for multiple UAVs or robots working collaboratively is a classic combinatorial optimization problem in mathematics and can be viewed as a 0–1 integer linear programming problem [25]. The primary method for solving linear programming problems in the early days was the Hungarian algorithm [26], which can be used to solve allocation optimization problems with one-to-one matching characteristics. Madridano designed a UAV task allocation system based on the Hungarian algorithm for building emergencies. This system assigned different tasks to UAVs based on parameters such as urgency and distance of the task [27]. Mixed-integer linear programming methods can handle more complex calculations and have been used to solve multi-objective task allocation problems for robots under uncertain conditions [28,29,30]. Linear programming is suitable for problems with solid constraint conditions, requiring the objective function of the problem to be linear.

Heuristic-based methods for solving MRTA problems can find optimal or near-optimal solutions while meeting specific constraints [31]. Based on heuristic information (such as robot positions and capabilities), heuristic methods evaluate and select heuristic information until a condition is met. For instance, the genetic algorithm for solving MRTA problems simulates the evolutionary mechanisms in nature to obtain relevant information during the search process, yielding the best allocation scheme [32,33,34]. Similarly, algorithms like particle swarm optimization (PSO) and ant colony optimization (ACO) can also be used to address MRTA problems [35,36,37]. However, heuristic-based methods converge slowly when faced with problems with large search spaces.

In recent years, increasing research has focused on addressing MRTA problems using auction algorithms [38]. Auction algorithms draw from the concept of market transactions, viewing tasks as bidding objects. Each robot bids based on its capabilities and requirements, ultimately forming an optimized task allocation scheme [31]. Based on auction algorithms, the consensus-based auction algorithm (CBAA) and consensus-based bundle algorithm (CBBA) have been used to solve single-task and multi-task allocation problems [39]. Moreover, Cheng successfully used auction algorithms to solve multi-UAV task allocation problems under multiple constraints [40]. Combining communication situations, some research has delved into impaired communication states’ impact on the performance of auction algorithms [41]. The Robot-Group Assignment Strategy also combines grouping strategies and auction algorithms, effectively solving MRTA problems [42].

With the continuous advancement of UAV technology and the increasing complexity of task allocation scenarios, the methods above have certain limitations when dealing with MRTA problems with various constraints. To address these challenges, some new approaches offer better solutions for MRTA problems, such as the Monte Carlo Tree Search (MCTS). MCTS was initially mainly used to improve computer performance in board games, especially in Go [43]. AlphaGo, developed by Google DeepMind, employed MCTS as its key component [44]. Moreover, MCTS has gradually been applied to broader domains. For instance, in ever-changing factory environments, MCTS can facilitate rapid automated decisions for human–machine collaboration [45]. Based on MCTS, signal control systems at intersections can be optimized, providing an effective solution for urban traffic management [46]. In autonomous driving technology, by combining reinforcement learning with MCTS, the unsafe behaviors of self-driving vehicles can be minimized, enhancing their performance in intricate settings [47]. In UAV-aided wireless systems, UAV planning problems are addressed using the random-MCTS (R-MCTS) strategy [48]. Furthermore, the greedy-MCTS (G-MCTS) strategy and a combined approach of the random–greedy simulation strategy (random–greedy-MCTS, RG-MCTS) can effectively tackle the weighted vertex coloring problem [49]. However, only some studies leverage MCTS to solve UAV task allocation challenges.

The information from the related literature and its method type is summarized in

Table 1. Summary of the related literature.

Reference	Method Type	Application Scenario	Objectives	Constraints
[27]	Hungarian method	UAV rescue service	Time consumption	Obstacle avoidance
[28]	mixed-integer programming	UAV fleet coordination	Travel distance	Obstacle avoidance
[29]	mixed-integer programming	Robot trajectory planning	Action cost	Robots action Feasibility
[30]	mixed-integer programming	Robot tasks allocation	Time consumption	Tasks feasibility
[32]	Genetic Algorithm	UAVs inspection service	Time consumption	Obstacle avoidance
[34]	Genetic Algorithm	UAVs inspection service	Time and energy consumption	Tasks feasibility
[35]	Particle Swarm Optimization	Robot task allocation	Travel distance	Computing capacity
[36]	Particle Swarm Optimization	Robot rescue service	Computing time	Robot capability
[39]	Auction algorithm	autonomous vehicles task allocation	Travel distance and feasible solutions	Communication capability
[40]	Auction algorithm	UAVs task allocation	Travel cost	Time window
[41]	Auction algorithm	Robot task allocation	Travel cost	Communication capability
[42]	Auction algorithm	Robot task allocation	Time consumption	Time window
[45]	Monte Carlo Tree Search	Human-robot collaboration	Time consumption	Task execution time
[46]	Monte Carlo Tree Search	Traffic signal optimization	Signal optimization	Computing capacity
[47]	Monte Carlo Tree Search	Autonomous driving	Driving safety	Obstacle avoidance
[48]	Monte Carlo Tree Search	UAV-aided wireless systems	UAV path and computing time	Energy consumption and user fairness
This paper	Monte Carlo Tree Search	City delivery	Energy consumption	Energy consumption and replenishment payload capacity

. Although there have been many studies on task allocation, some aspects have not been considered: (1) excess numbers of tasks can impact decision-making efficacy; (2) UAV task planning is strictly limited by energy and capacity; and (3) energy replenishment services of UAV docks should also be factored in. These aspects will introduce more uncertainties to the task allocation.

The core objective of this paper was to design an effective multi-task allocation method that considers the need for UAVs to be energized via UAV docks to meet the demands of performing longer-range and longer-time missions in the presence of limited UAV payload capacity and battery energy. In this paper, the process of task assignment was divided into two steps. Firstly, to reduce the search range of task allocation, this paper divided all the tasks into different sub-task groups and performed task allocation within each sub-task group. Secondly, for task allocation, this paper proposes the task allocation MCTS (TA-MCTS) method based on MCTS. TA-MCTS aims to minimize the energy consumption of UAVs, considers the constraints of load capacity and battery energy, and rationally formulates UAV flight strategies. The main methods are as follows:

(1) Based on the K-Means clustering method, this paper constrained the number of elements in each sub-task group so that it not only grouped all tasks but also ensured the same number of tasks were allocated to each sub-task group.

(2) Within the divided sub-task groups, task allocation was realized by multiple trees of MCTS corresponding to multiple UAVs.

(3) In the selection and expansion phase of MCTS, selection and expansion optimization strategies were proposed. By considering the energy state of the UAVs during the execution of the mission, the following possible nodes were selectively expanded to reduce the search range of the MCTS.

(4) In the simulation phase of MCTS, a simulation optimization strategy was proposed. The strategy utilized the simulation results during the multi-UAV task assignment process as heuristic information to guide the selection of tasks in the simulation phase.

The rest of the paper is organized as follows: Section 2 provides a problem description. An overview of the research methodology of the paper is given in Section 3. Section 4 analyzes the performance of the proposed methodology in different settings. Finally, the paper concludes in Section 5.

2. Problem Description

2.1. Problem Scenario

Urban UAV logistics is a complex system involving the interaction of multiple entities, including UAVs, distribution centers (DC), UAV docks, and task locations. In this paper, we mainly considered how to assign multiple tasks to UAVs to minimize energy consumption when UAVs have limited energy and UAV docks are available. As shown in Figure 1, although the energy of the UAV could meet the needs of tasks 1 and 2, it could not perform task 3. Therefore, the UAV should replace the battery at the dock after performing task 1 to reach the energy requirement for performing three tasks. Unlike the traditional allocation method, the consumption of UAV energy affects the task allocation process.

To simplify the problem, the following assumptions were made in this paper:

• The distribution center covers a specific area, and the distribution targets of the task are distributed in this area.
• UAV docks are distributed in this area and serve as service stations for UAVs to change their batteries.
• The UAVs have limited energy sources, and when low on power, the UAVs need to travel to docks for battery replacement.
• The UAVs have limited carrying capacity, and each can only carry a limited number of items of the same weight.
• The UAVs maintain a consistent flight speed throughout the delivery process, correlating energy consumption per kilometer to the number of items carried.

2.2. Task Model

The task model and symbol are described as follows: Let $U$ denote the set of m UAVs, with the UAVs denoted by $u_1, u_2, \cdots , u_m$. Each UAV has the same attributes: average flight speed $V_{avg}$, average flight altitude $z_{avg}$, maximum number of items to be carried $P_{max}$, maximum battery capacity $E_{max}$, and energy consumed per kilometer $E_{max}$. Let $T$ denote the set of $N$ tasks, with the tasks denoted by $t_1,t_2, \cdots , t_N$, and the task location denoted by their coordinates $l=(x, y, z)$. $E_{cost}$ is defined as Equation (1), where $E$ represents the energy consumed by the UAV per kilometer when no items are loaded, and $\alpha$ is the number of loaded items. $∆$ denotes the energy consumed by each item (based on the assumption that the weight of the items is the same).

$$E_{cost}=E+\alpha ∆$$

(1)

When task $t_j$ is assigned to UAV $u_i$, the energy $c_{ij}$ required for UAV $u_i$ to accomplish $t_j$ is shown in Equation (2), where $D\left(l_{j-1}, l_j\right)$ denotes the distance of the UAV from position $t_{j-1}$ to position $t_j$.

$$c_{ij}=D\left(l_{j-1}, l_j\right)E_{cost}$$

(2)

$$D\left(l_{j-1}, l_j\right)=\sqrt{{\left|x_j-x_{j-1}\right|}^2+{\left|y_j-y_{j-1}\right|}^2}+\left|2z_{avg}-z_j-z_{j-1}\right|$$

(3)

In this paper, all tasks are grouped, and each sub-task group is denoted by $S$, with the same number of tasks in each group denoted as $n$. The following constraints are satisfied if the task group $S$ is feasible:

$$\sum _{j=1}^n{x}_{ij}\le P_{max} \forall i\in U$$

(4)

$$\sum _{i=1}^m{x}_{ij}\le 1 \forall j\in S$$

(5)

$$x_{ij}\in \left\{0, 1\right\} \forall \left(i, j\right)\in U\times S$$

(6)

$$E\left(x_{i\left(j-1\right)}\right)-c_{ij}>0 \forall \left(i, j\right)\in U\times S$$

(7)

where $x_{ij}=1$ indicates that task $j$ is assigned to UAV $i$; otherwise, $x_{ij}=0$. Equation (4) indicates that the number of items carried by each UAV cannot exceed the maximum limit, $P_{max}$. Equation (5) indicates that, at most, one UAV is assigned to each task. $E\left(x_{ij}\right)$ indicates the energy left after UAV $i$ completes task $j$. Equation (7) indicates that after completing the previous task, the UAV should have enough energy to reach the following task location.

Considering that the MRTA problem can be expressed as a minimization objective function, in this paper, the objective function was defined as all the energy $C$ consumed by the UAV to perform the tasks, and the objective function is shown in Equation (8) under the above constraints:

$$C=\sum _{i=1}^m\sum _{j=1}^n{c}_{ij}{x}_{ij}$$

(8)

3. Methodology

Generally, a DC in an urban area with many required delivery tasks is responsible for a specific area. In order to accomplish all the tasks with minimum energy consumption, the core idea of TA-MCTS proposed in this paper was to group the tasks and solve the multi-task assignment problem by local search, as shown in Figure 2. This consisted of two main aspects. Firstly, the tasks were grouped into multiple sub-task groups by improved K-means. Then, using MCTS, all the tasks contained in each group were assigned to multiple UAVs.

3.1. Task Grouping Strategy

A direct assignment of all tasks in DC with many tasks would create a large-scale search range, leading to poor assignment results. Therefore, it was crucial to group all tasks according to UAV characteristics. These sub-task groups directly affected the way the UAV performed its tasks. Moreover, grouping results play a crucial role in reducing energy consumption.

According to the scenario in Section 2.1, the DC set in this paper was surrounded by multiple distribution task locations. Considering the load capacity of UAVs, this paper first clustered the tasks according to their locations. Currently, standard clustering methods include density-based, hierarchical, and partitioning clustering methods.

K-means, among the partitioned clustering methods, is widely used due to its efficiency. The K-means algorithm can randomly assign all tasks to K clusters, with K representing the number of groups. K-means bases its classification on the distances of each element from the center of each cluster without considering the number of elements in each cluster. However, during the task assignment process, if the clustering results in an unequal number of elements, it may result in the number of tasks exceeding the maximum load capacity of the UAV, which will thus fail to complete the task assignment.

In this paper, based on K-means, the uniform distribution K-means (UD-Kmeans) method was proposed to ensure that the number of elements in each cluster was the same according to the payload capacity of the UAV. $K$ is calculated as shown in Equation (9), where $K$ denotes the number of all sub-task groups, and the meanings of $m, N,P_{max}$ are shown in Section 2.2.

$$K=\mathit{min}\left(m, \frac{N}{m{P}_{max}}\right)$$

(9)

The details of the UD-Kmeans method are described in Algorithm 1. First, a point is randomly selected as the first clustering center from all task locations (Line 2). The shortest distance from each task location to the existing clustering center $o_i$ is calculated. Moreover, we select the next center point with a probability weighted by the square of the distance. The first clustering center is selected by the above method (Lines 3–9). When all the $K$ clustering centers are selected, each task is assigned to the nearest center to form a sub-task group $S$ (Lines 12–15). After that, whether the item number in each sub-task group $s_i$ exceeds the specified number is checked. If it is exceeded, the task $t_j$ that is farthest from the current $s_i$ is moved out of $s_i$, and $t_j$ is moved into $s_k$ closest to $t_j$ (Lines 16–20). Then, the mean value of $s_i$ is calculated, and the clustering center is updated (Lines 22–24). The next iteration is carried out until the clustering center does not change.

Algorithm 1: Uniform Distribution K-Means
1:	Initialization: $\mathrm{randomly} \mathrm{select} \mathrm{any} \mathrm{task} \mathrm{location} \mathrm{in} T$$\mathrm{as} \mathrm{the} \mathrm{first} \mathrm{center} \mathrm{point} o_1$
2:	Construct a list of clustering centers $O$$, O=\left\{o_1\right\}$
3:	for $i=2$ to $K$ do
4:	for each $t_j$ in $T$ do
5:	$\mathrm{Calculate} \mathrm{the} \mathrm{distance} \mathrm{from} t_i$$\mathrm{to} \mathrm{all} \mathrm{elements} O$,$\mathrm{take} \mathrm{the} \mathrm{minimum} \mathrm{value} {dis}_j$
6:	end for
7:	$\mathrm{Select} \mathrm{the} \mathrm{new} \mathrm{centroid} o_i$$\mathrm{according} \mathrm{to} \mathrm{the} \mathrm{probability} P\left(ci\right)\propto {dis}_j^2$
8:	Add new $\mathrm{center} \mathrm{point} o_i$$\mathrm{to} O$
9:	end for
10:	while $O$ is changed do
11:	Clear all elements in $s_i$$, s_i\in S$
12:	for each $o_i$ in $O$ do
13:	$i=argmin\left\{D\left(o_i, t_j\right), o_i\in O, t_j\in T\right\}$
14:	$s_i\leftarrow s_i\cup \left\{t_j\right\}$
15:	end for
16:	for each $s_i$ in $S$ do
17:	while $\left|s_i\right|>m{P}_{max}$ do
18:	$j=max\left\{D\left(o_i, t_j\right), t_j\in s_i\right\}$$, s_i\leftarrow s_i-\left\{t_j\right\}$
19:	$k=min\left\{D\left(o_k, t_j\right), o_k\in O,o_k\ne o_i\right\}$$, s_k\leftarrow s_k\cup \left\{t_j\right\}$
20:	end while
21:	end for
22:	for each $o_i$ in $O$ do
23:	$o_i={\sum }_{t_j\in s_i}(x_j,y_j,z_j)/\left|s_i\right|$
24:	end for
25:	end while

3.2. MCTS Task Allocation

MCTS is widely used as a powerful method in games and decision-making problems. The method constructs a tree and stores information about each action through iterations, allowing for more illuminating decisions to be made in subsequent decisions. Figure 3 shows one complete iteration in MCTS.

Selection: From the root node, MCTS selects the node with the highest value in each iteration, according to the selection strategy.

Expansion: If a node is found that has not been expanded, one or more child nodes are added to this node to indicate the following possible action.

Simulation: Starting from the selected node, a series of actions are simulated until an end state is reached. Usually, a random or greedy strategy is adopted [50].

Backpropagation: MCTS propagates the simulation results to the root node and updates all nodes on the path.

In the selection phase, the upper confidence bounds for trees (UCT) method is most commonly used to balance exploration and exploitation of the entire tree, as shown in Equation (10):

$$UCT={argmax}_{a\in A\left(s\right)}\left\{Q\left(s,a\right)+C\sqrt{\frac{\mathit{ln}\left[N\left(s\right)\right]}{N\left(s,a\right)}}\right\}$$

(10)

where $A\left(s\right)$ denotes the set of actions available in state $s$. $Q\left(s,a\right)$ denotes the average reward of action $a$ in state $s$. $N(s,a)$ denotes the number of executions of action $\mathrm{a}$ in state $s$. $N\left(s\right)$ denotes the number of all executions of state $s$. $C$ is a constant controlling the balance between exploration and exploitation. A larger $C$ implies exploring more unvisited nodes; otherwise, it is more inclined to utilize visited nodes. With the reward value set within the range $\left[\mathrm{0,1}\right]$, $C=1/\sqrt{2}$ favors the construction of the MCTS tree.

In this paper, based on MCTS, the TA-MCTS method was proposed to solve the multi-task assignment problem of UAVs. First, considering that there were multiple UAVs in the scene, constructing only one tree was expected to lead to a sharp increase in the number of nodes, affecting the search speed and efficiency. Therefore, this paper constructs a separate tree for each UAV.

Figure 4 shows the process of TA-MCTS proposed in this paper. When searching multiple trees in TA-MCTS, a public task set is established to record the changes of selected tasks in the TA-MCTS process corresponding to each UAV to prevent task selection conflicts. Firstly, in this paper, we select a TA-MCTS process corresponding to a UAV, and according to the selection rule, a task is selected and removed from the task set to prevent it from being selected by other trees. After that, the node will expand according to the selected tasks. In the simulation phase, based on the historical information generated in the iterative process and the tasks selected by the current tree, the subsequent tasks that may be executed are obtained and removed from the task set. Finally, a backpropagation phase is performed to update the information.

The TA-MCTS method has two essential aspects: the selection and expansion optimization strategy and the simulation optimization strategy.

Selection and Expansion Optimization Strategy (SEOS): In the selection phase of the MCTS, this strategy ensures that, in an iteration, in addition to considering the reward values of the nodes, there is no task conflict between each UAV. In the expansion phase, this strategy considers the energy consumption of UAVs and selectively expands the possible tasks according to the current energy status of UAVs. This method can efficiently build different nodes in the tree.

Simulation Optimization Strategy (SOS): During the MCTS tree construction, we recorded each node’s value as heuristic information. This information was used in subsequent simulations to compute more rational results.

Algorithm 2 shows the whole flow of the TA-MCTS method. TA-MCTS builds a search tree for each UAV and sets its state. A public task set $Vs$ is built to represent the tasks visited to avoid task selection conflicts (Lines 3–4). Define the matrix $M$ as an information repository to record the information for the simulation optimization strategy (Lines 5). Lines 6–15 represent all the processes during one iteration of TA-MCTS. First, a search tree corresponding to a UAV $u_i$ is selected. Following the node selection strategy, the tree selects and expands a task node. The simulation optimization strategy is used in the simulation process to guide nodes in choosing possible tasks. Finally, the backpropagation process updates the result based on the simulation outcome. During this process, $Vs$ is continuously updated to collaborate with the corresponding trees of different UAVs to prevent task conflicts. The notations and functions involved are further elaborated in Section 3.2.1 and Section 3.2.2.

Algorithm 2: TA-MCTS
1:	Initialization:
2:	$\mathrm{Set} \mathrm{the} \mathrm{energy} \mathrm{of} \mathrm{all} \mathrm{UAVs} \mathrm{in} U$$\mathrm{to} E_{max}$
3:	$\mathrm{Construct} \mathrm{an} \mathrm{empty} \mathrm{tree} \mathrm{for} \mathrm{each} \mathrm{UAVs} \mathrm{with} \mathrm{the} \mathrm{root} \mathrm{node} v_0$ representing the DC.
4:	$\mathrm{Build} \mathrm{a} \mathrm{public} \mathrm{task} \mathrm{set} Vs$ denoting visited tasks
5:	$\mathrm{Define} \mathrm{matrix} M$$\mathrm{with} \mathrm{dimensions} \left(n+L\right)\times \left(n+L\right)$
6:	while iterations are not completed do:
7:	$\hspace{1em}\hspace{1em}\mathrm{UAV} u_i$ is randomly selected
8:	$v_j\leftarrow Selection\left(v\right)$
9:	$Expand\left(v\right)$
10:	$\mathbf{if} \mathrm{the} \mathrm{position} x_j^i$$\mathrm{of} \mathrm{node} v_j$$\mathrm{belongs} \mathrm{to} S$ then
11:	$Vs\leftarrow Vs\cup \left\{x_j^i\right\}$
12:	end if
13:	$C\leftarrow Simulation\left(v\right)$
14:	$Backpropagation(v,C)$
15:	end while

3.2.1. Selection and Expansion Optimization Strategy

Combined with the energy situation of UAVs, this paper proposes a selection and expansion optimization strategy to optimize the selection and expansion phases of MCTS. Firstly, through SEOS, we ensured that there was no conflict in the task selection of each UAV in the respective node selection phase. Furthermore, during the expansion process, considering that UAV energy has limitations, not all locations satisfy the expansion needs. SEOS selectively added feasible nodes according to the UAV energy status, enabling the following simulation process to be effective and improving the multi-task assignment efficiency for UAVs.

According to the definition in Section 2.2, let $U$ denote the set of $m$ UAVs, with the UAVs denoted by $u_1,u_2, \cdots , u_m$. $p_i$ denotes the number of items the UAV $u_i$ currently carries. In this paper, task assignments for UAVs were performed in each sub-task group. We used $S$ to denote each sub-task group, and the number of tasks in each group is $n$, with the tasks denoted by $s_1,s_2, \cdots , s_n$ as shown in Equation (11).

$$S=\left[s_1,s_2, \cdots , s_n\right]$$

(11)

Let $R$ denote the set of $L$ UAV docks, with the UAV docks denoted by $r_1,r_2, \cdots , r_L$ as shown in Equation (12).

$$R=\left[r_1,r_2, \cdots , r_L\right]$$

(12)

The task location $j$ of UAV $u_i$ is denoted by $x_j^i$, where $j\in \left(S\cup R\right)$ denotes the UAV docks and task locations. As shown in Equation (13), the energy state of UAV $u_i$ at position $j$ is denoted using ${xe}_j^i$. When satisfy ${xe}_{j-1}^i-c_{ij}>0$, it means the UAV has enough energy to fly from the previous position to $x_j^i.$ When $x_j^i$ belongs to $R$, it means the UAV completes the battery exchange and the energy becomes $P_{max}$ again.

$${xe}_j^i=\left\{\begin{array}{l}{xe}_{j-1}^i-c_{ij}, if {xe}_{j-1}^i-c_{ij}>0 and j\in S\\ P_{max}, if {xe}_{j-1}^i-c_{ij}>0 and j\in R\end{array} \right.$$

(13)

As shown in Figure 5, when the UAV $u_i$ is in the node expansion phase, it selects tasks that have not been visited. Using Equation (13), it evaluates its energy and selects reachable nodes while removing inaccessible nodes due to insufficient energy. Based on this, we ensured the nodes’ validity and effectively reduced the search tree’s size, minimizing invalid and redundant searches.

Based on the above calculations, SEOS pseudo-code as shown in Algorithm 3.

Algorithm 3: Selection and Expansion Optimization Strategy
1:	function $Selection\left(v\right)$
2:	while state of $v$ satisfy ${p_i<P}_{max}$ do
3:	if $v$ not fully expanded then
4:	return  $v$
5:	else
6:	$v\leftarrow BestChild\left(v\right)$
7:	end while
8:	return $v$
9:	function $Expand\left(v\right)$
10:	choose $j\in \left\{\left(S\backslash Vs\right)\cup R\right\}$
11:	${xe}_j^i$ is obtained by Equation (13)
12:	if ${xe}_j^i>0$ then
13:	set state and position of the new node $v^{\prime }$ according to $j$ and ${xe}_j^i$
14:	add a new child $v^{\prime }$ to $v$
15:	return $v^{\prime }$
16:	else
17:	$Expand\left(v\right)$
18:	end if
19:	function $BestChild\left(v\right)$
20:	$v$ is obtained by Equation (10)
21:	return $v$

The above pseudo-code primarily outlines the SEOS method. Lines 1–7 depict the selection function. If a node has not reached its termination status, it means the UAV $u_i$ is currently carrying items not exceeding $P_{max}$. If this node is not fully expanded, it will proceed with expansion. Otherwise, the optimal node will be selected. Lines 8–16 illustrate the expansion function. An unvisited task location $j$ is chosen, and then ${xe}_j^i$ is computed based on Equation (13). If ${xe}_j^i>0$, a new node will be generated and added to the child list of the prior node. If not, a new task location $j$ that has not been visited will be reselected. Lines 17–19 depict the optimal node computation function, where its reward value is computed through Equation (10).

3.2.2. Simulation Optimization Strategy

Typically, the MCTS simulation phase starts at a selected node and carries out a series of actions, either randomly or according to some strategy, until it reaches the termination state. This process is also referred to as a rollout. However, the traditional simulation approach may not be optimal when faced with the multi-UAV task assignment problem.

The simulation optimization strategy proposed in this paper aims to enhance the simulation phase of MCTS for the multi-UAV task assignment problem. The following is the core idea of the strategy:

In solving the multi-UAV task assignment, the method proposed in this paper builds a separate tree for each UAV. Moreover, as the MCTS iterates the process many times, many different results are formed. This paper recorded and shared these simulation results with all UAVs. These results were used as heuristic information to form an information repository during multiple iterations, which could be used as historical experience to guide action selection in the subsequent simulation process. The simulation phase could be more effective by combining the historical experience with the current state information of a particular UAV. Thus, a stochastic simulation process was transformed into an intelligent simulation that combined historical heuristic information. SOS improved the effectiveness of the simulation phase of MCTS and provided a better solution for MCTS to solve the multi-task assignment problem.

Based on SOS, this paper recorded the results of each iteration in the tree for each UAV and updated the information repository. By defining matrix $M$ as the information repository with dimension $\left(n+L\right)\times \left(n+L\right)$, $n$ is the number of tasks in the sub-task group, and $L$ is the number of UAV docks. In matrix $M$, the information between each task position is computed as shown in Equation (14):

$$M_{ij}\left(t\right)=M_{ij}\left(t-1\right)+{\left(c_{ij}\right)}^{-1}$$

(14)

where $M_{ij}\left(t\right)$ denotes the value of information between positions $i$ and $j$ during the $t$-th iteration, $i,j\in \left(S\cup R\right)$. The matrix $M$ is constantly updated through multiple iterations of multiple UAVs, which guides the simulation process in MCTS.

During the simulation process, the information in the matrix $M$ helps guide the algorithm to quickly select the following possible actions based on the existing information to find a suitable allocation scheme. Specifically, the following action is selected in the simulation process according to a certain random probability, and the random probability is calculated as shown in Equation (15):

$$P_{ij}=\frac{M_{ij}\left(t\right)\times {\left(c_{ij}\right)}^{-1}}{\sum _{k\in allow}{M}_{ik}\left(t\right)\times {\left(c_{ik}\right)}^{-1}}$$

(15)

where $P_{ij}$ denotes the probability of selecting location $j$ based on the current location $i$, and $allow$ denotes the set of locations that have not yet been visited after removing location $j$.

Algorithm 4 is the simulation process of the TA-MCTS method. First, we determined whether node $v^{\prime }$ satisfied the termination state, and if it did, we computed $C$ directly according to Equation (8). Otherwise, we obtained the set of nodes $NV$ (Line 4) that had not been visited. We iterated through all the $NV$ nodes, computed each node’s current probability according to Equation (15), and stored it in the list of $probabilities$ (Lines 6–12). Subsequently, the next possible location $j$ was obtained based on the generated random number $rn$ and the list of $probabilities$. A new node was set up based on the current state, and the matrix $M$ was updated by Equation (14) (Lines 13–17).

Algorithm 4: Simulation Optimization Strategy
1:	function $Simulation\left(v\right)$
2:	set $v^{\prime }=v$
3:	while state of $v^{\prime }$ satisfy ${p_i<P}_{max}$ do
4:	get the set of not visited position $NV=\left\{\left(S\backslash Vs\right)\cup R\right\}$
5:	$\mathrm{define} \mathrm{a} \mathrm{list} probabilities$
6:	for each$j$$\mathrm{in} NV$ do
7:	${xe}_j^i$ is obtained by Equation (13)
8:	if ${xe}_j^i>0$ then
9:	$\mathrm{calculate} P_{ij}$ by Equation (15)
10:	$probabilities\left[j\right]=P_{ij}$
11:	end if
12:	end for
13:	$\mathrm{generate} \mathrm{a} \mathrm{random} \mathrm{number} rn$$\mathrm{in} \mathrm{the} \mathrm{range} \left[\mathrm{0,1}\right)$
14:	$\mathrm{get} j$$\mathrm{according} \mathrm{to} rn$$\mathrm{and} probabilities$
15:	set state and position of the new node $v^{\prime }$
16:	$Vs\leftarrow Vs\cup \left\{x_j^i\right\}$$, x_j^i$$\mathrm{is} \mathrm{the} \mathrm{position} \mathrm{of} \mathrm{node} v^{\prime }$
17:	update matrix $M$ by Equation (14)
18:	end while
19:	$\mathrm{get} \mathrm{the} \mathrm{value} \mathrm{of} C$ according to Equation (8)
20:	return $C$

In summary, based on the characteristics of multi-task assignment, this paper optimized the MCTS simulation phase. The results of multiple MCTS simulations were utilized to create an information repository, which guided the selection of actions during the simulation phase.

4. Experiments

In order to validate the TA-MCTS method, experiments and comparisons were performed. The experimental running environment was a 2.30 GHz CPU, 16.0 GB of RAM, and a 64-bit operating system.

4.1. Experimental Environment

The experimental area of this experiment was part of Hong Kong, as shown in Figure 6. In this paper, three square areas were selected in this region to simulate the experiment.

Table 2. Positions defined in different scenarios.

Scenarios	Area Size (km2)	Latitude and Longitude in the Lower-Left Corner	Relative Location of DC (km)	Relative Location of UAV Docks (km)	Number of Tasks
Scenario A	$\left(1\times 1\times 1\right)$	114.16885° E 22.27325° N	(0, 0)	(0.5, 0.5)	27
Scenario B	$\left(1.5\times 1.5\times 1.5\right)$	114.12077° E 22.35433° N	(0.75, 0.75)	(0.75, 1.25) (0.25, 0.25) (1.25, 0.25)	64
Scenario C	$\left(2\times 2\times 2\right)$	114.16395° E 22.30251° N	(1,1)	(0.6, 0.6) (1.3, 0.6) (0.6, 1.3) (1.3, 1.3)	125

shows the size and the latitude and longitude of the lower left corner of each scenario selected in this paper. Based on the location of the lower-left corner, this paper defines the local coordinates of the DC and UAV docks concerning the lower-left corner of the scenario. Moreover, a specific number of task locations were randomly generated in the region. With the expansion of scenarios and the increased number of tasks, UAVs need more energy to complete them all. Therefore, broadening the scenario, this paper establishes additional UAV docks. The increase in the number of tasks and UAV docks directly adds to the uncertainty of task allocation, which can more effectively evaluate the applicability and effectiveness of the TA-MCTS method in different environments.

Combining the above settings, the locations of DC, UAV docks, and randomly generated tasks for the three scenarios are shown in Figure 7. The red point represents the DC, the blue point represents the UAV dock, and the green point represents the task location.

Scenarios A, B, and C cover small to large areas where the number of UAVs involved and their flight capabilities differ.

Table 3. UAV parameters in different scenarios.

Parameters	Scenario A	Scenario B	Scenario C
Number of UAV docks	1	3	4
$m$	3	4	5
$E_{max}$	400 Wh	500 Wh	600 Wh
$E$	80 Wh/km	100 Wh/km	120 Wh/km
$∆$	30 Wh/km	50 Wh/km	50 Wh/km
$P_{max}$	3	4	5

shows the parameter settings of the UAVs in the three scenarios, with the number of UAVs as $m$, the maximum battery capacity as $E_{max}$, the maximum number of items to be carried as $P_{max}$, $E$ represents the energy consumed by the UAVs per kilometer when no items are loaded, $\alpha$ is the number of loaded items, and $∆$ denotes the energy consumed by each item. In order to better conform to actual conditions, as the payload capacity of UAVs increases, the required battery capacity and the energy consumption per kilometer also increase accordingly. Moreover, variations in the energy parameters of UAVs directly affect the frequency of their visits to UAV docks.

4.2. Task Grouping Comparison

In this paper, we first experimented with the task grouping strategy. Task grouping aims to cluster tasks according to their geographic location, narrowing the search range of task assignments for UAVs and improving the efficiency and accuracy of the assignment. Unlike the traditional K-means, the UD-Kmeans proposed in this paper ensured that the number of tasks within each grouping was consistent, thus creating favorable conditions for subsequent task allocation.

In order to verify the effectiveness of the UD-Kmeans method, this paper conducted group experiments for all tasks within the three scenarios. Based on the maximum load of UAVs and the number of UAVs, this paper determined that the number of groups in scenarios A, B, and C was 3, 4, and 5, respectively, according to Equation (9). The experimental results are shown in Figure 8 and

Table 4. Comparison of K-Means and UD-Kmeans.

Scenario	Group	K-Means	UD-Kmeans
Scenario A	Group1	9	9
	Group2	8	9
	Group3	10	9
Scenario B	Group1	12	16
	Group2	20	16
	Group3	15	16
	Group4	17	16
Scenario C	Group1	23	25
	Group2	25	25
	Group3	27	25
	Group4	33	25
	Group5	17	25

. Figure 8 indicates the grouping results with different colored dots, and Table 4 directly shows the number of elements in each group. UD-Kmeans made the number of tasks in each grouping result consistent. However, the K-means grouping results indicate inconsistent tasks in each group. Uneven grouping results may lead to difficulties in subsequent task assignments. Therefore, UD-Kmeans contributes to the stability and reliability of subsequent multi-task allocations.

4.3. MCTS Comparison

In order to validate the TA-MCTS, we conducted randomized experiments of TA-MCTS with R-MCTS, G-MCTS, and RG-MCTS in three scenarios. For RG-MCTS, the weights of random and greedy simulation strategies were set to 50%.

Compared with the original MCTS method, the TA-MCTS optimized the node selection and simulation strategies. Generally, MCTS builds a separate tree for node selection and simulation. However, in the case of task allocation considering multi-UAV and energy, a separate tree may result in too many nodes and a slow search. Therefore, this paper modified the three MCTS methods mentioned above to use multiple trees.

Due to the randomness of the methods, a single experiment could not fully reflect the differences between different methods. In order to achieve a more precise comparison, this paper executed 50, 100, 150, 200, 1000, and 2000 simulations for each method in each scenario and then analyzed the results comprehensively.

Figure 9 shows the average energy consumption of different algorithms for multiple experiments in different scenarios. Figure 9a–c represent the average energy consumption under scenarios A, B, and C, respectively. The difference between the four methods was relatively slight due to the small area covered by Scenario A. Figure 9b,c show that TA-MCTS consumed less energy on average than the other three methods, regardless of the number of experiments. Moreover, the results obtained by the TA-MCTS method were more stable, while the other three methods were subject to fluctuations.

In order to show the effect of each method more clearly, this paper used violin plots to show the characteristics of the data distribution of energy consumption for 2000 experiments; the results are shown in Figure 10, Figure 11 and Figure 12. As seen in Figure 10, due to the small area of Scenario A, the UAVs carrying energy could cover almost the whole area, so the difference in the data distribution obtained by the four methods was small. Figure 11 and Figure 12 show that in Scenarios B and C, the difference in the distribution of energy consumption among the four methods was more evident due to the expansion of the area. In Scenario B, the data concentration of TA-MCTS is better than the remaining three methods. The optimal and worst energy consumption of TA-MCTS was also better than the other three methods. In Scenario C, TA-MCTS indicated significantly lower energy consumption in the concentrated area than RG-MCTS and G-MCTS. R-MCTS performed less well than the other algorithms, and the distribution of results was relatively scattered due to its intrinsic randomness.

Figure 13a–c show the iterative process for a sub-task group in scenarios A, B, and C, respectively. In the three scenarios, the number of iterations required to converge was approximately the same for all four methods. However, the TA-MCTS method proposed in this paper obtained better results after a similar number of iterations, proving the superiority of the TA-MCTS method.

To further illustrate the impact of different data in Table 2 and Table 3 on the results, the average number of visits to UAV docks in different scenarios was counted in this paper, as shown in

Table 5. Average number of passes through the UAV docks in scenario A.

Number of Experiments	TA-MCTS	G-MCTS	R-MCTS	RG-MCTS
50	0.18	0.14	0.36	0.22
100	0.17	0.2	0.28	0.26
150	0.26	0.17	0.15	0.22
200	0.19	0.17	0.18	0.18
1000	0.22	0.16	0.21	0.25
2000	0.21	0.16	0.21	0.24

,

Table 6. Average number of passes through the UAV docks in scenario B.

Number of Experiments	TA-MCTS	G-MCTS	R-MCTS	RG-MCTS
50	0.42	1.01	0.56	0.62
100	0.46	1.02	0.64	0.64
150	0.44	1.01	0.49	0.57
200	0.49	0.96	0.58	0.57
1000	0.47	1.01	0.52	0.66
2000	0.44	0.99	0.55	0.67

and

Table 7. Average number of passes through the UAV docks in scenario C.

Number of Experiments	TA-MCTS	G-MCTS	R-MCTS	RG-MCTS
50	15.54	17.44	19.66	15.84
100	16.19	18.16	19.17	17.68
150	16.34	18.59	19.61	16.78
200	16.44	18.14	19.56	16.78
1000	16.49	18.63	19.36	16.91
2000	16.33	18.64	19.29	16.73

. In the descriptions in Table 2 and Table 3, Scenarios A and B, the smaller area and the small number of missions undertaken by the UAVs resulted in the UAVs not needing to visit the UAV dock as frequently to change batteries. Therefore, the four methods show fewer passes through the docks, but overall, the TA-MCTS led to fewer passes through the UAV docks than other methods. Combining the descriptions of Scenario C in Table 2 and Table 3, the increase in the number of missions and the size of the area resulted in the UAVs needing to visit the UAV docks more times to replenish their energy to complete the missions. Therefore, the data in Table 7 show that the average number of times the TA-MCTS method passes through the docks was better than the other algorithms.

Figure 14 illustrates the task allocation of TA-MCTS in three scenarios. Figure 14a–c represent the results under scenarios A, B, and C, respectively. The dotted lines of different colors represent different task groups, and the green dots represent the task locations.

After the above comparative analysis, the TA-MCTS proposed in this paper was shown to have significant advantages, and it can reasonably accomplish multi-UAV task allocation. TA-MCTS ensured the UAVs minimized energy consumption in accomplishing all the tasks, regardless of the scenario. These experiments highlight the innovation and effectiveness of the TA-MCTS method and provide a powerful solution for multi-UAV task allocation in urban logistics with practical applications.

5. Conclusions

This paper investigated the multi-task assignment method for UAV logistics in urban low-altitude environments. Based on the traditional K-means, this paper proposes UD-Kmeans to realize multi-task grouping and reduce the scope of multi-task allocation. On this basis, we proposed TA-MCTS by improving the selection and simulation strategies of MCTS. We utilized multiple collaborative trees for decision evaluation to effectively accomplish multi-task allocation. From Table 4 and Figure 8, UD-Kmeans exhibited stability in task grouping, ensuring the number of tasks within each grouping was consistent. The advantages of TA-MCTS were illustrated through comprehensive comparisons with other related MCTS methods. Table 5, Table 6 and Table 7 and Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 showed that TA-MCTS efficiently performs in solution optimality. The main findings are as follows:

(1) UD-Kmeans ensures that the number of tasks in each group is balanced while performing clustering, effectively avoiding uneven task allocation.

(2) In the TA-MCTS method, using SEOS, conflict-free node selection is achieved in the selection phase. In the expansion phase, selective expansion is performed according to the UAV’s energy situation to reduce the number of possible subsequent ineffective searches.

(3) TA-MCTS optimizes the simulation strategy by collecting the simulation results of multiple UAVs and constructing a historical information repository. The historical information is used to guide the subsequent simulation, thus improving the effectiveness of the simulation.

In subsequent research, we plan to optimize these methods further and explore the implementation of task allocation among multiple scenarios. Moreover, with the development of UAV technology, we plan to conduct more extensive research in the real world with real UAV performance situations.