Efficient Task Allocation in Multi-Agent Systems Using Reinforcement Learning and Genetic Algorithm

Abstract

The multi-agent task allocation problem has attracted substantial research interest due to the increasing demand for effective solutions in large-scale and dynamic environments. Despite advancements, traditional algorithms often fall short in optimizing efficiency and adaptability within complex scenarios. To address this shortcoming, we propose a genetic algorithm-enhanced PPO (GAPPO) algorithm, specifically developed to enhance decision-making in challenging task allocation contexts. GAPPO employs a deep reinforcement learning framework, enabling each agent to independently evaluate its surroundings, manage energy resources, and adaptively adjust task allocations in response to evolving conditions. Through iterative refinement, GAPPO achieves balanced task distribution and minimizes energy consumption across diverse configurations. Comprehensive simulations demonstrate that GAPPO consistently outperforms traditional algorithms, resulting in reduced task completion time and heightened energy efficiency. Our findings underscore GAPPO’s potential as a robust solution for real-time multi-agent task allocation.

1. Introduction

The multi-agent task allocation problem involves distributing tasks among multiple agents to optimize resource utilization and minimize operational costs [1,2]. Effective coordination among agents is essential to enhance performance across a range of applications [3,4]. Effective task allocation is crucial in these contexts, as it directly impacts the system’s overall performance, efficiency, and success [5]. The complexity of this problem lies in balancing task requirements, agent capabilities, and environmental constraints, each of which can vary significantly [6]. Developing a robust allocation scheme is crucial to avoid resource wastage and to ensure tasks are assigned to the most suitable agents. As the scale of agents and tasks increases, computational and logistical demands rise, emphasizing the need for adaptive and efficient solutions [7]. In recent years, multi-agent systems have shown significant potential in applications requiring rapid deployment and dynamic task allocation, such as agriculture [8,9], intelligent transportation [10,11], and disaster response—including rescue [12], search [13], tracking operations [14,15], etc. [16].

Existing algorithms for multi-agent task allocation primarily fall into centralized and distributed frameworks. Centralized algorithms leverage global information for optimization but struggle with scalability and adaptability in dynamic environments. In contrast, distributed algorithms enhance flexibility by allowing agents to make decisions based on local observations, though they often result in suboptimal allocations due to limited coordination. Reinforcement learning (RL) has gained attention as an alternative [17,18], enabling agents to learn allocation strategies through interaction with the environment. However, conventional RL algorithms often suffer from slow convergence and instability, particularly in large-scale and dynamic settings [19,20].

The multi-agent task allocation problem considered in this work is computationally challenging due to its combinatorial nature. The problem requires determining an optimal task allocation policy while considering constraints such as agent capabilities, dynamic task demands, and communication limitations. Similar combinatorial optimization problems, such as the multi-agent traveling salesman problem (mTSP) and job-shop scheduling, have been shown to be NP-hard [21]. Given the exponential growth of the solution space as the number of agents and tasks increases, solving this problem optimally in real time becomes computationally infeasible with traditional optimization techniques.

To overcome those limitations, we propose a novel algorithm that integrates genetic algorithms (GAs) with proximal policy optimization (PPO) to enhance task allocation efficiency in multi-agent systems. The GA facilitates exploration through selection, crossover, and mutation, improving policy diversity and search effectiveness. Meanwhile, PPO optimizes decision-making by leveraging real-time feedback, enabling agents to dynamically adjust allocations. This hybrid algorithm enhances adaptability and learning efficiency in dynamic task allocation scenarios. By combining the GA’s exploratory strengths with PPO’s optimization capabilities, our algorithm addresses the challenges posed by traditional RL algorithms and provides a more scalable solution to the task allocation problem.

The main contributions of this paper are as follows.

• We model the multi-agent task allocation problem as a Markov game, where each agent functions as an independent agent, interacting continuously with its environment to iteratively improve its task allocation. Under this framework, each agent determines its actions based on local observations, facilitating coordinated and efficient task distribution across the swarm.
• We propose a GA-PPO reinforcement learning algorithm, which combines genetic algorithms with proximal policy optimization, incorporating an attention mechanism and adaptive learning rate. This algorithm empowers each agent to autonomously adapt to changing task demands, improving learning efficiency and facilitating effective coordination with multi-agent environments. By leveraging these features, the system optimizes resource utilization while maintaining robust performance in dynamic scenarios.
• Numerical experiments demonstrate that our proposed algorithm performs well in terms of convergence speed and scalability compared with the existing representative algorithms.

This paper is organized as follows. Section 2 reviews related work. Section 3 describes the problem formulation. Section 4 presents the GAPPO algorithm and provides its detailed process. Section 5 discusses the performance of the algorithm in different environments. Section 6 presents the summary and future directions. The main notations of this paper are listed in

Table 1. Notation used in this paper.

Notation	Definitions
$\hspace{1em} n\left(t\right)$	the number of agents at time t
m	the number of tasks
N	the set of $n\left(t\right)$ agents
S	the set of m tasks
$N_j$	the number of agents participating in task j
${\left\{S\right\}}_i$	the task that agent i chooses
${\left\{I\right\}}_j$	the set of agents or the group for task j
${\omega }_i$	the work ability of agent i
$h_j$	the workload of task j
$T_j$	the completion time of task j
T	the max completion time of all tasks
$A_i$	the strategy set of agent i
A	the strategy space
$\tau$	the discount factor reflecting the rate of task reward change
$Ra{d}_i$	the communication radius of agent i j
$R_j$	the reward of task j
$E_i$	the battery level of the agent i
$v_i$	the speed of the agent i
$\theta$	the elevation angle of the agent

.

2. Related Work

2.1. Centralized Optimization Algorithms

Many traditional algorithms rely on centralized optimization algorithms, leveraging global information to optimize task allocation. These algorithms often employ mixed-integer programming [22] or centralized RL models [23], achieving high-quality solutions under well-structured environments.

For instance, Shabanighazikelayeh et al. [24] proposed a centralized algorithm for UAV placement under high-altitude constraints, ensuring efficient coverage and communication. Similarly, Consul et al. [25] applied a hybrid federated RL framework in a UAV-assisted mobile edge computing (MEC) network, improving computation offloading and resource allocation. These algorithms demonstrate that centralized optimization achieves high efficiency in controlled settings with predictable task demands.

However, centralized algorithms face scalability and adaptability challenges. Wang et al. [26] highlighted that centralized algorithms optimize path planning effectively but struggle with rapid environmental changes, necessitating frequent re-optimization. Al-Hussaini et al. [27] proposed an automated task reallocation system for multi-robot missions, yet the algorithm remains constrained by centralized control bottlenecks. Furthermore, centralized algorithms introduce single points of failure, increasing system vulnerability in large-scale and dynamic environments [28].

Comparison with GA-PPO: Unlike centralized algorithms that rely on a single controller, GA-PPO employs a decentralized framework, where genetic algorithms enhance exploration, and PPO dynamically refines policies. Compared to Shabanighazikelayeh et al.’s [24] UAV optimization algorithm, GA-PPO does not require predefined constraints, allowing greater adaptability to changing conditions. Furthermore, while Consul et al.’s [25] federated RL algorithm improves computational efficiency, it still relies on a central coordinator. GA-PPO eliminates this dependency, making it more robust to network failures and large-scale applications.

2.2. Distributed Optimization Algorithms

Distributed optimization algorithms decentralize decision-making, allowing agents to allocate tasks based on local information with limited communication. This structure enhances scalability, adaptability, and fault tolerance by enabling agents to act independently, without the need for global communication [29]. Ren et al. [30] demonstrated the effectiveness of distributed optimization in large-scale wireless sensor networks, where agents can allocate resources based on local observations, thus improving system efficiency. Luo et al. [31] applied similar principles to multi-robot systems, optimizing task allocation to increase operational efficiency while maintaining system scalability.

Despite these advantages, distributed optimization faces challenges in terms of coordination and convergence. Zhang et al. [32] highlighted inefficiencies arising from incomplete information, where agents’ local knowledge may lead to suboptimal allocations. Moreover, the absence of global coordination can cause slower convergence, as agents may not be able to adjust their strategies based on the broader system dynamics [33]. These limitations can result in inefficiencies, particularly in dynamic environments where global coordination is beneficial for maintaining optimal system performance.

Comparison with GA-PPO: GA-PPO overcomes many of these challenges through the integration of genetic algorithms for broad exploration and proximal policy optimization for continuous policy refinement. This combination allows GA-PPO to adapt dynamically to changing environments, ensuring both efficient exploration and targeted optimization over time. Unlike Ren et al.’s [30] distributed algorithm, which relies heavily on local communication between agents, GA-PPO does not require fixed communication patterns, providing more flexibility and robustness to network failures. Furthermore, while Luo et al.’s [31] work in multi-robot systems benefits from localized decision-making, GA-PPO’s hybrid framework allows it to perform effectively even in scenarios with high levels of environmental uncertainty and variable task demands, making it more adaptable and scalable in comparison.

2.3. Reinforcement Learning-Based Algorithms

Reinforcement learning (RL) has emerged as a powerful algorithm for multi-agent task allocation, allowing agents to adapt and optimize task distribution strategies through real-time interactions with dynamic environments. Its ability to manage complex, evolving scenarios and enhance decision-making processes makes it especially suitable for multi-agent systems [34,35].

For instance, Wu et al. proposed a multi-agent deep reinforcement learning algorithm for formation control, which integrates an attention mechanism and adaptive accuracy to enhance agent coordination and precision [5]. Similarly, Ning et al. developed a joint optimization algorithm for data acquisition and trajectory planning in UAV-assisted IoT networks. Their algorithm maximizes energy efficiency while adhering to mobility, safety, and task constraints, framing the problem as a constrained Markov decision process, and using multi-agent deep RL to optimize agent movement policies [36]. These algorithms demonstrate the potential of RL in optimizing agent behaviors for specific applications with controlled constraints.

Xu et al. introduced a federated deep RL algorithm for agent deployment and resource allocation in 6G networks, enabling agents to make real-time decisions based on local observations while pursuing a global optimal solution [37]. Their algorithm significantly improves network throughput and convergence, as demonstrated in simulations. Moreover, Dai et al. developed a multi-agent collaborative RL algorithm for agent resource allocation in wireless networks, focusing on optimizing interference management and network capacity. This framework integrates federated learning to facilitate data sharing and cooperation among agents, outperforming traditional RL algorithms in simulation [23].

Comparison with GA-PPO: Unlike the aforementioned RL-based algorithms that rely on centralized or federated frameworks, GA-PPO employs a decentralized algorithm, offering improved scalability and robustness. While Wu et al.’s [5] algorithm enhances coordination through an attention mechanism, GA-PPO’s use of genetic algorithms for exploration and proximal policy optimization (PPO) for dynamic policy refinement provides greater flexibility and adaptability. This allows GA-PPO to adjust in real time without the need for centralized control. In comparison to Ning et al.’s [36] constrained Markov decision process, which is limited to specific task constraints, GA-PPO does not require predefined constraints, enabling it to operate across a wider range of environments. Furthermore, while Xu et al.’s [37] federated RL algorithm relies on local coordination between agents, GA-PPO eliminates the need for central coordination, making it more robust to network failures. This decentralized nature makes GA-PPO particularly well suited for large-scale applications, where adaptability and resilience are critical.

3. Problem Formulation

In this paper, we address the multi-agent task allocation problem, where agents are assigned to distinct tasks and operate either independently or cooperatively to enhance task efficiency and coverage across the environment.

More specifically, agents have the following two tasks.

• Primary (Extrinsic) Task: High-quality execution of multiple operational tasks by agents, such as conducting surveillance on designated targets or performing search and rescue missions in critical scenarios. Those tasks necessitate that agents autonomously assess their environments and adapt to dynamic conditions, thereby ensuring effective and efficient performance.
• Secondary (Intrinsic) Task: Evaluation of task allocation decisions. This task focuses on evaluating the effectiveness of agents’ decisions in optimizing resource utilization and enhancing operational efficiency, thereby improving the overall performance of the multi-agent system.

To achieve these objectives, which include the high-quality execution of operational tasks and the evaluation of task allocation strategies, each agent undertakes the following steps: state estimation, task allocation (for scenarios involving multiple tasks), and strategy, as illustrated in Figure 1 and elaborated upon in the subsequent sections.

• State Estimation: Each agent’s state includes its position, an occupancy map, and target identifiers. At time t, the state $S_{ti}$ reflects its configuration and location, updated in real time. The transition function $f(S_{ti},a_i)$ models state evolution. The Actor–Critic network processes these data to optimize policies, enabling efficient navigation, obstacle avoidance, and target acquisition in dynamic environments.
• Task Allocation: A dynamic algorithm optimizes task distribution, preventing clustering. Agents evaluate tasks based on proximity, urgency, and workload, considering completion time and expected benefits. They share status updates for real-time reallocation, ensuring balanced task coverage and efficient resource use.
• Strategy: Agents optimize detection and mapping based on state estimates. Initially, they update policies independently via rewards. In collaborative settings, they share policy gradients to refine strategies. GAPPO, with attention mechanisms, enables autonomous and cooperative decision-making for better coordination.

3.1. System Model

Suppose there exist a set of agents formed by $N=\{1,2,\dots ,n\}$ and a set of tasks formed by $S=\{1,2,\dots ,m\}$. An agent cannot participate in multiple tasks simultaneously. The task allocation problem in this multi-agent system can be formulated by defining the strategy space for each agent. Let ${\left\{S\right\}}_i$ represent the task selected by agent $i\in N$, and let ${\left\{I\right\}}_j$ represent the set of agents allocated to task $j\in S$. Each task j is positioned at a fixed location ${pos}_j=(x_j,y_j,z_j)$, defined by its three-dimensional coordinates. The position of agent i, ${pos}_i$, coincides with the position of the task it selects. The Euclidean distance between agent i and task j is expressed as

$$d_{ij}=\parallel {\mathrm{pos}}_i-{\mathrm{pos}}_j\parallel .$$

(1)

Here, $d_{ij}$ also represents a constraint on the communication range of the agents. Specifically, an agent can only access tasks or interact with other agents that are within its communication range, as determined by $d_{ij}$.

The workload of task j, represented as $h_j\ge 0$, represents the total effort required to complete the task through the combined efforts of the agents. The work ability of agent i is represented by ${\omega }_i$, indicating the amount of work that agent i can execute per unit time. For task j, the work ability of the group allocated to the task is defined as the sum of the work capacities of all agents in that group ${\sum }_{i\in I_j}{\omega }_i$. To complete task j, the total work ability of the allocated group must satisfy

$${\sum }_{i\in {\left\{I\right\}}_j}{\omega }_i-h_j\ge 0.$$

(2)

For each task $j\in S$, it is crucial to allocate an appropriate number of agents to satisfy (2) and thereby minimize resource wastage due to inefficient allocations. Thus, the reward for task j is expressed as

$$R_j=-{\tau }^{{\sum }_{i\in {\left\{I\right\}}_j}{\omega }_i-h_j},$$

(3)

where discount factor $\tau \in (0,1)$ represents the rate of change in reward based on the work ability of the group ${\left\{I\right\}}_j$. It also indicates the timeliness of the task. When $\tau$ approaches one, the reward continues to increase even if (2) is satisfied, encouraging more agents to participate and thereby speeding up task completion. Conversely, when $\tau$ approaches 0, the reward $R_j$ becomes insensitive to the (2), meaning no additional agents would gain rewards from the task once the condition is satisfied.

The objective of the task allocation problem is to identify an optimal allocation that minimizes the total completion time $T=max\{T_1,T_2,\dots ,T_m\}$, where the completion time for task j is expressed as

$$T_j={\displaystyle \frac{h_j}{{\sum }_{i\in {\left\{I\right\}}_j{\omega }_i}}}.$$

(4)

Therefore, the problem can be formulated as follows:

$$\begin{array}{cc}\hfill & \underset{\left\{e_{ij}\right\}}{min}\left\{T_j\right\},\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\sum }_{j=1}^m{e}_{ij}\le 1,\forall i\in N,\hfill \\ \hfill & e_{ij}\in \{0,1\},\forall i\in N,\forall j\in S,\hfill \end{array}$$

(5)

where $e_{ij}$ is a binary decision variable, if $e_{ij}=1$ indicates that agent i is allocated to task j; otherwise, agent i is not allocated to task j.

3.2. State Action Model

In our agent swarm framework, each agent’s transition at time t is represented by the tuple $(s_t,a_t,r_t,s_{t+1})$, where the following hold:

• $s_t$ denotes the agent’s current state, which is a composite representation consisting of the following:

  –

  The agent’s location ${\mathrm{pos}}_i=(x_i,y_i,z_i)$, providing the agent’s spatial position in the environment. This information is essential for determining the agent’s proximity to tasks and other agents, which influences both task allocation and collision avoidance.

  –

  Task-related information, including task priorities, workload status, and the agent’s assigned task. This information allows the agent to evaluate its current workload and adjust its decisions based on the urgency and importance of tasks within its scope.

  –

  The occupancy map, encoding the spatial distribution of tasks and agents within the agent’s communication or sensing range. This map helps the agent understand its environment by providing real-time updates on the locations of other agents and tasks, enabling more informed decisions on task allocation and collaboration with other agents.

• $a_t$ is the action selected by the agent in state $s_t$. The action is defined as the selection of a task $j\in S$, where S represents the set of available tasks. Formally,

  $$a_t=arg\underset{j\in \mathcal{C}\left(i\right)}{max}\pi (s_t,j),$$

  (6)

  where $\pi (s_t,j)$ is the policy function that outputs the probability of selecting task j given the current state $s_t$, and $\mathcal{C}\left(i\right)$ is the set of tasks within the communication or sensing range of agent i. If no tasks are feasible, the agent may choose an idle action.
  The policy function $\pi (s_t,j)$ is learned using the Actor–Critic architecture, where the Actor is responsible for selecting actions based on the current state, while the Critic evaluates the actions taken by estimating the value function. Specifically, the Actor network takes the current state $s_t$ as input and outputs a probability distribution over possible actions (tasks). The policy function $\pi (s_t,j)$ is expressed as

  $$\pi (s_t,j)=P(a_t=j\mid s_t),$$

  (7)

  where $a_t$ is the action selected by the agent at time step t, and j is a task from the set of available tasks $\mathcal{C}\left(i\right)$. The Critic, on the other hand, estimates the expected cumulative reward for the agent starting from state $s_t$. This is achieved through the value function $V\left(s_t\right)$, which represents the expected long-term reward given the current state. It is formally expressed as

  $$V\left(s_t\right)=\mathbb{E}\left[\sum _{k=t}^T{\gamma }^{k-t}{r}_k\mid s_t\right],$$

  (8)

  where $r_k$ is the reward received at time step k, and $\gamma$ is the discount factor that determines the importance of future rewards.
  At each time step, the agent selects an action $a_t$ based on the policy $\pi (s_t,j)$ by maximizing the probability of selecting a task from the set $\mathcal{C}\left(i\right)$ based on the current state $s_t$. The interaction between the Actor and Critic helps in continuously optimizing the policy, with the Critic providing feedback on the expected rewards, guiding the Actor to improve its decision-making over time.
  If no feasible tasks are within the agent’s sensing or communication range, the agent may choose an idle action, which corresponds to not selecting any task. This idle action is implicitly represented within the action space, where one of the actions is designated as “idle”.
• $r_t$ represents the reward obtained after executing action $a_t$. This reward reflects the agent’s contribution to task completion, considering factors such as task importance, completion effectiveness, and collaborative efficiency with other agents.
• $s_{t+1}$ is the subsequent state resulting from the agent’s interaction with the environment after performing $a_t$.

This process encapsulates the agents’ decision-making as they interact with the environment, where each agent optimizes its actions to maximize cumulative rewards using a combination of Actor–Critic and genetic algorithms.

In the proximal policy optimization (PPO) framework, the advantage function $A(s_t,a_t)$ evaluates the relative benefit of selecting action $a_t$ in state $s_t$. PPO employs Generalized Advantage Estimation (GAE) to compute the advantage. The advantage function is given by

$$A(s_t,a_t)={\widehat{A}}_t={\sum }_{i=t}^T{\left(\gamma \lambda \right)}^{i-t}{\delta }_i,$$

(9)

where $\gamma$ is the discount factor, which controls the importance of future rewards relative to immediate rewards. It is a value between 0 and 1. $\lambda$ is the smoothing parameter, which helps balance the bias-variance trade-off in estimating the advantage. It is also between 0 and 1. ${\delta }_i$ is the temporal difference error (TD-error), which is expressed as

$${\delta }_i=r_i+\gamma V(s_{i+1};\omega )-V(s_i;\omega ),$$

(10)

where $\gamma$ is the discount factor. The value function $V(s_i;\omega )$ is updated iteratively as

$$V(s_i;\omega )\leftarrow V(s_i;\omega )+\alpha {\delta }_i,$$

(11)

where $\alpha$ is the learning rate. This update process ensures that each agent effectively optimizes its cumulative rewards, leading to more precise task execution and resource allocation. The initial value of $V(s_i;\omega )$ is typically set to zero.

Moreover, the GA enhances policy optimization across the agent swarm by selecting and evolving the most effective action strategies. Integrating the GA with the Actor–Critic framework inherent in the PPO algorithm achieves a refined balance between exploration and exploitation. Consequently, this framework significantly improves adaptability in dynamic task environments. The synergy between the GA and the Actor–Critic mechanism of PPO not only fosters effective strategy development but also contributes to the overall robustness and efficiency of the agent swarm in complex operational scenarios.

4. A Learning Algorithm

There exist various optimization algorithms for addressing multi-agent task allocation problems, including the leader–follower algorithm [38], the virtual structure algorithm [39], and behavior-based control algorithm [40]. In addition, reinforcement learning has demonstrated effectiveness in this area. To address the challenges associated with dynamic task allocation among agents, this paper presents a novel algorithm, the genetic algorithm-enhanced proximal policy optimization (GA-PPO) algorithm, the aim of which is to enhance decision-making and coordination among agents in dynamic environments.

4.1. Genetic Algorithm-Enhanced Proximal Policy Optimization

This algorithm leverages the efficiency of PPO for learning optimal policies while utilizing the evolutionary principles of the GA to select the best-performing agents, thereby enhancing decision-making in dynamic environments.

The algorithm begins by initializing a population P of Actor–Critic networks, with each individual representing a distinct policy tailored for agent operations. Several key hyperparameters guide the learning process, including the learning rate lr, discount factor $\gamma$, clip ratio $ϵ$, and KL divergence target ${\mathrm{KL}}_{\mathrm{target}}$. Those values are selected based on theoretical foundations and established practices in reinforcement learning and evolutionary algorithms, ensuring a stable and efficient learning process. The specific values for these hyperparameters are chosen to align with common standards in the literature, which have been shown to perform well in similar multi-agent systems tasks.

A replay buffer with a capacity of C = 10,000 is created to store observed state–action–reward sequences. This buffer facilitates experience sampling during training. Additionally, the genetic algorithm component employs a set of parameters, including mutation rates and fitness weights ($\alpha$, $\beta$, and $\gamma$), which govern the selection process and population evolution.

The learning rate lr determines the step size during policy updates, with a typical value in PPO implementations chosen to ensure stable convergence. The discount factor $\gamma$ reflects the importance of future rewards, and it is set to a commonly used value for long-term decision-making tasks. The clip ratio $ϵ$ is used to prevent excessive updates during the training process, ensuring stability while maintaining exploration. The KL divergence target ${\mathrm{KL}}_{\mathrm{target}}$ helps control the divergence between old and new policies, promoting gradual updates to the policy.

The genetic algorithm parameters, such as mutation rates and fitness weights, are designed to balance reward maximization, exploration, and resource efficiency. Those parameters are selected based on general practices in evolutionary algorithms, adapted for the task at hand. The fitness score of each individual is computed based on cumulative reward, policy entropy, and computational resource cost, with appropriate weights ($\alpha$, $\beta$, and $\gamma$) to guide the optimization process.

The fitness score of each individual is computed based on cumulative reward, policy entropy, and computational resource cost, with appropriate weights ($\alpha$, $\beta$, and $\gamma$) to guide the optimization process.

The selection of hyperparameters follows established guidelines in reinforcement learning and evolutionary algorithms. However, the exact values used may not be optimal across all scenarios. Specifically, the performance of the algorithm could benefit from fine-tuning the parameters to better adapt to varying environments and tasks. While the current work does not include a comprehensive optimization of these hyperparameters, future research could involve systematically varying them to assess their influence on the performance of GAPPO.

During each iteration, the algorithm collects trajectories for each individual $P_i$ with the population. Each agent observes the current state $s_t$ and selects an action $a_t$ based on its policy $P_i$. The selected action is executed, resulting in a reward $r_t$ and a subsequent state $s_{t+1}$. The transition $(s_t,a_t,r_t,s_{t+1})$ is appended to the trajectory T, and this process continues until a terminal state is reached. The accumulated trajectories are stored in the replay buffer, allowing the PPO component to sample batches for optimization.

Once the replay buffer contains a sufficient number of samples, the strategy update phase commences. A batch of transitions is sampled, and for each transition, the algorithm computes the returns $R_t$ using the discount factor $\gamma$. The advantages $\widehat{A_t}$ are calculated utilizing generalized advantage estimation (GAE). The action probabilities $\pi \left(a_t\right|s_t)$ and state values $V\left(s_t\right)$ are obtained to compute the surrogate loss $L\left(\theta \right)$, as previously defined. This surrogate loss plays a crucial role in the GAPPO algorithm by ensuring that policy updates remain stable while allowing for effective exploration. $P_i$ is evaluated in the environment to obtain fitness score $F_i$ using

$$F_i=\alpha ·{\displaystyle \frac{R_i}{R_{max}}}+\beta ·{\displaystyle \frac{H_i}{H_{max}}}-\gamma ·{\displaystyle \frac{E_i}{E_{max}}},$$

(12)

where $R_i$ is the cumulative reward obtained by the individual $P_i$, $H_i$ is the policy entropy encouraging exploration, and $E_i$ is the computational resource cost. The factors $\alpha ,\beta ,\mathrm{and}\gamma$ are weights balancing these components.

To ensure the stability of the training process, PPO employs a clipped objective function for policy updates, which is expressed as

$$\begin{array}{cc}\hfill & L_1={\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)}}\widehat{A_t},\hfill \\ \hfill & L_2=\mathrm{clip}\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)}},1-ϵ,1+ϵ\right)\widehat{A_t},\hfill \\ \hfill & L\left(\theta \right)=min(L_1,L_2),\hfill \end{array}$$

(13)

where $r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)}}$ denotes the ratio of the new and old policies, and $ϵ$ controls the permissible range of this ratio. The clip function clip(·) is expressed as

$$\mathrm{clip}(x,1-ϵ,1+ϵ)=\left\{\begin{array}{cc}1-ϵ\hfill & \mathrm{if}x<1-ϵ\hfill \\ x\hfill & \mathrm{if}1-ϵ\le x\le 1+ϵ\hfill \\ 1+ϵ\hfill & \mathrm{if}x>1+ϵ\hfill \end{array}\right..$$

(14)

This objective function (14) restricts the magnitude of policy updates, thereby preventing significant shifts in the policy and enhancing training stability. Furthermore, incorporating a GA can optimize policy exploration. In each generation, the GA evaluates the fitness of multiple strategies, selecting the best-performing individuals for crossover and mutation, which generates a new set of strategies. This algorithm effectively expands the policy space and enhances the diversity and global optimality of the search.

Additionally, the value loss $L_V$ is calculated as

$$L_V={\displaystyle \frac{1}{2}}{(R_t-V\left(s_t\right))}^2.$$

(15)

The total loss L is computed by combining the surrogate loss and the value loss, and the network parameters given as $\theta$ are updated through gradient descent.

As the generations progress, our proposed GAPPO algorithm evaluates each individual in the population to obtain fitness scores $F_i$. Based on those scores, a tournament selection strategy is employed to choose individuals for reproduction. A new population $P^{\prime }$ is generated through crossover and mutation, and the original population P is updated accordingly. This evolutionary process continues until the maximum number of generations is reached.

The enhanced PPO agent with the genetic algorithm enhances task allocation in multi-agent systems by effectively balancing exploration and exploitation while ensuring diversity within the agent population. This algorithm demonstrates robustness and convergence, even in scenarios with varying numbers of agents. The operational flow of the algorithm is encapsulated in the procedure outlined in Algorithm 1.

Algorithm 1 Enhanced PPO agent with genetic algorithm

1: Initialize: Population of Actor–Critic networks $P=\{P_1,P_2,\dots ,P_N\}$  2: Set hyperparameters: learning rate $lr$, discount factor $\gamma$, clip ratio $ϵ$, and KL divergence target $K{L}_{target}$  3: Initialize replay buffer with capacity $C=10,000$  4: Set generation counter $g=0$  5: while training do  6:     for each individual $P_i\in P$ do  7:        Initialize trajectory T and observe initial state $s_0$  8:        for each time step $t=0$ to $T_{max}$ do  9:          Select action $a_t$ using policy $P_i$ based on $s_t$ 10:         Execute $a_t$, observe reward $r_t$ and next state $s_{t+1}$ 11:         Append transition $(s_t,a_t,r_t,s_{t+1})$ to trajectory T 12:         if terminal state reached then 13:            Break 14:         end if 15:       end for 16:       Store trajectory T in replay buffer 17:    end for 18:    if len(replay buffer) ≥ BATCH_SIZE then 19:       Sample a batch of transitions from replay buffer 20:       for each transition in the batch do 21:          Compute returns $R_t$ using $\gamma$ 22:          Compute advantages $\widehat{A_t}$ using Generalized Advantage Estimation (GAE) 23:          Get action probabilities $\pi \left(a_t\right|s_t)$ and state values $V\left(s_t\right)$ 24:          Calculate surrogate loss $L\left(\theta \right)$ from (13) 25:          Calculate value loss from (15) 26:          Compute total loss $L=L\left(\theta \right)+L_V$ 27:          Update network parameters $\theta$ using gradient descent 28:       end for 29:    end if 30: end while 31: while$g<\mathrm{max\_generations}$do 32:    for each individual $P_i\in P$ do 33:       Evaluate $P_i$ in the environment to obtain fitness score $F_i$ 34:    end for 35:    Select individuals based on fitness using a selection strategy (e.g., tournament selection) 36:    Generate new population $P^{\prime }=\{P_1^{\prime },P_2^{\prime },\dots ,P_N^{\prime }\}$ via crossover and mutation 37:    Update population $P\leftarrow P^{\prime }$ and increment generation counter g 38: end while 39: Periodically: Update target networks to stabilize training

This optimization framework leverages genetic operations—selection, mutation, and recombination—enabling agents to evolve strategies that better adapt to dynamic task environments. By integrating the genetic algorithm with the Actor–Critic network, agents are equipped to refine their strategies through both gradient-based and evolutionary search, enhancing adaptability and ensuring efficient coverage and resource allocation in complex and variable environments.

This section provides a comprehensive overview of the enhanced PPO agent with the genetic algorithm, detailing its initialization, iterative interactions between agents and the environment, and the strategy update phase. The distinctive features of this algorithm in the context of multi-agent task allocation are emphasized, highlighting its potential for improving operational efficiency.

4.2. Robustness Under Failure Conditions

GAPPO is designed to operate reliably in dynamic environments and addresses potential failure conditions inherent in multi-agent systems. In the event of communication breakdowns, GAPPO’s decentralized architecture enables each agent to rely on local observations and previously learned policies. This design ensures that task execution can continue even when global communication is unavailable.

These design choices are based on well-established principles of decentralized control, which support system resilience under adverse conditions. Although our current experimental evaluation focuses on standard operating scenarios, the theoretical foundation of GAPPO provides a promising basis for robustness in real-world applications.

5. Experimental Results

In this section, we evaluate the established model alongside the proposed GAPPO algorithm through numerical simulations under varying experimental configurations. Specifically, four experiments were conducted to assess the performance of GAPPO in task allocation, considering scenarios with different numbers of agents and tasks. The results demonstrate that GAPPO consistently outperformed other representative algorithms, achieving efficient task allocations and maintaining robust performance across diverse conditions.

To provide additional insights into the algorithm’s effectiveness, a bar chart was employed to compare energy consumption across the scenarios. The findings highlight GAPPO’s capability to optimize task allocation while ensuring effective resource utilization. Overall, the evaluation underscores the practicality of GAPPO in addressing complex task allocation problems in multi-agent systems, further validating its potential to meet diverse system requirements.

Task Allocation

This subsection focuses on evaluating the effectiveness of our proposed algorithm in environment. The GAPPO algorithm is compared with the following representative algorithms.

• Advantage Actor–Critic (A2C) [41]: The A2C algorithm optimizes decision-making by integrating policy and value functions. The advantage function $A(s_t,a_t)$ is expressed as

  $$A(s_t,a_t)=Q(s_t,a_t)-V\left(s_t\right),$$

  (16)

  where $Q(s_t,a_t)$ denotes the expected return from action $a_t$ in state $s_t$, and $V\left(s_t\right)$ provides the baseline value of $s_t$. This structure enhances stability in learning by reducing gradient variance.
• Proximal Policy Optimization (PPO) [42]: PPO is an on-policy algorithm designed to balance exploration and exploitation by constraining policy updates, which limits excessive updates and enhances stability and sample efficiency in learning.
• Deep Q-Network (DQN) [43]: DQN is a value-based reinforcement learning algorithm that approximates the optimal action–value function $Q(s,a)$ using deep neural networks. Leveraging experience replay and a target network, DQN stabilizes learning and updates the Q-values as

  $$\begin{array}{cc}\hfill & \delta =r_t+\gamma \underset{a^{\prime }}{max}Q(s_{t+1},a^{\prime })-Q(s_t,a_t),\hfill \\ \hfill & Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \delta ,\hfill \end{array}$$

  (17)

  where $\alpha$ is the learning rate, $r_t$ is the reward, and $\gamma$ is the discount factor.
• Deep Deterministic Policy Gradient (DDPG) [44]: DDPG is an off-policy algorithm that addresses continuous action spaces by combining an Actor–Critic framework with deterministic policy gradients. The actor network directly optimizes the policy, while the critic network evaluates the action–value function, with the policy gradient update given by

  $${\nabla }_{\theta }J\approx {\mathbb{E}}_{s_t\sim {\rho }^{\pi }}\left[{\nabla }_{a}Q(s_t,a|{\theta }^Q){\nabla }_{\theta }\mu \left(s_t\right|{\theta }^{\mu })\right],$$

  (18)

  where $Q(s,a|{\theta }^Q)$ is the action–value function, $\mu \left(s_t\right|{\theta }^{\mu })$ is the policy, and ${\theta }^{\mu }$ and ${\theta }^Q$ are the parameters of the actor and critic networks, respectively.

Four scenarios are examined: (i) 100 agents are allocated to 20 tasks; (ii) 300 agents are allocated to 30 tasks; (iii) 500 agents are allocated to 30 tasks; (iv) 500 agents are allocated to 50 tasks.

Monte Carlo simulations are conducted for each scenario, with results generated from over 100 runs per scenario.

In each simulation trial, agents are randomly allocated, with an initial random allocation conducted for each agent at the starting moment, and the maximum number of iterations is set to 200 for the first scenario, while it is set to 300 for the subsequent three scenarios. The time unit $∆t$ is defined as 1 hour (h) across all scenarios.

To evaluate reinforcement learning algorithms, we compare GAPPO, A2C, PPO, DQN, and DDPG across four increasingly complex scenarios, where agent capacity and task load are balanced, and the optimal completion time remains 1 unit time. The results demonstrate GAPPO’s superior performance, rapid convergence, and stability in large-scale task allocation.

In the first scenario (Figure 2), GAPPO quickly minimizes completion time and stabilizes at a low value, whereas other algorithms show slower learning and fluctuations. As complexity increases in the second scenario (Figure 3), GAPPO achieves optimal performance with fewer iterations, while competing algorithms exhibit erratic learning curves.

In the third scenario (Figure 4), GAPPO maintains robust performance with rapid convergence and minimal fluctuations, while other algorithms struggle with instability. In the fourth scenario (Figure 5), GAPPO continues to excel, reaching optimal performance quickly despite increased task volume, while alternative algorithms progress slowly and experience frequent regressions.

Across all scenarios, GAPPO consistently reduces maximum completion time, demonstrating strong adaptability and efficiency in handling large-scale multi-agent task allocation problems. Its rapid convergence and stability establish it as a highly effective solution for dynamic reinforcement learning tasks in complex environments.

In conclusion, GAPPO outperforms other algorithms in scalability and efficiency, making it well suited for real-world applications that require fast, reliable task allocation in complex, dynamic settings.

GAPPO demonstrates a significant advantage in energy efficiency for multi-agent task allocation. As shown in Figure 6, it consistently outperforms other algorithms in battery retention. In a setup with 100 agents and 20 tasks, GAPPO maintained 91.9% battery by the end of the process, while A2C, PPO, DQN, and DDPG retained only 52.2–55.5%. This notable gap, observed under identical conditions, underscores GAPPO’s superior energy management without compromising task allocation performance.

Each agent’s battery level is initialized at 20, and the reduction in battery power is proportional to the distance traveled. Therefore, the battery consumption is represented as

$$E_i^{k+1}=E_i^k-k·d_{ij},$$

(19)

where k is a proportionality constant representing the battery consumption rate per unit distance.

In larger-scale setups with 300 and 500 agents, the initial battery capacity was increased from 20 to 30 units to match the increased agent count. Despite this, GAPPO maintained a clear edge in energy management. Its battery percentage remained consistently high, between 91.7% and 91.8%, while A2C, PPO, DQN, and DDPG ranged from 74.2% to 81.6%. This sustained advantage, even with a higher initial battery capacity and increased task load, highlights GAPPO’s scalability and robustness in energy optimization.

While the primary focus of GAPPO is to optimize energy efficiency in multi-agent systems, its broader implications must also be considered. One key aspect is the fairness of resource allocation, as reinforcement learning-based policies may inadvertently favor specific agents or regions if not properly designed. Ensuring that GAPPO maintains balanced allocations across all agents is crucial to prevent systemic biases in real-world applications. From a sustainability perspective, our algorithm reduces energy consumption, which is particularly beneficial for large-scale deployments, such as smart grids or autonomous fleets. However, it is important to consider the trade-offs between computational cost and energy savings. Training reinforcement learning models requires significant computational resources, and optimizing this aspect remains an open challenge for sustainable AI development. Future research could explore strategies such as adaptive model complexity or decentralized learning to further minimize computational overhead while maintaining performance.

Overall, the results demonstrate that GAPPO effectively manages energy consumption across different scales and battery configurations while maintaining performance as task complexity and agent numbers increase. Its ability to balance task allocation and energy efficiency makes it well suited for real-world applications with constrained computational and energy resources. This robustness highlights its potential for deployment in complex multi-agent environments requiring high efficiency across multiple dimensions.

In the conducted experiments, GAPPO outperformed the baseline algorithms, including PPO, DDPG, DQN, and A2C, across various metrics such as task completion time, resource efficiency, and scalability. GAPPO’s superior performance is attributed to its ability to combine the strengths of both reinforcement learning and evolutionary algorithms. The genetic algorithm component enables a more diverse exploration of the policy space, leading to more robust solutions, particularly in dynamic environments where task configurations and agent capabilities vary over time.

Compared to PPO and A2C, which rely solely on policy gradient algorithms, GAPPO benefits from the genetic algorithm’s ability to select and evolve the most promising policies across generations. This evolution not only improves the exploration of the solution space but also ensures better convergence, particularly in complex, multi-agent environments where standard reinforcement learning algorithms often struggle to balance exploration and exploitation.

DDPG and DQN, being value-based algorithms, face limitations in environments with continuous action spaces or complex task dependencies. Those algorithms are more prone to getting stuck in local optima due to their reliance on fixed Q-values or deterministic policies. In contrast, GAPPO overcomes those limitations by combining both policy gradient and evolutionary search. This flexibility allows it to dynamically adjust its policies to adapt to changes in the environment and task complexity, leading to improved performance in a wider range of scenarios.

In summary, GAPPO’s integration of a genetic algorithm with reinforcement learning enables it to effectively balance exploration and exploitation, outperforming traditional algorithms like PPO, DDPG, DQN, and A2C in terms of scalability, adaptability, and robustness. This makes GAPPO particularly well suited for dynamic multi-agent systems, where environment complexity and agent interactions are key factors for success.

6. Conclusions and Future Work

In conclusion, the GAPPO algorithm stands out as a robust solution to the challenges inherent in multi-agent task allocation within complex environments. Traditional algorithms often encounter significant limitations in scalability and adaptability, struggling to manage the intricacies associated with numerous agents and tasks efficiently. In contrast, the GAPPO algorithm effectively addresses these shortcomings by prioritizing energy efficiency and optimal resource distribution, leading to superior performance outcomes. Our comprehensive experimental results consistently demonstrate that GAPPO algorithm significantly reduces task completion time while maintaining high energy levels across agent fleets. This effectiveness highlights the algorithm’s capability to enhance operational efficiency in diverse scenarios, solidifying its role in advancing multi-agent systems.

Future research will concentrate on several pivotal points: (1) optimizing energy management strategies within GAPPO, particularly in scenarios with varying task loads and agent configurations; (2) validating GAPPO’s performance in real-world applications, such as UAV fleets or autonomous robotic systems, to ensure its effectiveness and reliability outside of controlled environments; (3) although GAPPO effectively handles communication breakdowns and unexpected agent behaviors through the strategies outlined above, there is room for improvement. Future work could explore more sophisticated fault-tolerant mechanisms, such as decentralized learning or collaborative filtering techniques, to enhance agent cooperation in extreme failure conditions. Additionally, adaptive strategies could be further refined to allow for proactive anomaly detection, enabling the system to anticipate potential failures before they disrupt performance.