A Hybrid Genetic Algorithm and Proximal Policy Optimization System for Efficient Multi-Agent Task Allocation

Abstract

Efficient task allocation remains a fundamental challenge in multi-agent systems, particularly under resource constraints and large-scale deployments. Classical methods, including market-based mechanisms, centralized optimization techniques, and game-theoretic strategies, have been widely applied to address the multi-agent task allocation problem. While effective in small-to-medium-sized settings, these approaches often encounter limitations in terms of scalability, adaptability to dynamic environments, and computational efficiency as the problem size increases. To address these limitations, this study introduces a proximal policy optimization system augmented with a genetic algorithm (GAPPO) that integrates evolutionary search with deep reinforcement learning. GAPPO enables agents to develop energy-efficient task allocation strategies by perceiving environmental states and optimizing their actions through iterative policy updates. The genetic component promotes broader policy exploration beyond local optima, while the proximal policy optimization ensures update stability and sample efficiency. To evaluate the proposed GAPPO algorithm, extensive simulations are conducted across four scenarios, with the largest involving 50 tasks and 500 agents. The results demonstrate that GAPPO achieves superior performance compared to baseline methods, particularly in reducing task completion time. These findings highlight the algorithm’s robustness and efficiency in handling large-scale and computationally intensive coordination tasks.

1. Introduction

Task allocation in multi-agent systems optimizes resource utilization and minimizes operational costs [1]. The challenge lies in balancing task requirements, agent capabilities, and environmental constraints, which can vary significantly [2]. Efficient approaches mitigate resource wastage by assigning tasks to appropriate agents [3]. As the number of agents and tasks increases, the computational complexity of coordination grows substantially—often at least quadratically—due to the combinatorial nature of agent–task interactions and inter-agent communication [4]. This trend underscores the necessity for adaptive and scalable solutions capable of maintaining performance in large-scale environments. Developing dynamic algorithms is therefore crucial for enhancing scalability and robustness across diverse applications [5]. Multi-agent systems have gained attention for their role in dynamic task allocation and rapid deployment. They have been applied in supply chain management [6], energy distribution [7], and autonomous vehicle coordination [8], among other fields [9].

Centralized optimization algorithms, such as mixed-integer programming and reinforcement learning algorithms, are common in agent-assisted networks, but they rely on global information from a central controller [10]. To address this, Jiang et al. developed distributed algorithms for large-scale semidefinite programming, enhancing efficiency with low-rank transformations [11]. Gao et al. combined potential games with MADDPG to optimize UAV trajectories, reducing delays and improving energy efficiency [12]. Hu et al. proposed MO-MIX, a CTDE-based framework for multi-objective reinforcement learning, achieving high-quality Pareto approximations with lower computational overhead [13]. However, centralized algorithms suffer from scalability issues and single points of failure in dynamic environments [14], highlighting the need for alternative solutions.

Distributed optimization decentralizes decision-making, allowing agents to allocate tasks based on local information with limited communication [15]. This improves scalability and adaptability and reduces failure risks. Ren et al. demonstrated its effectiveness in large-scale wireless sensor networks [16], while Zhang et al. applied it to multi-robot systems [17]. However, Liu et al. noted inefficiencies from incomplete information [18], and the lack of global coordination can lead to suboptimal allocation and slower convergence [19].

Reinforcement learning (RL) has emerged as a powerful approach for multi-agent task allocation, enabling agents to optimize task distribution through continuous interaction with dynamic environments [20]. Its adaptability enhances decision-making in complex scenarios, improving applicability to multi-agent systems [21]. Ke et al. combined combinatorial optimization with RL to improve ride-sourcing efficiency [22], while Gao et al. proposed an autonomous perception-based RL scheme for optimal consensus control [23]. Chang et al. leveraged RL for UAV trajectory and resource allocation, optimizing user association and power allocation [24]. Furthermore, Guo et al. integrated genetic algorithms with proximal policy optimization for differentiated traffic scheduling in TSN-5G industrial networks, achieving notable improvements in end-to-end delay and algorithm convergence [25]. The overview of optimization-based approaches in multi-agent systems can be seen in

Table 1. Overview of optimization-based approaches in multi-agent systems.

Technique	References	Year	Data Used
Distributed Optimization	[16]	2022	Wireless sensor networks
Mixed-Integer Programming	[11]	2023	Semidefinite programming
UAV Trajectory + Resource Allocation	[24]	2024	UAV data
MADDPG	[12]	2021	UAV trajectories
Combinatorial Optimization + RL	[22]	2022	Ride-sourcing data
MO-MIX (CTDE)	[13]	2023	Multi-objective optimization data
Autonomous Perception-based RL	[23]	2024	Autonomous perception data
GA + PPO for Traffic Scheduling	[25]	2024	TSN-5G traffic data

.

Motivated by the potential benefits of combining different algorithms, we propose a hybrid approach that integrates genetic algorithms (GAs) with proximal policy optimization (PPO) for multi-agent task allocation. GAs improve exploration through evolutionary operations, while PPO enables adaptive policy refinement. This combination enhances both adaptability and performance by uniting GAs’ exploration with PPO’s optimization.

The main contributions of this paper are as follows:

• We propose a Markov game formulation for the multi-agent task allocation problem, in which agents independently refine their strategies through continuous interaction with the environment. By relying on local observations to guide each agent’s decision-making, the proposed approach effectively supports coordinated and efficient task distribution throughout the swarm, thereby addressing scalability and resource allocation challenges in multi-agent systems.
• We propose the GAPPO algorithm, which integrates genetic algorithms with proximal policy optimization and incorporates an attention mechanism alongside an adaptive learning rate. This design enables agents to adjust to varying task requirements, enhancing learning stability and coordination efficiency.
• Numerical results validate the efficiency and scalability of the proposed method, highlighting its faster convergence and better adaptability compared to the prevailing reinforcement learning algorithms.

The remainder of this paper is organized as follows. Section 2 describes the problem formulation. Section 3 presents the GAPPO algorithm and provides a detailed overview of its process. Section 4 discusses the performance of the algorithm in different environments. Section 5 presents a summary of the research and mentions future research directions.

2. Problem Formulation

In this paper, we address the multi-agent task allocation problem, where agents are assigned to distinct tasks and operate either independently or cooperatively to improve overall task efficiency and coverage. Each agent performs both task execution and allocation assessment, aiming to adapt autonomously to dynamic environments while optimizing resource utilization. To achieve this, agents iteratively estimate their local states, allocate tasks, and update their strategies, as illustrated in Figure 1.

• State Estimation: At each time step t, the agent maintains a local state $S_{ti}$, comprising its spatial coordinates, surrounding occupancy information, and relevant task identifiers. This state is continuously updated and evolves according to a transition function $f(S_{ti},a_i)$. The Actor–Critic framework takes this state as input to support real-time motion planning, enabling effective navigation, obstacle circumvention, and target recognition.
• Task Allocation: To prevent agent congestion and redundant task assignments, a distributed allocation scheme is employed. Agents evaluate candidate tasks based on factors such as distance, priority level, and current workload. Decisions are dynamically adjusted through localized communication, promoting balanced task execution and optimal use of agent capabilities.
• Strategy: Agents refine their behavior through policy updates driven by local observations and rewards. In non-cooperative conditions, each agent adapts its policy independently. Under collaborative settings, shared policy gradients enhance collective learning. The GAPPO model incorporates attention mechanisms to facilitate both autonomous decision-making and inter-agent coordination.

2.1. System Model

Suppose a set of agents $N=\{1,2,\dots ,n\}$ and a set of tasks $S=\{1,2,\dots ,m\}$ are given. Each agent can participate in only one task at a time. Let ${\tau }_i\in S$ denote the task allocated to agent $i\in N$, and let $I_j\subseteq N$ represent the set of agents assigned to task $j\in S$. Each agent is restricted to a single task at any given time, whereas a task may accommodate multiple agents simultaneously. Tasks are located at fixed positions ${\mathrm{pos}}_j=(x_j,y_j,z_j)$, and the position of agent i is assumed to coincide with the location of its allocated task, i.e., ${\mathrm{pos}}_i={\mathrm{pos}}_{{\tau }_i}$. The Euclidean distance between agent i and task j is defined as

$$d_{ij}=\parallel {\mathrm{pos}}_i-{\mathrm{pos}}_j\parallel .$$

(1)

The above distance also constrains communication, limiting interactions to tasks and agents within range.

Each task j has a workload $h_j\ge 0$, while an agent’s work ability ${\omega }_i$ defines its execution capacity per unit time. The total work ability of the assigned agents must satisfy the following:

$${\sum }_{i\in {\left\{I\right\}}_j}{\omega }_i-h_j\ge 0.$$

(2)

To ensure efficient allocation, the reward for task j is

$$R_j=-{\tau }^{{\sum }_{i\in {\left\{I\right\}}_j}{\omega }_i-h_j},$$

(3)

where $\tau \in (0,1)$ controls the reward sensitivity to surplus work ability. A higher $\tau$ incentivizes faster completion, while a lower $\tau$ limits unnecessary agent participation.

The objective is to minimize the total completion time $T=max\{T_1,T_2,\dots ,T_m\}$, where

$$T_j={\displaystyle \frac{h_j}{{\sum }_{i\in {\left\{I\right\}}_j}{\omega }_i}}.$$

(4)

Thus, the optimization problem is formulated as follows:

$$\begin{array}{cc}\hfill & \underset{\left\{e_{ij}\right\}}{min}\left\{T_j\right\},\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.{\sum }_{j=1}^m{e}_{ij}\le 1,\forall i\in N,\hfill \\ \hfill & e_{ij}\in \{0,1\},\forall i\in N,\forall j\in S,\hfill \end{array}$$

(5)

where $e_{ij}$ is a binary decision variable indicating whether agent i is assigned to task j.

2.2. State Action Model

In the agent swarm framework, the decision-making process of each agent is represented by the tuple $(s_t,a_t,r_t,s_{t+1})$, where $s_t$ denotes the agent’s state at time t, including its position ${\mathrm{pos}}_i=(x_i,y_i,z_i)$, task-related features, and the environmental map. The agent selects action $a_t$ based on $s_t$ by maximizing the action-value function

$$a_t=arg\underset{j\in \mathcal{C}\left(i\right)}{max}Q(s_t,j),$$

(6)

where $Q(s_t,j)$ estimates the cumulative reward of selecting task j. If no tasks are available, the agent selects an idle action to conserve resources.

The policy function $\pi (s_t,j)$ maps $s_t$ to a probability distribution over actions

$$\pi (s_t,j)=P(a_t=j\mid s_t),$$

(7)

which is optimized using reinforcement learning. The Critic evaluates the expected reward starting from $s_t$ using the value function

$$V\left(s_t\right)=\mathbb{E}\left[\sum _{k=t}^T{\gamma }^{k-t}{r}_k\mid s_t\right],$$

(8)

guiding the Actor to refine the policy.

The reward $r_t$ reflects the agent’s performance, considering task importance $I_j$, completion effectiveness $E_i$, and collaborative efficiency $C_i$

$$r_t=\alpha ·I_j+\beta ·E_i+{\gamma }_i·C_i,$$

(9)

where $\alpha ,\beta ,{\gamma }_i$ are the respective weights for those components. The subsequent state $s_{t+1}$ reflects the updated environment after taking action $a_t$:

$$s_{t+1}=f(s_t,a_t),$$

(10)

where $f(·)$ represents the environment transition function.

This decision-making process models the interaction between agents and the environment, optimizing actions to maximize cumulative rewards through the Actor–Critic framework and genetic algorithms.

In proximal policy optimization (PPO), the advantage function $A(s_t,a_t)$ measures the benefit of taking action $a_t$ in state $s_t$ and is computed using Generalized Advantage Estimation (GAE)

$$A(s_t,a_t)={\hat{A}}_t={\sum }_{i=t}^T{\left(\gamma \lambda \right)}^{i-t}{\delta }_i,$$

(11)

where ${\delta }_i$ is the temporal difference error, given by the following:

$${\delta }_i=r_i+\gamma V(s_{i+1};\omega )-V(s_i;\omega ).$$

(12)

The value function $V(s_i;\omega )$ is updated iteratively as follows:

$$V(s_i;\omega )\leftarrow V(s_i;\omega )+{\alpha }_1{\delta }_i,$$

(13)

where ${\alpha }_1$ is the learning rate. This iterative update process allows agents to optimize their rewards, improving task execution and resource allocation.

3. Learning Algorithm

Various optimization algorithms, such as the leader–follower algorithm [26], virtual structure algorithm [27], and behavior-based control algorithm [28], have been proposed to address the multi-agent task allocation problem. Additionally, reinforcement learning has proven effective in this domain. This paper introduces a novel proximal policy optimization system augmented with a genetic algorithm (GAPPO) that combines the stability and sample efficiency of PPO with the global search capability of genetic algorithms to enhance decision-making in complex task environments. The pseudocode of GAPPO algorithm can be seen in Algorithm 1.

Algorithm 1 Proximal Policy Optimization Augmented With Genetic Algorithm.

1: Initialize: Population of Actor–Critic networks $P=\{P_1,P_2,\dots ,P_N\}$, where each policy is encoded as a real-valued vector. Structural infeasibility, such as instability or divergence, is mitigated by bounding weight perturbations and incorporating penalty terms within the fitness evaluation to discourage infeasible policies; 2: Set hyperparameters: learning rate $lr$, discount factor $\gamma$, clip ratio $ϵ$, and KL divergence target $K{L}_{target}$; 3: Initialize replay buffer with capacity $C=10000$; 4: Set generation counter $g=0$; 5: while training do 6:   for each individual $P_i\in P$ do 7:     Initialize trajectory T and observe initial state $s_0$; 8:     for each time step $t=0$ to $T_{max}$ do 9:       Select action $a_t$ using policy $P_i$ based on $s_t$; 10:      Execute $a_t$, observe reward $r_t$ and next state $s_{t+1}$; 11:      Append transition $(s_t,a_t,r_t,s_{t+1})$ to trajectory T; 12:      if terminal state reached then 13:        Break 14:      end if 15:    end for 16:    Store trajectory T in replay buffer; 17:  end for 18:  if len(replay buffer) ≥ BATCH_SIZE then 19:    Sample a batch of transitions from replay buffer; 20:    for each transition in the batch do 21:      Compute returns $R_t$ using $\gamma$; 22:      Compute advantages ${\hat{A}}_t$ using Generalized Advantage Estimation (GAE); 23:      Get action probabilities $\pi \left(a_t\right|s_t)$ and state values $V\left(s_t\right)$; 24:      Calculate surrogate loss $L\left(\theta \right)$ by (15); 25:      Calculate value loss by (16); 26:      Compute total loss $L=L\left(\theta \right)+L_V$; 27:      Update parameters using gradient descent by (17); 28:    end for 29:  end if 30: end while 31:  Global Evolution via Genetic Algorithm 32: while g < max_generations do 33:   for each individual $P_i\in P$ do 34:     Evaluate $P_i$ to obtain fitness score $F_i$; 35:   end for 36:   Select individuals based on fitness; 37:   Select top 10% individuals based on fitness and retain them as elites; 38:   Generate offspring to fill the remaining 90% of the population via: 39:   Selection (e.g., tournament or roulette), 40:   Crossover (with rate 0.8), and 41:   Mutation (with rate 0.05); 42:   Form new population $P^{\prime }=\mathrm{Elites}\cup \mathrm{Offspring}$; 43:   Update population $P\leftarrow P^{\prime }$ and increment generation counter g; 44: end while 45: Periodically: Update target networks to stabilize training

To enhance replicability, all components of the genetic algorithm used in GAPPO are systematically detailed in

Table 2. Key components of the genetic algorithm framework in GAPPO.

Component	Design Choice and Rationale
Encoding Scheme	Real-valued direct encoding of Actor–Critic network weights to ensure structural feasibility and smooth parameter space exploration.
Population Initialization	The population size N is scaled proportionally with the number of agents and tasks, ranging from 100 to 300 individuals. Each individual represents a complete Actor–Critic policy, encoded as a real-valued parameter vector.
Selection Strategy	Fitness-proportionate selection with soft penalty terms integrated to discourage unstable or divergent policies.
Crossover Mechanism	Uniform crossover applied with a rate of 0.8, enabling information exchange between high-performing individuals.
Mutation Strategy	Gaussian mutation with a probability of 0.05 per gene. Perturbations are clipped to ensure parameter validity and prevent instability.
Fitness Evaluation Function	$F_i=\lambda ·{\displaystyle \frac{R_i}{R_{\max }}}+\mu ·{\displaystyle \frac{H_i}{H_{\max }}}-\nu ·{\displaystyle \frac{E_i}{E_{\max }}}$, where $R_i$: reward, $H_i$: entropy, and $E_i$: computational cost.
Fitness Weights	$\lambda =0.6$ (performance), $\mu =0.3$ (exploration), and $\nu =0.1$ (efficiency). Tuned empirically for optimal balance.
Constraint Handling	Penalty of $-5$ for infeasible allocations (capacity, range). Mutation range constraints enforce feasibility.
Generational Control	The population evolves for 500 generations. A generation counter tracks convergence trends.
Termination Criteria	Maximum of 500 generations or earlier if population fitness variance falls below a threshold, $ϵ={10}^{-3}$.
Reinsertion Strategy	Elitism is applied: the top 10% of individuals (based on fitness) are directly carried over to the next generation. The remaining 90% of the population is replaced by offspring generated via selection, crossover (rate 0.8), and mutation (rate 0.05).
Actor–Critic Network Structure	Actor: Two-layer MLP (256–128 units, ReLU), output via softmax. Critic: Two-layer MLP (256–128 units, ReLU), scalar output. Parameters are flattened for GA operations.

. This includes the encoding scheme, selection and variation operators, fitness evaluation, constraint handling, and hyperparameter configurations.

While PPO is effective for local policy refinement, it is sensitive to initialization and may converge to suboptimal solutions in highly non-convex policy spaces. To mitigate this, GAPPO introduces a population-based genetic layer. Initially, policy networks are pretrained using proximal policy optimization (PPO), where agents interact with the environment and store state–action–reward tuples in the replay buffer. Once sufficient data are collected, Generalized Advantage Estimation (GAE) is applied to compute advantages ${\hat{A}}_t$, and policy parameters are updated via the surrogate objective $L\left(\theta \right)$. This pretraining helps ensure a reasonable starting point for the evolutionary process. The resulting population serves as the initial generation for the genetic algorithm, which then refines policies by directly optimizing network parameters through selection, crossover, and mutation operations, allowing for the simultaneous evolution of multiple Actor–Critic networks. This dual approach helps prevent premature convergence to suboptimal solutions by promoting diversity and global search in the policy space.

The combination of PPO’s local optimization with GA’s global search enhances the overall adaptability and robustness of the system, especially in dynamic and complex environments.

The process begins by initializing a population P of Actor–Critic networks, each representing an optimized policy. Key hyperparameters are selected based on prior studies: the learning rate $lr$ is set to $3\times {10}^{-4}$ for balanced convergence and stability; the discount factor $\gamma =0.99$ emphasizes long-term rewards; the clipping threshold $ϵ=0.2$ constrains policy updates to avoid abrupt changes; and the KL divergence target $K{L}_{\mathrm{target}}=0.01$ prevents over-optimization and supports stable learning.

To further optimize these parameters, we employ automated hyperparameter tuning techniques, including grid search and random search. These tools explore a range of values, such as $lr\in [{10}^{-5},{10}^{-3}]$, $\gamma \in [0.9,0.99]$, and $ϵ\in [0.1,0.3]$. After conducting multiple experiments, the final optimal values were found to be $lr=3\times {10}^{-4}$, $\gamma =0.98$, and $ϵ=0.2$, which achieved the best performance in terms of task completion time and energy efficiency.

A replay buffer with a capacity $C=10,000$ is used to store state–action–reward sequences for training, ensuring a sufficient amount of experience for the agents to learn effectively. A generation counter g tracks the evolution of the population, allowing for continuous improvement of the policies across generations. This process is repeated for a total of 500 generations to ensure that the policies converge to an optimal solution.

To enhance adaptability and prevent convergence to suboptimal solutions, a genetic algorithm is employed following PPO updates. Prior research has demonstrated the effectiveness of this hybrid paradigm in complex optimization tasks, as discussed in literature [25]. Although the referenced work targets TSN-5G traffic scheduling, the underlying challenge of navigating high-dimensional policy spaces is shared with multi-agent task allocation, justifying the applicability of the GA-PPO framework in this study.

In our implementation, each individual in the population encodes the weights of an Actor–Critic network as a real-valued vector. This direct encoding avoids structural infeasibility. During the evolutionary process, policy infeasibility (e.g., instability or divergence) is mitigated by bounding weight perturbations and applying soft penalties to the fitness function for high-variance or low-reward behaviors. Moreover, mutation operations are clipped within a fixed range to ensure parameter validity throughout generations. Each policy’s fitness is evaluated by

$$F_i=\lambda ·{\displaystyle \frac{R_i}{R_{\max }}}+\mu ·{\displaystyle \frac{H_i}{H_{\max }}}-\nu ·{\displaystyle \frac{E_i}{E_{\max }}},$$

(14)

where $R_i$ is the cumulative reward, $H_i$ is the policy entropy, and $E_i$ is the computational cost. The parameters $\lambda$, $\mu$, and $\nu$ balance those components.

The coefficients $\lambda$, $\mu$, and $\nu$ are introduced to balance the trade-offs among reward maximization, policy exploration, and energy efficiency. Specifically, $\lambda$ emphasizes the importance of task performance, guiding the selection toward high-reward policies. $\mu$ encourages exploration by favoring policies with higher entropy, thereby reducing the risk of premature convergence. $\nu$ penalizes computational cost to promote energy-aware behavior, which is critical in resource-constrained environments. Those weights are empirically tuned to ensure that the evolved policies achieve a desirable balance between task effectiveness and adaptability. PPO’s stability is ensured using the clipped objective

$$L\left(\theta \right)=min\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)}}{\hat{A}}_t,\mathrm{clip}\left({\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)}},1-ϵ,1+ϵ\right){\hat{A}}_t\right),$$

(15)

where $ϵ$ controls the clipping range, limiting policy updates for stability.

The total loss combines the surrogate and value loss

$$L_V={\displaystyle \frac{1}{2}}{(R_t-V\left(s_t\right))}^2,$$

(16)

and the total loss L is optimized using gradient descent.

After computing the total loss $L=L\left(\theta \right)+L_V$, where $L\left(\theta \right)$ is the clipped surrogate objective and $L_V$ is the value function loss, the policy and value network parameters are updated using gradient descent

$$\theta \leftarrow \theta -\eta {\nabla }_{\theta }L,$$

(17)

where $\eta$ is the learning rate. This update rule ensures that the parameters are optimized to improve policy performance while maintaining training stability.

Based on the computed fitness, a tournament selection strategy is used to identify parent policies. Crossover and mutation are then applied to generate a new population $P^{\prime }$, enabling broader exploration of the policy space beyond the local region optimized by PPO. This evolutionary step improves the diversity of candidate solutions and enhances the algorithm’s robustness to initial conditions and non-stationary environments.

Compared with standard PPO, GAPPO introduces a population-based optimization step that improves the policy search process in two aspects: (i) it enables the discovery of more globally optimal solutions through exploration beyond the gradient-based neighborhood, and (ii) it incorporates energy awareness by explicitly penalizing high energy consumption in the fitness evaluation.

Simulation results demonstrate that GAPPO achieves (i) reduced task completion time, resulting from more effective task distribution strategies; (ii) lower energy consumption, as evolved policies tend to favor energy-efficient behaviors; and (iii) enhanced robustness to initialization, attributed to the diversity maintained by the genetic population.

The optimization process iterates until the maximum number of generations is reached. The final policy is selected based on the highest average reward over multiple evaluation episodes.

4. Experimental Results

This section evaluates the model and the proposed GAPPO algorithm through numerical simulations across various configurations. Four experiments were conducted to assess GAPPO’s performance in task allocation with different agent–task scenarios. The experiments were conducted on a system equipped with an NVIDIA RTX 4090 GPU, using Python 3.11.4 with key libraries including PyTorch 2.1, NumPy. GAPPO integrates genetic algorithms with proximal policy optimization, enhanced by an attention mechanism and adaptive learning rate.

The genetic algorithm employs a population size of 100, a crossover rate of 0.8, and a mutation rate of 0.05, while PPO is trained with a learning rate of 0.0003. To balance the contributions of task importance, completion effectiveness, and collaborative efficiency in the reward function, we empirically set the weights as $\alpha =0.4$, $\beta =0.3$, and ${\gamma }_i=0.3$. This configuration reflects the design choice to prioritize timely and appropriate task execution ($I_j$) while also considering the agent’s execution quality ($E_i$) and the overall coordination level among agents ($C_i$). For the genetic evaluation function, we set $\lambda =0.6$, $\mu =0.3$, and $\nu =0.1$ to prioritize reward maximization while ensuring adequate exploration and minimizing computational cost. To manage infeasible allocations—such as agents exceeding task capacity constraints or violating spatial range limits—we incorporate a penalty term into the reward function. Specifically, if an agent selects an invalid task, a negative reward is applied. The penalty value was initially set to $-5$ based on empirical evaluation, considering that the immediate rewards for valid actions typically fall within the range of [0, 10]. While this fixed penalty has proven effective in discouraging invalid actions without significantly impeding exploration, its static nature may limit adaptability across varying scenarios. A potential direction for future enhancement is to adopt a dynamic penalty strategy, where the penalty magnitude is scaled relative to the current reward distribution. The modified reward function is expressed as

$$r_t=\alpha ·I_j+\beta ·E_i+\gamma ·C_i+\theta ·{\mathbb{I}}_{\mathrm{invalid}},$$

(18)

where ${\mathbb{I}}_{\mathrm{invalid}}=-1$ if the allocation is invalid and 0 otherwise, and $\theta =5$.

The GAPPO algorithm is compared with the following representative algorithms.

• Advantage Actor–Critic (A2C) [29]: A2C integrates policy and value functions, with the advantage function $A(s_t,a_t)$ defined as

  $$A(s_t,a_t)=Q(s_t,a_t)-V\left(s_t\right),$$

  (19)

  where $Q(s_t,a_t)$ is the expected return and $V\left(s_t\right)$ is the baseline value, enhancing learning stability by reducing gradient variance.
• Proximal Policy Optimization (PPO) [30]: PPO balances exploration and exploitation by constraining policy updates, which enhances stability and sample efficiency.
• Deep Q-Network (DQN) [31]: The DQN approximates the optimal action-value function $Q(s,a)$ using deep neural networks, stabilizing learning through experience replay and a target network. The update rule is as follows:

  $$\begin{array}{cc}\hfill & \delta =r_t+\gamma \underset{a^{\prime }}{max}Q(s_{t+1},a^{\prime })-Q(s_t,a_t),\hfill \\ \hfill & Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \delta .\hfill \end{array}$$

  (20)
• Deep Deterministic Policy Gradient (DDPG) [32]: The DDPG handles continuous action spaces with an Actor–Critic framework. The policy gradient update is as follows:

  $${\nabla }_{\theta }J\approx {\mathbb{E}}_{s_t\sim {\rho }^{\pi }}\left[{\nabla }_{a}Q(s_t,a|{\theta }^Q){\nabla }_{\theta }\mu \left(s_t\right|{\theta }^{\mu })\right].$$

  (21)

To evaluate the performance of GAPPO, we examine four distinct scenarios: (i) 100 agents are allocated to 20 tasks, (ii) 300 agents are allocated to 30 tasks, (iii) 500 agents are allocated to 30 tasks, and (iv) 500 agents are allocated to 50 tasks.

For each scenario, results are obtained by averaging over more than 100 independent runs, each initialized with random states and actions. This approach simulates the inherent randomness in multi-agent task allocation scenarios, ensuring a robust evaluation of the algorithm’s adaptability and effectiveness under dynamic and unpredictable conditions. By averaging the results across multiple runs, we obtain performance metrics that reflect the consistency and generalization of the algorithm in real-world dynamic environments. In each simulation trial, agents are randomly allocated at the start. The maximum number of iterations is set to 500 for the first two scenarios and 800 for the last two. In each scenario, the agents’ work abilities and the tasks’ workloads are initialized with distinct values. The optimal completion time for each scenario is determined through exhaustive search, and performance is assessed using key indicators: task completion time (T), the proportion of optimal time ($P_0$), and the number of iterations needed to reach the minimum value (min). The difference ratio $\delta$ is expressed as

$$\delta ={\displaystyle \frac{T-T_{\mathrm{opt}}}{T_{\mathrm{opt}}}},$$

(22)

and represents the deviation from the optimal time. The time unit $\Delta t$ is defined as 1 unit time, which corresponds to a fixed and consistent simulation step used across all scenarios. For simplicity, we do not map this unit to a real-world measure such as hours or seconds, as the focus is on relative performance across algorithms. Importantly, this abstraction ensures that the theoretical optimal value—representing the minimum task completion time under ideal allocation conditions—remains constant at 1 unit time in all experiments.

To assess the performance of the proposed approach, we conduct comparisons between GAPPO and the four aforementioned baseline algorithms under four progressively challenging scenarios. These scenarios are designed with balanced agent capacities and task demands, ensuring a consistent theoretical minimum completion time of one time unit. To facilitate a clear performance comparison, a red dotted line is included in the figures to represent this optimal solution. The optimal value is analytically derived based on the theoretical minimum task completion time under ideal allocation conditions—assuming perfect coordination among agents and optimal matching between task requirements and agent capabilities, without communication delay or resource contention. This value serves as a lower bound benchmark, allowing us to assess how closely the proposed algorithm approaches the theoretical optimum. The results demonstrate GAPPO’s superior performance, rapid convergence, and stability in large-scale task allocation.

In the first scenario (

Table 3. Allocation outcomes for the scenario ($n=100$, $m=20$) are compared across algorithms, with each algorithm evaluated over 100 simulation runs.

Algorithm	${\mathit{P}}_0$	$\mathit{\delta }$	T	min
GAPPO	7.5%	7.4%	1.074 h	45
A2C	6.8%	8.8%	1.088 h	107
PPO	4.4%	15.8%	1.158 h	186
DQN	2.2%	22.1%	1.221 h	184
DDPG	4.5%	16.2%	1.162 h	193
Optimal	100%	-	1.0 h	-

, Figure 2), GAPPO quickly minimizes the task completion time, stabilizing at a low value. In contrast, other algorithms exhibit slower convergence and greater fluctuations. Notably, PPO and DQN experience prolonged exploration phases, resulting in higher iteration counts and suboptimal time proportions.

In the second scenario (

Table 4. Allocation outcomes for the scenario ($n=300$, $m=30$) are compared across algorithms, with each algorithm evaluated over 100 simulation runs.

Algorithm	${\mathit{P}}_0$	$\mathit{\delta }$	T	min
GAPPO	20.3%	1.4%	1.014 h	86
A2C	2.6%	23.4%	1.234 h	277
PPO	2.3%	25.6%	1.256 h	281
DQN	2.1%	21.2%	1.212 h	272
DDPG	1.2%	50.2%	1.502 h	286
Optimal	100%	-	1.0 h	-

, Figure 3), where the complexity increases with a larger agent population, GAPPO achieves near-optimal performance with a minimal performance gap ($\delta =1.4\%$) and significantly fewer iterations compared to the alternatives. Other algorithms, such as A2C and PPO, demonstrate delayed convergence, while DDPG suffers from increased task completion time.

In the third scenario (

Table 5. Allocation outcomes for the scenario ($n=500$, $m=30$) are compared across algorithms, with each algorithm evaluated over 100 simulation runs.

Algorithm	${\mathit{P}}_0$	$\mathit{\delta }$	T	min
GAPPO	16.2%	2.4%	1.024 h	220
A2C	3.2%	40.2%	1.402 h	279
PPO	3.2%	40.8%	1.408 h	263
DQN	2.6%	51.1%	1.511 h	286
DDPG	2.6%	51.8%	1.518 h	290
Optimal	100%	-	1.0 h	-

, Figure 4), GAPPO maintains strong scalability, delivering competitive task completion times with minimal deviation from the optimal solution. On the other hand, other methods exhibit significant performance degradation, with some reaching over 50% in performance gap, indicating challenges in adapting to denser agent environments.

In the fourth scenario (

Table 6. Allocation outcomes for the scenario ($n=500$, $m=50$) are compared across algorithms, with each algorithm evaluated over 100 simulation runs.

Algorithm	${\mathit{P}}_0$	$\mathit{\delta }$	T	min
GAPPO	15.3%	3.9%	1.039 h	261
A2C	3.1%	43.6%	1.436 h	268
PPO	4.3%	32.6%	1.326 h	290
DQN	3.8%	37.3%	1.373 h	286
DDPG	2.4%	51.8%	1.518 h	295
Optimal	100%	-	1.0 h	-

, Figure 5), where the task-to-agent ratio increases, GAPPO continues to excel, demonstrating efficient task allocation with the fewest iterations. Meanwhile, DDPG and DQN display instability and fail to achieve a noticeable reduction in completion time.

In conclusion, GAPPO consistently outperforms competing algorithms in terms of scalability, convergence speed, and solution quality across all tested scenarios. It demonstrates minimal performance degradation even as the problem size grows, underscoring its suitability for real-world multi-agent systems requiring fast, stable, and energy-efficient task allocation in dynamic environments.

5. Conclusions and the Future Work

In conclusion, this paper introduces the GAPPO algorithm, which combines genetic algorithms with proximal policy optimization (PPO) for effective task allocation in multi-agent systems. The proposed method demonstrates superior performance in task completion time and energy consumption, particularly when the number of agents and tasks increases. Experimental results show that GAPPO outperforms traditional algorithms like PPO and DDPG, making it a promising approach for scalable and efficient task allocation in dynamic environments.

Future work will focus on further optimizing GAPPO’s performance, particularly in large-scale systems and real-world applications, to improve its adaptability and scalability. Additionally, integrating advanced optimization techniques could enhance its robustness in complex, dynamic environments.