Dual-Component Reward Mechanism Based on Proximal Policy Optimization: Resolving Head-On Conflicts in Multi-Four-Way Shuttle Systems for Warehousing

Abstract

Path planning for multiple four-way shuttles in high-density warehousing is frequently hampered by efficiency-degrading conflicts, particularly head-on deadlocks. To address this challenge, this paper proposes a multi-agent reinforcement learning (MARL) framework based on Proximal Policy Optimization (PPO). The core of our approach is a novel Cooperative Avoidance Reward Mechanism (CARM), which employs a dual-component reward structure. This structure integrates a distance-guided reward to ensure efficient navigation towards targets and a cooperative avoidance reward that uses both immediate and delayed returns to incentivize implicit collaboration. This design effectively resolves conflicts and mitigates the policy instability often caused by traditional collision penalties. Experiments in a 20 × 20 grid simulation environment demonstrated that, compared to a rule-based A* and Conflict-Based Search (CBS) algorithms, the proposed method reduced the average travel distance and total time by 35.8% and 31.5%, respectively, while increasing system throughput by 49.7% and maintaining a task success rate of over 95%. Ablation studies further confirmed the critical role of CARM in achieving stable multi-agent collaboration. This work offers a scalable and efficient data-driven solution for real-time path planning in complex automated warehousing systems.

1. Introduction

The rapid development of e-commerce, omni-channel retail, and on-demand delivery has imposed stringent requirements on modern warehousing systems, demanding high throughput, low latency, flexible scheduling, and rapid responsiveness [1]. To meet these demands, four-way shuttle-based automated storage and retrieval systems have been widely adopted due to their scalability and operational efficiency [2]. As a next-generation warehousing technology, four-way shuttles enable agile horizontal and vertical movements, seamless lane switching, and online replanning, making them particularly suitable for large-scale, high-density storage scenarios.

However, the large-scale deployment of four-way shuttles introduces significant multi-agent coordination and path planning challenges [3,4]. Classical single-agent methods, such as A* [5] and Dijkstra [6], along with rule-based scheduling strategies [7], are limited in handling high-density traffic flows, cross-layer interactions, and real-time conflict resolution [8]. Conflict-Based Search (CBS), a typical hierarchical multi-agent path planning algorithm, also struggles with computational scalability in large-scale shuttle deployments [9]. Common industrial solutions, like limiting vehicles per layer or enforcing unidirectional travel, reduce collision risks but compromise system throughput and flexibility. Thus, there is an urgent need for path planning methods that address local collision avoidance while optimizing global efficiency and enabling online deployment.

To overcome rule-based limitations, recent research has shifted toward reinforcement learning (RL) for multi-agent path finding (MAPF) [10,11,12,13]. Reinforcement learning provides a computational framework for agents to learn optimal behaviors through continuous interaction with an environment. In a standard RL model, an agent takes an action based on the current state, and the environment provides a reward and transitions to a new state. The core objective is to find an optimal policy that maximizes the long-term cumulative return. In the context of four-way shuttle MAPF, this process is typically modeled as a collaborative Markov Decision Process (MDP) or a more complex Decentralized Partially Observable MDP (Dec-POMDP). By transforming complex path planning into an optimization problem of expected returns within a dynamic search space, RL empowers the system to handle high uncertainty and large-scale dynamic conflicts.

While multi-agent RL (MARL) shows promise in learning decentralized, adaptive policies, its application in warehousing faces such challenges as sample inefficiency and training instability due to sparse rewards and credit assignment difficulties [14], as well as collaboration and safety issues, where agents struggle to form coordination strategies in narrow lanes or high-density conditions, leading to deadlocks, near-misses, or conservative behaviors [11,15]; and generalization and deployment hurdles, with methods often relying on expert demonstrations or environment-specific training, exhibiting poor transferability and lacking interpretable, tunable reward designs [11].

Current studies address these issues along three complementary lines: imitation learning (IL), inverse reinforcement learning (IRL), and causal reasoning.

A.

Imitation Learning (IL)

IL accelerates policy learning by leveraging expert demonstrations to inject domain knowledge and avoid inefficient random exploration [16,17,18]. Representative works include GAPPO, combining generative adversarial IL with proximal policy optimization (PPO) [17]. MixGAIL, a hybrid adversarial framework [18]; and decentralized approaches like PRIMAL, incorporating expert MAPF planners, reward shaping, and sampling for implicit collaboration under partial observability [19,20]. Gaussian process-based IL with Voronoi tessellation enhances data efficiency and generalization [21]. However, IL’s reliance on high-quality, comprehensive demonstrations is costly in dynamic, high-density scenarios, limiting coverage of multi-agent interactions and generalization across densities or topologies [16,18,19].

B.

Inverse Reinforcement Learning (IRL)

IRL infers underlying reward functions from expert demonstrations, bypassing manual reward engineering for tasks like socially compliant navigation [22]. Key works use bioelectric signals or human driving data to construct human-preferred rewards [23,24]; maximum entropy or deep models for cost maps in local planning [25,26]; and encodings of social rules or time-varying costs for human-aware paths [22,27]. IRL’s strength lies in explicit reward structures aligning with expert intent. Yet, in multi-agent warehousing, learned rewards lack interpretability and tunability, struggle with adaptation to changing conditions (e.g., task priorities or densities), and inadequately model delayed collaborative returns, resulting in conservative or locally optimal behaviors [22,25].

C.

Causal Reasoning

Causal reasoning identifies causal relationships between states, actions, and rewards to reduce confounding and enhance policy generalization [28,29]. Notable efforts in this field are as follows: Kansky et al. proposed Schema networks, a representative generative causal model, to realize zero-shot transfer for transferable physics inference [28]; Hua et al. developed a causal masking strategy to help models focus on true causal features in dynamic path planning tasks [29]; Tang et al. introduced structural causal models to optimize temporal relation modeling and communication efficiency in multi-agent motion planning scenarios [30]; in addition, causal insights have been incorporated into hierarchical conflict resolution mechanisms in relevant studies [31]. Furthermore, disentangled representations avoid UAV collision avoidance [32], and graph networks improve MAPF [33], with selective communication reducing redundancy [34]. Despite theoretical benefits in interpretability, causal methods face scalability issues in high-dimensional MDPs, lack systematic conversion of insights into trainable reward signals, and underemphasize delayed returns for long-term collaboration [32,33,34].

In contrast to the aforementioned methods that focus on external expert data or complex causal inference, this paper starts from the most fundamental aspect of reinforcement learning—reward function design (Reward Reshaping). Specifically, it encodes collaborative intent as continuous, semantic learning signals, which can guide agents to achieve efficient navigation and implicit collision avoidance without relying on costly expert data or complex heuristic algorithms. A key feature is the Cooperative Avoidance Reward Mechanism (CARM), which integrates immediate and delayed rewards to explicitly incentivize proactive yielding behaviors, addressing the limitations of existing approaches in modeling long-term collaborative benefits.

The contributions are as follows:

• Distance-aware reward shaping: Dense reward signals based on Manhattan distance are introduced, which are weighted to adapt to the warehouse “main-lane-sub-lane” topology structure for accelerating policy convergence.
• Immediate-delayed dual-component reward framework via CARM: Combines immediate navigation rewards with explicit delayed collaborative returns to capture long-term benefits of yielding actions, fostering implicit multi-agent collaboration and reducing deadlocks.
• Stability, performance, and interpretability guarantees: The method enhances training stability and efficiency in high-density scenarios, with theoretical links between local rewards and global metrics for improved explainability and tunability.

2. Methods

This section formalizes the multi-four-way shuttle path planning problem as a decentralized partially observable Markov decision process (Dec-POMDP) and elaborates on the observation space, action masking mechanism, deep network architecture, and core reward reshaping design.

2.1. Problem Formalization

The multi-agent path planning is modeled as a Dec-POMDP, defined as a septuple:

$$M=⟨N,S,{\left\{A_{\mathrm{i}}\right\}}_{i=1}^N,P,{\left\{O_i\right\}}_{i=1}^N,{\left\{r_i\right\}}_{i=1}^N,\gamma ⟩$$

(1)

where $N$ denotes the number of agents; $S$ the agent state space; $A_i$ the action set per agent; $P$ the state transition probability; $O_i$ the observation space per agent; $r_i\left(s,a\right)$ the immediate reward function per agent; and $\gamma \in \left(0,1\right]$ the discount factor.

At each timestep $t$, each agent selects an action based on its local observation, the environment transitions to a new state according to $P\left(s^{\prime }|s,a\right)$, and rewards are provided. The objective of the MDP is to find a joint policy ${\pi }^{\ast }$ that maximizes the expected cumulative return, which corresponds to the agents’ path policies.

2.2. Warehouse Grid Map and Collision Scenarios

The warehouse is modeled as a 2D grid map, with grid coordinates expressed as integer values, and cargo storage areas corresponding to impassable grid cells. The position of agent $i$ at time $t$ is denoted as $p_t^i=\left(x_i\left(t\right),y_i\left(t\right)\right)$.

In such discretized environments, collisions during multi-agent movement represent a core constraint for path planning algorithms. The most typical conflict types include vertex conflicts and swap conflicts, which can be quantified via indicator functions, and illustrative scenarios of these conflicts are shown in Figure 1.

Figure 1. Two conflict scenarios.

Vertex conflict: A vertex conflict occurs when two agents attempt to occupy the same grid cell at the same discrete time step. In the context of four-way shuttle systems, vertex conflicts typically arise from orthogonal movements at lane intersections and thus manifest as cross conflicts.

$$I_{\mathrm{vert}}(t;i,j)=1\left\{p_{t+1}^i=p_{t+1}^j\right\}$$

(2)

Swap conflict: A swap conflict occurs when two agents attempt to exchange their positions between two consecutive time steps, resulting in a transition-based collision. In narrow warehouse aisles, this situation corresponds to a head-on conflict, which is a primary cause of irreversible deadlocks.

$$I_{\mathrm{swap}}(t;i,j)=1\left\{p_{t+1}^i=p_t^j\wedge p_{t+1}^j=p_t^i\right\}$$

(3)

2.3. Observation Space

The observation space for each agent is a partially observable discrete grid, limited to a local field of view centered on the agent. This local observation setup reduces neural network input dimensionality and enhances policy generalization.

To simplify the learning process, environmental information is classified into two categories, and each observation consists of two binary matrices that, respectively, represent the obstacle distribution and the positions of observable agents within the local field of view. Boundary areas beyond the world are padded as obstacles. From the perspective of each agent, other observable agents function as dynamic obstacles whose positions evolve over time.

To avoid local observation limitations, agents receive absolute 2D target coordinates as directional guidance in the navigation strategy. The observation space is defined as:

$$o_t^i=\left(O_t^{\mathrm{obs}},O_t^{\mathrm{agent}},G_t^i\right)$$

(4)

where $O_t^{\mathrm{obs}}\in {\left\{0,1\right\}}^{H\times W}$ is the binary map of static obstacles in the local field of view; $O_t^{\mathrm{agent}}\in {\left\{0,1\right\}}^{H\times W}$ the positions of other observable agents; and $G_t^i\in {\mathbb{Z}}^2$ the absolute target coordinates. A corresponding scenario is illustrated in Figure 2.

Figure 2. Agent observation space.

2.4. Action Space

The action space consists of a discrete set: forward, backward, left, right, and stay, with each movement spanning one grid unit.

$$A=\left\{\mathrm{forward},\mathrm{backward},\mathrm{left},\mathrm{right},\mathrm{stay}\right\}$$

(5)

The warehousing system employs a “main-lane-sub-lane” layered structure, constraining shuttle movements: bidirectional in main lanes but unidirectional in sub-lanes, without cross-sub-lane switching (see Figure 3). Sampling from valid actions, rather than penalizing invalid ones by stasis, promotes stable training [19]. The valid action set per grid is $A_{valid}\left(p_t^i\right)\subseteq A$. The action mask is defined as

$$m_t^i[a]=\left\{\begin{array}{c}1,a\in A_{valid}\left(p_t^i\right)\\ 0,otherwise\end{array}\right.$$

(6)

Figure 3. Valid Actions of the Agent in Main and Sub Aisles.

The policy outputs log-probabilities $l_t^i\in {\mathbb{R}}^{\left|A\right|}$, masked as

$${\pi }_{\theta }^i\left(a|o\right)={\displaystyle \frac{\exp \left(l_t^i[a]+\log m_t^i[a]\cdot M\right)}{{\displaystyle {\sum }_{a^{\prime }}\exp \left(l_t^i[a^{\prime }]+\log m_t^i[a^{\prime }]\cdot M\right)}}}$$

(7)

where ${\pi }_{\theta }^i\left(a|o\right)$ is the probability that agent $i$ (parameterized by policy $\theta$) selects action $a$ given observation $o$; $M$ is practically set to −1 × 10⁹, which is commonly used in engineering to represent an extremely small value, effectively modeling an impossible or infeasible action.

2.5. Reward Function

The reward function at timestep $t$ comprises three components:

$$r_t^i=r_t^{\mathrm{step}}+r_t^{\mathrm{dist}}+r_t^{\mathrm{coop}}$$

(8)

The single-step reward follows standard grid-world logic: a penalty −1 per timestep for agents not at the goal to drive shorter-path exploration; a higher penalty −2 for stop actions to encourage movement and avoid stagnation; and a goal reward +50 upon reaching the target. Thus,

$$r_t^{\mathrm{step}}=\left\{\begin{array}{c}-1\\ -2\\ +50\end{array}\begin{array}{c}\\ \\ \end{array}\begin{array}{c}{a}_t^i=\mathrm{move}\\ a_t^i=\mathrm{stay}\\ p_{t+1}=G^i\end{array}\right.$$

(9)

where $p_{t+1}$ is the position of agent $i$ at time $t+1$; $G^i$ is the absolute target coordinates of agent $i$.

To alleviate sparse reward issues, a weighted Manhattan distance differential is adopted, providing continuous gradients and biasing toward main-lane movement to align with four-way shuttle topologies:

$$d_w\left(p,g\right)={\omega }_x\cdot \left|x_p-x_g\right|+{\omega }_y\cdot \left|y_p-y_g\right|$$

(10)

$$r_t^{\mathrm{dist}}=\kappa \cdot \left(d_w\left(p_{t+1}^i,g^i\right)-d_w\left(p_t^i,g^i\right)\right)$$

(11)

where $\kappa$ is a scaling factor, set to 0.125, and ${\omega }_x$, ${\omega }_y$ are horizontal/vertical coordinate weights, set to 1.5 and 0.5, respectively.

Traditional avoidance relies on severe post-collision penalties, but this approach has drawbacks: ambiguous attribution of collision actions (hindering precise strategy learning) and promotion of selfish behaviors that impede global convergence in multi-agent settings.

To solve this problem, we propose a Cooperative Avoidance Reward Mechanism (CARM) to facilitate the learning of proactive collision avoidance behaviors. The observation space of each agent is formulated as a partially observable discrete grid, confined to a local field of view centered on the agent itself. CARM consists of two functional components, with the core incentive rule defined as follows: when an agent is navigating the main lane and detects other agents ahead in its moving direction within this localized observation space (i.e., potential path overlap is identified), a positive immediate reward of +3 is conferred for executing lateral yielding actions—specifically, transitioning from the main lane to the adjacent sub-lanes to vacate the main passage for conflicting agents. This immediate reward helps incentivize proactive avoidance behaviors.

$$r_{\mathrm{inst}}^i\left(t\right)=+3$$

(12)

Furthermore, a 7-step action-reward buffer is maintained. If a potential deadlock resolves within 7 steps (i.e., no further conflict), the system backtracks to credit prior yielding actions with a high delayed reward, scaled by yielding steps to avoid prolonged passivity and resolve credit assignment. This delayed reward uses +5 as the base weight and dynamically adjusts the reward via a cumulative term of discount factors, thereby strengthening the incentive for effective avoidance.

$$r_{\mathrm{delay}}^i\left(t\right)=+5\cdot {\displaystyle \sum _{k=1}^K{s}^k}$$

(13)

where $K$ is the yielding step count, and $s$ a scaling factor, set to 0.9.

Thus, the cooperative reward is

$$r_t^{\mathrm{coop}}=r_{\mathrm{inst}}^i\left(t\right)+r_{\mathrm{delay}}^i\left(t\right)$$

(14)

2.6. PPO Training Objective and Advantage Estimation

PPO is employed as the RL algorithm, combined with Generalized Advantage Estimation (GAE) for advantage computation.

The importance ratio is

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a|o_{1:t}\right)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a|o_{1:t}\right)}}$$

(15)

where $\theta$ are policy parameters and $\varphi$ value parameters.

The clipped objective is

$$L^{\mathrm{CLIP}}\left(\theta \right)=E_t\left[\min \left(r_t\left(\theta \right){\widehat{A}}_t,clip\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t\right)\right]$$

(16)

where $\epsilon$ is the clip parameter.

The advantage is estimated as

$${\delta }_t=r_t+\lambda V_{\varphi }(s_{t+1})-V_{\varphi }\left(s_t\right)$$

(17)

$${\widehat{A}}_t={\displaystyle \sum _{l=0}^{\infty }{\left(\gamma \lambda \right)}^l{\delta }_{t+l}}$$

(18)

where ${\delta }_t=r_t+\lambda V_{\varphi }(s_{t+1})-V_{\varphi }\left(s_t\right)$ is the temporal-difference residual; $\lambda$ a scaling factor; $\gamma$ the GAE decay; and $V_{\varphi }\left(s_t\right)$ the value function.

The final objective is

$$L(\theta ,\varphi )=-L^{\mathrm{CLIP}}(\theta )+c\cdot {\rm E}{\left[V_{\varphi }\left(s_t\right)-R_t\right]}^2$$

(19)

where $R_t$ is the return at time $t$, and $c$ coefficients.

2.7. Deep Reinforcement Learning Framework

PPO serves as the core RL algorithm. In preprocessing, inspired by PRIMAL [19], we use the convolutional neural network (CNN) structure in Figure 4: 3 × 3 kernels initially extract features from 2-channel grid maps (two local observation subgraphs), yielding 64-dimensional feature maps for multi-scale fusion. Multi-scale CNNs (1 × 1, 3 × 3, 5 × 5 parallel kernels) extract spatial features, followed by ResBlock to prevent degradation. CBAM (Convolutional Block Attention Module) adaptively enhances focus on dynamic obstacles. Temporal features are processed via LSTM, outputting policy distributions and value estimates.

Figure 4. Deep reinforcement learning network.

This modular design extracts multi-scale spatial features, leverages residuals and attention for improved representation and generalization, and maintains temporal coherence for robust policy learning efficiently.

3. Results

3.1. Experimental Setup

Environment and Map Configuration.

All experiments are conducted in a discrete two-dimensional grid environment with a map size of 20 × 20, representing a simplified warehouse layout. Main lanes and sub-lanes are predefined according to the warehouse topology. Cargo locations in sub-lanes are uniformly randomly generated with a fixed density of 30%. Static obstacles are generated using a fixed random seed (seed = 41) to ensure reproducibility across different experimental runs. The maximum number of steps per episode is set to 50. An episode is regarded as a failure if the agent fail to reach the assigned target within this step limit.

Experimental Scenarios.

Two scenarios are designed to evaluate different aspects of the proposed method. To provide a clear visual representation of the experimental configurations, the spatial layouts and task logic for the two scenarios are illustrated in Figure 5. These schematics highlight the deterministic nature of the task instances, which allows for a focused analysis of conflict resolution and system efficiency.

Figure 5. Schematic of the experimental scenarios: (a) Configuration for Scenario 1, to evaluate the capability of collaborative collision avoidance in ablation studies; (b) Configuration for Scenario 2, demonstrating the continuous task flow and agent interaction.

(1)

Scenario 1 (Head-On Inevitable Conflict).

As shown in Figure 5a, two shuttles are initialized at opposite ends of the same main lane, with start positions at (10, 0) and (10, 19), and target positions at (10, 19) and (10, 0), respectively. Since both agents move toward each other along the same lane, a head-on conflict is inevitable if no avoidance behavior is adopted. This scenario represents a strong-conflict setting and is mainly used for ablation studies to validate the effectiveness of the proposed avoidance and coordination mechanisms. Scenario 1 is intentionally designed as a representative strong-conflict case to efficiently shape cooperative avoidance behaviors during training.

(2)

Scenario 2 (Warehouse Throughput Evaluation).

Figure 5b illustrates Scenario 2, which extends the first scenario to simulate continuous warehouse operations. Two shuttles operate under the same spatial configuration as in Scenario 1. Each shuttle is assigned 100 transportation tasks, with new tasks generated every 20 s, resulting in a total of 200 tasks. All tasks are generated at fixed pickup and delivery locations to ensure fair comparison across different methods. Two task types are considered: transportation from (10, 0) to (10, 19), and from (10, 19) to (10, 0). Throughput is measured as the number of completed tasks per unit time.

Training Protocol.

Policy training is performed using Proximal Policy Optimization (PPO). Each training configuration runs for 1000 iterations, with 4096 timesteps collected per iteration and 15 mini-batch updates for policy optimization. All hyperparameters are kept consistent across experiments and are summarized in Table 1.

Table 1. Reinforcement Learning Algorithm Parameters.

Parameter	Value	Description
Learning Rate	1 × 10−4	Step size of optimizer for parameter updates
Entropy Coefficient	0.05	Strength of entropy regularization
Training Batch Size	4096	Number of training samples per iteration for the learner
Value Function Loss Coefficient	0.8	Weight of value function loss in the total loss
Use GAE	True	Enable Generalized Advantage Estimation
Use Critic Network	True	Include value function network for evaluating state values
GAE Discount Factor (λ)	0.99	Discount factor for advantage estimation in GAE calculation
Return Discount Factor (γ)	0.995	Degree of emphasis on long-term rewards
Policy Clipping Parameter	0.2	Control the magnitude of policy updates
Value Function Clipping Parameter	3	Prevent excessive deviation in value estimation
Gradient Clipping Threshold	0.8	Avoid gradient explosion
Optimizer	Adam	Type of optimizer
Number of SGD Iterations	15	Number of SGD iterations performed per batch of data

Evaluation Protocol and Reproducibility.

To ensure statistical reliability, each experimental setting is repeated 10 times using different random seeds. Policy evaluation is conducted every 50 training iterations on a held-out test set consisting of 100 fixed initial states. Model checkpoints are saved periodically for performance comparison and analysis.

Implementation Details.

All experiments are implemented in Python 3.10 (Python Software Foundation, Wilmington, DE, USA) using PyTorch 2.1 (Meta Platforms, Inc., Menlo Park, CA, USA). The simulation environment is constructed based on Gymnastics v1.23 (Farama Foundation, New York, NY, USA), while distributed training and experiment management are supported by RayLib v2.41 (Anyscale, Inc., San Francisco, CA, USA). Experiments are executed on a workstation running Ubuntu 22.04 (64-bit) (Canonical Group Ltd., London, UK). The hardware configuration includes an Intel Core i9-14900K CPU (32 cores) (Intel Corporation, Santa Clara, CA, USA), an NVIDIA GeForce RTX 4090 GPU (NVIDIA Corporation, Santa Clara, CA, USA), and 128 GB of RAM (Samsung Electronics Co., Ltd., Suwon, Gyeonggi-do, Republic of Korea). All methods are evaluated under identical hardware conditions to ensure comparability.

3.2. Ablation Studies

To verify the contributions of the Distance Reward (DR) and Cooperative Reward (CR) modules to MAPF performance, we evaluate five ablation configurations in Scenario 1:

(1)

Full PPO: Vanilla PPO + CR + DR

(2)

VP + CR: Vanilla PPO + CR

(3)

VP + CP: Vanilla PPO + Collision Penalty

(4)

VP + DR: Vanilla PPO + Distance Reward

(5)

VP: Vanilla PPO

Training runs for 1000 iterations. Success rates are shown in Figure 6, and agent steps in Figure 7.

Figure 6. Success rates.

Figure 7. Agent steps. (a) Step count of Agent 1; (b) Step count of Agent 2.

The ablation results clearly demonstrate the distinct and complementary roles of the distance-based reward (DR) and the cooperative reward (CR) in multi-agent path finding under inevitable head-on conflict scenarios.

Vanilla PPO (VP) fails to converge throughout 1000 training iterations, with the success rate remaining near zero and the average step count stabilizing at a high level. This behavior indicates that, in strongly coupled multi-agent scenarios, sparse terminal rewards combined with per-step penalties are insufficient to provide meaningful learning signals. Agents are therefore prone to aimless exploration or convergence to locally optimal but globally infeasible behaviors.

By introducing distance rewards (VP + DR), the success rate increases to approximately 60% within the first 400 iterations, and the average step count is significantly reduced. This confirms that dense, Manhattan-distance-based shaping effectively mitigates reward sparsity and accelerates goal-oriented learning. However, the performance soon plateaus and exhibits noticeable fluctuations. This instability arises because DR primarily optimizes individual goal approaching behavior and does not explicitly encode inter-agent coordination. Consequently, agents tend to insist on forward motion in narrow main lanes, leading to persistent mutual blocking.

Both VP + CP and VP + CR reach success rates close to 90% within 200 iterations, highlighting the necessity of incorporating conflict-related signals into the reward design. Nevertheless, these two approaches differ fundamentally in training stability and long-term performance. The collision-penalty-based VP + CP adopts a reactive mechanism, in which negative rewards are triggered only after conflicts occur. Such penalties provide no anticipatory guidance and suffer from ambiguous credit assignment, often inducing overly conservative or erratic behaviors in later training stages, resulting in performance oscillations.

In contrast, VP + CR employs proactive positive incentives that explicitly reward yielding actions before collisions arise, complemented by delayed rewards that reinforce the long-term benefits of cooperation. This design leads to faster convergence, consistently low step counts, and significantly reduced variance. The results indicate that proactive cooperative rewards are substantially more effective than passive punishment mechanisms in fostering stable multi-agent coordination.

The Full PPO configuration achieves the best overall performance, maintaining a stable success rate of approximately 95% with the lowest average step count and minimal variance. These results confirm the strong complementarity between DR and CR: DR ensures navigation efficiency and accelerates convergence, while CR guarantees conflict resolvability through implicit collaboration. Together, they satisfy the dual objectives of efficiency and safety inherent to MAPF problems.

3.3. Comparative Experiments

To ensure the fairness and validity of comparative evaluations, all benchmarked methods are tested under identical experimental configurations, including shelf layout and simulation platform, with only the path planning and decision-making algorithms varied. This approach guarantees meaningful comparisons, free from biases introduced by platform discrepancies:

(1)

Baseline: A* with Rules—The grid-based A* algorithm with Manhattan distance heuristic [5] is employed, constrained by real-world operational rules. This baseline algorithm is fully implemented on the identical simulation platform as the proposed method, eliminating biases induced by platform differences.

(2)

Baseline: CBS (Conflict-Based Search)—The CBS algorithm [9], a well-established approach for multi-agent pathfinding (MAPF), is utilized as a second baseline. CBS works by decomposing the MAPF problem into individual agent pathfinding tasks, while resolving conflicts between paths through a hierarchical approach. At the first level, CBS solves each agent’s path using A*, and at the second level, it resolves conflicts between agents by iteratively adjusting the plans. This process ensures agents avoid collisions while satisfying global constraints. In our experiments, CBS provides a baseline for performance, as it remains one of the most effective algorithms for ensuring collision-free path planning in multi-agent systems.

(3)

Ours: Full PPO—Our proposed method, Full PPO, leverages deep reinforcement learning (DRL) for MAPF. This method is trained using PPO, which is known for its stability and efficiency in handling continuous action spaces, making it a robust choice for our path planning task. Post-training, the best policy is evaluated on Scenario 2 (10 repeats for statistical analysis). Full PPO is designed to reduce redundancy in agent paths, enabling flexible local yielding and adaptive decision-making, which leads to improved throughput compared to traditional rule-based methods.

To rigorously evaluate the performance of the proposed CARM framework, we select two representative baselines that encompass both academic optimality and industrial practicality.

First, we incorporate Conflict-Based Search, a foundational and optimal solver in the Multi-Agent Pathfinding (MAPF) domain. As a classical search-based method, CBS serves as a rigorous academic benchmark to assess the coordination efficiency and scalability of our learning-based approach.

Second, we employ A* with Rules as a primary practical baseline. While contemporary learning-based methods like PRIMAL [19] utilize imitation learning to bridge the gap between search and reinforcement learning, A* with Rules remains the de facto industrial standard for four-way shuttle systems due to its reliability and ease of deployment. In this study, A* with Rules is implemented with specific action constraints to ensure collision-free navigation. By comparing CARM against both a key academic solver (CBS) and a strong industrial baseline, we aim to demonstrate the theoretical superiority and the real-world application potential of our proposed method.

Post-training, the best policy from Full PPO is evaluated on Scenario 2. The results of the experiments are shown in Table 2 and Figure 8, where we compare the performance of A* with Rlues, CBS, and Full PPO across various metrics. Average steps, time, and throughput are measured to provide a comprehensive understanding of the relative advantages and disadvantages of each approach.

Table 2. Test Results for All Algorithms.

Metric	A* with Rules	CBS	Full PPO	Improvement (%) (Full PPO vs. A* with Rules)	Improvement (%) (Full PPO vs. CBS)
Avg. Steps	34.50 ± 0.0	24.1 ± 1.2	22.15 ± 2.1	35.8	8.1
Avg. Time (s)	3459 ± 0.0	2423 ± 20	2370 ± 25	31.5	2.2
Throughput (tasks/min)	0.0564 ± 0.0	0.0825 ± 0.001	0.0844 ± 0.002	49.7	2.4

Figure 8. Warehouse throughput performance comparison. (a) Completed tasks and (b) Completion rate, showing the efficiency improvements across different algorithms.

• Quantitative Results: To ensure the statistical reliability of the results, 10 repetitions were conducted for each experimental setup, and the results presented in the table are expressed as the mean ± standard deviation (SD). Different random seeds were adopted to ensure the robustness of the findings, thus, with the repeated experiments, providing a comprehensive characterization of the variability across experiments.
• Time (seconds) Calculation: The “time (seconds)” metric recorded in the table is computed based on the number of steps taken by the agents. In this experiment, we preset that each movement step consumes one second, and the total time is thus proportional to the total number of steps taken. This time-based calculation is used to reflect the throughput of the warehouse, as fewer steps indicate more efficient pathfinding and higher throughput.

The comparative experiments further validate the superiority of the proposed method at the system level, particularly in terms of efficiency and throughput. Compared with A* with Rules, Full PPO achieves equivalent reliability (≈95% success rate) while reducing average steps and total completion time by 35.8% and 31.5%, respectively, and increasing throughput by nearly 50%.

This performance gap stems from fundamentally different conflict resolution paradigms. A* with Rules relies on predefined heuristics and hard constraints to avoid collisions. Vehicles alternate transport tasks along fixed routes with deterministic travel times, and all task scheduling is fully predetermined. Consequently, repeated experiments yield identical results, resulting in zero standard deviation across runs. While deterministic and interpretable, such rigidity forces agents to adopt conservative detours or waiting behaviors, leading to path redundancy and underutilization of shared spatial resources in high-density environments.

CBS improves upon A* by explicitly resolving conflicts through hierarchical replanning, guaranteeing collision-free solutions. However, CBS relies on centralized conflict detection and repeated replanning, which limits its responsiveness and efficiency in continuous, online task execution scenarios. As reflected in the experimental results, CBS outperforms A* in efficiency but still falls short of Full PPO in both average steps and throughput.

Full PPO distinguishes itself by learning decentralized, context-aware coordination strategies. Through the Cooperative Avoidance Reward Mechanism (CARM), yielding actions are no longer treated as efficiency losses but as contributors to long-term system-level rewards. This “soft-constraint” formulation allows agents to dynamically negotiate shared space without invoking global replanning, thereby achieving superior throughput and scalability.

3.4. Scalability and Generalization Evaluation

Although the comparative and ablation experiments above mainly focus on the two-agent setting, real-world warehouse systems often involve multiple shuttles operating simultaneously under varying environment complexities. To further evaluate the scalability and generalization ability of the proposed method, we conduct additional experiments under increased numbers of agents and different cargo densities.

Based on the layout used in Scenario 1, the number of agents is extended to 2, 4, and 6. For each configuration, agents are initialized in symmetric start–goal pairs along multiple main lanes. Specifically, in the two-agent case, agents start at (10, 0) and (10, 19) with corresponding targets at (10, 19) and (10, 0). In the four-agent case, two pairs of agents are initialized at (10, 0) and (10, 19), and (3, 0) and (3, 19), with targets at (10, 19) and (10, 0), and (3, 19) and (3, 0), respectively. In the six-agent case, three pairs of agents are initialized at (10, 0) and (10, 19), (3, 0) and (3, 19), and (16, 0) and (16, 19), with corresponding targets at (10, 19) and (10, 0), (3, 19) and (3, 0), and (16, 19) and (16, 0).

To further investigate robustness under different environment complexities, the six-agent configuration is evaluated under three cargo density levels: 0.3, 0.6, and 0.9. All other environment settings, training protocols, and hyperparameters remain identical to those described in Section 3.1 to ensure fair comparison.

Additional experiments demonstrate that the proposed approach exhibits strong scalability and robustness under increasing agent density and environmental complexity. As illustrated in Figure 9a, when the cargo density is fixed at 0.3, the success rate remains consistently above 0.8 as the number of agents increases from 2 to 6, with only minor fluctuations. This indicates that the learned cooperative behaviors are not tightly coupled to a specific agent count and can generalize to more interaction-intensive scenarios.

Figure 9. Success rate curves under different agent quantities and cargo densities. (a) Success rate with varying agent numbers (cargo density = 0.3); (b) Success rate with varying cargo densities (agent number = 6), reflecting the policy’s performance under different interaction complexities and spatial constraints.

As shown in Figure 9b, when the number of agents is fixed at six, increasing the cargo density from 0.3 to 0.8 leads to a gradual decline in success rate, which nevertheless remains above 0.6 without catastrophic failure. This robustness can be attributed to the structural characteristics of four-way shuttle systems, where main lanes act as primary coordination corridors. As long as sufficient local vacancies exist near these lanes, agents can leverage them to resolve conflicts through learned yielding behaviors.

The dual structure of CARM plays a crucial role in this robustness. Immediate rewards guide agents to identify viable avoidance opportunities in real time, while delayed rewards reinforce the long-term benefits of orderly cooperation. As a result, the proposed method maintains stable performance even under tight spatial constraints, validating its potential for real-world deployment.

4. Discussion

This study systematically demonstrates that the proposed PPO-based MARL framework effectively addresses the deadlock-prone nature of multi-four-way shuttle systems in high-density warehouses. By integrating topology-aware distance rewards with the Cooperative Avoidance Reward Mechanism (CARM), the framework overcomes two long-standing challenges in Multi-Agent Pathfinding (MAPF): sparse rewards and ambiguous credit assignment.

A primary finding of this research is the performance gap between classical search-based methods and the proposed learning-based approach. Our comparative analysis includes Conflict-Based Search (CBS), a foundational optimal solver in the MAPF domain. While CBS provides a theoretical upper bound for path optimality in sparse scenarios, its computational overhead grows exponentially as agent density increases, often leading to search failures in complex warehouse grids. In contrast, CARM demonstrates superior scalability and real-time adaptability. By learning emergent coordination strategies, the framework maintains high system throughput in high-conflict environments where classical solvers struggle to converge.

Furthermore, the comparison with A* with Rules highlights the practical industrial value of our method. As the de facto standard in current automated warehousing, A* with Rules relies on handcrafted heuristics to resolve conflicts. While reliable, these rules are often overly conservative, leading to unnecessary wait times. CARM transcends these limitations by evolving more fluid and proactive yielding behaviors. While we acknowledge that specialized MARL architectures—such as MAPPO, IPPO, or QMIX—represent the cutting edge of collaborative decision-making, CARM adopts a streamlined, reward-centric approach. It achieves highly effective cooperative control without the need for complex centralized mixing networks or causal inference modules, offering a more interpretable and computationally efficient solution for large-scale shuttle systems.

At a mechanistic level, it is crucial to emphasize that these efficiency gains are achieved without relaxing any hard safety constraints. CARM adheres to the same strict kinematic and collision-avoidance protocols as the baselines. Instead of simplifying the physical problem, it introduces immediate and delayed reward signals that guide agents to prioritize global coordination over local distance minimization. This explains the framework’s ability to resolve head-on conflicts and complex deadlocks that typically paralyze rule-based systems.

Nevertheless, this study has limitations. Although the proposed reward design enables stable collaboration, a direct quantitative benchmarking against a full suite of state-of-the-art MARL architectures remains a primary direction for future work. Additionally, the current experiments are conducted in static environments. Future research will focus on incorporating attention-based architectures for long-range dependency modeling and exploring the framework’s robustness against real-world uncertainties such as sensor noise and communication delays.

5. Conclusions

This paper presents a robust PPO-based multi-agent reinforcement learning framework for multi-four-way shuttle systems, addressing head-on conflicts and deadlocks in high-density warehousing environments through a dual-component reward design. The proposed Cooperative Avoidance Reward Mechanism (CARM) integrates immediate yielding incentives with delayed cooperative returns, effectively resolving the credit assignment problem and enabling the stable emergence of implicit collaboration without expert demonstrations.

Extensive experiments demonstrate that the proposed approach significantly outperforms both the industry-standard A* with Rules and the academic benchmark Conflict-Based Search (CBS). Specifically, CARM reduces average travel distance by 35.8% and improves system throughput by 49.7% compared to rule-based methods while maintaining a success rate exceeding 95%. Ablation and scalability studies further confirm the robustness and generalization capability of the learned policies under increasing agent numbers and environmental complexity.

Despite these promising results, we acknowledge that this study focuses primarily on bridging the gap between classical search-based methods and decentralized reinforcement learning. A direct comparative analysis against a broader range of specialized MARL architectures, such as MAPPO, IPPO, or QMIX, remains a limitation of the current work. Future research will focus on benchmarking CARM against these state-of-the-art MARL baselines and exploring its transferability to even larger-scale, heterogeneous multi-robot systems.

In summary, this work provides an efficient, decentralized control paradigm that balances individual navigation efficiency and global coordination. Beyond four-way shuttle systems, the proposed reward design framework offers valuable insights for broader multi-robot coordination problems, including autonomous vehicle fleets and robotic swarms.