Reinforcement-Learning-Driven Improved MOEA/D Algorithm for Configuration Optimization of Four-Way Shuttle Warehouse Systems

Abstract

The four-way shuttle warehouse system (FWSWS), as a form of high-density automated warehousing equipment, is characterized by high storage density, high space utilization, and operational flexibility. The initial configuration of this warehousing system is constrained by spatial layout and equipment resources, exhibiting discrete, strongly coupled, and nonlinear characteristics. When solving such complex constrained problems, existing algorithms like MOEA/D and NSGA-II generally suffer from insufficient constraint-handling capabilities and low search efficiency for feasible solutions. To address these issues, this paper proposes the RL-IMOEA/D algorithm, integrating reinforcement learning and a knowledge-driven strategy (RL-IMOEA/D). This algorithm enhances the capability of searching for feasible solutions through a knowledge-driven feasible region repair mechanism and utilizes Q-learning to dynamically adjust the variable neighborhood search (VNS) strategy, thereby improving the convergence and diversity of the Pareto solution set. Experimental results show that, compared with other optimization algorithms, RL-IMOEA/D achieves an average hypervolume (HV) value of 1.2068—a 0.96% improvement over the standard MOEA/D baseline (1.1953)—while its inverted generational distance (IGD) value is reduced from 0.1487 to 0.0411. The proposed algorithm demonstrates superior performance in terms of convergence, solution diversity, and constraint-handling capability. The obtained configuration schemes satisfy the capacity and queueing stability constraints, verifying the effectiveness of the proposed algorithm in complex discrete optimization problems and providing an effective solution approach for the initial configuration optimization of four-way shuttle warehouse systems.

1. Introduction

In recent years, the four-way shuttle warehouse system (FWSWS) has become a mainstream solution for high-density warehousing due to its flexibility in intra-level transportation and adaptability across aisles. Unlike traditional automated storage and retrieval systems, the operational performance of FWSWS is strongly influenced by the coupling effects among warehouse space constraints, physical interference, and high-concurrency operations.

Research on the planning and optimization of automated warehouse systems has mainly focused on storage assignment, task scheduling, and path optimization. Cheng et al. [1] investigated the initial configuration optimization problem of cross-layer and cross-aisle shuttle warehouse systems. A multi-objective mathematical model was developed with the objectives of minimizing task completion time and system energy consumption, and an improved NSGA-II algorithm incorporating a clonal selection mechanism was proposed as a solution. Li [2] addressed the storage location assignment problem in four-way shuttle warehouse systems and developed an improved ant colony–greedy algorithm for optimization. Ren et al. [3] conducted an in-depth study on the integrated scheduling optimization problem of deep-lane cross-layer four-way shuttle systems and proposed a logistics equipment control strategy, considering both equipment selection and path planning. Queueing theory is a widely used analytical tool for evaluating resource competition, congestion propagation, and service capacity boundaries in discrete-event systems, and it has been applied to performance analysis of automated warehouse systems. Zhu [4] developed an open queueing network model for a mother–child shuttle system to analyze system operational characteristics under different storage strategies. Tian [5] further introduced a semi-open queueing network to characterize congestion behaviors caused by multi-node coupling in automated shuttle storage and retrieval systems. For shuttle-based storage and retrieval systems (SBS/RS), Ekren et al. [6] established a performance evaluation tool based on an open queueing network. Battarra et al. [7] developed a multi-queue single-server model for automated vehicle storage and retrieval systems (AVS/RS) to analyze storage and retrieval requests. Kumawat and Roy [8] proposed an approximate solution method for multi-stage semi-open queueing networks and applied it to the performance analysis of shuttle-based dense storage systems. However, existing studies mainly employ queueing models for posterior performance evaluation, while their integration into the configuration optimization process remains relatively limited. Regarding optimization algorithms, NSGA-II [9] and MOEA/D [10] provide classical frameworks for solving multi-objective combinatorial optimization problems. Zhou et al. [11] improved the MOEA/D algorithm to enhance the distribution performance of solutions for multi-aisle automated storage and retrieval system storage assignment problems.

However, several limitations remain in the existing studies regarding the initial hardware configuration of FWSWS: (1) the queueing performance indicators have rarely been integrated into the configuration optimization process to establish a closed-loop evaluation mechanism; and (2) existing multi-objective optimization algorithms tend to suffer from local search stagnation when solving highly constrained hardware configuration problems, while conventional local search strategies lack self-learning capability and environmental adaptability.

Accordingly, to address the aforementioned gaps, the primary research question of this study is how to efficiently search for feasible configuration schemes in a highly constrained, discrete decision space while ensuring queueing stability. To this end, a multi-objective collaborative configuration optimization model for FWSWS is established by integrating three-dimensional spatial interference constraints and M/G/c queueing steady-state constraints. Furthermore, an RL-IMOEA/D algorithm is proposed. The main methodological novelty lies in the deep integration of a Q-learning mechanism with a domain-knowledge-driven heuristic repair strategy. Unlike existing RL-assisted evolutionary algorithms that treat the discrete search space as a black box, the proposed RL-IMOEA/D intelligently selects search operators based on the evolutionary state and utilizes an infeasible-region mapping function to actively repair lethal solutions violating M/G/c queueing constraints. This approach significantly improves the exploration capability toward the Pareto front and enhances convergence accuracy.

The remainder of this paper is organized as follows: Section 2 describes the system layout and establishes the M/G/c queueing network model. Section 3 constructs the multi-objective configuration optimization model. Section 4 details the design of the proposed RL-IMOEA/D algorithm. Section 5 presents the experimental simulation and algorithm performance verification. Finally, Section 6 concludes the paper and discusses future research directions.

2. Overview of the Shuttle Warehouse System

2.1. System Layout

The four-way shuttle warehouse system (FWSWS) is a typical high-density, highly flexible “parts-to-picker” automated storage and retrieval system, and its physical layout structure is illustrated in Figure 1. To enhance storage and retrieval efficiency and eliminate interference caused by cargo relocation, this paper adopts a single deep configuration mode, ensuring that all storage locations are directly adjacent to the running tracks.

Figure 1. Structure diagram of the four-way shuttle warehouse system: (a) top view; (b) side view.

2.2. Inbound and Outbound Operation Process

Based on the actual operational logic of the system, the core outbound operation process consists of four main stages (as shown in Figure 2):

Figure 2. Outbound process flow of the four-way shuttle warehouse system.

The outbound operation process consists of four stages:

(1) Resource allocation: The warehouse control system (WCS) assigns tasks and monitors available shuttles and lifts.

(2) Level transfer and routing: The selected shuttle moves to the target level and performs conflict-free routing.

(3) Storage/retrieval operation: The shuttle retrieves the target bin.

(4) Return and delivery: The shuttle transports the bin to the lift, which delivers it to the picking station.

2.3. Symbol Definitions and 3D Coordinate Mapping

To accurately evaluate the system performance, the physical layout of the warehouse is abstracted into a three-dimensional spatial model in this paper. Let the origin of the coordinates at the bottom-layer lift transfer point be set as O(0, 0, 0), and the main system parameters are defined in Table 1.

Table 1. Main parameter description.

Parameter Category	Parameter Symbol	Parameter Description
Spatial structure parameters	WY, HZ, LX	Total width, total height, and total length of the storage racks
	C, R, T	Number of columns, rows, and tiers of the racks along the X, Y, and Z axes
	w, l, h	Width, depth, and height of a single storage location unit
Kinematic parameters	vx, vy, vz	Maximum travel velocity of the shuttle along the X and Y axes, and of the lift along the Z axis
	ax, ay, az	Constant acceleration/deceleration of the shuttle along the X and Y axes and of the lift along the Z axis
Operation time constants	tturn	Direction-changing time of the shuttle via wheel lifting at aisle intersections
	tp/d	Fixed time for the shuttle to pick up/drop off a bin at a storage location
	ttransfer	Cargo transfer time between the lift and the shuttle
System configuration and demand	Ns, Nl	Number of four-way shuttles and lifts configured in the system
	Qdemand	Minimum total storage capacity requirement of the warehousing system
	λ	Average arrival rate of inbound/outbound tasks from outside the system (throughput demand)
Queueing network parameters	μs, μl, μp	Average service rate per single equipment/worker for shuttles, lifts, and picking stations
	ρs, ρl, ρp	Steady-state service utilization rate of the shuttle, lift, and picking station nodes

Based on these parameters, any target storage location can be uniquely represented as K(x, y, z). The absolute horizontal distance for the shuttle traveling from the transfer point (0, 0, z) on the z-th layer to the target storage location is calculated as:

$$d_x=x\cdot w$$

(1)

$$d_y=y\cdot l+\left[{\displaystyle \frac{y}{2}}\right]\cdot W_{aisle}$$

(2)

The absolute vertical distance for the lift to travel from the bottom tier to the target tier z is:

$$d_z=\left(z-1\right)\cdot h$$

(3)

2.4. Optimal Solution Database Generation

2.4.1. Equipment Kinematics Model

When executing handling tasks, the kinematic characteristics of both the four-way shuttle and the lift follow the classic piecewise acceleration and deceleration profile. Let d denote the target travel distance of the equipment in any single dimension; the equipment travel time t(d) thus exhibits a typical piecewise characteristic, as illustrated in Figure 3.

Figure 3. Kinematic laws of equipment: (a) triangular velocity profile; (b) trapezoidal velocity profile.

As illustrated in Figure 3, when the target travel distance is relatively short (i.e, d ≤ v²_max/a), the equipment must initiate the deceleration phase before reaching its maximum speed limit, as shown in (a). Conversely, when the target distance is sufficiently long (i.e, d ≥ v²_max/a), the equipment can accelerate to the system’s speed limit and maintain a uniform velocity phase for a certain duration, as depicted in (b). The area under the velocity–time curve represents the actual travel displacement d of the equipment.

2.4.2. Expected Time Model for Single-Command Operations

(1)

Expected horizontal service time of the four-way shuttle:

In a single outbound task, the shuttle must cover the travel distance to the target location (x,y) and execute steering at intersections. Using double integration with the probability density function of a uniform distribution on a 2D plane, the expected travel time E(T_s) is:

$$E\left(T_s\right)={\displaystyle {\int }_0^{X_{\max }}{\displaystyle {\int }_0^{Y_{\max }}\left[2t_x\left(x\right)+2t_y\left(y\right)+2t_{\mathrm{turn}}\right]}}f\left(x,y\right)dydx+t_{\mathrm{p}/\mathrm{d}}$$

(4)

where t_x(x) is the travel time along the X-axis, t_y(y) is the travel time along the Y-axis, and 2t_x(x) + 2t_y(y) represents the horizontal travel time for the outbound and return trips.

(2)

Expected vertical travel time of the lift:

The single-trip expected travel time for the lift from the bottom tier to the target tier z is expressed via discrete expectation summation:

$$E\left(T_l\right)={\displaystyle \sum _{z=1}^T\left[t_z\left(\left(z-1\right)h\right)+t_{transfer}\right]}{P}_z$$

(5)

where P(z) is the probability of the target being on the z-th tier; in this study, we assume random storage, meaning P(z) follows a uniform distribution across all tiers (i.e., P(z) = 1/T). t_z((z−1)h) is the vertical travel time.

2.5. Queueing Network Model

In high-concurrency scenarios, equipment serves as shared bottleneck resources. To evaluate the actual order completion cycle, including queueing waiting times, this paper constructs a multi-server tandem queueing model based on the “decoupling assumption.”

2.5.1. Queueing Network Structure

The outbound/inbound processes are abstracted into a three-stage decoupled tandem queueing network consisting of shuttles, lifts, and picking stations (as shown in Figure 4):

Figure 4. Diagram of the three-stage decoupled tandem queueing network model.

(1)

Shuttle Node (M/G/N_s): Task arrivals follow a Poisson distribution with parameter λ. N_s shuttles act as parallel servers; service times follow a general distribution with an expected service time of E(T_s).

(2)

Lift Node (M/G/N_l): N_l lifts are considered a queueing system with Nl parallel servers. Service times follow a general distribution with expectation E(T_l).

(3)

Picking Station Node (M/G/N_p): N_p picking stations process totes from the bottom buffer zone. Manual picking service times follow a general distribution with an expected time denoted as t_pick.

2.5.2. Calculation of Expected Order Completion Cycle

For analytical tractability in the optimization model, we assume that task arrivals follow a Poisson process (i.e., squared coefficient of variation C_a² = 1) and service times approximate an exponential distribution (C_s² ≈ 1). Consequently, the variance multiplier (C_a² + C_s²)/2 simplifies to 1. Using this specific case of the Allen–Cunneen approximation method, the average queue waiting times at the shuttle node E(W_s) and the lift node E(W_l) are expressed as:

$$E\left(W_s\right)=\left({\displaystyle \frac{{{\rho }_s}^{\sqrt{2\left(N_s+1\right)}-1}}{N_s\cdot \left(1-{\rho }_s\right)}}\right)\cdot 2E\left(T_s\right)$$

(6)

$$E\left(W_l\right)=\left({\displaystyle \frac{{{\rho }_l}^{\sqrt{2\left(N_l+1\right)}-1}}{N_l\cdot \left(1-{\rho }_l\right)}}\right)\cdot 2E\left(T_l\right)$$

(7)

Based on the independence after decoupling, the expected completion cycle for a single outbound task in the system is expressed as the linear superposition of the waiting times and service times at each node:

$$CT=E\left(W_s\right)+2\cdot E\left(T_s\right)+E\left(W_l\right)+2\cdot E\left(T_l\right)+t_{pick}$$

(8)

To ensure the queueing system reaches a steady state, the utilization rates of the first two core resources must be strictly less than 1. This forms the physical upper limit for the system’s task arrival rate λ.

$${\rho }_s={\displaystyle \frac{\lambda \cdot 2E\left(T_s\right)}{N_s}}<1, {\rho }_l={\displaystyle \frac{\lambda \cdot 2E\left(T_i\right)}{N_l}}<1$$

(9)

3. Construction of Multi-Objective Configuration Optimization Model

3.1. Decision Variables and Parameter Definitions

The core of system configuration optimization lies in finding the optimal 3D physical dimensions and a reasonable investment in handling equipment. The system configuration is defined as a five-dimensional discrete decision variable vector X = [x₁,x₂,x₃,x₄,x₅]^T, representing the number of rack rows, columns, tiers, shuttles, and lifts, respectively.

3.2. Baseline Parameter Settings

To ensure the scientific validity and solvability of the multi-objective optimization model, the boundary parameters and benchmark cost coefficients were extracted and calibrated based on a typical tote-level four-way shuttle automated storage and retrieval system (AS/RS) engineering case, as listed in Table 2.

Table 2. Benchmark parameters of the FWSWS.

Parameter Category	Symbol	Description	Baseline Value	Unit
Space Parameters	w, l, h	Unit width, depth, and height of a single cargo location	0.6, 0.8, 0.5	m
	Waisle	Width of the shuttle running aisle	1.0	m
Motion Parameters	vx, vy, vz	Max horizontal speeds of shuttle, max vertical speed of lift	2.0, 2.0, 3.0	m/s
	ax, ay, az	Horizontal acceleration of shuttle, vertical acceleration of lift	1.5, 1.5, 2.0	m/s2
	tturn, tp/d	Shuttle turning time, fixed picking/dropping time	3.0, 3.0	s
	cshuttle	Comprehensive procurement and implementation cost per shuttle	120,000	CNY/Unit
Cost Parameters	clift	Comprehensive procurement and implementation cost per lift	180,000	CNY/Unit
	cshelf	Steel structure manufacturing cost per standard location	150	CNY/Location
	cland	Land cost per unit area of facility (including infrastructure amortization)	2500	CNY/m2
Engineering Constraints	Lmax, Wmax	Max available internal length and width of the warehouse	40, 20	m
	Hmax	Max available internal height of the warehouse	12	m
	Kratio	Engineering empirical upper limit for shuttle-to-lift ratio	5	-

3.3. Objective Function Settings

3.3.1. Objective 1: Minimize System Comprehensive Construction Cost

The system’s initial investment cost mainly consists of rack structure costs, automated equipment procurement costs, and warehouse footprint costs. To simplify the model, the footprint of lift terminal accessories and maintenance aisles is not separately calculated.

(1)

Rack structure cost: Proportional to the total number of storage locations:

$$C_1=c_{\mathrm{shelf}}\cdot \left(x_1\cdot x_2\cdot x_3\right)$$

(10)

(2)

Equipment procurement cost: Determined by the number of shuttles and lifts:

$$C_2=c_{\mathrm{shuttle}}\cdot x_4+c_{\mathrm{lift}}\cdot x_5$$

(11)

(3)

Footprint cost: The projected area on the X-Y plane is determined by the unit dimensions and aisle width. The total projected area and cost are:

$$S=\left(x_2\cdot w\right)\cdot \left(x_1\cdot l+\left[{\displaystyle \frac{x_1}{2}}\right]\cdot W_{\mathrm{aisle}}\right)$$

(12)

$$C_3=c_{land}\cdot S$$

(13)

In summary, the objective function F₁(X) for minimizing construction costs is:

$$\min F_1\left(X\right)=C_1+C_2+C_3$$

(14)

3.3.2. Objective 2: Minimization of Expected Order Completion Time

Based on the three-stage decoupled tandem queueing network model established in Section 2, the expected order completion cycle is affected by the nonlinear coupling of the 3D rack dimensions and the equipment quantities. Therefore, the objective function F₂(X) represents the expected system completion cycle CT_sys.

$$\min F_2\left(X\right)=C{T}_{sys}\left(x_1,x_2,x_3,x_4,x_5\right)$$

(15)

$$C{T}_{sys}=\left[E\left(W_s\right)+2\cdot E\left(T_s\right)\right]+\left[E\left(W_l\right)+2\cdot E\left(T_l\right)\right]+\left[E\left(W_p\right)+t_{pick}\right]$$

(16)

3.4. Constraint Conditions

To ensure the engineering feasibility of the configuration scheme, the vector X must strictly satisfy the following four core constraints:

(1)

Storage capacity constraint: Total designated locations must meet the minimum demand Q_demand:

$$x_1\cdot x_2\cdot x_3\ge Q_{demand}$$

(17)

(2)

Physical space constraint: The 3D dimensions must not exceed the maximum allowed physical boundaries of the facility:

$$\left\{\begin{array}{l}{x}_2\cdot w\le L_{\max }\\ x_1\cdot l\cdot \left[{\displaystyle \frac{x_1}{2}}\right]\cdot W_{aisle}\le W_{\max }\\ x_3\cdot h\le H_{\max }\end{array}\right.$$

(18)

(3)

Equipment logic ratio constraint: To prevent over-configuration of a single type of equipment forming an absolute bottleneck, the shuttle-to-lift ratio must not exceed K_ratio:

$$1\le {\displaystyle \frac{x_4}{x_5}}\le K_{ratio}$$

(19)

(4)

Variable attribute constraint: Rack dimensions and equipment quantities are indivisible physical entities; decision variables must be strictly positive integers:

$$x_i\in Z^{+},i=1,2,3,4,5$$

(20)

4. Algorithm Design

This model is a highly constrained Mixed-Integer Nonlinear Programming (MINLP) problem. Traditional algorithms tend to generate massive lethal solutions violating the queueing steady state. To address these challenges, this chapter proposes an improved MOEA/D algorithm (RL-IMOEA/D) that integrates heuristic constraint repair, Q-learning-based operator scheduling, and an adaptive weight adjustment mechanism.

4.1. Tchebycheff Scalarization

Standard MOEA/D uses uniform weight vectors to decompose multi-objectives. Given the stark magnitude disparities and non-convex Pareto front characteristics of this model, the Tchebycheff decomposition method is applied. The scalarization function is defined as:

$$\begin{array}{l}\min g^{te}\left(x|{\lambda }^i,z^{*}\right)=\underset{1\le j\le 2}{\max }\left\{{\lambda }_j^i|F_j\left(x\right)-z_j^{*}|\right\}\\ s.t.x\in \mathsf{\Omega }\end{array}$$

(21)

where x is the 5D decision variable vector X; λⁱ = (λ₁ⁱ,λ₂ⁱ)^T is the weight vector for the i-th subproblem; F_j(x) represents the bi-objective functions; zⁱ = (z₁*,z₂*)^T is the ideal reference point comprising historical minimums of the population; and Ω is the discrete feasible solution space satisfying constraints.

4.2. Design of Heuristic Constraint Repair Strategy in Discrete Space

The decision variables of the proposed system are indivisible positive integers. To balance the exploration capability in continuous space and engineering feasibility, a hybrid mechanism combining real-number encoding, integer mapping, and heuristic repair is developed. First, the SBX crossover and polynomial mutation operators are employed to generate continuous offspring, which are then mapped into the discrete integer space through rounding before evaluation. For infeasible solutions caused by random perturbations, a heuristic repair strategy is designed: boundary truncation is first applied to variables exceeding spatial limits; the rack rows/columns are then preferentially adjusted to satisfy capacity constraints; subsequently, the number of devices is increased to restore queuing stability constraints (ρ < 1). Finally, the shuttle-to-lift ratio is corrected to meet engineering limitations. This mechanism effectively reduces invalid searches and ensures that all individuals entering the evaluation process satisfy the predefined constraints.

4.3. Design of Local Search and Adaptive Weight Adjustment Strategies

Conventional variable neighborhood search (VNS) generally employs a single perturbation operator triggered with a fixed probability, lacking the capability to perceive different evolutionary stages of the population and to adaptively schedule search strategies. To address this limitation, a Q-learning-based reinforcement learning mechanism integrated with an adaptive weight adjustment strategy is incorporated into the proposed algorithm.

4.3.1. Q-Learning State and Action Space Design

(1) State space: The state S is defined to represent the evolutionary stage of the algorithm. It is obtained by discretizing the ratio between the current generation g and the maximum number of generations G_max, i.e.,:

$$S=\left(s_{early},s_{mid},s_{late}\right)$$

(22)

$$s_t=\left({\displaystyle \frac{g}{G_{\max }}}\times 3\right)$$

(23)

This formulation divides the evolutionary process into three discrete stages (early, middle, and late), enabling the learning agent to perceive the global evolutionary progress.

(2) Action Space: The action A is defined as the candidate set of mutation and variable neighborhood search (VNS) operators available to the learning agent:

$$A=\left(a_1,a_2,a_3\right)$$

(24)

Action a₁ (Congestion Alleviation Operator): Reduces the number of rack levels (x₃) while increasing the numbers of shuttles (x₄) and lifts (x₅) to improve system throughput under high-load conditions.

Action a₂ (Cost Compression Operator): Removes redundant operating equipment (x₄ and x₅) while maintaining the queue stability constraint (ρ < 1) to reduce the overall system cost.

Action a₃ (Structural Mutation Operator): Randomly perturbs the numbers of rack rows (x₁) and columns (x₂) with adaptive step sizes to escape local optima.

4.3.2. Reward Function and Q-Table Updating

(1)

Reward Function:

An asymmetric reward–penalty mechanism is adopted to evaluate the effectiveness of action a_t executed under state s_t. The improvement effect of each action is evaluated based on the change in the Chebyshev scalar value g^te before and after execution.

$$R\left(s_t,a_t\right)=\left\{\begin{array}{l}+1.0,{\mathrm{if}\mathrm{g}}^{te}\left(x_{\mathrm{new}}\right)<{\mathrm{g}}^{te}\left(x_{\mathrm{old}}\right)\\ -0.5,{\mathrm{if}\mathrm{g}}^{te}\left(x_{\mathrm{new}}\right)\ge {\mathrm{g}}^{te}\left(x_{\mathrm{old}}\right)\end{array}\right\}$$

(25)

This reward formulation ensures that only actions leading to substantial improvements in solution quality receive positive rewards, whereas actions resulting in stagnation or deterioration are penalized.

(2)

Q-Table Update Mechanism:

An ε-greedy policy is employed to balance exploration and exploitation during action selection. The learning agent updates the action-value matrix asynchronously using the Temporal Difference (TD) learning rule:

$$Q\left(s_t,a_t\right)\leftarrow Q\left(s_t,a_t\right)+\alpha \left[R\left(s_t,a_t\right)+\gamma \underset{a}{\max }Q\left(s_{t+1},a_t\right)-Q\left(s_t,a_t\right)\right]$$

(26)

where α denotes the learning rate and γ denotes the discount factor. With the reinforcement learning mechanism, the search operator can be adaptively selected to match the current evolutionary stage of the population.

4.4. Algorithm Flow

By integrating adaptive weight adjustment, heuristic repair, and Q-learning-based operator scheduling, the proposed RL-IMOEA/D algorithm establishes a closed-loop optimization framework that combines continuous offspring generation with mandatory feasibility repair, autonomous operator selection toward Pareto-optimal regions, and dynamic weight adaptation for disconnected Pareto front completion. The overall workflow of the proposed algorithm is illustrated in Figure 5.

Figure 5. Overall flowchart of the improved MOEA/D algorithm.

(1)

Initialization and state perception stage:

The population, weight vectors, and Q-table are initialized. During the main evolutionary loop, the current evolutionary state (early, middle, or late stage) is dynamically identified, providing temporal information for subsequent intelligent operator scheduling.

(2)

Intelligent operator scheduling and heuristic repair stage:

The learning agent adaptively selects mutation operators according to the current evolutionary state to generate offspring solutions. Subsequently, domain-knowledge-based mapping is employed to perform strict spatial and queue-stability repairs, forcing infeasible solutions caused by deadlock constraints back into the feasible region.

(3)

Evaluation and self-learning feedback stage:

The local neighborhoods and external archive (Archive) are updated according to the Chebyshev scalarization value. Meanwhile, the improvement in solution quality is used as the reward signal to update the Q-table, enabling the algorithm to acquire environmental self-learning capability.

(4)

Dynamic weight adjustment and population evolution stage:

The sparsity of the Archive is periodically monitored, and the weight vectors are dynamically adjusted to guide the population toward unexplored regions of the Pareto front. The evolutionary process continues until the maximum number of iterations is reached, after which the Pareto-optimal configuration solution set is obtained.

5. Experimental Simulation

5.1. Case Study Parameters

This simulation is based on a real parts line-side warehouse project provided by a cooperating enterprise. The available spatial constraints of the warehouse are limited to L_max = 40 m, W_max = 20 m, and H_max = 12 m. The rigid operational requirements include a storage capacity of Q_demand ≥ 10,000 and a throughput of λ = 400 bins/h. The remaining kinematic and cost parameters of the equipment are strictly consistent with the baseline values defined in Section 3.

Simulation Environment and Algorithm Control Parameters

The simulation platform was developed in Python 3.9 using an object-oriented programming framework. The hardware environment consisted of an Intel Core i7-12700H processor, 16 GB memory, and the Windows 11 operating system. The common parameters of the standard MOEA/D and NSGA-II algorithms were uniformly configured as follows: population size N = 100, maximum generation number Gen_max = 500, crossover probability P_c = 0.9 and mutation probability P_m = 0.1.

For the parameters specific to RL-IMOEA/D, sensitivity experiments (Figure 6) were conducted to determine appropriate parameter settings. As shown in (a), p_vns = 0.2 provides a good balance between convergence speed and optimization performance, while further increases bring limited improvement. As shown in (b), T_weight = 50 achieves smoother convergence compared with excessively frequent or sparse weight updates. Therefore, p_vns = 0.2 and T_weight = 50 are selected for subsequent experiments.

Figure 6. Pre-experiments on core parameters of the I-MOEA/D algorithm (a) sensitivity analysis of P_vns; (b) convergence analysis of T_weight.

To ensure a fair comparison, all baseline algorithms were subjected to an identical penalty-based constraint-handling system. Solutions violating the physical boundaries or queueing stability constraints (ρ ≥ 1) were heavily penalized to eliminate them from the Pareto front, while the specific domain-knowledge heuristic repair mechanism was exclusively deployed in our proposed RL-IMOEA/D.

5.2. Evaluation Metrics and Algorithm Performance Verification

To evaluate the convergence performance and solution distribution capability of the algorithm, two metrics, inverted generational distance (IGD) and hypervolume (HV), are introduced. Five representative multi-objective optimization algorithms, including NSGA-II, NSGA-III, SPEA2, MOPSO, and the standard MOEA/D, are selected for comparison. Under the same constraints and computational budget, all six algorithms are independently executed 30 times, and the corresponding statistical results are presented in Table 3.

Table 3. Statistical results of IGD and HV indicators for six algorithms over 30 independent runs.

Evaluation Metric	Statistic	NSGA-II	NSGA-III	SPEA2	MOPSO	Standard MOEA/D	RL-IMOEA/D
HV (Maximize)	Mean	1.2004	1.2009	1.2003	1.1783	1.1953	1.2068
	Std	0.0003	0.0008	0.0005	0.0351	0.0033	0.0016
IGD (Minimize)	Mean	0.1388	0.1644	0.1387	0.1595	0.1487	0.0411
	Std	0.0002	0.0005	0.0004	0.0217	0.0044	0.0152

Comprehensive Performance Analysis:

(1)

Hypervolume (HV) Analysis for Front Coverage Capability:

RL-IMOEA/D achieves the best performance in terms of the HV metric, with an average value of 1.2068, representing improvements of 0.96% and 0.49% over the standard MOEA/D (1.1953) and NSGA-III (1.2009), respectively. Although the improvement is relatively limited, further enhancement becomes increasingly difficult when the HV value is already at a high level. This result demonstrates that the proposed adaptive weight adjustment strategy contributes to improving the coverage capability and distribution quality of the Pareto solution set.

(2)

Inverted Generational Distance (IGD) Analysis for Convergence Accuracy:

Regarding the IGD metric, RL-IMOEA/D achieves the best performance with an average value of 0.0411, reducing IGD by approximately 70.4% compared with SPEA2 (0.1387) and 72.4% compared with MOEA/D (0.1487). This demonstrates that the proposed algorithm provides a more accurate approximation to the Pareto front. The improvement is mainly attributed to the knowledge-driven feasible-region repair mechanism and the Q-learning-based VNS operator selection strategy, which enhance feasible solution quality and local search capability under complex constraints.

5.3. Ablation Study of Algorithmic Components

To quantitatively evaluate the individual contribution of each proposed mechanism in the RL-IMOEA/D algorithm, an ablation study was conducted. We compared the standard MOEA/D with three algorithm variants: MOEA/D with heuristic repair only (MOEA/D-HR), MOEA/D with Q-learning-based VNS (MOEA/D-QL), and MOEA/D with adaptive weight adjustment (MOEA/D-AW). The statistical results of the HV and IGD metrics over 30 independent runs are presented in Table 4.

Table 4. Performance comparison of different algorithm components in the ablation study.

Evaluation Metric	Statistic	Standard MOEA/D	MOEA/D-HR	MOEA/D-QL	MOEA/D-AW	RL-IMOEA/D
HV (Maximize)	Mean	1.1953	1.2039	1.2079	1.1997	1.2068
	Std	0.0033	0.0019	0.0009	0.0009	0.0016
IGD (Minimize)	Mean	0.1487	0.1807	0.0532	0.2064	0.0411
	Std	0.0044	0.0066	0.0167	0.0021	0.0152

As shown in Table 4, the standard MOEA/D performs poorly due to the generation of lethal solutions. The addition of the heuristic repair mechanism (MOEA/D-HR) significantly improves the valid solution rate, leading to a primary reduction in the IGD value. The incorporation of Q-learning (MOEA/D-QL) further enhances convergence speed, while the adaptive weight adjustment (MOEA/D-AW) mainly contributes to the uniform distribution of the Pareto front (reflected in a higher HV value). The complete RL-IMOEA/D algorithm achieves the best comprehensive performance, demonstrating that these components are highly complementary.

5.4. Multi-Objective Decision-Making and Mechanism Analysis of System Configuration

The Pareto solution set obtained by the RL-IMOEA/D algorithm is mapped into the objective function space. To analyze the trade-off relationship between system construction cost and order completion cycle, a three-dimensional density surface of the Pareto solution set and a two-dimensional Pareto front distribution are constructed, as shown in Figure 7.

Figure 7. Dual-perspective distribution plot of the Pareto front. (a) 3D view; (b) 2D view.

As shown in (a), the Pareto solutions obtained by RL-IMOEA/D are mainly concentrated in the frontier region of the objective space, indicating that the proposed algorithm can generate a stable set of non-dominated solutions. Image (b) demonstrates an apparent trade-off relationship between system construction cost and order completion cycle. Compared with NSGA-II and MOEA/D, the Pareto front obtained by RL-IMOEA/D exhibits better convergence and diversity. Based on the obtained Pareto front, three representative configurations (Solutions A, B, and C) are selected for engineering-oriented inverse analysis, and the corresponding results are presented in Table 5.

Table 5. Detailed inversion data of typical Pareto-optimal configuration schemes.

Core Variables and Metrics	Scheme A (Cost Priority)	Scheme B (Balanced)	Scheme C (Efficiency Priority)
Rack Rows x1	11	12	11
Rack Columns x2	38	35	38
Rack Tiers x3	24 (Max height limit)	24	24
Shuttles x4	4	6	18
Lifts x5	1	2	7
Actual Capacity Q	10032	10080	10032
Cost F1 (10k CNY)	300.8	341.1	576.8
Expected Cycle F2 (s)	193.8	66.4	53.7
Shuttle Utilization ρs	0.88 (High load)	0.58	0.22 (Low load)
Lift Utilization ρl	0.95 (Near congestion)	0.52	0.18

(1)

Scheme A: Cost-Oriented Configuration (Cost Reduction)

Solution A is located in the low-cost region of the Pareto front, adopting an 11-row × 38-column × 24-layer storage layout with four shuttles and one lift. The total investment is 3.008 million yuan, representing the lowest cost among all candidate solutions. However, the single-lift configuration results in a high utilization rate of ρ_l = 0.95, indicating that the system operates near the queue stability limit. Consequently, the order completion cycle increases to 193.8 s due to significant lift waiting time. This solution minimizes construction cost but is more prone to congestion under peak conditions, making it suitable for scenarios with relatively low throughput and less demanding response time requirements.

(2)

Scheme C: Time-Oriented Configuration (Eliminating Queue Waiting)

Solution C is located in the high-efficiency region of the Pareto front, adopting a configuration of 18 shuttles and 7 lifts to enhance system parallel processing capability. With sufficient equipment resources, the shuttle and lift utilization rates decrease to 0.22 and 0.18, respectively, effectively reducing queue waiting and shortening the order completion cycle to 53.7 s. However, the high investment cost of 5.768 million yuan leads to severe resource underutilization. Therefore, this configuration is only suitable for high-value-added applications with extremely stringent response time requirements, such as pharmaceutical cold-chain warehousing.

(3)

Scheme B: Balanced Configuration (Recommended Solution)

Solution B is located at the “knee point” of the Pareto front and adopts a 12-row × 35-column × 24-layer storage layout equipped with six shuttles and two lifts, achieving a balanced configuration. Compared with Solution A, only an additional 13.4% investment (0.403 million yuan) is required, while the order completion cycle is reduced by 65.7% (127.4 s). The utilization rates of the shuttles and lifts are maintained within the range of 0.52–0.58, respectively, which ensures economic efficiency while providing sufficient operational redundancy to accommodate peak demand fluctuations. Therefore, Solution B is considered the most suitable recommended configuration for this case study.

5.5. Chapter Summary

This chapter conducts comparative simulation experiments among six algorithms based on a real-world line-side warehouse project. The results show that RL-IMOEA/D achieves the best performance in both HV and IGD metrics, demonstrating its ability to obtain high-quality Pareto solutions under complex constraints. Pareto front analysis reveals the trade-off between system construction cost and order completion cycle, providing a recommended configuration scheme that balances economic efficiency and operational performance. Sensitivity analysis further verifies the effectiveness of the proposed model in capturing the impacts of external demand and equipment parameter variations on system configuration.

6. Conclusions

This study addresses the configuration optimization problem of the four-way shuttle warehouse system (FWSWS) under physical space constraints and high-concurrency operations. A multi-objective configuration optimization model integrating three-dimensional discrete mapping and the M/G/c queueing network is established to achieve the coordinated optimization of system construction cost and order completion cycle. An RL-IMOEA/D algorithm is proposed by incorporating heuristic constraint repair, Q-learning-driven variable neighborhood search, and adaptive weight adjustment strategies, thereby improving the search capability and Pareto solution quality under complex discrete constraints. Experimental results based on a real-world line-side warehouse case demonstrate that RL-IMOEA/D achieves the best performance in both HV and IGD metrics, verifying the effectiveness of the proposed model and algorithm. Furthermore, Pareto front analysis provides a recommended configuration scheme that balances construction cost and operational efficiency.

Despite the promising results, this study has certain limitations that should be explicitly stated. First, the experimental validation relies on a single engineering warehouse case with fixed spatial, capacity, and throughput conditions. While this case is highly representative of typical line-side warehousing, further research is required to evaluate the algorithm’s generalization capability across varying scales and dynamic demand profiles. Second, the performance of the proposed algorithm currently depends on manually selected hyperparameters, such as the variable neighborhood search trigger probability (P_vns) and the weight adjustment period (T_weight). Future research will focus on developing fully adaptive, parameter-free learning mechanisms to mitigate this dependence and investigate the collaborative optimization of storage location assignment and dynamic task scheduling under stochastic disturbances.