An Enhanced NSGA-II Driven by Deep Reinforcement Learning to Mixed Flow Assembly Workshop Scheduling System with Constraints of Continuous Processing and Mold Changing

Abstract

Mixed-flow assembly lines are widely employed in industrial manufacturing to handle diverse production tasks. For mixed flow assembly lines that involve mold changes and greater processing difficulties, there are currently two approaches: batch production and production according to order sequence. The first approach struggles to meet the processing constraints of workpieces with higher production difficulty, while the second approach requires the development of suitable scheduling schemes to balance mold changes and continuous processing. Therefore, under the second approach, developing an excellent scheduling scheme is a challenging problem. This study addresses the mixed-flow assembly shop scheduling problem, considering continuous processing and mold-changing constraints, by developing a multi-objective optimization model to minimize additional production time and customer waiting time. As this NP-hard problem poses significant challenges in solution space exploration, the conventional NSGA-II algorithm suffers from limited local search capability. To address this, we propose an enhanced NSGA-II algorithm (RLVNS-NSGA-II) integrating deep reinforcement learning. Our approach combines multiple neighborhood search operators with deep reinforcement learning, which dynamically utilizes population diversity and objective function data to guide and strengthen local search. Simulation experiments confirm that the proposed algorithm surpasses existing methods in local search performance. Compared to VNS-NSGA-II and SVNS-NSGA-II, the RLVNS-NSGA-II algorithm achieved hypervolume improvements ranging from 19.72% to 42.88% and 12.63% to 31.19%, respectively.

1. Introduction

Emerging digital technologies such as blockchain and AI-driven optimization transform supply-side production model from manufacturing resource allocation to personalized orders. To meet the challenges associated with personalized orders, manufacturing enterprises typically use mixed-flow assembly lines to complete production tasks in scenarios such as automotive assembly. Compared with traditional assembly lines, mixed-flow production lines can manufacture a variety of product types. Therefore, under the premise that high productivity, high efficiency, and high quality can increase competitiveness, the scheduling optimization problem of mixed-flow assembly lines has attracted widespread attention from both academia and industry [1,2,3]. The production approaches for mixed-flow assembly lines can generally be divided into two types. The first approach is to produce according to the order of the orders. The second approach involves categorizing similar products into the same batch for production. Both production approaches have their respective limitations. The first approach involves frequent changes in tools and molds, and existing research has pointed out the negative impacts of this approach [4]. Although the second approach reduces the frequency of tool and mold changes, different machined workpieces have varying levels of processing difficulty, making it challenging for some workstations to continuously process multiple workpieces that are more difficult to machine. To address these limitations, optimizing scheduling on mixed-flow assembly lines while considering both workstation overload and mold changes has become a critical issue for production-oriented enterprises.

The mixed-flow production line scheduling optimization problem is a subclass of flexible workshop scheduling optimization problems, and academia has conducted research on different mixed-flow assembly scenarios. Next, this article introduces the research status from the perspectives of the characteristics of the research scenarios and the characteristics of the solution methods. Wu et al. [5] developed a multi-stage model for personalized orders, whereas Shen et al. [6] and Zhang et al. [7] created balanced scheduling frameworks for shop floors. Su et al. [8] discussed the constraints of continuous processing in certain production scenarios and constructed a hybrid flow shop scheduling problem considering continuous processing and resource constraints. Wang et al. [9] further advanced the field by integrating logistics constraints with production scheduling. Many scholars have investigated production scheduling optimization for mixed-flow lines under multi-objective frameworks. For example, Guan et al. [10] constructed a multi-objective scheduling optimization model considering the continuous casting scenario, using iterative information to enhance the solving effectiveness of the heuristic algorithm. Laili et al. [11] examined flexible scheduling in shared manufacturing equipment scenarios and solved the resulting multi-objective problem. Zhang et al. [12] studied multi-objective scheduling for vehicle production under unexpected disruptions. Unlike most studies that focus on minimizing the makespan, Liu et al. [13] prioritized the optimization of assembly waiting time as the primary objective.

Currently, heuristic algorithms have emerged as a primary solution approach for mixed-flow assembly line scheduling problems with particular emphasis on enhancing local search mechanisms. For example, Geng et al. [14] developed an automated guided vehicle (AGV) compatible hybrid heuristic incorporating process skipping to optimize both global and local search performance. Wu et al. [15] addressed the long-cycle, high-energy-consumption characteristics of steel pipe production by developing a hybrid flow shop scheduling model considering batch processing and re-entrant flows, improving heuristic algorithms via a diversity-based local search mechanism. Wallrath et al. [16] studied a flow shop scheduling optimization problem that considers batch and resource allocation, using an improved heuristic algorithm for solving it. Gao et al. [17] implemented a digital twin-oriented local search operator for dynamic scheduling scenarios. Ding et al. [18] designed a multi-operator genetic algorithm specifically for resource-constrained flow shop scheduling. However, these studies focused primarily on refining local search rules without leveraging iterative algorithm data to guide heuristic optimization. Reinforcement learning (RL) offers potential through its self-learning capability via environmental interaction [19], showing promise for heuristic enhancement. Specifically, RL has been applied to parameter tuning, fitness evaluation, solution initialization, and local search operator selection [20]. Traditional random or exhaustive operator selection methods either risk missing optimal solutions or require excessive computation time [21]. To address this, several studies have adopted RL for operator selection, including: (1) Q-table-based hybrid heuristics by Yüksel et al. [22] and Li et al. [23] for local search decisions; and (2) Q-learning-guided PSO by Li et al. [24]. However, Q-table approaches suffer from adaptability constraints due to discrete state requirements. Deep reinforcement learning (DRL) resolves this limitation through deep learning. Ding et al. [25] developed a DRL-based parameter adaptation mechanism to balance global and local search, and Zheng et al. [26] used iterative algorithm states as DRL inputs to guide optimization. Yuan et al. [27] improved the genetic algorithm via deep reinforcement learning operators and designed a hybrid reward mechanism that combines delayed immediate rewards with global rewards. This paper summarizes the similarities and differences between existing studies and the present work, as outlined in

Table 1. Current and Previous Studies.

Literature	Model Building					Solution Approach
	Personalized Order	Multiple Production Cycles	Change Mold Constraints	Continuous Processing Constraints	Multiple Objectives	Heuristic Algorithm	Experience Guidance	Reinforcement Learning	Deep Reinforcement Learning
Wu et al. [5]	✔	✔	✖	✖	✖	✔	✔	✖	✖
Shen et al. [6]	✔	✔	✔	✖	✖	✔	✖	✖	✖
Zhang et al. [7]	✔	✖	✖	✖	✔	✔	✖	✖	✖
Su et al. [8]	✖	✖	✔	✔	✖	✖	✖	✖	✖
Guan et al. [10]	✔	✖	✖	✔	✔	✔	✔	✖	✖
Laili et al. [11]	✖	✖	✖	✖	✔	✔	✖	✖	✖
Zhang et al. [12]	✖	✖	✖	✖	✔	✔	✖	✖	✖
Liu et al. [13]	✔	✔	✖	✖	✔	✔	✖	✖	✖
Geng et al. [14]	✔	✖	✖	✖	✔	✔	✖	✖	✖
Wu et al. [15]	✔	✖	✖	✖	✔	✔	✖	✖	✖
Wallrath et al. [16]	✖	✖	✖	✖	✖	✔	✖	✖	✖
Gao et al. [17]	✔	✖	✖	✖	✔	✔	✖	✖	✖
Ding et al. [18]	✖	✖	✖	✖	✖	✔	✖	✖	✖
Yüksel et al. [22]	✖	✖	✖	✖	✖	✔	✔	✔	✖
Li et al. [23]	✖	✖	✖	✖	✖	✔	✔	✔	✖
Li et al. [24]	✖	✖	✖	✖	✖	✔	✔	✔	✖
Ding et al. [25]	✔	✔	✖	✖	✖	✔	✔	✔	✔
Zheng et al. [26]	✖	✖	✖	✖	✔	✔	✔	✔	✔
Yuan et al. [27]	✖	✖	✖	✖	✔	✔	✔	✔	✔
this paper	✔	✔	✔	✔	✔	✔	✔	✔	✔

. Among them, ✖ and ✔ indicate whether the literature did not consider/considered the elements, and ‘experience guidance’ means that the algorithm used effective information from the iterative process to guide subsequent algorithmic solutions.

Prior research has rarely integrated workload balancing, efficiency, and delivery performance within scheduling models. From a methodological perspective, some studies have employed DRL-driven operators to leverage iterative optimization data for solving mixed-flow assembly line scheduling problems, but this idea has not yet been applied to solve the problem studied in this paper.

In mixed-flow assembly lines, consecutive processing of high-difficulty workpieces can cause overload constraints, reducing overall efficiency. To address this: (1) This study constructs a mixed-flow assembly line scheduling optimization model considering overload constraints, with objectives to minimize additional production time and delivery deviation (a multi-objective optimization problem). (2) To overcome NSGA-II’s computational complexity and weak local search, we enhance it with a DRL operator that utilizes real-time optimization data to direct search processes. The experimental results validate the algorithm’s superior performance.

2. Problem Description and Modeling

2.1. Problem Description

The growing diversification of market demand has driven widespread adoption of mixed-flow assembly lines in manufacturing enterprises. In contrast, this approach enhances production flexibility, but significant variations in workpiece processing complexity present operational challenges. Consecutive scheduling of high-complexity workpieces can induce overload constraints, resulting in unplanned production interruptions and degraded overall efficiency.

This study addresses a mixed-flow assembly line scheduling optimization problem, which is defined as follows: A manufacturing plant produces $\{\mathrm{1,2},\dots K\}$ types of workpieces, where each type $k$ has $m_k$ configurations, denoted as $C_K^M=\{C_1^k,C_2^k\dots C_{m_k}^k\}$. The layout and processing flow of the mixed-flow assembly line are known. These workpieces require $V$ types of components in total. The demand for each configuration is known and represented as $\{D_1^k,D_2^k,\dots ,D_{m_k}^k\}$. Customer orders are time sensitive, meaning that the quantity of each workpiece type to be delivered by period t is given by $\{D_1^{kt},D_2^{kt},\dots ,D_{m_k}^{kt}\}$. Workstation tooling requirements and assembly times vary significantly across different workpiece configurations. Configurations with higher assembly complexity frequently induce overload constraint conditions. While assembly times remain fixed (being process-dependent), production efficiency improvements are achieved through sequence optimization, measured via two key metrics: (1) additional production time (combining overload and mold-changing durations) and (2) delivery deviation. The mixed-flow assembly line scheduling problem studied in this work is illustrated in Figure 1.

To formulate the mixed-flow assembly line scheduling optimization problem, the following assumptions are made:

• The quantity of workpieces to be produced in each period is known in advance.
• Each assembly task at any workstation involves installing only one component.
• A workstation may handle one or multiple types of components.
• If a workstation consecutively assembles components exceeding a predefined difficulty threshold, overload time is needed, and production must wait until completion.
• If the component type changes at a workstation, a mold-changing time is incurred before the next assembly task can begin.
• Owing to uncertainties in order placement and delivery timelines, the customer waiting time in the model considers only scheduling-related delays (not external factors).
• No stochastic disruptions are considered; the completion time of each order depends solely on the production schedule.

2.2. Notations

This article constructs a mathematical model based on the optimization model of flexible workshop production scheduling. The key symbols and their definitions used in the mixed-flow assembly line scheduling problem with continuous processing constraints and mold-changing constraints are summarized in

Table 2. Symbol Definition Table.

Symbols	Explanations	Symbols	Explanations
$x_i^t$	Decision variable, type and configuration number of the workpiece in median position $i$ of the production cycle	$B_{jv}$	0–1 variable, indicating whether the component $v$ can be assembled at station $j$
$k_m$	Upper limit of workpiece type number	${Pt}_{ij}^t$	0–1 variable, where 0 indicates that the workpiece at position $i$ in period $t$ has no overload at station $j$, and vice versa.
$T_m$	Upper limit of the production cycle	$F_{mv}$	0–1 variable, indicating whether the workpiece type $m$ contains the accessory $v$
$V$	Accessory number limit	$R_{jv}$	The load limit of the workstation $j$ assembly accessories $v$
$J$	Workstation number upper limit	$W_{ij}^t$	The waiting time for mold change at station $j$ for workpiece at position $i$ produced in cycle $t$
$I$	The maximum number of workpieces produced in each production batch	$S_{ij}^t$	The time when the workpiece at position $i$ is processed at station $j$ during cycle $t$
$v$	Accessory number, $v\in \{\mathrm{1,2},\dots ,V\}$	$P_{ij}^t$	The processing time of the workpiece at position $i$ produced in cycle $t$ at workstation $j$
$k$	Workpiece Type Number, $k\in \{\mathrm{1,2},\dots k_m\}$	$P_M^{KT}$	The output of type $K$ $M$ configured within cycle $T$
$j$	Workstation number, $j\in \{\mathrm{1,2},\dots ,J\}$	${Pe}_v$	The time for the normal assembly of part $v$
$L$	Workstation set, $j\in \{\mathrm{1,2},\dots ,J\}$	$O_{ij}^t$	The time when the workpiece at position $i$ is produced in cycle $t$ and leaves workstation $j$
$M$	A sufficiently large positive integer	$C_M^K$	The configuration set of workpieces produced by the mixed flow assembly line, $C_M^K\in \{C_1^k,C_2^k\dots C_{m_k}^k\}$
$H$	Extra assembly time caused by overload	$D_M^K$	Total demand quantity of $M$ configuration of type $K$,$D_M^K\in \{D_1^k,D_2^k,\dots ,D_{m_k}^k\}$
$m_k$	The quantity of configuration for workpiece type $k$	$E_{ij}^t$	The time for the workpiece produced at position $i$ during cycle $t$ to enter station $j$.
$D_M^{KT}$	The total demand quantity of type $K$ configuration $M$ at time $T$	${Pr}_{jv}$	Load statistical parameters of assembly parts $v$ at workstation $j$

.

2.3. Model formulation

The mixed-flow assembly line scheduling optimization model is a type of flexible workshop production scheduling optimization model, typically based on production scenarios such as electric vehicle manufacturing. Mixed-flow assembly line scheduling optimization traditionally prioritizes production efficiency and customer satisfaction. While improvements in the assembly process are beyond the scope of this study, we optimized the assembly sequence to reduce the additional production time. The integer programming model is formulated as follows:

Objective Functions:

$$\mathit{min}{\mathit{Fo}}_1={\sum }_{t=1}^{T_m}{\sum }_{i=1}^I{\sum }_{j=1}^J(W_{\left(i+j-1\right)j}^t+H\ast {Pt}_{\left(i+j-1\right)j}^t)$$

(1)

$$\min {Fo}_2=\min {\sum }_{t=1}^T{\sum }_{k=1}^K\left({\sum }_{m=1}^{m_k}\left(\right|D_m^{kt}-{\sum }_{t^{\prime }=1}^t{P}_m^{k{t}^{\prime }}\right|\left)\right)$$

(2)

Equation (1) calculates the additional production time ${Fo}_1$, combining mold-changing time and overload processing time. The specific calculation approach is to sum the mold change waiting time and the overload processing time for each workpiece at each station for each cycle. Equation (2) computes the completion deviation ${Fo}_2$, quantifying the discrepancy between the expected and actual production quantities per workpiece type per period. The specific calculation method calculates the difference between the order quantity for each configuration and the actual production quantity for each period.

The model incorporates virtual workpieces to represent workstation waiting states, occurring in two scenarios: (1) when awaiting upstream completion before starting, or (2) when awaiting downstream completion before finishing. These artificial elements enable computational efficiency while avoiding representation of actual workpieces.. They have zero processing time. When a virtual workpiece is processed at workstation $j$, it indicates that the workstation is waiting for workpieces to arrive. For workstation $j$, any workpiece processed in a position smaller than its workstation number is considered a virtual workpiece.

To accurately characterize the time constraints for workpieces entering workstations, we divide the constraints into two cases: one is when the workpiece is in the first position of a production cycle and at the first workstation, and the other is when the workpiece is in other positions of a production cycle or in the first position workpiece entering other workstations. For the first case, we have Constraints (3) and (4):

$$E_{11}^t=0,\forall t\in \left\{\mathrm{1,2},\dots ,T_m\right\}$$

(3)

$$E_{i^{\prime }j}^t=0,\forall t\in \left\{\mathrm{1,2},\dots ,T_m\right\},\forall j\in \left\{1,\dots ,J\right\},\forall i^{\prime }\in \left\{1,\dots ,I+J-1\right\},i^{\prime }<j$$

(4)

Constraint (3) indicates that the start time of the pipeline is 0 in any cycle. Constraint (4) represents the calculation constraint for the entry time of virtual workpieces.

For the second case, the entry time of a workpiece into a workstation is determined by the maximum completion time of the previous production positions across all workstations, as shown in Constraint (5):

$$\begin{gathered} E_{\left(i+j-1\right)j}^t={max}_{j^{\prime }\in \left\{1,\dots ,J\right\}}{O}_{(i{+j-2)j}^{\prime }}^t\forall t\in \left\{\mathrm{1,2},\dots ,T_m\right\}, \\ \forall j\in \left\{1,\dots ,J\right\},\forall i\in \left\{1,\dots ,I\right\},i\ge j,i+j>2 \end{gathered}$$

(5)

Constraint (6) is established to characterize the mold-changing time between different production tasks. The specific calculation approach is to determine whether the task-related component $\mathrm{v}$ is processed on $\mathrm{j}$, and whether this component is different from the component processed previously.

$$W_{(i+j-1)j}^t=\left\{\begin{array}{c}wt,(\forall j\in \left\{1,\dots ,J\right\},\forall i\in \left\{2,\dots ,I\right\},\\ \forall t\in \{\mathrm{1,2},\dots ,T_m\},\exists v\in V,B_{jv}=1,\\ F_{x_{i-1}^{t}v}=1,F_{x_i^{t}v}=0)\\ 0,\mathrm{else}\end{array}\right.$$

(6)

The processing start time for a workpiece at a workstation is determined by both its entry time and the required mold-changing time. Constraint (7) accordingly defines the workpiece processing start time:

$$\begin{gathered} S_{(i+j-1)j}^t=E_{(i+j-1)j}^t+W_{\left(i+j-1\right)j}^t\forall t\in \{1,\dots ,T_m\}, \\ \forall j\in \{1,\dots ,J\},\forall i\in \{1,\dots ,I\} \end{gathered}$$

(7)

The assembly time calculation accounts for both normal and overload conditions by distinguishing between two operational modes: (1) normal assembly time without overload, whose calculation constraint is the normal processing time. (2) Overload the assembly time when consecutive processing exceeds the threshold, which is calculated as the normal processing time plus the overload time. The assembly time calculation constraints are formulated as follows:

$$P_{(i+j-1)j}^t=\left({\sum }_{v=1}^V{F}_{x_i^{t}v}\ast {Pe}_v\ast B_{jv}\right)i<{Pr}_{lv},\forall t\in \{1,\dots ,T_m\},\forall j\in \left\{1,\dots ,J\right\},\forall i\in \{1,\dots ,I\}$$

(8)

$$\begin{gathered} P_{(i+j-1)j}^t=\left({\sum }_{v=1}^V{F}_{x_i^{t}v}\ast {Pe}_v\ast B_{jv}\right)+H\ast {Pt}_{(i+j-1)j}^t,i\ge {Pr}_{lv} \\ \forall t\in \{1,\dots ,T_m\},\forall j\in \{1,\dots ,J\},\forall i\in \{1,\dots ,I\} \end{gathered}$$

(9)

where ${Pt}_{ij}^t$ represents the overload time penalty. The calculation of overload time depends on the number of consecutive workpieces processed at the workstation. When workstation j processes more than threshold ${Pr}_{jv}$ of component $v$, ${Pr}_{jv}=1$, and vice versa. Therefore, Constraint (10) is set as follows:

$${Pt}_{ij}^t=\left\{\begin{array}{c}0,{\sum }_{v=1}^V{B}_{jv}\ast {\sum }_{I^{\prime }=i-{Pr}_{lv}}^i{F}_{x_{I^{\prime }}^{t}v}\le {Pr}_{jv}\\ 1,\mathrm{else}\end{array}\right.,\forall t\in \{1,\dots ,T_m\},\forall j\in \{1,\dots ,J\},\forall i\in \{1,\dots ,I\}$$

(10)

The completion time of workpiece $i$ at workstation $j$ depends on its start time and processing time. Therefore, constraint (11) is set to characterize the time when the workpiece leaves the workstation. The calculation method for the workpiece leaving the workstation is the start processing time plus the processing time.

$$O_{(i+j-1)j}^t=S_{(i+j-1)j}^t+P_{(i+j-1)j}^t\forall t\in \{1,\dots ,T_m\},\forall j\in \{1,\dots ,J\},\forall i\in \{1,\dots ,I\}$$

(11)

Each accessory is completed by only one workstation. Therefore, constraint (12) is set to describe the processing constraints of the workstation and the operation:

$${\sum }_{j=1}^J{B}_{jv}=1 \forall v\in \{1,\dots ,V\}$$

(12)

In addition, any workstation can process one or more components. Constraint (13) is set as follows:

$${\sum }_{v=1}^V{B}_{jv}\ge 1 \forall j\in \{1,\dots ,J\}$$

(13)

For the workpieces produced within each production cycle, constraint (14) is formulated to represent the output of workpieces during a given cycle. The calculation method for production during the period is the sum of the indicator variables. $1_{{\mathrm{x}}_{\mathrm{i}}^{\mathrm{t}}={\mathrm{C}}_{\mathrm{m}}^{\mathrm{k}}}$ represents the indicator variable, returning 1 when ${\mathrm{x}}_{\mathrm{i}}^{\mathrm{t}}={\mathrm{C}}_{\mathrm{m}}^{\mathrm{k}}$ is satisfied, and 0 otherwise.

$$P_m^{kt}={\sum }_{i=1}^I\left(1_{x_i^t=C_m^k}\right),\forall t\in \{1,\dots ,T_m\},\forall j\in \{1,\dots ,J\},\forall i\in \{1,\dots ,I\}$$

(14)

Furthermore, the model has constraints on the values of decision variables (15):

$$x_i^t\in \{C_1^1,\dots ,C_{m_1}^1,\dots ,C_{m_k}^k\} ,\forall t\in \{1,\dots ,T_m\},\forall j\in \{1,\dots ,J\},\forall i\in \{1,\dots ,I\}$$

(15)

3. Deep-Reinforcement Learning-Driven Enhanced NSGA-II

The nondominated sorting genetic algorithm (NSGA-II) represents an improved variant of genetic algorithms. Compared with conventional genetic algorithms, NSGA-II enhances its performance in solving multi-objective optimization problems through three key improvements: nondominated sorting, crowding distance comparison, and the elitist strategy. These characteristics render the approach particularly effective for solving the mixed-flow assembly line scheduling optimization problem under investigation. However, NSGA-II suffers from inherent limitations in local search capability, which manifests in two primary aspects when solving our proposed optimization model. First, the designed model requires decision-making for production planning of multiple workpiece types across various production cycles, resulting in an extensive solution space. Second, permutations of production sequences among nonoverloaded positions may not affect the objective function values, a characteristic further amplified in multi-objective optimization scenarios.

To address these challenges, we introduce a deep reinforcement learning-driven variable neighborhood search (VNS) operator. Compared with conventional VNS algorithms, our proposed operator achieves superior performance because it intelligently selects neighborhood search operators on the basis of multiple types of information collected during the iterative process. Figure 2 presents the solution framework of the enhanced NSGA-II algorithm incorporating deep reinforcement learning. The left side of Figure 2 shows the solving process of NSGA-II, whereas the right side presents the RL-VNS operator designed in this paper. Notably, since the various operators designed in this paper do not alter the nondominated sorting and crowding distance calculations of NSGA-II, their algorithmic complexity remains $\mathrm{O}\left(\mathrm{M}{\mathrm{N}}^2\right)$ [28]. Here, M is the number of objectives, and N is the population size. The following subsections detail three core components: (1) the encoding scheme, (2) variable neighborhood search operators, and (3) the deep reinforcement learning-driven operator selection mechanism. The pseudocode of the solution process is shown in Algorithm 1.

Algorithm 1 The pseudocode of improved NSGA-II process driven by deep reinforcement learning

Input: Demand data, processing technology data for various workpieces, and processing capability data for workstations. Output: Production scheduling scheme. 1 Initialize relevant parameters. 2 Load the deep reinforcement learning driven operator. 3 Initialize population. 4 Calculate the objective function of the initial population. 5 While the iteration limit is not reached, do: 6   Use crossover operations to search the current population. 7   Use mutation operations to search the current population. 8   Perform local search on the population using RL-VNS and update the gradient. 9   Calculate the crowding degree and perform fast non-dominated sorting. 10   Select the next generation individuals based on the elite retention strategy. 11 Output results and obtain statistical indicators.

3.1. Encoding Scheme

The encoding scheme selection significantly influences the computational efficiency of the enhanced NSGA-II algorithm for mixed-flow assembly line scheduling optimization. The proposed approach implements a dual-layer integer encoding strategy for chromosome representation. The first layer elements represent workpiece types corresponding to order positions, whereas the second layer elements indicate production quantities for respective order positions. Compared with single-layer chromosome encoding, this approach effectively reduces chromosome length. For example, Figure 3 presents a chromosome encoding example for a workshop that produces workpieces with five different configurations.

As shown in Figure 3, the above chromosome indicates that the production sequence of the production plan is 1->3->2->3…, producing 2, 3, 1, 4… pieces respectively. The highlighted positions indicate overloaded positions, meaning that the quantity of workpieces being processed exceeds the threshold, and some production tasks will have additional production time.

3.2. Variable Neighborhood Search Operators

For heuristic algorithms exhibiting limited local search capabilities, performance enhancement typically involves integrating specialized local search operators. Accordingly, this study develops an enhanced NSGA-II algorithm through the implementation of variable neighborhood search operators to improve local search effectiveness. Specifically, we introduce six neighborhood search operators from two perspectives: production sequence and production quantity. These operators include the insertion search operator, reverse search operator, shuffling search operator, workload adjustment operator based on overload conditions, production position adjustment operator based on the satisfaction level, and production sequence merging operator.

The insertion, reverse, and shuffling search operators aim to identify superior production schemes by adjusting production sequences and quantities. The algorithm operates in two phases: (1) identification of search points across production cycles, followed by (2) generation of new solutions through the application of insertion, reversal, or shuffling operations. These operators implement a variable neighborhood search strategy, progressively expanding from smaller to larger search spaces. The above ideas are reflected in the first to third parts of Figure 4.

The workload adjustment operator, designed based on overload conditions, modifies production tasks in response to the overload status of workstations. The implementation proceeds as follows: (1) randomly select an overloaded position in the current solution; (2) identify potential positions capable of transferring the entire overloaded quantity; (3) if found, transfer the overload; otherwise, partially transfer the workload until reaching the position’s capacity limit. The above idea is reflected in the fourth part of Figure 4.

Two production position adjustment operators (Operator 1 and Operator 2) were developed based on satisfaction level criteria. While both operators modify production sequences and quantities in response to delivery deviations across cycles, they employ distinct strategies: Operator 1 reallocates production by decreasing one workpiece’s quantity to increase another's, whereas Operator 2 achieves balance by redistributing specific workpieces’ production quantities across cycles. The specific mechanisms are as follows:

• Operator 1: (1) randomly select a production cycle with surplus  production; (2) identify workpieces with production deficits in subsequent  cycles; (3) swap their production positions.
• Operator 2: (1) Identify workpieces with surplus production; (2)  locate cycles where the same workpiece shows production deficits; (3) exchange  positions with other workpieces having production surplus in those cycles.

The above idea is reflected in the fifth and sixth parts of Figure 4.

The production sequence merging operator combines production quantities of the same workpiece across different positions. The procedure includes (1) randomly selecting a nonoverloaded position; (2) identifying merge-compatible positions that will not cause overload after merging; and (3) randomly selecting and merging with a compatible position. The above idea is reflected in the final part of Figure 4.

Figure 4 demonstrates the VNS operator solution process, with production positions and quantities in red font indicating merged encoding results. Unlike fixed-neighborhood local search operators, VNS employs multiple neighborhood structures. Since solutions optimal within a single neighborhood may not represent global optima, the VNS algorithm enhances search capability through diverse neighborhood exploration. This approach increases the probability of discovering superior solutions while improving overall algorithmic performance.

3.3. Deep-Reinforcement Learning-Driven Operator

Previous research has extensively explored heuristic algorithm improvements. However, growing problem complexity and incomplete understanding of problem characteristics make heuristic design increasingly challenging [26]. Reinforcement learning offers potential solutions through environment-state-based action selection, though traditional Q-table implementations face limitations in terms of environmental definition and action space handling. Deep reinforcement learning overcomes these constraints by employing neural networks to support continuous environment representations. Building on these advances, this study develops a deep reinforcement learning-driven operator to enhance heuristic algorithms. The proposed operator dynamically collects and analyzes information about both search objects and population states during iterations, using these insights to intelligently select search operators. A schematic diagram of this approach is shown in Figure 5. The deep reinforcement learning-driven operator first obtains the network input on the basis of the population information of the current search target and then acquires the action values of each local search operator through forward propagation. A roulette wheel decision-making process is used to select the search operator on the basis of the current action values. The selected search operator is used to obtain a new population. Finally, on the basis of this new population, the effectiveness of the previous search operator is evaluated, and the gradient of the neural network is updated.

In the deep reinforcement learning-driven operator in this study, the state space is defined via two key information types during iterations: (1) population diversity metrics (hypervolume change percentage and crowding distance change percentage from the previous action), and (2) the current search object’s objective function information. Specifically, the environmental components are as follows:

$${\mathrm{s}}_{\mathrm{t}}=\text{{}\Delta {\mathrm{H}}_{\mathrm{t}},\Delta {\mathrm{C}}_{\mathrm{t}}, \cos (\text{∠}{\mathrm{A}}_{\mathrm{t}}), \cos (\text{∠}{\mathrm{B}}_{\mathrm{t}})\text{}}$$

(16)

where $\Delta {Hv}_t$ represents the percentage change in hypervolume after the previous action, and where $\Delta C_t$ denotes the percentage change in the population crowding distance after the previous action. The hypervolume serves as a performance metric for multi-objective optimization algorithms and quantifies the volume of space enclosed between the optimal solution set and a reference point. The calculation formulas for hypervolume (${\mathrm{HV}}_{\mathrm{t}}$), $\Delta {\mathrm{Hv}}_{\mathrm{t}}$, and $\Delta {\mathrm{C}}_{\mathrm{t}}$ are as follows:

$${\mathrm{HV}}_{\mathrm{t}}=\delta ({\bigcup }_{\mathrm{i}=1}^{\left|{\mathrm{S}}_{\mathrm{t}}\right|}{\mathrm{v}}_{\mathrm{i}})$$

(17)

$$\Delta {\mathrm{Hv}}_{\mathrm{t}}=({\mathrm{HV}}_{\mathrm{t}}-{\mathrm{HV}}_{\mathrm{t}-1})/{\mathrm{HV}}_{\mathrm{t}-1}$$

(18)

$$\Delta {\mathrm{C}}_{\mathrm{t}}=({\mathrm{C}}_{\mathrm{t}}-{\mathrm{C}}_{\mathrm{t}-1})/{\mathrm{C}}_{\mathrm{t}-1}$$

(19)

where $\left|S_t\right|$ represents the number of optimal solutions at time $t$, and where $v_i$ indicates the volume of space enclosed between the $i$-th solution and the reference point.

Furthermore, to characterize the relative positional relationship of a solution within the Pareto front, this study employs $\cos (\text{∠}{\mathrm{A}}_{\mathrm{t}})$ and $\cos (\text{∠}{\mathrm{B}}_{\mathrm{t}})$ to define the deep reinforcement learning state. Specifically, $\text{∠}{\mathrm{A}}_{\mathrm{t}}$ represents the angle formed between the search target at time t and the two extreme solutions of the optimal Pareto front before the search initiation, whereas $\mathrm{\angle }{B}_t$ denotes the angle formed between the search target at time t and the two extreme solutions of the current optimal Pareto front. $\cos \left(\text{∠}{\mathrm{A}}_{\mathrm{t}}\right)$ and $\cos (\text{∠}{\mathrm{B}}_{\mathrm{t}})$ correspond to the cosine values of these angles.

In the deep reinforcement learning operator, the action ($a_t$) is defined as the selection of neighborhood search operators: ${\mathrm{a}}_{\mathrm{t}}\text{∈}\mathrm{Ls}=\text{{}{\mathrm{Ls}}_1,\text{…},{\mathrm{Ls}}_7\text{}}$. When the selected search operator yields improvements at time t, a reward (${\mathrm{r}}_{\mathrm{t}}=1$) is assigned; otherwise, ${\mathrm{r}}_{\mathrm{t}}=0$. Upon completing the search, the neural network weights and biases in the deep reinforcement learning operator are updated accordingly. The pseudocode for the deep reinforcement learning-driven local search operator is presented as Algorithm 2:

Algorithm 2 The pseudocode of deep reinforcement learning-driven local search operator

Input: Population to be searched, deep reinforcement learning agent Output: Population after search 1 While the search task for all search objectives is not completed, do: 2   if the number of iterations < the upper limit of iterations, do: 3     Update ${\mathrm{s}}_{\mathrm{t}}$ 4     Calculate the return values of various actions under ${\mathrm{s}}_{\mathrm{t}}$ by the deep reinforcement learning operator. 5     Select the corresponding local search operator for local search based on the return values. 6     Check if the new population has improvements? Determine the reward value ${\mathrm{r}}_{\mathrm{t}}$. 7     Determine ${\mathrm{s}}_{\mathrm{t}+1}$. 8     Use $({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}},{\mathrm{r}}_{\mathrm{t}},{\mathrm{s}}_{\mathrm{t}+1})$ to update the gradient of deep reinforcement learning.

4. Simulation Experiments

4.1. Test preparation and parameter settings

The coding was implemented via MATLAB 2022b. The experimental environment for the simulation tests was configured as follows: CPU: 13600KF (@3.5 GHz), RAM: DDR4 32GB 4000 MHz, Hard Disk: ZHITAI Tiplus5000 2TB. The parameter settings were as follows: maximum number of iterations: 200, local search attempts: 20, population size: 20, number of hidden layers: 3, number of neurons in the hidden layers: $\left(\mathrm{64,32,32}\right)$, number of output layers: $7$, Discount Factor: $0.9$, learning rate: $0.00001$ and activation functions: $(relu,relu,sigmoid,sigmoid)$.

4.2. Algorithm Comparison Experiments

The case study data for this comparative experiment comes from the mixed-flow assembly line of BYD electric vehicles located in the Yuhua District of Changsha City, Hunan Province, China, which is responsible for the production tasks of 11 different models. Specifically, this article performs comparative experiments using 7 cases, with the experimental scenarios covered by each case shown in

Table 3. Comparison of the number of nondominated solutions (NNSs).

Example	$\mathbf{Demand} \mathbf{Scales} {\mathit{D}}_{\mathit{M}}^{\mathit{K}\mathit{T}}$			$\mathbf{Cycle} {\mathit{T}}_{\mathit{m}}$
	$\mathit{m}\mathit{i}\mathit{n}$	$\mathit{m}\mathit{a}\mathit{x}$	$\mathit{a}\mathit{v}\mathit{g}$
1	18	60	37.59	2
2	30	60	38.55	2
3	30	66	41.96	5
4	30	66	41.96	5
5	15	33	20.94	10
6	15	33	21.11	10
7	5	11	7.0364	10

below.

In order to verify that the deep reinforcement learning-driven operator designed in this article can effectively and accurately evaluate various states, this paper first ran the RLVNS-NSGA-II 80 times. Since the maximum number of iterations was set to 200 and the number of local searches to 30, each run of the algorithm trained the deep reinforcement learning model for $200\times 30=6000$ times. Furthermore, this paper recorded the average estimation error of the deep reinforcement learning operator for the State Value during each run, resulting in the convergence graph of prediction errors shown in Figure 6 below.

The above prediction error diagram shows that the deep reinforcement learning-driven operator designed in this paper effectively converges to the prediction error during the learning process. This proves that the deep reinforcement learning operator gradually converges during the training process. In other words, the deep reinforcement learning operator progressively and accurately evaluates the effectiveness of each local search operator within the solution space during the iterative process, thereby guiding the algorithm to select the most appropriate operator to enhance search efficiency.

To evaluate the algorithm's performance in multi-objective optimization, the following metrics were selected: the number of nondominated solutions ($NNS$), where larger values indicate more nondominated solutions in the results; the generational distance ($GD$), where smaller values indicate results closer to the Pareto front and convergence of solutions; and the hypervolume ($HV$), where larger values indicate better comprehensive performance. NSGA-II, VNS-NSGA-II improved by variable neighborhood search, and SVNS-NSGA-II with random variable neighborhood search were selected as comparison algorithms, with each algorithm running 40 times per test case. The comparison results of these metrics are shown in

Table 4. Comparison of the number of nondominated solutions (NNSs).

Example	RLVNS-NSGA-II			VNS-NSGA-II			SVNS-NSGA-II
	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$
1	1	20	12.95	3	20	13.25	1	20	14.425
2	3	20	15.675	0	20	11.05	1	20	12.375
3	4	20	14.475	8	20	16.425	8	20	15.75
4	8	20	16.575	6	20	15.275	7	20	15.375
5	8	20	17	8	20	15.675	7	20	15.925
6	6	20	16.9	6	20	16.6	6	20	16.6
7	10	20	17.1	6	20	17	6	20	17

,

Table 5. Comparison of generation distance (GD).

Example	RLVNS-NSGA-II			VNS-NSGA-II			SVNS-NSGA-II
	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$
1	6.19 × 10−4	1.73 × 10−1	3.03 × 10−2	2.17 × 10−2	4.55 × 10−1	1.00 × 10−1	1.05 × 10−2	6.69 × 10−1	6.91 × 10−2
2	1.83 × 10−8	2.08 × 10−1	3.16 × 10−2	6.11 × 10−3	5.78 × 10−1	9.50 × 10−2	4.94 × 10−3	2.77 × 10−1	8.21 × 10−2
3	1.75 × 10−4	1.36 × 10−1	2.17 × 10−2	1.21 × 10−2	1.96 × 10−1	6.81 × 10−2	1.29 × 10−2	2.92 × 10−1	6.25 × 10−2
4	4.59 × 10−6	2.01 × 10−1	3.93 × 10−2	2.87 × 10−2	8.25 × 10−1	2.16 × 10−1	4.61 × 10−3	5.35 × 10−1	1.57 × 10−1
5	6.47 × 10−4	1.39 × 10−1	3.45 × 10−2	3.63 × 10−2	4.04 × 10−1	1.50 × 10−1	3.53 × 10−2	2.88 × 10−1	1.10 × 10−1
6	1.27 × 10−3	7.96 × 10−2	3.05 × 10−2	1.65 × 10−2	2.53 × 10−1	9.56 × 10−2	1.17 × 10−2	3.52 × 10−1	9.58 × 10−2
7	0.00	1.36 × 10−1	4.04 × 10−2	7.52 × 10−3	2.21 × 10−1	9.72 × 10−2	2.03 × 10−3	2.21 × 10−1	6.13 × 10−2

and

Table 6. Comparison of hypervolume (HV).

Example	RLVNS-NSGA-II			VNS-NSGA-II			SVNS-NSGA-II
	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$	$\mathbf{min}$	$\mathbf{max}$	$\mathbf{avg}$
1	7.23 × 10−1	1.21	1.08	5.93 × 10−1	1.02	8.80 × 10−1	7.94 × 10−1	1.10	9.63 × 10−1
2	8.12 × 10−1	1.20	1.06	6.00 × 10−1	1.12	8.82 × 10−1	5.44 × 10−1	1.10	8.67 × 10−1
3	9.13 × 10−1	1.16	1.05	4.66 × 10−1	9.54 × 10−1	7.86 × 10−1	3.60 × 10−1	1.01	8.01 × 10−1
4	8.46 × 10−1	1.19	1.03	6.30 × 10−1	9.35 × 10−1	7.61 × 10−1	6.37 × 10−1	9.58 × 10−1	8.13 × 10−1
5	7.98 × 10−1	1.17	1.01	4.14 × 10−1	9.18 × 10−1	7.06 × 10−1	6.49 × 10−1	1.01	7.80 × 10−1
6	7.83 × 10−1	1.09	9.56 × 10−1	4.92 × 10−1	9.38 × 10−1	7.27 × 10−1	4.98 × 10−1	9.82 × 10−1	7.53 × 10−1
7	8.27 × 10−1	1.20	1.01	4.21 × 10−1	1.02	7.76 × 10−1	5.30 × 10−1	1.15	8.66 × 10−1

and Figure 7. The data for Example 1 to Example 7 are represented by (a) to (g) in Figure 7.

According to Table 4, the proposed RLVNS-NSGA-II outperforms the other two algorithms in terms of the number of nondominated solutions for most test cases. This metric indicates that RLVNS-NSGA-II can obtain more nondominated solutions during optimization, thereby providing better solutions.

According to Table 5 and Figure 7, the proposed RLVNS-NSGA-II demonstrates better solving performance for most test cases: from both mean values and median distributions, RLVNS-NSGA-II’s results show smaller generational distances, and from the distributions of upper and lower quartiles, RLVNS-NSGA-II's results are all smaller than those of the other two algorithms. This metric indicates that RLVNS-NSGA-II has better convergence capability and stability than the other two algorithms.

While the above results demonstrate RLVNS-NSGA-II’s convergence capability, further evidence from Table 6 shows that the solution set obtained by this algorithm has better distribution uniformity and spread: the hypervolume of RLVNS-NSGA-II is greater than that of VNS-NSGA-II and SVNS-NSGA-II across all the examples. When VNS-NSGA-II is compared with SVNS-NSGA-II, the leading margins of hypervolume are 19.72–42.88% and 12.63–31.19%, respectively.

The aforementioned predictive errors demonstrate that the deep reinforcement learning operator can accurately assess the current state value. Furthermore, RLVNS-NSGA-II drives the operator’s decision-making process with deep reinforcement learning, whereas the other two algorithms determine the search operators on the basis of fixed or random rules. All experiments employed an identical local search operator library. The results collectively demonstrate that the proposed deep reinforcement learning-driven operator effectively guides the algorithm to: (1) leverage population diversity information, (2) utilize objective function characteristics of search objects, and (3) select appropriate search operators. This adaptive mechanism consistently improves algorithmic performance across diverse test cases.

5. Conclusion

This study develops a mixed-flow assembly line scheduling optimization model incorporating mold-changing and overload constraints, balancing production efficiency with customer satisfaction. The dual-objective formulation minimizes both additional production time and delivery deviation. To solve this model effectively, we enhance NSGA-II through: (1) integration of six neighborhood search operators, and (2) implementation of a deep reinforcement learning mechanism that guides operator selection based on population diversity and objective function information. The results demonstrate that the proposed algorithm significantly improves NSGA-II’s local search capability and outperforms both VNS-NSGA-II and SVNS-NSGA-II variants: RLVNS-NSGA-II outperforms VNS-NSGA-II and SVNS-NSGA-II with hypervolume improvements of 19.72–42.88% and 12.63–31.19%, respectively.

The practical value of this study is demonstrated in several ways. In production settings involving personalized orders, such as electric vehicle manufacturing, the proposed models and methods can support decision-making for production planning. Leveraging the characteristics of multi-objective scheduling optimization, decision-makers can generate scheduling plans that effectively balance production efficiency with customer satisfaction. Moreover, the models and methods presented in this study can be integrated into manufacturing execution systems, thereby enhancing their functionality and intelligence.

However, this study has several limitations. Future research could be enriched in the following aspects:

• Research on scheduling strategies considering inventory constraints in line-side warehouses in mixed-flow assembly scenarios is conducted.
• A bilevel programming model considering AGV routing in mixed-flow assembly scenarios is established, and scheduling strategies are studied.