Flexible Job Shop Scheduling Problem Based on Deep Reinforcement Learning Using Dual Attention Network

Abstract

Industry 4.0 is transforming the way companies manufacture, improve, and distribute products, moving toward fast, intelligent, and flexible manufacturing, which will bring about fundamental changes in enterprises’ production capabilities. The Flexible Job Shop Scheduling Problem (FJSP) allows a single job to be divided into multiple operations, each of which can be processed on multiple machines. Due to its high flexibility and complexity, traditional scheduling methods are difficult to meet the needs of dynamic production. Dispatching rules struggle to effectively perceive the global precedence relationships among jobs and the distribution of machine workloads; metaheuristic approaches suffer from slow iterative convergence; existing deep reinforcement learning methods often employ a single policy network to handle both operation sequencing and machine assignment in a coupled manner, which tends to cause training instability and slow convergence. This paper proposes a deep reinforcement learning model that integrates Multi-Proximal Policy Optimization (MPPO) and Dual Attention Network (DAN) to address the FJSP. The model uses the operation message attention block and machine message attention block of DAN to capture the dependency relationships between operations and the dynamic competitive relationships between machines, respectively, and extract deep features. At the same time, MPPO designs dual actor networks to handle operation sequencing and machine assignment decisions separately, and combines a centralized critic to optimize the policy. This balances exploration and exploitation and improves training stability. Experiments are conducted based on the SD1 and SD2 datasets. In FJSP instances of four scales, the model is compared with PPO-DAN, PPO-HGNN, traditional scheduling rules, and OR-Tools. The results show that the algorithm reduces makespan by up to 4.2% on SD1 and 10.1% on SD2. Moreover, it achieves better performance than traditional scheduling rules. Its comprehensive performance is superior to that of the comparison methods, verifying its effectiveness and practical application potential in solving the FJSP.

1. Introduction

Research on production scheduling problems and their solution methods has existed since the 1950s. With the evolution of the industrial market and intensified competition among manufacturing enterprises, the Job Shop Scheduling Problem (JSP) emerged to ensure efficient and stable operation of manufacturing systems while maximizing production benefits [1]. JSP, one of the most common traditional scheduling problems, involves a finite set of machines processing a set of jobs. According to the technological process of each job, each operation is processed on a given machine, and each machine can only perform one operation at a time [2]. However, with changes in production models, traditional JSP has gradually evolved into the Flexible Job Shop Scheduling Problem (FJSP). FJSP involves a set of machine groups that can be selected for each operation, meaning multiple operations of a job can be processed on different machines, increasing scheduling flexibility and complexity [3]. Constantly changing production conditions and personalized customer demands introduce more complex factors into FJSP.

Chen et al. [4] proposed an improved Genetic Algorithm (GA) that updates the population through random methods like crossover and mutation, refining the population update process to obtain optimal solutions during iteration, achieving commendable results. Engin et al. [5] studied flow shop scheduling by introducing crossover and mutation operators from genetic algorithms into a hybrid Ant Colony Optimization algorithm. Chaouch et al. [6] investigated the application of an improved Ant Colony Optimization algorithm in distributed job shops to explore a larger search space. Wang et al. [7] addressed dynamic scheduling under random job arrivals by adding random weights to particles and redesigning the swarm transformation mechanism to improve Particle Swarm Optimization (PSO). Yu et al. [8] further enhanced PSO’s capability for FJSP by employing enhanced nonlinear inertia weights and Gaussian mutation to improve search ability and reduce the probability of falling into local optima. Li et al. proposed an improved multi-objective Lightning Search Algorithm [9], utilizing digital twins [10] and cat mapping [11], considering workers’ multi-memory processes. They aimed to minimize makespan, total carbon emissions, total production cost, and maximize product quality stability, proposing a digital twin-driven dynamic scheduling strategy for Multi-Process Flexible Job Shop Scheduling (MPFJSP) under machine failures [12]. Zhang et al. [13] proposes a discrete-time distributed PI+Reset control scheme that accelerates convergence and avoids overshoot through marginal cost consensus, while designing a routing algorithm to achieve supply-demand balance in a grid-connected BESS network. Metaheuristic algorithms aim to find the current best solution through multiple iterations, taking the current best result as the optimal solution for FJSP. However, they require re-iteration when problems change, are time-consuming, lack generalization capability, and suffer from exponentially increasing solving times with problem scale, making it difficult to meet response requirements and potentially leading to local optima. Machine learning methods can quickly classify or make decisions once the model is trained, making their application to FJSP a novel research approach.

FJSP is an important branch of job shop scheduling problems. Current main solution methods for FJSP include scheduling rules, heuristic algorithms, metaheuristic algorithms, and machine learning methods. However, due to practical needs, scheduling rules, heuristics, and metaheuristics can no longer fully meet actual requirements. In recent years, with continuous improvements in computing power, neural networks can be trained and inferred more efficiently and rapidly, sparking a new wave of artificial intelligence technology. Reinforcement learning has achieved remarkable success in various industries and fields, with researchers using it to solve complex real-world problems, including workshop scheduling issues.

Luo et al. [14] proposed a machine optimization method based on the THDQN model [15], optimizing two practical objectives: weighted total tardiness and average machine utilization. They proposed four different objectives corresponding to four different forms of reward functions, each optimizing a tardiness or machine utilization metric. Zhang et al. [16] designs a distributed PI controller with a reset mechanism for marginal cost consensus, where the integral term is reset when the proportional term crosses zero, thereby improving convergence speed, control accuracy, and suppressing overshoot. Zhang et al. [17] proposed a Contract Net Protocol(CNP)-based self-organizing negotiation mechanism [18] to guide cooperation and competition among multi-agents, combined with PPO [19] to improve the decision-making performance of AI schedulers under order and resource disruptions. Lei et al. [20] proposed a Multi-Pointer Graph Network (MPGN) [21], formulating the multi-action problem as a Multi-Markov Decision Process (MMDP). Computational experiments showed that the agent could learn high scheduling accuracy and track local states encountered during scheduling in real-time. Li et al. [22] established a multi-objective optimization model for DFJSP-ITR and proposed a Hybrid Deep Q-Network(HDQN) combining Deep Q-Network with three extended forms. Results showed that HDQN outperformed existing optimization-based methods, demonstrating superiority and generality in effectively handling disruption events and unseen situations through learning. Zhang et al. [23] designed six state transition functions and eight agent actions, testing their effectiveness and the agent’s generalization ability in FJSP of different scales considering variable processing times. Gui et al. [24] provided a continuous rule space and SDR weight selection through composite scheduling behavior aggregated from SDR and continuous weight variables. They designed a reward function related to the average tardiness performance metric, making cumulative reward maximization equivalent to average tardiness minimization, and constructed a policy network taking state as input and weights as output, using DDPG [25] to train the policy network to select the most appropriate weights at each decision point. Yue et al. [26] proposed an improved deep reinforcement learning algorithm, TS-DDQN, using a two-stage decision-making method to solve complex multi-objective DFJSP optimization problems. They used an external non-dominated set to improve network training speed and solution quality. Extensive experiments on benchmark datasets and comparisons with existing scheduling methods showed that the algorithm performed well in solution quality, convergence speed, and selection of optimal solutions. Zhang et al. [27] proposed an improved Multi-Agent Dueling Double Deep Q-Network [28] algorithm to optimize scheduling strategies during long-term training. Experimental results indicated that the algorithm effectively reduced weighted flow time in both training and unseen scenarios compared to existing methods. Zhou et al. [29] proposed a DQN-based [30] DQNDEMA, aiming to minimize makespan and total carbon emissions. They established a coupled mathematical model to study the complex coupling effects among factories, jobs, machines, and AGVs caused by machine failures. Performance validation on 20 instances showed that the algorithm obtained higher quality and more diverse Pareto fronts in 15 instances.

Although Deep Reinforcement Learning has demonstrated promising decision-making capabilities in solving the Flexible Job Shop Scheduling Problem, its standalone application in such structured scheduling environments still encounters significant representational bottlenecks. Conventional DRL models typically encode scheduling states as one-dimensional vectors or grid-like inputs. This flattened representation struggles to effectively preserve the complex topological relationships inherent in job shop scheduling, including the precedence constraints among operations, the optional processing affiliations between operations and machines, and the resource contention among different machines within the same time window. Without a dedicated graph representation learning mechanism, DRL agents are often limited to extracting information from global statistical features or handcrafted features, resulting in insufficient structural perception of the scheduling state. This deficiency subsequently impairs the efficiency of policy learning and the generalization capability of the model. Consequently, incorporating Graph Neural Networks to enhance the model’s ability to capture the topological structure of the scheduling environment has emerged as a crucial pathway for improving the performance of DRL in addressing the FJSP.

With the recent popularity of Graph Neural Networks (GNNs), which are frameworks using deep learning to analyze graph-structured data, GNNs can handle various types of graph data, including social networks, protein interaction networks, road networks, etc. The basic idea of GNNs is to utilize node features and relationships between nodes to learn a global representation. FJSP can model instances as disjunctive graphs, treating job operations as nodes and machine processing sequences as directed edges, making it highly compatible with GNNs. Several scholars have combined Graph Isomorphism Networks, Graph Attention Networks, Heterogeneous Graph Neural Networks, Dual Attention Networks, etc., with DRL algorithms to solve FJSP.

Song et al. [31] combined PPO with a Heterogeneous Graph Neural Network [32], integrating operation selection and machine assignment into a single decision process and using a heterogeneous graph to represent scheduling states to capture complex operation-machine relationships. This method outperformed traditional scheduling rules on datasets of different scales and characteristics with high computational efficiency, demonstrating good generalization performance and practical application value. Wang et al. [33] proposed a PPO combined with DAN framework [34], leveraging complex relationships between operations and machines, extracting deep features via DAN, and combining them with the PPO algorithm to optimize scheduling decisions, simultaneously achieving operation sequencing and machine assignment. Experiments showed that this method not only outperformed traditional PDRs but also matched the performance of exact methods in some scenarios, demonstrating strong generalization ability and practical application potential. Zhang et al. [35] develops a distributed PI+R economic dispatch strategy for isolated BESS networks, employing a centralized reset mechanism to ensure supply-demand balance, and introducing a dwell time to avoid Zeno behavior and handle input delays. Lv et al. [36] proposes the Flexible Job-shop Scheduling Problem with Parallel Operations (FJSP-PO), focusing on flexible job-shop scheduling involving parallel operations. It designs the AR-HGAT method, which combines heterogeneous graph attention networks with reinforcement learning. Through node-level and semantic-level attention mechanisms, it learns features, and the AR mechanism optimizes time-slot scheduling. Validated on simulated data, benchmark datasets, and real enterprise data, the method outperforms existing approaches in inference time and solution effectiveness, offering new ideas for discrete manufacturing scheduling. Zhao et al. [37] address the insufficient global representation of machines and operations in FJSP, this paper proposes the Dual Dynamic Attention Network (DDAN). It adopts an in-channel dynamic attention mechanism to capture complex dependencies and trains the model under the Actor-Critic framework. Evaluated on synthetic datasets and public benchmarks, DDAN outperforms traditional Priority Dispatching Rules (PDRs) and state-of-the-art DRL algorithms. It even surpasses deterministic algorithms in some scenarios and exhibits strong generalization ability on large-scale unseen FJSP instances. Li et al. [38] aiming to resolve the imbalance between local and global dependencies in FJSP, this paper presents the DAN-Mamba hybrid framework. It integrates the local feature extraction capability of the Dual Attention Network (DAN) with the linear-complexity global temporal modeling of the Mamba model. Experiments on synthetic and classic benchmark datasets demonstrate that the framework significantly shortens the makespan. Trained on small-scale instances, it can adapt to large-scale scenarios, showing excellent generalization and scalability. Wan et al. [39] decomposes FJSP into two subtasks: operation selection and machine allocation. It uses a meta-path-based heterogeneous graph neural network (MHGNN) to embed global scheduling states and models the problem as a dual Markov decision process. A soft double-actors critic algorithm is designed to train dual policy networks. Validated on three public benchmarks (Hurink, Behnke, Brandimarte), the method outperforms heuristic and advanced DRL approaches in both solution quality and efficiency, especially in large-scale problems. Yang et al. [40] proposes the heterogeneous graph attention-enhanced DRL framework (HGADRL). It designs an improved heterogeneous disjunctive graph to represent states and a dual-scale graph attention network to capture complex dependencies, combined with an Actor-Critic network for decision-making. Validated on 500 synthetic instances, 16 unseen instances, and 4 real-world instances, HGADRL outperforms hybrid PDRs, meta-heuristics, and DRL methods, demonstrating strong generalization. Zeng et al. [41] aiming at the dynamic multi-objective flexible job-shop scheduling problem with new job arrivals and urgent insertions, this paper proposes a method integrating the Dynamic Dual Attention Network (DDAN) with multi-agent reinforcement learning. The DDAN captures global features and responds to dynamic events, while two agents optimize makespan and average tardiness respectively, collaborating through a state-sharing mechanism and customized reward functions. Experiments show the method converges quickly and outperforms traditional scheduling rules, metaheuristic algorithms, and advanced DRL methods on large-scale instances. Tang et al. [42] to solve the Flexible Job-shop Scheduling Problem (FJSP), this paper presents a framework fusing the Relation-based Heterogeneous Graph Neural Network (HGNNR) with deep reinforcement learning. It models FJSP as a heterogeneous graph, decomposes it into relation-specific subgraphs, extracts features via graph convolution, and fuses cross-relation information using a multi-attention mechanism, combined with the PPO algorithm for optimization. Validated on public benchmark datasets, it exhibits superior scheduling efficiency and generalization compared to mainstream DRL methods and rule-based heuristics. Li et al. [43] focusing on the Multi-objective Flexible Job-shop Scheduling Problem (MOFJSP), this paper proposes an end-to-end framework based on Graph Attention Networks (GAT) and reinforcement learning. A Dual Graph Attention Network (DGAT) is designed to deeply extract heterogeneous graph features, and a weighted sum decomposition strategy is adopted to balance three objectives: makespan, maximum machine load, and total machine load. Experiments demonstrate the method can quickly approximate the Pareto front, outperforming five multi-objective evolutionary algorithms on synthetic and public instances with strong generalization. Huang et al. [44] to address the redundancy of disjunctive graphs and overestimation bias in FJSP, this paper proposes the Mean Multi-Channel Graph Attention-Proximal Policy Optimization (MCGA-PPO) model. Channel attention filters redundant information, multi-channel weighting balances estimation bias, and entropy loss optimizes exploration in large action spaces. Validated on synthetic and classic datasets, it achieves performance improvements of 1.22% and 1.29% respectively, effectively solving optimization challenges in complex scheduling scenarios. Zhang et al. [45] proposes a solution for FJSP combining the Meta-relation Heterogeneous Graph Neural Network (MRHGNN) with reinforcement learning. It defines four meta-relations to encode node semantics, uses GAT and Graph Transformer to extract heterogeneous features, and trains the decision network via the REINFORCE algorithm. Experiments on multiple public benchmark datasets show it outperforms heuristic, metaheuristic, and advanced DRL methods. Trained on small-scale instances, it can adapt to large-scale scenarios with strong generalization.

Table 1 summarizes the classified two categories labeled “DRL algorithms”.

Table 1. Research on FJSP & DFJSP by Various Scholars.

Author	Objective	Problem	Algorithm	Agent
Luo [14]	Multiple	FJSP	THDQN	Single
Zhang [17]	Single	DFJSP	PPO	Single
Lei [20]	Single	FJSP	MPPO	Multiple
Li [22]	Multiple	FJSP	HDQN	Multiple
Zhang [23]	Single	DFJSP	PPO	Single
Gui [24]	Single	FJSP	DDPG	Single
Yue [26]	Single	FJSP	TS-DDQN	Single
Zhang [27]	Single	FJSP	D3QN	Single
Zhou [29]	Multiple	DFJSP	DQNDEMA	Single
Song [31]	Single	FJSP	PPO-HGNN	Single
Wang [33]	Single	FJSP	PPO-DAN	Single
Lv [36]	Single	FJSP	AR-HGAT	Single
Zhao [37]	Single	FJSP	DDAN	Single
Li [38]	Single	FJSP	DAN-Mamba	Single
Wan [39]	Single	FJSP	PPO-MHGNN	Single
Yang [40]	Single	FJSP	HGADRL	Single
Zeng [41]	Single	FJSP	PPO-DDAN	Single
Tang [42]	Single	FJSP	PPO-HGNNR	Single
Li [43]	Single	FJSP	PPO-GAT	Single
Huang [44]	Single	FJSP	PPO-HGNN	Single
Zhang [45]	Single	FJSP	PPO-HGNN	Single
Our	Single	FJSP	MPPO-DAN	Multiple

The above-mentioned relevant studies cover both single-objective and multi-objective optimization scenarios, with the vast majority taking makespan as the core optimization objective. The research objects include both FJSP and DFJSP, while the solution methods cover meta-heuristic algorithms, DRL, and the integrated architectures of DRL combined with GNN. Despite the significant progress achieved in integrating Deep Reinforcement Learning with Graph Neural Networks for solving the Flexible Job Shop Scheduling Problem, the current research frameworks still exhibit limitations that constrain further improvements in overall performance and generalization capability.

First, existing mainstream architectures, such as those combining PPO with GNN, predominantly rely on a single-Actor network to simultaneously handle the two parallel decision sub-tasks of operation sequencing and machine assignment. However, the types of information and constraints associated with operation nodes and machine nodes in the scheduling environment are inherently different. A single-Actor network is required to produce a unified representation for both types of nodes, which prevents the optimization gradients of the operation selection policy and the machine assignment policy from being independently optimal during parameter updates. This entanglement increases training instability and may adversely affect the quality of the final scheduling solutions.

Second, the structural design of the DAN is intended to employ two independent attention modules to separately capture the complex interactions on the operation side and the machine side. Nevertheless, a single-Actor architecture cannot fully exploit this structural advantage of DAN. The capability of the operation attention module to perceive sequential constraints among operations, as well as the capability of the machine attention module to discriminate machine load and availability, are both compromised.

Therefore, it is imperative to develop a more targeted multi-policy framework that aligns with the inherent dual attention modules of DAN, thereby enhancing the expressiveness and decision-making accuracy of the policy. In this study, we first address the issue that the single Actor in the traditional PPO algorithm cannot effectively perform node embedding with DAN in scheduling problems. We propose a novel combination of MPPO with two Actors integrated with DAN. Unlike traditional PPO, which uses a single Actor and a single Critic, MPPO employs two independent Actor networks to embed machine nodes and operation nodes separately, accompanied by a centralized Critic for value evaluation. This architectural innovation more naturally adapts to the characteristics of the two heterogeneous entities—operations and machines—in scheduling problems, fully leveraging the advantages of DAN’s attention mechanism.

Secondly, by separating the representation learning processes for machines and operations, the two Actors can focus on generating embeddings for their respective node types (machines and operations), significantly improving the quality and discriminativity of node representations. Based on this, DAN can more effectively capture complex interaction relationships between nodes, particularly the matching compatibility between operations and machines. This design not only enhances the perception of heterogeneous information in the scheduling environment but also improves the learning efficiency and expressiveness of the policy function. Experimental results show that the proposed MPPO combined with DAN method outperforms baseline models of traditional PPO combined with DAN and traditional PPO combined with HGNN across multiple scheduling metrics, validating its superior adaptability and performance in complex scheduling tasks.

Finally, we designed a scale-agnostic decision-making framework to simultaneously address two key sub-problems in FJSP: operation sequencing and machine assignment. Two different types of information are incorporated into the policy network (MPPO’s input) to ensure high-quality decisions: the deep features extracted by DAN and the raw features of compatible operation-machine pairs. Experimental results demonstrate that our method outperforms traditional PDRs and traditional PPO, showing considerable advantages on two synthetically generated datasets with different distributions. Furthermore, it exhibits good generalization ability, as models trained on small-scale instances can yield high-quality solutions when directly applied to solve large-scale or out-of-distribution instances. The main contributions of this paper are:

1.

This paper proposes a deep reinforcement learning approach for FJSP based on the combination of MPPO and DAN. Since the DAN structure contains two independent attention networks, the traditional PPO algorithm did not achieve ideal performance when combined with it. Therefore, this paper introduces the MPPO framework, enabling multiple Actors to embed and collaborate with the two attention modules separately, thereby enhancing the diversity of policy expression and training stability.

2.

MPPO was validated on multiple datasets, including SD1 with variable operations, SD2 generated from synthetic data, and public benchmark datasets such as mk, Hurink_edata, Hurink_rdata, and Hurink_vdata. Experiments show that MPPO-DAN, as an end-to-end deep reinforcement learning algorithm, can train scale-agnostic scheduling policies, significantly outperforming traditional manually designed Priority Dispatching Rules (PDRs) in solving effectiveness while maintaining high computational efficiency.

3.

Compared to baseline methods PPO-DAN and PPO-HGNN, MPPO-DAN demonstrates superior performance in key metrics such as makespan and reward value. Additionally, this paper compares four typical scheduling rules, further validating the superiority of the proposed method in comprehensive scheduling performance. The results indicate that MPPO-DAN not only significantly improves solution quality but also possesses good generalization ability and stability, providing a new approach for solving complex scheduling problems.

2. Relate Work

2.1. Mathematical Model of FJSP

FJSP can be defined as follows: There are n jobs to be scheduled on m machines. The set of machines is denoted as M, and the set of jobs is denoted as J, Each job $J_i$ consists of $n_i$ operations, writted as $O_{ij}$, each requiring assignment to a machine. For each operation $O_{ij}$, there are multiple machines available for processing, and it is assigned to only one of them. The goal is to complete all jobs while maximizing or minimizing an optimization objective. In single-objective FJSP, the optimization objective is typically to minimize the makespan of the schedule. In multi-objective FJSP, other objectives may include minimizing energy consumption, minimizing carbon emissions, minimizing production costs, or maximizing machine utilization.

This is an example of a quote, the selection of action space is shown in Figure 1.

Figure 1. Illustration of the action space for FJSP.

FJSP can be viewed as a disjunctive graph. The disjunctive graph model can be described as: Given a graph G = {O, C, D} representing the initial FJSP, where O is a set including all operation nodes plus virtual start node S and end node T, C is the set of conjunctive arcs, and D is the set of disjunctive arcs. Figure 2 shows the disjunctive graph, where the undirected arcs in the left graph represent all possible processing sequences, and the directed arcs in the right graph represent the suitable processing sequence after solving.

Figure 2. Model the Flexible Job Shop Scheduling Problem (FJSP) as a disjunctive graph, where: (a) represents the sequential relationships of all possible processing operations; (b) illustrates the optimized operation execution order after problem solving.

The relevant symbolic definitions are shown in Table 2.

Table 2. Symbol meaning.

Symbol	Meaning
N	Number of jobs to be processed
M	Total number of machines
$J_i$	The i-th job
$n_i$	Number of operations for i-th job
$M_k$	The k-th machine
$O_{ij}$	The j-th operation of the i-th job
$P_{ijk}$	Processing time of the j-th operation of job i on machine k
$C_{ij}$	Completion time of the j-th operation of job i
$C_i$	Completion time of job i
$S_{ij}$	Start time of the j-th operation of job i
$C_{ijk}$	Completion time of the j-th operation of job i on machine k

The objective function is defined as the maximum completion time among all processed jobs, which is referred to as the makespan. This objective should be as short as possible; therefore, the objective function is to be minimized, as shown in the following equation:

$$C_{max}=min\left(max\left(C_i\right)\right)$$

(1)

The constraints are expressed as follows:

$$X_{ijk}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{O}_{ij}\mathrm{is}\mathrm{allocated}\mathrm{to}\mathrm{machine}{M}_k,\hfill \\ 0\hfill & \mathrm{Otherwise}.\hfill \end{array}\right.$$

(2)

$${\displaystyle \sum _{k=1}^M}{X}_{ijk}=1$$

(3)

$$C_{ij}\ge S_{ij}+P_{ijk}-(1-X_{ijk})\times N$$

(4)

$$S_{ij}\ge C_{i^{\prime }{j}^{\prime }}-Y_{ij{i}^{\prime }{j}^{\prime }}\times N$$

(5)

$$S_{ij}+X_{ijk}\times P_{ijk}\le C_{ijk}$$

(6)

$$C_{ijk}\le C_{i(j+1)k}$$

(7)

$$S_{ij}\ge C_{i(j-1)}$$

(8)

among them, Equation (3) describes the relationship between the start time and end time of an operation; Equation (4) ensures the sequential order of operations on the same machine to avoid processing conflicts; Equation (5) indicates that each operation can only be processed on a single machine; Equation (6) specifies that the next operation can start only after the current machine finishes its processing; Equation (7) means that a machine can process only one operation at the same time; Equation (8) states that the subsequent operation can start only after the completion of the preceding operation. The framework diagram for solving FJSP via DRL is shown in Figure 3.

Figure 3. Overview of Scheduling Framework.

2.2. Markov Decision Process

Reinforcement learning enables an agent to learn optimal behaviors through trial and error interactions with an environment to maximize cumulative reward, defined by the tuple(S,A,P,R,$\gamma$). In reinforcement learning, at each decision point t, the agent selects an action $a_t\in A$ based on the current state $s_t\in S$ according to a policy $\pi (S\to A)$. The environment provides feedback on the action’s outcome, transitions to a new state st+1 with probability $p\left(s_{t+1}\right|s_t,a_t)\in P(S\times A\to S)$, and yields a reward $r_t\in R(S\times A\times S\to R)$, subsequently updating the state.

A Markov Decision Process (MDP) is a mathematical model describing sequential problems with stochasticity and decision-making, serving as a theoretical foundation for reinforcement learning. A key property of MDPs is that the future state depends only on the current state and current action, independent of past state sequences. This property suits FJSP, which involves processing sequences. A standard MDP is defined by the following quintuple:

1.

State: Operations can be categorized into three groups based on their state: completed, processing, and unscheduled. $O_u\left(t\right)$, $M_u\left(t\right)$ are the sets of relevant operations and machines at step t, A(t) is the set of compatible operation-machine pairs, and state $s_t$ is a collection of feature vectors.

2.

Action: The set of all compatible operation-machine pairs A(t) defines the action space at step t. Over time, as more operations are scheduled, the action space shrinks.

3.

State Transition: An action at corresponds to processing an operation on a machine. Upon taking this action, the environment (i.e., the production state of all operations and machines) changes. Consequently, the sets $O_u\left(t\right)$, $M_u\left(t\right)$, and A(t), along with the features of relevant entities, are updated, resulting in a new state s_t+1.

4.

Reward: The reward rt should be designed to guide the agent towards actions that help reduce the maximum completion time of all operations in the task, i.e., the makespan denoted by Cmax. The formula is as follows:

$$r_t=C_{max}(O_{ij},s_t)-C_{max}(O_{ij},s_{t+1})$$

(9)

5.

Policy: A stochastic policy $\pi \left(a_t\right|s_t)$ is adopted, whose distribution is generated by a DRL algorithm (actor-critic network) with trainable parameters $\theta$. Given a state st, this distribution returns the probability of selecting each action in A(t).

The status, actions, and rewards of MDP are embedded in the FJSP jobs, machines, and operations as shown in the Figure 4.

Figure 4. MDP framework for FJSP.

This section establishes the core theoretical system for the FJSP and lays the research foundation from two aspects: mathematical modeling of the problem and the basic framework of reinforcement learning. Firstly, the standard mathematical model of the FJSP is clarified, defining the core elements and symbolic notations of jobs, operations, and machines, and visualizing the FJSP using a disjunctive graph to clearly sort out the variable meanings and logical relationships of the problem. With minimizing the makespan as the core optimization objective, it systematically elaborates key constraints such as the processing sequence of operations, exclusive occupation of machine resources, and unique assignment of a single operation, fully defining the scheduling rules, optimization direction, and problem boundaries of the FJSP.

Secondly, the theory of MDP is introduced, dissecting its five core elements: state, action, state transition, reward function, and policy. Combined with the scheduling characteristics of the FJSP, it corresponds the states of unscheduled operations and machines, operation-machine matching actions, production environment state updates, makespan-oriented reward design, and stochastic policy decision-making one by one, demonstrating the high compatibility between MDP and the sequential decision-making characteristics of the FJSP. This provides a standardized modeling approach and theoretical support for solving the FJSP via deep reinforcement learning in the subsequent research. The content of this section builds a complete theoretical foundation for the network design, training process, and practical application of the MPPO-DAN model in the following chapters.

3. Method

The Graph Attention Network (GAT) is one of the most popular and effective architectures in the GNN domain. The graph attention layer is the core block of GAT, using an attention mechanism to aggregate neighboring information and compute node embeddings. The attention coefficient eij for each node i and j is calculated as in equation:

$$e_{ij}=LeakyReLU\left(a^T\left[W{h}_1\left|\right|W{h}_2\right]\right)$$

(10)

where $W\in R^{F^{\prime }\times F}$ is a linear transformation shared by all nodes, the operation [$•\left|\right|•$] concatenates two vectors to form a larger vector, and $a\in R^{2F^{\prime }\times 1}$ is the weight vector of a single-layer neural network with LeakyReLU as the activation function. These attention coefficients are further normalized using the softmax function as shown in equation:

$${\alpha }_{ij}=softmax\left(e_{ij}\right)={\displaystyle \frac{exp\left(e_{ij}\right)}{{\sum }_{j\in N}exp\left(e_{ik}\right)}}$$

(11)

Finally, the output is computed as the weighted sum of the transformed neighboring features using the normalized attention coefficients, which is then fed into a nonlinear activation function $\sigma$: R→R (e.g., Exponential Linear Unit [ELU]) to produce the new feature vector.

$$h_i^{\prime }=\sigma \left(\sum _{j\in N}{\alpha }_{ij}W{h}_j\right)$$

(12)

3.1. Dual Attention Network

Traditional GAT typically contains a single graph neural network structure. When applied to FJSP, the node feature extraction effect is often suboptimal because it usually only utilizes GAT to embed operation node features, while machine nodes rely on MLP for embedding. This differing embedding approach—using graph neural networks and multilayer perceptrons separately—limits performance in solving FJSP. In contrast, DAN decouples the complex relationships between operations and machines and between operations themselves, processing them separately with two independent attention modules—the Operation Message Attention Block and the Machine Message Attention Block—significantly improving node embedding capability. Furthermore, compared to the combination of PPO and DAN, the integration of MPPO and DAN leverages its dual Actor network architecture to achieve more optimal embedding of machine nodes and operation nodes separately, demonstrating significant superiority over the most basic traditional graph neural network methods.

• Operation Message Attention Block:
  For each operation $O_{ij}\in O_u$ with input feature $h_{O_{ij}}\in R^{2d}$, this block models the relationship between it and its predecessor operation $O_{i(j-1)}$ and successor operation $O_{i(j+1)}$ by calculating the attention coefficients:

  $$e_{ijp}=LeakyReLU\left(a^T\left[W{h}_{O_{ij}}\left|\right|W{h}_{O_{ip}}\right]\right)$$

  (13)
  The softmax function is used to normalize all $e_{ijp}$ obtaining the normalized attention coefficients ${\alpha }_{ijp}$. Finally, the output feature vector is obtained through a weighted linear combination of the transformed input features $W{h}_{O_{i(j-1)}}$, $W{h}_{O_{ij}}$ and $W{h}_{O_{i(j+1)}}$, followed by the application of a nonlinear activation function $\sigma$:

  $$h_{O_{ij}}^{\prime }=\sigma \left({\displaystyle \sum _{p=j-1}^{j+1}}{\alpha }_{ijp}W{h}_{O_{ip}}\right)$$

  (14)
• Machine Message Attention Block:
  The Machine Message Attention Block utilizes $c_{kq}$ to calculate the attention coefficients $u_{kq}$. or each machine $M_k$ with input feature $h_{M_k}$, the attention coefficients $u_{kq}$ for all $q\in N_k$ are calculated as follows:

  $$u_{kp}=LeakyReLU\left(b^T\left[Z^1{h}_{M_k}\left|\right|Z^1{h}_{M_q}\left|\right|Z^2{h}_{K_q}\right]\right)$$

  (15)

3.2. Proximal Policy Optimization

Policy gradient methods work by computing an estimate of the policy gradient and plugging it into a stochastic gradient ascent algorithm.

$$\widehat{g}={\widehat{E}}_t\left|{▽}_{\theta }log{\pi }_{\theta }\left(a_t\right|s_t){\widehat{A}}_t\right|$$

(16)

where ${\pi }_{\theta }$ is a stochastic policy, ${\widehat{A}}_t$ is an estimator of the advantage function at time step t, and the expectation denotes the empirical average over a finite batch of samples in algorithms that alternate between sampling and optimization. The estimator is obtained by differentiating the objective function.

PPO is a policy gradient-based deep reinforcement learning algorithm proposed by OpenAI. Its core idea is to limit the magnitude of policy updates, ensuring the difference between the new and old policies remains within a controllable range, thereby balancing exploration and exploitation and improving training stability and sample efficiency. The policy update is as shown in equation:

$$L^{CPI}\left(\theta \right)={\widehat{E}}_t\left[{\displaystyle \frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{old}\left(a_t|s_t\right)}}{\widehat{A}}_t\right]={\widehat{E}}_t\left[r_t\left(\theta \right){\widehat{A}}_t\right]$$

(17)

The above update method might lead to excessively large updates, potentially causing destructive large policy updates. Therefore, a clipping method is proposed to define the PPO update as in equation:

$$L^{CLIP}\left(\theta \right)={\widehat{E}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,clip\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t\right)\right]$$

(18)

where $\epsilon$ is a hyperparameter. The first term is LCPI, and the second term is the clipping. By clipping the probability ratio, the agent’s objective is modified, avoiding moving the reward outside the interval $(1-\epsilon ,1+\epsilon )$ and preventing excessively large policy updates. Finally, the minimum of the clipped and unclipped objectives is taken, so the final objective is a lower bound.

A policy gradient style often seen in [46], well-suited for use with recurrent neural networks, involves running the policy for T timesteps and using the collected samples for an update. This style requires an advantage estimator that does not look beyond timestep T. The advantage estimator used in [46] is:

$${\widehat{A}}_t=-V\left(s_t\right)+r_t+\gamma r_{t+1}+\cdots +{\gamma }^{T-t+1}{r}_{T-1}+{\gamma }^{T-t}V\left(s_t\right)$$

(19)

where T specifies the time index in [0, T] within a trajectory segment of length T.

3.3. Multi-Proximal Policy Optimization

Due to the use of the DAN model for machine-operation and operation-operation relationships, employing different operation message attention heads and machine message attention heads, each agent typically only has access to local observations, while the global state of the environment determines the overall reward. MPPO adopts the “Centralized Training with Decentralized Execution” (CTDE) framework: (1) Training Phase: Utilizes global information, such as observations and actions of all agents and the global state, to assist policy optimization. (2) Execution Phase: Each agent makes decisions independently based solely on its own local observations, without requiring global information. MPPO maintains independent policies for each agent but uses a shared “centralized critic” to estimate value leveraging global information, balancing local decision-making and global optimization.

The single Actor network in traditional PPO needs to handle the embedding tasks for both machine nodes and operation nodes simultaneously. The characteristic attributes of these two node types differ significantly—machine nodes emphasize static features like capacity constraints and equipment status, while operation nodes emphasize dynamic features like process dependencies and time parameters. The dual Actor architecture of MPPO achieves specialization in feature extraction: the Actor responsible for machine nodes can focus on learning the embedding patterns of inherent device attributes and states, forming more accurate machine feature vectors; the Actor handling operation nodes can concentrate on encoding process features such as precedence relationships between operations and resource demand matching, generating operation feature representations that better align with actual production logic.

Building upon MPPO, we propose here the use of two different message attention heads applied to the two different Actor networks. To generate the discrete job operation policy ${\pi }_{{\theta }_o}$ and the discrete machine operation policy ${\pi }_{{\theta }_m}$, the two actor networks of Multi-PPO output probability distributions over the job operation space and machine operation space, respectively. Then, job operations and machine operations are selected via a decoding strategy based on these generated probability distributions. The formulas for operations and machines are as follows:

$$L_{CLIP}^h\left({\theta }_h\right)={\widehat{E}}_t\left[min\left({\delta }_t^h\left({\theta }_h\right){\widehat{A}}_t,clip\left({\delta }_t^h\left({\theta }_h\right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t\right)\right]$$

(20)

$$L_E^h\left({\theta }_h\right)={\widehat{E}}_t\left[Entropy\left({\pi }_{{\theta }_h}\left(a_t^h\right|s_t)\right)\right]$$

(21)

Existing studies on FJSP differ mainly in three aspects, namely state representation, decision-making strategy, and adaptability to dynamic environments:

1.

From the perspective of state representation, dispatching rules and metaheuristic algorithms typically rely on handcrafted features, which are insufficient to capture complex relationships among operations and machines. Although DRL-based methods improve representation by learning from data, early approaches still adopt relatively simple state descriptions. Recent studies incorporate GNN or attention mechanisms to enhance representation capability; however, these methods mainly focus on feature extraction and do not explicitly consider its interaction with decision-making. In contrast, the proposed MPPO-DAN framework employs a DAN to capture both local and global dependencies, enabling more expressive and task-relevant state representation.

2.

From the perspective of decision-making, dispatching rules depend on predefined heuristics, limiting flexibility. Metaheuristic algorithms improve solution quality through iterative search but suffer from high computational cost and lack real-time responsiveness. DRL-based methods learn scheduling policies and improve decision efficiency; however, most approaches adopt single-policy learning frameworks, which may restrict their ability to handle complex and dynamic decision spaces. Different from these methods, the proposed MPPO-DAN framework introduces a MPPO mechanism to enhance policy learning, improving decision flexibility and robustness in complex environments.

3.

From the perspective of the interaction between state representation and decision-making, most existing methods treat these two components independently. Dispatching rules and metaheuristic algorithms rely on predefined features and separate optimization procedures, while many DRL-based approaches focus either on improving representation or enhancing policy learning, without explicitly modeling their interaction. Such decoupled designs limit the effectiveness of learned policies, as the representation may not be fully aligned with decision requirements. In contrast, the proposed MPPO-DAN framework jointly optimizes representation learning and policy learning, enabling better coordination between feature extraction and decision-making, thereby improving overall scheduling performance.

Overall, existing methods either rely on limited representation, inefficient decision-making, or decoupled optimization of representation and policy. In contrast, the proposed MPPO-DAN framework provides a unified solution that integrates these aspects, thereby achieving superior scheduling performance and generalization ability.

The overall flow of the model is as follows: An FJSP instance is divided into n jobs and m machines, where each job has different operations $O_{ij}$. It is converted into a disjunctive graph. The Dual Attention Network (DAN) is used to embed machine nodes and operation nodes. Subsequently, MPPO is used to train two different actor networks and one critic network for machines and operations, respectively. Finally, it is applied to FJSP. The process is illustrated in the Figure 5 below.

Figure 5. Flow Chart of MPPO Algorithm.

The pseudocode for this Algorithm 1 is as follows:    

Algorithm 1: Training with MPPO-DAN

Input: Initialize parameters

Output: Generate schedule using optimal policy, compute Cmax.

1  Load FJSP instance (SD1/SD2)

2  Initialize machine set M and job set J

3  Construct disjunctive graph G = {O,C,D}

4  Build Dual Attention Network

5  Initialize Operation Message Attention Block parameters

6  Initialize Machine Message Attention Block parameters

7  Build Actor Networks

8  Actor_O ← MLP(hidden_layers [64, 64])

9  Actor_M ← MLP(hidden_layers [64, 64])

10  Critic ← MLP(hidden_layers [64, 64])

11  for episode in 1 to Episodes do:

12     s = get_initial_production_state()

13     trajectory = []

14     while unscheduled operations exist:

15        Extract relevant operation/machine features → state vector;

16        Compute operation dependencies & machine competition features via DAN;

17        action = sample (Actor_O(s), Actor_M(s));

18        s_next, reward = environment.execute (action);

19        trajectory.append((s, action, reward, s_next));

20     s = s_next;

21  end for

22  returns = compute GAE(trajectory, $\gamma$, $\lambda$);

23  Update policy network parameters in batch

4. Experiments

4.1. Datasets

An FJSP instance with n jobs and m machines is denoted as n × m. Two datasets are used here: SD1 and SD2. SD1 is adapted from literature [29] for experimental validation of FJSP. Its core characteristic is allowing jobs to contain a variable number of operations, meaning the number of operations can vary dynamically between jobs (e.g., some jobs have multiple operations, others have few). SD2 contains tasks with a wider range of processing times. For an n × m FJSP instance, each job contains m operations. This paper proposes applying MPPO to SD1 with varying job numbers and SD2 with wide processing time ranges to comprehensively evaluate the algorithm’s performance in diverse scheduling environments. The number of available machines |$M_{ij}$| and processing time $p_{ijk}$ for each operation Oijare sampled from uniform distributions U(1, m) and U(1, 99), respectively. This constructs scheduling scenarios with wide processing time ranges and high machine assignment flexibility, used to test the algorithm’s robustness under high uncertainty and large numerical variations.

Similarly, to validate the model’s generalization ability in real scenarios, this paper proposes applying MPPO to public benchmarks representing problems of different scales: mk for evaluating the algorithm’s universal performance under common job-machine configurations; Hurink_edata [47] with relaxed constraints for testing scheduling effectiveness in highly flexible environments; Hurink_rdata [47] with resource constraints for examining adaptability under limited machine availability; and Hurink_vdata [47] with mixed constraints for comprehensive evaluation of overall performance in complex heterogeneous scenarios. These four public benchmark datasets can test the model’s generalization ability on out-of-distribution data, verify whether the model truly learns the structural knowledge of FJSP rather than overfitting to the training distribution, and enable fair comparison with traditional methods (e.g., OR-Tools, scheduling rules), comprehensively validating the advantages of the proposed method in structural diversity, constraint complexity, and out-of-distribution generalization.

On the specific SD1 and SD2 datasets [29], as well as the five public benchmark datasets, we use 6 different scales of FJSP instances: $10\times 5$, $20\times 5$, $15\times 10$, $20\times 10$, $30\times 10$ and $40\times 10$.

4.2. Parameters

Parameter settings are: Episodes = 1000, DAN uses L = 2 dual attention layers, each block uses H = 4 attention heads per layer, ELU is used as the activation function $\sigma$. The output dimension for each attention head in both blocks is d(1)o = d(1)m = 32 for the first layer and d(2)o = d(2)m = 8 for the second layer. MLP$\theta$ and MLP$\varphi$ both have two hidden layers with dimension 64, using tanh as the activation. For PPO parameters, the policy, value, and entropy coefficients in the loss function are set to 1, 0.5, and 0.01, respectively. The clipping parameter $\epsilon$ = 0.2, GAE parameter $\lambda$ = 0.98, discount factor $\gamma$ = 0.99.

All experiments were conducted on a machine with an Intel 12600KF and Nvidia RTX4070super.

4.3. Compared Algorithms

In terms of scheduling rules, four PDRs [48,49] that have shown good performance in FJSP were selected. Descriptions of these four PDRs are as follows:

1.

First-In-First-Out (FIFO): Selects the earliest ready candidate operation and the earliest ready compatible machine;

2.

Most Operations Remaining (MOPNR): Selects the candidate operation with the most remaining successors and a machine that can process it immediately;

3.

Shortest Processing Time (SPT): Selects the compatible operation-machine pair with the shortest processing time;

4.

Most Work Remaining (MWKR): Selects the candidate operation with the highest average processing time of remaining successors and a machine that can process it immediately.

In terms of DRL algorithms, the HGNN combined with PPO proposed by Song et al. [31] and the DAN combined with PPO proposed by Wang et al. [33] were selected for comparison.

4.4. Convergence Curve

To begin with, the convergence of reward functions and makespan for MPPO-DAN and PPO-DAN in the aforementioned datasets was verified, and Figure 6 presents their convergence curves. As shown, MPPO-DAN achieves a better convergence value than PPO-DAN.

Figure 6. Reward and Makespan; (a) reward of size 10 × 5; (b) makespan of size 10 × 5; (c) reward of size 20 × 5; (d) makespan of size 20 × 5; (e) reward of size 15 × 10; (f) makespan of size 15 × 10.

Figure 6 for the 10 × 5 problem scale, MPPO-DAN demonstrates a significantly faster convergence speed, outperforming PPO-DAN remarkably.

4.5. Performance Comparison

Using dispatching rules for comparison is essential because they serve as well-established, intuitive, and widely adopted baselines in scheduling research. These heuristic rules, such as Shortest Processing Time, are deeply ingrained in industrial practice due to their simplicity, computational efficiency, and ease of implementation. By comparing a new sophisticated algorithm against these classical rules, researchers can concretely demonstrate its relative performance improvement, whether in solution quality, efficiency, or robustness. This comparison provides a critical reality check, validating that the proposed advanced method offers tangible benefits over simple, understandable, and practical approaches already in use. Experiments were conducted on six test instances of different scales to compare the proposed MPPO-DAN algorithm with four traditional scheduling rules. The makespan was used as the evaluation metric, and the experimental results are listed in Table 3. The results demonstrate that the scheduling performance of MPPO-DAN is superior to that of the traditional scheduling rules, verifying the advantages of the MPPO-DAN algorithm. The bolded text in the following tables indicates the best performance.

Table 3. Comparison of Scheduling Rules and Google: OR-Tools.

Size			SPT	MWKR	MOR	FIFO	Ours	OR-Tools
SD1	10 × 5	makespan	129.82	113.23	115.38	119.40	103	96.32
		time	0.16	0.16	0.16	0.16	3.42	205
	20 × 5	makespan	230.48	209.78	214.16	216.08	199.95	188.15
		time	0.32	0.33	0.33	0.33	6.36	411
	15 × 10	makespan	198.33	171.25	173.15	184.55	161.85	143.53
		time	0.51	0.51	0.52	0.52	5.31	659
	20 × 10	makespan	255.17	216.11	219.80	233.48	195.98	195.8
		time	0.70	0.70	0.70	0.71	8.3	887
SD2	10 × 5	makespan	514.38	549.27	557.48	569.40	413.8	326.24
		time	0.16	0.16	0.16	0.16	0.94	202
	20 × 5	makespan	835.94	1026.03	1045.94	1045.83	667.75	602.04
		time	0.32	0.33	0.33	0.33	2.13	408
	15 × 10	makespan	703.07	830.53	571.14	845.16	590.65	377.17
		time	0.51	0.51	0.52	0.52	9.83	656
	20 × 10	makespan	829.14	1040.69	1059.68	1088.05	610.05	464.16
		time	0.70	0.70	0.70	0.71	14.51	883

From Table 3 can be seen that the computation times for the four scheduling rules can all be controlled within a small range, but their makespan values show a certain gap compared to the three DRL algorithms mentioned above. Although OR-Tools yields the best results, its excessively long computation time lacks practical significance. MPPO-DAN’s results are relatively close to OR-Tools’ optimal solutions while having much shorter training times. Among these three types, DRL algorithms exhibit stronger comprehensive performance.

The above experiments have confirmed the superiority of MPPO-DAN over traditional scheduling rules. To further validate the effectiveness of MPPO-DAN in deep reinforcement learning (DRL)-based scheduling, we conducted a comparative analysis between the proposed MPPO-DAN and two state-of-the-art methods, namely PPO-HGNN [31] and PPO-DAN [33]. The three algorithms were tested on multiple instances with different scales, and the comparative results are presented in Table 4 and Table 5. The results show that the performance of MPPO-DAN is better than that of both PPO-HGNN and PPO-DAN.

Table 4. Comparison of MPPO-DAN, PPO-DAN, and PPO-HGNN.

Size			Ours	PPO-HGNN [31]	PPO-DAN [33]
SD1	10 × 5	makespan	103.00	111.67	103.3
		time	3.42	0.45	0.9
	20 × 5	makespan	199.95	211.22	201.6
		time	6.36	0.9	5.15
	15 × 10	makespan	161.85	166.92	164.75
		time	5.31	1.43	5.09
	20 × 10	makespan	195.8	215.78	197.9
		time	8.3	1.91	8.36
SD2	10 × 5	makespan	413.8	553.61	415.45
		time	0.94	0.46	0.96
	20 × 5	makespan	667.75	1059.04	670.1
		time	2.13	0.94	1.9
	15 × 10	makespan	590.65	807.47	592.55
		time	9.83	1.5	4.5
	20 × 10	makespan	610.05	1045.82	617.4
		time	14.51	1.95	8.29

Table 5. Comparison for Large-Scale Instances.

Size			Ours	PPO-HGNN [31]	PPO-DAN [33]
SD1	30 × 10	makespan	285.00	313.04	288.61
		time	4.79	2.84	2.78
	40 × 10	makespan	379.28	416.18	385.35
		time	7.85	3.81	3.77
SD2	30 × 10	makespan	761.25	1543.69	794.62
		time	4.67	2.93	2.8
	40 × 10	makespan	983.37	2032.54	1094.11
		time	7.59	3.93	3.76

From Table 5, it can be seen that MPPO-DAN still maintains an advantage in large-scale comparisons. Public benchmark [49] datasets are essential to ensure fairness, comparability, and reproducibility in research. They provide a standardized set of test instances that are widely recognized and often derived from real-world industrial scenarios or established literature. By evaluating algorithms on these common grounds, researchers can objectively compare the performance of different methods, avoid overfitting to specific synthetic data distributions, and rigorously assess an algorithm’s generalization capability across various problem scales and structures.

Furthermore, using benchmarks facilitates transparency and allows the research community to track progress over time, ensuring that advancements are meaningful and verifiable.The comparison results are shown in Table 6, Table 7 and Table 8.

Table 6. Comparison of DRL Makespan for 5 Scales of BenchData (edata).

Size	BenchData	Ours	PPO-DAN [33]	PPO-HGNN [31]
	Hurink_edata1	1046.48	1055.63	1116.18
	Hurink_edata2	973.3	978.79	1015.38
10 × 5	Hurink_edata3	879.99	887.31	973.3
	Hurink_edata4	925.73	933.05	969.64
	Hurink_edata5	847.06	856.21	927.56
	Hurink_edata6	1961.23	1964.89	2014.29
	Hurink_edata7	1714.25	1717.91	1738.03
20 × 5	Hurink_edata8	1900.86	1904.52	1926.47
	Hurink_edata9	1959.4	1963.06	1986.85
	Hurink_edata10	1994.16	1997.82	2032.58
	Hurink_edata11	1511.17	1566.06	1622.77
	Hurink_edata12	1373.96	1428.85	1450.8
15 × 10	Hurink_edata13	1525.81	1569.72	1569.72
	Hurink_edata14	1456.29	1505.69	1615.46
	Hurink_edata15	1423.36	1470.92	1615.46
	Hurink_edata16	1948.43	1970.38	1992.33
	Hurink_edata17	2010.63	2023.44	2054.54
20 × 10	Hurink_edata18	1983.19	1999.65	2023.44
	Hurink_edata19	1842.31	1864.27	1919.15
	Hurink_edata20	1981.36	2003.31	2043.56
	Hurink_edata21	2791.83	2795.49	2855.86
	Hurink_edata22	3047.96	3045.96	3097.36
30 × 10	Hurink_edata23	2751.58	2750.41	2800.98
	Hurink_edata24	2819.27	2822.93	2857.69
	Hurink_edata25	2846.71	2843.71	2879.65

Table 7. Comparison of DRL Makespan for 5 Scales of BenchData (rdata).

BenchData	Size	Ours	PPO-DAN [33]	PPO-HGNN [31]
	Hurink_rdata1	924.58	932.66	986.48
	Hurink_rdata2	859.92	864.77	897.1
10 × 5	Hurink_rdata3	777.49	783.95	859.92
	Hurink_rdata4	817.9	824.36	856.69
	Hurink_rdata5	748.39	756.47	819.51
	Hurink_rdata6	1732.77	1736.01	1779.65
	Hurink_rdata7	1514.56	1517.79	1535.57
20 × 5	Hurink_rdata8	1679.43	1682.67	1702.06
	Hurink_rdata9	1731.16	1734.39	1755.4
	Hurink_rdata10	1761.87	1765.1	1795.81
	Hurink_rdata11	1335.14	1383.63	1433.74
	Hurink_rdata12	1213.91	1262.4	1281.8
15 × 10	Hurink_rdata13	1348.07	1386.87	1386.87
	Hurink_rdata14	1286.65	1330.29	1427.28
	Hurink_rdata15	1257.55	1299.58	1427.28
	Hurink_rdata16	1721.46	1740.86	1760.25
	Hurink_rdata17	1776.42	1787.73	1815.21
20 × 10	Hurink_rdata18	1752.17	1766.72	1787.73
	Hurink_rdata19	1627.71	1647.1	1695.6
	Hurink_rdata20	1750.55	1769.95	1805.51
	Hurink_rdata21	2469.85	2466.62	2523.19
	Hurink_rdata22	2692.91	2692.91	2736.55
30 × 10	Hurink_rdata23	2432.67	2431.06	2474.7
	Hurink_rdata24	2490.86	2494.1	2524.81
	Hurink_rdata25	2515.11	2515.41	2544.2

Table 8. Comparison of DRL Makespan for 5 Scales of BenchData (vdata).

BenchData	Size	Ours	PPO-DAN [33]	PPO-HGNN [31]
	Huvink_vdata1	873.94	881.58	932.45
	Huvink_vdata2	812.83	817.41	847.97
10 × 5	Huvink_vdata3	734.9	741.02	812.83
	Huvink_vdata4	773.1	779.21	809.77
	Huvink_vdata5	707.4	715.04	774.63
	Huvink_vdata6	1637.88	1640.93	1682.18
	Huvink_vdata7	1431.61	1434.67	1451.48
20 × 5	Huvink_vdata8	1587.46	1590.51	1608.85
	Huvink_vdata9	1636.35	1639.4	1659.27
	Huvink_vdata10	1665.38	1668.43	1697.46
	Huvink_vdata11	1262.02	1307.86	1355.22
	Huvink_vdata12	1147.43	1193.27	1211.6
15 × 10	Huvink_vdata13	1274.24	1310.91	1310.91
	Huvink_vdata14	1216.18	1257.44	1349.11
	Huvink_vdata15	1188.68	1228.41	1349.11
	Huvink_vdata16	1627.18	1645.51	1663.85
	Huvink_vdata17	1679.13	1689.82	1715.8
20 × 10	Huvink_vdata18	1656.21	1669.96	1689.82
	Huvink_vdata19	1538.56	1556.9	1602.73
	Huvink_vdata20	1654.68	1673.02	1706.63
	Huvink_vdata21	2331.53	2334.58	2385
	Huvink_vdata22	2545.43	2542.43	2586.68
30 × 10	Huvink_vdata23	2299.44	2297.91	2339.17
	Huvink_vdata24	2354.45	2357.5	2386.53
	Huvink_vdata25	2377.36	2373.36	2404.87

From Table 4 and Table 5, it can be observed that MPPO-DAN consistently achieves superior performance to both PPO-DAN and PPO-HGNN across small-scale problem sizes (i.e., 10 × 5, 20 × 5, 15 × 10, and 20 × 10). Notably, for the 15 × 10 problem size, MPPO-DAN outperforms PPO-DAN by an average of 3.84%.

4.6. Comparison Curves for Different Parameters

Subtle differences in parameter configuration often significantly impact the final results. To verify the optimality of the current parameter settings, we conducted comparative experiments on several common parameters, including different learning rates, hidden layer dimensions, and actor/critic network depths. Through systematic testing of these parameter variables under the same experimental conditions, we performed quantitative analysis from key performance metrics of the model. The comparison results clearly show that the current parameter combination performs best in balancing training efficiency and performance stability, proving its rationality and superiority in adapting to task requirements compared to other parameter configurations. As shown in Figure 7.

Figure 7. Comparison of A-C Layers (1/2/3); (a) Reward; (b) makespan.

From Figure 7, experimental results show that under settings of 1, 2, and 3 layers, the models with 2 and 3 layers achieve similar final convergence performance, both significantly outperforming the model with 1 layer. Furthermore, the model with 3 layers converges noticeably faster than the one with 2 layers. This is primarily because deeper network structures (e.g., 3 layers) possess stronger nonlinear representation capabilities, enabling more effective learning of complex features. Additionally, with reasonable initialization and optimization, the cumulative effect of gradient propagation in deep networks can sometimes accelerate convergence in the early training stages, improving training efficiency. As shown in Figure 8.

Figure 8. Comparison of Hidden Layers (32/64/128); (a) Reward; (b) makespan.

From Figure 8, when the hidden layer dimensions for both Actor and Critic are set to 32, 64, and 128, the final convergence results of the three are relatively close, indicating that the model’s fitting capability within this dimension range meets the demands of the scheduling problem. However, further analysis of the convergence process reveals that with a hidden layer dimension of 32, the model converges significantly slower. This is because the lower dimension limits feature expression capability, requiring more iterations for the network to fully learn effective information. In contrast, hidden layers of 64 and 128 dimensions can complete the training process more efficiently. As shown in Figure 9.

Figure 9. Comparison of Learning Rates ($1\times {10}^{-4}$/$3\times {10}^{-4}$/$1\times {10}^{-3}$); (a) Reward; (b) makespan.

From Figure 9, when the learning rates are set to $1\times {10}^{-4}$, $3\times {10}^{-4}$, and $1\times {10}^{-3}$, the final convergence values of the models show small differences, indicating that these three learning rates can achieve similar precision in final optimization effects. However, in terms of convergence efficiency, the model with a learning rate of $1\times {10}^{-4}$ shows a significant gap in convergence speed compared to the other two. This is because an excessively small learning rate leads to limited parameter update steps, requiring more iteration steps to approach the optimal solution. In contrast, learning rates of $3\times {10}^{-4}$ and $1\times {10}^{-3}$ achieve a better balance between update magnitude and stability, thereby accelerating the convergence process.

5. Conclusions

Addressing the complexity and dynamics of the Flexible Job Shop Scheduling Problem (FJSP), this paper proposed the MPPO-DAN model to enhance the efficiency and quality of scheduling decisions. The main research contents and conclusions are as follows:

Multi-Proximal Policy Optimization (MPPO) was introduced into the Dual Attention Network (DAN). DAN’s dual attention blocks model operation dependencies and machine competition relationships separately, extracting key features. MPPO’s dual actor networks achieve joint decision-making for operation sequencing and machine assignment, while the centralized critic utilizes global information to optimize the strategy, balancing training efficiency and decision accuracy. Based on the SD1 and SD2 datasets, across four different scales (10 × 5 to 20 × 10) of FJSP instances, MPPO-DAN was compared with PPO-DAN, PPO-HGNN, traditional scheduling rules, and OR-Tools. The results show that MPPO-DAN performs better in the makespan metric (e.g., in the 20 × 10 instance of SD1, MPPO-DAN’s makespan is 195.8, PPO-DAN’s 197.9 and PPO-HGNN’s 215.78), has lower loss values, a more stable training process, and stronger generalization ability.

Main contributions: First, the effective integration of MPPO and DAN was achieved, enhancing FJSP solving performance. Second, it was verified that this model has superior scheduling effects and stability compared to methods combining standard PPO with HGNN or DAN. In summary, MPPO-DAN provides an efficient and stable new method for solving FJSP, applicable to production scenarios of different scales, with strong practical application value. Future work could extend it to multi-objective optimization scenarios and explore its solving capability for larger-scale FJSP instances.