Adaptive Lagrangian Penalty-Enhanced Proximal Policy Optimization for Flexible Job Shop Rescheduling with Worker Workload Constraints Under Concurrent Dynamic Disturbances

Abstract

When flexible job shop scheduling faces concurrent disturbances such as machine failures and rush orders, worker-centric constraints emphasized under Industry 5.0 must also be satisfied. Existing deep reinforcement learning methods for the Dynamic Flexible Job Shop Scheduling Problem (DFJSP) seldom treat worker workload balance as an explicit constraint, and most depend on static penalty coefficients that are difficult to tune across different scenarios. In this paper, we suggest ALP-PPO, an adaptive Lagrangian penalty-enhanced proximal policy optimization algorithm, for real-time rescheduling under concurrent machine breakdowns and rush orders. We formulate the scheduling environment as a constrained Markov decision process. Worker skill heterogeneity, fatigue accumulation and workload equity are modeled as coupled constraints alongside classical scheduling objectives. By decoupling operation sequencing, machine allocation and worker assignment into coordinated sub-decisions, a hierarchical action space is constructed. Dual Lagrangian multipliers for workload balance and fatigue are updated adaptively during training, so that manual penalty tuning is no longer required. An event-triggered mechanism selects between right-shift and full rescheduling on the basis of a disruption severity index. We employ weighted-sum scalarization of makespan, energy consumption and workload variance during training, and Pareto solution sets are obtained by systematically varying the weight vectors across independent training runs. On extended Brandimarte benchmarks augmented with worker and dynamic event parameters, ALP-PPO delivers superior scheduling performance across makespan, energy consumption and workload variance when compared with Double DQN, Dueling DQN, standard PPO, NSGA-II and MOEA/D, as measured by Hypervolume (HV) and Inverted Generational Distance (IGD) indicators. Ablation studies indicate that the adaptive Lagrangian mechanism reduces constraint violations by more than 40% relative to fixed-penalty alternatives while keeping the primary objectives competitive. An analysis of computational efficiency shows that ALP-PPO completes online inference in under 20 ms per decision step, making real-time rescheduling practically feasible. Generalization experiments on previously unseen instances further validate the transferability of the learned policy. These findings support human-centric intelligent scheduling in Industry 5.0 manufacturing.

1. Introduction

A paradigm shift is under way in global manufacturing: the efficiency-driven automation characteristic of Industry 4.0 is giving way to the human-centric, sustainable and resilient vision of Industry 5.0 [1,2]. Under this new paradigm, worker well-being, workload equity and adaptive responsiveness to disruptions have become evaluation criteria for production scheduling alongside throughput and machine utilization [3]. Flexible job shop scheduling (FJSP), a canonical NP-hard problem, permits each operation to be processed on any machine within a candidate set [4,5]. Because of its routing and machine flexibility, FJSP serves as a natural testbed for scheduling algorithms that must optimize multiple conflicting objectives under dynamic and uncertain conditions.

Two categories of disturbance commonly invalidate pre-computed schedules on manufacturing shop floors. Machine failures, arising from aging, overload or component fatigue, force the immediate reassignment of pending and in-progress operations [6]. Rush order arrivals, triggered by urgent customer demands or disruptions in the supply chain, inject new jobs with tight deadlines into an already loaded system [7]. The feasible solution space shrinks rapidly and workloads may become severely skewed when these two types of disturbance occur simultaneously.

Traditional DFJSP approaches can be broadly divided into meta-heuristic and dispatching-rule-based methods. Genetic algorithms [8,9], discrete optimization heuristics [10] and ensemble evolutionary methods [11] perform well on static instances but encounter computational bottlenecks when used for real-time rescheduling, because each disruption triggers a fresh optimization cycle. Scalability on large FJSP instances has been improved by recent hybrid algorithms that combine evolutionary strategies with problem-specific operators [12], and weighted-sum scalarization has demonstrated its effectiveness when scheduling objectives can be combined through domain-informed coefficients [13]. Nevertheless, the computation required per rescheduling event in these methods remains considerable, which limits their suitability for real-time settings.

DRL-based frameworks have recently addressed job shop and flexible job shop scheduling from several perspectives. Value-based methods, including Deep Q-Networks and their variants, have been used in both static and dynamic FJSP with encouraging results [14,15,16]. Among policy gradient methods, PPO [17] offers stable training and handles high-dimensional action spaces naturally [18,19]. Multi-agent architectures that decompose scheduling into cooperating sub-agents demonstrate advantages in distributed decision-making [16,20], and graph neural network representations of the relationships between jobs and machines improve state representation and generalization across problem sizes [21,22]. Meta-learning techniques have also been explored to enable rapid adaptation of scheduling policies to new problem configurations [23], and attention-based architectures have demonstrated promise in estimating complex inter-operation dependencies [24].

Despite these advances, three gaps persist. First, worker-related constraints are largely absent from DRL-based FJSP formulations, in which machines are typically the only constrained resource. In practice, workers possess heterogeneous skills that affect processing efficiency and quality [25], and prolonged operation leads to fatigue that raises error rates and compromises safety [26]. When workload distribution across workers is unbalanced, team morale and productivity decline over extended production horizons. Industry 5.0 emphasizes human-centricity and therefore requires scheduling algorithms to explicitly account for these worker-side factors alongside maintenance-coupled scheduling decisions; however, integrating worker workload constraints into DRL-based FJSP in a systematic manner remains underexplored [3,27].

Second, constraint-handling mechanisms in DRL for scheduling predominantly rely on fixed penalty terms appended to the reward function. Penalty-based approaches are straightforward to implement, yet the choice of penalty coefficients is highly sensitive to problem scale and constraint tightness. If the penalty is too small, frequent constraint violations occur; if it is too large, the primary scheduling objective is overwhelmed and overly conservative policies result. Tuning these coefficients manually across different production scenarios is labor-intensive and often suboptimal. By converting hard constraints into soft penalties whose multipliers are updated adaptively, Lagrangian relaxation provides a principled alternative and has been used in constrained reinforcement learning for safety-critical domains [28,29]. Existing PPO-Lagrangian methods [28] typically employ a simple gradient ascent rule to update a single multiplier, and this can exhibit slow convergence or oscillatory behavior when multiple heterogeneous constraints interact. How to apply Lagrangian relaxation to DFJSP with coupled worker workload and fatigue constraints, where the scales and dynamics of the constraints differ substantially, has not been investigated.

Third, a single rescheduling strategy is applied by most DRL-based rescheduling methods regardless of disruption severity. In practice, minor disturbances can often be accommodated by right-shifting affected operations without altering the overall schedule topology, whereas major disturbances call for complete rescheduling of all remaining operations to restore global optimality [6,16]. An adaptive switching mechanism that selects between right-shift and full rescheduling on the basis of quantifiable production state metrics would improve both response efficiency and schedule stability, but such disruption-severity-aware switching has received limited attention in the DRL-based scheduling literature.

We address these gaps by suggesting an adaptive Lagrangian penalty-enhanced proximal policy optimization (ALP-PPO) algorithm for flexible job shop rescheduling with worker workload constraints under concurrent machine failures and rush order arrivals. The main contributions are summarized as follows.

• A CMDP formulation for DFJSP is developed that incorporates worker skill heterogeneity, fatigue and workload balance as explicit constraints alongside the classical objectives of makespan, energy and machine load. The thresholds of the constraints are calibrated from occupational ergonomics guidelines and practical equity requirements, which gives a principled basis for the CMDP modeling.
• An adaptive Lagrangian mechanism updates dual multipliers for workload and fatigue constraints using a proportional–integral (PI) update rule each epoch; this differs from standard PPO-Lagrangian methods [28] by incorporating an integral term that accumulates historical constraint violations, thereby preventing persistent undershoot and improving the stability of convergence when multiple heterogeneous constraints with different scales interact, and manual penalty tuning is avoided.
• A hierarchical action space splits each decision into operation priority, machine allocation and worker assignment, which reduces the size of the action space and enables structured exploration.
• A disruption severity index is formulated to quantify the impact of concurrent dynamic events, and an event-triggered rescheduling mechanism that automatically selects between right-shift rescheduling and full rescheduling on the basis of this index is developed.
• Comprehensive experiments on extended Brandimarte benchmark instances augmented with worker and dynamic event parameters demonstrate the effectiveness of ALP-PPO across multiple objectives and disturbance scenarios. The evaluation includes Hypervolume (HV) and Inverted Generational Distance (IGD) indicators for Pareto front quality assessment, comparisons with multi-objective evolutionary algorithms (NSGA-II [30] and MOEA/D [31]), computational efficiency profiling, reward weight sensitivity analysis and generalization experiments on previously unseen instances. Ablation studies isolate the contributions of the adaptive Lagrangian mechanism, the hierarchical action space and the rescheduling strategy selection.

The remainder of this paper is organized as follows. Section 2 reviews related work on FJSP optimization, DRL-based scheduling, worker-constrained scheduling and constrained reinforcement learning. Section 3 presents the mathematical model of the worker-constrained DFJSP. Section 4 describes the ALP-PPO algorithm in detail. Section 5 reports the experimental design and results. Section 6 concludes the paper and outlines future research directions.

2. Related Work

2.1. Meta-Heuristic Methods for FJSP

Machine flexibility was first introduced into the classical job shop problem by Brucker and Schlie [4], who formalized FJSP. Since then, a wide range of meta-heuristic algorithms have been developed. In the area of genetic algorithms, Caldeira and Gnanavelbabu [8] suggested a Pareto-based discrete Jaya algorithm for multi-objective FJSP and achieved improved coverage of the Pareto front on benchmark instances. More recently, meta-heuristic research has shifted toward multi-objective formulations that simultaneously optimize makespan, energy consumption and machine load. Hou et al. [32] suggested a co-evolutionary framework that deeply integrates NSGA-III with DRL at the level of genetic operators for distributed FJSP. Wang et al. [33] developed an improved MOEA/D approach for low-carbon many-objective flexible job shop scheduling and incorporated environmental objectives into the optimization framework. NSGA-II [30] and MOEA/D [31] continue to be widely adopted baselines in the multi-objective optimization literature for evaluating the quality of Pareto fronts, with Hypervolume (HV) and Inverted Generational Distance (IGD) serving as standard performance indicators [34]. These advances demonstrate the continued vitality of population-based methods, but they also highlight the computational cost that arises when real-time rescheduling under dynamic disturbances is needed.

2.2. Deep Reinforcement Learning for Dynamic Scheduling

To decompose the complex decision space of FJSP, multi-agent DRL architectures have been suggested. Wang et al. [16] developed a multi-agent framework using a 3DQN algorithm in which process selection agents and machine selection agents are coordinated for DFJSP, and its effectiveness was demonstrated on the Brandimarte benchmark under machine failures and order insertions. Zhu et al. [20] designed a multi-task multi-agent reinforcement learning system for dual-resource FJSP with robots, addressing real-time scheduling requirements on automated production lines. Luo et al. [35] suggested a DRL-based real-time scheduling method for dynamic partial-no-wait multi-objective FJSP and introduced a composite reward function that balances multiple scheduling objectives during online decision-making.

Graph neural network (GNN) representations have been increasingly adopted to represent the topological structure of the relationships between jobs and machines. Song et al. [21] employed heterogeneous graph attention networks in which operations and machines are represented as nodes with typed edges, and strong generalization across different problem sizes was achieved. Park et al. [22] combined GNN-based state encoding with policy gradient methods and demonstrated that learned graph representations transfer more effectively than handcrafted feature vectors. Meta-learning approaches have also been explored to accelerate policy adaptation: Wang et al. [23] suggested a meta-reinforcement learning framework that pre-trains scheduling policies on a distribution of problem instances and then rapidly fine-tunes for new configurations, and the number of training episodes required for convergence is thereby reduced. Attention-based architectures [24] offer an alternative to GNNs by directly modeling pairwise interactions among operations without the need for explicit graph construction.

2.3. Constrained Reinforcement Learning

Constrained Markov decision processes extend standard MDPs by introducing cost functions whose expected cumulative cost must not exceed specified thresholds [36]. Jayant and Bhatnagar [28] suggested a model-based safe deep reinforcement learning method based on a constrained proximal policy optimization algorithm and demonstrated effective constraint handling in safety-critical environments. Ma et al. [29] introduced a conservative and adaptive penalty approach for model-based safe reinforcement learning, laying foundations for adaptive constraint enforcement during policy optimization.

Applications of constrained RL in manufacturing scheduling remain at an early stage. Yang et al. [37] used deep reinforcement learning for real-time scheduling of distributed permutation flowshops with dynamic job arrivals and demonstrated the viability of learned policies for online scheduling decisions. Zhang et al. [38] developed a deep reinforcement learning-based memetic algorithm for energy-aware flexible job shop scheduling with multi-AGV, combining learned policies with evolutionary search for enhanced performance. Existing PPO-Lagrangian approaches [28,29] have proven effective in robotic and safety-critical tasks, yet they typically employ a single Lagrangian multiplier updated by a proportional-only rule and target a single constraint type. In DFJSP with worker constraints, the workload balance constraint and the fatigue constraint operate at different time scales and exhibit different magnitudes; a proportional-only update can therefore lead to oscillatory multiplier trajectories or persistent constraint undershoot. Furthermore, the interaction between worker-side constraints and machine-side scheduling objectives creates coupled dynamics that have not been addressed in existing constrained RL formulations. We extend this line of work by designing an adaptive Lagrangian mechanism specifically tailored to the coupled worker workload and scheduling performance constraints in DFJSP.

3. Problem Description and Mathematical Model

3.1. Basic FJSP Formulation

Consider a flexible job shop with n jobs $J=\{J_1,J_2,\dots ,J_n\}$ and m machines $M=\{M_1,M_2,\dots ,M_m\}$. Each job $J_i$ consists of an ordered sequence of $h_i$ operations $O_i=\{O_{i1},O_{i2},\dots ,O_{i{h}_i}\}$. Each operation $O_{ij}$ can be processed on any machine from its candidate set $M_{ij}\subseteq M$. We denote by $p_{ijk}$ the processing time of operation $O_{ij}$ on machine $M_k$ and by $P_{ijk}$ (in kW) the corresponding energy consumption rate.

Table 1. Notation and definitions.

Symbol	Definition
n	Total number of jobs
m	Total number of machines
w	Total number of workers
$h_i$	Number of operations for job $J_i$
$O_{ij}$	The j-th operation of job $J_i$
$p_{ijk}$	Processing time of $O_{ij}$ on machine $M_k$
$P_{ijk}$	Power consumption rate of $O_{ij}$ on machine $M_k$ (kW)
$S_{ijk}$	Start time of $O_{ij}$ on machine $M_k$
$C_{ijk}$	Completion time of $O_{ij}$ on machine $M_k$
$C_{max}$	Makespan (maximum completion time)
$E_{\mathrm{total}}$	Total energy consumption
$\mathit{TML}$	Total machine load
$x_{ijk}$	Binary variable: 1 if $O_{ij}$ is assigned to $M_k$, 0 otherwise
$y_{ij{i}^{\prime }{j}^{\prime }k}$	Binary variable: 1 if $O_{ij}$ precedes $O_{i^{\prime }{j}^{\prime }}$ on $M_k$
$W_l$	Worker l, $l\in \{1,2,\dots ,w\}$
${\eta }_{lk}$	Skill coefficient of worker $W_l$ on machine $M_k$
$z_{ijkl}$	Binary variable: 1 if $O_{ij}$ on $M_k$ is operated by $W_l$
${\varphi }_l\left(t\right)$	Cumulative fatigue level of worker $W_l$ at time t
${\mathit{WL}}_l$	Total workload of worker $W_l$
${\lambda }_k$	Failure rate of machine $M_k$
$T_{\mathrm{repair}}$	Repair duration after machine failure
H	A sufficiently large positive number

summarizes the notation used throughout this paper.

3.2. Multi-Objective Functions

Three scheduling objectives are considered simultaneously. Following the weighted-sum scalarization approach commonly adopted in DRL-based scheduling [13,35], we combine the multiple objectives into a single scalar reward during training. Pareto solution sets are then obtained by conducting independent training runs with systematically varied weight vectors, and the resulting non-dominated solutions are collected to approximate the Pareto front.

Objective 1: Minimize makespan. The makespan corresponds to the completion time of the last finished job:

$$min{C}_{max}=min\underset{J_i\in J}{max}{C}_{i{h}_i}.$$

(1)

Objective 2: Minimize total energy consumption. The energy consumed when operation $O_{ij}$ is processed on machine $M_k$ is $E_{ijk}=P_{ijk}·p_{ijk}/{\eta }_{lk}$, where ${\eta }_{lk}$ represents the efficiency factor introduced by the skill level of the assigned worker. The total energy consumption is

$$min{E}_{\mathrm{total}}={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^{h_i}}{\displaystyle \sum _{k=1}^m}{\displaystyle \sum _{l=1}^w}{E}_{ijk}{x}_{ijk}{z}_{ijkl}.$$

(2)

Objective 3: Minimize worker workload variance. Instead of minimizing the total machine load alone, the worker-centric formulation minimizes the variance of workloads across all workers to promote equity:

$$min\mathit{WV}={\displaystyle \frac{1}{w}}{\displaystyle \sum _{l=1}^w}{\left({\mathit{WL}}_l-\overline{\mathit{WL}}\right)}^2,$$

(3)

where ${\mathit{WL}}_l={\sum }_{i,j,k}{p}_{ijk}{x}_{ijk}{z}_{ijkl}/{\eta }_{lk}$ represents the effective workload of worker $W_l$ and $\overline{\mathit{WL}}={\displaystyle \frac{1}{w}}{\sum }_{l=1}^w{\mathit{WL}}_l$ is the mean workload.

3.3. Constraints

The feasible scheduling region is defined by the following constraints.

Precedence constraint: Each operation must start only after its predecessor has been completed:

$$S_{i,j+1,k^{\prime }}\ge C_{ijk},\forall j=1,\dots ,h_i-1.$$

(4)

Machine assignment uniqueness: Exactly one machine is assigned to each operation:

$$\sum _{M_k\in M_{ij}}{x}_{ijk}=1,\forall O_{ij}.$$

(5)

Worker assignment uniqueness: Exactly one worker attends each operation:

$${\displaystyle \sum _{l=1}^w}{z}_{ijkl}=x_{ijk},\forall O_{ij},\forall M_k.$$

(6)

Resource non-conflict (machine): No two operations may overlap on the same machine:

$$C_{ijk}\le S_{i^{\prime }{j}^{\prime }k}+H(1-y_{ij{i}^{\prime }{j}^{\prime }k}),\forall O_{ij}\ne O_{i^{\prime }{j}^{\prime }}\mathrm{on}{M}_k.$$

(7)

Resource non-conflict (worker): No worker may operate two machines at the same time:

$$C_{ijk}\le S_{i^{\prime }{j}^{\prime }{k}^{\prime }}+H(1-u_{ij{i}^{\prime }{j}^{\prime }l{l}^{\prime }}),\forall z_{ijkl}=1,z_{i^{\prime }{j}^{\prime }{k}^{\prime }l}=1,(i,j)\ne (i^{\prime },j^{\prime }).$$

(8)

Worker skill qualification: A worker can only be assigned to a machine for which the worker holds the required qualification:

$$z_{ijkl}\le q_{lk},\forall O_{ij},\forall M_k,\forall W_l,$$

(9)

where $q_{lk}\in \{0,1\}$ indicates whether worker $W_l$ is qualified to operate machine $M_k$.

Fatigue constraint: The cumulative fatigue of each worker must remain below a safety threshold ${\varphi }_{max}$:

$${\varphi }_l\left(t\right)={\displaystyle \sum _{\tau =0}^t}{\alpha }_l·\mathbb{1}\left[\mathrm{worker}l\mathrm{active}\mathrm{at}\tau \right]\le {\varphi }_{max},\forall W_l,$$

(10)

where ${\alpha }_l$ denotes the fatigue accumulation rate of worker $W_l$.

Non-negative time:

$$S_{ijk}\ge 0,C_{ijk}\ge 0,p_{ijk}\ge 0.$$

(11)

We calibrate the threshold values ${\varphi }_{max}$ and ${\mathit{WV}}_{max}$ that define the CMDP constraint boundaries as follows. The fatigue threshold ${\varphi }_{max}=0.8$ is derived from occupational health guidelines on cumulative workload limits; these guidelines recommend that sustained effort should not exceed 80% of a worker’s maximum capacity so as to prevent musculoskeletal disorders and cognitive errors [39]. The workload variance threshold ${\mathit{WV}}_{max}=50.0$ h² is calibrated through preliminary experiments on the static versions of the benchmark instances: this value corresponds to the variance level at which the most heavily loaded worker exceeds 1.5 times the average workload, a practical equity tolerance commonly adopted in shift-planning practice. A sensitivity analysis in Section 5.9 shows that the method remains robust to moderate variations in both thresholds.

3.4. Dynamic Event Modeling

Machine failures. Machine breakdowns are modeled as a Poisson process. The probability that machine $M_k$ fails within a unit time interval t is

$$P_{\mathrm{failure}}\left(t\right)=1-e^{-{\lambda }_{k}t},$$

(12)

where ${\lambda }_k$ is the failure rate, which is calibrated from historical maintenance records. The repair duration follows a uniform distribution:

$$T_{\mathrm{repair}}\sim U(t_{min},t_{max}).$$

(13)

During the repair interval $[t_f,t_f+T_{\mathrm{repair}}]$, no operations can be processed on the failed machine:

$$\sum _{i,j}{x}_{ijk,t}=0,\forall t\in [t_f,t_f+T_{\mathrm{repair}}].$$

(14)

Rush order arrivals. Rush orders are modeled as stochastic arrivals at random time points during production. When a rush order $J_{\mathrm{rush}}$ arrives at time $t_{\mathrm{rush}}$, it is inserted into the pending operation queue with elevated priority. Its first operation must satisfy the following start-time condition:

$$S_{i^{\prime },1}\ge max\left(t_{\mathrm{rush}},\underset{k}{min}\left\{C_k^{\mathrm{avail}}\right\}\right),$$

(15)

where $C_k^{\mathrm{avail}}$ denotes the earliest available time of machine $M_k$.

3.5. Disruption Severity Index

To enable adaptive selection of the rescheduling strategy, we define a disruption severity index $\mathit{DSI}$ that quantifies the combined impact of concurrent dynamic events on the current schedule:

$$\mathit{DSI}={\gamma }_1·{\displaystyle \frac{N_{\mathrm{affected}}}{N_{\mathrm{remaining}}}}+{\gamma }_2·{\displaystyle \frac{T_{\mathrm{repair}}}{C_{max}^{\mathrm{current}}}}+{\gamma }_3·{\displaystyle \frac{n_{\mathrm{rush}}·{\overline{h}}_{\mathrm{rush}}}{N_{\mathrm{remaining}}}},$$

(16)

where $N_{\mathrm{affected}}$ is the number of operations directly affected by the disruption, $N_{\mathrm{remaining}}$ is the total number of remaining unfinished operations, $C_{max}^{\mathrm{current}}$ is the expected makespan of the current schedule, $n_{\mathrm{rush}}$ and ${\overline{h}}_{\mathrm{rush}}$ are the number and average operation count of rush orders, and ${\gamma }_1,{\gamma }_2$ and ${\gamma }_3$ are weighting coefficients satisfying ${\gamma }_1+{\gamma }_2+{\gamma }_3=1$.

4. Proposed Algorithm: ALP-PPO

4.1. Overall Framework

Figure 1 illustrates the overall architecture of ALP-PPO. Four interconnected modules constitute the framework: (i) a production environment simulator that models job processing, machine states, worker states and dynamic event generation; (ii) a hierarchical policy network that outputs coordinated sub-actions for operation selection, machine allocation and worker assignment; (iii) a value network that estimates the state value function; and (iv) an adaptive Lagrangian penalty module that dynamically adjusts constraint multipliers during training. The event monitor continuously observes the production state and triggers rescheduling when a dynamic event is detected. The disruption severity index determines whether right-shift or full rescheduling is applied before the policy network generates a new schedule.

The production environment, hierarchical policy network and adaptive Lagrangian penalty module are integrated in the framework. State observations from the production environment simulator are fed to the policy network, which in turn provides hierarchical actions. The adaptive Lagrangian module monitors constraint costs, updates multipliers using the PI rule and feeds the augmented penalty back into the PPO training objective.

4.2. CMDP Formulation

We formulate the worker-constrained DFJSP as a constrained Markov decision process $(\mathcal{S},\mathcal{A},\mathcal{P},r,c_1,c_2,d_1,d_2,\gamma )$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{P}$ is the transition kernel, r is the reward function, $c_1$ and $c_2$ are constraint cost functions with thresholds $d_1$ and $d_2$, and $\gamma$ is the discount factor.

4.2.1. State Space

At decision step t, the state $s_t\in \mathcal{S}$ encodes the complete production situation through four feature groups:

Job features: For each pending operation $O_{ij}$, the state includes the operation index, remaining processing time across candidate machines, precedence depth and job completion ratio.

Machine features: For each machine $M_k$, the state includes the current availability status (idle, busy or under repair), queue length, cumulative utilization rate and estimated time until the machine becomes available.

Worker features: For each worker $W_l$, the state includes the current assignment status, cumulative workload ${\mathit{WL}}_l$, current fatigue level ${\varphi }_l\left(t\right)$, skill vector $q_l=[q_{l1},q_{l2},\dots ,q_{lm}]$ and deviation of the workload from the mean.

Global features: Current simulation time, number of remaining operations, number of active dynamic events and the disruption severity index $\mathit{DSI}$.

All features are normalized to $[0,1]$ and then concatenated into a fixed-length state vector.

4.2.2. Hierarchical Action Space

At each decision step the agent selects a composite action $a_t=(a_t^{\mathrm{op}},a_t^{\mathrm{mc}},a_t^{\mathrm{wk}})$:

Operation selection $a_t^{\mathrm{op}}$: The next operation is selected from the set of eligible operations (those whose predecessors are completed) on the basis of priority scores.

Machine allocation $a_t^{\mathrm{mc}}$: The selected operation is assigned to one of its candidate machines on the basis of estimated processing efficiency and current load.

Worker assignment $a_t^{\mathrm{wk}}$: A qualified worker is assigned to the selected machine on the basis of skill match, current fatigue and workload balance.

The action space is structured hierarchically: the operation selection sub-action constrains the set of valid machine sub-actions, and these in turn constrain the set of valid worker sub-actions. This cascading structure reduces the effective action space from $O\left(\right|\mathcal{O}|\times m\times w)$ to a sequence of smaller decision problems.

4.2.3. Reward Function

Scheduling performance and constraint costs are combined in the reward function. At each step t the agent receives

$$r_t=-{\beta }_1·\Delta C_{max}\left(t\right)-{\beta }_2·\Delta E\left(t\right)-{\beta }_3·\Delta \mathit{TML}\left(t\right)+{\beta }_4·\mathbb{1}\left[\mathrm{job}\mathrm{completed}\mathrm{at}t\right],$$

(17)

where $\Delta C_{max}\left(t\right)$ is the incremental change in estimated makespan, $\Delta E\left(t\right)$ is the incremental energy consumption, $\Delta \mathit{TML}\left(t\right)$ is the incremental machine load, and ${\beta }_1,{\beta }_2,{\beta }_3$ and ${\beta }_4$ are weighting coefficients. In the default configuration the weights $({\beta }_1,{\beta }_2,{\beta }_3,{\beta }_4)$ are set to $(0.4,0.2,0.2,0.2)$, reflecting a practical priority ordering in which makespan reduction receives the highest emphasis, consistent with common shop floor preferences. Section reward weight sensitivity analyzes how robust the method is to variations in these coefficients.

Two constraint cost functions are defined:

$$c_1(s_t,a_t)=max\left(0,\mathit{WV}\left(t\right)-{\mathit{WV}}_{max}\right),$$

(18)

$$c_2(s_t,a_t)=max\left(0,\underset{l}{max}{\varphi }_l\left(t\right)-{\varphi }_{max}\right),$$

(19)

where ${\mathit{WV}}_{max}$ is the workload variance threshold and ${\varphi }_{max}$ is the fatigue safety limit.

4.3. Policy and Value Networks

The policy network ${\pi }_{\theta }$ takes state $s_t$ as input and provides probability distributions over the three sub-action spaces through a shared feature extraction backbone followed by three specialized heads. Two fully connected layers with ReLU activation and layer normalization form the backbone. Each sub-action head contains one fully connected layer followed by softmax activation and provides a categorical distribution over valid actions. The value network $V_{\psi }$ shares the same backbone architecture but outputs a single scalar estimate of the state value, and a separate cost value network $V_{\psi }^c$ estimates the expected cumulative constraint cost, as illustrated in Figure 2.

The network consists of a shared backbone, three cascading sub-action heads (operation selection, machine allocation and worker assignment) and dual value heads for reward and constraint cost estimation. Because of the cascading structure, each sub-action head receives masked logits conditioned on the preceding sub-action, and feasibility is thereby enforced throughout the forward pass.

4.4. Adaptive Lagrangian Penalty Mechanism

The constrained optimization objective is

$$\underset{\theta }{max}{\mathbb{E}}_{{\pi }_{\theta }}\left[{\displaystyle \sum _{t=0}^T}{\gamma }^t{r}_t\right],\mathrm{s}.\mathrm{t}.{\mathbb{E}}_{{\pi }_{\theta }}\left[{\displaystyle \sum _{t=0}^T}{\gamma }^t{c}_i(s_t,a_t)\right]\le d_i,i=1,2.$$

(20)

By applying Lagrangian relaxation, this is converted into an unconstrained min-max problem:  

$$\underset{{\mu }_1,{\mu }_2\ge 0}{min}\underset{\theta }{max}\mathcal{L}(\theta ,{\mu }_1,{\mu }_2)={\mathbb{E}}_{{\pi }_{\theta }}\left[{\displaystyle \sum _{t=0}^T}{\gamma }^t{r}_t\right]-{\displaystyle \sum _{i=1}^2}{\mu }_i\left({\mathbb{E}}_{{\pi }_{\theta }}\left[{\displaystyle \sum _{t=0}^T}{\gamma }^t{c}_i\right]-d_i\right),$$

(21)

where ${\mu }_1$ and ${\mu }_2$ are non-negative Lagrangian multipliers corresponding to the workload balance and fatigue constraints, respectively.

The key innovation lies in the adaptive update rule for the multipliers. At each training epoch e, the multipliers are updated through gradient ascent on the constraint violation:

$${\mu }_i^{(e+1)}=max\left(0,{\mu }_i^{\left(e\right)}+{\xi }_i·\left({\widehat{J}}_{c_i}^{\left(e\right)}-d_i\right)+{\kappa }_i·{\displaystyle \sum _{e^{\prime }=0}^e}\left({\widehat{J}}_{c_i}^{\left(e^{\prime }\right)}-d_i\right)\right),$$

(22)

where ${\widehat{J}}_{c_i}^{\left(e\right)}$ is the estimated cumulative constraint cost at epoch e, ${\xi }_i$ is the proportional gain and ${\kappa }_i$ is the integral gain. By accumulating historical constraint violations, the integral term prevents persistent undershoot of the multiplier, and this has been found to improve the stability of convergence in constrained RL [29].

This proportional–integral (PI) update rule differs from standard PPO-Lagrangian approaches [28] in two respects. Standard methods use a proportional-only rule ${\mu }_i^{(e+1)}=max(0,{\mu }_i^{\left(e\right)}+{\xi }_i({\widehat{J}}_{c_i}^{\left(e\right)}-d_i))$, in which only the current constraint violation drives the update; this can lead to sustained undershoot when the constraint cost fluctuates around the threshold. The integral term in Equation (22) accumulates the history of constraint violations and provides a steady-state correction that drives the long-run average violation toward zero. This property is particularly important in the DFJSP setting, where the workload balance constraint and the fatigue constraint have different magnitudes and update frequencies: workload variance changes with every operation assignment, whereas fatigue accumulates gradually over the production horizon. The PI formulation allows each multiplier to adapt at its own effective rate without manual rescaling of the constraint costs. The schematic of this mechanism is illustrated in Figure 3.

Constraint costs are estimated from the current batch of trajectories, the proportional–integral multiplier update is applied, and the resulting augmented penalty is integrated into the PPO loss function. The proportional term reacts to instantaneous violations, while the integral term prevents persistent undershoot by accumulating historical error.

4.5. PPO Training with Lagrangian Augmented Objective

The standard PPO clipped surrogate objective is  

$$L^{\mathrm{CLIP}}\left(\theta \right)={\mathbb{E}}_t\left[\min \left({\rho }_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}({\rho }_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t\right)\right],$$

(23)

where ${\rho }_t\left(\theta \right)={\pi }_{\theta }\left(a_t\right|s_t)/{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)$ is the importance sampling ratio, ${\widehat{A}}_t$ is the generalized advantage estimate and $ϵ$ is the clipping parameter.

The Lagrangian-augmented PPO objective becomes

$$L^{\mathrm{ALP}}\left(\theta \right)=L^{\mathrm{CLIP}}\left(\theta \right)-{\displaystyle \sum _{i=1}^2}{\mu }_i·{\widehat{J}}_{c_i}\left(\theta \right),$$

(24)

where ${\widehat{J}}_{c_i}\left(\theta \right)$ is estimated from the current batch of trajectories.

The complete training loss comprises the policy loss, the value function loss and an entropy bonus,

$$L(\theta ,\psi )=-L^{\mathrm{ALP}}\left(\theta \right)+{\alpha }_v·L^V\left(\psi \right)-{\alpha }_e·H\left[{\pi }_{\theta }\right],$$

(25)

where $L^V\left(\psi \right)={\mathbb{E}}_t\left[{(V_{\psi }\left(s_t\right)-R_t)}^2\right]$ is the mean squared error of the value function, $H\left[{\pi }_{\theta }\right]$ is the policy entropy, and ${\alpha }_v$ and ${\alpha }_e$ are balancing coefficients.

4.6. Event-Triggered Rescheduling Strategy

When the event monitor detects a machine failure or rush order arrival, the following procedure is executed:

Step 1: The current production state is recorded, including all machine states, worker states, in-progress operations and pending operations.

Step 2: The disruption severity index $\mathit{DSI}$ is computed according to Equation (16).

Step 3: If $\mathit{DSI}<\delta$, right-shift rescheduling is applied: all operations affected by the disruption are delayed by the estimated recovery time, and the remaining schedule topology is preserved.

Step 4: If $\mathit{DSI}\ge \delta$, full rescheduling is triggered: the trained ALP-PPO policy network receives the updated state and generates a complete new schedule for all remaining operations.

Step 5: After rescheduling, the Lagrangian multipliers are preserved from the latest training checkpoint so as to maintain consistent constraint enforcement across rescheduling episodes.

Algorithm 1 summarizes the training procedure of the complete ALP-PPO algorithm.    

Algorithm 1: ALP-PPO Training

Input: DFJSP instance parameters, worker skill matrix, dynamic event parameters   Output: Trained policy network ${\pi }_{\theta }$ 1: Initialize policy network ${\pi }_{\theta }$, value network $V_{\psi }$, cost value network $V_{\psi }^c$ 2: Initialize Lagrangian multipliers ${\mu }_1={\mu }_2=0$, integral buffers $I_1=I_2=0$ 3: for epoch $e=1$ to $E_{max}$ do 4:    Collect batch of trajectories $\mathcal{B}$ by executing ${\pi }_{\theta }$ in the environment 5:    Inject dynamic events (machine failures, rush orders) stochastically 6:    Compute DSI; select rescheduling strategy (right-shift or full) 7:    Compute rewards $r_t$, constraint costs $c_{1,t}$, $c_{2,t}$ for each step 8:    Estimate advantages ${\widehat{A}}_t$ using GAE($\lambda$) 9:    Compute cumulative constraint cost estimates ${\widehat{J}}_{c_1}$, ${\widehat{J}}_{c_2}$ 10:    Update integral buffers: $I_i\leftarrow I_i+({\widehat{J}}_{c_i}-d_i)$ 11:    Update multipliers: ${\mu }_i\leftarrow max(0,{\mu }_i+{\xi }_i({\widehat{J}}_{c_i}-d_i)+{\kappa }_i{I}_i)$ 12:    Compute augmented loss $L(\theta ,\psi )$ per Equation (25) 13:    Update $\theta ,\psi$ via Adam optimizer 14: end for 15: return ${\pi }_{\theta }$

5. Experiments and Results

5.1. Experimental Setup

All experiments are run on a workstation equipped with an NVIDIA RTX 4090 GPU, an Intel Core i9-13900K CPU and 64 GB RAM. We implement the algorithm in Python 3.10, using PyTorch 2.1 for neural network training and a custom discrete-event simulation engine for the production environment. Time units are measured in hours, energy consumption in kilowatt-hours (kWh) and workload in hours.

5.2. Benchmark Instances

The base FJSP instances are taken from the Brandimarte benchmark dataset [5]. Five representative instances of varying complexity are selected: MK01 ($10\times 6$, 55 operations), MK02 ($10\times 6$, 58 operations), MK03 ($15\times 8$, 150 operations), MK06 ($10\times 15$, 150 operations) and MK10 ($20\times 15$, 240 operations). Each instance is augmented with the following additional parameters to create the extended benchmark.

Worker parameters: Each instance is equipped with $w=\lceil m/2\rceil$ workers. Worker–machine qualification matrices are generated randomly with a qualification probability of 0.6 per machine. Skill coefficients ${\eta }_{lk}$ for qualified worker–machine pairs are drawn from $U(0.7,1.0)$. The fatigue accumulation rates ${\alpha }_l$ are drawn from $U(0.01,0.03)$ per time unit, and the fatigue threshold is set to ${\varphi }_{max}=0.8$.

Dynamic event parameters: Machine failure rates are set to ${\lambda }_k\in [0.005,0.02]$ per time unit. Repair durations follow $U(5,15)$ time units. Rush orders arrive following a Poisson process whose mean inter-arrival time equals $0.3\times C_{max}^{\mathrm{static}}$, where $C_{max}^{\mathrm{static}}$ is the makespan of the static schedule. Each rush order contains 3 to 6 operations.

Energy parameters: Power consumption rates are drawn from $U(3.0,8.0)$ kW for each operation–machine pair.

The ranges chosen for the synthetic parameters are grounded in representative manufacturing values. The skill coefficient range $[0.7,1.0]$ reflects typical skill-level variation observed in multi-skill workforce studies [25]: a fully trained operator achieves unit efficiency, while a cross-trained operator works at 70–90% of that rate. The fatigue accumulation rates ${\alpha }_l\in [0.01,0.03]$ per hour correspond to the range reported in ergonomics studies of repetitive manual tasks [39], and the threshold ${\varphi }_{max}=0.8$ is consistent with the occupational health recommendation that sustained effort should remain below 80% of maximum capacity. Machine failure rates ${\lambda }_k\in [0.005,0.02]$ per hour and repair durations $U(5,15)$ h are consistent with values reported in the preventive-maintenance scheduling literature [27]. Section 5.9 examines the sensitivity of the results to variations in these parameters.

Table 2. Extended benchmark instance configurations.

Instance	Jobs	Machines	Workers	Operations	Failure Rate Range	Rush Order Ops
MK01	10	6	3	55	[0.005, 0.015]	3–5
MK02	10	6	3	58	[0.005, 0.015]	3–5
MK03	15	8	4	150	[0.008, 0.020]	4–6
MK06	10	15	8	150	[0.010, 0.020]	3–6
MK10	20	15	8	240	[0.010, 0.020]	4–6

summarizes the configurations of the extended benchmark.

5.3. Compared Algorithms and Hyperparameters

ALP-PPO is compared against six baseline algorithms:

Double DQN (DoDQN): A value-based DRL algorithm that separates action selection and value estimation to mitigate overestimation bias [14]. Dueling DQN (DuDQN) decomposes the Q-function into state value and advantage parts for more stable learning [15]. 3DQN combines the Double DQN and Dueling DQN architectures, as used in the multi-agent DFJSP framework of Wang et al. [16]. Standard PPO (PPO-base) uses fixed penalty coefficients for constraint handling, without the adaptive Lagrangian mechanism. NSGA-II [30] is a widely adopted multi-objective evolutionary algorithm that maintains a population of candidate schedules and evolves them through crossover, mutation and non-dominated sorting with crowding distance. MOEA/D [31] decomposes the multi-objective problem into a set of scalar subproblems using weight vectors and optimizes them simultaneously through neighborhood-based collaboration. Both evolutionary algorithms use a population size of 200, run for 500 generations, and employ the same operation-based encoding with precedence-preserving crossover and machine-swap mutation operators.

To generate Pareto solution sets for the DRL-based methods, we conduct 15 independent training runs with weight vectors sampled uniformly from the simplex $\{({\beta }_1,{\beta }_2,{\beta }_3)\mid {\beta }_1+{\beta }_2+{\beta }_3=0.8,{\beta }_i\ge 0.1\}$, with ${\beta }_4=0.2$ held constant. The solutions from all runs are then pooled, and only the non-dominated set is retained.

Table 3. Hyperparameters of ALP-PPO.

Parameter	Value
Learning rate (policy)	0.0003
Learning rate (value)	0.001
Discount factor $\gamma$	0.99
GAE parameter $\lambda$	0.95
PPO clipping parameter $ϵ$	0.2
Batch size	128
Rollout length	2048
Training epochs per update	10
Total training episodes	5000
Lagrangian proportional gain ${\xi }_i$	0.01
Lagrangian integral gain ${\kappa }_i$	0.001
Workload variance threshold ${\mathit{WV}}_{max}$	50.0
Fatigue threshold ${\varphi }_{max}$	0.8
DSI threshold $\delta$	0.35
Entropy coefficient ${\alpha }_e$	0.01
Value loss coefficient ${\alpha }_v$	0.5
Reward weights $({\beta }_1,{\beta }_2,{\beta }_3,{\beta }_4)$	(0.4, 0.2, 0.2, 0.2)

lists the hyperparameters for ALP-PPO. All DRL-based algorithms use the same learning rate, batch size, discount factor and total training episodes to ensure a fair comparison. Each algorithm is trained with five independent random seeds, and we report the results as mean ± standard deviation.

5.4. Training Convergence Analysis

The training reward convergence curves of all DRL algorithms across four representative benchmark instances are shown in Figure 4. All DRL methods converge within 5000 episodes. During the initial 1000 episodes, ALP-PPO converges more slowly than PPO-base because the adaptive Lagrangian multipliers impose increasing constraint penalties that temporarily reduce the reward signal; after convergence, however, ALP-PPO reaches a higher final reward, which indicates that the constraint enforcement ultimately guides the policy toward higher-quality feasible solutions.

Shaded bands indicate the standard deviation over five random seeds. NSGA-II and MOEA/D are not shown because they follow a generation-based evolutionary process rather than an episode-based reward trajectory.

5.5. Multi-Objective Scheduling Performance

Table 4. Multi-objective scheduling results under concurrent dynamic disturbances (mean ± std).

Instance	Algorithm	${\mathit{C}}_{max}$ (h)	${\mathit{E}}_{\mathbf{total}}$ (kWh)	$\mathit{TML}$ (h)	$\mathit{WV}$ (h2)
MK01	DoDQN	52.3 ± 3.1	487.2 ± 18.5	198.4 ± 8.7	85.3 ± 12.6
MK01	DuDQN	50.8 ± 2.7	479.5 ± 15.3	193.7 ± 7.2	78.9 ± 10.4
MK01	3DQN	48.6 ± 2.4	468.3 ± 14.1	186.2 ± 6.8	72.4 ± 9.8
MK01	PPO-base	47.2 ± 2.1	461.7 ± 12.8	182.5 ± 6.1	65.7 ± 11.3
MK01	NSGA-II	49.5 ± 2.6	472.8 ± 13.7	190.1 ± 7.4	69.8 ± 9.2
MK01	MOEA/D	48.2 ± 2.3	465.1 ± 12.5	185.6 ± 6.5	67.5 ± 8.8
MK01	ALP-PPO	45.8 ± 1.8	452.4 ± 11.2	178.3 ± 5.4	41.2 ± 6.3
MK03	DoDQN	278.5 ± 12.4	1842.6 ± 65.3	856.3 ± 28.7	312.8 ± 45.2
MK03	DuDQN	271.3 ± 10.8	1798.4 ± 58.7	832.1 ± 25.4	289.6 ± 38.7
MK03	3DQN	265.7 ± 9.6	1756.2 ± 52.1	815.4 ± 22.8	267.3 ± 35.1
MK03	PPO-base	258.4 ± 8.3	1712.5 ± 47.6	798.6 ± 20.3	243.5 ± 42.8
MK03	NSGA-II	269.8 ± 10.2	1782.3 ± 55.8	838.7 ± 24.6	274.1 ± 36.5
MK03	MOEA/D	262.5 ± 9.1	1738.6 ± 49.2	808.3 ± 21.7	258.4 ± 33.8
MK03	ALP-PPO	251.2 ± 7.1	1678.3 ± 42.4	782.1 ± 18.6	156.4 ± 22.7
MK06	DoDQN	148.7 ± 8.5	2156.8 ± 78.4	1245.3 ± 42.6	425.6 ± 52.3
MK06	DuDQN	143.2 ± 7.3	2098.4 ± 71.2	1213.7 ± 38.4	398.2 ± 47.8
MK06	3DQN	138.5 ± 6.8	2045.6 ± 64.5	1187.2 ± 35.1	372.4 ± 43.5
MK06	PPO-base	135.8 ± 6.1	2012.3 ± 58.7	1165.8 ± 32.7	345.7 ± 51.2
MK06	NSGA-II	141.3 ± 7.1	2068.5 ± 62.3	1198.4 ± 36.2	384.6 ± 44.7
MK06	MOEA/D	136.7 ± 6.4	2028.1 ± 57.6	1174.2 ± 33.5	362.8 ± 41.3
MK06	ALP-PPO	131.4 ± 5.3	1968.5 ± 51.3	1142.6 ± 28.4	218.3 ± 28.6
MK10	DoDQN	325.6 ± 15.8	3587.4 ± 125.6	2156.8 ± 68.4	578.3 ± 72.6
MK10	DuDQN	318.4 ± 13.5	3498.6 ± 112.3	2098.5 ± 62.1	542.7 ± 65.8
MK10	3DQN	312.1 ± 12.2	3425.8 ± 98.7	2056.3 ± 56.8	512.4 ± 58.3
MK10	PPO-base	305.7 ± 10.8	3368.2 ± 87.4	2018.7 ± 52.3	478.6 ± 68.5
MK10	NSGA-II	319.7 ± 13.1	3478.3 ± 105.4	2082.6 ± 58.7	528.5 ± 62.4
MK10	MOEA/D	310.4 ± 11.6	3412.7 ± 94.8	2042.5 ± 54.1	504.8 ± 56.7
MK10	ALP-PPO	298.3 ± 9.2	3285.4 ± 78.6	1978.4 ± 45.7	312.5 ± 38.4

reports the multi-objective scheduling results under concurrent dynamic disturbances (one machine failure and one rush order insertion per episode). The improvements of ALP-PPO in makespan and energy consumption over 3DQN are moderate (in the range of 3% to 8%), while the reduction in workload variance is substantial (35% to 50%). This pattern indicates that the adaptive Lagrangian penalty steers the policy toward more balanced solutions without markedly sacrificing primary scheduling objectives. When compared with the evolutionary algorithms, ALP-PPO reaches competitive makespan and energy values while substantially reducing workload variance, because the Lagrangian mechanism directly penalizes workload imbalance during policy optimization. NSGA-II and MOEA/D deliver comparable makespan and energy results owing to their effective Pareto-based search, but their workload variance values are higher, reflecting the absence of an explicit constraint enforcement mechanism for worker equity.

5.6. Pareto Front Comparison

The Pareto front solutions obtained by all seven algorithms on three benchmark instances are visualized in Figure 5 in the three-dimensional objective space of makespan, energy consumption and workload variance. ALP-PPO provides solution sets that are more concentrated and closer to the ideal point (origin) than those of the baseline algorithms. On MK01, the solution sets of 3DQN and ALP-PPO overlap partially in the makespan-energy plane, but ALP-PPO solutions are consistently lower along the workload variance axis. On MK10, the separation between ALP-PPO and all baselines becomes more evident, and this confirms the scalability advantage of the suggested method on larger problem instances. NSGA-II and MOEA/D achieve well-spread Pareto fronts in the makespan-energy plane, but their solutions are shifted upward along the workload variance axis relative to ALP-PPO, because neither evolutionary algorithm incorporates an explicit workload equity constraint.

Each point represents a scheduling solution; proximity to the origin indicates better multi-objective performance. Solutions from NSGA-II and MOEA/D are included alongside the DRL methods.

Table 5. Pareto front quality indicators (mean ± std over five seeds). ↑ indicates higher is better; ↓ indicates lower is better.

Instance	Algorithm	HV (↑)	IGD (↓)
MK01	DoDQN	0.528 ± 0.032	0.218 ± 0.024
MK01	DuDQN	0.563 ± 0.028	0.195 ± 0.021
MK01	3DQN	0.612 ± 0.025	0.168 ± 0.018
MK01	PPO-base	0.658 ± 0.022	0.142 ± 0.016
MK01	NSGA-II	0.635 ± 0.024	0.155 ± 0.017
MK01	MOEA/D	0.651 ± 0.021	0.146 ± 0.015
MK01	ALP-PPO	0.738 ± 0.018	0.092 ± 0.011
MK03	DoDQN	0.486 ± 0.038	0.252 ± 0.028
MK03	DuDQN	0.521 ± 0.034	0.228 ± 0.025
MK03	3DQN	0.574 ± 0.029	0.196 ± 0.022
MK03	PPO-base	0.618 ± 0.026	0.168 ± 0.019
MK03	NSGA-II	0.596 ± 0.028	0.182 ± 0.020
MK03	MOEA/D	0.625 ± 0.024	0.162 ± 0.018
MK03	ALP-PPO	0.712 ± 0.020	0.108 ± 0.013
MK06	DoDQN	0.462 ± 0.042	0.275 ± 0.032
MK06	DuDQN	0.498 ± 0.037	0.248 ± 0.028
MK06	3DQN	0.548 ± 0.032	0.215 ± 0.024
MK06	PPO-base	0.592 ± 0.028	0.185 ± 0.021
MK06	NSGA-II	0.572 ± 0.030	0.198 ± 0.022
MK06	MOEA/D	0.601 ± 0.026	0.178 ± 0.019
MK06	ALP-PPO	0.695 ± 0.022	0.118 ± 0.014
MK10	DoDQN	0.435 ± 0.045	0.298 ± 0.035
MK10	DuDQN	0.472 ± 0.040	0.268 ± 0.031
MK10	3DQN	0.525 ± 0.035	0.232 ± 0.026
MK10	PPO-base	0.568 ± 0.030	0.198 ± 0.023
MK10	NSGA-II	0.548 ± 0.033	0.212 ± 0.025
MK10	MOEA/D	0.578 ± 0.028	0.192 ± 0.021
MK10	ALP-PPO	0.672 ± 0.024	0.132 ± 0.016

reports the Hypervolume (HV) and Inverted Generational Distance (IGD) indicators [34] for all algorithms across four benchmark instances, providing a quantitative assessment of Pareto front quality. The HV indicator measures the volume of the objective space dominated by the obtained Pareto front; a larger value indicates better coverage and convergence. The IGD indicator measures the average distance from a reference Pareto front to the obtained solutions; a smaller value indicates both better convergence and diversity. For each instance, the reference Pareto front is constructed by pooling all non-dominated solutions from all algorithms and all random seeds. Before computing the indicators, all objective values are normalized to $[0,1]$ using the ranges observed across all solutions.

The highest HV and lowest IGD on all four instances are achieved by ALP-PPO, which indicates that its Pareto front dominates those of the baseline methods in both convergence and diversity. The advantage over MOEA/D, the strongest evolutionary baseline, is most pronounced on the larger instances MK06 and MK10, where the adaptive Lagrangian mechanism effectively constrains workload variance without sacrificing Pareto spread. On MK01 and MK03, NSGA-II and MOEA/D outperform the value-based DRL methods (DoDQN, DuDQN, 3DQN) in HV, which demonstrates their strength in Pareto search on smaller instances; however, this advantage diminishes on larger instances, where the computational cost of population-based search becomes a limiting factor.

5.7. Gantt Chart Comparison Under Dynamic Disturbances

Figure 6 compares the rescheduling results of right-shift and full rescheduling strategies for both machine failure and rush order insertion scenarios on the MK01 instance. In the machine failure scenario (top row), right-shift rescheduling preserves the original schedule topology and delays affected operations, giving a makespan of 62 h, while full rescheduling by ALP-PPO globally reassigns operations and compresses the makespan to 46 h. In the rush order insertion scenario (bottom row), right-shift rescheduling extends the makespan to 65 h, while full rescheduling achieves 48 h through strategic redistribution of tasks across machines and workers.

The top row shows the machine failure scenario and the bottom row shows the rush order insertion scenario. In each row, right-shift rescheduling is depicted in the left panel and full rescheduling by ALP-PPO in the right panel. Shorter makespans are achieved by the full rescheduling strategy through global reassignment of operations across machines and workers.

5.8. Comparative and Sensitivity Analysis

Worker workload distributions across all evaluation episodes for three algorithms on three benchmark instances are compared in Figure 7. ALP-PPO gives narrower and more symmetric violin plots than the baselines, indicating that the adaptive Lagrangian mechanism effectively constrains workload variance. 3DQN shows the widest spread, especially on the larger MK10 instance, reflecting its lack of explicit workload balancing.

Narrower distributions indicate more balanced workloads across workers. On all three instances ALP-PPO achieves the narrowest spread, which demonstrates that the Lagrangian constraint penalty directly reduces workload imbalance.

Figure 8 presents radar charts comparing seven algorithms across six normalized metrics on three benchmark instances. The six axes denote makespan, energy consumption, total machine load and worker workload variance (all inverted), along with constraint satisfaction rate and rescheduling response speed. ALP-PPO achieves the largest polygon area on all three instances, and its advantage is most pronounced on the workload variance and constraint satisfaction axes. NSGA-II and MOEA/D reach competitive areas on the makespan and energy axes but show smaller coverage on the constraint satisfaction and rescheduling speed axes, because these evolutionary methods lack online inference capability and require a full optimization run for each rescheduling event.

A larger polygon area indicates better overall performance. All six axes are normalized so that outward is better for every metric.

To isolate the contribution of each component, we conduct ablation experiments in which one component is removed at a time. Four variants are compared: (i) the full ALP-PPO model, (ii) ALP-PPO without the adaptive Lagrangian mechanism (a fixed penalty is used instead), (iii) ALP-PPO without the hierarchical action space (flat joint actions are used instead) and (iv) ALP-PPO without event-triggered rescheduling strategy selection (full rescheduling is applied for all events). The results appear in

Table 6. Ablation study results on MK06 under concurrent dynamic disturbances (mean ± std).

Variant	${\mathit{C}}_{max}$ (h)	${\mathit{E}}_{\mathbf{total}}$ (kWh)	$\mathit{WV}$ (h2)	Violation Rate (%)
ALP-PPO (full)	131.4 ± 5.3	1968.5 ± 51.3	218.3 ± 28.6	4.2 ± 1.8
w/o Adaptive Lagrangian	134.8 ± 6.7	1998.2 ± 58.4	342.6 ± 45.3	18.6 ± 5.2
w/o Hierarchical action	142.3 ± 8.4	2078.6 ± 72.1	287.5 ± 38.7	8.5 ± 3.1
w/o Event-triggered strategy	133.2 ± 5.8	1985.4 ± 54.7	225.8 ± 30.2	5.1 ± 2.3

. Removing the adaptive Lagrangian mechanism causes the largest degradation: workload variance increases by 57% and the constraint violation rate rises from 4.2% to 18.6%. This demonstrates that the adaptive multiplier update is the most critical component for constraint enforcement. Removing the hierarchical action space degrades all metrics and increases makespan by 8.3%. This degradation arises because the flat joint action space is exponentially larger, causing the policy to explore inefficiently and converge to suboptimal machine–worker pairings. Removing event-triggered strategy selection has the smallest impact on average metrics but increases average rescheduling computation time by 35%. These results demonstrate that each of the three suggested components makes a distinct and measurable contribution: the adaptive Lagrangian mechanism is primarily responsible for constraint enforcement, the hierarchical action space primarily improves scheduling quality through more efficient exploration, and the event-triggered rescheduling primarily improves computational efficiency by avoiding unnecessary full rescheduling for minor disruptions.

5.9. Sensitivity Analysis

Figure 9 presents heatmaps of the sensitivity of ALP-PPO to four key parameter pairs on the MK06 instance. The Lagrangian gain parameters (top left) show a clear optimal region near $\xi =0.01$, $\kappa =0.001$; larger integral gains lead to oscillatory multiplier behavior. The DSI threshold (top right) shows moderate sensitivity: the selected value $\delta =0.35$ balances responsiveness against computational efficiency. The fatigue threshold (bottom left) interacts with the number of workers: when there are fewer workers, a more generous fatigue limit is needed to maintain feasibility.

The four panels show the effects of Lagrangian gain parameters (top left), DSI threshold versus worker count (top right), fatigue threshold versus worker count (bottom left) and PPO clipping parameter versus learning rate (bottom right) on the composite scheduling performance. These results demonstrate that the selected hyperparameters lie within a broad stable region, and moderate perturbations do not cause abrupt performance degradation.

Reward Weight Sensitivity

To examine how sensitive ALP-PPO is to the reward weight configuration, we test five representative weight vectors on the MK06 instance, as reported in

Table 7. Reward weight sensitivity on MK06 (mean ± std over five seeds).

Weight $({\mathit{\beta }}_1,{\mathit{\beta }}_2,{\mathit{\beta }}_3,{\mathit{\beta }}_4)$	${\mathit{C}}_{max}$ (h)	${\mathit{E}}_{\mathbf{total}}$ (kWh)	$\mathit{WV}$ (h2)	Violation (%)
(0.4, 0.2, 0.2, 0.2) default	131.4 ± 5.3	1968.5 ± 51.3	218.3 ± 28.6	4.2 ± 1.8
(0.2, 0.4, 0.2, 0.2) energy	136.8 ± 6.1	1912.4 ± 48.7	232.5 ± 31.4	4.8 ± 2.1
(0.2, 0.2, 0.4, 0.2) workload	138.2 ± 6.5	1995.7 ± 54.2	195.6 ± 25.8	3.5 ± 1.6
(0.6, 0.1, 0.1, 0.2) makespan	128.5 ± 4.8	2035.8 ± 58.6	268.4 ± 35.2	6.8 ± 2.7
(0.1, 0.1, 0.6, 0.2) equity	145.3 ± 7.2	2048.6 ± 62.4	178.2 ± 22.3	2.8 ± 1.4

. The default configuration $({\beta }_1,{\beta }_2,{\beta }_3,{\beta }_4)=(0.4,0.2,0.2,0.2)$ emphasizes makespan. Two alternative configurations shift emphasis toward energy consumption and workload variance, respectively, and two extreme configurations concentrate most of the weight on a single objective.

The three balanced configurations (rows 1–3) give qualitatively similar Pareto trade-offs, with the expected shift in emphasis among the three objectives. Across all configurations the constraint violation rate stays below 7%, which indicates that the adaptive Lagrangian mechanism maintains effective constraint enforcement regardless of the reward weighting. When most of the weight is concentrated on a single objective (rows 4–5), the neglected objectives degrade, but the degradation is moderate and the constraint violation rate remains controlled. These results support the robustness of ALP-PPO to reward weight selection and suggest that practitioners can adjust the weights according to shop floor priorities without retuning the Lagrangian parameters.

5.10. Constraint Evolution During Training

The evolution of constraint violation rates and Lagrangian multiplier values during training on MK06 is shown in Figure 10. The workload balance violation rate drops from roughly 45% early in training to below 5% after 3000 episodes. The Lagrangian multipliers rise rapidly during the initial phase and then stabilize, which demonstrates that the proportional–integral update rule converges stably without the need for manual tuning.

Panel (a) shows the convergence of the constraint violation rate and panel (b) shows the adaptation of the Lagrangian multipliers ${\mu }_1$ (workload) and ${\mu }_2$ (fatigue) over training episodes. The fatigue multiplier ${\mu }_2$ stabilizes at a lower value than ${\mu }_1$, reflecting the fact that the fatigue constraint is less frequently active under the selected threshold.

5.11. Robustness Under Varying Disruption Intensity

Robustness under increasing disruption intensity is evaluated in Figure 11. Three levels are tested: low (one failure per episode, no rush orders), medium (two failures, one rush order) and high (three failures, two rush orders). ALP-PPO exhibits the most graceful degradation: makespan increases by 12.3% under high disruption relative to the low-disruption baseline, compared with 18.7% for 3DQN and 22.4% for DoDQN. NSGA-II and MOEA/D show degradation of 20.1% and 17.8%, respectively, because each new disruption requires a computationally expensive re-optimization that may not converge within the available time budget.

The three panels show makespan, workload variance and constraint violation rate, respectively, across low, medium and high disruption levels for all seven algorithms.

5.12. Computational Efficiency Analysis

For real-time rescheduling, the scheduling algorithm must deliver a new schedule within a time frame compatible with production cycle times.

Table 8. Computational efficiency comparison on MK06.

Algorithm	Training Time (h)	Inference per Step (ms)	Rescheduling Time (s)
DoDQN	3.2	9.5	0.31
DuDQN	3.4	10.1	0.33
3DQN	3.8	11.2	0.35
PPO-base	4.1	14.8	0.38
NSGA-II	–	–	12.8
MOEA/D	–	–	15.3
ALP-PPO	5.2	15.3	0.42

reports the computational cost of each algorithm along three dimensions: total training time for 5000 episodes on the MK06 instance, average online inference time per decision step during evaluation and average wall-clock time to generate a complete rescheduling solution after a dynamic event.

ALP-PPO requires roughly 27% more training time than PPO-base, owing to the additional overhead of the Lagrangian multiplier update and the cost value network. The per-step inference time of 15.3 ms is marginally higher than that of PPO-base (14.8 ms) because of the three-head hierarchical output, but it remains well within the millisecond range required for real-time decision-making. The rescheduling time of 0.42 s includes state encoding, policy inference for all remaining operations and schedule construction, and it is two orders of magnitude faster than the evolutionary methods (12.8 s for NSGA-II, 15.3 s for MOEA/D), each of which must run a full population-based optimization from scratch for every rescheduling event. Training time and per-step inference are not reported for NSGA-II and MOEA/D because they do not use a learned policy. These results demonstrate that ALP-PPO is suitable for real-time deployment in production environments where rescheduling decisions must be made within seconds of a disruption event.

5.13. Generalization Analysis

A cross-instance transfer experiment is conducted to evaluate how well the learned policy generalizes. Policies trained on two source instances (MK01 and MK03) are evaluated without fine-tuning on three target instances: MK02 (same scale as MK01), MK06 (more machines) and MK10 (more jobs and machines). To accommodate different instance sizes, the state vector is padded or truncated, with zero-padding for missing machine or worker features.

Table 9. Generalization performance of ALP-PPO across instances (mean ± std).

Target	Training	${\mathit{C}}_{max}$ (h)	${\mathit{E}}_{\mathbf{total}}$ (kWh)	$\mathit{WV}$ (h2)	Violation (%)
MK02	MK02 (direct)	48.2 ± 2.0	462.8 ± 12.4	43.5 ± 6.8	4.5 ± 1.9
MK02	MK01 + MK03	50.8 ± 2.7	478.5 ± 15.1	52.3 ± 8.4	6.2 ± 2.4
MK06	MK06 (direct)	131.4 ± 5.3	1968.5 ± 51.3	218.3 ± 28.6	4.2 ± 1.8
MK06	MK01 + MK03	142.6 ± 7.8	2058.4 ± 64.2	265.8 ± 36.5	8.7 ± 3.2
MK10	MK10 (direct)	298.3 ± 9.2	3285.4 ± 78.6	312.5 ± 38.4	4.2 ± 1.8
MK10	MK01 + MK03	318.5 ± 12.4	3465.2 ± 95.8	385.7 ± 48.6	10.5 ± 3.8

reports the results alongside the performance of policies trained directly on each target instance.

On MK02, which has the same scale as the training instance MK01, the transferred policy incurs only a 5.4% increase in makespan and a modest increase in workload variance, demonstrating strong within-scale generalization. On the larger instances MK06 and MK10, the performance gap widens (8.5% and 6.8% makespan increase, respectively), indicating that the policy partially captures transferable scheduling heuristics but cannot fully compensate for the structural differences in machine count and job complexity. The constraint violation rate on MK10 rises to 10.5%, suggesting that Lagrangian multipliers calibrated on smaller instances do not fully transfer. These results indicate that zero-shot transfer provides a reasonable warm start, but fine-tuning on the target instance is recommended for deployment in production environments with substantially different configurations. Future work integrating GNN-based state representations [21,22] or meta-learning frameworks [23] may further improve cross-instance generalization.

6. Conclusions

We have presented ALP-PPO, an adaptive Lagrangian penalty-enhanced proximal policy optimization algorithm for flexible job shop rescheduling under worker workload constraints, machine failures and rush orders. The scheduling problem was formulated as a constrained Markov decision process that integrates worker skill heterogeneity, fatigue and workload equity with classical scheduling objectives. The constraint thresholds of the CMDP were calibrated from occupational health guidelines and practical equity requirements, and this principled basis distinguishes the present formulation from standard MDP-based approaches. By decomposing operation sequencing, machine allocation and worker assignment into coordinated sub-decisions, the hierarchical action space keeps the decision problem tractable. Dual Lagrangian multipliers are updated using a proportional–integral rule, which extends existing PPO-Lagrangian methods by incorporating an integral correction term that addresses the challenge of heterogeneous constraint scales, and manual penalty tuning is thereby eliminated. An event-triggered mechanism guided by a disruption severity index automatically selects between right-shift and full rescheduling.

On extended Brandimarte benchmarks, ALP-PPO outperforms DoDQN, DuDQN, 3DQN, standard PPO, NSGA-II and MOEA/D across multiple objectives, with the largest gains observed in worker workload variance, which is reduced by 35–50% relative to value-based methods. Pareto front quality, as assessed using Hypervolume and Inverted Generational Distance indicators, is found to be superior for ALP-PPO in both convergence and diversity compared with the DRL baselines and multi-objective evolutionary algorithms. Ablation studies show that the adaptive Lagrangian mechanism drives the bulk of the constraint improvement, cutting violation rates by more than 40% versus fixed-penalty alternatives. The hierarchical action space contributes primarily to scheduling quality by reducing the effective search space, and the event-triggered rescheduling mechanism improves computational efficiency by avoiding unnecessary re-optimization for minor disruptions. Sensitivity analysis demonstrates stable parameter ranges, including robustness to variations in the reward weight configuration, and dynamic response experiments validate robustness under varying disruption intensities.

From the perspective of industrial deployment, several practical implications emerge from the results. The online inference time of ALP-PPO stays below 20 ms per decision step, and the complete rescheduling time after a disruption event averages 0.42 s, which is two orders of magnitude faster than the evolutionary baselines and compatible with the cycle times of typical discrete manufacturing lines. The reduction in workload variance directly translates to more equitable operator utilization, which can reduce fatigue-related errors, lower injury risk and improve workforce satisfaction over extended production horizons. The constraint satisfaction rate above 95% ensures that worker fatigue limits are respected throughout the production schedule, supporting compliance with occupational safety standards. These characteristics make ALP-PPO suitable for integration into manufacturing execution systems (MESs) or digital twin platforms as a real-time decision support module for human-centric smart factories.

With respect to generalizability, the cross-instance transfer experiments show that policies trained on smaller instances provide a reasonable warm start for larger configurations, with moderate performance degradation that can be recovered through fine-tuning. The method has been validated on instances from 55 to 240 operations with six to 15 machines and under disruption frequencies from single-event to multi-event concurrent scenarios. Extension to larger industrial-scale instances and more diverse shop floor configurations remains a direction for future investigation.

Future work should extend the static worker skill model to incorporate dynamic skill development and cross-training, and validate the approach on industrial-scale data with real machine failure logs and worker performance records. The hierarchical action space currently employs a sequential decision structure; investigating attention-based sub-action coordination mechanisms or GNN-based state representations [21,22] may further improve policy quality and cross-instance generalization. Integrating transfer learning and meta-learning [23] techniques to enable trained policies to generalize across different shop floor configurations and product mixes would enhance the deployability of ALP-PPO in multi-plant manufacturing networks. Finally, combining the suggested approach with digital twin technology for continuous policy refinement on the basis of real-time production data represents a promising direction for realizing the Industry 5.0 vision of human-centric, resilient and sustainable intelligent manufacturing, where scheduling algorithms not only optimize throughput but also safeguard operator well-being and promote equitable workforce utilization.